Almost 2 years ago (how time flies) Tsahi had a great post on “why hardware beats software every day of the week”. Although he’s a software guy (me too!), when it comes to hardcore number crunching, as is the case of video processing, it’s hard to beat hardware-based solutions.
Tsahi was mainly dealing with maintaining a software-based client vs. a hardware-based “appliance”, but I have been thinking about it a lot recently, following a great deal of conversations with customers and potential customers about mobile video clients.
Hardware-based vs. Software-based codecs
There are a LOT of mobile clients out there. Some are off-the-shelf clients, tied to video services and over-the-top solutions (Skype, Qik, Fring, Tango, etc.). Some are in the form of SDKs, client frameworks, media engines, codecs. Some are coming from well-known software vendors, with industry track record, while others are based on open-source projects. But the most important distinction between all of these solutions is whether they are software-based or hardware-based, and I will explain.
Most mobile devices today, especially smartphones and tablets, come with a chipset inside them. This chipset, coming from vendors like Qualcomm, NVIDIA, Texas Instruments and others, has a general purpose processor (usually ARM Cortex A8 or A9), which runs the operating system, handles the UI and runs the applications. In fact, if you compare, for instance, two Android devices from different vendors, they would be pretty similar in that sense, and so application developers can develop an application for the Android Market (assuming they deal well with fragmentation).
But these chipsets usually come as system-on-chip solutions, where there are other co-processors, and usually one of them will be a hardware-based accelerator for video coding or an entire video codec in hardware, usually for H.264/AVC. This means that you can encode and decode using a different processor, without “burdening” the main processor.
Using the hardware-based codec has a lot of significant advantages:
- Quality of Experience: Hardware-based codecs today support 720p high definition video, even on relatively “weak” devices, while software-based codecs usually support only CIF (and not even in 30fps).
- CPU: Running video on a hardware-based codec means that the CPU is free to do other things, which is very important in a real multi-tasking OS such as Android (no offense, Apple!). Running video on the main CPU means that practically you can’t do anything during a call, not even run tasks in the background.
- Battery Life: With video running on a hardware-based codec the battery utilization is optimized, which means you get more battery life. Running video on the main CPU will drain the battery quite severely (example), which is really not a very good experience.
Of course all of this comes with a few disadvantages, not for the end-user but the developer and/or service provider:
- Portability: Hardware-based codecs are different between vendors and even between chipsets from the same vendor. This means you don’t get that “plug-and-play” portability as in software, and you really need to know how to write your media engine to make porting feasible.
- Interoperability: Hardware-based codecs are… well… hardware-based codecs. Which means you can’t really fix issues like interoperability, as you can with software. Here, again, you need to know what you’re getting and how to handle it, and it takes a lot of know-how in managing interoperability to properly handle.
HD video on the tablet – good. SD video on the tablet – bad.
(sources: Crackberry, MattMacintosh)
It’s The User Experience, Stupid!
But the bottom line is pretty clear, and it has all to do with user experience and nothing to do with things like interoperability or portability — At the end of the day the user wants a high quality video call on the latest and greatest device he just stood in line for hours to purchase. If I have a 10” tablet with a high quality camera, I want to see high definition video, I want a 30fps experience and I want to feel I’m getting a superior experience. You can’t have any of that with software, with all due respect.
So whenever I’m asked why we are focusing on hardware-based client frameworks, although we are a software company, I always tell people “because we don’t focus on software, we focus on user experience. And if hardware beats software any day of the week, user experience beats everything else (including software). Ask Steve.
The issue of hardware vs software has long been with us. I certainly agree that hardware acceleration of standards-based things like video encode/decode are key to user experience. Too often ambitious software companies think that they can deliver a similar user experience leveraging only the CPU. Often that approach fails to consistently deliver the goods.
Stepping away from the mobile space for a moment, there’s a middle ground that may be appropriate for fixed installations. That is, soft programmable hardware…like FPGAs… can deliver the performance of real hardware but in a manner that’s extensible through firmware updates.
My employer has leveraged this for many years. The key advantage is that the life cycle of our products is extended beyond those of our competitors, whose hardware platforms age much more quickly. Their hardware systems drop from the scope of their software development efforts often years before we cease development on our comparable systems.
Such programmability of hardware in the field also lets you adapt your product to deal with ever-evolving standards for media