WANTED: Multi-core for video communication clients

Danny Loeb provided his insights on the effect of multicore on SIP servers, but what about video communication clients? Up until today, there is a tendency to either push multi-core hardware as the next best thing (mainly by the relevant chipset vendors) or to dismiss it for most uses. While that has been true up until today, it’s popularity is changing - the latest evidence being Skype version 4.0 beta release, which is video-centric.

There is a multitude of other indications that video communications is becoming the “next big thing,”  both in the consumer market and for enterprises. I get a lot more requests from customers for video applications and most concern high definition. This move to high definition has made multi-core platforms essential - there’s no way around it (unless you are willing to settle for crappy video).

Why is multi-core that important? The answer is that there’s no other way to encode high definition video; at least not if you are planning something that is cost effective.

Ariel already gave his views on the bandwidth requirements of high definition at Video over Enterprise. There are also computing requirements that need to be taken into account: assuming we want to encode 720p at 30 frames per second, this would require to handle 1280×720 pixels per frame which gives us 27,648,000 pixels each second to process. That’s over 27 MILLION pixels to encode and another 27 MILLION pixels to decode. High definition is usually done using the H.264 standard (in order to lower the bandwidth required while keeping high quality) and H.264 is quite complex to compute.

Modern Video encoding techniques take each frame,split it into smaller blocks called macroblocks - usually in sizes of 8×8 - and then analyze and encode each such macroblock on its own. This allows for splitting an image in order to perform parallel processing. In other words, multi-core processors can be utilized in a fairly simple way: you split the pictures into regions and encode the regions in parallel. This is not trivial because sometimes the actual processing of a macroblock is dependent on previous images or its neighbor macroblocks (especially for motion estimation algorithms), but other than that, it is quite straightforward.

 
A frame split into 4 areas, each being processed by a different core processor

In today’s x86 architecture there is no way one can encode a high definition video stream in real-time - at least not if you want to maintain video quality and allow for 30 frames per second. Chip architectures move towards adding additional processing cores instead of increasing clock speeds. But programmers are still not embracing these changes in their applications. Jeff Atwood even points out the diminishing returns of using multicore. A video conferencing client, supporting high definition, really needs these multicores - there’s no way around it today.

Developing a video telephony client that is going to do high definition? You will probably use multicore technology in the underlying hardware sooner or later.