The Missing Hardware For Next-Level Phone Photography

Now that you understand what computer vision and computational photography are, you must have figured out that both might require substantial amounts of energy when highest quality is the goal. Think about it: the suite of imaging algorithms that are put in motion just by opening the camera and looking for a subject can be insane! So, should we wait for better battery technologies? Or is a specialized processor necessary? To answer these questions, we must understand how we got here in the first place.

For instance, we can’t imagine a camera device today without a GPU. This piece of hardware wasn’t always seen as indispensable. In fact, it only came into existence when loads of gamers felt the need for an arcade games experience at home.

The CPU could provide that experience all by itself, saving companies the trouble of drafting new objectives and postponing corporate plans. But the CPU was already a veritable workhorse.

To understand the pressure engineers put on its shoulders, just imagine a day at the workplace. You’re multitasking from dawn to dusk, trying to cover as much ground as you can, knowing fully that you should delegate some work to avoid burnout. But there’s no one around, so you keep doing it yourself… until the day your manager hires help.

That help, that guy who does one thing really well, came to be known as the GPU.

So, what made NVIDIA launch the “world’s first GPU”? The company knew that it would win on the long term because the market had a real need – gamers wanted high quality graphics on their PCs and the GPU paved the way.

FOUR THINGS THAT ARE HOLDING BACK SMARTPHONE CAMERAS

With this idea in mind, we believe that the necessary technology to make smartphone photography truly incredible exists. So, why do we feel like something’s missing? I’m not talking here about the three big vulnerabilities of smartphones in their race to achieve DSLR quality – zoom, low light, bokeh – but about the overall perceived quality of a picture.

Why aren’t manufacturers using the best camera software in their smartphones? Well, if they were to take the most complex imaging algorithms and run them to the fullest extent of their abilities, there would be a price for it – increase bandwidth requirements, huge power consumption, compromised flexibility and lack of scalability.

The increase in requirements for larger resolutions and larger frame rates has a direct impact on image analytics and enhancements performance while running on embedded platforms. It would appear there’s no easy way out of this conundrum – platforms’ limitations are forcing an approach that affects directly the quality and the performance of all camera related applications and core technologies.

But what if we went back and learned from history? Video games were the experience trigger, now it is photography. The desire to play arcade games at home is similar to the one of bringing DSLR quality on a smartphone. So, if GPU was the way of making those things possible then, could a specialized processor for imaging be the path, now?

Well, to differentiate their smartphone camera from another’s, manufacturers need to be able to unleash the full potential of their camera’s features. I reckon this can happen only with a dedicated hardware aboard, a piece that can keep the power consumption and bandwidth in check, while enabling the best user experience.

THE ANSWER: A GPU FOR IMAGE PROCESSING

It’s not just me saying that. Companies are getting the same feeling, too – that they need another piece of the puzzle. Qualcomm has publicly talked about its belief that a dedicated processor is more efficient at running a specific task than a general purpose processor. Heterogeneous computing is their attempt at it, trying to save power by delegating tasks to different processors.

Apple understood the importance of human faces in mobile photography so in iPhone 6 and iPhone 6 Plus it added a dedicated hardware within the A8 processor to accelerate the face detection algorithm.

For Phantom 2, DJI used the Movidius Myriad 2 chip to optimize particular computer vision algorithms in order to enhance the DJI drones’ capabilities to avoid objects.

In retrospective, these efforts could be similar to the ones Atari and Fujitsu did in the 70’s. While those were respectable ideas, they didn’t turn into the GPU even if they solved parts of the problem. What we need is a well-thought GPU for image processing, an IPU (Image Processing Unit) if you’d like, that does more than just execute computational photography, computer vision and machine learning tasks, compromising power, heat and bandwidth requirements; we need a piece of hardware that handles these tasks efficiently, with minimal loss.

Let’s say that our devices – smartphones, AR glasses – would have an embedded IPU now.  In what way it would improve our user experience, you ask. In what way it wouldn’t?! An IPU could run parallel workloads with minimum power.  With five dedicated engines, like this one from FotoNation, it would do on the fly processing and analytics on small input resolution images. The hybrid approach (software-hardware) means it could detect and track faces, people, cars, bikes, objects and employ HDR, all simultaneously.

For videos, it could do tone mapping, stabilization and wide field of view corrections. It would ensure the best performance possible and would be cost effective in the long term since it would guarantee full programmability and scalability based on future resolutions and frame rates.

Now imagine enabling machine learning at the core of image understanding and enhancement, by using a dedicated engine as part of the IPU. This would allow dedicated CNN implementations with an order of magnitude faster than classic GPU / DSP implementations.

When wearing an AR headset the barrier between real and digital would fade away. Just imagine how technology could personalize your device at pixel level, learning and adapting to you and your preferences. It would learn what, why and how to track, for what and how to enhance it, in ways that would be derived from your personal taste. Beauty might be subjective, but with the help of technology, your devices could also understand it.

If you want to read more about the future of our world with IPU around, don’t miss our next interview with Simon Fitzpatrick, Senior Director of Product Management at FotoNation. He will discuss that particular subject but also, give us a glimpse on how we will interact with smart devices in 2027.