GoPro SuperView-like adaptive aspect ratio

For Christmas, my parents got me a fantastic gift for photographers and outdoors enthusiasts, a GoPro Hero3+ Silver Edition digital camera (Amazon). If you are not familiar with GoPros, they are small action cameras that have a very wide-angle lens and come with a water-resistant case. It’s a fantastic little camera that packs a lot of punch for such a compact package.

There are a number of GoPro editions, but the newest one are the Hero3+ Silver and Black. The Silver Edition is very similar to the GoPro Hero3+ Black Edition (Amazon), but omits a few features, including some very high-resolution video recording modes and a feature that GoPro calls “Superview”. This post talks about a way that I attempted to emulate their Superview mode in MATLAB and put together an adaptive aspect ratio function that allows one to change the image’s aspect ratio while maintaining “safe regions” with minimal distortion.

This function allowed me to resize 4:3 images and video, like this one:

To a wider aspect ratio, for example 16:9:

Click through for info on how I implemented the code.

Let’s say we have an original, 4:3 image or frame from a camera (GoPro in this instance):

If we want to convert it to 4:3 format, we have a couple of options. The simplest is cropping out the top and bottom of the image:

The issue with cropping is that it cuts down on the vertical field of view. We could alternatively scale the image in the horizontal direction (or compress in the vertical, the effect is the same):

As you can see, scaling causes distortion in the image that can be distracting, especially when there are known points of reference (like people) in the frame.

An alternative to cropping or linear scaling, is the GoPro Hero3+ Black’s Superview mode, which is intended to non-linearly scale the image to retain as much vertical image information while attempting to minimize the perceived scaling distortion, especially in the center of the image frame.

Here is a description of the GoPro Superview feature, according to GoPro:

SuperView is a new feature introduced with HERO3+ Black Edition which allows you to capture an immersive wide angle perspective. What this mode does is it takes a 4:3 aspect ratio and dynamically stretches it to a 16:9 aspect ratio. This can be a great choice because it uses the height of the camera’s sensor that you get with 4:3 meaning that you will see more of the sky and ground assuming you are pointed at the horizon. … The way it works is that the camera automatically stretches out the sides of the video to fit into the 16:9 frame. The center of the frame is unchanged, only the edges are adjusted.

This sounds as though there is 1:1 mapping of pixels in the center of the frame with gradual distortion/interpolation towards the right and left edges. This would maintain the total field of view and information content in the image, while attempting to conform to a widescreen aspect ratio.

Abe Kislevitz has a nice discussion of 4:3 to 16:9 footage conversion with the GoPro and specifically discusses a plugin called Elastic Aspect. I chose to do a similar transformation in MATLAB to see if I could reproduce the Superview effect in “post production” (after an image or video has been captured).

To do this, I created a 1-Dimensional mapping of the input pixels to the output pixels along the y-axis and remapped the locations of pixels that were not linearly mapped to “Safe Zones’ using cubic spline interpolation.

Let’s say I want to stretch GoPro image of my wife and I from 4:3 to 16:9 widescreen aspect ratio. Here’s the original 4:3 image:

Using linear interpolation, this would be the expected mapping (and resulting image). Note the distortion in our faces:


Created using the following code (from the below function):

[code lang=matlab]
outImage = bSplineResizeWidth([1080,1920], inImage, [50 50]);
[/code]

With cubic spline mapping with a very small “Safe Zone” that is linearly mapped (5% of the image width), we would have this mapping and resulting image:


Created using the following code (from the below function):

[code lang=matlab]
outImage = bSplineResizeWidth([1080,1920], inImage, [47.5 52.5]);
[/code]

Notice the difference in perceived quality already? Now, let’s protect the center 25% of the image:


Created using the following code (from the below function):

[code lang=matlab]
outImage = bSplineResizeWidth([1080,1920], inImage, [37.5,62.5]);
[/code]

Looking good! Let’s say our subject is off to the side of the image, we can explicitly choose those regions to “protect” by ensuring a 1:1 mapping to avoid any distortion for our subject. For example, here’s a picture of my cousin’s husband, located slightly off-center:

I can explicitly select a “Safe Zone” around his body to avoid distorting him further. Here’s the remapping and resulting image:


Created using the following code (from the below function):

[code lang=matlab]
outImage = bSplineResizeWidth([1080,1920], inImage, [], 1);
[/code]

So, although there is still inevitable distortion around the outside of the image, it seems as this may be an effective way to emulate GoPro’s Superview mode and perform an adaptive aspect ratio adjustment.

For the sake of discussion, there is another method called seam carving (commonly known as content-aware scaling with Adobe Photoshop), but this can introduce some artifacts, depending on the scene. It is also quite slow. Here is an example implementation in MATLAB by Danny Luong: Seam Carving for content aware image resizing: GUI implementation demo. In general, it appears as though seam carving is better suited for images rather that videos (note that the original authors of the algorithm developed a similar method for processing video):

Here is the code for both procedures: