Tech Center Current

At the 2007 SIGGRAPH conference, an annual convention for computer graphics specialists to discuss new computer graphics techniques and applications, Shai Avidan and Ariel Shamir of Mitsubishi Electric Research Laboratories demonstrated an innovative new method for resizing the dimensions of images. The new method, called content-aware image resizing, preserves the size and visual quality of the more important regions of the image by sacrificing accuracy in the less important areas. It’s difficult to describe in words alone, so here is a video from the creators illustrating the impressive results:

Classic resizing methods work by removing vertical and/or horizontal columns of pixels at regular intervals as needed to account for the difference between the original dimensions and the desired dimensions. As you shrink the image, you inevitably lose some of the detail. If the image consists of a single person and some unimportant surroundings, you could maximize the detail of the person in the resized image by first cropping out some of the surroundings so they don’t take up the precious pixel real estate in the final image.

But what about when an image consists of a person and a house with some featureless landscape in between? In order to preserve the detail of the person and the house, you would ideally want to remove some of that space between them. Simply cropping out that space and shoving the two sides of the image together would leave a very apparent and ugly division in the middle. There isn’t any good solution using classic resizing and slicing methods.

This is where content-aware image resizing comes in. Rather than just removing entire columns of pixels at regular intervals, this algorithm analyzes the image and determines the least noticeable sets of pixels to remove. It does this by calculating the “energy” of pixels. The energy relates to the relative visual importance of those pixels (in other words, how much the pixels stand out). By removing the pixels with the lowest energy, the image can be resized with minimal change to the most visually significant parts of the image.

There’s one more key aspect of the algorithm: Let’s imagine for a moment what would happen if it simply removed the lowest-energy pixels from each horizontal row. What if the lowest-energy pixel in one row were on one side of a house and the lowest-energy pixel on the next row were on the other side of the house? Suddenly, after removing those pixels, the edges of the house would be noticeably zig-zagged. So in order to also preserve the shapes in the image, the algorithm makes sure that it only removes pixels along continuous paths from one side of the image to the other side. The creators of this technique call these continuous paths “seams”.

This works great for shrinking images, but what about enlarging them? It turns out this algorithm is also well-suited for enlarging images. It follows the same procedure, except instead of removing the lowest-energy seams, it doubles them (with a little mixing to smooth them out). Since the seams are low-energy, doubling them has a minimal visual effect. In other words, the result doesn’t look nearly as blocky or blurry as it would with traditional resizing methods, and the sizes and shapes of the most important parts of the page are preserved.

The above video also illustrates how this concept can be used to convincingly remove whole objects from an image by manually forcing the algorithm to treat the particular object areas as being low-energy and then resizing appropriately.

An obvious limitation of the algorithm used in the above video is that it simply removes whole pixels rather than performing some sort of resampling. As I explained above, the classic resizing technique is to remove columns of pixels at regular intervals. While this is the fastest method, it also introduces significant visual data loss, even compared to other traditional resizing techniques.

Image resampling is a way to resize the image without completely discarding any pixel data, by instead calculating each pixel on the resized image as a combination of pixels on the original image. For example, if you were resizing the image to half its original height and width, each pixel on the resized image would be an equal mixture of four pixels (two by two) from the original image.

The content-aware image resizing algorithm could be modified to incorporate some aspect of this technique, by condensing pixels near the seams rather than stripping out the pixels entirely. This would probably improve the outcome of the last example in the above video, where the current algorithm’s limitations were quite apparent.

For more information, see the PDF. The download may be temporarily unresponsive since it has recently been linked to by several very high-traffic websites.

Update: Following the huge attention this new technique has received, co-inventor Shai Avidan has just been hired by Adobe, presumably to add this technology and other work to the popular Adobe Photoshop image editing software.