From Information to Intelligence

If you have ever watched NCIS, CSI, numb3rs or any modern police related TV show you probably have seen them take a digital photo and enlarge it to reveal details. Because it is a TV show the photo magnification work perfectly and show wonderful details. Well this is fiction and in real world this is not going to append. However over the years, magnification algorithms have made serious progress so I thought it could be interesting to see how everyday softwares handle magnification.

The problem

The problem with photo magnification is the following: the computer has to infer information that does not exist. A photo as you probably already know is made of pixels (elementary points), hence an upscaled photo has more pixels that the original one. The question is which color theses new pixels should have ? Since there were not in the original photo, you have to guess. That exactly what does a magnification algorithm: It tries to guess what the new pixels should look like according to the original image. That is why TV show techniques are fiction: you can’t find information that wasn’t in the original set of pixels.
I have to precise that in this post I am talking about photo magnification, also know as interpolation, not video upscaling. Video interpolation algorithms can in some case combine multiples frames of the video to have a bigger pictures. (I save this for a future post).

How magnification work ?

The basic idea behind every magnification (interpolation) algorithm is to guess the value of the new pixel according to the real pixels that you find around it. To make it more efficient, many techniques have been tried such as bicubic, Lanczos, XinLin, DDL ones . If you want learn how an interpolation algorithm works read this excellent tutorial made by Sean. If you just want the basic idea, take a look at the diagram below taken from this tutorial:

Objectives

There is a bunch of comparison between extrapolation algorithm in research publication. This post is a little different because it has a pragmatic approach of the problem. It seek to answer the following question:

“What result can I expect from my everyday software on a standard photo”

If you want an analysis of high end research algorithms that is all about numbers and variations take a look at this page.

Software tested

I choose three widely used photo software and a research soft wares to compare their results on average photos:

1. Windows paint as baseline
2. Gimp 2.4.4 which is the open source reference
3. Photo shop CS3 which is the commercial reference
4. SAR 3.4 which implement blending edge research algorithm

Methodolgy

In order to make the test the most objective possible, I did the following:

• I took three photos with a resolution equal or above 400×400.
• I downscale them to 200×200.
• I upscale them to 400×400 using the different soft wares.
• I made a visual comparison of each algorithm result for a relevant part of the photo.

Every photos during the process have been saved using the PNG algorithm to ensure that no distortion is introduced by a compression algorithm. Complete results and raw material are provided so you can judge by yourself, do the same, and even made more tests (I you do, I would be happy to publish them). Two of the three photos are human portrait because face are known to be hard to upscale and this is of course a very popular topic Please note that because the test was made on three images, this post cannot be considered as a reference benchmark (The methodology is correct but there is simply not enough data).

The photos

Photo 1 Evangeline Lilly aka Kate

The first photo is a portrait of Evangeline Lilly which play Kate in Lost. Here is the original (400×400) photo:

The resulting downscaled photo (200×200)

Photo 2 Matthew Fox aka Jake

I used a photo of Matthew Fox as the second photo, because unlike Evangeline, he have short hairs which should be easier to upscale. The original photo (400×400):

The resulting downscaled photo (200×200)

Photo 3 Nightwish Dark Passion play cover

As the last photo, I used the cover of the album Dark passion play because I wanted to see how upscaling algorithm handle light diffusion and text. The original photo (400×400):

The resulting downscaled photo (200×200)

The results

For each software, I tried various interpolation algorithms if possible. For the visual comparison I kept, the one that seems to be the better for each software. On the right of the comparison, you have the expected photo which is directly taken from the original file. Of course no algorithm is expected to be that perfect, but it allow to put things in perspective. In the visual comparison, I kept the recommended resize algorithm for Photo shop CS, the most advanced method (sic) for Gimp, and the back projected Jensen DDL for SAR. Complete results of extrapolation algorithm are visible in the photo galleries.

Result for Evangeline

As expected the SAR algorithm is clearly better. What is unexpected it that the paint algorithm is not so bad. You can found all the interpolations results below

Result for Matthew

Here the logic is respected, and there is a clear difference of quality, this is particular visible for the shirt and around the eyes. . You can found all the interpolations results below

Result for Nightwish

Once again SAR is better, and this time the paint algorithm is clearly behind. This is obvious when you look at the word play. . You can found all the interpolations results below

Conclusion

As expected, the quality of the magnification is dependent of the software you use. Photo shop and Gimp are pretty close and SAR is way above. Note that SAR algorithms are way more slower than photoshop and GIMP algorithms because they performs a significant amount of computation. SAR is also much harder to use than the other products. If you dont understand the algorithms, it can give you hard time. If you want to see more comparison, AmericasWonderlands have made a very neat post about it. Feel free to send me your interpolation results so we will have a more significant corpus.