Multi-image or multi-frame super resolution (SR) is a technique used to restore detail, while upscaling low resolution images or video frames. Although SR can be applied to native low resolution images or video frames, here we will focus on the case where a high resolution source has been downscaled to a low resolution, as is the case for commercial DVDs.
We will start with an example of a single image. When you downscale a high resolution image to a lower resolution you lose information. Each pixel in the low resolution image becomes a weighted average of a number of frames in the high resolution image. When we upscale the low resolution image to the resolution of the high resolution image the lost information does not magically reappear. The result is a blurry image that lacks detail, as can be seen in the following example. Here the resolution has first been reduced by a factor of two, and than upscaled to the original size.
http://screenshotcomparison.com/comparison/125890
The information about the stripes on the girl's pants and scarf has obviously been lost as a consequence of the downscale. However, if we have multiple images with subpixel shifts much of this information can be retrieved. This effect can be simulated by shifting the high resolution image in multiple directions by one pixel. Each of these shifted images is then downscaled and upscaled in the same way as before. If we then align, average, and deblur these shifted images, we obtain the following result:
http://screenshotcomparison.com/comparison/125895
As you can see much of the original detail has been retrieved.
We have seen that it is possible to obtain a high resolution image by combining multiple low resolution images. The same principle can be applied to video. Video can be viewed as a set of shifted images. By aligning very similar objects in multiple frames, averaging the pixel information, and deblurring, the resolution of a video can be increased. In practise this is of course quite complicated, but the basic concept is the same as the above example for a single image.
An example using the bluray for Star Wars can be seen in the following screenshot comparisons. As before the video has first been downscaled by a factor of two, and then upscaled using SR.
Simple upscale versus SR:
http://screenshotcomparison.com/comparison/125527
The methods used for image and video super resolution have been extensively described in scientific literature. Just type in (video) super resolution in Google Scholar, if you're interested. Commercial software is also available at a reasonable price. As a Virtualdub and Avisynth user I have found that for video upscaling the best performing option at this point is the Infognition software (standalone, Virtualdub plugin, Avisynth plugin). Why don't you try it out?
We will start with an example of a single image. When you downscale a high resolution image to a lower resolution you lose information. Each pixel in the low resolution image becomes a weighted average of a number of frames in the high resolution image. When we upscale the low resolution image to the resolution of the high resolution image the lost information does not magically reappear. The result is a blurry image that lacks detail, as can be seen in the following example. Here the resolution has first been reduced by a factor of two, and than upscaled to the original size.
http://screenshotcomparison.com/comparison/125890
The information about the stripes on the girl's pants and scarf has obviously been lost as a consequence of the downscale. However, if we have multiple images with subpixel shifts much of this information can be retrieved. This effect can be simulated by shifting the high resolution image in multiple directions by one pixel. Each of these shifted images is then downscaled and upscaled in the same way as before. If we then align, average, and deblur these shifted images, we obtain the following result:
http://screenshotcomparison.com/comparison/125895
As you can see much of the original detail has been retrieved.
We have seen that it is possible to obtain a high resolution image by combining multiple low resolution images. The same principle can be applied to video. Video can be viewed as a set of shifted images. By aligning very similar objects in multiple frames, averaging the pixel information, and deblurring, the resolution of a video can be increased. In practise this is of course quite complicated, but the basic concept is the same as the above example for a single image.
An example using the bluray for Star Wars can be seen in the following screenshot comparisons. As before the video has first been downscaled by a factor of two, and then upscaled using SR.
Simple upscale versus SR:
http://screenshotcomparison.com/comparison/125527
The methods used for image and video super resolution have been extensively described in scientific literature. Just type in (video) super resolution in Google Scholar, if you're interested. Commercial software is also available at a reasonable price. As a Virtualdub and Avisynth user I have found that for video upscaling the best performing option at this point is the Infognition software (standalone, Virtualdub plugin, Avisynth plugin). Why don't you try it out?
