■ Sharpening by amplifying existing image details.
This is the change in the spatial frequency amplitude
spectrum of an image associated with image sharp-
ening. Existing high frequencies in the image are
amplified. This is often useful to do, provided noise
isn’t amplified.
■ Aggregating from multiple frames. Extracting a single
high-resolution frame from a sequence of low-
resolution video images adds value and is sometimes
referred to as super-resolution.
■ Single-frame super-resolution. The goal of this arti-
cle is to estimate missing high-resolution detail that
isn’t present in the original image, and which we can’t
make visible by simple sharpening.
We feel researchers should use each method wherever
applicable, but in this article, we focus on single-frame
super-resolution. Although integrating resolution
information over multiple frames is sometimes called
super-resolution, for the purposes of this article, super-
resolution will refer to the single-frame enlargement
problem.
Because the richness of real-world images is difficult
to capture analytically, for the past several years, we’ve
been exploring a learning-based approach for enlarg-
ing images.
1-3
In a training set, the algorithm learns the
fine details that correspond to different image regions
seen at a low-resolution and then uses those learned
relationships to predict fine details in other images.
(Recently, Hertzmann et. al
4
also used a training-based
method to perform super-resolution, in the context of
analogies between images. Baker and Kanade
5
focused
on enlarging images of a known model class—for exam-
ple, faces. Liu et. al
6
built on their and our work.)
To understand why this approach should work at all,
IEEE Computer Graphics and Applications 57
2 (a) An image (100 × 100 pixels)
of a real-world teapot shows a
richness of texture but yields a
blocky or blurred image when we
enlarge it by a factor of 8 in each
dimension by (b, c) pixel replication
or (d, e) cubic-spline interpolation.
((b) through (i) were 32 × 32 pixel
original subimages, zoomed by 8 to
256 × 256 images). Sharpening the
cubic-spline interpolation might not
help to increase the perceptual
sharpness; we used the “sharpen
more” option in Adobe Photoshop
(f, g). (h, i) The results of our one-
pass super-resolution algorithm,
maintaining edge and line sharp-
ness as well as inventing plausible
texture details.
(a)
(b) (c)
(d) (e)
(f) (g)
(h) (i)
Related Approaches
The cubic spline
1
is a common image
interpolation function, but it suffers from blurring
edges and image details. Recent attempts to
improve on cubic-spline interpolation
2-4
have met
with limited success. Schreiber
3
proposed a
sharpened Gaussian interpolator function to
minimize information spillover between pixels
and optimize flatness in smooth areas. Schultz
and Stevenson
5
used a Bayesian method for
super-resolution, but it hypothesizes rather than
learns the prior probability.
These analytic approaches often suffer from
perceived loss of detail in textured regions. A
proprietary, undisclosed algorithm, Altamira
Genuine Fractals 2.0 (an Adobe Photoshop
plug-in, http://www.altamira.com), does as well
as any of the nontraining-based methods, but
can cause blurring in texture regions and at fine
lines. Recently, image interpolation-based level-
set methods
6
have shown excellent results for
edges.
References
1. R. Keys, “Cubic Convolution Interpolation for Digital
Image Processing,” IEEE Trans. Acoustics, Speech, Sig-
nal Processing, vol. 29, no. 6, 1981, pp. 1153-1160.
2. F. Fekri, R.M. Mersereau, and R.W. Schafer, “A Gen-
eralized Interpolative Vq Method for Jointly Optimal
Quantization and Interpolation of Images,” Proc. Int’l
Conf. Acoustics, Speech, and Signal Processing
(ICASSP), vol. 5, IEEE Press, Piscataway, N.J., 1998,
pp. 2657-2660.
3. W.F. Schreiber, Fundamentals of Electronic Imaging
Systems, Springer-Verlag, New York, 1986.
4. S. Thurnhofer and S. Mitra, “Edge-Enhanced Image
Zooming,” Optical Engineering, vol. 35, no. 7, July
1996, pp. 1862-1870.
5. R.R. Schultz and R.L. Stevenson, “A Bayesian
Approach to Image Expansion for Improved Defin-
ition,” IEEE Trans. Image Processing, vol. 3, no. 3, May
1994, pp. 233-242.
6. B. Morse and D. Schwartzwald, “Image Magnifica-
tion Using Levelset Reconstruction,” Proc. Interna-
tional Conf. Computer Vision (ICCV), IEEE CS Press,
Los Alamitos, Calif., 2001, pp. 333-341.