Image Synthesis Considerations for Image Refocusing
Thomas Brow
TEBROW@STANFORD.EDU
Abstract
Image refocusing from a single image is an
underdetermined problem. Out-of-focus
images of a scene do not capture high-
frequency details that should appear in a
focused image. In this paper, we describe a
system for incorporating additional, focused
images into the refocusing task in order to
inform high-frequency detail. We use the
additional inputs to determine a mapping
from low-frequency to high-frequency
features, and also to define a space of
natural images to which we constrain our
output. We find that the synthesized image
accurately reproduces high-frequency
textures from the scene when the focused
inputs capture the scene from the same
view, but achieves mixed results otherwise.
1. Introduction
Given an image in which some object appears out of
focus, we would like to synthesize an image in which
that object appears in focus. Since the image in a
lens camera of an object at a given depth is the
convolution of a pinhole image of that object by the
point spread function (PSF) of the lens, a popular
method for recovering the sharp image is
deconvolution by the PSF. In deconvolution, blur is
modeled by the equation
!
f * g = h
,
(1)
where f is the sharp image and g is the PSF. The
output image is some solution for f given g and h.
However, the modulation transfer function (the
magnitude of the Fourier transform of the PSF) of a
typical photography lens falls to zero at a
significantly lower frequency when the image is not
in focus than when it is in focus [1]. Thus a band of
spatial frequencies that should appear in a sharp
image of the object is not present in the input image.
Refocusing from a single image is therefore an
underdetermined problem, and as a result,
deconvolution algorithms are diverse and highly
tailored to their respective applications.
We present a system that incorporates additional,
focused images of the target object in order to inform
the high-frequency detail of the synthesized image.
Following [2] and [3], we use the additional inputs to
define a space of natural focused images to which we
constrain our synthesized image. As in those papers,
we implement this constraint as a library of local
patches appearing in the input images, requiring that
every patch in the synthesized image resembles some
patch in the library.
Our system further differs from deconvolution in that
we do not generate candidates for the synthesized
image by finding global solutions for (1). Instead, we
find for each local patch in the unfocused image a set
of patches from our focused patch library that are
likely to resemble that patch when convolved by the
PSF. This is analogous to the local approach toward
super-resolution in [3], in which the set of high-
resolution patches which downsample to a given low-
resolution patch in the input image are considered.
From this set of focused patches we derive a list of
candidate values and corresponding probabilities for
the center pixel, which can then be selected from
according to the natural image constraint.
The primary advantage of our refocusing method
over deconvolution methods is that it does not
assume a particular strategy for recovering the lost
high-frequency detail from the misfocused image, but
instead lets a probabilistic mapping from low-
frequency features to high-frequency features be
specified by the additional inputs. Our method
should thus retain greater generality across
applications than any one deconvolution algorithm.
In section 2, we describe the components of our
method in detail. In section 3, we compare the
results produced by different variations of our
system. In section 4, we evaluate the results of our
best system. In section 5, we conclude the paper and
suggest future work.
2. System
In the previous section, we described two
probabilistic constraints on the synthesized image:
our mapping of blurred to sharp neighborhoods, and
our natural image constraint. Ideally, we would like
to output the image which best satisfies the
constraints across the entire image. In practice, we
reduce the dimensionality of our search space
considerably by instead considering pixels
