Chapter 2
Feature Tracking
As the camera moves, the patterns of image intensities change in a complex way. In general, any function of three
variables I(x, y, t), where the space variables x and y as well as the time variable t are discrete and suitably bounded,
can represent an image sequence. However, images taken at near time instants are usually strongly related to each
other, because they refer to the same scene taken from only slightly different viewpoints.
We usually express this correlation by saying that there are patterns that move in an image stream. Formally, this
means that the function I(x, y, t) is not arbitrary, but satisfies the following property:
I(x, y, t + τ ) = I(x − ξ, y − η, t) ; (2.1)
in plain English, a later image taken at time t + τ can be obtained by moving every point in the current image, taken
at time t, by a suitable amount. The amount of motion d = (ξ, η) is called the displacement of the point at x = (x, y)
between time instants t and t + τ , and is in general a function of x, y, t, and τ.
Even in a static environment under a constant lighting, the property described by equation (2.1) is violated in many
situations. For instance, at occluding boundaries, points do not just move within the image, but appear and disappear.
Furthermore, the photometric appearance of a region on a visible surface changes when reflectivity is a function of the
viewpoint.
However, the invariant (2.1) is by and large satisfied at surface markings, and away from occluding contours. At
locations where the image intensity changes abruptly with x and y, the point of change remains well defined even in
spite of small variations of overall brightness around it.
Surface markings abound in natural scenes, and are not infrequent in man-made environments. In our experiments,
we found that markings are often sufficient to obtain both good motion estimates and relatively dense shape results.
As a consequence, this report is essentially concerned with surface markings.
The Approach
An important problem in finding the displacement d of a point from one frame to the next is that a single pixel cannot
be tracked, unless it has a very distinctive brightness with respect to all of its neighbors. In fact, the value of the pixel
can both change due to noise, and be confused with adjacent pixels. As a consequence, it is often hard or impossible
to determine where the pixel went in the subsequent frame, based only on local information.
Because of these problems, we do not track single pixels, but windows of pixels, and we look for windows that
contain sufficient texture. In chapter 4, we give a definition of what sufficient texture is for reliable feature tracking.
Unfortunately, different points within a window may behave differently. The corresponding three-dimensional
surface may be very slanted, and the intensity pattern in it can become warped from one frame to the next. Or the
window may be along an occluding boundary, so that points move at different velocities, and may even disappear or
appear anew.
This is a problem in two ways. First, how do we know that we are following the same window, if its contents change
over time? Second, if we measure ”the” displacement of the window, how are the different velocities combined to give
2
