1
Contour Detection and
Hierarchical Image Segmentation
Pablo Arbel
´
aez, Member, IEEE, Michael Maire, Member, IEEE,
Charless Fowlkes, Member, IEEE, and Jitendra Malik, Fellow, IEEE.
Abstract—This paper investigates two fundamental problems in computer vision: contour detection and image segmentation. We
present state-of-the-art algorithms for both of these tasks. Our contour detector combines multiple local cues into a globalization
framework based on spectral clustering. Our segmentation algorithm consists of generic machinery for transforming the output of
any contour detector into a hierarchical region tree. In this manner, we reduce the problem of image segmentation to that of contour
detection. Extensive experimental evaluation demonstrates that both our contour detection and segmentation methods significantly
outperform competing algorithms. The automatically generated hierarchical segmentations can be interactively refined by user-
specified annotations. Computation at multiple image resolutions provides a means of coupling our system to recognition applications.
F
1 INTRODUCTION
This paper presents a unified approach to contour de-
tection and image segmentation. Contributions include:
• A high performance contour detector, combining
local and global image information.
• A method to transform any contour signal into a hi-
erarchy of regions while preserving contour quality.
• Extensive quantitative evaluation and the release of
a new annotated dataset.
Figures 1 and 2 summarize our main results. The
two Figures represent the evaluation of multiple con-
tour detection (Figure 1) and image segmentation (Fig-
ure 2) algorithms on the Berkeley Segmentation Dataset
(BSDS300) [1], using the precision-recall framework in-
troduced in [2]. This benchmark operates by compar-
ing machine generated contours to human ground-truth
data (Figure 3) and allows evaluation of segmentations
in the same framework by regarding region boundaries
as contours.
Especially noteworthy in Figure 1 is the contour de-
tector gP b, which compares favorably with other leading
techniques, providing equal or better precision for most
choices of recall. In Figure 2, gP b-owt-ucm provides
universally better performance than alternative segmen-
tation algorithms. We introduced the gP b and gP b-owt-
ucm algorithms in [3] and [4], respectively. This paper
offers comprehensive versions of these algorithms, mo-
tivation behind their design, and additional experiments
which support our basic claims.
We begin with a review of the extensive literature on
contour detection and image segmentation in Section 2.
• P. Arbel´aez and J. Malik are with the Department of Electrical Engineering
and Computer Science, University of California at Berkeley, Berkeley, CA
94720. E-mail: {arbelaez,malik}@eecs.berkeley.edu
• M. Maire is with the Department of Electrical Engineering, California
Institute of Technology, Pasadena, CA 91125. E-mail: mmaire@caltech.edu
• C. Fowlkes is with the Department of Computer Science, University of
California at Irvine, Irvine, CA 92697. E-mail: fowlkes@ics.uci.edu
Section 3 covers the development of the gP b contour
detector. We couple multiscale local brightness, color,
and texture cues to a powerful globalization framework
using spectral clustering. The local cues, computed by
applying oriented gradient operators at every location
in the image, define an affinity matrix representing the
similarity between pixels. From this matrix, we derive
a generalized eigenproblem and solve for a fixed num-
ber of eigenvectors which encode contour information.
Using a classifier to recombine this signal with the local
cues, we obtain a large improvement over alternative
globalization schemes built on top of similar cues.
To produce high-quality image segmentations, we link
this contour detector with a generic grouping algorithm
described in Section 4 and consisting of two steps. First,
we introduce a new image transformation called the
Oriented Watershed Transform for constructing a set of
initial regions from an oriented contour signal. Second,
using an agglomerative clustering procedure, we form
these regions into a hierarchy which can be represented
by an Ultrametric Contour Map, the real-valued image
obtained by weighting each boundary by its scale of
disappearance. We provide experiments on the BSDS300
as well as the BSDS500, a superset newly released here.
Although the precision-recall framework [2] has found
widespread use for evaluating contour detectors, con-
siderable effort has also gone into developing metrics
to directly measure the quality of regions produced by
segmentation algorithms. Noteworthy examples include
the Probabilistic Rand Index, introduced in this context
by [5], the Variation of Information [6], [7], and the
Segmentation Covering criteria used in the PASCAL
challenge [8]. We consider all of these metrics and
demonstrate that gP b-owt-ucm delivers an across-the-
board improvement over existing algorithms.
Sections 5 and 6 explore ways of connecting our
purely bottom-up contour and segmentation machinery
