1
Bilinear CNNs for Fine-grained Visual
Recognition
Tsung-Yu Lin Aruni RoyChowdhury Subhransu Maji
Abstract—We present a simple and effective architecture for fine-grained visual recognition called Bilinear Convolutional Neural
Networks (B-CNNs). These networks represent an image as a pooled outer product of features derived from two CNNs and capture
localized feature interactions in a translationally invariant manner. B-CNNs belong to the class of orderless texture representations but
unlike prior work they can be trained in an end-to-end manner. Our most accurate model obtains 84.1%, 79.4%, 86.9% and 91.3%
per-image accuracy on the Caltech-UCSD birds [67], NABirds [64], FGVC aircraft [42], and Stanford cars [33] dataset respectively and
runs at 30 frames-per-second on a NVIDIA Titan X GPU. We then present a systematic analysis of these networks and show that (1)
the bilinear features are highly redundant and can be reduced by an order of magnitude in size without significant loss in accuracy, (2)
are also effective for other image classification tasks such as texture and scene recognition, and (3) can be trained from scratch on the
ImageNet dataset offering consistent improvements over the baseline architecture. Finally, we present visualizations of these models
on various datasets using top activations of neural units and gradient-based inversion techniques. The source code for the complete
system is available at http://vis-www.cs.umass.edu/bcnn.
Index Terms—Fine-grained recognition, Texture representations, Second-order pooling, Bilinear models, Convolutional networks
F
1 INTRODUCTION
F
INE-GRAINED recognition involves classification of in-
stances within a subordinate category. Examples in-
clude recognition of species of birds, models of cars, or
breeds of dogs. These tasks often require recognition of
highly localized attributes of objects while being invariant
to their pose and location in the image. For example, dis-
tinguishing a “California gull” from a “Ringed-bill gull”
requires the recognition of patterns on their bill, or subtle
color differences of their feathers [1]. There are two broad
classes of techniques that are effective for these tasks. Part-
based models construct representations by localizing parts
and extracting features conditioned on their detected lo-
cations. This makes subsequent reasoning about appear-
ance easier since the variations due to location, pose, and
viewpoint changes are factored out. Holistic models on the
other hand construct a representation of the entire image
directly. These include classical image representations, such
as Bag-of-Visual-Words [12] and their variants popularized
for texture analysis. Most modern approaches are based
on representations extracted using Convolutional Neural
Networks (CNNs) pre-trained on the ImageNet dataset [54].
While part-based models based on CNNs are more accurate,
they require part annotations during training. This makes
them less applicable in domains where such annotations are
difficult or expensive to obtain, including categories without
a clearly defined set of parts such as textures and scenes.
In this paper we argue that the effectiveness of part-
based reasoning is due to their invariance to position and
pose of the object. Texture representations are translationally
invariant by design as they are based on aggregation of
local image features in an orderless manner. While classical
• T.-Y. Lin, A. RoyChowdhury, and S. Maji are with the College of Informa-
tion and Computer Sciences, University of Massachusetts Amherst, USA.
E-mails: {tsungyulin, arunirc, smaji}@cs.umass.edu
texture representations based on SIFT [40] and their recent
extensions based on CNNs [11], [24], have been shown to be
effective at fine-grained recognition, they have not matched
the performance of part-based approaches. A potential rea-
son for this gap is that the underlying features in texture
representations are not learned in an end-to-end manner
and are likely to be suboptimal for the recognition task.
We present Bilinear CNNs (B-CNNs) that address several
drawbacks of existing deep texture representations. Our key
insight is that several widely-used texture representations
can be written as a pooled outer product of two suitably
designed features. When these features are based on CNNs
the resulting architecture consists of standard CNN units
for feature extraction, followed by a specially designed
bilinear layer and a pooling layer. The output is a fixed
high-dimensional representation which can be combined
with a fully-connected layer to predict class labels. The
simplest bilinear layer is one where two identical features
are combined with an outer product. This is closely re-
lated to the Second-Order Pooling approach of Carreira et
al. [8] popularized for semantic image segmentation. We
also show that other texture representations can be written
as B-CNNs once suitable non-linearities are applied to the
underlying features. This results in a family of layers which
can be plugged into existing CNNs for end-to-end training
on large datasets, or domain-specific fine-tuning for transfer
learning. B-CNNs outperform existing models, including
those trained with part-level supervision, on a variety of
fine-grained recognition datasets. Moreover, these models
are fairly efficient. Our most accurate model implemented
in MatConvNet [66] runs at 30 frames-per-second on a
NVIDIA Titan X GPU and obtains 84.1%, 79.4%, 86.9%
and 91.3% per-image accuracy on Caltech-UCSD birds [67],
NABirds [64], FGVC aircraft [42], and Stanford cars [33]
dataset respectively.
arXiv:1504.07889v6 [cs.CV] 1 Jun 2017
剩余13页未读，继续阅读
评论0