Faster R-CNN Towards Real-Time Object Detection with Region Prop...

1

Faster R-CNN: Towards Real-Time Object

Detection with Region Proposal Networks

Shaoqing Ren, Kaiming He, Ross Girshick, and Jian Sun

Abstract—State-of-the-art object detection networks depend on region proposal algorithms to hypothesize object locations.

Advances like SPPnet [1] and Fast R-CNN [2] have reduced the running time of these detection networks, exposing region

proposal computation as a bottleneck. In this work, we introduce a Region Proposal Network (RPN) that shares full-image

convolutional features with the detection network, thus enabling nearly cost-free region proposals. An RPN is a fully convolutional

network that simultaneously predicts object bounds and objectness scores at each position. The RPN is trained end-to-end to

made publicly available.

Index Terms—Object Detection, Region Proposal, Convolutional Neural Network.

F

1 INTRODUCTION

Recent advances in object detection are driven by

the success of region proposal methods (e.g., [4])

and region-based convolutional neural networks (R-

CNNs) [5]. Although region-based CNNs were com-

putationally expensive as originally developed in [5],

their cost has been drastically reduced thanks to shar-

Region proposal methods typically rely on inex-

pensive features and economical inference schemes.

Selective Search [4], one of the most popular meth-

ods, greedily merges superpixels based on engineered

low-level features. Yet when compared to efficient

detection networks [2], Selective Search is an order of

magnitude slower, at 2 seconds per image in a CPU

implementation. EdgeBoxes [6] currently provides the

best tradeoff between proposal quality and speed,

at 0.2 seconds per image. Nevertheless, the region

2

(Figure 1, b), we introduce novel “anchor” boxes

that serve as references at multiple scales and aspect

ratios. Our scheme can be thought of as a pyramid

of regression references (Figure 1, c), which avoids

enumerating images or filters of multiple scales or

aspect ratios. This model performs well when trained

and tested using single-scale images and thus benefits

running speed.

To unify RPNs with Fast R-CNN [2] object detec-

tion networks, we propose a training scheme that

We comprehensively evaluate our method on the

PASCAL VOC detection benchmarks [11] where RPNs

with Fast R-CNNs produce detection accuracy bet-

ter than the strong baseline of Selective Search with

Fast R-CNNs. Meanwhile, our method waives nearly

all computational burdens of Selective Search at

test-time—the effective running time for proposals

is just 10 milliseconds. Using the expensive very

deep models of [3], our detection method still has

a frame rate of 5fps (including all steps) on a GPU,

A preliminary version of this manuscript was pub-

lished previously [10]. Since then, the frameworks of

RPN and Faster R-CNN have been adopted and gen-

eralized to other methods, such as 3D object detection

[13], part-based detection [14], instance segmentation

[15], and image captioning [16]. Our fast and effective

object detection system has also been built in com-

1. Since the publication of the conference version of this paper

R-CNN networks leading to less training time.

2

. These results suggest that our method

3

composed of two modules. The first module is a deep

fully convolutional network that proposes regions,

and the second module is the Fast R-CNN detector [2]

that uses the proposed regions. The entire system is a

4

position. For a convolutional feature map of a size

W × H (typically ∼2,400), there are W Hk anchors in

total.

An important property of our approach is that it

is translation invariant, both in terms of the anchors

and the functions that compute proposals relative to

the anchors. If one translates an object in an image,

the proposal should translate and the same function

should be able to predict the proposal in either lo-

an object is translated.

The translation-invariant property also reduces the

model size. MultiBox has a (4 + 1) × 800-dimensional

fully-connected output layer, whereas our method has

a (4 + 2) × 9-dimensional convolutional output layer

in the case of k = 9 anchors. As a result, our output

layer has 2.8 × 10

4

for VGG-16), two orders of magnitude fewer than

MultiBox’s output layer that has 6.1 × 10

parameters

to have less risk of overfitting on small datasets, like

PASCAL VOC.

up to the network’s total stride.

6. Considering the feature projection layers, our proposal layers’

6

;

6

.

5

any ground-truth box. Note that a single ground-truth

assign a negative label to a non-positive anchor if its

Anchors that are neither positive nor negative do not

1

N

p

(t

, t

).

(1)

Here, i is the index of an anchor in a mini-batch and

p

is the predicted probability of anchor i being an

object. The ground-truth label p

is 1 if the anchor

is positive, and is 0 if the anchor is negative. t

is a

vector representing the 4 parameterized coordinates

where x, y, w, and h denote the box’s center coordi-

nates and its width and height. Variables x, x

a

, and

x

are for the predicted box, anchor box, and ground-

truth box respectively (likewise for y, w, h). This can