POLY-YOLO: HIGHER SPEED, MORE PRECISE DETECTION AND
INSTANCE SEGMENTATION FOR YOLOV3
A PREPRINT
Petr Hurtik*, Vojtech Molek*, Jan Hula*, Marek Vajgl*, Pavel Vlasanek
∗
, and Tomas Nejezchleba
†
June 1, 2020
ABSTRACT
We present a new version of YOLO with better performance and extended with instance segmentation
called Poly-YOLO. Poly-YOLO builds on the original ideas of YOLOv3 and removes two of its
weaknesses: a large amount of rewritten labels and inefficient distribution of anchors. Poly-YOLO
reduces the issues by aggregating features from a light SE-Darknet-53 backbone with a hypercolumn
technique, using stairstep upsampling, and produces a single scale output with high resolution. In
comparison with YOLOv3, Poly-YOLO has only 60% of its trainable parameters but improves
mAP by a relative 40%. We also present Poly-YOLO lite with fewer parameters and a lower
output resolution. It has the same precision as YOLOv3, but it is three times smaller and twice
as fast, thus suitable for embedded devices. Finally, Poly-YOLO performs instance segmentation
using bounding polygons. The network is trained to detect size-independent polygons defined on
a polar grid. Vertices of each polygon are being predicted with their confidence, and therefore
Poly-YOLO produces polygons with a varying number of vertices. Source code is available at
https://gitlab.com/irafm-ai/poly-yolo.
Keywords
Object detection
·
Instance segmentation
·
YOLOv3
·
Bounding box
·
Bounding polygon
·
Realtime
detection
Figure 1: The figure shows instance segmentation performance of the proposed Poly-YOLO algorithm applied on
Cityscapes dataset and running 22FPS on a mid-tier graphic card. Image was cropped due to visibility.
∗
University of Ostrava, Centre of Excellence IT4Innovations, Institute for Research and Applications of Fuzzy Modeling, 30.
dubna 22, Ostrava, Czech Republic
†
Varroc Lighting Systems, Suvorovova 195, Šenov u Nového Ji
ˇ
cína, Czech Republic.
arXiv:2005.13243v2 [cs.CV] 29 May 2020
A PREPRINT - JUNE 1, 2020
1 Problem statement
Object detection is a process where all important areas containing objects of interest are bounded while the background
is ignored. Usually, the object is bounded by a box that is expressed in terms of spatial coordinates of its top-left corner
and its width and height. The disadvantage of this approach is that for the objects of complex shapes, the bounding
box also includes background, which can occupy a significant part of the area as the bounding box does not wrap the
object tightly. Such behavior can decrease the performance of a classifier applied over the bounding box [
1
] or may
not fulfill requirements of precise detection [
2
]. To avoid the problem, classical detectors such as Faster R-CNN [
3
]
or RetinaNet [
4
] were modified into a version of Mask R-CNN [
5
] or RetinaMask [
6
]. These methods also infer the
instance segmentation, i.e., each pixel in the bounding box is classified into object/background classes. The limitation
of the methods is their computation speed, where they are unable to reach real-time performance on non-high-tier
hardware. The problem we focus on is to create a precise detector with instance segmentation and the ability of real-time
processing on mid-tier graphic cards.
In this study, we start with YOLOv3 [
7
], which excels in processing speed, and therefore it is a good candidate for
real-time applications running on computers [
8
] or mobile devices [
9
]. On the other hand, the precision of YOLOv3 lags
behind detectors such as RetinaNet [
4
], EfficientDet [
10
], or CornerNet [
11
]. We analyze YOLO’s performance and
identify its two drawbacks. The first drawback is low precision of the detection of big boxes [
7
] caused by inappropriate
handling of anchors in output layers. The second one is rewriting of labels by each-other due to the coarse resolution. To
solve these issues, we design a new approach, dubbed Poly-YOLO, that significantly pushes forward original YOLOv3
abilities. To tackle the problem of instance segmentation, we propose a way to detect tight polygon-based contour. Our
contributions and benefits of our approach are as follows:
•
we propose Poly-YOLO that increases the detection accuracy of the previous version, YOLOv3. Poly-YOLO
has a brand-new feature decoder with a single output tensor that goes to head with higher resolution that solves
two principal YOLO’s issues: rewriting of labels and incorrect distribution of anchors.
•
We produce a single output tensor by a hypercolumn composition of multi-resolution feature maps produced
by a feature extractor. To unify the resolutions of the feature maps, we utilize stairstep upscaling, which allows
us to obtain slightly lower loss with comparison to direct upscaling while the computation speed is preserved.
•
We design an extension that realizes instance segmentation using bounding polygon representation. The
number of maximal polygon vertices can be adjusted according to a requirement to a precision.
•
The bounding polygon is detected within a polar grid with relative coordinates that allow the network to learn
general, size-independent shapes. The network produces a dynamic number of vertices per bounding polygon.
Figure 2: Examples of Poly-YOLO inference on the Cityscapes testing dataset.
Figure 3: Examples of Poly-YOLO inference on the India driving testing dataset.
2
A PREPRINT - JUNE 1, 2020
2 Current state and related work
2.1 Object detection
Models for object detection can be divided into two groups, two-stage, and one-stage detectors. Two-stage detectors
split the process as follows. In the first phase, regions of interest (RoI) are proposed, and in the subsequent stage,
bounding box regression and classification is being done inside these proposed regions. One-stage detectors predict the
bounding boxes and their classes at once. Two-stage detectors are usually more precise in terms of localization and
classification accuracy, but in terms of processing are slower then one-stage detectors. Both of these types contain a
backbone network for feature extraction and head networks for classification and regression. Typically, the backbone is
some SOTA network such as ResNet [
5
] or ResNext [
12
], pre-trained on ImageNet or OpenImages. Even though, some
approaches [13], [14] also experiment with training from scratch.
2.1.1 Two-stage detectors
The prototypical example of two-stage architecture is Faster R-CNN [
3
], which is an improvement of its predecessor
Fast R-CNN [
15
]. The main improvement lies in the use of Region Proposal Network (RPN), which replaced a much
slower selective search of RoIs. It also introduced the usage of multi-scale anchors to detect objects of different sizes.
Faster R-CNN is, in a way, a meta-algorithm that can have many different incarnations depending on a type of the
backbone and its heads. One of the frequently used backbones, called Feature Pyramid Network (FPN) [
16
], allows
to predict RoIs from multiple feature maps, each with a different resolution. This is beneficial for the recognition of
objects at different scales.
2.1.2 One-stage detectors
Two best-known examples of one-stage detectors are YOLO [
7
] and SSD [
17
]. The architecture of YOLO will be
thoroughly described in Section 3. Usually, one-stage detectors divide the image into a grid and predict bounding boxes
and their classes inside them, all at once. Most of them also use the concept of anchors, which are predefined typical
dimensions of bounding boxes that serve as apriori knowledge. One of the major improvements in the area of one-stage
detectors was a novel loss function call Focal Loss [
4
]. Because of the fact that two-stage detectors produce a sparse
set of region proposals in the first step, most of the negative locations are filtered out for the second stage. One-stage
detectors, on the other hand, produce a dense set of region proposals which they need to classify as containing objects or
not. This creates a problem with the non-proportional frequency of negative examples. Focal Loss solves this problem
by adjusting the importance of negative and positive examples within the loss function. Another interesting idea was
proposed in an architecture called RefineDet [
18
], which performs a two-step regression of the bounding boxes. The
second step refines the bounding boxes proposed in the first step, which produces more accurate detection, especially
for small objects. Recently, there has been a surge of interest in approaches that do not use anchor boxes. The main
representative of this trend is the FCOS framework [
19
], which works by predicting four coordinates of a bounding
box for every foreground pixel. These four coordinates represent a distance to the four boundary edges of a bounding
box in which the pixel is enclosed in. The predicted bounding boxes of every pixel are subsequently filtered by NMS.
Similar anchor-free approach was proposed in CornerNet [
11
], where the objects are detected as a pair of top-left and
bottom-right corners of a bounding box.
2.2 Instance Segmentation
In many applications, a boundary given by a rectangle may be too crude, and we may instead require a boundary
framing the object tightly. In the literature, this task is called Instance Segmentation, and the main approaches also
fit into the one-stage/two-stage taxonomy. The prototypical example of a two-stage method is an architecture called
Mask R-CNN [
5
], which extended Faster R-CNN by adding a separate fully-convolutional head that predicts masks of
objects. Note, the same principle is also applied to RetinaNet, and the improved net is called RetinaMask [
6
]. One of
Mask R-CNN innovations is a novel way for extracting features from RoIs using the RoIAlign layer, which avoids
the problem of misalignments of the RoI due to its quantization to the grid of the feature map. One-stage methods
for instance segmentation can be further divided into top-down methods, bottom-up methods, and direct methods.
Top-down methods [
20
,
21
] work by first detecting an object and then segmenting this object within a bounding box.
Prediction of bounding boxes either uses anchors or is anchor free following the FCOS framework [
19
]. Bottom-up
methods [
22
,
23
], on the other hand, work by first embedding each pixel into a metric space in which are these pixels
subsequently clustered. As the name suggests, direct methods work by directly predicting the segmentation mask
without bounding boxes or pixel embedding [
24
]. We also mention that independently of our instance segmentation,
PolarMask [
25
] introduces instance segmentation using polygons, which are also predicted in polar coordinates. In
comparison with PolarMask, Poly-YOLO learns itself in general size-independent shapes due to the use of the relative
3
A PREPRINT - JUNE 1, 2020
size of a bounding polygon according to the particular bounding box. The second difference is that Poly-YOLO
produces a dynamic number of vertices per polygon, according to the shape-complexity of various objects.
3 Fast and precise object detection with Poly-YOLO
Here, we firstly recall YOLOv3 fundamental ideas, describe issues that block reaching higher performance, and propose
our solution that removes them.
3.1 YOLO history
First version of YOLO (You Only Look Once) was introduced in 2016 [
26
]. The motivation behind YOLO is to create a
fast object detector with an emphasis on speed. The detector is made of two essential parts: the convolutional neural
network (CNN) and specially designed loss function. The CNN backbone is inspired by GoogleNet [
27
] and has 24
convolutional layers followed by 2 fully connected layers. The network output is reshaped into two dimensional grid
with with the shape of
G
h
× G
w
, where
G
h
is number of cells in vertical side and
G
w
in horizontal side. Each grid
cell occupies a part of the image, as depicted in Fig. 4. Every object in the image has its center in one of the cells, and
Figure 4: The left image illustrates the YOLO grid over the input image, and yellow dots represent centers of detected
objects. The right image illustrates detections.
that particular cell is responsible for detecting and classifying said object. More precisely, the responsible cell outputs
N
B
bounding boxes. Each box is given as a tuple
(x, y, w, h)
and a confidence measure. Here,
(x, y)
is the center of
the predicted box relative to the cell boundary and
(w, h)
is the width and height of the bounding box relative to the
image size. The confidence measures how much is the cell confident that it contains an object. Finally, each cell outputs
N
c
conditional class probabilities, i.e. probabilities that detected object belongs to certain class(es). In other words,
cell confidence tells us that there is object in the predicted box and conditional class probabilities tells us that the box
contains, e.g., vehicle – car. The final output of the model is a tensor with dimensions
G
h
× G
w
× (5N
B
+ N
c
)
, where
constant five is used because of (x, y, w, h) and a confidence.
YOLOv2 [
28
] brought a couple of improvements. Firstly, the architecture of the convolutional neural network was
updated to Darknet-19 – a fully convolutional network with 19 convolutional layers containing batch normalization and
five max-pooling layers. The cells are no longer predicting plain
(x, y, w, h)
directly, but instead scales and translates
anchor boxes. The parameters
(a
w
, a
h
)
, i.e., width and height of an anchor box for all anchors boxes are extracted
from a training dataset with usage of
k
-means algorithm. The clustering criterion is IoU. Lastly, YOLOv2 uses skip
connections to concatenate features from different parts of the CNN to create final tensor of feature maps, including
features across different scales and levels of abstraction.
The most recent version of YOLO [
7
] introduces mainly three output scales and a deeper architecture – Darknet-53.
Each of the scale/feature-map has its own set of anchors – three per output scale. Compared with v2, YOLOv3 reaches
higher accuracy, but due to the heavier backbone, its inference speed is decreased.
3.2 YOLOv3 issues blocking better performance
YOLOv3, as it is designed, suffers from two issues that we discovered and that are not described in the original papers:
rewriting of labels and imbalanced distribution of anchors across output scales. Solving these issues is crucial for
improvement of the YOLO performance.
4
A PREPRINT - JUNE 1, 2020
Figure 5: The image illustrates the label rewriting problem for the detection of cars. A label is rewritten by other if
centers of two boxes (with the same anchor box) belong to the same cell. In this illustrative example, blue denotes grid,
red rewritten label, and green preserved label. Here, 10 labels out of 27 are rewritten, and the detector is not trained to
detect them.
Table 1: Amount of rewritten labels for various datasets
Rewritten labels [%]
Poly Poly
Dataset Resolution YOLOv3 YOLO YOLO lite
Simulator 416×416 16.36 0.22 2.31
Simulator 608×800 12.55 0.00 0.61
Cityscapes 416×416 9.51 2.79 9.50
Cityscapes 608×832 3.92 0.97 2.75
Cityscapes 640×1280 2.56 0.59 1.44
India Driving 416×416 23.07 5.80 13.78
India Driving 448×800 13.54 1.92 4.96
India Driving 704×1280 9.16 1.12 2.44
3.2.1 Label rewriting problem
here, we discuss situation, when a bounding box given by its label from a ground truth dataset can be rewritten
by other box and therefore the network is not trained to detect it. For the sake of simplicity and explanation, we
avoid the usage of the anchors notation in the text bellow. Let us suppose an input image with a resolution of
r × r
pixels. Furthermore, let
s
k
be a scale ratio of an
k
-th output to the input, where YOLOv3 uses the following
ratios:
s
1
= 1/8, s
2
= 1/16, s
3
= 1/32
. These scales are given by the YOLOv3 architecture, namely by strided
convolutions. Finally, let
B = {b
1
, . . . , b
n
}
be a set of boxes presented in an image. Each box
b
i
is represented
as a tuple
(b
x
1
i
, b
y
1
i
, b
x
2
i
, b
y
2
i
)
that defines its top-left and bottom-right corner. For simplicity, we also derive centers
C = {c
1
, . . . , c
n
}
where
c
i
= (c
x
i
, c
y
i
)
is defined as
c
x
i
= 0.5(b
x
1
i
+ b
x
2
i
)
and the same for
c
y
i
. With this notation, label
is rewritten, if the following holds:
∃(c
i
, c
j
∈ C) : ξ(c
x
i
, c
x
j
, s
k
) + ξ(c
y
i
, c
y
j
, s
k
) = 2, (1)
where
ξ(x, y, z) =
1, bxzc = byzc
0, else
, (2)
and
b·c
denotes the lowest integer of the term. The purpose of function
ξ
is to check if both boxes are assigned to the
same cell of a grid on the scale
s
k
. In simple words, if two boxes on the same scale are assigned to the same cell, then
one of them will be rewritten. Introducing anchors, both must belong to the same anchor. As a consequence, the network
is trained to ignore some objects, which leads to a low number of positive detections. According to Equations
(1)
and
(2)
, there is a crucial role of
s
k
that directly affects the number and the resolution of cells. Considering standard
resolution of YOLO
r = 416
, then, for
s
3
(the coarsest scale) we obtain a grid of
13 × 13
cells with size of
32 × 32
pixels each. Also, the absolute size of boxes does not affect the label rewriting problem; the important indicator is the
box center. The practical illustration for such a setting and its consequence for the labels is shown in Figure 5. The ratio
of rewritten labels in the datasets used in the benchmark is shown in Table 1.
5
