2020 International Conference on Computer Network, Electronic and Automation (ICCNEA)
State-of-the-art of Object Detection Model Based on YOLO
Peng Chen
Unit 32382, Beijing, China
E-mail: beimingke@163.com
Yuejun Shi
Unit 32382, Beijing, China
E-mail: 519752292@qq.com
Abstract—Object detection has got rapid development these
years, and target classification and location become gradually
mature. Fast R-CNN, which is a landmark in accuracy, is a
kind of classic two-stage model and has relatively low speed.
To improve the detection efficiency, YOLO (You Only Look
Once) model is proposed. In this article, the research starts
from the evolution of CNN for object detection and YOLO
model is sorted out. Then we compare the structures, target
outputs and network training of each versions. The functional
layers evolved into a structure constructed with residual mod-
ules, which keep the accuracy. At last, we discuss the system
advantage future develop trend. Form the discussion, we con-
clude that YOLO learns very general representations of ob-
jects, and it integrates various mature new technologies. With
the improvement of hardware technology, it will be applied
more widely.
Keyword-Classification and Location; Object Detection;
CNN
I. INTRODUCTION
These years, as a core problem in computer vision, ob-
ject detection based on CNN (Convolutional Neural Net-
works) develops fast and new framework emerges massively.
Generally, it implements by two steps: one is feature extrac-
tion, and the other one is classification and location. For
extracting a set of robust features from input images in the
first step, Haar [1]
, SIFT [2]
, HOG [3]
, convolutional features
are the mainstream method. Then, for classification or loca-
tion are used to identify objects in the feature space. The
classifiers or localizers are run either in sliding window
fashion over the whole image or on some subset of regions
in the image [5-7]
. Based on this essential structure, there are
many framework. This paper will discuss the one of the
978-1-7281-7083-1/20/$31.00 ©2020 IEEE
DOI 10.1109/ICCNEA50255.2020.00030
Authorized licensed use limited to: UNIVERSIDAD DE OVIEDO. Downloaded on December 30,2020 at 17:07:46 UTC from IEEE Xplore. Restrictions apply.
Qian Zheng
Unit 32382, Beijing, China
E-mail: 1763987144@qq.com
Qingchun Wu
Unit 32382, Beijing, China
E-mail: 3247029516@qq.com
most popular YOLO framework, referred to as You Only
Look Once. It spreads fast for its speed and relatively accu-
rate effect.
II. THE EVOLUTION OF CNN FOR OBJECT DETECTION
The first practical CNN is LeNet-5, which is constructed
with two convolutional layers, two subsampling layers and
two full connection layers, designed for training for gray-
scale. The first landmark framework is AlexNet, which is
the champion of ILSVRC 2012, deeper than LeNet. It is
constructed with five convolutional layers and three full
connection layers, importing three pooling layers, abandon-
ing subsampling layers, and replacing the large convolution
with the multilayer small convolution superposition. Until
now, the CNN structure is no longer a simple combination
of layers, but an organic composition of "modules". AlexNet,
which derivatives ZF net, firstly uses ReLU activation and
CUDA, speeding up the training greatly.
The second major improvement is VGG-16 and Goog-
leNet, which is a famous framework in ILSVRC 2014. In
contrast, VGG-16 is slightly less efficient than GoogleNet,
[4] the champion of ILSVRC 2014, however it is simple and
faster. VGGNet explores the relationship between the depth
of the convolutional neural network and its performance. By
repeatedly stacking the small convolutional kernels and
maximum pooling layers, VGG-16 successfully constructs a
16~19 layers deep convolutional neural network. Compared
with the previous state-of-the art network structure, the error
101
rate of VGG-16 is greatly reduced. VGG-16 structure is very
neat, with not so many super parameters. It focus on build-
ing a simple network, which is constructed with several
convolutional layers followed by one pooling layer for com-
pressing image size. Unlike VGG, GoogleNet is a bolder try,
which not only increase the depth, it also increases the width.
For avoiding the curse of massive parameters, excessive
depth and huge net size, GoogleNet designs the Inception
structure, which adds two loss at different depths to ensure
the disappearance of the gradient return and adds 1x1 con-
volution kernel of to reduce the thickness of the feature map.
VGG-16 and GoogleNet make people aware that deepening
the network is an effective way to improve the quality of
models, laying the foundation for the follow-up.
The CNN framework such as LeNet, AlexNet, VGG and
GoogleNet has effectively finishes the feature extraction and
classification. However, we need to add another important
function of object location. The framework which can com-
[8]
plete the task is R-CNN (Region with CNN feature)
R-CNN divides a picture into S×S grids, continuously
merging the small regional grids using SS (Selective Search),
and then extracting the features of each small grid. When we
get the feature map, border regression are used to complete
target location. A border regression is usually a prediction of
(x,y,w,h). Regions are derived from the concept of "receptive
field", and different receptive fields correspond to different
regions. However, R-CNN is not so ideal. Firstly, it needs to
consume a lot of computer resources to extract candidate
areas because of this sensitive field processing. Secondly,
although the image normalization of the human eye nerve
processing is very clever, the computer processing lead to a
great increase in the amount of calculation because of the
size normalization problem. In addition, a large number of
overlapping areas will also lead to double counting of the
CNN process. Then Fast R-CNN uses the ROI Pooling layer
to decrease the computer load and increase the speed. In
Faster R-CNN, the SS algorithm is replaced by RPN (Re-
gion Proposal Net), which is a full convolutional neural
network, increase the speed and further improve the effect.
For R-CNN framework, the image feature is extraction
twice, which classifies the target first and then does the loca-
Authorized licensed use limited to: UNIVERSIDAD DE OVIEDO. Downloaded on December 30,2020 at 17:07:46 UTC from IEEE Xplore. Restrictions apply.
tion. For speedup the course, YOLO framework is proposed
to combine the two steps together, which abandons the RPN
and use convolutions to achieve classification and location.
Until now, YOLO framework has developed to the firth
[9-12]
generation, called YOLOv1~YOLOv4 . In this paper,
we will analysis this framework in several aspects.
III. THE STRUCTURE OF YOLO
A. Basic Structure
As R-CNN, YOLOv1 also divides the image into S×S
grids, but this segmentation is not so dense, for we don't
need to merge candidate boxes in advance. The structure of
YOLOv1 is shown as Fig. 1.
.
Figure 1. The structure of YOLOv1
The convolution layers in the front extracts features, and
the full connection layer in the back calculates the output
probabilities and coordinates. The network main body
adopts the structure of 24 volume basic units and 2 full con-
nection layers. There is also a Fast YOLO version of
YOLOv1, which cuts the previous 24 convolutions to 9
convolutions, with all the rest being the same.
YOLOv2 was improved on the basis of YOLOv1, whose
accompanying version YOLO9000 model was trained on
COCO detection data set and ImageNet classification data
set, which can detect more than 9000 kinds of objects, im-
proving mAP in maintaining a good consistency of speed.
The basic classification structure of this model is darknet-19,
including 19 convolutional layers, 5 maxpooling layers, one
global avgpool layer, and one softmax classification activa-
tion layer.
YOLOv3 was expanded to darknet53 on the basis of
YOLOv2, which is relatively large. Darknet53 refers to the
52-layer convolutional layer +1 fully connected layer
(achieved by 1x1 convolution). Compared with YOLOv2,
102
YOLOv3 no longer has the maxpooling layer and the reorg clearer as it moves forward. In order to balance this, you
layer. Compared with the residual layer, the multi-resolution need to take the front information back and mix them with
method is adopted in the structure, and YOLO layers of the information from the back. As discussed in the previous
large, medium and small sizes are set. section, the route layer allows YOLO to stop being a pure
"you only look at it once," and instead “looking back two
B. Improved module
more times”, adjusting the results and improving the accu-
Nearly all the improvement based on the basic structure
racy of the predictions.
is on the module, which is a better combination of added 3) Residual layer
layers. We all know, the depth of the network could not be infi-
1) Batch-normalize layer nite. In fact, the plain network stack begins to degenerate
The batch-normalize layer is added to each implementa-
when the network is deep enough. Because the deeper the
tion of the convolution layer. The primary use of this layer is
network is, the more obvious the gradient disappears. Until
to prevent the gradient from exploding and disappearing.
now, the shallow network is not able to significantly im-
The calculation error is needed to calculate the gradient for
prove the network recognition effect, so the problem to be
stochastic gradient descent, and the error is also propagated
solved is how to control gradient disappearance while deep-
in the back propagation process. According to the principle
ening the network for better recognition effect. On ILSVRC
of back propagation, there will be cumulative and multi-
2014, the new champion ResNet, which import the Residual
plexing effects. If the error calculation at each layer is not
block, of which network depth of ResNet is unlimited theo-
controlled, the increment of gradient will gradually disap-
retically.
pear or become infinite. Batch-normalize is to normalize by
From YOLOv3, residual layer was added to realize re-
batch to control this problem in the back propagation pro-
sidual network. Residual layer is also called shortcut layer. It
cess.
sounds like a bypass layer or a short circuit layer. Generally,
Each layer is a joint probability function in practice,
the output of the previous several layers is directly directed
multiplicative the whole system is a huge joint probability
to the output layer of the current layer. The residual model
function, element is the input of each layer. For normaliza-
scheme is as Fig. 2.
tion, it needs to put the batch elements together to find the
mean and variance, then computes elements minus divided
by average variance and divided by the variance. In order to
avoid changing the characteristics of probability distribution,
scale and offset should be added to the normalized results.
2) Route layer
From YOLOv2, route layer begins to appear in the Figure 2. Residual model scheme

framework. Route layers take the output of the specified

If there is no residual layer, the input of the above mod-
layer and places it in current layer. If the specified layers are
ule is x, and the output is ( x) , our optimization goal is
multiple, it will follow one by one. For multiple layers, one
( x) . With the residual layer, )= ( x)  x , the resid-
( x)=
of the key rules is that the dimensions of the two layers must
ual is ( x)=
)= ( x)--x , and if you train
))-x ( x) to a small
be the same in width×height, otherwise you zero both l. w
enough value, ( x) will get closer to the desired output.
and l.h. Because for YOLO the feature extraction is carried
4) Yolo layer
out in sequence, continuous convolution and pooling. In the
YOLO series author is very interesting, v1 output layer is
end, there may be some information pasted together. It is
called detection layer, v2 output layer is called region layer,
quite true that the detailed information gets clearer and
v3 output layer is called YOLO layer, and perhaps really
clearer as it moves forward, and the classification becomes
think v3 has been successful cultivation.

103

Authorized licensed use limited to: UNIVERSIDAD DE OVIEDO. Downloaded on December 30,2020 at 17:07:46 UTC from IEEE Xplore. Restrictions apply.
 Each yolo layer corresponds to three clustering anchor
boxes, which can also be seen at a glance. The
pre-determined size of anchor is relative to the input layer of
the whole network. During the training, it needs to deter-
mine which one of the three boxes should be used, and then
predict the ratio between the width and height.
C. New technology in YOLOv4
In YOLOv4, CSPDarknet53 is chosen as backbone,
which is added over by the SPP block. It imports and veri-
fies Bag-of-Freebies and Bag-of-Specials methods of object
detection. It is also a more efficient and powerful object de-
tection model. It makes everyone can use a 1080 Ti or 2080
Ti GPU to train a super-fast and accurate object detector. As
Alexey said[12]ˈComparison of the proposed YOLOv4 and
other state-of-the-art object detectors.
IV. TARGET OUTPUT
The target output of YOLOv1 is 7×7×30 results. The
7×7 output layer contains the results that can be detected in
each corresponding region after the corresponding position
of the original input by 64 times and features are extracted.
This is a 30-dimensional tensor. The first 10 dimensions are
divided into 2 groups, which are respectively used for detec-
tion. The confidence of 2 targets, the coordinate of the center
(x,y), and the border (w,h) are respectively used. The next
20 dimensions are used to categorize the 20 objects. In short,
the detection capability of YOLOv1 is to detect 2 results out
of 20 targets in a region.
For YOLOv2, taking coco data set as an example, the
last layer outputs the dimension of the last layer is 425,
which means that up to 80 species can be detected, and 5
targets can be detected at the same time, 5*(80+5). The out-
put feature map is a tensor of 425 channels.
Here, Anchor boxes are used, which are priori boxes.
K-means clustering algorithm is used in advance to cluster
various targets into 5 categories. When the region layer im-
plements the detection, the closest one is selected to calcu-
late the ratio of the target size to the prior box. The Bound-
ing boxes with dimension priors and location prediction is as
Fig. 3.
104
Authorized licensed use limited to: UNIVERSIDAD DE OVIEDO. Downloaded on December 30,2020 at 17:07:46 UTC from IEEE Xplore. Restrictions apply.
Figure 3. Bounding boxes with dimension priors and location prediction
The output that border regression needs to process is
(tx,ty,tw,th), which is computed by
bx V (t x )  cx
by V (t y )  c y
bw pw e tw
bh ph e t h
For YOLOv3, the target output of each yolov layer is
255 channels. We use 255 channels because the system out-
put targets of 3 out of 80 categories at the same time. In the
output result, the first 5 channels of each group are (x, y, w,
h, score), and the last 80 channels are 80 categories, a total
of (4+1+80) ×3 = 255. Also, according to the darknet con-
figuration, each diagram can display up to 90 targets.
V. NETWORK TRAINING
The model training of YOLO series is actually a
two-step training. The initial parameters of the convolutional
layer are pre-trained through ImageNet 1000-class, using the
model of the first 20 winding bases + the average pooling
layer + a full connection layer. The newly added weights are
all randomly generated. Users can use YOLO network ac-
cording to their actual data set needs and train with YOLO's
pre-trained weight file.
For YOLOv2 and YOLOv3, the method is the same as
YOLOv1, the difference is LOSS function. Here we take
YOLOv3 for example the LOSS function is as follow.
 Here binary cross entropy loss is used, different from the
front version, the background is no longer distinguished and
the multi-classification loss function based on softmax is no
longer used. Instead of specifically configuring the lambda
coefficients in the configuration file, they all take the con-
stant 1.
A controversial point is that MSE loss function was used
to replace the cross entropy loss function. One explanation is
that for Logistic regression, the derivative form of variance
loss and cross entropy loss is exactly the same. In darknet,
the variance loss is replaced, because the numerical trend is
the same.
The border information loss is multiplied by a (2-wi×hi)
scaling factor, where w and h are the width and height of GT,
respectively. This is in order to improve prediction accuracy
for small object a little skill, for wi, hi < 1, if wi, hi is small-
er, wi, hi, the greater the this part of the greater the weight,
the impact of error can be amplified, some experiments for
YOLOv3, if not minus wi×hi, the AP will have a signifi-
cantly lower, if continue to add up, such as(2-wi×hi)× 1.5,
general AP will still rise a PM (including validation set and
test set).
VI. SYSTEM ADVANTAGE
Here use YOLOv3 and YOLOv4 compare with the pop-
ular frameworks, comparison of the proposed YOLOv3 and
YOLOv4 other state-of-the-art object detectors. YOLOv4
runs twice faster than other top frameworks with comparable
performance. Improves YOLOv3’s AP and FPS by 10% and
12%, respectively, is shown in Fig. 4 [12].
Figure 4. Effect of contrast with other frameworks
105
Authorized licensed use limited to: UNIVERSIDAD DE OVIEDO. Downloaded on December 30,2020 at 17:07:46 UTC from IEEE Xplore. Restrictions apply.
VII. CONCLUSION
In this article, it starts from the evolution of CNN for
object detection and YOLOv1~YOLOv3 are sorted out from
structure, Target output and network training. YOLOv4 es-
sentially inheriting the structure of YOLOv3, adding more
current new uses of trick. YOLOv4 is mainly for industrial
applications. It is applied in TensorFlow, pytorch, caffe and
other frameworks. Different versions of YOLOv4 have dif-
ferent implementations and are essentially the same. It's
been reported that YOLOv3 have been constructed in Pad-
dlePaddle, which is independently developed by Baidu, and
the popularity has skyrocketed over the last two years, and
with the improvement of hardware technology, it will be
applied more widely.
REFERENCES
[1] C. P. Papageorgiou, M. Oren, and T. Poggio. A General Framework
for Object Detection[C] //IEEE International Conference on Com-
puter vision, April 1998: 555–562.
[2] D. G. Lowe. Object recognition from local scale-invariant features[C]
//The proceedings of the seventh IEEE international conference on
Computer vision, April 1999: 1150–1157.
[3] N. Dalal and B. Triggs. Histograms of oriented gradients for human
detection[C] // IEEE Computer Society Conference on In Computer
Vision and Pattern Recognition, 2005. CVPR 2005. April 2005:
886–893.
[4] J. Donahue, Y. Jia, O. Vinyals, J. Hoffman, N. Zhang, E. Tzeng, and
T. Darrell. Decaf: A deep convolutional activation feature for generic
visual recognition [J]. arXiv preprint arXiv:1310.1531, 2013. 4.
[5] S. Gould, T. Gao, and D. Koller. Region-based Segmentation and
Object Detection [J]. Advances in Neural Information Processing
Systems. 2009, 4:655–663.
[6] J. R. Uijlings, K. E. van de Sande, T. Gevers, and A. W. Smeulders.
Selective search for object recognition [J]. International journal of
computer vision, 2013, 104(2):154–171.
[7] C. L. Zitnick and P. Dollár. Edge boxes: Locating object proposals
from edges[C]// Computer Vision–ECCV 2014: 391–405.
[8] Girshick R. Fast R-CNN[C] // International conference on computer
vision, 2015: 1440-1448.
[9] J Redmon, S K Divvala, R Girshick, et al. You Only Look Once:
Unified, Real-Time Object Detection[C] //Computer vision and pat-
tern recognition, 2016: 779-788
[10] J Redmon, A Farhadi. YOLO9000: Better, Faster, Stronger[C].
Computer vision and pattern recognition, 2017: 6517-6525.
[11] J Redmon, A Farhadi. YOLOv3: An Incremental Improvement [J].
arXiv: Computer Vision and Pattern Recognition, 2018.
[12] A Bochkovskiy, C Y Wang, H Y Mark Liao. YOLOv4: Optimal
Speed and Accuracy of Object Detection [J]. arXiv preprint
arXiv:2004.10934, 2020. 4.