The 2018 International Conference On Control Automation & Information Sciences (ICCAIS 2018)

October 24-27, 2018, Hangzhou, China.

The Analysis Between Traditional Convolution

Neural Network and CapsuleNet
Feng Yang1,2 Wentong Li Weikang Tang Xinliang Wu
1. Key Laboratory of Key Laboratory of Key Laboratory of China National
Information Fusion Tech- Information Fusion Tech- Information Fusion Tech- Aeronautical Radio Electronics
nology Ministry of Education nology Ministry of Education nology Ministry of Education Research Institute
Northwestern Poly- Northwestern Poly- Northwestern Poly- Shanghai, China
technical University technical University technical University wuxinliang51@163.com
Xi’an, China Xi’an, China Xi’an, China
2. Science and Technology liwentong@mail.nwpu.edu.cn tangweikang725@foxmail.com
on Electro-optic
Control Laboratory
Luoyang, China
yangfeng@nwpu.edu.cn

Abstract—Convolution neural networks(CNNs) have made a
series of breakthroughs in the fields of autonomous driving,
robotics, medical treatment and so on. The powerful feature
learning and classification abilities of CNNs have attracted wide
attention of scholars. With the improvement of the networks
structure and the increasing of the networks depth, the per-
formance of tasks such as image classification, detection, recog-
nition, and segmentation has been improved greatly. Recently,
CapsuleNet, which is different from CNNs, has been proposed.
In CapsuleNet, vectors replace traditional scalar points to better
characterize the relationships between feature information and
acquire more attributes of things. In this paper, the basic
structure and algorithm principle of classical CNN models were
described firstly, and then the basic neural network, network
structure, parameter update and distribution process for the
CapsuleNet model were analyzed and summarized. Finally,
several group of training experiments were conducted to compare
and analyze the image classification and feature expression
capabilities of CapsuleNet and CNN models.
Index Terms—CNN, CapsuleNet, VGGNet, GoogleNet, ResNet
I. I NTRODUCTION
The extraction and classification of image features have
always been a fundamental and important research direction in
the field of computer vision. The CNN provides an end-to-end
learning model. The parameters in the model can be trained to
complete feature extraction and classification at different levels
of image. The information processed by the CNN models is
the scalar. The point-and-product convolution and the local
information extraction are achieved through weight sharing
and pooling respectively, which greatly reduces the training
parameters and improves the perfermance of model training
The paper is sponsored by National Natural Science Foundation of
China (No. 61374159, No. 61374023), Aviation Science Foundation
(No. 20165153034), Natural Science Foundation of Shaanxi province
(No.2018MJ6048) and the Seed Foundation of Innovation and Creation
for Graduate Students in Northwestern Polytechnical University (No. Z-
Z2018149).
process, thereby speeding up image feature extraction and
learning. However, this also makes the CNN always activate
local information, have a weak global information association
capability, and unable to express more complex and abstract
attribute characteristics of objects.
In the CapsuleNet [1] model, a capsule is a set of neurons
whose input and output vectors implement the encapsulation
of the entity. The length of vectors is used to describe the
probability distribution, and the direction of vectors represents
the property of corresponding entities. In the process of
information processing, CapsuleNet model keeps convolution
to get the information of receptive field. The processing
and transformation process of vectors information replace the
traditional max-pooling, which preserves the precise feature
information of the entity’s posture (position, size, orientation),
deformation, velocity, albedo, hue, texture and so on. In terms
of parameters update, CapsuleNet model uses dynamic routing
algorithm in addition to the traditional back-propagation gra-
dient descent method [2]. It solves the problem of gradient dis-
sipation in training process to a certain degree, and improves
the expression ability of the model. In order to accurately learn
the features of higher-order abstractions, solve difficult tasks
related to artificial intelligence such as image classification [3],
object detection [4], and semantic understanding [5], it is of
great significance for the comparative analysis of traditional
CNNs and CapsuleNet.
II. T RADITIONAL C NN M ODEL
As an important research branch in the field of neural
networks, the CNN model has been rapidly developed from
LeNet-5 [6], AlexNet [7], VGGNet [8], GoogleNet [9] to
deeper ResNet [10], DenseNet [11]. CNN models have been
greatly improved and developed in depth, network structure,
generalization capabilies. In this section, the classic CNN

978-1-5386-6020-1/18/$31.00 ©2018 IEEE 210

 The 2018 International Conference On Control Automation & Information Sciences (ICCAIS 2018)
October 24-27, 2018, Hangzhou, China.
models including VGGNet, Google Inception Net, and ResNet
were analysised.
A. VGGNet
As shown in Fig.1, the relationship between the depth
and the performance of CNN was explored in VGGNet. By
repeatedly stacking the 3*3 convolution and the 2*2 max-
pooling layers, VGGNet successfully built a deep CNN model,
the number of weight layers from 11 to 19. VGGNet has 5
convolution segments. Each of segments has 2-3 layers, and a
max-pooling layer is connected to the end of each segment to
reduce the picture size. The number of kenels in each segment
is the same, followed by 64 - 128 - 256 - 512 - 512. There is
a very useful design in VGGNet that stacks multiple identical
convolutional layers together. Two 3*3 convolutional layers
in series are equivalent to a 5*5 convolutional layer, i.e. one
pixel will be associated with the surrounding 5*5 pixels. The
three 3*3 convolutional layers in series is equivalent to a 7*7
convolutional layer, but the quantity of parameters is less than
a 7*7 convolutional layer and the model has more nonlinear
transformations. In this way, the learning ability of the CNN
is more robust. VGGNet shows that the deeper the network,
the better the performance.
Fig. 1. The structure of VGGNet [8].
B. Google Inception Net
The inception module, which is the core network struc-
ture in the Google Inception Net model, combines different
convolutional layers in parallel. As shown in Fig.2, the in-
ception module structure has 4 parallel branches. The first
branch performs a 1*1 convolution on the input, which is
an important structure proposed by the NIN [12]. The 1*1
978-1-5386-6020-1/18/$31.00 ©2018 IEEE 211
convolution can not only increase the expressive ability of the
network by organizing information across channels, but also
reduce the dimension of the output channels to reduce the
training parameters. For the second branch, a 1*1 convolution
is used to reduce the channel dimensions firstly, and then
a 3*3 small-size receptive fields is concatenated to achieve
convolution. In this case, two nonlinear transformations of
model information are carried out. The third branch is similar
to the second branch. Firstly, a 1*1 convolution is used to
reduce the dimension of the channel, and then a 5*5 large
size receptive field is concatenates to achieve convolution. The
fourth branch is different. A 3*3 max-pooling is used to extract
the salient features first, and then a 1*1 convolution is used to
reduce the channel dimensions. Finally, the four branches are
merged in the channel dimension. The inception module output
incorporates multiple types of features extracted through three
different sizes of convolutional layers and one max-pooling
layer to increase the adaptability of the network to different
scales. In this way, the depth and width of the network can
be expanded efficiently, and the accuracy can be increased
without overfitting. During the development of the Google
Inception Net model, the inception module has also been
gradually improved.
Fig. 2. Inception module structure [9]
Google Inception Net includes four versions of Inception
V1 [9], Inception V2 [13], Inception V3 [14] and Inception
V4 [15]. All of those are spliced by the inception module.
The depth of the models gradually increased, and the top-
5 error rate dropped from 6.67% to 3.08% [9]. Among
them, the Batch Normalization (BN) method was proposed
to standardize the internal processing of the data in Inception
V2. The output of each layer can be normalized to a Gaussian
distribution to ensure the stability of the input data by the BN
method. This not only accelerates the convergence process of
the training, but also increases the generalization ability of
the model, which provides strong support for the training of
large-scale CNN models.
C. ResNet
If the layer is added to the previous layer simply to
increase the depth of the network, there will be a problem of
gradient dissipation during the training process, which makes
 The 2018 International Conference On Control Automation & Information Sciences (ICCAIS 2018)
October 24-27, 2018, Hangzhou, China.
the training process of the deep network more difficult. As the
depth of the network increases, its performance will gradually
become saturated, and even begin to decline. ResNet can solve
the problem of ”with the network becoming deeper and deeper,
the accuracy rate does drop”. It successfully trained a 152-
layer deep neural network, making the top-5 error rate as low
as 3.57% [10]. At the same time, the amount of parameters
is lower than VGGNet, and the effect of the model is very
prominent. The core module of ResNet is residual unit, and
its core idea is to introduce a shortcut connection that can skip
one or more layers. As shown in Fig.3, the input of a certain
neural network is x and the desired output is H (x). The input
x bypasses weight layer to H (x) as the initial result. At this
time, the learning objective is only the difference between the
input and output, that is, the residual F (x) = H(x) − x.
The learning goals have been changed, which allows the
stacked network layer to gradually adapt to a residual mapping.
This way of this shortcut connection is to directly send the
input information to the output to protect the integrity of
the information. The entire network only needs to learn the
difference between the input and output.The Learning goal is
simplified. The residual unit effectively solved the problem
of gradient dissipation to some extent, and made significant
contributions to deepening the network.
Fig. 3. The residual unit structure [10]
The number of layers evolved from 18 (ResNet18), 34
(ResNet34), and 50 (ResNet50) to 101 (ResNet101) and
152 (ResNet152). Though the depth has been significantly
increased, ResNet152 is still lower than VGG16 and VGG19
in the complexity, and a significant accuracy gain has also
been achieved. Because ResNet is simple and practical, many
methods of detection, segmentation and recognition are built
on the basis of ResNet50 or ResNet101, and ResNet is also
applied in AlphaGo zero [16].
III. C APSULENET
Compared with the traditional CNN model, CapsuleNet has
quite differences in the aspects of neuron structure, network
structure, and the way of propagation and distribution between
978-1-5386-6020-1/18/$31.00 ©2018 IEEE
layers. The above aspects of CapsuleNet are focused on to
analysis in this chapter.
A. The capsule neuron
Fig. 4. Parametric connection of capsule neurons
The key structure in CapsuleNet is the capsule neuron, as
shown in Fig.4. Unlike the traditional neuron, the input and
output parameters are vectors.The internal parameters are also
corresponding to vectors, and the activation functions used are
also different. The capsule neuron input and output vectors
represent instantiation parameters for a particular entity type.
The length of the vector represents the probability of existence
of the entity, and the direction of the vector represents the
existence of the attribute of the entity. The activation function
of capsule neurons is the squashing function, by which the
vectors are compressed and non-linearized. The same level of
capsules predict the instantiation parameters of higher level
capsules through matrix transformation.
B. CapsuleNet structure
Fig. 5. Structure of the CaptureNet mode [1]
At present, the structure of CapsuleNet is relatively simple.
As shown in Fig.5, the CapsuleNet model consists of three
layers [1]: Conv Layder, PrimaryCaps, and DigitCaps. The
Conv1 is a conventional convolution layer and uses the ReLU
activation function to implement local feature extraction in
pixels. The second layer, which is the PrimaryCaps layer,
serves as the input to the Capsule layer and constructs the
corresponding vector structure. The PrimaryCaps layer also
uses convolution to encapsulate multiple convolution units into
a new capsule unit, which is output as a vector and used as the
input to the next layer. The third layer, which is the DigitCaps
layer, is the output layer of the Capsule. It is connected to
the upper layer through a full connection. The calculation
and parameter update between the full connections uses the
dynamic routing algorithm [1]. The length of the final capsule
output vector is used as the probability distribution of the
entity.
212
 The 2018 International Conference On Control Automation & Information Sciences (ICCAIS 2018)
October 24-27, 2018, Hangzhou, China.

C. Propagation and distribution between Capsule units of parameter update, in addition to the traditional back-
The procedure of propagation and distribution between propagation algorithm to update the weight parameters, Capsu-
Capsule units is divided into three phases: linear combination, leNet adopts a dynamic routing algorithm to iteratively update
routing parameter update, and nonlinear function activation. the coupling coefficients in the forward propagation, which
_
The first stage is a linear combination. As in (1), u j|i is makes the network model easier to be trained and converge,
_
a linear combination of ui . The prediction vector u j|i was but brings a larger amount of calculation.
obtained by multiplying the vector ui of the i − th Capsule of
the previous layer output and the corresponding weight vector IV. C OMPASION A ND A NALYSIS OF R ESULTS OF
Wij . D IFFERENT DATASETS
_
u j|i = Wij ui (1)
In this section, the traditional classical CNN models and
In the second phase, the routing parameters are updated by CapsuleNet are used to conduct three groups of training
using the dynamic routing algorithm. The coupling coefficient experiments in different datasets. For the training results, this
denoted as cij is iteratively updated and determined by the chapter conducts corresponding comparation and analysis.
dynamic routing process. The dynamic routing process is:
First, bij is initialized by 0. As in (2), bij is depended on
the positions and types of the two capsules, and is updated A. Introduction to datasets
iteratively through the consistency between the current output This training experiment selected three different image
_
vj of each capsule j and the prediction vector u j|i of the datasets:MNIST [17], small NORB [18], and CIFAR-10 [19].
previous layer of the capsule i. Equation (3) is the softmax The three kinds of datasets are described in detail below.
function. We can see that the sum of the coupling coefficient 1) MNIST database of handwritten digits: The MNIST
between Capsule i and all Capsules in the same level is 1. dataset is a hand-written digital database consisting of 250
_
According to (4), the prediction vector u j|i and the corre- different handwritten digits 0-9, the training set with 60,000
sponding coupling coefficient cij are weighted sum to obtain handwritten digital images, and the test set with 10,000
the input vector sj of the next layer of the unit. According handwritten digital images. Each picture in the dataset is a
to (5), the output vector vj of the next layer of the Capsule blackwhite picture, the number of channels is 1, and the picture
unit is obtained. Output vectors are used to update bij , so pixel size is 28*28. An example diagram is shown in Fig.6.
the coupling coefficient cij is also updated to complete the
dynamic routing process. The algorithm is easy to converge
through 3 iterations.
_
bij =bij + u j|i vj (2)
exp(bij )
cij = P (3)
exp(bik )
k
X _
sj = cij u j|i (4)
i
2 Fig. 6. Example of MNIST [17]
ksj k sj
vj = 2 ks k (5)
1 + ksj k j
2) V1.0-small NORB: The dataset is a 3D database con-
The third stage is the activation process of nonlinear func- taining 50 toys in five categories:four-legged animals, human
tions. Since the length of the output vector of the Capsule figures, airplanes, trucks, and cars. The training and test
unit represents the probability that an entity exists, its value set contain 48,600 images. Each image has two channels
must be between 0 and 1. In order to achieve this vector and is composed of two grayscale images taken at different
compression, and complete the nonlinear activation function of angles. The pixel size of the image is 96*96. And the V1.0-
the Capsule layer, the activation function used by CapsuleNet small NORB is commonly used in 3D object recognition
is squashing function expressed as (5). The nonlinear function experiments. The example image is shown in Fig.7.
ksj k2
is divided into two parts. The former part is 1+ks 2 to
jk
complete the compression reassignment of the input vector
s
sj . The second part is ksjj k to unitize the input vector sj . This
function therefore preserves the direction of the input vector
and compresses the length of the input vector to the interval
[0, 1).
In general, CapsuleNet uses vectors instead of scalars of Fig. 7. Example of V1.0-small NORB [18]
traditional CNNs to obtain more information links. In terms

978-1-5386-6020-1/18/$31.00 ©2018 IEEE 213

 The 2018 International Conference On Control Automation & Information Sciences (ICCAIS 2018)
October 24-27, 2018, Hangzhou, China.
3) CIFAR-10: The dataset which is a color picture set
contains 10 categories:airplanes, cars, birds, cats, deer, dogs,
frogs, zebras, boats and trucks. The training set contains
50,000 images and the test set contaions 10,000 images. Each
picture has 3 channels and the picture pixel size is 32*32. The
example image is shown in Fig.8.
Fig. 8. Example of CIFAR-10 [19]
B. Experimental background and results
For the traditional CNN, this section selects the classic 16-
layer VGG16, 42-layer Google Inception V3, and 152-layer
ResNet, and contrast training experiment with CapsuleNet.
These four network models are trained by the same dataset.
Each group of experiments ensures the consistent training
parameters. The training parameters and results are shown
below.
1) The training experiment on the MNIST dataset: The
MNIST dataset is used for training in this experiment. The
parameters are set as follows: batch-size is 64, the number
of training steps is 50000, learning rate is 0.0001, learning
optimization algorithm is Adam. The training experiment
results are shown in Fig.9 below.
2) The training experiment on V1.0-small NORB dataset:
The V1.0-small NORB dataset is used for training in this
experiment. The parameters are set as follows: batch-size is
32, the number of training steps is 30000, learning rate is
0.0001, learning optimization algorithm is Adam. The training
experiment results are shown in Fig.10 below.
3) The training experiment on CIFAR-10 dataset: The
CIFAR-10 dataset is used for training in this experiment. The
parameters are set as follows: batch-size is 64, the number
of training steps is 150000, learning rate is 0.0001, learning
optimization algorithm is Adam. The training experiment
results are shown in Fig.11 below.
978-1-5386-6020-1/18/$31.00 ©2018 IEEE 214
Fig. 9. Comparison of training results of the four models on the MNIST
dataset
Fig. 10. Comparison of training results of the four models on the V1.0-small
NORB dataset
After training experiments, the corresponding training mod-
els were got in this paper, and then test experiments were
carried out on the training models. The test results show that
the accuracy of the test set has declined as a whole, but the
overall curve trend and performance are consistent with the
training experiment results.
C. Analysis of experimental results
According to the above three groups of experimental results,
the training results of the traditional CNN models and Cap-
suleNet perform well on the two gray image datasets MNIST
and small NORB. In 10K steps, the training accuracy rate can
reach 98%. And whether it is in the convergence speed, or
in training accuracy, CapsuleNet is better than the traditional
CNN models. For the CIFAR-10 RGB color picture dataset,
the overall convergence speed of the four neural network
models is slower than that of the gray picture dataset. However,
the traditional CNN models show good performance. Within
40K steps, the training accuracy rate of the traditional CNN
models reaches 95%. The CapsuleNet is about 140K steps,
and the training accuracy rate is only about 90%.
The reasons are analyzed as follows: First, CapsuleNet tends
to explain everything in the picture, and it can perform well for
MNIST and small NORB datasets with a single background.
 The 2018 International Conference On Control Automation & Information Sciences (ICCAIS 2018)
October 24-27, 2018, Hangzhou, China.
Fig. 11. Comparison of training results of the four models on the CIFAR-10
dataset
For the CIFAR-10 dataset, the background of the picture is
chaotic, and it changes too much for the fixed size model.
Second, the CapsuleNet model is in its infancy. Compared
with VGG16, inception-v3 and ResNet CNN models, the
CapsuleNet model structure only with three layers is more
simple. Although the vector in the network can express more
information and attributes of images, the number of network
layers is less. For complex picture information, the feature
extracted by learning model is not enough. Therefore, the
CapsuleNet model performs poorly on the CIFAR-10 and other
RGB datasets.
V. C ONCLUSION
This paper compares the classical CNN models (VGGNet,
Google Inception Net, and ResNet) with CapsuleNet in the
aspects of network structure, parameter update and propaga-
tion, principle analysis and training results. The traditional
CNN models show powerful feature extraction and learning
ability with the increasing of network depth and improvement
of algorithm. While the traditional pooling is discarded in
CapsuleNet to preserve precise location information. In the
form of vector, the capsule unit maintains the relationship
between feature information and implements the encapsulation
of the entity, which makes up for the lack of the CNN
models. The experimental results show that a simple three-
layer CapsuleNet gets a significant effect on the gray image
with a single background. However, for the RGB image with
a complex background, the features of the image cannot be
completely learned by CapsuleNet to achieve high-precision
classification. Therefore, CapsuleNet needs to be improved in
both the model structure and the parameter updating algorithm,
and it will also promote the advancement of neural network.
ACKNOWLEDGMENT
The paper is sponsored by National Natural Science Foun-
dation of China (No. 61374159, No. 61374023), Aviation Sci-
ence Foundation (No. 20165153034), Natural Science Foun-
dation of Shaanxi province (No.2018MJ6048) and the Seed
Foundation of Innovation and Creation for Graduate Students
in Northwestern Polytechnical University (No. ZZ2018149).
978-1-5386-6020-1/18/$31.00 ©2018 IEEE
R EFERENCES
[1] S. Sabour, N. Frosst, and G. E. Hinton, “Dynamic Routing Between
Capsules,” arXiv:1710.09829v2 [cs.CV] 7 Nov.2017.
[2] D. E. Rumelhart, G. E. Hinton, and R. J. Williams, “Learning representa-
tions by back-propagating errors,” Nature, vol. 323(6088), pp. 533-536,
1986.
[3] A. S. Razavian, H. Azizpour, J. Sullivan, and S. Carlsson, “Cnn features
off-the-shelf: An astounding baseline for recognition,” IEEE Conference
on Computer Vision and Pattern Recognition. IEEE Computer Society,
pp. 512-519, 2014.
[4] W. Liu, D. Anguelov, D. Erhan, C. Szegedy, S. Reed, C. Y. Fu, and A.
C. Berg, “SSD: Single shot multibox detector,” European Conference
on Computer Vision. Springer, Cham, pp. 21-37, 2016.
[5] J. Long, E. Shelhamer, T. Darrell, “Fully convolutional networks for
semantic segmentation,” IEEE Conference on Computer Vision and
Pattern Recognition. IEEE Computer Society, pp. 3431-3440, 2015.
[6] Y. Lecun, L. Bottou, Y. Bengio, and P. Haffner, “Gradient-based learning
applied to document recognition,” Proceedings of the IEEE, vol. 86(11),
pp. 2278-2324, 1998.
[7] A. Krizhevsky, I. Sutskever, and G. E. Hinton, “ImageNet Classification
With Deep Convolutional Neural Networks,” International Conference
on Neural Information Processing Systems. Curran Associates Inc. pp.
1097-1105, 2012.
[8] K. Simonyan and A. Zisserman, “Very deep convolutional networks for
large-scale image recognition,” In ICLR, 2015.
[9] C. Szegedy, W. Liu, Y. Jia, P. Sermanet, S. Reed,D. Anguelov, D. Erhan,
V. Vanhoucke, and A. Rabinovich, “Going deeper with convolutions,”
In Proceedings of the IEEE Conference on Computer Vision and Pattern
Recognition, pp. 1-9, 2015.
[10] K. He, X. Zhang, S. Ren, and J. Sun, “Deep residual learning for
image recognition,” In IEEE Conference on Computer Vision and Pattern
Recognition, pp. 770-778, 2016.
[11] G. Huang, Z. Liu, L. V. D. Maaten, and K. Q. Weinberger, “Densely
Connected Convolutional Networks,” arXiv: 1608.06993v5 [cs.CV] 28
Jan 2018.
[12] M. Lin, Q. Chen, and S. Yan, “Network In Network,” Computer Science,
2013.
[13] S. Ioffe, and C. Szegedy, “Batch normalization: Accelerating deep
network training by reducing internal covariate shift,” International
Conference on Machine Learning, pp. 448-456, 2015.
[14] C. Szegedy, V. Vanhoucke, S. Ioffe, J. Shlens, and Z. Wojna, “Rethinking
the inception architecture for computer vision,” arXiv preprint arXiv:
1512.00567, 2015.
[15] C. Szegedy, S. Ioffe, V. Vanhoucke, and A. Alemi, “Inception-v4,
Inception-resnet and the impact of residual connections on learning,”
arXiv: 1602.07261v2 [cs.CV], 23 Aug 2016.
[16] D. Silver, J. Schrittwieser, K. Simonyan, I. Antonoglou, A. Huang, A.
Guez, T. Hubert, L. Baker, M. Lai, and A. Bolton, “Mastering the game
of go without human knowledge,” Nature, vol. 550(7676), pp. 354-359,
2017.
[17] Y. LeCun, and C. Cortes, “The mnist database of handwritten digits,”
1998.
[18] Y. LeCun, F. J. Huang, and L. Bottou, “Learning methods for generic ob-
ject recognition with invariance to pose and lighting,” IEEE Conference
on Computer Vision and Pattern Recognition. IEEE Computer Society,
vol. 2, pp. II-104, 2004.
[19] M. D. Zeiler and R. Fergus, “Stochastic pooling for regularization of
deep convolutional neural networks,” arXiv preprint arXiv: 1301.3557,
2013.
215