ORMEDIAN RESEARCH INSTITUTE

TECHNICAL REPORT ON DENSENET ARCHITECTURE

RESEARCH-BASED ON: IMAGE RECOGNITION/CLASSIFICATION

BY

ADEKOLA OLUWASEUN OLUWAFEMI

DECEMBER 16, 2022.

i
MEANING OF DENSENET

DenseNet: The term DenseNet is the short form of densely connected Convolutional Network
(DenseNet). This is a type of convolutional neural network that utilizes Dense Blocks to establish
dense connections between layers that are present in the network. The configuration of the
DenseNet is done in such a way that every layer except the very first layer in the network receives
additional inputs from preceding layers and each of these layers passes on its feature maps to all
concomitant layers. This previously explained technique is generally utilized in DenseNet
architecture to maintain the feed-forward nature and correct matching feature map sizes. Unlike
the traditional convolutional networks that usually have L layers with L connections, one between
𝐿(𝐿+1)
each layer and its concomitant layers. DenseNet architecture has . DenseNet offers
2
several compelling advantages that give it more overhead when compared to other convolutional
networks. In short, we can see DenseNet as the modification of the standard convolutional neural
network architecture.

ADVANTAGES OF DENSENET

(i) It minimizes the vanishing gradient problems that are associated with other deep
neural networks.
(ii) Reusability of features is possible with DenseNet.
(iii) The DenseNet approach ensures that there is a substantial reduction in the number of
parameters used.
(iv) It fortifies feature propagation.

1
REASONS FOR DENSENET

As the depth of the deep neural network increases, there is always an increase in the accuracy of
such network and it gets saturated, but rapid degradation occurs afterward to such network. This
is because the gradients of such a network tend to zero i.e. vanish as the number of layers
increases. Also, training is not only slower in deep networks but they are also prone to errors;
the drawback to the deep network is a result of the backpropagation. DenseNet was designed for
a specific purpose which is to strongly improve the waning accuracy caused by the vanishing
gradients in high-level neural networks. Vanishing gradients are one phenomenon affecting the
efficiency of deep networks and this is a result of the long path between the input layer and the
output layer which causes information to vanish before reaching its destination.

DenseNet has multiple dense blocks in its architecture. Every single dense block inside the
network has interconnected layers of the same feature map sizes.

TYPICAL STRUCTURE OF DENSENET WITH BLOCKS ARRANGEMENT

By definition of DenseNet, uses a dense connection of layers through some modules called dense
blocks. These blocks connect layers with matching feature map sizes together. In summary,
DenseNet can be regarded as an architecture that is divided into multiple dense blocks.
DesnseNet becomes a good approach to achieve more accuracy based on the fact that it has the
capability of reusing the output of each layer because a layer output parameter relies on output
features from the adjacent bottom layer and also relies on other bottom layers.

Figure 1: Architecture of DenseNet [5].

2
What is most important to take note of is the fact that “the input layer inside the DenseNet is the
concatenation of feature maps from the previous layers”.

Inputs of Lth in a dense network are the results of the concatenations of feature maps of previous

layers.

Figure 2: DenseNet Channel-wise Concatenation [7].

Each block i.e dense block in a densely connected convolutional network must have the same
feature map sizes before concatenation can be done between blocks. While keeping feature map
sizes to be the same, we must also bear in mind that the concept of down-sampling is an important
factor in a convolutional neural network that should be achieved by performing (convolution +
pooling) outside the dense blocks. The layers that do the convolution and pooling are called
transition layers.

In the DenseNet architecture, the transition layer used consists of a batch-norm layer, 1 x 1
convolution, and 2 x 2 average pooling respectively. It is this 1 x 1 convolution that performs the
down-sampling from some input features to some output features.

3
WHY THE NEED FOR DOWN-SAMPLING IN DENSENET

Generally speaking, down-sampling is an important term in the convolutional neural network,

because there's always a need to randomly remove observations from the majority class to prevent
its signal from dominating the learning algorithm, hence this will reduce the count of training
samples falling under the majority class. Down-sampling is applied by the transition layer through
batch normalization.

Figure 3: Pictorial Representation of Down-sampling in DenseNet [3].

In DenseNet architecture, the interrelationship between the output, dense blocks, and input is
expressed as:

Xl = Hl ([X0, X1… Xl-1])

Where [x0, x1, x2...] are the concatenation of the feature maps produced by [0, 1 …Lth].

4
SINGLE-DENSE BLOCK ARCHITECTURE

Layer_0 Layer_1 Layer 2

Figure 4: Typical Dense block and layers within it.

A dense block is simply one of the blocks in a DenseNet architecture. We have each layer connected
to every other layer in such a way that the feature map size remains the same. By observing Figure
4, Layer_0 receives some blue input features and performs a non-linear transformation to add the
purple features to the blue features. The output result of layer_0 then becomes the input to Layer_1,
this Layer_1 then performs a non-linear transformation to add green features to the purple and
blue features. The trend continues that way until the output of Layer_3 of the dense block is a
concatenation of blue, purple, green, and black as demonstrated in Figure 4. As this is being done,
the feature map size continues to grow as the new feature is being concatenated with the existing
features. The rate at which feature map size grows is regarded as the growth rate (k)

5
STEPS REQUIRED IN IMPLEMENTING DENSENET

i. Import necessary libraries and modules

ii. Create a bottleneck layer class: it's a huge computation requirement, this is where 1 x 1
convolution is introduced as the bottleneck layer, this 1x1 convolution is then used to
produce 4*k feature maps. 3 x 3 convolution is then performed to have more reduced
feature maps. This 3 x 3 convolution is performed on the 4*k size feature maps which were
first produced by the 1 x 1 convolution to improve computational efficiency. batchNorm2d,
ReLU, and Conv2d operations are performed in this block.

iii. Creation of Transition Layer Class: This is used to perform the down-sampling by
convolution and pooling on the output of the previous block.
iv. Creation of the DenseNet Class: This is where functions are created for stacking resulting
layers and blocks in an orderly manner by taking into consideration several operations
such as convolution, batch-normalization, relu, pooling, and other mathematical
operations to produce desired outputs based on parameters defined.
v. Transformation settings of training and testing dataset
vi. Data loading for training and testing
vii. Definition of relevant parameters e.g epochs, training sample, testing sample,
learning_rate, optimizer, and criterion, etc.
viii. Model training and testing
ix. Obtain results.

6
 CONCLUSION

The key benefit of DenseNet over ResNet is that it enables us to train our model in a better way
and model learn better things due to the concatenation used instead of the ordinary summing up
approach. Not only does DenseNet use few parameters, but also was able to eliminate
the issue of vanishing gradients to a very large extent.

The DenseNet121 Model has been implemented in this paper to test how accurate predictions can
be. The publicly available CIFAR-10 dataset has been utilized to perform training and testing. The
accuracy obtained was over 98%. Hence this shows that DenseNet is a better model compared to
other traditional convolutional neural networks. Implementation codes are available in the
appendix section of this paper.

7
 REFERENCES

[1] G. Huang, Z. Liu, and L. van der Maaten, "Densely Connected Convolutional Networks,"
2018.

[2] R. Kumar, K. Gredd and J. Schmidhuber, “Highway Networks,” 2015.

[3] Chandrakar, R., Raja, R. & Miri, R. Animal detection based on deep convolutional neural
networks with genetic segmentation. Multimed Tools Appl 81, 42149–42162 (2022).
https://doi.org/10.1007/s11042-021-11290-4

[4] K. He, X. Zhang, S. Ren, and Jian Sun, "Deep Residual Learning for Image Recognition,"
2015.

[5] Understanding and visualizing DenseNets. Available at

https://towardsdatascience.com/understanding-and-visualizing-densenets-7f688092391a

[6] Dense Blocks and Layers. https://www.analyticsvidhya.com/blog/2022/03/introduction-to-

densenets-dense-cnn/

[7] The generic architecture of dense block https://arthurdouillard.com/post/densenet/

[8] G. Larsson, M. Maire, and G. Shakhnarovich, "FractalNet: Ultra-Deep Neural Networks

without Residuals," 2017.

APPENDIX

https://colab.research.google.com/drive/13v7D_jxoi_uOD-dLFxhvKfB_RcB-Fqq_