Real-time Marine Animal Images Classification by

Embedded System Based on Mobilenet and Transfer

Learning
Xuefeng Liu
College of Automation &
Electronic Engineering
Qingdao University of Science
and Technology
Country Qingdao, China
nina.xf.liu@hotmail.com
Zhenqing Jia
College of Automation &
Electronic Engineering
Qingdao University of Science
and Technology
Country Qingdao, China
jia101728@163.com
Xiaoke Hou
College of Automation &
Electronic Engineering
Qingdao University of Science
and Technology
Country Qingdao, China
574741423@qq.com
Min Fu
College of Information Science &
Engineering
Ocean University of China
Country Qingdao, China
fumin@ouc.edu.cn

Li Ma
College of Information Science &
Engineering
Ocean University of China
Country Qingdao, China
sugemali@hotmail.com
Qiaoqiao Sun
Institute Fresnel
Ecole Centrale de Marseille
ECM, Marseille, France
qiaoqiao.sunny@foxmail.com

Abstract—In marine aquaculture, the classification and I. INTRODUCTION

identification of marine animals image is one of the important
research, which can help monitor the growth and fishing of marine
animals, as well as water conditions. Generally, the video and
image information of marine animals transmitted by underwater
cameras are identified by a human or a computer, which is hardly
to do real-time processing. To classify the marine animal images
efficiently and in real time, a method combining an embedded
system and deep learning based on MobileNetV2 and transfer
learning is proposed in this paper. Firstly, the marine animal
images are colleced by an underwater robot equipped with an
embedded device. Next, a MobileNetV2 model based on
convolutional neural network (CNN) is constructed according to
the marine animal images, which can ensure the real-time
requirement. Thirdly, transfer learning is used to further improve
the classification. Then, the model can be trained by the collected
marine animal images. Finally, the trained model could be
downloaded to the embedded device and real-time classify the
marine animal images under water. To evaluate the performance
of the proposed method, InceptionV3 and MobilenetV1 models are
used for the comparison in the experiments, and the identification
accuracy rate and average classification time are calculated. The
results show that the MobilenetV2 model plus transfer learning
could be a better choice for the real-time classification of the
marine animal images than the other two considered models.
Furthermore, the size of the MobilenetV2 model is only about 40M
and suitable for the embedded device.
Keywords—CNN, parameter transfer, fine tuning, inception,
underwater image
Efficient classification and identification the marine animals
from the underwater images can help monitor the growth of
marine animals and water conditions. [1,2]. Manual recognition
is time consuming and not accurate enough. Therefore, a high-
precision and real-time method is necessary for marine animal
images classification.
At present, some methods of machine learning have
achieved outstanding results in marine animal image
classification. The BP neural network is used to fish recognition
[3]. A maximum probability partial ranking method based on
sparse representation (SRC-MP), is proposed for real-world fish
recognition and identification[4]. A linear SVM classifier with
spatial pyramid pooling (SPP) is used for the classification on a
real-world fish recognition dataset [5]. These methods can
achieve high identification accuracy, but they only extract for
one or several features of the image and the requirements for the
data set are high. If the number of images is insufficient, the
accuracy will be reduced.
With the improvement of computer performance, deep
learning shows great potential in large data processing.
Convolutional neural network (CNN) is one of the most popular
neural networks for image classification and target detection[6].
The CNN extracts the features of the original image through the
convolution layer and the pooling layer. The training of CNN is
to improve the recognition accuracy by continuously updating
the weights [7]. In recent years, it has been widely used in face
recognition[8], text recognition[9], license plate detection[10],
medical image[11] and other fields. Transfer learning is a

The work was supported by the Nation Natural Science Foundation of

China (Grant No. 61401244, 61773277).

978-1-7281-1450-7/19/$31.00 ©2019
Authorized licensed use limited to: IEEE
University of Tasmania. Downloaded on March 12,2021 at 23:12:55 UTC from IEEE Xplore. Restrictions apply.
method of transferring weights from a trained network to
another untrained network. It only needs a small amount of the
training samples of the source and target data .
Therefore, we propose a method of the marine animal image
classification based on transfer learning and MobileNet based
on CNN. This method uses MobileNetV2 as pre-training
models. Some parameters are transferred from the deep learning
model trained by the source data in the ImageNet dataset. The
other parameters are trained by the images captured by the
underwater robot. Then, we compare the accuracy and
recognition time in the experiments to evaluate the performance
of the proposed method.
II. CONVOLUTIONAL NEURAL NETWORK AND TRANSFER
LEARNING
A. Convolutional Neural Network
Convolutional neural network (CNN) is a kind of deep
feedforward neural network. It imitates the hierarchical structure
of biological neural networks. The structure of CNN model
consists of input layer, convolution layer, pooling layer, fully
connection layer and output layer[12]. CNN extracts image
information layer by layer through convolution kernel. In recent
years, the deep convolution neural network model trained by
massive sample sets has reached an unprecedented high level in
speed and recognition accuracy.
B. IncpetionV3
InceptionV3 network is developed from GoogleNet network
[13]. Before the advent of GoogleNet, the networks performance
was improved by increasing the number of network layers and
neurons, but there were the disadvantages of the huge amount of
calculation and the phenomenon of over-fitting. Therefore, the
main idea of InceptionV3 model is to replace the large size
convolution kernel with the small size convolution kernel. For
example, the 5×5 convolution kernel is decomposed into two-
layer 3×3 convolution kernel. The 3×3 convolution kernels are
decomposed into the form of 1×3 and 3×1 convolution kernels
[14]. In addition, batch normalization (BN) layer is added to
normalize the output of each layer to the gaussian distribution
from 0 to 1.
C. MobileNetV1
MobileNetV1 is an efficient model for mobile and embedded
devices. Based on streamlined architecture, MobileNets uses
depthwise separable convolutions to construct lightweight deep
neural networks [15]. Depthwise separable convolutions include
two parts: depthwise convolution and pointwise convolution.
The convolution kernel parameters of traditional
convolution are DK×DK×M×N, in which DK×DK is the size of
convolution core, M is the number of input channels and N is
the number of output channels. The convolution kernel
parameter in depthwise convolution is DK×DK×1×M, where M
cores of DK×DK and corresponding channels of input
characteristics are convoluted. In pointwise convolution, the
convolution kernel parameter is 1×1×M×N.
Standard convolutions have the computational cost of:
DK×DK×M×N×DF×DF. Depthwise separable convolutions cost:
DK×DK×M×DF×DF+M×N×DF×DF. We get a reduction in
computation:
Authorized licensed use limited to: University of Tasmania. Downloaded on March 12,2021 at 23:12:55 UTC from IEEE Xplore. Restrictions apply.
× × × × ×
= + (1)
× × × × × × ×
When using 3×3 convolution kernel, Depthwise separable
convolutions can reduce the computational complexity to 1/8-
1/9 of the original one. So MobileNetV1 is essentially designed
based on this decomposition hypothesis. MobileNet's separable
convolution module compresses both parameters and
computational complexity. It enables modern mobile devices to
give full play to the computing power of CPU and GPU, and
speeds up image recognition without the accuracy reduction.
D. MobileNetV2
MobileNetV2 is also a lightweight convolutional neural
network. On the one hand, MobileNetV2 proposes the concept
of inverted residual block based on MobileNetV1. The
traditional residual block workflow is: (1) using the 1×1
convolution kernel to reduce the dimensionality of the high-
dimensional features; (2) applying the 3×3 convolution kernel to
filter; (3) using the 1×1 convolution kernel to improve the
dimensions and rectified linears units (ReLU) being included in
these convolution operations; (4) adding the output features (the
input of the next layer) in element wise manner [16]. But, in the
workflow of inverted residual block, 1×1 convolution kernel is
used to increase the dimension of low-dimensional features
(excluding ReLU), and then 1×1 convolution kernel is applied
to reduce the dimensions of features. On the other hand,
mobileNetV2 uses linear bottleneck instead of ReLU to avoid
damage to features. The purpose of the improvements is to avoid
information loss and increase the expressive ability of the
model. Fig.1. shows the structural changes from MobileNetV1
to MobileNetV2 [16].
MobileNetV1 MobileNetV2
Fig.1 Basic structural changes from MobileNetV1 to
MobileNetV2.
E. Transfer Learning
The research of transfer learning focuses on reusing the
knowledge acquired in the source domain and solving new
problems in the target domain with better solutions [17]. The
methods of transfer learning can be divided into transfer sample,
transfer feature and transfer parameter [18]. When the data of
 Fig.2 Classification based on transfer learning.
source domain and target domain are very similar, sample
migration fuses source samples and target samples, then adjusts
the source domain weights to get the target domain weights.
Feature migration finds feature association between source and
target domains by reconstructing features, minimizing the
difference between them. Parameter migration is to share the
parameters between the source and the target domains, and
automatically adjust the weights to get the optimal results [19].
Transfer learning can alleviate the problem of insufficient
data. Therefore, transfer learning has gradually become the
preferred technology for artificial intelligence (AI) projects with
insufficient data or computational power. As a branch of
machine learning, transfer learning is increasingly integrated
with neural networks. For image recognition, migration learning
is a method to solve the problem of less labeled sample data and
high cost of model training. Pre-training model is the model that
has been trained by large data sets. We find some network layers
which can reuse feature vectors, and then transfer these network
layers and parameters to train networks with smaller data sets
[20]. Therefore, training costs are reduced and resource
utilization is improved.
III. IMAGE CLASSIFICATION BASED ON TRANSFER LEARNING
A. Data Enhancement
In this paper, we would like to: (1) train model from small-
scale marine animal’s data sets. (2) do the real-time image
classification by downloaded the model to embedded devices.
Because the data set is small in scale, we first enhance the data
set. Data enhancement is a method to improve the overall
performance of training network when the original image data
set is insufficient. The main methods of data enhancement are
rotation, flipping, translation, zooming, noise addition, etc. In
this paper, we combine rotation, flipping and translation to
expand the sample space.
B. Transfer Parameter
To solve the problem of insufficient data sets, transfer
learning is introduced for the classification in this paper. In
convolutional neural networks, MobileNet is an efficient model
for mobile and embedded devices. Therefore, we propose a
transfer learning method based on MobileNet model. In order to
Authorized licensed use limited to: University of Tasmania. Downloaded on March 12,2021 at 23:12:55 UTC from IEEE Xplore. Restrictions apply.
verify the effectiveness of the MobileNet model, the Inception
V3 model combining with the transfer learning is evaluated in
the experiment for comparison.
Firstly, we enhance the data of the images collected by the
underwater vehicle. Then, the method of parameter transfer
learning is applied to small-scale marine animal data sets. We
use the model parameters of InceptionV3, MobileNetV1 and
MobileNetV2, which have been trained by the large samples of
source data, to train the network of target data. In order to
improve the training accuracy, we adopt a fine-tuning method in
parameter transfer. If it is the underlying structure of the
convolution module, its parameters could be kept. If it is a high-
level convolution module close to the classifier, these modules
are set as trainable modes, including matrix weight, bias term
and other regularization term coefficients. Then the model can
be used for the target data and the optimum value can be
obtained by adjusting the parameters in a small range.
Taking the MobileNetV1 model as an example, the specific
transfer learning method is shown in Fig. 2. According to the
data set category, we replace the full connection layer of the
source model with the 7-class Softmax classifier. According to
the structure of different networks, the weight of high-level
convolution module is set to be trainable for adaptive
adjustment. Then the full connection layer of the model is
modified. Through experiments, it is found that InceptionV3
model can get the highest accuracy of validation set when
training from the level 175, MobileNetV1 is from level 122 and
MobileNetV2 is from level 130.
IV. EXPERIMENT
A. Data Set
Some of the data used in the experiment are obtained by
underwater cameras for underwater marine animals and others
are collected by the Internet. The whole data set is divided into
seven categories: fish, shrimp, scallop, crab, lobster, abalone and
sea cucumber. Each category ranges from 1000 to 1400 sheets,
totaling 8455 sheets. 80% of them are training set and 20% are
validation set. We enhanced the training set data. Each original
image is generated into three deformed images by three
processing methods: rotation, translation and flipping. The
training set can be expanded to 27056 pictures. In addition, for
the embedded devices, the testing data are seven types of images
collected by the network, each type of image includes five sizes,
each size has 10 pictures, a total of 350. When selecting training
data, a part of the image containing non-target samples is used
to simulate random noise in order to improve the generalization
ability of the model. Because the resolution of the image taken
by underwater camera is different from that collected by
network, we normalize the original image. According to the
requirements of different models, reshape is 224 × 224 or 299 ×
299.
Fig.3 Classification Fig.4 Classification loss of
In this paper, the experimental environment adopts a Ubuntu
16.04 version computer. It carries two GTX 1080ti video cards. accuracy of testing data validation data
The experiment could be completed in the framework of
TensorFlow + Keras. In addition, the embedded device is Jeston-
TX2 produced by NVIDIA. The device has a NVIDIA Pascal From Figure 3 and Figure 4, it can be seen that the accuracy
GUP, 256 NVIDIA CUDA cores, dual-core Denver 2, four-core of validation set increases with the number of iterations in the
ARM Cortex-A57 processor. All these configurations are network training process. The loss rate of validation set
designed to better adapt to convolution operations and improve gradually stabilizes in a certain interval with the increase of
the speed of operations. iterations. According to Table 1, Inception V3 model has the
best training set accuracy, but this does not mean that the best
B. Experimental Process verification set accuracy can be obtained. Under the same
Training Convolutional Neural Network conditions, MobileNetV2 model obtained the highest accuracy
We have selected the bath_size = 16 and epoch = 200 to train of validation set : 92.89%.
the network model. After the training, the values of accuracy and C. Classification Time Test in the Embedded Device
loss rate are recorded. The formulas for calculating the accuracy
and loss rate is as follows:
acc = (2)
where n represents the number of correctly classified
images and N represents the total number of images. Loss is
calculated by the following formulas with yi being the predicted
value and y_hati being the original target value：

Softmax( )=∑ (3)

Loss(y, y )=− ∑ _ℎ × log ( ( )) (4)

The accuracy of training set, the loss rate of training set, the
accuracy rate of validation set and the loss rate of validation set
are illustrated in Table 1. The change process of the accuracy
rate and the loss rate of the validation set in the training process
are shown in Fig. 3 and Fig. 4.

TABLE I. ACCURACY AND LOSS OF TRAINING SET AND VALIDATION SET

FOR EACH MODEL

Types of data Fig. 5 Some real-time classification results. On the top of each
Accuracy of Loss of Accuracy of Loss of
Model
training training validation validation
image, there are species, accuracy and classification time.
data data data data
InceptionV3 99.73% 0.0072 91.13% 0.372
MobileNetV1 99.29% 0.0199 89.92% 0.5113
MobileNetV2 99.58% 0.0085 92.89% 0.3165

Authorized licensed use limited to: University of Tasmania. Downloaded on March 12,2021 at 23:12:55 UTC from IEEE Xplore. Restrictions apply.
 TABLE II . CLASSIFICATION ACCURACY AND TIME OF THE MARINE ANIMAL IMAGES BY EACH MODEL IN THE EMBEDDED SYSTEM

Species
Model
Abalone Crab Fish Lobster Scallop Sea_cucumber Shrimp Average
Time(s) 0.091 0.116 0.174 0.181 0.139 0.114 0.135 0.1361
InceptionV3
Accuracy 96.8% 95.6% 100% 98.2% 92.2% 92.6% 91.6% 93.6%
Time(s) 0.046 0.055 0.077 0.046 0.032 0.089 0.051 0.0569
MobileNetV1
Accuracy 96.2% 94.8% 100% 93.6% 96.4% 89.6% 86.8% 92.6%
Time(s) 0.052 0.052 0.052 0.073 0.053 0.075 0.045 0.0578
MobileNetV2
Accuracy 96.6% 96.4% 99.2% 96.8% 100% 89.6% 92.4% 95.0%

The trained model is downloaded into the embedded device
Jeston TX2 and the test data is classified. Four classification
results are listed in Fig. 5. The average classification time of
each model are shown in Table 2.
From Table 2, the average classification time of the
InceptionV3, MobilenetV1 and MobilenetV2 models for image
are: 0.136s, 0.0569s and 0.0578s respectively. The
MobileNetV1 model has the best performance for classifying
runtime and the speed of MobileNetV2 is close to that of
MobileNetV1.
D. Discussion of the Experimental Results
In this experiment, three models are selected for transfer
learning and the parameters of marine animal images are
retrained to classify the images in the embedded devices. The
experimental results show that the MobileNetV2 model trained
by transfer learning has the best validation set accuracy of
92.89%. In terms of image classification speed, The
MobileNetV1 model achieved the least average classification
time. But MobilenetV2 is only 0.001 seconds slower than
MobileNetV1. Therefore, considering both the classification
accuracy and computing time, it can be concluded that the
MobilenetV2 model plus transfer learning could be a better
choice for the real-time classification of the marine animal
images than the other two considered models. Furthermore, the
size of the MobilenetV2 model is only about 40M and much
suitable for the embedded device.
REFERENCES
[1] Yajuan Wei, “Study of Zooplankton Automatic Recognition Method for
Dark Field Image,” Ocean University of China, 2013.
[2] Xi Qiao, “Sea cucumber identification in real-time based on underwater
machine vision techinque,” China Agricultural University, 2017.
[3] Peng Wan, Hailong Pan, Changjiang Long, et al, “Design of the on-line
identification device of freshwater fish species based on machine vision
technology,” Food and Machinery, vol. 28, pp.164-167, 2012.
[4] Yihao Hsiao , ChaurChin Chen , Sunin Lin and Fangpang Lin, “Real-
world underwater fish recognition and identification, using sparse
representation,” Ecological Informatics, vol. 23, pp.13-21, 2014.
[5] Hongwei Qin, Xiu Li, Jian Liang , Yigang Peng and Changshui Zhang,
“DeepFish: Accurate Underwater Live Fish Recognition with a Deep
Architecture,” Neurocomputing, 2015.
[6] Christian Szegedy, Wei Liu, Yangqing Jia, et al, “Going deeper with
convolutions,” Proceedings of the 2015 IEEE Conference on Computer
Vision and Pattern Recognition, pp. 1-9, 2015.
[7] A. I. Kukharenko and A. S. Konushin, “Simultaneous classification of
several features of a person’s appearance using a deep convolutional
neural network,” Pattern Recognition and Image Analysis, vol. 25, pp.
461-465, 2015.
[8] Siyue Xie and Haifeng Hu, “Facial expression recognition with FRR-
CNN,” Electronics Letters, vol. 53, pp. 235-237, 2017.
[9] Yichao Wu, Fei Yin and Chenglin Liu, “Improving handwritten Chinese
text recognition using neural network language models and convolutional
neural network shape models,” Pattern Recognition, vol. 65, pp. 251-264,
2017.
[10] Lele Xie, Tasweerie, Lianwen Jin , Yuliang Liu and Sheng Zhang, “A
New CNN-Based Method for Multi-Directional Car License Plate
Detection,” IEEE Transactions on Intelligent Transportation Systems,
vol. 19, pp. 507-517, 2018.
[11] Tajbakhsh Nima, Shin JaeY, Gurudu, Suryakanth R and et al,
“Convolutional Neural Networks for Medical Image Analysis: Fine
Tuning or Full Training?” IEEE Transactions on Medical Imaging, vol.
35, pp. 1299-1312, 2016.
[12] Keqing Zhu, Jie Tian and Haining Huang, “Underwater object Images
Classification Based on convolutional neural network,” 2018 IEEE 3rd
International Conference on Signal and Image Processing (ICSIP), pp.
301-305, 2018.
[13] Christian Szegedy , Wei Liu, Yangqing Jia , et al, “Going Deeper with
Convolutions,” Conference on Computer Vision and Pattern Recognition
(CVPR), pp. 1-9, 2014.
[14] Szegedy Christian, Vanhoucke Vincent, Ioffe Sergey and et al,
“Rethinking the inception architecture for computer vision,” Proceedings
of the 2016 IEEE Conference on Computer Vision and Pattern
Recognition, pp. 2818-2826, 2016.
[15] Howard A G, Zhu Menglong, Chen Bo and et al. “MobileNets: Efficient
Convolutional Neural Networks for Mobile Vision Applications,” http://
arxiv.org/abs/1704.04861, 2017.
[16] Sandler M, Howard A, Zhu Menglong and et al, “MobileNetV2: Inverted
Residuals and Linera bottlenecks,” http ： //arxiv.org/abs/1801.04381,
2018.
[17] Xin Sun , Junyu Shi , Lipeng Liu and et al, “Transferring deep knowledge
for object recognition in Low-quality underwater videos,”
Neurocomputing, vol. 275, pp. 897-908, 2017.
[18] HooChang Shin, Roth, Holger Roth, Mingchen Gao and Ronald M
Summers, “Deep convolutional neural networks for computer-aided
detection: CNN architectures, dataset characteristics and transfer
learning,” IEEE Transactions on Medical Imaging, vol. 35, pp. 1285-
1298, 2016.
[19] Ling Shao, Fan Zhu and Xuelong Li, “Transfer Learning for Visual
Categorization: A Survey,” IEEE Transactions on Neural Networks and
Learning Systems, vol. 26, pp. 1019-1034, 2015.
[20] Zhongling Huang, Zongxu, Pan and Bin Lei, “Transfer learning with deep
convolutional neural network for SAR target classification with limited
labeled data,” Remote sensing, vol. 9, pp. 1-21, 2017.

Authorized licensed use limited to: University of Tasmania. Downloaded on March 12,2021 at 23:12:55 UTC from IEEE Xplore. Restrictions apply.