2018 18th International Conference on Control, Automation and Systems (ICCAS 2018)

Oct. 17~20, 2018; YongPyong Resort, PyeongChang, GangWon, Korea

A Sea Creatures Classification Method

using Convolutional Neural Networks
Jonghyun Ahn*, Yuya Nishida, Kazuo Ishii and Tamaki Ura
Center for Socio-Robotic Synthesis, Kyushu Institute of Technology,
Kitakyushu, Japan
{ahn@brain, ynishida@lsse, ishii@brain, ura@lsse}.kyutech.ac.jp

Abstract: Recently, Autonomous Underwater Vehicles (AUVs) provide high resolution seafloor images to the
researchers. The provided images are classified by the researchers one by one, and this task increases the burden on the
researchers. In this research, we propose an automatic classification method for the sea creatures using‫ٻ‬convolutional
neural networks to provide the‫ٻ‬requested information, such as crab images, to researcher. The proposed method is
comprised image enhancement process, segmentation process and classification process. In the image enhancement
process, which is used Retinex model, the visibility of seafloor images is improved. For the candidate’s area selection,
the Saliency map is employed to extract the represented areas in seafloor images. In the classification process, the
selected areas are recognized based on the biological group. The total candidate detection rate was 64%, and the total
recognition accuracy was 67% by the evaluation.

Keywords: AUV mission, Seafloor imaging, Classification system

1. INTRODUCTION 2. PROPOSED METHOD

Recently, Autonomous Underwater Vehicles (AUVs)
are employed to observe seafloor environment [1-3].
AUVs equipped with camera and light system
photograph high resolution seafloor images in the
observation mission. After the observation mission is
completed, the researchers check the obtained images
which are contained various creatures. The number of
acquired images by AUV is more than 3,000 images
based on the 5-hour observation mission, and more than
20,000 images are obtained during one sailing (a week).
Currently, the researchers classify the provided seafloor
images one by one. Therefore, we propose an automatic
system for simplification of classification task. The
concept of system is shown in Fig.1.
In the Fig.1, AUV provides seafloor images which
are photographed during observation mission. The
images are processed, and requested information such as
sea creature’s images is provides to researchers. In this
paper, we present a part of the proposed system, and the
accuracy of presented system is evaluated.
Fig. 1 Concept of the proposed automatic
classification system for sea creatures
In the proposed system, the photographed creatures in
an input image are automatically classified, and
information such as creature images, total number of
specific creatures, is provided. In this paper, we present
a part of the proposed system, which is comprised of
several processes. The processes of system are shown in
Fig. 2.
In the image enhancement process, the seafloor image
visibility, which is affected by underwater light
attenuation, is improved. Then candidate areas are
selected by the segmentation process based on Saliency
map. The selected areas are classified based on the
biological group.
Input Image
Segmentation Classification
image enhancement
Fig. 2 Processes of the proposed method
2.1 Image enhancement process
The obtained seafloor images by AUVs are affected
by underwater light attenuation. Therefore, the seafloor
images look blue, and they have low intensity. The
obtained seafloor images by AUV are shown in Fig. 3.
The seafloor images were photographed by AUV:
TUNA-SAND [3] in the sea of Okhotsk. The images in
the Fig. 3 look dark and blue because each color has a
different attenuation rate. Furthermore, the seafloor
image has uneven illumination by difference of light
travel distance. Recently, Retinex model [4], which is
famous color constancy model, is employed to enhance
seafloor images [5,6]. In this image enhancement
process, Retinex model is used to solve problems of
underwater light attenuation.
The image enhancement process using Retinex model
is shown in Fig. 4.

978-89-93215-15-1/18/$31.00 ⓒICROS 420

Authorized licensed use limited to: University of Tasmania. Downloaded on March 12,2021 at 23:25:32 UTC from IEEE Xplore. Restrictions apply.
 In the Retinex model, an image comprises an
illumination factor and a reflectance factor. The
illumination factor, which is from light source, causes
low visibility because illumination contains light
attenuation. The reflectance is the uninfluenced color of
object by illumination. The Retinex model is;
ܴఒ ሺ‫ݔ‬ǡ ‫ݕ‬ሻ ൌ ߙሼŽ‘‰‫ܫ‬ఒ ሺ‫ݔ‬ǡ ‫ݕ‬ሻ െ Ž‘‰‫ܮ‬ఒ ሺ‫ݔ‬ǡ ‫ݕ‬ሻሽ ൅ ߚ.
where I represents input image, λ is color plane, (x, y) is
the image coordinates, L is the estimated illumination, α
is scaling gain parameter, β is scaling offset parameter,
and R is the output of process.
L is estimated by the a Gaussian filters, as follows;
ଵ ି൫௫ మ ା௬ మ ൯
‫ܮ‬ఒ ሺ‫ݔ‬ǡ ‫ݕ‬ሻ ൌ ‫ܫ‬ఒ ሺ‫ݔ‬ǡ ‫ݕ‬ሻ ‫כ‬ ή ‡š’ ቀ ቁ, (2)
ଶగఙ మ ଶఙ మ
The symbol “‫ ”כ‬represents convolution and σ is the
standard deviation of the distribution. The image
enhancement process using Retinex model is shown in
Fig.4, and the processed results are shown in Fig. 5.
(1) For the image enhancement, the used parameters are
σ: 128, α: 384, β: 128. The parameters are selected
manually based on image size and approximate
photographed altitude. The enhanced images show
improved color and contrast.

Fig. 3 Examples of obtained seafloor images by AUV

Fig. 4 Image enhancement process using Retinex model

Fig. 5 Processed results by image enhancement (Used parameters- σ: 128, α: 384, β: 128)

2.2 Segmentation process process using saliency map is shown in Fig. 6.

In the Fig.6, enhanced image is processed image by
In the segmentation process, the candidate areas
image enhancement process, Saliency map is shown
are selected using the saliency map which is a visual
object’s conspicuity, and the circles in selected
attention model of human. The saliency map
interesting area represent saliency area and red circle
estimates the conspicuity of objects as a
show most interesting area in an image. In this
two-dimensional map. The saliency map is composed
process, the areas, where the saliency value is larger
from the simple features such as intensity, color, and
than 0.5, are selected as the interest areas (Saliency
orientation based on Gaussian pyramid images for
map scale 0.0~1.0).
scale invariance. An example of segmentation

421

Authorized licensed use limited to: University of Tasmania. Downloaded on March 12,2021 at 23:25:32 UTC from IEEE Xplore. Restrictions apply.
2018 18th International Conference on Control, Automation and Systems (ICCAS 2018)
Oct. 17~20, 2018; YongPyong Resort, PyeongChang, GangWon, Korea

Fig. 6 An example of segmentation process using saliency map

2.3 Classification process Start

Na(i) = Number of
Recently, CNN (Convolutional Neural Networks) is
areas (BW(i))
shown high recognition rate in the application for Dset = Seafloor images
classification of objects. In this classification process, j == Na(i)
the AlexNet [7], which is name of a CNN, is used.
Ns = number of
Seafloor images j++
37,394 candidate area images, which are extracted
from 3,866 obtained seafloor images by AUV, are used
for training data. The flowchart for composing classifier i = 0, j = 0 Ac (j) =
is shown in Fig. 7. candidate area image
Dset is the data set which contains 3,866 enhanced using BW(i) and Dset(i)
i == Ns
seafloor images, Ns is number of seafloor images in Dset,
i denotes a running indexes for seafloor image number, j
represent a running indexes for interesting area number i++
in a seafloor image. The first loop is repeated until i ==
Ns is satisfied. Smap is saliency map using an image in
Smap(i) = Ga = Classification Ac
Dset and, BW is binary images using thresholding Saliency map(Dset(i))
(threshold parameter: 0.5), and Na is number of areas in
a BW. The second loop is repeated until j == Na(i) is C = Classifier
BW(i) = Thresholding composition(Ga)
satisfied. Ac, which is interesting area images, is (Smap(i))
classified by human, then the classified Ac are grouped
End
into each Ga. Ga is composed serval groups such as fish
group, crab group, and C, which denotes AlexNet
classifier, is composed using these groups. The Fig. 7 Flowchart of composing classifier
composed classifier is employed for sea creature
classification process. The flow chart of sea creature Start
classification is shown in Fig. 8.
I is enhanced seafloor image, Lc is represents j == Na
recognized label which is shown the biological group. I I = Seafloor image
is segmented using Saliency map, and selected area Ac is j++
classified using C. The process results are shown in Fig.
Smap =
9.
Saliency map(I)
The used classifier can be recognized 6 targets such Ac (j) = candidate area
as crab, fish, sea cucumber, starfish, seaweed and image using BW and I
BW =
seafloor. In the (a) image in Fig. 9, a channel rockfish is
Thresholding (Smap) Lc (j) =
photographed, and it is recognized as fish. In the (b)
image, a ray is taken, and it also is recognized as fish. Classification using C
N a=
However, (c) image is shown error of recognition,
Number of areas (BW)
which a crab is recognized as starfish.

End

Fig. 8 Flowchart of sea creature classification process

422

Authorized licensed use limited to: University of Tasmania. Downloaded on March 12,2021 at 23:25:32 UTC from IEEE Xplore. Restrictions apply.
 Fig. 7 Classification process result

3 EVALUATION As the future works, the recognition species will be

classified. The recognized areas as the fish will be
In the evaluation, the candidate detection, which is classified again for scientific name such as channel rockf
caused by segmentation process using Saliency map, ish or ray.
and creatures recognition accuracy are evaluated. For
the evaluation, 15 seafloor images are randomly ACKNOWLEDGMENT
selected. The result of evaluation is shown in Table 1. This research is supported by JST CREST
In the table 1, “No.” represents image number, “Establishment of core technology for the preservation
“Number of Targets” is number of creatures in an image. and regeneration of marine biodiversity and ecosystems”
“Number of Detected targets” denotes number of directed by Prof. Koike, and Hokkaido National
detected target by segmentation process, “Number of Fisheries Research Institute, Japan Fisheries Research
Detection error” is number of misdetected area and and Education Agency.
“Recognition accuracy” is recognition accuracy of
detected areas using CNN.
REFERENCES
[1] T. Ura, "Observation of Deep Seafloor by
The total candidate detection rate is 64%, and the
Autonomous Underwater Vehicle." Indian
total recognition accuracy is 67%.
Journal of Geo-Marine Sciences, vol.42,
Table 1 Result of the proposed classification method pp.1028-1033, 2013.
[2] T. Maki, A Kume and T. Ura, "Volumetric
Number of
No.
Recognition mapping of tubeworm colonies in Kagoshima
Detected Detection
Targets accuracy Bay through autonomous robotic surveys."
targets error
1 2 2㻌 0㻌 100%㻌 Deep Sea research part I: seaographic research
2 2 2㻌 0㻌 50%㻌 papers, 58.7, pp.757-767, 2011.
3 1 1㻌 0㻌 100%㻌 [3] T. Nakatani, et al. "AUV TUNA-SAND and its
4 2 2㻌 0㻌 50%㻌 Exploration of hydrothermal vents at
5 1 1㻌 0㻌 100%㻌 Kagoshima Bay." Oceans 2008-MTS/IEEE
6 1 3㻌 2㻌 66%㻌 Kobe , 2008.
7 1 2㻌 1㻌 100%㻌
[4] E. H. Land, J. J. McCann. "Lightness and
8 2 3㻌 1㻌 66%㻌
9 3 3㻌 0㻌 66%㻌
retinex theory," Journal of the Optical Society
10 1 2㻌 1㻌 100%㻌 of America, Vol. 61, pp.1-11, 1971.
11 1 1㻌 0㻌 0%㻌 [5] J. Ahn, et al. “Enhancement of deep-sea floor
12 1 1㻌 0㻌 0%㻌 images obtained by an underwater vehicle and
13 8 5㻌 3㻌 40%㻌 its evaluation by crab recognition.” Journal of
14 7 3㻌 4㻌 66%㻌 Marine Science and Technology, vol.22, pp1
15 2 1㻌 1㻌 100%㻌 -13, 2017
4 CONCLUSION [6] X. Fu, et al. "A retinex-based enhancing ap
proach for single underwater image," 2014 I
In this paper, we proposed a sea creature CIP, pp. 4572-4576, 2014.
classification method using CNN for simplification of [7] A . Krizhevsky, I. Sutskever, G . E . Hinton ,
classification task by researchers. For the composing "Imagenet classification with deep
classifier, 3,866 images, which are obtained by AUV, convolutional neural networks," Advances in
are employed. Then, the total detection rate was 64%, neural information processing systems, pp. 109
and the total recognition accuracy was 67% in the 7-1105, 2012.
evaluation using 15 seafloor images. The detection error
is happen when the seafloor shape is complicated.

423

Authorized licensed use limited to: University of Tasmania. Downloaded on March 12,2021 at 23:25:32 UTC from IEEE Xplore. Restrictions apply.