Journal of
Marine Science
and Engineering
Article
Research on Underwater Image Restoration Technology Based
on Multi-Domain Translation
Teng Xue 1,2 , Tianchi Zhang 3, * and Jing Zhang 1,2, *
Citation: Xue, T.; Zhang, T.; Zhang, J.
Research on Underwater Image
Restoration Technology Based on
Multi-Domain Translation. J. Mar. Sci.
Eng. 2023, 11, 674. https://doi.org/
10.3390/jmse11030674
Academic Editor: Alessandro Ridolfi
Received: 22 February 2023
Revised: 18 March 2023
Accepted: 20 March 2023
Published: 22 March 2023
Copyright: © 2023 by the authors.
Licensee MDPI, Basel, Switzerland.
This article is an open access article
distributed under the terms and
conditions of the Creative Commons
Attribution (CC BY) license (https://
creativecommons.org/licenses/by/
4.0/).
J. Mar. Sci. Eng. 2023, 11, 674. https://doi.org/10.3390/jmse11030674
1 School of Information Science and Engineering, University of Jinan, Jinan 250022, China
2 Shandong Provincial Key Laboratory of Network-Based Intelligent Computing, Jinan 250022, China
3 School of Information Science and Engineering, Chongqing Jiaotong University, Chongqing 400074, China
* Correspondence: zhangtianchi@cqjtu.edu.cn (T.Z.); ise_zhangjing@ujn.edu.cn (J.Z.)
Abstract: Underwater images are crucial in various underwater applications, including marine
engineering, underwater robotics, and subsea coral farming. However, obtaining paired data for
these images is challenging due to factors such as light absorption and scattering, suspended particles
in the water, and camera angles. Underwater image recovery algorithms typically use real unpaired
dataset or synthetic paired dataset. However, they often encounter image quality issues and noise
labeling problems that can affect algorithm performance. To address these challenges and further
improve the quality of underwater image restoration, this work proposes a multi-domain translation
method based on domain partitioning. Firstly, this paper proposes an improved confidence estimation
algorithm, which uses the number of times a sample is correctly predicted in a continuous period as a
confidence estimate. The confidence value estimates are sorted and compared with the real probability
to continuously optimize the confidence estimation and improve the classification performance of
the algorithm. Secondly, a U-net structure is used to construct the underwater image restoration
network, which can learn the relationship between the two domains. The discriminator uses full
convolution to improve the performance of the discriminator by outputting the true and false images
along with the category to which the true image belongs. Finally, the improved confidence estimation
algorithm is combined with the discriminator in the image restoration network to invert the labels
for images with low confidence values in the clean domain as images in the degraded domain. The
next step of image restoration is then performed based on the new dataset that is divided. In this
way, the multi-domain conversion of underwater images is achieved, which helps in the recovery of
underwater images. Experimental results show that the proposed method effectively improves the
quality and quantity of the images.
Keywords: underwater image restoration; deep learning; confidence estimation; domain translation
1. Introduction
The study and application of marine resources is crucial aspect of the academic field of
human development. The use of Autonomous Underwater Vehicles (AUVs) has emerged
as a significant tool for exploring and studying marine resources, particularly in marine
development and coastal defense. The visual information collected by the AUV’s vision
system plays a pivotal role in detecting and understanding the underwater environment.
However, due to challenges posed by light absorption and scattering in underwater envi-
ronments, images captured by AUVs often suffer from blurring and color degradation. To
address these issues, numerous academic methods have been proposed, which are broadly
categorized into three approaches: the first is based on the Jaffe–McGlamery underwater
imaging model [1–3], the second is based on the dark channel prior (DCP) [4–6], and the
third is based on deep learning methods. In recent years, the field of computer vision has
seen significant advancements in the application of deep learning, which make it a highly
effective tool for addressing underwater image problems in the academic realm.
https://www.mdpi.com/journal/jmse
J. Mar. Sci. Eng. 2023, 11, x FOR PEER REVIEW 2 of 14

J. Mar. Sci. Eng. 2023, 11, 674 2 of 14

learning, which make it a highly effective tool for addressing underwater image problems
in the academic realm.
ForFor
deep learning
deep methods,
learning methods,data is crucial,
data and
is crucial, high-quality
and high-quality datasets cancan
datasets significantly
significantly
enhance
enhancethethe
efficacy
efficacyof of
model training.
model training. However,
However, obtaining
obtaining thethe
paired
pairedreal datasets
real datasetsforfor
underwater
underwater image
imageacquisition poses
acquisition a significant
poses challenge.
a significant As a result,
challenge. unpaired
As a result, real da-
unpaired real
tasets are often
datasets obtained
are often usingusing
obtained subjective methods
subjective [7] or a[7]combination
methods of subjective
or a combination and
of subjective
and objective
objective [8] division
[8] division methods.
methods. Unfortunately,
Unfortunately, such division
such division methodsmethods
can leadcan lead to
to poor-
poor-quality datasets, which are shown
quality datasets, which are shown in Figure 1. in Figure 1.

(a) (b) (c) (d)

Figure 1. Examples
Figure of the
1. Examples underwater
of the synthetic
underwater dataset
synthetic EUVP
dataset [7].[7].
EUVP (a,b): distorted
(a,b): image.
distorted (c,d):
image. clean
(c,d): clean
image.
image.

In In
Figure
Figure1c,d areare
1c,d both in the
both clean
in the domain,
clean domain,but their clarity
but their differs,
clarity withwith
differs, (c) being of
(c) being
poor quality.
of poor Most Most
quality. of theof current research
the current on underwater
research image restoration
on underwater image restorationemploys syn-
employs
synthetic
thetic datasets,
datasets, whichwhich are categorized
are categorized as paired
as paired and and unpaired.
unpaired. TheThe SUIDSUID dataset
dataset [9] [9]
and and
thethe
Li Li dataset
dataset [10]synthesize
[10] synthesizeunderwater
underwaterimages images using
using ground truthtruth images.
images.Moreover,
Moreover,the
theproposed
proposedCyclic Cyclic Consistency
Consistency Network
Network (CycleGAN)
(CycleGAN) [11][11]
addresses
addresses the the
issueissue
of theofpaired
the
dataset’s
paired unavailability.
dataset’s CycleCycle
unavailability. consistency enables
consistency the conversion
enables of images
the conversion between
of images two
be-
distinct
tween two domains. For EUVP’s
distinct domains. paired paired
For EUVP’s dataset,dataset,
CycleGANCycleGANuses realusesunderwater
real underwater images
to obtain
images reduced
to obtain ground
reduced truth.
ground On On
truth. the the
other hand,
other hand,UGANUGAN [12] and
[12] andUFO-120
UFO-120 [13]
[13]use
CycleGAN to generate distorted underwater datasets based
use CycleGAN to generate distorted underwater datasets based on real ground images. on real ground images. The
degraded images in the UIEB [14] dataset were real underwater
The degraded images in the UIEB [14] dataset were real underwater images, while the images, while the ground
truth was
ground truthobtained
was obtainedfrom various conventional
from various methods.
conventional The UUIE
methods. The[15]
UUIE dataset is unpaired
[15] dataset is
and comprises images from EUVP and UIEB datasets,
unpaired and comprises images from EUVP and UIEB datasets, as well as some high-as well as some high-quality in-air
images
quality in this
in-air dataset
images fromdataset
in this ImageNet from [16] or that were
ImageNet [16] collected
or that were fromcollected
the internet.
fromPaired
the
synthetic datasets were generated after processing using several
internet. Paired synthetic datasets were generated after processing using several algo- algorithms. It is crucial
to note
rithms. that
It is theretoexists
crucial note athat
difference in data
there exists distribution
a difference between
in data synthetic
distribution and real
between data,
syn-
which leads to variations in training results.
thetic and real data, which leads to variations in training results.
ForFor classification
classification tasks
tasks forfor deep
deep learning,
learning, validating
validating a model
a model notnot only
only requires
requires correct
correct
predictions, but also requires an awareness of the confidence level
predictions, but also requires an awareness of the confidence level during the predictions. during the predictions.
Overconfident
Overconfident predictionscan
predictions canmake
make thethe network
network model
modelunreliable;
unreliable;hence,
hence,there
thereis the need
is the
to address
need to address this this
issue. Moon
issue. et al.et[17]
Moon al.proposed a solution
[17] proposed by training
a solution deep neural
by training deep networks
neural
using a novel loss function named Correctness Ranking Loss. This method regularizes
networks using a novel loss function named Correctness Ranking Loss. This method reg-
class probabilities to provide a better confidence estimate by ranking them based on the
ularizes class probabilities to provide a better confidence estimate by ranking them based
ordinal number of confidence estimation. However, as the number increases in the training
on the ordinal number of confidence estimation. However, as the number increases in the
epoch, the ranking process changes steadily and successively, and subsequent sequential
training epoch, the ranking process changes steadily and successively, and subsequent
numerical rankings do not work.
sequential numerical rankings do not work.
When the data distributions of the source and target domains differ but the tasks are
When the data distributions of the source and target domains differ but the tasks are
the same, domain adaptation enables the interconversion between different domains and
the same, domain adaptation enables the interconversion between different domains and
utilizes information-rich source domain samples to improve the performance of the target
utilizes information-rich source domain samples to improve the performance of the target
domain model. The success of Generative Adversarial Networks (GANs) [18] in the field
domain model. The success of Generative Adversarial Networks (GANs) [18] in the field
of image-to-image translation [19] provides a feasible approach to performing image trans-
of image-to-image translation [19] provides a feasible approach to performing image
lation between two or more domains in an unsupervised manner. While traditional GAN
translation
networksbetween two or more
are unidirectional, domains [11]
CycleGAN in anconsists
unsupervised manner. While traditional
of two mirror-symmetric GANs that
GANform a ring network. The purpose of this model is to learn theofmapping
networks are unidirectional, CycleGAN [11] consists two mirror-symmetric
relationship from
GANs that form
the original a ring
input to thenetwork. The purpose
target domain from theof this model
source is to learn
domain. the years,
In recent mapping rela-
CycleGAN
tionship
and itsfrom the original
modifications haveinput to the target
successfully domain
restored from theimages
underwater sourceusingdomain. In recent
synthetic paired
datasets for training [20–23]. Unpaired datasets were also used for training [24,25]. During
GAN training, pattern collapse is prone to occur, which leads to high image degradation
J. Mar. Sci. Eng. 2023, 11, 674 3 of 14

when the image orientation changes. This degradation is not ideal for underwater image
datasets. CycleGAN utilizes cycle consistency to achieve conversion between two domains,
but for domains that number more than two, more than one model needs to be learned.
The StarGAN [26] proposal solves the multi-domain problem by using only one model to
translate between different domains.
Taking inspiration from StarGAN and considering the variations in the quality of
underwater image datasets, the objective of this work is to partition the dataset and conduct
training in several domains to accomplish the restoration of underwater images. First,
an improved confidence estimation algorithm is proposed for the dataset classification
process. The algorithm attends to the current state of each sample to capture the dynamic
changes in the network. It uses the frequency of consecutive correctly predicted samples
during the training process as the probability of correctness, which is ranked with the
confidence estimate to obtain reliable predictions. Secondly, an image restoration network
with an auxiliary classifier in the discriminator is built. The auxiliary classifier can output
the class of the image based on the judgment of whether the image is true or false. Lastly,
the improved confidence estimation algorithm is combined with the image restoration
network discriminator to improve the classification performance of the dataset. On the one
hand, the improved confidence estimation algorithm can obtain the confidence value of the
sample, and use the inversion label of the sample with a low confidence value in the clean
domain as the distorted domain to classify the image. On the other hand, the ability of the
discriminator to discriminate between true and false images is also enhanced. This restores
the distorted underwater images and enhances their visual quality. The subsequent sections
of this academic paper are structured as follows: The methodology is described in detail in
Section 2. The experimental results, along with quantitative and qualitative evaluations,
are presented in Section 3. Finally, Section 4 presents the conclusion and potential avenues
for future research.

2. Proposed Method
This section presents the details of multi-domain translation in this work. The pro-
posed improved confidence estimation method is first described in Section 2.1. Then,
Section 2.2 describes the details of the multi-domain translation network module, including
the network module used for training and the loss function.

2.1. Confidence Estimation Improvement Method

In accordance with [17], the authors proposed a new method of training deep neural
networkswith a novel loss function, called Correctness Ranking Loss, which regularizes
class probabilities to be better confidence estimates in terms of ordinal ranking according
to confidence. The authors’ proposed method is easy to implement and can be applied
to existing architectures without any modification. The Correctness Ranking Loss LCR
function proposed by the authors is shown in Equation (1).





LCR xi , x j = max 0, − g ci , c j k i − k j + ci − c j (1)

In (1), for a pair of xi and x j , ci and c j are the proportion of correct events of xi over the
total number of examinations (i.e., ci ), and k i , k j represent the confidence of each sample,
respectively. ci − c j represents margin. For a pair with ci > c j , the CRL will
be zero when
k i is larger than k j + ci − c j , but, otherwise, a loss will be incurred. g ci , c j is shown in
Equation (2).


 1, i f ci > c j
g ci , c j = 0, i f ci = c j (2)
−1, otherwise


The method in this paper is to add a coefficient in front of ci − c j , as is shown in

Equations (3) and (4).





LCR xi , x j = max 0, − g ci , c j k i − k j + λ ci − c j (3)
 J. Mar. Sci. Eng. 2023, 11, 674 4 of 14

 n
1
λ = 1− (4)
2
In Equations (3) and (4), n represents the number of consecutive correct predictions on
a sample, and λ is expressed as the stability of correct predictions. Specifically, assume that
√
“×” represents that the classifier predicts the sample label incorrectly, and “ ” represents
that the classifier predicts the sample label correctly. Then, if the history of continuous
√
  is [×], then λ = 0; if the history of continuous judgment is [×  ], then λ =
judgment
1 √√ 2
1 − 21 ; if the history of continuous judgment is [× ], then λ= 1 − 21 ; if the
√√ √√√
 
3
history of judgment is [× ×× ], then λ = 1 − 21 . The purpose is that the more
consecutive correct times there are, the higher the confidence. Therefore, the improved
algorithm takes the number of consecutive correct predictions of samples during training
as a coefficient to obtain a good confidence estimation in terms of ordinal ranking according
to confidence to obtain more accurate predictions.
During extended periods of training in the deep neural network, the network’s state
is constantly changing, thus resulting in dynamic shifts in the network’s evaluation of
the samples. In the later stages of training, the confidence estimation may not change
significantly. So, unlike Moon’s method, the purpose of the improvement is to emphasize
the current state of the sample rather than its historical state, which is more indicative of
the network’s current performance. This is accomplished by pushing the number of correct
predictions from the back to the front, thereby resulting in a more accurate representation
of the network’s current state.

2.2. Underwater Image Restoration Network

This section mainly introduces the network used for underwater image restoration,
including the generator and discriminator network architecture and loss function.

2.2.1. Generator Network

The generator translates an input image into an output image in the target domain
and uses Adaptive Instance Normalization (AdaIN) [27] to inject label information into
J. Mar. Sci. Eng. 2023, 11, x FOR PEER REVIEW 5 is
the generator. Then, it generates fake images. The network architecture of the generator of 14
shown in Figure 2.

Figure 2. The architecture of the generator.

Figure 2. The architecture of the generator.

As is shown in Figure 2, the generator in this study employs the U-net [28] architec-
ture, which includes both up sampling and down sampling. The network undergoes three
rounds of up sampling and three rounds of down sampling. The “Upconv” layer repre-
sents the up-sample layer that uses a transposed convolution operator. The “Tanh” layer
represents the non-linear activation function, Tanh. Additionally, intermediate skip con-
J. Mar. Sci. Eng. 2023, 11, 674 Figure 2. The architecture of the generator. 5 of 14

As is shown in Figure 2, the generator in this study employs the U-net [28] architec-
ture, which includes
As is shown both up
in Figure 2, sampling
the generatorandindown sampling.
this study The the
employs network
U-netundergoes three
[28] architecture,
rounds of up sampling and three rounds of down sampling. The “Upconv”
which includes both up sampling and down sampling. The network undergoes three rounds layer repre-
sents
of upthe up-sample
sampling and layer
three that
rounds usesofadown
transposed convolution
sampling. operator.
The “Upconv” Therepresents
layer “Tanh” layerthe
represents
up-sample thelayernon-linear
that usesactivation
a transposed function, Tanh. operator.
convolution Additionally, intermediate
The “Tanh” skip con-
layer represents
nections enableactivation
the non-linear the network to retain
function, Tanh. low-level features
Additionally, while performing
intermediate feature fusion
skip connections enable
with high-level
the network information.
to retain low-level Instance
features normalization [29] isfeature
while performing denoted as “IN”.
fusion withDuring up
high-level
sampling, the IN layer is replaced with the Adaptive Instance Normalization
information. Instance normalization [29] is denoted as “IN”. During up sampling, the IN (AdaIN)
layer,
layer iswhich consists
replaced with ofthethe IN andInstance
Adaptive Linear layers. The AdaIN
Normalization layerlayer,
(AdaIN) incorporates label in-
which consists of
formation
the IN andthrough one-hot
Linear layers. Theencoding,
AdaIN layerthereby allowinglabel
incorporates the information
generator tothrough
learn the label
one-hot
information of the allowing
encoding, thereby domain. the generator to learn the label information of the domain.

2.2.2. Discriminator
2.2.2. Discriminator Network
Network
The discriminator learns to
The discriminator learns to distinguish
distinguish between
between real
real and
and fake
fake images,
images, whether
whether itit is
is aa
real image
real image oror aa fake
fake image
image generated
generated by
by the
the generator,
generator, and
and classifies
classifies the
the real
real images.
images. The
The
network architecture
network architecture of
of the
the discriminator
discriminator isis shown
shown in
in Figure
Figure 3.
3.

Figure 3. The architecture of the discriminator.

Figure 3. The architecture of the discriminator.

As
As is
is shown
shown in in Figure
Figure 3,3, the
the discriminator
discriminator inputs
inputs an
an image
image that
that passes
passes through
through the
the
convolution layer and finally outputs values. The discriminator adopts
convolution layer and finally outputs values. The discriminator adopts the output mode the output mode of
of ACGAN
ACGAN [30]
[30] and
and hashas
twotwo outputs.
outputs. TheThe
firstfirst
3*33*3 convolution
convolution output
output judges
judges the the authen-
authenticity
ticity
of theof the image,
image, with
with the realthe real judged
image image judged as true
as true and and the
the false false
image image is
is judged asjudged as
false. The
false.
second The1*1second 1*1 convolution
convolution outputs the outputs the probability
probability that this that this belongs
picture picture belongs to two
to two classes,
classes,
and theand the maximum
maximum class probability
class probability is usedisas used
theas the confidence
confidence estimation.
estimation. Further-
Furthermore,
more, spectral normalization [31] is added to the discriminator during
spectral normalization [31] is added to the discriminator during training to stabilize training to stabilize
the
the training
training process.
process.

2.2.3. Loss
Adversarial Loss: The adversarial loss is a general loss function of the GAN, which
makes the generated image more consistent with the distribution of the target domain, as
is shown in Equation (5). The generator G tries to minimize L adv , and the discriminator D
tries to maximize it.

L adv = Ex [log( D ( x ))] + Ez [log(1 − D ( G (z)))] (5)

Domain Classification Loss: For a given input image x and a domain label c, the goal
is to translate x into an output image y that is properly classified to the target domain c.
To achieve this condition, add an auxiliary classifier on top of D and impose the domain
classification loss when optimizing both D and G. The domain classification loss for real
images is used to optimize D, as is shown in Equation (6).

Lrcls = Ex,c0 − log Dcls c0 x



(6)

The domain classification loss for fake images is used to optimize G, as is shown in
Equation (7).
f
Lcls = Ex,c [− log Dcls (c| G ( x, c))] (7)
J. Mar. Sci. Eng. 2023, 11, 674 6 of 14

The term Dcls (c0 | x ) represents a probability distribution over domain labels computed
by D. By minimizing this objective, D learns to classify a real image x to its corresponding
original domain c0 . Assume that the input image and domain label pair (x, c0 ) are given by
the training data. G tries to minimize this objective to generate images that can be classified
as the target domain c.
Reconstruction Loss: As is shown in Equation (8), G takes as its input the translated
image G ( x, c) and the original domain label c0 , and it tries to reconstruct the original image
x. The L1 norm is used as the reconstruction loss.

Lrec = Ex,c,c0 = x − G G ( x, c), c0 ||1



(8)

Identity Loss: When calculating the identity loss of generator G, G inputs y to generate
ŷ. Then, use the L1 loss of y and ŷ as the identity loss of G. Correspondingly, the identity
loss of generator F is the L1 loss of input x and generated x 0 , as is shown in Equation (9).

Liden ( G, F ) = Ey∼ pdata (y) [|| G (y) − y||1 ] + Ex∼ pdata ( x) [|| F ( x ) − x ||1 ] (9)

The objective functions to optimize G and D are shown in Equations (10) and (11).

L D = − L adv + λcls Lrcls (10)

f
LG = L adv + λcls Lcls + λrec Lrec + λiden Liden (11)
where λcls , λrec , and λiden are hyper-parameters that control the relative importance of the
domain classification, reconstruction loss, and identity loss. λcls = 1, λrec = 10, and λiden = 1
in the underwater image restoration experiments.

3. Experimental Results
In this section, experiments have been conducted. Section 3.1 comprises the experi-
ments to obtain the improved confidence estimation. Section 3.2 combines the improved
confidence estimation method with the built underwater image restoration network to con-
duct image restoration experiments. Section 3.3 is an experiment that consists of dividing
the dataset and then performing multi-domain translation in two stages.

3.1. Confidence Improvement Experiment

In this section, the related issues of training details are described in detail, and then
the experimental results are evaluated using different evaluation metrics.

3.1.1. Training Details

In this experiment, PyTorch libraries [32] were used to implement the experiment.
The ResNet50 [33] model was trained using SGD, the momentum was 0.55, the initial
learning rate was 0.1, the weight decay was 10−5 , and it was trained for 100 epochs. The
initial learning rate was set to 0.001 for VGG16 [34] training and ResNet50. The dataset
was unpaired from the EUVP dataset [6], 90% of which was used as a training set, and
10% was used as the test set. For data augmentation, the size of the images was adjusted
to (128 + 16) × (128 + 16) and then randomly cropped to 128 × 128. When testing, we
adjusted the image to (128 + 16) × (128 + 16) and then cropped the center to 128 × 128.

3.1.2. Evaluation Metrics

In [17], to measure ordinal ranking performance, five common metrics were used: the
accuracy, the Area Under the Risk–Coverage Curve (AURC) that was defined to be the
error rate as a function of coverage, the Excess-AURC (E-AURC) that was a normalized
AURC [35], the Area Under the Precision–Recall Curve that used errors as the positive
class (AUPR-Error), and the False Positive Rate at 95% True Positive Rate (FPR-95%-TPR).
For calibration, we used the Expected Calibration Error (ECE) [36] and the Negative Log
Likelihood (NLL).
J. Mar. Sci. Eng. 2023, 11, 674 7 of 14

The improved method proposed in this paper was compared with the method of [17]
and the baseline model without the improved method. In this experiment, the baseline
model used the ResNet50 network and VGG16 network to do a binary classification task.
The dataset used the unpaired files in the EUVP dataset. The loss function used Cross
Entropy Loss. Moon’s [17] method added a CRL loss to the baseline, and the method
proposed in this paper is based on Moon’s improved method. The obtained comparison
results are shown in Tables 1 and 2.

Table 1. Comparison of ResNet50.

Accuracy AURC EAURC AUPR FPR ECE NLL Brier

Method
(↑) (↓) (↓) (↑) (↓) (↓) (↓) (↓)
Baseline [17] 93.82 9.17 7.22 44.11 41.1 4.77 3.43 10.67
Moon [17] 94.06 6.1 4.3 46.37 39.06 1.2 1.84 8.83
Proposed 94.06 5.84 4.03 47.43 38.72 0.96 1.8 8.77

Table 2. Comparison of VGG16.

Accuracy AURC EAURC AUPR FPR ECE NLL Brier

Method
(↑) (↓) (↓) (↑) (↓) (↓) (↓) (↓)
Baseline [17] 98.58 1.67 1.67 26.7 33.33 1.24 0.96 2.53
Moon [17] 97.95 1.4 1.19 36.59 23.08 1.48 0.76 3.32
Proposed 98.58 0.91 0.81 21.1 22.22 1.09 0.56 2.66

In Tables 1 and 2, the AURC and E-AURC values were multiplied by 103 , and the NLL
was multiplied by 10 for clarity. All remaining values are percentages. The symbol “↑”
indicates that higher values are better, and the symbol “↓” indicates that lower values are
better. The results in the first row represent the results obtained by the baseline model. The
second line is the improvement made by Moon [17] for calculating confidence estimation,
and the third line is the improvement proposed in this paper. It can be seen that the accuracy
of the improved method is the same as that of Moon’s method, and other indicators had
good performance.
In addition, to verify the effect of the proposed method on Deep Ensembles [37], a
comparison of experiments was carried out in Table 3. In the training process, all models
were trained with SGD with a momentum of 0.9, an initial learning rate of 0.1, and a
weight decay of 10−4 . The batch size was 128, and 300 epochs were trained. At 100 epochs
and 150 epochs, the learning rate was reduced by a factor of 10. All training results were
averaged from experiments using five different random seeds.
The CIFAR-10/100 datasets [38] and the SVHN dataset [39] were exported from
the torchvision package of the PyTorch library. For their data augmentation, a random
horizontal flip and a 32 × 32 after-filling of 4 pixels per side were used. The standard data
augmentation scheme of random cropping was used.
In Table 3, the CIFAR-10 dataset showed improvements in all metrics. Additionally,
the SVHN and CIFAR-100 datasets also showed significant improvements in most metrics.
Thus, the standard depth model trained with the proposed method improved the quality of
confidence estimation and, thus, improved the better performance of the model. However,
the improved confidence value estimation algorithm had some limitations, such as a
degradation in the performance of the method when for multiple classification tasks.
J. Mar. Sci. Eng. 2023, 11, 674 8 of 14

Table 3. Experimental results based on DenseNet-121 network.

Model Accuracy AURC EAURC AUPR FPR ECE NLL Brier

Method
Dataset (↑) (↓) (↓) (↑) (↓) (↓) (↓) (↓)
Baseline [17] 88.76 ± 0.44 20.21 ± 2.12 13.61 ± 0.46 49.08 ± 2.41 56.58 ± 2.39 7.86 ± 0.19 5.71 ± 0.09 18.81 ± 1.04
DenseNet-121
Moon [17] 89.17 ± 0.19 18.62 ± 2.4 12.52 ± 1.24 49.13 ± 2.38 54.57 ± 5.10 3.21 ± 0.09 3.59 ± 0.03 16.15 ± 0.48
CIFAR-10
Proposed 89.22 ± 0.1 18.25 ± 0.85 12.22 ± 0.34 49.96 ± 0.82 53.39 ± 2.48 2.99 ± 0.04 3.54 ± 0.01 15.99 ± 0.21
Baseline [17] 60.49 ± 0.79 158.38 ± 54.42 67.33 ± 8.87 74.51 ± 0.37 67.46 ± 0.35 18.59 ± 0.13 18.67 ± 0.23 57.23 ± 1.22
DenseNet-121
Moon [17] 62.63 ± 0.34 141.51 ± 15.41 60.87 ± 1.62 74.29 ± 0.56 65.78 ± 3.25 14.36 ± 0.02 15.95 ± 0.08 52.39 ± 0.6
CIFAR-100
Proposed 62.6 ± 0.52 140.53 ± 18.34 59.75 ± 2.29 74.43 ± 0.32 66.21 ± 0.77 14.54 ± 0.33 15.85 ± 0.1 52.32 ± 0.82
Baseline [17] 95.45 ± 0.03 8.85 ± 2.5 7.8 ± 2.5 40.47 ± 5.13 37.54 ± 6.7 3 ± 0.02 2.45 ± 0.01 7.7 ± 0.08
DenseNet-121
Moon [17] 95.83 ± 0.01 5.53 ± 0.04 4.65 ± 0.05 45.07 ± 4.14 31.91 ± 4.64 0.65 ± 0.003 1.64 ± 0.001 6.43 ± 0.02
SVHN
Proposed 95.99 ± 0.03 5.29 ± 0.02 4.47 ± 0.02 43.87 ± 3.76 31.29 ± 3.64 0.63 ± 0.001 1.59 ± 0.003 6.20 ± 0.04
J. Mar. Sci. Eng. 2023, 11, 674 9 of 14

3.2. Underwater Image Restoration Experiment

In image restoration experiments, it is first necessary to combine the improved confi-
dence estimation algorithm with the image restoration network. Specifically, the improved
algorithm is applied to the discriminator of the network, and the class probability of the real
image output from the discriminator is continuously optimized to improve the capability
of the discriminator.
In the process of image restoration, the training is divided into two stages. The first
stage is to add the method of Section 3.1 to the auxiliary classifier of the discriminator to
improve the classification performance of the auxiliary classifier. The translation performed
in the first stage is the translation of the undivided poor domain and the good domain.
After the first stage is completed, the dataset is divided, and the second stage of training is
J. Mar. Sci. Eng. 2023, 11, x FOR PEER carried
REVIEW out according to the newly divided dataset. A schematic diagram of each stage 9 of 14
is
shown in Figure 4.

Overviewof
Figure4.4.Overview
Figure ofdomain
domaintranslation.
translation.

Asisisshown
As shownin inFigure
Figure4,4, the
thefirst
firststage
stageofof the
the training
trainingprocess
process isis the
the transformation
transformation
between the distorted domain and the clean domain, and the second stage isisthe
between the distorted domain and the clean domain, and the second stage theinverse
inverse
labeling of samples with low confidence values in the clean domain to
labeling of samples with low confidence values in the clean domain to the distorted do- the distorted domain.
Then, both domains are transformed to the domain with high confidence values in the clean
main. Then, both domains are transformed to the domain with high confidence values in
domain. For these images, it did not particularly conform to the characteristics of the good
the clean domain. For these images, it did not particularly conform to the characteristics
domain, and the label needed to be reversed into a distorted domain, so the method in the
of the good domain, and the label needed to be reversed into a distorted domain, so the
right figure of Figure 4 was used in the second stage of training. According to the confidence
method in the right figure of Figure 4 was used in the second stage of training. According
value, domains below 0.55 are considered domains with low confidence values in the clean
to the confidence value, domains below 0.55 are considered domains with low confidence
domain, and inverting their labels as distorted domains can improve the final restoration
values in the clean domain, and inverting their labels as distorted domains can improve
effect. The label adopted one-hot encoding, and the largest class probability among the
the final restoration effect. The label adopted one-hot encoding, and the largest class prob-
two probabilities output by the auxiliary classifier in the discriminator was used as the
ability among the two probabilities output by the auxiliary classifier in the discriminator
confidence value. In this section, experiments were performed using Moon’s [17] method
was used as the confidence value. In this section, experiments were performed using
and the improved confidence value method combined with image restoration networks,
Moon’s [17] method and the improved confidence value method combined with image
respectively.
restoration networks, respectively.
3.2.1. Training Details
3.2.1. Training Details
The dataset used in the experiments and the image restoration process in this section
Thesame
are the datasetas used in the experiments
in Section 3.1. This work anduses
the image
Adamrestoration
optimizer process
[40] with in this
beta1section
= 0.5,
are the=same
beta2 0.999asandin Section
learning 3.1. This
rate lr work
= 0.0001.usesAdditionally,
Adam optimizer the [40]
weightwithdecay
beta1value
= 0.5, beta2
of the
=discriminator
0.999 and learning rate was
optimizer 1 × 10−Additionally,
lr = 0.0001. 5. the weight decay value of the discrim-
inator optimizer was 1 × 10−5.
3.2.2. Evaluation Metrics
3.2.2. We
Evaluation
invertedMetrics
the label value of the sample with a low confidence value, set five random
seeds,
Weconducted
inverted the fivelabel
experiments,
value of and took thewith
the sample average
a lowvalue. The experimental
confidence value, set fivesetup
ran-is
the same as in Section 3.2. The results are shown in Table 4.
dom seeds, conducted five experiments, and took the average value. The experimental
setup is the same as in Section 3.2. The results are shown in Table 4.

Table 4. Comparison between the improved method and the original method in the underwater
image restoration network.
J. Mar. Sci. Eng. 2023, 11, 674 10 of 14

Table 4. Comparison between the improved method and the original method in the underwater
image restoration network.

Model Random Seed Accuracy AURC EAURC AUPR FPR ECE NLL Brier
100 94.49 7.42 5.87 33.71 48.57 2.61 1.69 8.66
101 94.33 7.85 6.21 32.63 61.11 3.03 1.84 9.13
Moon [17] 102 94.96 8.16 6.87 37.9 40.62 2.47 1.76 7.64
103 94.02 7.79 5.97 39.09 50 3.28 1.84 9.21
104 91.34 15.2 11.34 51.14 50.91 4.76 2.8 12.44
93.83 9.28 7.25 38.89 50.24 3.23 1.99 9.42
Average Value
93.76 8.37 6.26 39.82 46.58 3.38 1.79 9.24
100 96.06 4.5 3.72 27.57 48 2.32 1.2 6.67
101 94.49 10.32 8.77 38.42 51.43 2.8 1.88 8.73
Proposed 102 92.28 7.92 4.87 57 30.61 4.56 1.9 10.01
103 94.17 7.6 5.87 33.6 43.24 2.84 1.7 8.68
104 91.81 11.5 8.05 42.52 59.62 4.37 2.27 12.11

In Table 4, the two average values represent the average metrics of Moon’s method and
the method mentioned in this paper after training with five random seeds, respectively. It
can be seen that the proposed method was well improved compared to the Moon’s method.
Therefore, the next step of the division experiment could be carried out.

3.3. Division Experiment of Underwater Image Restoration

This section builds on Section 3.2, but differs by dividing the training process into two
stages. After completion of the first stage, the dataset was divided based on the confidence
estimation, and the corresponding index was stored. A threshold was set based on the
confidence for division, with a value of 0.55 used in this case, thereby separating higher
and lower confidence domains. At the beginning of the second stage, which starts between
50–80 epochs, respectively, the lower confidence domain labels in the good domain were
reversed to indicate distorted domains and continued to run through 100 epochs. The test
set used the “Inp” file in the “test_sample” folder in EUVP. Tables 5 and 6 present the
comparison results of the experiment’s quantitative and qualitative evaluation.

3.3.1. Quantitative Evaluation

For a quantitative evaluation of the restored images, the UIQM [41] can be used for
comparison. UIQM includes three underwater image attribute metrics: Underwater Image
Colorimetric Metric (UICM), Underwater Image Sharpness Metric (UISM), and Underwater
Image Contrast Metric (UIConM). The higher the UIQM score, the more consistent the
results are with human visual perception.
In Table 5, the first row shows the UIQM values obtained from the test set after training
using the Section 3.2 method for images without label inversion, which demonstrate an
improvement in values. The second and third rows display the UIQM values after label
inversion using different epochs. The results show that UIQM values with label inversion
performed better than those without inversion, regardless of the epoch. This is because the
divided dataset filtered out more blurred images in the clean domain, thereby demonstrating
the effectiveness of the proposed method.

3.3.2. Qualitative Evaluation

For the qualitative assessment, the results of the second phase of the test starting from
the 80th epoch were chosen to be compared with the results of the unreversed labels, in
addition to the DCP method using as a comparison. The results are shown in Table 6.
 104
104
104
104 91.81
91.81
91.81
91.81 11.5
11.5
11.5
11.5 8.05
8.05
8.05
8.05 42.52
42.52
42.52
42.52 59.62
59.62
59.62
59.62 4.37
4.37
4.37
4.37 2.27
2.27
2.27
2.27 12.11
12.11
12.11
12.11
In
In Table
In
In Table
Table 4,
Table 4,
4, the
4, the
the two
the two
two average
two average
average values
average values
values represent
values represent
represent the
represent the
the average
the average
average metrics
average metrics
metrics of
metrics of
of Moon’s
of Moon’s
Moon’s method
Moon’s method
method
method
and
and In Table
In
the
the Table
method4, the
4,
method the two average
two average
mentioned
mentioned in
in values
values
this
this paper
paperrepresent
represent
after
after the average
the
training
trainingaverage
with
with metrics
metrics
five
five of Moon’s
of
random
random Moon’s
seeds,
seeds, method
method
respec-
respec-
and
and
and
and the
the
the
the method
method
method
method mentioned
mentioned
mentioned
mentioned in
in
in
in this
this
this
this paper
paper
paper
paper after
after
after
after training
training
training
training with
with
with
with five
five
five
five random
random
random
random seeds,
seeds,
seeds,
seeds, respec-
respec-
respec-
respec-
tively.
tively.
tively.
tively. It
It
It
It can
can
can
can be
be
be
be seen
seen
seen
seen that
that
that
that the
the
the
the proposed
proposed
proposed
proposed method
method
method
method was
was
was
was well
well
well
well improved
improved
improved
improved compared
compared
compared
compared to
to the
to
to the
the
the
J. Mar. Sci. Eng. 2023, 11, 674
tively.
tively.
Moon’s It
It
Moon’s method.can
can
method. be
be seen
seen that
that
method. Therefore,
Therefore, the
the
Therefore, the
the proposed
proposed
the next
next
next step
step
step of
ofmethod
method
of the
the was
was
the division
division well
well improved
improved
division experiment
experiment
experiment could compared
compared
could
could be
be
be carried
carried
carriedto
to the
the
11out.
out.
of 14
Moon’s
Moon’s method.
Moon’s method.
Moon’s Therefore,
method. Therefore, the
Therefore, the next
the next step
next step of
step of the
of the division
the division experiment
division experiment could
experiment could be
could be carried out.
carried
be carried out.
out.
out.
3.3.
3.3. Division
3.3.
3.3. Division
Division Experiment
Division Experiment
Experiment of
Experiment of
of Underwater
of Underwater
Underwater Image
Underwater Image
Image Restoration
Image Restoration
Restoration
Restoration
3.3. Division
3.3. Division Experiment
Experiment of of Underwater
Underwater Image Image Restoration
Restoration
This
This section
Table
This section
section
6 comparesbuilds
buildsthe
builds on
onrestoration
on Section
Section 3.2,
Section 3.2,
3.2,result
but
but differs
but differs
differs
obtained by
by without
by dividing
dividingdividing
the
the training
training
training process
processwith
the dataset into
into
two
two
This
stages.
stages.
section
This section
This section
After
After
builds
builds on
builds
completion
completion
on Section
on Section
Section
of
of the
the
3.2,
first
first
but
3.2, but
3.2, but
stage,
stage,
differs
differs
differs
the
the by dividing
by
by dividing
dividing
dividing
dataset
dataset was
was
the
the
the
divided
divided
training
the training
training
based
based
process
process
process
process
on
on the
the
into
into
into
into
con-
con-
the
two
two restoration
stages.
stages. After
Afterresult obtained
completion
completion of
of by
the
the dividing
first
first stage,
stage,it into
the
the 80s.
datasetThe
dataset wastraditional
was divided
divided method
based
based on
on DCP
the
the was
con-
con-
two
two
fidence
fidencestages.
stages. After
After
estimation,
estimation, completion
completion
and
and thethe
the of
of the
the
corresponding
correspondingfirst
first stage,
stage,
index the
the
index was dataset
dataset
was
was was
was
stored.
stored. A Adivided
divided
A threshold
threshold based
based
thresholdimage,was on
on
was set the
the
set
setandbased
based con-
con-
also
fidence used for comparison.
estimation, and The first column shows the original underwater the
fidence
onfidence
fidence
on the
the
estimation,
estimation,for
estimation,
confidence
confidence
and
and
and
for the corresponding
the
the
division,
division,
corresponding
corresponding
corresponding
with
with a
a value
value
index
index
index
index
of
of
was
was
was
0.55
0.55
stored.
stored.
stored.
stored.
used
used in
in
A
A
this
this
threshold
A threshold
threshold
case,
case, thereby
thereby
was
was
was set based
set
wasseparating
set based
based
based
separating
onsecond column shows the repair resultvalue
of the low-confidence sample withoutseparating
reversing
onon the
on
higherthe confidence
the
the
higher
confidence
confidence
confidence
and
and lower
lower
for
for division,
for
for division,
division,
division,
confidence
confidence
with
with aaaa value
with
with
domains.
domains. value
value
At
At the
the
of
of
of
of 0.55
0.55 used
0.55
0.55
beginning
used
usedof
used
beginning
in
in
in
of
this
inthethis
this
this
the
case,
case, thereby
case,
case,
second
second
thereby
thereby
thereby
stage,
stage,
separating
separating
separating
which
which starts
starts
and
higher low-confidence samples. Although the undivided restoration results improved, the
higher and
higher
higher
between
between
overall
and
and
and lower
lower
50–80lower
lower
50–80
appearance
confidence
confidence
confidence
confidence
epochs,
epochs,was
domains.
domains.
domains.
domains.
respectively,
respectively,
red, while the
the
the
At
At
At
AtDCP
the
the
the
lower
lower
beginning
beginning
themethod
beginning
beginning
confidence
confidence
of
of
of the
the second
the
ofdomain
the
domain
produced
second
second
second
a
stage,
labels
stage,
stage,
stage,
labels
dark in
in
which
which
which
which
the
the
restoration good
good
starts
starts
starts
starts
result. do-
do-
between
between
between
between 50–80
50–80
50–80
50–80 epochs,
epochs,
epochs,
epochs, respectively,
respectively,
respectively, the
the lower
the lower
lower confidence
confidence
confidence domain
domain
domain labels
labels
labels in
in the
in the
the good
good
good do-
do-
do-
main
main were
were
Bycomparison,
main
main were
were reversed
reversed torespectively,
to
the division
reversed
reversed to
to indicate
indicate
results
indicate
indicate
the
distorted
distorted
exhibited
distorted
distorted
lower confidence
domains
domains
a better
domains
domains and
visual
and
and
domain
and continued
continued
continued
restoration
continued
labels
to
to
to
in
to effect.
run
run
run
run
the good
through
through
The
through
through results do-
100
100
100
100
main
main
show
epochs.
epochs. were
were
that
The
The reversed
reversed
thetest
proposed
test
test set
set usedto
to
used
used indicate
indicate
method
the
the “Inp”
“Inp”
“Inp” distorted
distorted
is effective
file
file in
in domains
domains
for“test_sample”
in the
the
the and
and
underwater image
“test_sample” continued
continued
folder
folder to
to
restoration,
in run
run
EUVP.as
in EUVP.
EUVP. through
through
it achieved
Tables
Tables 100
100
epochs.
epochs.
epochs.
epochs. The
The
The
The test
test
test set
set
set
set used
used
used the
the
the
the “Inp”
“Inp”
“Inp” file
file
file
file in
in
in the “test_sample”
the
the “test_sample”
“test_sample”
“test_sample” folder
folder
folderand
folder in
in EUVP.
in EUVP.
in EUVP. Tables
Tables
Tables
Tables 555555 and
and
and
and
and
and
6good
6 outcomes
present
present
6666 present
present the
the
the
the in both
comparison
comparison
comparison
comparison qualitative
results
results
results
results of
of
of
ofand
the
the
the
the quantitative
experiment’s
experiment’s
experiment’s
experiment’s evaluations.
quantitative
quantitative
quantitative
quantitative and
and
and qualitative
qualitative
qualitative
qualitative evalua-
evalua-
evalua-
evalua-
present the
present
tion.
tion. the comparison
comparison results results ofof the
the experiment’s
experiment’s quantitative
quantitative and and qualitative
qualitative evalua-
evalua-
tion.
tion.
tion. 5. The UIQM value comparison of a different epoch.
tion.
Table
Table
Table 5.
Table
Table 5.
5. The
5. The
The UIQM
The UIQM
UIQM value
UIQM value
value comparison
value comparison
comparison of
comparison of
of aaaa different
of different
different epoch.
different epoch.
epoch.
epoch.
Table
Table5.
Table 5.
5.The
The
TheUIQM
UIQM
UIQMNot
value
value
value comparison
comparisonof
comparison
Inversion of
ofaaadifferent
different epoch. 3.22 ± 0.33
differentepoch.
epoch.
Input UIQM Not
Not Inversion
Inversion 3.22
3.22 ±±±± 0.33
0.33
0.33 70th
Input
Input
2.57 ±UIQM
UIQM
0.55
Not
Not Inversion
Not
Not Inversion
Inversion
Inversion 50th 60th3.22
3.22 0.33
3.22 ±± 0.33
3.22 0.33 80th
Input
Input UIQM
Input
Input UIQM
UIQM Label Inversed
UIQM 50th
50th
3.24 ± 0.34 60th
60th
3.24 ± 0.32 70th
70th
3.24 ± 0.31 80th
80th
3.26 ± 0.4
2.57
2.57 ±±±± 0.55
0.55
0.55 Label 50th
50th
Label Inversed
Inversed
Inversed 50th
50th 60th
60th 70th
70th
60th
60th 70th
70th 80th
80th
80th
80th
2.57
2.57 0.55
2.57 ±± 0.55
2.57 0.55 Label
Label Inversed
Label Inversed
Label 3.24
3.24
Inversed 3.24 ±± 0.34
0.34 3.24
3.24 ±± 0.32
0.32 3.24
3.24 ±± 0.31
0.31 3.26
3.26 ±± 0.4
0.4
3.24 ±±±± 0.34
3.24
3.24 0.34
0.34
0.34 3.24
3.24 ±±±± 0.32
3.24
3.24 0.32
0.32
0.32 3.24
3.24 ±±±± 0.31
3.24
3.24 0.31
0.31
0.31 3.26
3.26 ±±±± 0.4
3.26
3.26 0.4
0.4
0.4
Table
Table 6.
Table 6. Qualitative
Qualitative comparison
6. Qualitative
Qualitative comparison of
comparison of the
of the underwater
the underwater images.
underwater images.
images.
Table
Table6.
Table
Table 6.
6. Qualitativecomparison
Qualitative
6.Qualitative
Qualitative comparisonof
comparison
comparison ofthe
of
of theunderwater
the
the underwaterimages.
underwater
underwater images.
images.
images.
Table 6. comparison of the underwater images.
Original Image
Original
Original Image
Image NotReverse
Not
Not Reverse
Reverse DCP
DCP
DCP 80th
80th
80th
Original
Original Image
Original
Original Image
Image
Image Not
Not Reverse
Not
Not Reverse
Reverse
Reverse DCP
DCP
DCP
DCP 80th
80th
80th
80th

J.J.
J.
J.J.
Mar.
J.Mar.
Mar.
Mar.
Mar.
Sci.
Mar.Sci.
Sci.
Sci.
Sci.
Eng.
Sci.Eng.
Eng.
Eng.
Eng.
2023,
Eng.2023,
2023,
2023,
2023,
11,
2023,11,
11,
11,x
11,
11, xx FOR
xxxFOR
FOR
FOR
FOR
PEER
FORPEER
PEER
PEER
PEER
REVIEW
PEERREVIEW
REVIEW
REVIEW
REVIEW
REVIEW
11
11
11 of
11 of
11
11 of 14
of 14
of
of 14
14
14
14

3.3.1.
3.3.1.
3.3.1.
3.3.1. Quantitative
3.3.1.Quantitative
Quantitative
Quantitative Evaluation
QuantitativeEvaluation
Evaluation
Evaluation
Evaluation
For
For
For aa quantitative
quantitative
For aaa quantitative
For quantitative evaluation
quantitative evaluation
evaluation of
evaluation
evaluation of
of
of the
of the
the
the restored
the restored
restored
restored images,
restored images,
images,
images, the
images, the
the
the UIQM
the UIQM
UIQM
UIQM [41]
UIQM [41]
[41]
[41] can
[41] can
can
can be
can be
be
be used
be used for
used for
used
used for
for
for
comparison.
comparison.
comparison.
comparison. UIQM
comparison. UIQM
UIQM includes
includes
UIQM includes
UIQM three
three
includes three
includes underwater
underwater
three underwater
three image
image
underwater image
underwater attribute
attribute
image attribute
image metrics:
metrics:
attribute metrics:
attribute Underwater
Underwater
metrics: Underwater
metrics: Underwater Im-
Underwater Im-
Im-
Im-
Im-
age
age
age
age Colorimetric
age Colorimetric
Colorimetric
Colorimetric Metric
Colorimetric Metric
Metric
Metric (UICM),
Metric (UICM),
(UICM),
(UICM), Underwater
(UICM), Underwater
Underwater Image
Underwater
Underwater Image
Image
Image Sharpness
Image Sharpness
Sharpness
Sharpness Metric
Sharpness Metric
Metric
Metric (UISM),
Metric (UISM),
(UISM), and
(UISM),
(UISM), and
and
and Un-
and Un-
Un-
Un-
Un-
derwater
derwater
derwater
derwater Image
derwater Image
Image
Image Contrast
Image Contrast
Contrast
Contrast Metric
Contrast Metric
Metric
Metric (UIConM).
Metric (UIConM).
(UIConM). The
(UIConM).
(UIConM). The
The
The higher
The higher
higher
higher the
higher the
the
the UIQM
the UIQM
UIQM
UIQM score,
UIQM score,
score,
score, the
score, the
the
the more
the more
more
more con-
more con-
con-
con-
con-
sistent
sistent
sistent the
the
sistentthe
sistent results
results
theresults
the are
are
resultsare
results with
with
arewith
are human
human
withhuman
with visual
visual
humanvisual
human perception.
perception.
visualperception.
visual perception.
perception.
J. Mar. Sci. Eng. 2023, 11, 674 12 of 14

4. Conclusions
This work aimed to improve the visual quality of images by introducing an improved
confidence estimation algorithm. The algorithm divides the dataset using the number
of consecutive correct predictions during the training period as a confidence estimation,
which serves as a real probability ranking to obtain better prediction results. This method
improved the classification performance. Moreover, this work built an image restoration
network that demonstrated an enhancement in image quality. Finally, the performance of
the final image restoration was further improved by applying the improved confidence
estimation algorithm to the image restoration network and inverting labels for images
J.J. Mar.
Mar. Sci.
Sci. Eng.
Eng. 2023,
2023, 11,
11, xx FOR
FOR PEER REVIEW
PEERwith
REVIEW
low confidence values in the clean domain as degenerated domain. The experimental 12
12 of
of 14
14
J.J. Mar.
Mar. Sci.
Sci. Eng.
Eng. 2023,
2023, 11,
11, xx FOR
FOR PEER
PEER REVIEW
REVIEW 12
12 of
of 14
14
results show that the multi-domain transformation method based on domain partitioning
is highly effective. Both qualitative and quantitative evaluations confirm improved image
restoration
restoration quality. It is important to note that the proposed method can preprocess the
restoration quality.
restoration
restoration quality. It
quality.
quality. ItIt is
It is important
is
is important to
important
important to note
to
to note that
note
note that the
that
that the proposed
the
the proposed method
proposed
proposed method can
method
method can preprocess
can
can preprocess the
preprocess
preprocess the
the
the
dataset
dataset before network trainingtrainingto toenhance
enhancethe therecovery
recoveryresults,
results,and
and it can be extended
dataset before network training to enhance the recovery results, and it can be extended to
dataset before
before network
network training to enhance the recovery results, and it
it can
can be
be extended
extended to
to
to other
other underwater image restoration networks without the need for additional nodes to
other underwater
other
other underwater image
underwater
underwater image restoration
image
image restoration networks
restoration
restoration networks without
networks
networks without the
without
without the need
the
the need for
need
need for additional
for
for additional nodes
additional
additional nodes to
nodes
nodes to
to
to
output
output
output confidence
confidence
confidence values.
values.
values. Therefore,
Therefore,
Therefore, this
this
this method
method
method can
can
can be
be
be added
added
added to
to
to the
the
the algorithms
algorithms
algorithms of
of
of other
other
other
output
output
authors. confidence
confidence
However, values.
values.
it does
does Therefore,
Therefore,
not performthis
this method
method
well for can
can be
be
extremely added
added
blurryto
to the
the algorithms
algorithms
images, as is of
of
shownother
other
in
authors.
authors.
authors.
authors. However,
However,
However,
However, it
itit does
it does
does not
not
not
not perform
perform well
perform
perform well
well
well for
for extremely
for
for extremely
extremely
extremely blurry
blurry
blurry
blurry images,
images, as
images,
images, as is
as
as is shown
is
is shown in
shown
shown in
in
in
Table
Table
Table 7.
7.
7. Future
Future
Future research
research
research will
will
will concentrate
concentrate
concentrate on
on
on restoring
restoring
restoring blurred
blurred
blurred images.
images.
images.
Table
Table 7.
7. Future
Future research
research will will concentrate
concentrate on on restoring
restoring blurred
blurred images.
images.
Table
Table 7.Badly
Table7.
7. Badlytranslated
Badly translatedimages.
translated images.
images.
Table
Table 7.
7. Badly
Badly translated
translated images.
images.

Original
Original image
Original
Original
Original image
image
image
image

Restored
Restored
Restored
Restored image
image
image
image
Restored image

Table
Table 7777 shows
Table
Table shows that
shows
shows that the
that
that the network’s
the
the network’s recovery
network’s
network’s recovery of
recovery
recovery of highly
of
of highly blurred
highly
highly blurred images
blurred
blurred images is
images
images is unsatisfactory
is
is unsatisfactory
unsatisfactory
unsatisfactory
and Table 7in
results shows
furtherthat the network’s
distortion. recovery
Therefore, our of highly
future workblurred
will images is unsatisfactory
concentrate on
and
and
and
and
results
results
results in
in
in further
further
further
results in further
distortion.
distortion.
distortion.
distortion.
Therefore,
Therefore,
Therefore, our
our future
future
future work
work
work will
will
will concentrate
Therefore, our future work will concentrate on
our concentrate
concentrate on enhancing
on
on
enhancing
enhancing
enhancing
enhancing
the
the
the processing
processing
processing
the processing
processing of of
of
of such
such
such
ofsuch images.
images.
images.
suchimages.
images.
the
Author
Author Contributions:
Author Contributions: Conceptualization,
Contributions: Conceptualization, methodology,
Conceptualization, methodology, software,
methodology, software, validation,
software, validation, investigation,
validation, investigation, data
investigation, data
data
Author
curation,Contributions:
curation, writing—original
writing—originalConceptualization,
draft
draft preparation,
preparation,methodology,
T.X.;
T.X.; review
review andsoftware,
and editing,
editing,validation,
T.X.,
T.X., T.Z. investigation,
T.Z. and
and J.Z.
J.Z. All data
All authors
authors
curation,
curation,writing—original
writing—originaldraftdraftpreparation,
preparation,T.X.;
T.X.;review
reviewand andediting,
editing,T.X.,
T.X.,T.Z.
T.Z.and
andJ.Z.
J.Z.All
Allauthors
authors
curation,
have readwriting—original
have read and
and agreed
agreed to
to the
thedraft preparation,
published
published version
version T.X.;
of review
of the
the and editing, T.X., T.Z. and J.Z. All authors
manuscript.
manuscript.
have
have read
read and
and agreed
agreed to
to the
the published
published version
version of of the
the manuscript.
manuscript.
have read and agreed to the published version of the manuscript.
Funding: This research was supported by the Natural Science Foundation of China under Grand
Funding:
Funding: This
This research
research was
was supported
supported byby the
the Natural
Natural Science
Science Foundation
Foundation of of China
China under
under Grand
Grand
Funding:
(No. This research
(No. 52171310),
52171310), the was supported
the National
National Natural by theFoundation
Natural Science
Science Natural Science
Foundation of Foundation
of China
China (No. of ChinaResearch
(No. 52001039),
52001039), under Grand
Research Fund
Fund
(No. 52171310), the National Natural Science Foundation of China (No. 52001039), Research Fund
(No.
from52171310),
from Science
Science and
andthe National Natural
Technology
Technology on Science Vehicle
on Underwater
Underwater Foundation of ChinaLaboratory
Vehicle Technology
Technology (No. 52001039),
Laboratory (No. Research Fund
(No. 2021JCJQ-SYSJJ-
2021JCJQ-SYSJJ-
from
from Science
Science and
and Technology
Technology on on Underwater
Underwater Vehicle
Vehicle Technology
Technology Laboratory
Laboratory (No.
(No. 2021JCJQ-SYSJJ-
2021JCJQ-SYSJJ-
from Science
LB06903).
LB06903). and Technology
Shandong
Shandong Natural on Underwater
Natural Science
Science Foundation
FoundationVehicle
in Technology
in China
China (No. Laboratory (No. 2021JCJQ-SYSJJ-
(No. ZR2019LZH005).
ZR2019LZH005).
LB06903).
LB06903). Shandong
Shandong Natural
Natural Science
Science Foundation
Foundation in in China
China (No.(No. ZR2019LZH005).
ZR2019LZH005).
LB06903). Shandong Natural Science Foundation in China (No. ZR2019LZH005).
Institutional
Institutional Review
Review Board
Board Statement:
Statement: Not
Not applicable.
applicable.
Institutional
Institutional Review
Review Board
Board Statement:
Statement: Not
Not applicable.
applicable.
Institutional Review Board Statement: Not applicable.
Informed
Informed Consent
Consent Statement:
Statement: NotNot applicable.
applicable.
Informed
Informed Consent
Consent Statement:
Statement: NotNot applicable.
applicable.
Informed Consent Statement: Not applicable.
Data
Data Availability
Availability Statement:
Statement: Suggested
Suggested Data
Data Availability
Availability Statements
Statements areare available
available atat https://ir-
https://ir-
Data
Data Availability
Availability Statement:
Statement: Suggested
Suggested Data
Data Availability
Availability Statements
Statements areare available
available atat https://ir-
https://ir-
Data Availability Statement: Suggested Data
vlab.cs.umn.edu/resources/euvp-dataset
vlab.cs.umn.edu/resources/euvp-dataset Availability
(accessed
(accessed on
on 10 Statements2021).
10 December
December are available at https://irvlab.
2021).
vlab.cs.umn.edu/resources/euvp-dataset
vlab.cs.umn.edu/resources/euvp-dataset (accessed
(accessed on on 10
10 December
December 2021).
2021).
cs.umn.edu/resources/euvp-dataset (accessed on 10 December 2021).
Acknowledgments:
Acknowledgments: The The authors
authors would
would like
like to
to thank
thank thethe editors
editors and
and reviewers
reviewers for
for their
their advice.
advice.
Acknowledgments:
Acknowledgments: The The authors
authors would
would like
like to
to thank
thank the
the editors
editors and
and reviewers
reviewers for
for their
their advice.
advice.
Acknowledgments: The authors would like to thank the editors and reviewers for their advice.
Conflicts
Conflicts of
of Interest:
Interest: The
The authors
authors declare
declare no
no conflict
conflict ofof interest.
interest.
Conflicts
Conflicts of
of Interest:
Interest: The
The authors
authors declare
declare no
no conflict
conflict ofof interest.
interest.
Conflicts of Interest: The authors declare no conflict of interest.
References
References
References
References
1.
1. McGlamery,
McGlamery, B.L.B.L. A
A computer
computer model
model for
for underwater
underwater camera
camera systems[C]//Ocean
systems[C]//Ocean Optics
Optics VI.
VI. Int.
Int. Soc.
Soc. Opt.
Opt. Photonics
Photonics 1980,
1980, 208,
208, 221–
221–
1.
1. McGlamery,
McGlamery, B.L.B.L. A
A computer
computer model
model for
for underwater
underwater camera
camera systems[C]//Ocean
systems[C]//Ocean Optics
Optics VI.
VI. Int.
Int. Soc.
Soc. Opt.
Opt. Photonics
Photonics 1980,
1980, 208,
208, 221–
221–
231.
231.
231.
231.
2.
2. Jaffe,
Jaffe, J.S.
J.S. Computer
Computer modeling
modeling and
and the
the design
design of
of optimal
optimal underwater
underwater imaging
imaging systems.
systems. IEEE
IEEE J.J. Ocean.
Ocean. Eng.
Eng. 1990,
1990, 15,
15, 101–111.
101–111.
2.
2. Jaffe,
Jaffe, J.S.
J.S. Computer
Computer modeling
modeling and
and the
the design
design of
of optimal
optimal underwater
underwater imaging
imaging systems.
systems. IEEE
IEEE J.J. Ocean.
Ocean. Eng.
Eng. 1990,
1990, 15,
15, 101–111.
101–111.
https://doi.org/10.1117/12.958279.
https://doi.org/10.1117/12.958279.
https://doi.org/10.1117/12.958279.
https://doi.org/10.1117/12.958279.
J. Mar. Sci. Eng. 2023, 11, 674 13 of 14

References
1. McGlamery, B.L. A computer model for underwater camera systems. Int. Soc. Opt. Photonics 1980, 208, 221–231.
2. Jaffe, J.S. Computer modeling and the design of optimal underwater imaging systems. IEEE J. Ocean. Eng. 1990, 15, 101–111.
[CrossRef]
3. Zhou, J.; Wei, X.; Shi, J.; Chu, W.; Lin, Y. Underwater image enhancement via two-level wavelet decomposition maximum
brightness color restoration and edge refinement histogram stretching. Opt. Express 2022, 30, 17290–17306. [CrossRef] [PubMed]
4. He, K.; Sun, J.; Tang, X. Single image haze removal using dark channel prior. IEEE Trans. Pattern Anal. Mach. Intell. 2010, 33,
2341–2353. [CrossRef] [PubMed]
5. Liang, Z.; Ding, X.; Wang, Y.; Yan, X.; Fu, X. GUDCP: Generalization of Underwater Dark Channel Prior for Underwater Image
Restoration. IEEE Trans. Circuits Syst. Video Technol. 2021, 32, 4879–4884. [CrossRef]
6. Zhou, Y.; Wu, Q.; Yan, K.; Feng, L.; Xiang, W. Underwater image restoration using color-line model. IEEE Trans. Circuits Syst.
Video Technol. 2018, 29, 907–911. [CrossRef]
7. Islam, M.J.; Xia, Y.; Sattar, J. Fast underwater image enhancement for improved visual perception. IEEE Robot. Autom. Lett. 2020,
5, 3227–3234. [CrossRef]
8. Chen, L.; Tong, L.; Zhou, F.; Jiang, Z.; Li, Z.; Lv, J.; Dong, J.; Zhou, H. A Benchmark dataset for both underwater image
enhancement and underwater object detection. arXiv 2020, arXiv:2006.15789. [CrossRef]
9. Hou, G.; Zhao, X.; Pan, Z.; Yang, H.; Tan, L.; Li, J. Benchmarking underwater image enhancement and restoration, and beyond.
IEEE Access 2020, 8, 122078–122091. [CrossRef]
10. WaterGAN: Unsupervised generative network to enable real-time color correction of monocular underwater images. IEEE Robot.
Autom. Lett. 2017, 3, 387–394. [CrossRef]
11. Zhu, J.Y.; Park, T.; Isola, P.; Efros, A.A. Unpaired image-to-image translation using cycle-consistent adversarial networks. In
Proceedings of the IEEE International Conference on Computer Vision, Venice, Italy, 22–29 October 2017; pp. 2223–2232. [CrossRef]
12. Fabbri, C.; Islam, M.J.; Sattar, J. Enhancing underwater imagery using generative adversarial networks. In Proceedings of the
2018 IEEE International Conference on Robotics and Automation (ICRA), Brisbane, Australia, 21–25 May 2018; IEEE: Piscataway,
NJ, USA, 2018; pp. 7159–7165. [CrossRef]
13. Islam, M.J.; Luo, P.; Sattar, J. Simultaneous enhancement and super-resolution of underwater imagery for improved visual
perception. arXiv 2020, arXiv:2002.01155. [CrossRef]
14. Li, C.; Guo, C.; Ren, W.; Cong, R.; Hou, J.; Kwong, S.; Tao, D. An underwater image enhancement benchmark dataset and beyond.
IEEE Trans. Image Process. 2019, 29, 4376–4389. [CrossRef] [PubMed]
15. Hong, L.; Wang, X.; Xiao, Z.; Zhang, G.; Liu, J. WSUIE: Weakly supervised underwater image enhancement for improved visual
perception. IEEE Robot. Autom. Lett. 2021, 6, 8237–8244. [CrossRef]
16. Deng, J.; Dong, W.; Socher, R.; Li, L.-J.; Li, K.; Fei-Fei, L. Imagenet: A large-scale hierarchical image database. In Proceedings of
the 2009 IEEE Conference on Computer Vision and Pattern Recognition, Miami, FL, USA, 20–25 June 2009; IEEE: Piscataway, NJ,
USA, 2009; pp. 248–255. [CrossRef]
17. Moon, J.; Kim, J.; Shin, Y.; Hwang, S. Confidence-aware learning for deep neural networks. In Proceedings of the International
Conference on Machine Learning, PMLR, Virtual, 13–18 July 2020; pp. 7034–7044. [CrossRef]
18. Goodfellow, I.; Pouget-Abadie, J.; Mirza, M.; Xu, B.; Warde-Farley, D.; Ozair, S.; Courville, A.; Bengio, Y. Generative adversarial
networks. Commun. ACM 2020, 63, 139–144. [CrossRef]
19. Isola, P.; Zhu, J.Y.; Zhou, T.; Efros, A.A. Image-to-image translation with conditional adversarial networks. In Proceedings of the
IEEE Conference on Computer Vision and Pattern Recognition, Honolulu, HI, USA, 21–26 July 2017; pp. 1125–1134. [CrossRef]
20. Li, C.; Guo, J.; Guo, C. Emerging from water: Underwater image color correction based on weakly supervised color transfer. IEEE
Signal Process. Lett. 2018, 25, 323–327. [CrossRef]
21. Liu, P.; Wang, G.; Qi, H.; Zhang, C.; Zheng, H.; Yu, Z. Underwater image enhancement with a deep residual framework. IEEE
Access 2019, 7, 94614–94629. [CrossRef]
22. Park, J.; Han, D.K.; Ko, H. Adaptive weighted multi-discriminator CycleGAN for underwater image enhancement. J. Mar. Sci.
Eng. 2019, 7, 200. [CrossRef]
23. Maniyath, S.R.; Vijayakumar, K.; Singh, L.; Sharma, S.K.; Olabiyisi, T. Learning-based approach to underwater image dehazing
using CycleGAN. Arab. J. Geosci. 2021, 14, 1908. [CrossRef]
24. Wang, P.; Chen, H.; Xu, W.; Jin, S. Underwater image restoration based on the perceptually optimized generative adversarial
network. J. Electron. Imaging 2020, 29, 033020. [CrossRef]
25. Zhai, L.; Wang, Y.; Cui, S.; Zhou, Y. Enhancing Underwater Image Using Degradation Adaptive Adversarial Network. In Proceedings
of the 2022 IEEE International Conference on Image Processing (ICIP), Bordeaux, France, 16–19 October 2022; IEEE: Piscataway, NJ,
USA, 2022; pp. 4093–4097. [CrossRef]
26. Choi, Y.; Choi, M.; Kim, M.; Ha, J.W.; Kim, S.; Choo, J. Stargan: Unified generative adversarial networks for multi-domain
image-to-image translation. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Salt Lake City,
UT, USA, 18–22 June 2018; pp. 8789–8797. [CrossRef]
27. Huang, X.; Belongie, S. Arbitrary style transfer in real-time with adaptive instance normalization. In Proceedings of the IEEE
International Conference on Computer Vision, Venice, Italy, 22–29 October 2017; pp. 1501–1510. [CrossRef]
J. Mar. Sci. Eng. 2023, 11, 674 14 of 14

28. Ronneberger, O.; Fischer, P.; Brox, T. U-net: Convolutional networks for biomedical image segmentation. In Proceedings of the
Conference on Medical image Computing and Computer-Assisted Intervention, Munich, Germany, 5–9 October 2015; Springer:
Cham, Switzerland, 2015; pp. 234–241. [CrossRef]
29. Ulyanov, D.; Vedaldi, A.; Lempitsky, V. Instance normalization: The missing ingredient for fast stylization. arXiv 2016, arXiv:1607.08022.
[CrossRef]
30. Odena, A.; Olah, C.; Shlens, J. Conditional image synthesis with auxiliary classifier gans. In Proceedings of the International
Conference on Machine Learning, PMLR, Sydney, Australia, 6–11 August 2017; pp. 2642–2651. [CrossRef]
31. Miyato, T.; Kataoka, T.; Koyama, M.; Yoshida, Y. Spectral normalization for generative adversarial networks. arXiv 2018,
arXiv:1802.05957. [CrossRef]
32. Paszke, A.; Gross, S.; Massa, F.; Lerer, A.; Bradbury, J.; Chanan, G.; Killeen, T.; Lin, Z.; Gimelshein, N.; Antiga, L.; et al. Pytorch:
An imperative style, high-performance deep learning library. Adv. Neural Inf. Process. Syst. 2019, 32. [CrossRef]
33. He, K.; Zhang, X.; Ren, S.; Sun, J. Deep residual learning for image recognition. In Proceedings of the IEEE Conference on
Computer Vision and Pattern Recognition, Las Vegas, NV, USA, 27–30 June 2016; pp. 770–778. [CrossRef]
34. Simonyan, K.; Zisserman, A. Very deep convolutional networks for large-scale image recognition. arXiv 2014, arXiv:1409.1556.
[CrossRef]
35. Geifman, Y.; Uziel, G.; El-Yaniv, R. Bias-reduced uncertainty estimation for deep neural classifiers. In Proceedings of the
International Conference on Learning Representations, New Orleans, LA, USA, 6–9 May 2019. [CrossRef]
36. Naeini, M.P.; Cooper, G.; Hauskrecht, M. Obtaining well calibrated probabilities using bayesian binning. In Proceedings of the
Twenty-Ninth AAAI Conference on Artificial Intelligence, Austin, TX, USA, 25–30 January 2015. [CrossRef]
37. Lakshminarayanan, B.; Pritzel, A.; Blundell, C. Simple and scalable predictive uncertainty estimation using deep ensembles. Adv.
Neural Inf. Process. Syst. 2017, 30, 6402–6413. [CrossRef]
38. Krizhevsky, A.; Hinton, G. Learning Multiple Layers of Features from Tiny Images; Technical Report TR-2009; University of Toronto:
Toronto, ON, Canada, 2009.
39. Netzer, Y.; Wang, T.; Coates, A.; Bissacco, A.; Wu, B.; Ng, A.Y. Reading Digits in Natural Images with Unsupervised Feature
Learning. In Proceedings of the NIPS Workshop on Deep Learning and Unsupervised Feature Learning, Granada, Spain, 12–17
December 2011; Available online: https://storage.googleapis.com/pub-tools-public-publication-data/pdf/37648.pdf (accessed
on 5 January 2023).
40. Kingma, D.P.; Ba, J. Adam: A method for stochastic optimization. arXiv 2014, arXiv:1412.6980. [CrossRef]
41. Panetta, K.; Gao, C.; Agaian, S. Human-visual-system-inspired underwater image quality measures. IEEE J. Ocean. Eng. 2015, 41,
541–551. [CrossRef]

Disclaimer/Publisher’s Note: The statements, opinions and data contained in all publications are solely those of the individual
author(s) and contributor(s) and not of MDPI and/or the editor(s). MDPI and/or the editor(s) disclaim responsibility for any injury to
people or property resulting from any ideas, methods, instructions or products referred to in the content.