IEEE TRANSACTIONS ON MEDICAL IMAGING, VOL. 41, NO.
11, NOVEMBER 2022 3357
Triplet Cross-Fusion Learning for Unpaired
Image Denoising in Optical
Coherence Tomography
Mufeng Geng , Xiangxi Meng , Lei Zhu , Zhe Jiang , Mengdi Gao, Zhiyu Huang,
Bin Qiu, Yicheng Hu, Yibao Zhang , Qiushi Ren, and Yanye Lu
Abstract — Optical coherence tomography (OCT) is a
widely-used modality in clinical imaging, which suffers from
the speckle noise inevitably. Deep learning has proven
its superior capability in OCT image denoising, while the
difficulty of acquiring a large number of well-registered
OCT image pairs limits the developments of paired learning
methods. To solve this problem, some unpaired learning
methods have been proposed, where the denoising net-
works can be trained with unpaired OCT data. However,
majority of them are modified from the cycleGAN frame-
work. These cycleGAN-based methods train at least two
generators and two discriminators, while only one gener-
ator is needed for the inference. The dual-generator and
dual-discriminator structures of cycleGAN-based methods
demand a large amount of computing resource, which may
be redundant for OCT denoising tasks. In this work, we pro-
Manuscript received 18 March 2022; revised 10 May 2022 and 9 June
2022; accepted 15 June 2022. Date of publication 20 June 2022; date
of current version 27 October 2022. This work was supported in part
by the Beijing Natural Science Foundation under Grant Z210008; in
part by the Shenzhen Science and Technology Program under Grant
KQTD20180412181221912 and Grant JCYJ20200109140603831; and
in part by the Open Project through the Key Laboratory of Carcinogen-
esis and Translational Research, Ministry of Education/Beijing, under
Grant 2022 Open Project-2. (Corresponding author: Yanye Lu.)
This work involved human subjects or animals in its research. Approval
of all ethical and experimental procedures and protocols was granted by
the Ethics Committee of Peking University First Hospital under Applica-
tion No. 2017-47, and performed in line with the Ethical Standards of the
Institutional and/or National Research Committee, and the Declaration
of Helsinki.
Mufeng Geng, Lei Zhu, Zhe Jiang, Mengdi Gao, Zhiyu Huang, Bin Qiu,
Yicheng Hu, and Qiushi Ren are with the Institute of Medical Technology,
Peking University Health Science Center, Peking University, Beijing
100191, China, also with the Department of Biomedical Engineering,
College of Future Technology, Peking University, Beijing 100871, China,
also with the Institute of Biomedical Engineering, Peking University
Shenzhen Graduate School, Shenzhen 518055, China, also with
the Institute of Biomedical Engineering, Shenzhen Bay Laboratory,
Shenzhen 518071, China, and also with the National Biomedical
Imaging Center, Beijing 100871, China (e-mail: gmfpku@pku.
edu.cn; zhulei@stu.pku.edu.cn; gjiang47@163.com; 1901111963@
pku.edu.cn; hzy_bme@pku.edu.cn; qiub@pku.edu.cn; huyc@stu.
pku.edu.cn; qren@pku.edu.cn).
Xiangxi Meng and Yibao Zhang are with the Key Laboratory of Car-
cinogenesis and Translational Research (Ministry of Education), Peking
University Cancer Hospital and Institute, Beijing 100142, China (e-mail:
mengxiangxi@pku.edu.cn; zhangyibao@pku.edu.cn).
Yanye Lu is with the Institute of Medical Technology, Peking Univer-
sity Health Science Center, Peking University, Beijing 100191, China,
and also with the Institute of Biomedical Engineering, Peking Univer-
sity Shenzhen Graduate School, Shenzhen 518055, China (e-mail:
yanye.lu@pku.edu.cn).
Digital Object Identifier 10.1109/TMI.2022.3184529
1558-254X © 2022 IEEE. Personal use is permitted, but republication/redistribution requires IEEE permission.
See https://www.ieee.org/publications/rights/index.html for more information.
Authorized licensed use limited to: Hong Kong Polytechnic University. Downloaded on June 01,2024 at 05:15:34 UTC from IEEE Xplore. Restrictions apply.
pose a novel triplet cross-fusion learning (TCFL) strategy
for unpaired OCT image denoising. The model complexity
of our strategy is much lower than those of the cycleGAN-
based methods. During training, the clean components and
the noise components from the triplet of three unpaired
images are cross-fused, helping the network extract more
speckle noise information to improve the denoising accu-
racy. Furthermore, the TCFL-based network which is trained
with triplets can deal with limited training data scenarios.
The results demonstrate that the TCFL strategy outperforms
state-of-the-art unpaired methods both qualitatively and
quantitatively, and even achieves denoising performance
comparable with paired methods. Code is available at:
https://github.com/gengmufeng/TCFL-OCT.
Index Terms — Unpaired learning, optical coherence
tomography, image restoration.
I. I NTRODUCTION
O PTICAL coherence tomography (OCT), a non-invasive
cross-sectional imaging technique, plays an essential role
in clinical imaging [1], such as brain imaging, ophthalmol-
ogy, cardiology, etc. In practice, OCT images are inevitably
susceptible to the speckle noise, which severely degrades the
image quality and affects clinical diagnosis. Existing OCT
denoising methods can be divided into two categories: the
hardware-based methods and the software-based methods.
As for the hardware-based approaches such as improving
the light source, the noise of detectors and scanners can be
reduced. However, the speckle or white noise in the imaging
system cannot be eliminated [2], limiting the development of
hardware-based methods. In contrast, software-based methods
post-process OCT images directly, and are not restricted to
specific acquisition processes. A range of traditional algorith-
mic methods have been developed for OCT image denoising,
including block-matching 3D (BM3D) [3], nonlocal-means
(NLM) [4], noise adaptive wavelet thresholding algorithm [5],
etc. Although these methods can reduce the speckle noise to
various extents, they always cause certain noise residuals and
the loss of many structural details in the denoised images.
Recently, deep learning has been widely utilized in the
medical imaging field due to its powerful modeling capability,
such as low-dose reconstruction [6], material decomposi-
tion [7], super-resolution [8], segmentation [9], as well as
OCT image denoising. Compared with traditional algorithms,
3358
deep learning-based OCT denoising methods have demon-
strated their superior capabilities in both speckle noise reduc-
tion and structure preservation. In order to improve the
denoising effect, researchers have made some attempts in
optimizing deep network structures [10], [11] or design-
ing structure-sensitive loss functions [12], [13]. Specifically,
Xu et al. combined deep nonlinear convolutional neural net-
work (CNN) mapping and residual learning and proposed
the OCTNet, realizing speckle noise reduction and texture
preservation for low-quality OCT images [10]. Shen et al.
reported a double-path parallel CNN (DPNet) to extract deeper
denoising information from the noisy OCT images [11].
Ma et al. designed an edge-sensitive loss function inspired by
the edge preservation index (EPI) to make the deep denoiser
sensitive to edge-related details [12]. Qiu et al. presented a
perceptually-sensitive loss function to preserve more structure
information in the denoised OCT images [13]. Meanwhile,
there are also a number of OCT denoising works based on
generative adversarial networks (GANs) to better regularize
the training process of denoising tasks [14], [15]. Although
the above-mentioned deep learning methods have achieved
impressive results in OCT denoising tasks, they all follow
the supervised training strategy, requiring noisy-clean paired
OCT images. Clean labels are usually obtained by multi-frame
averaging, which is time-consuming. It has been repeatedly
demonstrated that the averaging operation might introduce
motion artifacts and lose some critical structure details includ-
ing some small lesions or abnormal features (e.g., hyper reflec-
tion), due to unconscious eye movement or body jitter during
scanning [16], [17]. Thus, it is hard to acquire the ideally-
registered noisy-clean OCT image pairs in practice. When the
supervised deep learning networks are trained with such non-
ideal noisy-clean image pairs, the trained networks tend to
smooth some critical structure details [16], [17]. Furthermore,
some weakly-supervised OCT denoising methods have been
proposed based on the Noise2Noise (N2N) strategy [18]–[20].
Although the N2N strategy mitigates the need of noisy-clean
image pairs, capturing well-registered noisy-noisy OCT image
pairs remains a very challenging problem due to the uncon-
scious eye movement in clinical scannings. Both supervised
and weakly-supervised methods belong to paired methods,
where the requirement of paired training data limits their
developments.
In order to avoid the above-mentioned problem of paired
training data, some unpaired OCT image denoising methods
have been proposed. Guo et al. reported the nonlocal-GAN
model to denoise OCT images [21], which considered that
the background areas of OCT images mainly contained pure
“real” noise, and used the background areas to guide the whole
image denoising process. However, in practice, the background
areas of OCT images seldom purely contain noise information.
Therefore, the background areas inevitably introduced some
wrong noise priors, limiting the denoising accuracy. Most
existing unpaired OCT image denoising methods are based
on the cycleGAN framework [16], [17], [22]–[25]. Although
cycleGAN was not originally designed for OCT image denois-
ing, some researchers have tried to optimize it in terms of
network structures [17], [23], [24] or objective loss functions
Authorized licensed use limited to: Hong Kong Polytechnic University. Downloaded on June 01,2024 at 05:15:34 UTC from IEEE Xplore. Restrictions apply.
IEEE TRANSACTIONS ON MEDICAL IMAGING, VOL. 41, NO. 11, NOVEMBER 2022
[16], [25] to fit OCT image denoising tasks. They treat the
OCT denoising problem as a domain translation between the
noisy domain and the clean domain, which requires at least
two generators (one for noisy-to-clean translation and the other
for clean-to-noisy translation) and at least two discriminators
(one for noisy image discrimination and the other for clean
image discrimination). As a result, the network structures
of these methods are always complex with huge parameters,
which may be redundant for OCT image denoising tasks and
may cause overfitting and poor generality [15].
In this work, we propose a novel triplet cross-fusion learn-
ing (TCFL) strategy for unpaired OCT image denoising. The
model complexity of our strategy is much lower than those of
the cycleGAN-based methods, where only one generator and
one discriminator are utilized in training. In addition, unlike
most existing denoising methods whose training is based
on two paired/unpaired images, the TCFL-based network is
trained based on three unpaired images (i.e., a triplet), includ-
ing two unpaired noisy images and one clean image. During
training, the clean components and the noise components
from the triplets are cross-fused, helping the network extract
more speckle noise information to improve the denoising
accuracy. To enable the capabilities of noise reduction and
structure preservation, based on the TCFL strategy, a global
residual learning is utilized, and an identity loss is designed.
The proposed TCFL strategy is verified over three OCT
datasets. The results demonstrate that the proposed strategy
outperforms existing unpaired state-of-the-art methods both
qualitatively and quantitatively, and even achieves denoising
performance comparable with paired methods. This might be
because the information interaction of clean components and
noise components among the training samples gets enhanced
through our TCFL strategy, which helps deep learning net-
works better distinguish between the clean components and
the noise components. In summary, the contributions of this
paper are as follows:
• We propose a TCFL strategy for unpaired OCT image
denoising, where three unpaired images are cross-fused
during training, helping the denoising network extract
more speckle noise information to improve the denoising
accuracy.
• Based on the TCFL strategy, an identity loss is designed
to enable the convergence of the denoising network.
Ablation studies demonstrate that the identity loss is
essential for effective denoising.
• Compared with cycleGAN-based methods which involve
at least two generators and two discriminators, the
proposed TCFL strategy can be implemented with
only one generator and one discriminator, ensuring
lower model complexity. Extensive experimental results
demonstrate that the proposed strategy has reached the
state-of-the-art in the unpaired OCT image denoising
field.
• The use of three unpaired images results in a large
number of triplet combinations based on the fixed training
data, which can deal with the problem of limited train-
ing data. Related experiments are conducted to confirm
this.
GENG et al.: TRIPLET CROSS-FUSION LEARNING FOR UNPAIRED IMAGE DENOISING IN OCT
Fig. 1. Training data requirements of (a) paired image denoising and (b) unpaired image denoising.
II. R ELATED W ORKS
Over the years, numerous deep learning-based image
denoising methods have been proposed, which can be divided
into paired and unpaired learning methods. Fig. 1 compares
their training data requirements.
A. Paired Image Denoising
Paired learning methods can be further divided into super-
vised methods and weakly-supervised methods. As for the
supervised image denoising, noisy-clean image pairs are
needed to train the network. Zhang et al. proposed DnCNN
which first used a CNN and a global residual learning in image
denoising [26]. DnCNN verifies the effectiveness of deep
CNNs for image denoising tasks, and has been optimized and
widely used in various medical image denoising scenarios [6].
In the field of OCT image denoising, a number of supervised
learning-based methods have been proposed [10]–[15], which
have been reviewed in Section I. However, in practice, it is
always hard and expensive to acquire a large number of
noisy-clean image pairs to train the deep learning network,
limiting the development of supervised image denoising.
Weakly-supervised denoising networks can be trained using
a pair of noisy images. Lehtinen et al. first statistically
demonstrated that under some reasonable assumptions on
noise distribution, it is possible to retrieve the underlying clean
image using only noisy images during training, and proposed
a weakly-supervised denoising strategy, N2N. After that, N2N
was extended to various image denoising tasks [19], [27].
Qiu et al. comprehensively studied the denoising performance
of four common deep networks in the OCT image denoising
task following the N2N strategy [19]. Ahmed et al. reported
a collaborative technique to train multiple N2N denoisers
to address low-dose computed tomography (CT) denoising
tasks [27]. Although N2N avoids the use of clean-noisy
paired images, the collection of a large number of well-
registered noisy-noisy image pairs might be also challenging,
especially for the medical imaging due to the unconscious
body movement during scanning.
B. Unpaired Image Denoising
Unpaired learning methods mainly include self-supervised
methods and cycleGAN-based methods [28]. Self-supervised
denoising methods train the deep denoising networks based on
a single noisy image, including Noise2Self [29], Recorrupted-
to-Recorrupted (R2R) [30], Neighbor2Neighbor (NBR) [31],
etc. Noise2Self utilized a blind-spot strategy to make deep
networks learn the clean image from a single noisy image,
while R2R and NBR manually created paired noisy-noisy
Authorized licensed use limited to: Hong Kong Polytechnic University. Downloaded on June 01,2024 at 05:15:34 UTC from IEEE Xplore. Restrictions apply.
3359
images from a single noisy image to meet the N2N con-
dition. These self-supervised methods were mainly used in
natural image denoising and involved requirements in terms
of noise characteristics, such as zero-mean noise distribution.
These requirements are implausible for some medical image
denoising tasks, limiting the applications in this area.
Some researchers tried to use cycleGAN in unpaired image
denoising [22], [32]–[34], and the training of this method
is based on two unpaired noisy-clean images. Zhu et al.
first proposed cycleGAN for unpaired image-to-image trans-
lation [22]. After that, cycleGAN was widely adopted in the
field of image denoising, including OCT image denoising.
Specifically, Guo et al. proposed a structure-aware noise reduc-
tion GAN (SNRGAN) for unpaired OCT image denoising,
which learned the mapping function between two domains
(the clean and the noisy domains), and a structure-aware
loss based on the structural similarity (SSIM) index was
introduced [16]. Similarly, Manakov et al. developed a hybrid
discriminator cycleGAN (HDcycleGAN) to realize unpaired
OCT image denoising, which introduced skip connections
into generators and utilized a shared discriminator [23].
Das et al. reported an unpaired denoising framework inspired
by cycleGAN to perform fast and reliable OCT denoising [24].
Huang et al. presented a disentangled representation GAN
(DRGAN), which also introduced the cycle-consistency idea
and transformed the speckle noise reduction problem to a noise
disentanglement problem [17]. Wu et al. proposed a structure-
persevered cycleGAN (SPcycleGAN) to denoise OCT images
and used a structure-specific cross-domain description to pre-
serve retinal details [25]. These works have demonstrated the
capability of the cycleGAN framework to address the OCT
image denoising tasks.
III. M ETHODS
A. OCT Speckle Noise Model
In clinical settings, considering the limited dynamic range
of display monitors and the perception of the human eyes,
projecting the measured OCT data into the logarithm space
to acquire the finally-displayed OCT images is widely
adopted [35]. As a result, it causes the conversion of speckle
noise from multiplicative (data-dependent) to additive (data-
independent) [36]. Thus, in this work, we simplify a speckle
noise-corrupted OCT image I as the sum of its clean compo-
nent C and its speckle noise N.
I = C + N. (1)
We aim to restore C from I to improve the OCT image
quality. Fig. 2 shows a noisy OCT image and its corresponding
3360
Fig. 2. Speckle noise-corrupted OCT image (left), the corresponding
clean OCT image (middle), and the speckle noise (right).
clean image and speckle noise. The clean image is generated
with 50-frame averaging, and the speckle noise is the differ-
ence between the noisy image and the clean image. We can
hardly see any structural information from the speckle noise
image, which supports the rationality of the additive noise
model in (1).
B. Motivation
In the field of OCT image denoising, in contrast with
paired learning, unpaired learning breaks the limit of paired
training data, reducing the difficulty of data acquisition.
Among unpaired OCT image denoising methods, cycleGAN
and its variants (such as HDcycleGAN [23], SNRGAN [16],
DRGAN [17], SPcycleGAN [25], etc.) have become the
mainstream solutions, due to the similarities between image-
to-image translation and image denoising. In detail, cycleGAN
was first proposed to solve unpaired image-to-image transla-
tion problems, which aimed to translate images between two
domains, X = {I1 , I2 , . . . , Im } and Y = {C1 , C2 , . . . , Cm },
in the absence of paired samples. It is noteworthy that the
image-to-image translation through cycleGAN is bidirectional,
consisting of both the mapping from X to Y and the inverse
mapping from Y to X . As a result, cycleGAN-based methods
contain at least two generators (G 1 for X → Y and G 2 for
Y → X ) and at least two discriminators (D1 for discrimination
of X and D2 for discrimination of Y). Migrating to OCT
image denoising, X is regarded as the noisy domain, and Y
corresponds to the clean domain. That is, the OCT image
denoising problem is treated as a special case of image-
to-image translation through cycleGAN. Existing cycleGAN-
based OCT image denoising methods train both G 1 for
X → Y and G 2 for Y → X , while only G 1 is needed
for the inference. The dual-generator and dual-discriminator
structures of cycleGAN-based methods demand a large amount
of computing resource, which may be redundant for OCT
denoising tasks. Thus, there is a demand of exploring an alter-
native unpaired image denoising method to cycleGAN-based
denoising methods, reducing the model complexity with fewer
generators and fewer discriminators.
In this work, based on the observation that the speckle
noise in OCT images is additive in the logarithm space,
we design a simple yet effective clean-to-noisy translation
approach. A fake noisy OCT image can be synthesized by
adding a clean OCT image and the speckle noise of another
noisy OCT image. In order to enable the loss constraint and
the convergence of the network, the denoising network is
trained based on three unpaired OCT images including two
Authorized licensed use limited to: Hong Kong Polytechnic University. Downloaded on June 01,2024 at 05:15:34 UTC from IEEE Xplore. Restrictions apply.
IEEE TRANSACTIONS ON MEDICAL IMAGING, VOL. 41, NO. 11, NOVEMBER 2022
Fig. 3. The proposed triplet cross-fusion learning (TCFL) strategy, whose
training utilizes only one generator (i.e., noise predictor) and one discrim-
inator. The training of noise predictor consists of three sequential parts:
(a) Noisy2Predicted-Clean (N2P) part, (b) Predicted-Clean2Predicted-
Clean (P2P) part, and (c) Clean2Predicted-Clean (C2P) part.
(d) Discriminator and loss calculation.
unpaired noisy images and one clean image. A discriminator
is also introduced to help the denoising network generate
realistic fake images. The proposed strategy involves only one
generator and one discriminator. Obviously, compared with
cycleGAN-based methods, the model complexity is reduced
significantly with our proposed strategy.
C. Triplet Cross-Fusion Learning
1) Overview: The proposed TCFL strategy is able to utilize
concise structures to finish the training of the denoising
network in an end-to-end manner. Three unpaired OCT images
are used to train the TCFL-based network, including one
noisy image IA , the other one noisy image IB , and one
clean image CC . As shown in Fig. 3, only one generator
(i.e., noise predictor) and one discriminator are utilized under
the proposed TCFL strategy. The training of noise predic-
tor consists of three sequential parts: Noisy2Predicted-Clean
(N2P) part, Predicted-Clean2Predicted-Clean (P2P) part, and
Clean2Predicted-Clean (C2P) part. Algorithm 1 presents the
pseudocode of the training and inference of the TCFL-based
network.
2) N2P Part: In this part, IA and IB first go through noise
predictor pn (·) to generate two speckle noise images NA and
NB , respectively.
NA = pn (IA ),
NB = pn (IB ). (2)
Then, two fake clean images and CA1 CB1
can be estimated
by subtracting noisy inputs from predicted noise (i.e., global
GENG et al.: TRIPLET CROSS-FUSION LEARNING FOR UNPAIRED IMAGE DENOISING IN OCT 3361

Algorithm 1 Training and Inference of the TCFL-Based two clean images CA2 and CB2 , respectively.
Network
CA2 = IA − pn (IA ),
Require: Three unpaired OCT images, IA , IB , and CC ;
Require: A noise predictor pn and a discriminator D; CB2 = IB − pn (IB ). (5)
Require: Initialize parameters θ pn and θ D randomly; Although the speckle noise components in IA and IA are
for each epoch do different, their clean components (i.e., CA1 and CA2 ) should be
Sample the batch;
identical. Analogically, CB1 and CB2 should also be identical.
for each iteration do
4) C2P Part: Similar to the synthesis of IA and IB , we can
Noisy2Predicted-Clean (N2P) part:
pn (IA ) → NA ; synthesize two fake noisy images IC1 and IC2 based on CC ,
pn (IB ) → NB ; NA , and NB .
IA − NA → CA1 ; IC1 = CC + NA ,
IB − NB → CB1 ; IC2 = CC + NB . (6)
Predicted-Clean2Predicted-Clean (P2P) part:
CA1 + NB → IA (cross-fusion); Then, IC1 and IC2 go through pn (·) and subtraction operations
CB1 + NA → IB (cross-fusion); to estimate two clean images CC1 and CC2 , respectively.
IA − pn (IA ) → CA2 ;
CC1 = IC1 − pn (IC1 ),
IB − pn (IB ) → CB2 ;
Clean2Predicted-Clean (C2P) part: CC2 = IC2 − pn (IC2 ). (7)
CC + NA → IC1 (cross-fusion); Ideally, CC , CC1 and CC2 should be identical.
CC + NB → IC2 (cross-fusion); 5) Discriminator: In this work, we adopt a discriminator to
IC1 − pn (IC1 ) → CC1 ; realize the adversarial learning with the generator (i.e., noise
IC2 − pn (IC2 ) → CC2 ; predictor). The generator in TCFL is designed to estimate
Calculate identity loss and GAN loss; noise, whereas the discriminator of TCFL is used to identify
Update pn : θ pn ← Adam; real clean OCT images CC from the predicted OCT images
Update D: θ D ← Adam; C = {CA1 , CA2 , CB1 , CB2 , CC1 , CC2 }, as shown in Fig. 3 (d).
Return: Trained pn ; 6) Loss Function: The total loss of TCFL L total can be
Inference: Given the noisy image ID from the test set, expressed as:
predict the clean image CD : ID − pn (ID ) → CD . L total = L GAN + λL iden , (8)
where L GAN denotes the GAN loss contributed by the discrim-
inator D; L iden indicates the specifically-designed identity loss
residual learning). It has been reported in the literature that of generator G; λ is the weight of L iden .
the global residual learning is able to help the deep denoisers Inside (8), L GAN can be written as:
protect more structural details [10], [26].
L GAN = E[logD(C)] + E[log(1− D(C ))], (9)
CA1 = IA − NA , where E[·] represents the expectation operator; C is the real
CB1 = IB − NB . (3) clean image; C denotes the fake clean image. D tries to
maximize this objective, while G tries to minimize it.
It can be seen that in this part the noise predictor pn (·) predicts The identity loss L iden is defined as follows:
two speckle noise images (NA and NB ) and two clean images
(CA1 and CB1 ) with two noisy images (IA and IB ) as inputs L iden = L iden A + L iden B + μ(L iden C1 + L iden C2 ), (10)
in an independent and parallel manner. where μ is the loss weight, and {L iden A , L iden B , L iden C1 ,
3) P2P Part: Due to the data-independence of speckle L iden C2 } can be expressed as:
noise in the logarithm space, we can synthesize two fake
noisy images by a cross-fusion mechanism. Specifically, the L iden A = CA1 − CA2 1 ,
cross-fusion mechanism aims to synthesize a new noisy image L iden B = CB1 − CB2 1 ,
by adding the clean component from one image and the L iden C1 = CC1 − CC 1 ,
noise component from another image. That is, one fake
L iden = CC2 − CC 1 , (11)
noisy image IA can be obtained by adding CA1 and NB ; the C2

other one fake noisy image IB can be obtained by adding
CB1 and NA .
IA = CA1 + NB ,
IB = CB1 + NA .
After that, IA and IB are inputted into the noise predictor pn (·)
and subsequently performed subtraction operations to estimate
where  · 1 represents L1 norm, L iden A indicates the identity
loss between CA1 and CA2 ; L iden B denotes the identity loss
between CB1 and CB2 ; L iden C1 is the identity loss between
CC1 and CC ; L iden C1 represents the identity loss between
(4) CC2 and CC .
The designed identity loss L iden enables the conver-
gence of the denoising network. Under the constraints of
L iden , the clean components and the noise components from

Authorized licensed use limited to: Hong Kong Polytechnic University. Downloaded on June 01,2024 at 05:15:34 UTC from IEEE Xplore. Restrictions apply.
3362 IEEE TRANSACTIONS ON MEDICAL IMAGING, VOL. 41, NO. 11, NOVEMBER 2022

Fig. 4. An representative implementations of the noise predictor

and the discriminator. (a) DnCNN (noise predictor). (b) PatchGAN
(discriminator).

{IA , IB , CC } can be cross-fused diversely, resulting in that the

network can extract more speckle noise information. Thus,
the clean components and noise components can be better
distinguished.

Fig. 5. Example images from PKU37, DUKE17, and DUKE28. For each
IV. E XPERIMENTAL S ETUP AND R ESULTS image, four ROIs are chosen to calculate unsupervised metrics.

A. Experimental Setup
1) Network Implementation: The proposed TCFL strategy
can be implemented based on existing CNNs. In this work,
we adopted a classical denoising CNN, DnCNN [26], as the
noise predictor and utilized PatchGAN [22] as the discrimina-
tor. DnCNN has been widely used in many image denoising
tasks, which is simple yet effective. PatchGAN constrains the
image style and texture at the scale of patches, and has also
been widely used. Fig. 4 shows their network architectures.
Compared with the original version in [26], the DnCNN
adopted in this work was modified: 1) the residual learning was
discarded to output noise; 2) ReLU was replaced with Leaky
ReLU to fix the ReLU dying problem [37]. The PatchGAN
used in this work kept the same architecture as the version
in [22]. We named the network implementation of TCFL using
DnCNN and PatchGAN as TCFL-DnCNN.
2) Data Preparation: In this work, we evaluated the proposed
TCFL strategy over three OCT datasets, including one not-yet-
public dataset (namely PKU371) and two publicly available
datasets (namely DUKE17 [38] and DUKE28 [39]). Fig. 5
shows the example images from the three datasets. PKU37 was
acquired with a customized spectral-domain OCT (SDOCT)
system, whose lateral resolution and axial resolution were
16 μm and 6 μm, respectively. The central wavelength of the
light source was 845 nm, and the full width at half maximum
bandwidth was 45 nm. Averaging 50 frames was adopted to
acquire the clean images. That is, in PKU37, one clean image
corresponds to 50 noisy images with independent speckle
noise. More details about the acquisition modes of PKU37
can be found in [13]. Table I presents main information of the
three datasets, and the detailed information of the other two
public datasets can be found in [38] and [39].
1 PKU37 will be released at https://wiki.milab.wiki/display/LF/Open+Source
+Project soon
TABLE I
T HREE OCT D ATASETS U SED IN T HIS W ORK
For the division of training set, validation set, and test
set, the images from 20 subjects in PKU37 were selected
for training (namely PKU37-train), and the images from
the rest 17 subjects in PKU37 were used for validation
(namely PKU37-validation). PKU37-validation was used for
the hyperparameter tuning of all deep learning-based methods.
In order to verify the generalization of the deep learning-based
methods, both DUKE17 and DUKE28 were used for cross-
domain test.
3) Training Details: We implemented the proposed
TCFL-based network through the PyTorch framework [40].
An NVIDIA GTX 3090 GPU was used for all the training.
Adam [41] was adopted as the optimizer, with momentum 1
and momentum 2 as 0.5 and 0.999, respectively. Batch
size was 2, and learning rate was set to 0.0002. For
model initialization, the weights in convolutions and batch
normalization were random numbers which followed the
normal distributions of N(0, 0.02) and N(1.0, 0.02),
respectively, and the biases were both initialized as 0.
The training epoch number was 100, and the training
time was 40 hours. The weights λ and μ in (8) and (10)
were empirically set as 6 and 1, respectively, which were
determined by a number of experiments.

Authorized licensed use limited to: Hong Kong Polytechnic University. Downloaded on June 01,2024 at 05:15:34 UTC from IEEE Xplore. Restrictions apply.
GENG et al.: TRIPLET CROSS-FUSION LEARNING FOR UNPAIRED IMAGE DENOISING IN OCT
TABLE II
Q UANTITATIVE C OMPARISON OF THE P ROPOSED TCFL-D N CNN AND S EVEN T RADITIONAL OR U NPAIRED M ETHODS ON PKU37-VALIDATION
4) Baselines and Quantitative Measurements: To better ver-
ify the effectiveness of the proposed TCFL strategy, nine
methods were implemented for comparison, including two
traditional algorithms and seven deep learning-based meth-
ods. The traditional algorithms include BM3D [3] and
NLM [4]. The deep learning-based methods consist of one
supervised method (OCTNet [10]), one weakly-supervised
method (N2N-DnCNN [19]), and five unpaired methods
(nonlocal-GAN [21], R2R-DnCNN [30], NBR-DnCNN [31],
DRGAN [17], and SPcycleGAN [25]). For the traditional
algorithms, we determined their parameters with a range of
experiments to achieve the best denoising performance. The
deep learning-based methods were implemented according to
the publicly available codes released by the authors. All deep
learning-based methods adopted the same division of training
set, validation set, and test set as TCFL-DnCNN. For a fair
comparison, we tried different hyperparameters for each deep
learning-based method to obtain the optimal denoising per-
formance on PKU37-validation. The hyperparameters leading
to the optimal results on PKU37-validation were selected as
the final hyperparameters. The image pairs during one training
epoch were also the same, 1000 pairs.
In terms of quantitative comparison, three unsupervised
metrics and three supervised metrics were used. The adopted
unsupervised metrics included signal-to-noise ratio(SNR),
contrastto-noise ratio(CNR), and EPI. Referring to [10], [15],
[17], the unsupervised metrics were calculated based on three
signal regions of interests (ROIs) and one background ROI.
As shown in Fig. 5, the signal ROIs (red rectangles) located
at or near the retinal layers, and the background ROI (green
rectangles) was the homogeneous region. As for supervised
metrics, in addition to two commonly-used metrics, peak
signal-to-noise ratio (PSNR) and SSIM, we also calculated
the gradient conduction mean square error (GCMSE) [42],
which is very sensitive to edges and borders. Smaller value
of GCMSE indicates the better denoising performance, and
larger values of the other quantitative metrics denote higher
performance.
B. Comparison With Traditional/Unpaired Methods
As shown in Fig. 6, we selected one representative OCT
image from PKU37-validation to visualize the denoising
performance of all methods. Two ROIs were magnified
for a clearer comparison. Figs. 6 (b1) and (b2) illustrate the
denoised results of traditional algorithms, and Figs. 6 (c1)-(c6)
show the denoised results of six unpaired methods. Most
Authorized licensed use limited to: Hong Kong Polytechnic University. Downloaded on June 01,2024 at 05:15:34 UTC from IEEE Xplore. Restrictions apply.
3363
unpaired methods generate fewer noise residues than tra-
ditional algorithms. Specifically, BM3D introduces some
streak artifacts after denoising, and NLM is not competent
to the speckle noise reduction task, with a lot of noise
residues. Among unpaired methods, the image denoised by
nonlocal-GAN loses many retinal layer signals; both R2R-
DnCNN and NBR-DnCNN enhance the original noisy OCT
image slightly, while there are many noise residues; the
denoising result of DRGAN has low contrast which heavily
weakens the blood flow information, as indicated by the
arrows in Fig. 6 (c4); SPcycleGAN reduces speckle noise to
a certain extent, while it also loses some structural details,
as indicated by the arrows in Fig. 6 (c5). In contrast, the
proposed TCFL-DnCNN not only reduces the speckle noise
effectively, but also well preserves many structural details and
blood flow information.
Table II summarizes the quantitative results
(mean±standard deviation) of traditional and unpaired
methods over PKU37-validation. It can be observed that
the proposed TCFL-DnCNN outperforms the traditional and
unpaired reference methods in terms of most quantitative
metrics except EPI. NLM achieves a higher EPI value than
TCFL-DnCNN. Considering that EPI is calculated using
the images before and after denoising, we speculate that
this is because NLM could not effectively eliminate the
speckle noise and the difference between the images before
and after denoising was not significant. We also compared
the inference time of different algorithms, as shown in
Table II. The proposed TCFL-DnCNN, R2R-DnCNN, and
NBR-DnCNN are much faster than other methods, which
have the potential for in vivo OCT denoising.
C. Comparison With Paired Methods
Fig. 6 (d1) presents the denoising result of the supervised
OCTNet, and Fig. 6 (d2) shows the denoising result of the
weakly-supervised N2N-DnCNN. It can be seen that, although
OCTNet and N2N-DnCNN generate smoother images than the
proposed TCFL-DnCNN, they (especially N2N-DnCNN) suf-
fer from the over-smoothness problem, losing some structural
details or blood flow information, as indicated by the arrows in
Figs. 6 (d1) and (d2). In contrast, TCFL-DnCNN achieves a
good balance between denoising and structure preservation,
not only improving the visual quality but also preserving
essential structures.
Table III presents the quantitative results of TCFL-DnCNN,
OCTNet, and N2N-DnCNN on PKU37-validation. TCFL-
DnCNN achieves the highest EPI and the lowest GCMSE,
3364 IEEE TRANSACTIONS ON MEDICAL IMAGING, VOL. 41, NO. 11, NOVEMBER 2022

Fig. 6. Denoising performance comparison of different methods on PKU37-validation. (a1) Clean. (a2) Noisy. (b1) BM3D. (b2) NLM. (c1) nonlocal-
GAN. (c2) R2R-DnCNN. (c3) NBR-DnCNN. (c4) DRGAN. (c5) SPcycleGAN. (c6) Our TCFL-DnCNN. (d1) OCTNet. (d2) N2N-DnCNN.

both of which are sensitive to the edge and texture. In terms of TABLE III
SNR, CNR, and SSIM, the proposed TCFL-DnCNN presents Q UANTITATIVE C OMPARISON OF THE P ROPOSED TCFL-D N CNN AND
T WO PAIRED M ETHODS ON PKU37-VALIDATION
the second best. Although OCTNet and N2N-DnCNN achieve
higher PSNR as a result of over-smoothness, the higher PSNR
values do not represent that the denoised images have a better
structural and texture preservation. The quantitative results
confirm our visual observations that TCFL-DnCNN tends to
preserve more structural information. As for the inference
time, as shown in Table III, both TCFL-DnCNN and N2N-
DnCNN are faster than OCTNet.
D. Generalization Comparison
Medical imaging including OCT always suffers from the methods by two cross-domain tests. Specifically, the deep
domain-shift problem, due to different scanning protocols, learning-based models were trained with PKU37-train, and
reconstruction methods, patients, etc. Thus, we also com- then were tested over DUKE17 and DUKE28, to verify their
pared the generalization capabilities of the deep learning-based capabilities of denoising “unseen” OCT images.

Authorized licensed use limited to: Hong Kong Polytechnic University. Downloaded on June 01,2024 at 05:15:34 UTC from IEEE Xplore. Restrictions apply.
GENG et al.: TRIPLET CROSS-FUSION LEARNING FOR UNPAIRED IMAGE DENOISING IN OCT
TABLE IV
Q UANTITATIVE C OMPARISON OF G ENERALIZATION FOR THE P ROPOSED TCFL-D N CNN
AND F IVE U NPAIRED M ETHODS ON DUKE17 AND DUKE28
Fig. 7. Denoising performance comparison of cross-domain test
with different deep learning-based methods on DUKE17. (a1) Clean.
(a2) Noisy. (b1) nonlocal-GAN (b2) R2R-DnCNN. (b3) NBR-DnCNN.
(b4) DRGAN. (b5) SPcycleGAN. (b6) Our TCFL-DnCNN. (c1) OCTNet.
(c2) N2N-DnCNN.
Fig. 7 and Fig. 8 illustrate the denoising examples of
different methods over DUKE17 and DUKE28, respectively.
The OCT images of DUKE28 are normal, while those of
DUKE17 are pathological. All deep learning-based methods
Authorized licensed use limited to: Hong Kong Polytechnic University. Downloaded on June 01,2024 at 05:15:34 UTC from IEEE Xplore. Restrictions apply.
3365
showed certain generalization capabilities. Similar to the
denoising performance on PKU37-validation, OCTNet and
N2N-DnCNN generate over-smoothed images which lose
some structural information; nonlocal-GAN discards a lot
of meaningful signals; there are a lot of noise residuals in
the denoised images by R2R-DnCNN and NBR-DnCNN;
the images denoised by DRGAN have low contrast; SPcy-
cleGAN loses some structural details. However, the pro-
posed TCFL-DnCNN not only reduces the speckle noise
effectively but also preserves more structural details (espe-
cially some layer information), as indicated by the arrows in
Fig. 7 and Fig. 8.
Table IV presents the quantitative results of TCFL-DnCNN
and five unpaired reference methods on DUKE17 and
DUKE28, where TCFL-DnCNN reaches the best scores in
all metrics. Table V summerizes the quantitative results of
TCFL-DnCNN, OCTNet and N2N-DnCNN on DUKE17 and
DUKE28. On DUKE17, compared with OCTNet and N2N-
DnCNN, the EPI and GCMSE of TCFL-DnCNN are the best,
and the SNR, CNR, and SSIM of TCFL-DnCNN achieves the
second best. On DUKE28, TCFL-DnCNN yields better scores
in EPI, SSIM, and GCMSE than OCTNet and N2N-DnCNN,
which are sensitive to edge and texture. Although OCTNet
and N2N-DnCNN are superior to TCFL-DnCNN in terms of
SNR, CNR, and PSNR, they over-smooth the denoised images.
These results reveal that the proposed TCFL-DnCNN has a
superior generalization capability to all reference methods,
in terms of reducing speckle noise, preventing over-smoothing
images, as well as preserving structural details. This might be
because the information interaction of clean components and
noise components among the training samples gets enhanced
through our TCFL strategy, which helps deep learning net-
works better distinguish between the clean components and
the noise components.
V. D ISCUSSIONS
A. Loss Weight Setting and Ablation Studies
As indicated in (8) and (10), there are two loss weights,
λ and μ, in our proposed loss function. We investigated
3366
Fig. 8. Denoising performance comparison of cross-domain test
with different deep learning-based methods on DUKE28. (a1) Clean.
(a2) Noisy. (b1) nonlocal-GAN (b2) R2R-DnCNN. (b3) NBR-DnCNN.
(b4) DRGAN. (b5) SPcycleGAN. (b6) Our TCFL-DnCNN. (c1) OCTNet.
(c2) N2N-DnCNN.
TABLE V
Q UANTITATIVE C OMPARISON OF G ENERALIZATION FOR THE
P ROPOSED TCFL-D N CNN AND T WO PAIRED M ETHODS
ON DUKE17 AND DUKE28
the influence of different loss weights on the TCFL-DnCNN
denoising performance, and the related quantitative results
over PKU37-validation, DUKE17, and DUKE28 are shown
in Fig. 9. We first set μ to 1, and tried different λ from {2, 3,
4, 5, 6, 7, 8, 9, 10}. As illustrated in Fig. 9 (a), when λ = 6,
TCFL-DnCNN achieves the best denoising performance in
Authorized licensed use limited to: Hong Kong Polytechnic University. Downloaded on June 01,2024 at 05:15:34 UTC from IEEE Xplore. Restrictions apply.
IEEE TRANSACTIONS ON MEDICAL IMAGING, VOL. 41, NO. 11, NOVEMBER 2022
Fig. 9. The influence of different (a) λ and (b) μ on the TCFL-DnCNN
denoising performance on PKU37-validation, DUKE17, and DUKE28.
terms of PSNR, SSIM, and GCMSE. Then, we fixed λ to 6,
and explored different μ from {0.6, 0.7, 0.8, 0.9, 1.0, 1.1,
1.2, 1.3, 1.4}. As shown in Fig. 9 (b), μ = 1 is the optimal
value for TCFL-DnCNN. Thus, we set λ and μ to 6 and 1,
respectively.
As indicated in (10), there are four identity loss terms,
L iden A , L iden B , L iden C1 , and L iden C2 . We also carried out
the ablation studies for the four identity loss terms. PKU37
was used for training and validation. It is noteworthy that
during the ablation studies, λ and μ were fixed to 6 and 1,
respectively. Table VI summarizes the quantitative results
of different combinations of identity loss terms on PKU37-
validation. Fig. 10 shows the related visual results.
From Table VI and Fig. 10, we can observe that when
only L iden A or L iden B was used as L iden (i.e., L iden =
L iden A or L iden = L iden B ), the denoising performance of
TCFL-DnCNN is relatively poor. Taking L iden = L iden A as
an example, only IA → NA → CA1 → IA → CA2 could
form a closed-loop constraint by calculating the identity loss
between CA1 and CA2 , while there is no loss constraint for
IB → NB → CB1 → IB → CB2 . As a result, it could not
be guaranteed that NB bears the wanted speckle noise, and
the subsequently synthesized IA might contain some structural
information of IB .
When both L iden A and L iden B were adopted as L iden
(i.e., L iden = L iden A + L iden B ), the denoising performance
of TCFL-DnCNN is significantly improved compared with
the situations of L iden = L iden A or L iden = L iden B . This
is because both IA → NA → CA1 → IA → CA2 and
IB → NB → CB1 → IB → CB2 get closed-loop constraints.
By adding L iden C1 or L iden C2 to L iden A and L iden B (i.e.,
L iden = L iden A + L iden B + μL iden C1 or L iden = L iden A +
L iden B + μL iden C2 ), the generator was able to look at real
GENG et al.: TRIPLET CROSS-FUSION LEARNING FOR UNPAIRED IMAGE DENOISING IN OCT
Fig. 10. Visual comparison of ablation study for different identity loss
combinations on PKU37-validation.
TABLE VI
Q UANTITATIVE R ESULTS OF A BLATION S TUDY FOR D IFFERENT
I DENTITY L OSS C OMBINATIONS ON PKU37-VALIDATION
clean images and generate more real-like clean images. Thus,
the denoising performance could be further improved.
When all of the four identity loss terms were utilized as
L iden (i.e., L iden = L iden A + L iden B +μ(L iden C1 + L iden C2 )),
the clean components and the noise components from three
unpaired images are cross-fused more diversely. The network
could extract more speckle noise information. As a result,
the noise could be reduced more effectively, so the structural
details could be preserved better, achieving the best denoising
performance. These results have revealed that the identity loss
is essential for the denoising performance of TCFL-DnCNN.
B. Universality Study
In order to verify the universal applicability of the pro-
posed TCFL strategy, besides DnCNN, we also implemented
the strategy with two other widely-used CNNs as the noise
predictor, i.e., U-Net [43] and SRDenseNet [44]. These three
CNN models represent different network architectures, where
DnCNN, U-Net, and SRDenseNet correspond to the single-
path, the U-shaped, and the multi-path architectures, respec-
tively. In addition, we also compared the situations where
CNNs are used to predict noise and the situations where
Authorized licensed use limited to: Hong Kong Polytechnic University. Downloaded on June 01,2024 at 05:15:34 UTC from IEEE Xplore. Restrictions apply.
3367
Fig. 11. Model architecture comparison of CNN as the noise predictor
and as the content predictor.
Fig. 12. Denoising performance comparison of different CNNs as the
noise (or content) predictor through the proposed TCFL strategy on
PKU37-validation.
CNNs are adopted to estimate clean images (content) directly,
as shown in Fig. 11. That is, the ablation studies for the
residual learning were carried out.
Fig. 12 shows the denoising examples of different CNNs as
the noise (or content) predictor through the proposed TCFL
strategy on PKU37-validation. Generally speaking, the three
CNNs show effective denoising capabilities. DnCNN and
U-Net outperform SRDenseNet in speckle noise reduction.
Furthermore, all CNNs improve their denoising performance
when they were used as the noise predictors, especially for
SRDenseNet. That is, the effectiveness of the global resid-
ual learning has been verified. Table VII summarizes the
quantitative results of different CNNs on PKU37-validation,
which support our visual observations. These studies have
demonstrated the universality of the proposed TCFL strategy.
That is to say, the strategy can be implemented by CNNs with
different structures.
In addition to exploring the impact of different network
architectures on the denoising performance of the TCFL
strategy, we also compared the denoising performance of two
TCFL-DnCNNs with different activation functions (i.e., Leaky
ReLU and ReLU). In terms of quantitative metrics, when
the activation function of TCFL-DnCNN was replaced from
3368 IEEE TRANSACTIONS ON MEDICAL IMAGING, VOL. 41, NO. 11, NOVEMBER 2022

TABLE VII
Q UANTITATIVE R ESULTS OF D IFFERENT CNN S U NDER THE P ROPOSED TCFL S TRATEGY ON PKU37-VALIDATION

TABLE VIII
Q UANTITATIVE R ESULTS OF TCFL-D N CNN U NDER D IFFERENT S PECKLE N OISE L EVELS ON DUKE17

Leaky ReLU to ReLU, PSNR is decreased from 29.418 to

29.035, SSIM is decreased from 0.635 to 0.618, and GCMSE
is improved from 7.723 to 8.984, on PKU37-validation.
The visual denoising comparison is presented in Fig. 12.
We can observe that although Leaky ReLU outperforms ReLU
quantitatively, ReLU preserves more structural information
visually. In the field of medical imaging, the macroscopic
image appearance is more informative than the commonly
used quantitative denoising evaluation metrics. These results
reveal that Leaky ReLU might tend to obtain better quantitative
results, while ReLU is preferred to Leaky ReLU with respect
to structure preservation.

C. Robustness Analysis
1) Noise Robustness: It is necessary to explore the noise
robustness for a trained deep learning network. Thus,
we applied the TCFL-DnCNN trained by PKU37-train to a
set of DUKE17 test scenarios with different noise levels. The
simulation of pseudo-speckle noise-corrupted OCT images
was referred to [17], [45], where a parameter σ 2 was employed
to control the noisy level, and the larger σ 2 value corresponds
to the noisier image. We tried different σ 2 from {0.02, 0.05,
0.1, 0.2}. Based on the clean images in DUKE17, four test
sets with different noise levels are simulated.
Fig. 13 illustrates the simulated noisy OCT images and
the corresponding denoised results by TCFL-DnCNN. It can
be seen that the proposed TCFL-DnCNN can deal with dif-
ferent noise levels and restore the noisy images effectively.
As the value of σ 2 increased, the quality of the simulated
noisy images gets worse, and the denoising performance of Fig. 13. The denoising performance of TCFL-DnCNN under different
TCFL-DnCNN exhibits slight decay. However, TCFL-DnCNN speckle noise levels on DUKE17.
still denoises the simulated noisy images effectively even
when σ 2 was 0.2. Table VIII presents the related quantitative
results, which support the visual observations. These results including two noisy images and one clean image. In contrast,
have demonstrated the noise robustness of the proposed TCFL existing denoising methods used only one (e.g., nonlocal-
strategy. GAN [21], R2R-DnCNN [30], and NBR-DnCNN [31]) or two
2) Training With Limited Data: As we have elaborated in images (e.g., OCTNet [10], N2N-DnCNN [19], DRGAN [17],
Section III, the TCFL-based network was trained using triplets, and SPcycleGAN [25]) for training. For example, the

Authorized licensed use limited to: Hong Kong Polytechnic University. Downloaded on June 01,2024 at 05:15:34 UTC from IEEE Xplore. Restrictions apply.
GENG et al.: TRIPLET CROSS-FUSION LEARNING FOR UNPAIRED IMAGE DENOISING IN OCT
Fig. 14. Quantitative results of different methods trained in lim-
ited training data scenarios. (a)-(c) correspond to PKU37-validation,
(d)-(f) correspond to DUKE17, and (g)-(i) correspond to DUKE28.
denoising training set D contains n noisy images and m clean
images. When the unpaired denoising networks are trained
based on two unpaired noisy-clean images (i.e., the doublet),
the number of all the possible doublet combinations from D
will be C1n ×C1m in Combinatorics. As for the proposed TCFL-
based networks, the number of all the possible triplet combina-
tions from D will be C2n × C1m , which is obviously larger than
the number of the doublet combinations. Thus, our strategy can
deal with some limited training data scenarios. We compared
TCFL-DnCNN with other seven deep learning-based methods
(i.e., OCTNet, N2N-DnCNN, nonlocal-GAN, R2R-DnCNN,
NBR-DnCNN, DRGAN, and SPcycleGAN) over six limited
training data scenarios, where the numbers of images in the
training set are reduced to 75%, 50%, 25%, 10%, 5%, and
2%, respectively. For a fair comparison, the reference methods
adopted some routine augmentation methods (including flip-
ping, cropping, shifting, zooming, and some combinations) to
keep the numbers of their training pairs consistent with that
of TCFL-DnCNN.
Fig. 14 presents the quantitative results of different meth-
ods over PKU37-validation, DUKE17, and DUKE28. It can
be seen that, the supervised OCTNet or weakly-supervised
Authorized licensed use limited to: Hong Kong Polytechnic University. Downloaded on June 01,2024 at 05:15:34 UTC from IEEE Xplore. Restrictions apply.
3369
Fig. 15. Denoising performance comparison of different methods in
limited training data scenarios on DUKE28.
N2N-DnCNN outperform TCFL-DnCNN in some metrics at
the beginning (i.e., 100% training data). However, as the
proportion of training data decreased, TCFL-DnCNN shows
slight declines in all metrics, while the declines of the ref-
erence methods are much heavier. When the proportion is
less than 50%, TCFL-DnCNN is superior to all reference
methods in all quantitative metrics. Fig. 15 visually compares
TCFL-DnCNN with seven reference methods on DUKE28.
In Fig. 15, all reference methods suffer from over-smoothness
and structural loss problems as the proportion decreased.
However, the proposed TCFL-DnCNN still denoises the OCT
images effectively, even when the proportion was 2%. These
results have proved the tolerance of the TCFL strategy to
scenarios with limited training data.
D. Application in Pronounced Retinal Pathologies
In this work, three OCT denoising datasets were used.
The clean OCT images are always obtained by multi-frame
averaging, which is very time-consuming and not easy to
be obtained. The difficulty of acquiring clean OCT images
limits the diversity of the OCT denoising datasets to some
extent, especially for the patients with severe pathologies.
3370
Fig. 16. Visual comparison before and after TCFL-DnCNN denoising
for two pronounced retinal pathologies (retinal fluid and vitreomacular
traction syndrome) and a non-foveal region.
Thus, the three datasets used in this work do not comprise
pronounced pathologies as can be frequently found in clinical
routine, as well as they are restricted to the region of the
fovea. In order to further verify the generalization capabilities
of the proposed TCFL strategy, we applied TCFL-DnCNN
trained on PKU37-train to denoising two noisy OCT images
with pronounced retinal pathologies (i.e., retinal fluid and
vitreomacular traction syndrome) and a noisy non-foveal OCT
image, which are provided by our clinical collaborators.
Fig. 16 shows the related visual comparison before and after
TCFL-DnCNN denoising. It can be seen that TCFL-DnCNN
improves the visual quality obviously. It is noteworthy that
although PKU37-train does not contain pathological images,
the trained TCFL-DnCNN can still effectively denoise various
OCT images with different pathologies, shapes, and structures.
This is because the characteristics of speckle noise in OCT
images are largely determined by the imaging system rather
than the samples [5].
E. Application in Retinal Layer Segmentation
Image denoising can be a pre-processing step for some
high-level vision tasks (such as segmentation, detection, clas-
sification, etc.) to mitigate the deterioration due to noise.
In order to further demonstrate the effectiveness of our TCFL
strategy, we compared the influence of different OCT image
denoising methods on the downstream retinal layer segmenta-
tion task. That is, we first trained different deep learning-based
denoising networks using PKU37-train. Then we applied the
Authorized licensed use limited to: Hong Kong Polytechnic University. Downloaded on June 01,2024 at 05:15:34 UTC from IEEE Xplore. Restrictions apply.
IEEE TRANSACTIONS ON MEDICAL IMAGING, VOL. 41, NO. 11, NOVEMBER 2022
Fig. 17. Comparison of the segmentation results after the denoising
by various methods. (a1) ground truth. (a2) no denoising. (b1) BM3D.
(b2) NLM. (c1) nonlocal-GAN. (c2) R2R-DnCNN. (c3) NBR-DnCNN.
(c4) DRGAN. (c5) SPcycleGAN. (c6) Our TCFL-DnCNN. (d1) OCTNet.
(d2) N2N-DnCNN.
trained deep denoisers and two traditional denoising algo-
rithms to denoise the OCT images of a public retinal layer
segmentation dataset [46]. Finally, we adopted the denoised
OCT images as the inputs of a trained segmentation network
(U-Net) to acquire the retinal layer segmentation results. In the
retinal layer segmentation dataset [46], there are 220 image
pairs from 20 subjects, and each pair consists of an OCT
retinal image as the input and a retinal layer segmentation
mask as the label. The layer segmentation masks include the
background area, the total retina area, and the retinal pigment
epithelium and drusen complex (RPEDC) area. Fig. 17 shows
the visual segmentation results with the help of the denois-
ing by various methods. We can observe that the proposed
TCFL-DnCNN achieves the most obvious improvement for
the downstream segmentation task. Two quantitative evaluation
metrics were calculated, including pixel accuracy (PA) and
mean intersection over union (MIoU). Table IX presents the
quantitative results, which also demonstrate the superiority of
our TCFL-DnCNN in serving for the downstream segmenta-
tion task.
F. Model Complexity
As mentioned above, most unpaired OCT image denoising
works are based on the cycleGAN frameworks which have
GENG et al.: TRIPLET CROSS-FUSION LEARNING FOR UNPAIRED IMAGE DENOISING IN OCT
TABLE IX
Q UANTITATIVE R ESULTS OF THE R ETINAL L AYER S EGMENTATION
A FTER VARIOUS D ENOISING M ETHODS
TABLE X
M ODEL C OMPLEXITY C OMPARISON OF TCFL-D N CNN AND
T WO CYCLE GAN-B ASED D EEP N ETWORKS
at least two generators and two discriminators, whereas the
proposed TCFL strategy contains only one generator and
one discriminator. To better demonstrate that the model com-
plexity of the TCFL strategy is much lower than those of
the cycleGAN-based methods, we compare TCFL-DnCNN
with two cycleGAN-based methods (i.e., DRGAN [17] and
SPcycleGAN [25]) in terms of model parameters. As listed
in Table X, the parameters of TCFL-DnCNN is significantly
smaller than those of DRGAN and SPcycleGAN.
G. Outlook
The proposed image denoising strategy is based on the
assumption that the OCT speckle noise in logarithm space is
additive and data-independent. In practice, considering factors
such as OCT signal acquisition and image reconstruction, the
speckle noise in logarithm space might not be completely
signal-independent. As a result, the synthesized OCT images
by the simple adding operation might contain a few structural
residues from the other OCT images. However, these structural
residues can be ignored by the deep learning networks, due
to their powerful modeling capabilities. The above denoising
results confirm this conclusion.
As for some other medical imaging modalities such as low-
dose CT, the noise is always data-independent. Our strat-
egy cannot deal with these situations with data-independent
noise. A novel synthetic method for the fake noisy images
should be explored instead of the simple adding opera-
tion in (4) and (6). We will explore a more generalized
unpaired medical image denoising strategy which can deal
with both signal-independent and signal-dependent noise in
future research.
Furthermore, the proposed TCFL can be extended to
quadruple or quintuple cross-fusion learning. This brings
two benefits. First, it will further enhance the information
Authorized licensed use limited to: Hong Kong Polytechnic University. Downloaded on June 01,2024 at 05:15:34 UTC from IEEE Xplore. Restrictions apply.
3371
interaction among training samples by quadruple or quintu-
ple cross-fusion. Second, quadruple or quintuple cross-fusion
learning can deal with the scenarios of fewer training samples.
We will conduct further studies toward this direction.
VI. C ONCLUSION
In this work, we propose a novel triplet cross-fusion learn-
ing (TCFL) strategy to denoise OCT images with unpaired
data. Compared with cycleGAN-based methods, the model
complexity of our strategy is reduced significantly. Three
widely-used CNNs were adopted to implement the TCFL strat-
egy, and three datasets were used for evaluation. The results
demonstrate that the proposed strategy outperformed exist-
ing unpaired methods both qualitatively and quantitatively.
Compared with existing supervised and weakly-supervised
methods, the proposed strategy has competitive quantitative
results, and demonstrates better structure preservation capa-
bility visually. We have also verified the universality and
generalization capability of the TCFL strategy. Furthermore,
the TCFL strategy can deal with some limited training data
scenarios.
R EFERENCES
[1] A. M. Zysk, F. T. Nguyen, A. L. Oldenburg, D. L. Marks, and
S. A. Boppart, “Optical coherence tomography: A review of clinical
development from bench to bedside,” J. Biomed. Opt., vol. 12, no. 5,
2007, Art. no. 051403.
[2] T. Klein, R. André, W. Wieser, T. Pfeiffer, and R. Huber, “Joint aperture
detection for speckle reduction and increased collection efficiency in
ophthalmic MHz OCT,” Biomed. Opt. Exp., vol. 4, no. 4, pp. 619–634,
2013.
[3] B. Chong and Y.-K. Zhu, “Speckle reduction in optical coherence
tomography images of human finger skin by wavelet modified BM3D
filter,” Opt. Commun., vol. 291, pp. 461–469, Mar. 2013.
[4] J. Aum, J.-H. Kim, and J. Jeong, “Effective speckle noise suppression in
optical coherence tomography images using nonlocal means denoising
filter with double Gaussian anisotropic kernels,” Appl. Opt., vol. 54,
no. 13, pp. D43–D50, 2015.
[5] F. Zaki, Y. Wang, H. Su, X. Yuan, and X. Liu, “Noise adaptive
wavelet thresholding for speckle noise removal in optical coherence
tomography,” Biomed. Opt. Exp., vol. 8, no. 5, pp. 2720–2731, 2017.
[6] M. Geng et al., “Content-noise complementary learning for medical
image denoising,” IEEE Trans. Med. Imag., vol. 41, no. 2, pp. 407–419,
Feb. 2022.
[7] M. Geng et al., “PMS-GAN: Parallel multi-stream generative adversarial
network for multi-material decomposition in spectral computed tomog-
raphy,” IEEE Trans. Med. Imag., vol. 40, no. 2, pp. 571–584, Feb. 2021.
[8] B. Qiu et al., “N2NSR-OCT: Simultaneous denoising and super-
resolution in optical coherence tomography images using semisu-
pervised deep learning,” J. Biophoton., vol. 14, no. 1, Jan. 2021,
Art. no. e202000282.
[9] L. Zhu et al., “Synergistically segmenting choroidal layer and vessel
using deep learning for choroid structure analysis,” Phys. Med. Biol.,
vol. 67, no. 8, Apr. 2022, Art. no. 085001.
[10] M. Xu, C. Tang, F. Hao, M. Chen, and Z. Lei, “Texture preservation
and speckle reduction in poor optical coherence tomography using the
convolutional neural network,” Med. Image Anal., vol. 64, Aug. 2020,
Art. no. 101727.
[11] Z. Shen, M. Xi, C. Tang, M. Xu, and Z. Lei, “Double-path parallel
convolutional neural network for removing speckle noise in different
types of OCT images,” Appl. Opt., vol. 60, no. 15, pp. 4345–4355,
May 2021.
[12] Y. Ma, X. Chen, W. Zhu, X. Cheng, D. Xiang, and F. Shi, “Speckle
noise reduction in optical coherence tomography images based on edge-
sensitive cGAN,” Biomed. Opt. Exp., vol. 9, no. 11, pp. 5129–5146,
Nov. 2018.
[13] B. Qiu et al., “Noise reduction in optical coherence tomography images
using a deep neural network with perceptually-sensitive loss function,”
Biomed. Opt. Exp., vol. 11, no. 2, pp. 817–830, 2020.
3372
[14] A. K. Nilesh, D. Rupali, D. Ambedkar, and K. Y. Phaneendra,
“SiameseGAN: A generative model for denoising of spectral domain
optical coherence tomography images,” IEEE Trans. Med. Imag., vol. 40,
no. 1, pp. 180–192, Jan. 2021.
[15] M. Wang et al., “Semi-supervised capsule cGAN for speckle noise
reduction in retinal OCT images,” IEEE Trans. Med. Imag., vol. 40,
no. 4, pp. 1168–1183, Apr. 2021.
[16] Y. Guo et al., “Structure-aware noise reduction generative adversarial
network for optical coherence tomography image,” in Proc. Int. Work-
shop Ophthalmic Med. Image Anal. Cham, Switzerland: Springer, 2019,
pp. 9–17.
[17] Y. Huang et al., “Noise-powered disentangled representation for unsu-
pervised speckle reduction of optical coherence tomography images,”
IEEE Trans. Med. Imag., vol. 40, no. 10, pp. 2600–2614, Oct. 2021.
[18] J. Lehtinen et al., “Noise2Noise: Learning image restoration without
clean data,” in Proc. ICML, 2018, pp. 1–12.
[19] B. Qiu et al., “Comparative study of deep neural networks with
unsupervised Noise2Noise strategy for noise reduction of optical coher-
ence tomography images,” J. Biophoton., vol. 14, no. 11, Nov. 2021,
Art. no. e202100151.
[20] Z. Jiang et al., “Weakly supervised deep learning-based optical coher-
ence tomography angiography,” IEEE Trans. Med. Imag., vol. 40, no. 2,
pp. 688–698, Feb. 2021.
[21] A. Guo, L. Fang, M. Qi, and S. Li, “Unsupervised denoising of opti-
cal coherence tomography images with nonlocal-generative adversarial
network,” IEEE Trans. Instrum. Meas., vol. 70, pp. 1–12, 2021.
[22] J.-Y. Zhu, T. Park, P. Isola, and A. A. Efros, “Unpaired image-to-image
translation using cycle-consistent adversarial networks,” in Proc. IEEE
Int. Conf. Comput. Vis. (ICCV), Oct. 2017, pp. 2223–2232.
[23] I. Manakov, M. Rohm, C. Kern, B. Schworm, K. Kortuem, and V. Tresp,
“Noise as domain shift: Denoising medical images by unpaired image
translation,” in Domain Adaptation and Representation Transfer and
Medical Image Learning With Less Labels and Imperfect Data. Cham,
Switzerland: Springer, 2019, pp. 3–10.
[24] V. Das, S. Dandapat, and P. K. Bora, “Unsupervised super-resolution
of OCT images using generative adversarial network for improved
age-related macular degeneration diagnosis,” IEEE Sensors J., vol. 20,
no. 15, pp. 8746–8756, Aug. 2020.
[25] M. Wu, W. Chen, Q. Chen, and H. Park, “Noise reduction for SD-OCT
using a structure-preserving domain transfer approach,” IEEE J. Biomed.
Health Informat., vol. 25, no. 9, pp. 3460–3472, Sep. 2021.
[26] K. Zhang, W. Zuo, Y. Chen, D. Meng, and L. Zhang, “Beyond a
Gaussian denoiser: Residual learning of deep CNN for image denoising,”
IEEE Trans. Image Process., vol. 26, no. 7, pp. 3142–3155, Jul. 2017.
[27] A. M. Hasan, M. R. Mohebbian, K. A. Wahid, and P. Babyn, “Hybrid-
collaborative Noise2Noise denoiser for low-dose CT images,” IEEE
Trans. Radiat. Plasma Med. Sci., vol. 5, no. 2, pp. 235–244, Mar. 2021.
[28] C. Tian, L. Fei, W. Zheng, Y. Xu, W. Zuo, and C.-W. Lin, “Deep learning
on image denoising: An overview,” Neural Netw., vol. 131, pp. 251–275,
Nov. 2020.
[29] J. Batson and L. Royer, “Noise2Self: Blind denoising by self-
supervision,” in Proc. Int. Conf. Mach. Learn., 2019, pp. 524–533.
[30] T. Pang, H. Zheng, Y. Quan, and H. Ji, “Recorrupted-to-
recorrupted: Unsupervised deep learning for image denoising,” in Proc.
IEEE/CVF Conf. Comput. Vis. Pattern Recognit. (CVPR), Jun. 2021,
pp. 2043–2052.
Authorized licensed use limited to: Hong Kong Polytechnic University. Downloaded on June 01,2024 at 05:15:34 UTC from IEEE Xplore. Restrictions apply.
IEEE TRANSACTIONS ON MEDICAL IMAGING, VOL. 41, NO. 11, NOVEMBER 2022
[31] T. Huang, S. Li, X. Jia, H. Lu, and J. Liu, “Neighbor2Neighbor:
Self-supervised denoising from single noisy images,” in Proc.
IEEE/CVF Conf. Comput. Vis. Pattern Recognit. (CVPR), Jun. 2021,
pp. 14781–14790.
[32] Y. Li, H. Wang, and X. Dong, “The denoising of desert seismic data
based on cycle-GAN with unpaired data training,” IEEE Geosci. Remote
Sens. Lett., vol. 18, no. 11, pp. 2016–2020, Nov. 2020.
[33] J. Song, J.-H. Jeong, D.-S. Park, H.-H. Kim, D.-C. Seo, and J. C. Ye,
“Unsupervised denoising for satellite imagery using wavelet direc-
tional CycleGAN,” IEEE Trans. Geosci. Remote Sens., vol. 59, no. 8,
pp. 6823–6839, Aug. 2021.
[34] J. Gu, T. S. Yang, J. C. Ye, and D. H. Yang, “CycleGAN
denoising of extreme low-dose cardiac CT using wavelet-assisted
noise disentanglement,” Med. Image Anal., vol. 74, Dec. 2021,
Art. no. 102209.
[35] S. Aumann, S. Donner, J. Fischer, and F. Müller, “Optical coherence
tomography (OCT): Principle and technical realization,” in High Reso-
lution Imaging in Microscopy and Ophthalmology. Cham, Switzerland:
Springer, 2019, pp. 59–85.
[36] H. M. Salinas and D. C. Fernandez, “Comparison of PDE-based non-
linear diffusion approaches for image enhancement and denoising in
optical coherence tomography,” IEEE Trans. Med. Imag., vol. 26, no. 6,
pp. 761–771, Jun. 2007.
[37] B. Xu, N. Wang, T. Chen, and M. Li, “Empirical evaluation of rectified
activations in convolutional network,” 2015, arXiv:1505.00853.
[38] L. Fang, S. Li, Q. Nie, J. A. Izatt, C. A. Toth, and S. Far-
siu, “Sparsity based denoising of spectral domain optical coherence
tomography images,” Biomed. Opt. Exp., vol. 3, no. 5, pp. 927–942,
May 2012.
[39] L. Fang et al., “Fast acquisition and reconstruction of optical coherence
tomography images via sparse representation,” IEEE Trans. Med. Imag.,
vol. 32, no. 11, pp. 2034–2049, Nov. 2013.
[40] A. Paszke et al., “PyTorch: An imperative style, high-performance deep
learning library,” in Proc. Adv. Neural Inf. Process. Syst., vol. 32,
Dec. 2019, pp. 8026–8037.
[41] Z. Zhang, “Improved Adam optimizer for deep neural networks,” in
Proc. IEEE/ACM 26th Int. Symp. Quality Service (IWQoS), Jun. 2018,
pp. 1–2.
[42] J. Lopez-Randulfe, C. Veiga, J. J. Rodriguez-Andina, and J. Farina,
“A quantitative method for selecting denoising filters, based on a new
edge-sensitive metric,” in Proc. IEEE Int. Conf. Ind. Technol. (ICIT),
Mar. 2017, pp. 974–979.
[43] O. Ronneberger, P. Fischer, and T. Brox, “U-Net: Convolutional net-
works for biomedical image segmentation,” in Proc. Int. Conf. Med.
Image Comput. Comput.-Assist. Intervent. Cham, Switzerland: Springer,
2015, pp. 234–241.
[44] T. Tong, G. Li, X. Liu, and Q. Gao, “Image super-resolution using
dense skip connections,” in Proc. IEEE Int. Conf. Comput. Vis. (ICCV),
Oct. 2017, pp. 4799–4807.
[45] M. Li, R. Idoughi, B. Choudhury, and W. Heidrich, “Statistical model for
OCT image denoising,” Biomed. Opt. Exp., vol. 8, no. 9, pp. 3903–3917,
2017.
[46] S. J. Chiu, J. A. Izatt, R. V. O’Connell, K. P. Winter, C. A. Toth,
and S. Farsiu, “Validated automatic segmentation of AMD pathology
including drusen and geographic atrophy in SD-OCT images,” Invest.
Ophthalmol. Vis. Sci., vol. 53, no. 1, pp. 53–61, 2012.