4668 IEEE JOURNAL OF BIOMEDICAL AND HEALTH INFORMATICS, VOL. 26, NO.
9, SEPTEMBER 2022
A Multi-Task Feature Fusion Model for Cervical
Cell Classification
Jian Qin , Yongjun He , Jinping Ge , and Yiqin Liang
Abstract—Cervical cell classification is a crucial tech-
nique for automatic screening of cervical cancer. Although
deep learning has greatly improved the accuracy of cell
classification, the performance still cannot meet the needs
of practical applications. To solve this problem, we propose
a multi-task feature fusion model that consists of one aux-
iliary task of manual feature fitting and two main classifica-
tion tasks. The auxiliary task enhances the main tasks in a
manner of low-layer feature fusion. The main tasks, i.e., a
2-class classification task and a 5-class classification task,
are learned together to realize their mutual reinforcement
and alleviate the influence of unreliable labels. In addition,
a label smoothing method based on cell category similarity
is designed to bring inter-cell class information into the
label. Comparative experimental results with other state-
of-the-art models on the HUSTC and SIPaKMeD datasets
prove the effectiveness of the proposed method. With a
high sensitivity of 99.82% and a specificity of 98.12% for
the 2-class classification task on the HUSTC dataset, our
method shows potential to reduce cytologist workload.
Index Terms—Cervical cell classification, feature fusion,
label smoothing, manual features, multi-task.
I. INTRODUCTION
ERVICAL cancer is now a significant health threat to
C women worldwide. It is accountable for 311,000 deaths
in 2018, with 85% of these deaths are occurring in low- and
middle-income countries [1]. Nevertheless, cervical cancer can
be effectively managed through preventive clinical management
strategies. Cervical cytopathology (pap smear or liquid-based
cytology) is the most common method for identifying precan-
cerous lesions of cervical cancer. It is tedious and laborious for
cytologists to manually screen abnormal cells from a cervical
cytology sample containing 20,000–50,000 cells. Moreover, the
missed rate of manual cervical screenings is still up to 20–30%
[2]. Hence, it is necessary to develop automatic screening tech-
niques to help cytologists diagnose smears.
Manuscript received 15 October 2021; revised 4 May 2022 and 31
May 2022; accepted 2 June 2022. Date of publication 7 June 2022;
date of current version 9 September 2022. This work was supported in
part by the National Natural Science Foundation of China under Grant
61673142 and in part by the Natural Science Foundation of Heilongjiang
Province of China under Grant JJ2019JQ0013. (Corresponding author:
Yongjun He.)
The authors are with the School of Computer Science and Tech-
nology, Harbin University of Science and Technology, Harbin 150080,
China (e-mail: jian.qin1@qq.com; holywit@163.com; 13079655433@
163.com; 984013452@qq.com).
Digital Object Identifier 10.1109/JBHI.2022.3180989
2168-2194 © 2022 IEEE. Personal use is permitted, but republication/redistribution requires IEEE permission.
See https://www.ieee.org/publications/rights/index.html for more information.
Authorized licensed use limited to: Charles Darwin University. Downloaded on January 20,2024 at 14:55:31 UTC from IEEE Xplore. Restrictions apply.
Cervical cells can be categorized according to The Bethesda
System [3], and some examples are shown in Fig. 1. The cate-
gories of cells are negative for intraepithelial lesion for malig-
nancy (NILM), atypical squamous cells of undetermined sig-
nificance (ASC-US), low-grade squamous intraepithelial lesion
(LSIL), atypical squamous cells cannot exclude HSIL (ASC-H),
and high-grade squamous intraepithelial lesion (HSIL). NIML
indicates normal cells, and the others indicate abnormal cells.
ASC-US and LSIL are primarily found in epidermic squamous
cells generally at the beginning of HPV infection. ASC-H and
HSIL from the basal squamous cells are usually warning signs
that cancerous processes may occur.
Recently, deep learning has brought significant progress in
cervical cell classification, but the performance still cannot meet
the needs of practical applications. There are two main reasons.
First, current deep learning networks need large amounts of
labeled data. This is hard to be satisfied because data label
needs professional cytologists. Second, cervical cancer lesions
are a continuous process, so there is no clear threshold to deter-
mine the boundary of cell category. This makes it challenging
to obtain accurate category annotations, resulting in degraded
performance. Currently, two hotspots are how to train a better
model without increasing labeled data and how to reduce the
impact of mislabeling.
Multi-task learning can effectively improve the performance
of deep learning. This is because related tasks can benefit from
each other by jointly learning certain shared, or mutually related
representations. Multiple signals originating from different tasks
can be considered as implicit data augmentation or additional
regularization [4]. This enables models to learn mutually related
representations for multiple tasks, thus avoiding overfitting and
leading to better generalizability.
Manual features defined in The Bethesda System rules [3] can
effectively assist in identifying normal and abnormal cells. Many
works have been done to classify manual features of cervical
cells by machine learning [5], [6]. These manual features about
cells are highly relevant to the cell classification task, so we
consider using a multi-task model for simultaneous manual fea-
ture and category prediction. This allows the cell classification
task to learn relevant hidden features through predicting man-
ual features, thus improving performance. In addition, current
label smoothing methods treat non-target categories equally by
assigning them with fixed identical probability [7]. If the prior
knowledge of category similarity is known, we can assign non-
target categories with different probabilities to further improve
classification performance.
QIN et al.: MULTI-TASK FEATURE FUSION MODEL FOR CERVICAL CELL CLASSIFICATION
Fig. 1. Illustration of the different cells. Left: normal cells defined as NILM. Right: abnormal cells including ASC-US, LSIL, HSIL and ASC-H.
ASC-US and LSIL are usually found in epidermal squamous cells, while ASC-H and HSIL are generally found in basal squamous cells.
In this paper, we propose a multi-task feature fusion model
that fully uses manual features. Specifically, the proposed model
contains a manual features fitting branch and a classification
branch. Rather than directly fusing manual features with deep
learning features, we fuse the middle layer features of the manual
features fitting branch with those of the classification branch.
The classification branch simultaneously performs 2-class and
5-class classification tasks. In the proposed model, features in
the manual feature fitting branch containing cervical cell domain
knowledge are incorporated into the classification branch by
connection module. To reduce the influence of incorrect label-
ing, we propose a new label smoothing method, which assigns
non-target categories with a different probability according to
the prior knowledge of similarity between categories. Further-
more, supervised contrastive learning is used to pre-train the
backbone network parameters. Our contributions are summa-
rized as follows:
1) We propose a multi-task model for cervical cell classifi-
cation, which uses manual features fitting as an auxiliary
task to help the model extract discriminative features.
2) The classification branch of the proposed model performs
2-class and 5-class classification tasks simultaneously.
This allows the two tasks to promote each other and
reduce the influence of incorrect annotations among intra-
group classes.
3) We propose a new label smoothing method that sets
non-target categories to different probabilities, which are
determined by the similarity of cell categories. This can
improve model performance by bringing inter-cell class
information into the label.
Authorized licensed use limited to: Charles Darwin University. Downloaded on January 20,2024 at 14:55:31 UTC from IEEE Xplore. Restrictions apply.
4669
4) The proposed method is evaluated on the HUSTC and
SIPaKMeD datasets [8]. The effectiveness of the pro-
posed multi-task feature fusion model is confirmed by
comparing it with other state-of-the-art models.
II. RELATED WORK
A. Cervical Cell Classification
Research on automatic cervical cancer screening techniques
began in the 1950-ies and entered its active research phase
in the 1970-ies [2]. Afterward, a series of cervical screening
systems were launched and some success was achieved [9].
In recent decades, automation technology and deep learning
have achieved remarkable progress in medicine [10], which has
powerfully promoted the continuous development of automatic
cervical cancer screening.
In the early days, most literature involved manual features and
traditional machine learning classifiers such as support vector
machine [11] and AdaBoost [12], etc. Wang et al. [13] used
feature selection algorithms to filter the features of shape, texture
and Gabor features, then classified the features through the sup-
port vector machine. Arya et al. [14] used multiple texture fea-
tures, including first-order histogram, grey-level co-occurrence
matrix, local binary pattern, Laws energy texture feature and
discrete wavelet transform for the cervical cell classification.
Win et al. [15] used a random forest algorithm as a feature
selection method. Then they used a bagging integrated classifier
to combine the results of five classifiers to obtain the final results.
These methods essentially use one or more classifiers to classify
4670 IEEE JOURNAL OF BIOMEDICAL AND HEALTH INFORMATICS, VOL. 26, NO. 9, SEPTEMBER 2022

Fig. 2. Overview of the proposed method for cervical cell classification. Firstly, target cells are marked on whole slide images and cropped out.
After that, the cells are segmented and used to compute manual features. Finally, the manual features and the smoothed labels are used to train
our proposed multi-task model.

features. As a result, they are inevitably affected by feature information contained in the manual features can be brought into
extraction and feature selection choices. the model.
With the development of computer hardware, deep learning
is widely used in image recognition. Deep learning methods B. Label Smoothing
are designed to discover intricate discriminative information
Training deep learning models with hard labels (with 1 in-
from raw data automatically. They have been used successfully
dicating the target category and 0 non-target categories) often
in biomedical image processing, such as classifying lung dis-
results in over-confident models. For supervised learning, the
eases [16] and lymph nodes [17], etc. Deep learning models such
quality of labels determines the performance of models. Boost-
as ResNet [18] and EfficientNet [19] have been successfully used
ing and changing labels is a simple and effective way to alleviate
in cervical cell classification tasks. DeepCervix [20] achieves
overfitting problems. Reed et al. [27] used predicted distribution
leading performance in cervical cell classification tasks by fus-
or predicted category to smooth hard labels to improve the
ing hybrid deep features from different models. DeepPap [21]
performance on noisy data. Dubey et al. [28] perturbed a small
can directly classify a single cervical cell into normal and
portion of the labels during model training to prevent overfitting
abnormal from image patches centered on the nucleus centroid.
of the model. Wang et al. [29] proposed an innovative framework
Lin et al. [22] proposed a CNN-based method that combines
that learned feature representation by predicting non-spatial and
cell image appearance with cell morphology for cervical cell
spatial transformation parameters. Zhang et al. [30] proposed an
classification. Basak et al. [23] used the gray wolf optimizer to
Online Label Smoothing (OLS) strategy, which generates soft
select the reduced-dimensional CNN features and thus improve
labels based on the statistics of the model prediction for the target
the classification performance. More recent methods have shown
category. Unlike the mentioned approaches above, the soft labels
the effectiveness of Transformer-based architectures for image
generated by our proposed method utilize a priori knowledge of
classification [24]–[26]. The transformer directly models long-
cervical cells.
range dependencies of local patches through the self-attention
mechanism, providing greater flexibility in modeling visual
contents. III. METHODS
However, most current methods only use category labels to The block diagram of the proposed work is shown in Fig. 2.
train the model and do not use the manual features of cervi- We first label cells on the whole slide images to obtain training
cal cells to guide the training. In this paper, we use manual data. Secondly, the labeled cells are cropped out from the whole
features as an auxiliary learning task for the model so that the slide images (WSIs), and the cells are segmented for subsequent

Authorized licensed use limited to: Charles Darwin University. Downloaded on January 20,2024 at 14:55:31 UTC from IEEE Xplore. Restrictions apply.
QIN et al.: MULTI-TASK FEATURE FUSION MODEL FOR CERVICAL CELL CLASSIFICATION
Fig. 3. The architecture of the proposed multi-task model. The model comprises a classification branch and a manual features fitting branch.
Rest1-Rest4 represent the layers of the Resnet model, respectively. ‘Conv’ denotes a 1 × 1 convolution operation with a step size of 1. ‘BN’ and
‘FC’ denote batch normalization and fully connected layers, respectively.
processing. Thirdly, according to the rules of The Bethesda
System, we calculated the manual features of the cells. Finally,
the multi-task model is trained using smoothed classification
labels and manual features.
A. Multi-Task Model Architecture
Multi-task learning can improve deep learning models by
simultaneously learning multiple related tasks. Manual features
of cervical cells which contain a large amount of a priori knowl-
edge are highly relevant to the classification task. Therefore, we
propose a multi-task mode that uses manual feature fitting as
an auxiliary learning task, allowing the model to learn the mu-
tually related representations between tasks to improve model
performance.
The proposed model borrows the idea from the
NDDR-CNN [31], using Neural Discriminative Dimensionality
Reduction (NDDR) as a layerwise feature fusion scheme. NDDR
is modulated by combining existing CNN components with clear
mathematical interpretability as discriminative dimensionality
reduction. In the proposed model, we use NDDR to implement
feature fusion between the classification branch and the
manual feature fitting branch. Compared with NDDR-CNN,
the difference is that we only input the fused features into the
classification task instead of all tasks. This is because in our
proposed model, we expect the performance of cell classification
to be improved rather than the manual feature prediction. The
proposed model comprises a classification branch and a manual
features fitting branch (Fig. 3), which are used for manual feature
prediction and cell classification tasks, respectively. Cervical
cell domain knowledge in the manual feature fitting branch
is incorporated into the classification branch by connection
Authorized licensed use limited to: Charles Darwin University. Downloaded on January 20,2024 at 14:55:31 UTC from IEEE Xplore. Restrictions apply.
4671
module. The manual features fitting branch and classification
branch are explained explicitly in the following.
1) Manual Features Fitting Branch: Resnet-50 can effec-
tively avoid gradient disappearance and improve model feature
extraction to predict cellular manual features better. The manual
features fitting branch uses the pre-trained Resnet-50 to predict
manual features from the input image. We retain all the fea-
ture extracting blocks of Resnet-50, and only change the fully
connected layer.
2) Classification Branch: The classification branch also uses
Resnet-50 as the feature extraction module, where Resnet-50 is
pre-trained on the 5-class dataset through supervised compara-
tive learning. As shown in Fig. 3, the feature extraction blocks in
the classification branch and the manual features fitting branch
are connected by a connection module. After that, we use 1 × 1
convolution operation to reduce the number of feature channels
to half. Finally, the fused features are input into the classification
branch.
The development of cervical cancer is a continuous process,
so there is no clear boundary between the categories of abnormal
cells. The relationships between various categories of cervical
cells are shown in Fig. 1. ASC-US and LSIL belong to the
epidermal group, while ASC-H and HSIL belong to the basal
group. Furthermore, cells in these two groups have a similar
intra-group appearance. In data labeling, if a cytologist does not
find enough evidence to determine an LSIL cell, he is prone
to label this cell as ASC-US. Therefore, manual labeling of
cells is subjective, which inevitably results in mislabeling. An
LSIL falling into HSIL should be penalized more than falling
into ASC-US. The classification loss of the model consists of
the 5-class and 2-class classification losses, which implements
a similar function to the synergistic grouping loss (SGL) [32]
4672 IEEE JOURNAL OF BIOMEDICAL AND HEALTH INFORMATICS, VOL. 26, NO. 9, SEPTEMBER 2022
Fig. 4. Different kinds of label distributions on the HUSTC dataset. The target category is ASC-US, and we scale the y-axis using the log function
for visualization. (a), (b), (c) show the original hard label, soft label generated by [7], and soft label generated by our method, respectively.
in the classification task. For example, when an ASC-H cell
is incorrectly labeled as HSIL instead of ASC-US, it is still a
correct label for a 2-class classification task. Such an operation
can make an ASC-H cell that falls into ASC-US gaining more
punishment than its misclassification as HSIL. In this way, the
synergistic loss of 2-class and 5-class classification can better
optimize the model and reduce the influence of inter-class noise
on fine-grained classification.
Since our model is a multi-task model, the model loss consists
of classification loss and manual features fitting loss. The loss
function of the model is shown as follows:
Loss = λLmf + L5−class + L2−class

0.5 (xf − yf )2 , if |xf − yf | < 1
Lmy =
|xf − yf | − 0.5, otherwise
where xf and yf denote the prediction and label of manual fea-
tures, respectively. Manual features are a N × 1 vector that com-
bines different features, N represents the number of features.
λ is used to control the weight of manual features prediction
loss in the overall loss of the model, and we set λ to 0.5 in this
paper. L5−class and L2−class are the weighted cross-entropy loss
of 5-class and 2-class classification, respectively. The weighted
cross-entropy loss is shown below

C
L = −wc yi log(pi )
i=1
where pi is the predicted result, yi is the ground truth, wc is the
weight, C is the number of classes. We set the weight of the
normal class to 0.6 and the weight of the abnormal class to 0.4.
B. Smoothing Noisy Label Regularization
Unlike natural images, various categories of cervical cells are
similar. Due to the complexity of cervical cells, noisy labels are
often present, especially intra-group mislabeling in the epider-
mal (ASC-US and LSIL) and basal groups (ASC-H and HSIL).
The label smoothing operation can effectively reduce the effect
of mislabeling on the classification model [30]. Xiang et al. [33]
used label smoothing to reduce the influence of mislabeling on
Authorized licensed use limited to: Charles Darwin University. Downloaded on January 20,2024 at 14:55:31 UTC from IEEE Xplore. Restrictions apply.
cervical cell classification. However, the original label smooth-
ing operation [7] treats the non-target categories equally by as-
signing them with identical probability without considering the
connection between the categories. Such a smoothing operation
would limit the improvement of the classification performance.
To address this problem, we used different probabilistic smooth-
ing labels by summarizing the prior between different classes of
cervical cells.
The cross-entropy loss function will make the models too
confident about the prediction, reducing their generalization
ability, especially when there are many uncertain labels on the
training data. The label smoothing operation [7] is often used
(1)
to alleviate this problem. Fig. 4(b) shows this label smoothing
result, and its equation is shown below.
(2) α
YiLS = Yi (1 − α) + (4)
K
where α is a small constant, K indicates the number of label
categories, and YiLS is the label after smoothed. This smoothing
operation does not consider the similarity between classes. ASC-
US and LSIL belong to the epidermal group, while ASC-H and
HSIL enter the basal group. The cells of the same group have
similar pathological characteristics, so the labels of the same
group should be given greater confidence in the label smoothing
operation. Therefore, we smooth the label distribution with:
αβ 
(3) Yi LS = Yi (1 − α) + Yj
Gi − 1
Yj ∈gi ,j=i
α(1 − β) 
+ Yj (5)
K − Gi − 1
Yj ∈
/ gi
where α and β are the parameters used to control the smoothing,
K indicates the number of label categories. gi indicates the
labels in the same group as Yi , and Gi is the number of the
labels in the gi . In our 5-class classification task, α and β are set
to 0.1 and 0.7, respectively. For a cell belonging to ASC-US, the
smoothed label means that the probability of this cell belonging
to ASC-US is 90%, and the probability belonging to LSIL is 7%.
To determine the parameters of α and β, we try different
combinations of parameters to search the approximate global
QIN et al.: MULTI-TASK FEATURE FUSION MODEL FOR CERVICAL CELL CLASSIFICATION
optimization of the hyper-parameters on the validation dataset.
Firstly, the parameter a and β is set to the minimum value in the
searching scope. Then we increase the parameter α and β with
rα and rβ respectively. Where rα and rβ denote parameters of
incremental size for each increment of α and β during the search.
The searching scope of α and β is 0–1. Finally, we calculate
the performance of all parameter combinations, and take the
combination with the highest accuracy on the validation set as
the optimized hyper-parameters.
C. Supervised Contrastive Learning
In recent years, contrastive learning, widely used in self-
supervised representation, has performed well on many visual
tasks [34]. The common idea of these methods is to pull together
anchors and positive samples in the embedding space and push
apart the anchors from negative samples. A positive pair often
consists of data augmentations of the sample, and negative pairs
are formed by the anchor and randomly chosen samples from
the minibatch. Self-supervised comparative learning methods
are extended to fully supervised tasks that can fully use label in-
formation. Due to the inclusion of label information, supervised
contrastive learning can make normalized embeddings from the
same class tighter than those from different classes [35].
In this paper, we use supervised contrastive learning to
pre-train the backbone network Resnet-50 of the classification
branch on the training set of HUSTC. The supervised contrastive
loss contrasts all samples of the same class as positive samples
against the negative samples from the remainder of the batch.
Such sample selection takes class label information into account
to construct an embedding space where samples of the same class
are more closely aligned. First, we apply data enhancement twice
on the input batch to obtain two copies. The encoded representa-
tions of the two copies are then obtained through Resnet-50, and
such representations are further propagated through the projec-
tion network during training. Finally, the supervised contrastive
loss is computed on the output of the projection network. This
enables the model to acquire rich feature information and thus
improve the classification performance.
D. Manual Features
To extract manual features of cervical cells, we first need
to segment cells. In this paper, we use the CE-Net [36] as the
segmentation model. The CE-Net consists of three major parts:
feature encoder, context extractor, and feature decoder. CE-Net
has been trained with enough cell segmentation annotation data
and can segment cell nuclei well.
According to the rules of The Bethesda System, we select
some discriminatively manual features. These features include
morphological features (e.g., area of nucleus, nuclear to cy-
toplasm ratio, nucleus roundness), integral optical dens, and
texture features. The formulas for these features are shown
below.
1) Area of Nucleus: It is defined as the number of pixels in
the region of a nucleus.
Authorized licensed use limited to: Charles Darwin University. Downloaded on January 20,2024 at 14:55:31 UTC from IEEE Xplore. Restrictions apply.
4673
2) Nuclear to Cytoplasm Ratio: It is defined as the ratio of
nucleus area to cytoplasm area:
Anu
N/C = (6)
Acy
where Anu is the nucleus area, and Acy is the cytoplasm area.
3) Nucleus Roundness: The roundness is calculated as the
ratio of the actual area to the area inside the circle. In this case,
the area inside the circle is approximately calculated by the target
perimeter. The roundness is given by
4π · Anu
Nrd = (7)
Ncir
where Anu is the nucleus area, and Ncir is the perimeter of the
nucleus.
4) Integral Optical Dens (IOD): To calculate the IOD of the
nucleus, we need to obtain the brightness of the image. In our
experiments, the brightness [37] is calculated by
Y(i,j) = 0.299R(i,j) + 0.587G(i,j) + 0.114B(i,j) (8)
where Y(i,j) is the brightness of the image at the position (i, j),
R(i,j) , G(i,j) , and B(i,j) are the pixel values of the image at
position (i, j) on the three RGB channels.
The optical density (OD) is derived from the Beer-Lambert
law, which measures the degree of light absorption after parallel
monochromatic light passes through a uniform, non-scattering
absorber. The optical density is related to the content of cell
chromatin. IOD is the sum of the optical densities of the points
within the indicated measurement area, and the IOD value of the
nucleus is calculated via
 Yμ
IOD = lg (9)
Y(i,j)
(i,j)∈S
where Y(i,j) denote the brightness at the position (i, j), and
S represents the nucleus region.Yμ denotes the average bright-
ness of the image background, approximating the incident light
brightness.
5) Texture Features: To obtain the texture features of
the cells, we select the gray-level co-occurrence matrix
(GLCM) [38] feature to represent texture features. GLCM con-
tains the conditional joint probabilities of all pair wise combi-
nations of gray levels given two parameters: interpixel distance
d and interpixel orientation θ. The probability measure can be
defined as:
P ro(x) = {p(i, j)|(d, θ)} (10)
where p(i, j) is defined as:
C(i, j)
p(i, j) = Ng −1 Ng −1 (11)
i=0 j=0 C(i, j)
where C(i, j) represents the number of occurrences of gray
levels and within the window. Ng is the total number of gray
levels.
The features of contrast and entropy are selected from the
GLCM feature. Contrast is a measure of the local variations
presented in an image, and entropy measures the randomness of
4674 IEEE JOURNAL OF BIOMEDICAL AND HEALTH INFORMATICS, VOL. 26, NO. 9, SEPTEMBER 2022

TABLE I be classified into three category groups, Normal (Superficial,

DESCRIPTIONS OF THE HUSTC DATASET
Parabasal), Abnormal (Koilocytotic, Koilocytotic), and Benign
(Metaplastic).

B. Implementation Details
We adopt Pytorch library and Pytorch image models library
(timm) [39] to implement our models and conduct all experi-
ments. Our experiments are conducted on a computer configured
with four GeForce RTX 2080-TI GPUs and dual Inter Xeon
TABLE II E5-2678 V3 (12-core 2.50 GHz) CPUs. For fair comparisons, we
DESCRIPTIONS OF THE SIPAKMED DATASET implement the same training scheme for all models. During the
training process, the input image size of the model is 224 × 224,
and all methods apply data enhancement including flip, rotation,
Gaussian blur, and color perturbation. We train these models for
310 epochs, using an SDG optimizer with a mini-batch size
of 32. The initial learning rate of SDG is 0.02 and decayed
according to cosine annealing. In addition, we use a linear
warm-up performed in the first three epochs.

the image texture. They can be defined as: C. Evaluation Metrics

⎧ ⎫ In this work, we adopt accuracy, sensitivity, specificity, pre-
Ng −1
 ⎨Ng −1 Ng −1
 ⎬ cision, and F1-score to assess the performance of the models.
Contrast = k2 p(i, j)| |i − j| = k (12)
⎩ ⎭ The definition of these metrics is as follows:
k=0 i=0 j=0
TP + TN
Ng −1 Ng −1 Accuracy = (15)
  TP + TN + FP + FN
Entropy = − p(i, j) log (p(i, j)) (13)
TP
i=0 j=0 Sensitivity = (16)
TP + FN
In the proposed method, every feature needs to be normalized.
TN
The normalization method is as follows: Specif icity = (17)
TN + FP
x i − μi
xi = (14) 2T P
σi F 1 − Score = (18)
2T P + F P + F N
where, xi represents the features that need to be normalized,
where T P is the number of accurately detected positive samples,
μi and σi represent the average and variance of the features,
F P represents the number of negative samples classified as
respectively.
positive, T N is the number of correctly classified negative
samples, and F N represents the number of positive instances
IV. EXPERIMENTS AND ANALYSIS
predicted as negative.
A. Dataset
Two datasets of cervical cells, i.e., the HUSTC dataset and D. Results and Analysis
the SIPaKMeD dataset, are employed to evaluate the proposed We have conducted extensive experiments to evaluate the
method. These two datasets cover different staining methods and proposed model. Firstly, we conduct ablation experiments to
imaging conditions. evaluate each module of our method. Secondly, we show the
1) HUSTC Dataset: HUSTC data set is a cervical cell classi- effect of label smoothing with different parameters. Finally, our
fication dataset built by our team. We collected 800 slides from proposed method is compared with existing methods.
The Second Affiliated Hospital of Harbin Medical University, 1) Ablation Study: In the experiment, we analyze the role
and these slides are scanned using digital slide scanners (WS-10, of the four key modules in our model and verify their perfor-
WISILEAP) with 20× objective lenses. These data are labeled mance. These modules are: 1) manual features fitting branch, 2)
and verified by multiple experienced cytologists. The data set classification branch, 3) pre-training using supervised contrast
contains 70197 single-cell images. The normal class is NILM, learning, 4) our proposed label smoothing operation.
and the abnormal classes are ASC-US, LSIL, ASC-H and HSIL. Our proposed method is based on the Resnet-50, so we use
The image number of each category is shown in Table I. Resnet-50 as the baseline model. The results of the ablation
2) Sipakmed: SIPaKMeD is an open-source pap smear im- study are shown in Table III. As we can see, the proposed
age database that contains 4049 cell images [8]. On the model has achieved the best performance in both the 5-class
SIPaKMeD dataset, cells are classified into five categories ac- and 2-class classification tasks. Our proposed method obtains
cording to morphology and cellular appearance. The division a 2.83% and 0.43% improvement of accuracy over the Resnet-
of these cell categories is shown in Table II. SIPaKMeD can 50 in the 5-class and 2-class classification tasks. As seen from
Authorized licensed use limited to: Charles Darwin University. Downloaded on January 20,2024 at 14:55:31 UTC from IEEE Xplore. Restrictions apply.
QIN et al.: MULTI-TASK FEATURE FUSION MODEL FOR CERVICAL CELL CLASSIFICATION
TABLE III
ABLATION ANALYSIS FOR THE PROPOSED METHOD
‘MF’ indicates the addition of manual features fitting branch, ‘ML’ denotes the addition of multi-classification branch, ‘SC’ means the pre-training using supervised contrast
learning, and ‘SL’ refers to using our proposed label smoothing operation.
Fig. 5. Confusion matrices for classification on the HUSTC dataset.
We treat abnormal cells as positive samples in the binary classification
task.
the confusion matrix in Fig. 5, our proposed method can improve
the classification of indistinguishable ASC-US, while the false
positive rate of prediction is effectively reduced. After adding
the manual features fitting branch to the model, the accuracy rate
of the 5-class classification is improved to 80.89%. The accuracy
rate is improved by 1.11% and 0.12% over the baseline model on
5-class and 2-class classification tasks, with the addition of the
multi-classification branch prediction module to the model. Fur-
thermore, supervised contrastive learning pre-training and label
smoothing operation can help the model further improve its per-
formance. Finally, our model achieves 81.88% and 99.52% accu-
racy on the 5-class and 2-class classification tasks, respectively.
2) Smoothing Label Regularization: Instead of using hard
labels in model training, the labels are processed with the
smoothing operations shown in Eq. (5). To use our proposed
Authorized licensed use limited to: Charles Darwin University. Downloaded on January 20,2024 at 14:55:31 UTC from IEEE Xplore. Restrictions apply.
4675
label smoothing operation, we divide cells into different groups.
The detailed division is shown in Table I and Table II. The
HUSTC dataset includes three category groups of epidermic,
basal, and NILM. Moreover, the SIPaKMeD dataset includes
normal, abnormal, and benign category groups.
To reduce the search space of the algorithm, we set α to 0.1 and
only search for β, where rβ is set to 0.1. When α = 0, the label
smoothing does not work. Our proposed method approximates
the use of uniform distribution to smooth hard labels [7], when
α = 0.1 and β = 0.2. As shown in Fig. 6(a), the model achieves
the highest accuracy of 0.8188 on the HUSTC dataset when
β is set to 0.7. As shown in Fig. 6(b), our proposed label
smoothing operation effectively improves the performance on
the SIPaKMeD dataset when β is between 0.3 and 1 and achieves
the best accuracy of 0.9867 when β = 0.6. Experimental results
further confirm the effectiveness of our proposed label smooth-
ing method.
3) Quantitative Comparison: We compare our model with
other methods on the HUSTC and SIPaKMeD datasets. On
the HUSTC dataset, the compared models are: Multi-tasking
model NDDR-CNN [31], CNN-based models (e.g. ResNet-152,
EfficientNet-B3 [19], VOLO-D2 [40]), Mixer-B/16 [41], and
Transformer-based models (e.g. ViT-B/16 [24], Swin-B [25],
XCiT-S24 [26]). The results are shown in Table IV. Compared
with the advanced multi-task learning model NDDR-CNN, our
proposed model improves the way of information interaction
between tasks, enabling classification tasks to acquire more
prior knowledge from manual features. As a result, our model
performs better than NDDR-CNN in all metrics. Among the
compared CNN-based models, EfficientNet-B3 achieves the
best results thanks to neural architecture search techniques.
Compared to EfficientNet-B3, our model performs better on
both the 5-class and 2-class classification tasks, with a 0.3%
and 0.28% improvement in specificity. XCiT-S24 combines the
accuracy of conventional transformers with the scalability of
convolutional architectures, and achieves the best performance
among the compared Transformer-based models in the 5-class
classification task. Compared to these mainstream classifica-
tion models, our method achieves the best performance on the
HUSTC dataset.
On the SIPaKMeD dataset, we use 5-fold cross-validation
for model verification. Our method is compared with
several conventional classifiers(e.g., DenseNet-121, Inception
4676 IEEE JOURNAL OF BIOMEDICAL AND HEALTH INFORMATICS, VOL. 26, NO. 9, SEPTEMBER 2022

Fig. 6. Label smoothing performance with different β.

TABLE IV
COMPARISON RESULTS WITH DIFFERENT CLASSIFICATION METHODS ON THE HUSTC DATASET

TABLE V
COMPARISON RESULTS WITH DIFFERENT CLASSIFICATION METHODS ON THE SIPAKMED DATASET

v3, EfficientNet-B3 [19], XCiT-S24 [26], Swin-B [25]), and
models designed for cervical cells (e.g., Plissiti et al. [8], Win et
al. [15], DeepPap [21], CompactVGG [42]). On the SIPaKMeD
dataset, we classify Benign as the normal class to complete the
2-class classification task. The experiment results are shown in
Table V. Win et al. [15] used a marker-controlled watershed
approach to segment cells and extracted manual features. After
that, a bagging ensemble classifier which combined the results
of five classifiers was applied achieved an accuracy of 94.09%.
CompactVGG makes VGG more compact through reasonable
design. While reducing the computational effort, the accuracy
of the model exceeds that of Inception v3, DenseNet-121 and
EfficientNet-B3. However, the Specificity of CompactVGG is
lower than that of EfficientNet-B3, XCiT-S24 and Swin-B,
which is not conducive to the practical application. In the 2-class
classification, our proposed model achieved the highest accuracy
rate of 98.96%. Among the contrasting methods of the 5-class
classification, Swin-B achieves the best results. However, the
performance of Swin-B in the 5-class classification task is still
lower than that of our proposed model.
V. DISCUSSION
Pap smear screening plays a fundamental role in preventing
women from cervical cancer. It is widely believed that computer-
assisted diagnosis can reduce the workload of cytologists and

Authorized licensed use limited to: Charles Darwin University. Downloaded on January 20,2024 at 14:55:31 UTC from IEEE Xplore. Restrictions apply.
QIN et al.: MULTI-TASK FEATURE FUSION MODEL FOR CERVICAL CELL CLASSIFICATION
decrease the rate of potential misdiagnosis. Deep learning mod-
els are used in cervical cell classification tasks and have been
proven to be more effective than traditional cell morphological
features. Moreover, deep learning models show better clas-
sification performance with continuous network architecture
optimization. However, these models are trained only with the
category labels of the cells and cannot learn using other valid
features.
Considering the rich classification information contained in
the manual features, we aim to use the manual features of cells
to assist classification models. Win et al. [15] achieved 94.09%
accuracy in the five-classification task on the SIPaKMeD
dataset just by using manual features. Basak et al. [23]
combined features from multiple deep learning models to
improve classification performance. Unlike these models,
we consider whether we can fuse the information in manual
features with the features extracted by the deep learning model.
Therefore, we propose a multi-task feature fusion deep learning
model that fully uses manual features. This enables our model
to achieve a 98.67% accuracy rate in the 5-class classification
task on the SIPaKMeD dataset.
For supervised learning, the quality of labels determines the
performance of models. The label smoothing operation can
effectively improve the performance of the classification model.
As shown in Fig. 6, after using the label smoothing operation, the
accuracy of the model has a great improvement compared to the
original one. However, existing label smoothing methods (e.g.,
OLS [30], LS [7]) do not make use of the prior knowledge of the
labels. Therefore, we smoothed the labels according to the sim-
ilar relationship of different classes of cervical cells. Compared
to the traditional smoothing method [7], our proposed method
achieves better performance on the HUSTC dataset after apply-
ing the label smoothing operation paired with prior knowledge.
Although the proposed method has achieved improvement,
there are still some limitations. First, manual features are
required as supervised information during training, and the
calculation of manual features usually requires the segmentation
of nuclei and cytoplasm. In our future work, cell segmentation
under the interference of overlapping and impurities needs to
be investigated for accurate manual features. Secondly, our
proposed label smoothing method adopts a search approach
to determine the best hyper-parameters, resulting in high
computational complexity. Future research can focus on more
stable and fast label smoothing algorithms. Finally, our method
shows high specificity and sensitivity in the 2-class classification
task of cervical cells. However, on complex datasets such as
the HUSTC, the 5-class classification sensitivity needs further
improvement. We think the reasons are as follows. 1) The data
on the HUSTC is very complex, which reduces the performance.
2) There is no obvious boundary between cells of different
categories. For example, ASC-US cells are similar to LSIL
cells, while ASC-H cells are similar to HSIL cells. In future
work, we will explore how cytologists distinguish different
cervical cells and combine their professional knowledge with
deep learning to improve performance.
In computer-aided diagnosis systems, the requirement for
sensitivity and specificity in the cell classification is different
Authorized licensed use limited to: Charles Darwin University. Downloaded on January 20,2024 at 14:55:31 UTC from IEEE Xplore. Restrictions apply.
4677
from that in the patient classification. In cell classification, we
expect sensitivity and specificity to be as high as possible. Low
sensitive results in missing diagnosis cells, while low specificity
results in high false positive cells. The latter will bring a heavy
workload to cytologists. The cell classification is the base of
the patient classification. The patient classification can be made
by considering the amounts and the lesion degree of abnormal
cells. In this situation, high sensitivity of the diagnostic system
is preferable because more true positive patients can be found.
The combination of the high sensitivity of the diagnostic system
and the high specificity of cytologists can effectively improve
diagnostic accuracy.
Our method can predict both 5-class and 2-class classification
tasks simultaneously. With the help of the high-precision 2-class
classification task, abnormal cells can be accurately screened,
and then the recommended fine-grained classification results
are given according to the 5-class classification results. This
allows automated computer-aided diagnostic methods to give
more refined predictions with guaranteed screening accuracy.
Our proposed model can effectively promote the development
of computer-aided diagnosis methods for cervical cancer and
improve the accuracy of cytologists in cancer diagnosis.
VI. CONCLUSION
In this paper, we propose a multi-task model that can effec-
tively classify cervical cells. We train a sub-network that fits
manual features as an auxiliary task and then fuses the middle
layer features of the sub-network with those of the classification
sub-network. The proposed multi-task model consists of two
branches, a manual features fitting branch and a classifica-
tion branch. By learning the mutually related representations
between the branches, the performance of cell classification
is effectively improved. In particular, the classification branch
allows for 5-class and 2-class classification tasks for cervical
cells. Furthermore, we propose a novel label smoothing method
that sets non-target classes to different probabilities determined
by the similarity of the cell categories. Smoothing the labels
provides the model with more inter-category information and
prevents the over-fitting of the model. The proposed method is
evaluated on the HUSTC dataset and the SIPaKMeD dataset.
It has achieved a better performance than other methods. Our
method is characterized by high sensitivity and high specificity
on the HUSTC dataset for 2-class classification tasks. This
shows the potential to reduce the workload of cytologists and
help the development of automatic cervical cancer screening.
REFERENCES
[1] M. Arbyn et al., “Estimates of incidence and mortality of cervical cancer
in 2018: A worldwide analysis,” Lancet Glob. Health, vol. 8, no. 2,
pp. e191–e203, 2020.
[2] E. Bengtsson and P. Malm, “Screening for cervical cancer using automated
analysis of PAP-smears,” Comput. Math. Methods Med., vol. 2014, 2014,
Art. no. 842037 .
[3] R. Nayar and D. C. Wilbur, The Bethesda System for Reporting Cervical
Cytology: Definitions, Criteria, and Explanatory Notes. Berlin, Germany:
Springer, 2015.
[4] S. Ruder, “An overview of multi-task learning in deep neural networks,”
2017, arXiv:1706.05098.
4678 IEEE JOURNAL OF BIOMEDICAL AND HEALTH INFORMATICS, VOL. 26, NO. 9, SEPTEMBER 2022
[5] K. Bora, M. Chowdhury, L. B. Mahanta, M. K. Kundu, and A. K. Das, “Au-
tomated classification of pap smear images to detect cervical dysplasia,”
Comput. Methods Prog. Biomed., vol. 138, pp. 31–47, 2017.
[6] T. Chankong, N. Theera-Umpon, and S. Auephanwiriyakul, “Automatic
cervical cell segmentation and classification in pap smears,” Comput.
Methods Prog. Biomed., vol. 113, no. 2, pp. 539–556, 2014.
[7] C. Szegedy, V. Vanhoucke, S. Ioffe, J. Shlens, and Z. Wojna, “Rethinking
the inception architecture for computer vision,” in Proc. IEEE Conf.
Comput. Vis. Pattern Recognit., 2016, pp. 2818–2826.
[8] M. E. Plissiti, P. Dimitrakopoulos, G. Sfikas, C. Nikou, O. Krikoni,
and A. Charchanti, “SIPAKMED: A new dataset for feature and image
based classification of normal and pathological cervical cells in pap
smear images,” in Proc. 25th IEEE Int. Conf. Image Process., 2018,
pp. 3144–3148.
[9] A. D. Brown and A. M. Garber, “Cost-effectiveness of 3 methods to
enhance the sensitivity of papanicolaou testing,” JAMA, vol. 281, no. 4,
pp. 347–353, 1999.
[10] M. M. Rahaman et al., “A survey for cervical cytopathology image analysis
using deep learning,” IEEE Access, vol. 8, pp. 61687–61710, 2020.
[11] J. A. K. Suykens and J. Vandewalle, “Least squares support vector machine
classifiers,” Neural Process. Lett., vol. 9, no. 3, pp. 293–300, 1999.
[12] Y. Freund and R. E. Schapire, “A decision-theoretic generalization of on-
line learning and an application to boosting,” J. Comput. Syst. Sci., vol. 55,
no. 1, pp. 119–139, 1997.
[13] P. Wang, L. Wang, Y. Li, Q. Song, S. Lv, and X. Hu, “Automatic cell nuclei
segmentation and classification of cervical pap smear images,” Biomed.
Signal Process. Control, vol. 48, pp. 93–103, 2019.
[14] M. Arya, N. Mittal, and G. Singh, “Texture-based feature extraction of
smear images for the detection of cervical cancer,” IET Comput. Vis.,
vol. 12, no. 8, pp. 1049–1059, 2018.
[15] K. P. Win, Y. Kitjaidure, K. Hamamoto, and T. Myo Aung, “Computer-
assisted screening for cervical cancer using digital image processing of
pap smear images,” Appl. Sci., vol. 10, no. 5, 2020, Art. no. 1800.
[16] J. Yao, X. Zhu, J. Jonnagaddala, N. Hawkins, and J. Huang, “Whole
slide images based cancer survival prediction using attention guided deep
multiple instance learning networks,” Med. Image Anal., vol. 65, 2020,
Art. no. 101789.
[17] P. Bandi et al., “From detection of individual metastases to classi-
fication of lymph node status at the patient level: The Camelyon17
challenge,” IEEE Trans. Med. Imag., vol. 38, no. 2, pp. 550–560, Feb.
2019.
[18] K. He, X. Zhang, S. Ren, and J. Sun, “Deep residual learning for image
recognition,” in Proc. IEEE Conf. Comput. Vis. Pattern Recognit., 2016,
pp. 770–778.
[19] M. Tan and Q. Le, “Efficientnet: Rethinking model scaling for convolu-
tional neural networks,” in Proc. Int. Conf. Mach. Learn., 2019, pp. 6105–
6114.
[20] M. M. Rahaman et al., “Deepcervix: A deep learning-based framework
for the classification of cervical cells using hybrid deep feature fusion
techniques,” Comput. Biol. Med., vol. 136, 2021, Art. no. 104649.
[21] L. Zhang, L. Lu, I. Nogues, R. M. Summers, S. Liu, and J. Yao,
“DeepPap: Deep convolutional networks for cervical cell classifica-
tion,” IEEE J. Biomed. Health Informat., vol. 21, no. 6, pp. 1633–1643,
Nov. 2017.
[22] H. Lin, Y. Hu, S. Chen, J. Yao, and L. Zhang, “Fine-grained classification
of cervical cells using morphological and appearance based convolutional
neural networks,” IEEE Access, vol. 7, pp. 71541–71549, 2019.
Authorized licensed use limited to: Charles Darwin University. Downloaded on January 20,2024 at 14:55:31 UTC from IEEE Xplore. Restrictions apply.
[23] H. Basak, R. Kundu, S. Chakraborty, and N. Das, “Cervical cytology
classification using PCA and GWO enhanced deep features selection,” SN
Comput. Sci., vol. 2, no. 5, pp. 1–17, 2021.
[24] A. Dosovitskiy et al., “An image is worth 16 × 16 words: Transformers
for image recognition at scale,” 2020, arXiv:2010.11929.
[25] Z. Liu et al., “Swin transformer: Hierarchical vision transformer using
shifted windows,” in Proc. IEEE/CVF Int. Conf. Comput. Vis., 2021,
pp. 10012–10022.
[26] A. Ali et al., “XCiT: Cross-covariance image transformers,” in Proc. Adv.
Neural Inf. Process. Syst., vol. 34, pp. 20014–20027, 2021.
[27] S. E. Reed, H. Lee, D. Anguelov, C. Szegedy, D. Erhan, and A. Rabinovich,
“Training deep neural networks on noisy labels with bootstrapping,” 2014,
arXiv:1412.6596.
[28] A. Dubey, O. Gupta, P. Guo, R. Raskar, R. Farrell, and N. Naik, “Pair-
wise confusion for fine-grained visual classification,” in Proc. Eur. Conf.
Comput. Vis., 2018, pp. 71–88.
[29] X. Wang, D. Kihara, J. Luo, and G.-J. Qi, “EnAET: A self-trained
framework for semi-supervised and supervised learning with ensemble
transformations,” IEEE Trans. Image Process., vol. 30, pp. 1639–1647,
2021.
[30] C.-B. Zhang et al., “Delving deep into label smoothing,” IEEE Trans.
Image Process., vol. 30, pp. 5984–5996, 2021.
[31] Y. Gao, J. Ma, M. Zhao, W. Liu, and A. L. Yuille, “NDDR-CNN: Layerwise
feature fusing in multi-task CNNs by neural discriminative dimensionality
reduction,” in Proc. IEEE/CVF Conf. Comput. Vis. Pattern Recognit., 2019,
pp. 3205–3214.
[32] H. Lin, H. Chen, X. Wang, Q. Wang, L. Wang, and P.-A. Heng, “Dual-path
network with synergistic grouping loss and evidence driven risk stratifica-
tion for whole slide cervical image analysis,” Med. Image Anal., vol. 69,
2021, Art. no. 101955.
[33] Y. Xiang, W. Sun, C. Pan, M. Yan, Z. Yin, and Y. Liang, “A novel
automation-assisted cervical cancer reading method based on convolu-
tional neural network,” Biocybernetics Biomed. Eng., vol. 40, no. 2,
pp. 611–623, 2020.
[34] K. He, H. Fan, Y. Wu, S. Xie, and R. Girshick, “Momentum contrast for
unsupervised visual representation learning,” in Proc. IEEE/CVF Conf.
Comput. Vis. Pattern Recognit., 2020, pp. 9729–9738.
[35] P. Khosla et al., “Supervised contrastive learning,” in Proc. Adv. Neural
Inf. Process. Syst., 2020, pp. 18661–18673.
[36] Z. Gu et al., “CE-Net: Context encoder network for 2D medical image
segmentation,” IEEE Trans. Med. Imag., vol. 38, no. 10, pp. 2281–2292,
Oct. 2019.
[37] J. Jantzen, J. Norup, G. Dounias, and B. Bjerregaard, “Pap-smear bench-
mark data for pattern classification,” in Nature inspired Smart Inf. Syst.,
2005, pp. 1–9.
[38] R. M. Haralick, K. Shanmugam, and I. H. Dinstein, “Textural features for
image classification,” IEEE Trans. Syst., Man, Cybern., vol. SMC-3, no. 6,
pp. 610–621, Nov. 1973.
[39] R. Wightman, “Pytorch image models,” GitHub repository, 2019. [Online].
Available: https://github.com/rwightman/pytorch-image-models
[40] L. Yuan, Q. Hou, Z. Jiang, J. Feng, and S. Yan, “Volo: Vision outlooker
for visual recognition,” 2021, arXiv:2106.13112.
[41] I. O. Tolstikhin et al., “Mlp-mixer: An all-MLP architecture for vision,”
Adv. Neural Inf. Process. Syst., vol. 34, pp. 24261–24272, 2021.
[42] H. Chen et al., “CytoBrain: Cervical cancer screening system based
on deep learning technology,” J. Comput. Sci. Technol., vol. 36, no. 2,
pp. 347–360, 2021.