Biomedical Signal Processing and Control 68 (2021) 102674

Contents lists available at ScienceDirect

Biomedical Signal Processing and Control

journal homepage: www.elsevier.com/locate/bspc

Locating abnormal heartbeats in ECG segments based on deep weakly

supervised learning
Yanni Tong a, 1, Yinan Sun a, 1, Peng Zhou a, Yang Shen a, Hua Jiang b, Xianzheng Sha a, *,
Shijie Chang a, *
a
Division of Biomedical Engineering, China Medical University, Shenyang, PR China
b
Department of Cardiovascular Medicine, The First Affiliated Hospital of China Medical University, Shenyang, PR China

A R T I C L E I N F O A B S T R A C T

Keywords: Electrocardiogram (ECG) examination has played a routine and crucial role in many aspects of clinic diagnosis.
Electrocardiogram An auxiliary diagnosis system that can extract effective information from ECG is valuable. In this study, we
Multiple instance learning propose to design a novel multi-instance neural network (MINN) model capable of detecting abnormal ECG
CNN
segments, meanwhile, locating abnormal heartbeats in them. The model is constructed by convolutional neural
network (CNN) and trained under the framework of multiple instance learning. It takes the interaction between
the individual heartbeat and whole ECG segment into consideration during the training process, making them
constrain each other. MIT-BIH arrhythmia database and CMUH database supported by the First Hospital of China
Medical University are used as data resources in this study. 44,332 ECG segments are extracted from both da­
tabases to exploit the model. We test our model on ECG segments with various number of heartbeats which are 5,
10, 15 and 20 respectively. The best performance of MINN on detecting ECG segments can achieve a AUC and
sensitivity up to 0.9922 and 0.9809, while on locating abnormal heartbeats can achieve a sensitivity up to
0.9473. The test results indicate our system can offer available classification and location messages, having the
potential to be applied in the analysis of long-term ECG records.

1. Introduction
Electrocardiogram (ECG) is a foundational tool with 12 channels of
signals used for revealing the overall situation of cardiac, recording
electrophysiological activities of myocardial cells which compose the
elementary functional units of cardiac. As for diagnosis of ECG, it either
depends on experienced cardiologists directly drawing conclusions from
the ectopic morphologies and rhythms, or needs to combine specific
parameters computed by algorithms. Typically, the explicit information
provided by those algorithms are heart rate, interval duration, and axis
deviation, disease categories may be given occasionally but with limited
accuracy.
Over the years, many algorithms directing at ECG analysis have
emerged. Recent trends to construct those analysis models are basing on
deep learning (DL) method, where projects such as ECG features gen­
eration and fusion, data augmentation, location of key joints of waves,
and arrhythmia detection can be fulfilled [1–5]. Since models may
perform well on one dataset obtained from particular acquisition facil­
ities and nations, but barely acceptable when switching dataset, the
practicability of those models still needs further exploration. Currently,
subject to such situations as misjudging disease types and appearance of
false negative cases, it is unrealistic to rely entirely on automatic com­
puter diagnosis over experts clinically. Thus, the ECG analysis algo­
rithms are also in the process of sustaining optimization, which mainly
reflects in trim of classifier structures, transformation and diversity of
datasets, and options of learning style. For example, establishing a larger
ECG training dataset including data collected from over 50,000 patients
and employing a modified residual network architecture, these opera­
tions elevated performance of the classifier effectively and competitively
[6]. As another example, during the training process, a generic model
could be pre-trained with a common database and then retrained with a
diverse dataset containing ECG of specific patients, aiming at making the
model dedicated [7]. Models in aforementioned researches were both
trained under supervised learning framework, which is a universal

* Corresponding authors at: Division of Biomedical Engineering, China Medical University, Shenyang, 110121, PR China.
E-mail addresses: xzsha@cmu.edu.cn (X. Sha), sjchang@cmu.edu.cn (S. Chang).
1
These two authors contributed equally to this work.

https://doi.org/10.1016/j.bspc.2021.102674
Received 29 January 2021; Received in revised form 7 April 2021; Accepted 24 April 2021
Available online 4 May 2021
1746-8094/© 2021 Elsevier Ltd. All rights reserved.
Y. Tong et al.
Fig. 1. Workflow of our proposed method.
behavior in incipient DL fields.
Usually, under supervised learning mode, plentiful data with precise
annotations such as categories, profiles, and position information of
target areas are required. However, data possessing high quality labels
need experts in relevant aspects to assist marking, which is a compli­
cated mission. With the development of DL, the implementation of
arithmetic is gradually inclined to reduce the proportion of manual work
in the overall project, which means the manual labels are needed but do
not entirely depend on them to trace all the details. Researchers strive to
utilize datasets with rough labels to train models, making them suffice to
speculate fine information. Thus, researches applying weakly supervised
learning emerge constantly. One common form in those studies is giving
the categories of training objects and then based on them to formulate
arithmetic with the capacity to predict which parts of the objects are the
dominant factors to identify classifications. For clinical utilization, the
weakly supervised learning is usually used for segmenting and classi­
fying medical images such as pathology, microscopy, and ultrasound
images [8–10]. This learning style simplifies the tag procedure of
datasets, achieving competitive effects compared to supervised learning
[11–14].
During ECG examination, the changes existing in a period of ECG
such as atypical waves and arrhythmia denoting some kind of cardio­
vascular diseases are not evident, sometimes, not easily perceived by
inexperienced physicians. Thus, it is practical to indicate the abnormal
heartbeats of ECG signals in case the occurrence of missed diagnosis. In
this study, we proposed to create an ECG screener (MINN) having the
function of giving the probabilities of ECG segments being positive,
meanwhile picking out which specific heartbeats in them are abnormal.
The screener is embedded with convolutional neural network (CNN),
and trained under the way of multiple instance learning (MIL), which is
a form of weakly supervised learning. That learning method can make
the screener own the property of training ECG segments with only
category labels, but ultimately having the ability to predict the inner fine
labels of ECG segments, that is, locating abnormal heartbeats in the
corresponding segments. The analysis results offered by our model could
offer more intuitive and helpful information to physicians, aiding the
diagnosis process more validly.
2
Biomedical Signal Processing and Control 68 (2021) 102674
2. Materials and methods
2.1. Overview
The workflow of our study was designed as delineated in Fig. 1.
Notice that the model possesses two output ports, while one port judges
whether the whole ECG segments are healthy, the other port outputs
heat maps denoting the probability values of each heartbeat being
anomalous. A total of two databases were used to train and test the
proposed model.
2.2. Data source
In this study, ECG signals from MIT-BIH arrhythmia database [15]
and CMUH database have been used. MIT-BIH arrhythmia database
includes 48 half-hour 2-channel ECG recordings of 47 subjects with a
sampling rate of 360 Hz. Referring to previous researches [1,17–19] that
used MIT-BIH database, signals of lead II were often utilized as the
analysis object, so for all ECG records we also utilized lead II signals in
this study. CMUH database contains 12-lead ECG records with a sam­
pling rate of 560 Hz collected from the patients hospitalized between
2013 and 2017 in First Hospital of China Medical University. Here, we
only used signals of lead II from CMUH database, and the length of them
is 10 s. For both databases, cardiologists provided annotation files for all
heartbeats.
2.3. Preprocessing
In the first step, wavelet transform was used to eliminate baseline
drift [16]. The next step was to crop the long signals to relatively short
segments with different lengths. The purpose of that operation was to
observe and compare the analytical performance of the model for signals
of different lengths. Since we would use the model trained by datas from
MIT-BIH to test datas from CMUH, we should make the sampling rate of
these signals uniform. Thus, the signals of CMUH are down sampled to
360 Hz.
2.4. Data partition
2.4.1. Dataset derived from MIT-BIH database
We selected 45 records (except for records ‘102’, ‘104’, ‘114’) that
contain signals of lead II from MIT-BIH database. Various types of
Y. Tong et al. Biomedical Signal Processing and Control 68 (2021) 102674

Fig. 2. Illustration of ECG segments in four datasets.

Table 1
Number of heartbeats and ECG segments involved in each dataset.
Type of heartbeat A B C D

training test training test training test training test

N 58414 14405 58064 14649 58235 14386 58267 14262

L 6412 1642 6352 1691 6283 1733 6498 1517
R 5700 1538 5974 1253 5759 1454 5546 1657
a 116 34 114 34 122 25 120 27
V 5680 1381 5626 1422 5497 1524 5729 1292
F 634 162 600 195 638 157 636 158
A 2032 498 2053 476 2078 428 1933 579
j 176 53 190 39 184 45 184 43
/ 2824 788 2869 733 2925 672 2818 766
！ 359 90 350 70 345 45 280 80
x 157 36 173 19 167 24 154 37
f 197 63 222 38 219 41 177 83
Other Ab 1954 475 1943 481 1908 511 1918 499
Total Ab 26241 6760 26466 6451 26125 6659 25993 6738
Segments 16931 4233 8453 2110 5624 1403 4213 1050

This table lists the number of healthy and abnormal heartbeats. The abbreviations of heartbeat types were achieved from the documents in MIT-BIH database, which
provided the annotations for each heartbeat. For heartbeat types with less number, we consolidated them into “other Ab” type showing their total. “Segments” stands
for the total of ECG segments in each dataset. “Ab” is the abbreviation for abnormal.

heartbeats such as right bundle branch block (R), left bundle branch
block (L), atrial premature contraction (A), premature ventricular
contraction (V), junctional premature (J), and junctional escape (j) are
included in the ECG signals. After cutting these long signals into seg­
ments with different number of heartbeats, we divided these segments
into dataset A, B, C and D, which contained 5, 10, 15 and 20 heartbeats,
respectively. Fig. 2 shows samples of the preprocessed data from those
four datasets. Table 1 counts the number of heartbeat types contained in
each dataset. The training and test segments were assigned at a ratio of
8: 2.
2.4.2. Dataset derived from CMUH database
This dataset consists of II lead ECG data of 315 people extracted from
CMUH database. Fig. 3 displays a sample of ECG segment from the
dataset. The number of different types of heartbeats in the dataset is
exhibited in Table 2.

3
Y. Tong et al. Biomedical Signal Processing and Control 68 (2021) 102674

Yi and Yik could be stated as Yi = 1 ≡ ∃k, s.t. Yik = 1 and Yi = 0 ≡ ∀k, s.

Fig. 3. Illustration of ECG segments in CMUH dataset.
Table 2
Number of heartbeats for different types.
Type of heartbeat Number
N 1801
L 700
R 590
V 59
Segments 315
2.5. Multiple instance learning
The form of weakly supervised learning we took in this study was
t. Yik = 0. Then we defined Yˆi as probability of an ECG segment being
aberrant, and Yˆik as probability of the kth heartbeat being aberrant in the
ith segment. These two types of probabilities were yield by two ports of
MINN. The Yˆik is interpreted as Yˆik = p(Yik = 1|Xik , θ), where θ = {ωz ,
bz , ∀z} represents the parameters of the zth layer of MINN. During the
training of model MINN, only coarse labels Yi that indicate the class of
ECG segments were offered, the role of MIL was to make the model learn
the ability of predicting fine labels Yik that signify which specific
heartbeats are problematic.
For the relation between Yˆi and Yˆik , generally there are two modal­
ities to express, one is using max pooling which is represented as Yˆi =
maxk Yˆik . This expression emphatically considers the impact of the most
discriminative instances but may overlook the devotion of other in­
stances, and make the derivative course ∂Yˆi /∂Yˆik exist truncation [9]. In
view of enabling the model to concurrently consider the impacts of all
the instances to one bag, not just pay attention to the instance with
maximum activation value, the other modality uses global functions to
gather instance-level predictions aiming at making more instances
participate in the derivation of bag-level prediction. We employed both
modalities to link Yˆi and Yˆik in this study, observing and comparing their
performances. The global functions we picked to express the relation of
Yˆi and Yˆik were integrated segmentation and recognition (ISR) [21],
long-sum-exponential (LSE) [22] with r equal to 1, 2.5, 5 and noisy-or
[22]. The relevant equations are:
/( )
( ) ∑ m
Yˆik ∑m
Yˆik
Yˆi = f Yˆik = 1 + ISR (1)
1 − Yˆik 1 − Yˆik
k=1 k=1

multiple instance learning (MIL), which was first proposed by Dietterich

( ) 1 1∑ m
( )
et al. to research questions of predicting pharmaceutical activities [20]. Yˆi = f Yˆik = ln exp rYˆik LSE (2)
There are two essential definitions in ideology of MIL, namely bag and r m k=1
instance. A bag, consisting of multiple instances, is deemed as negative if ( ) ∏m ( )
all the instances in it are negative, and positive provided one instance is Yˆi = f Yˆik = 1 − k=1
1 − Yˆik noisy − or (3)
positive. Similarly, an ECG segment would be diagnosed as healthy if
These formulations could decently aggregate instance probabilities
there is no abnormality in each heartbeat. Therefore, conforming to the
Yˆik into bag-level probability Yˆi , and assign similar weights to similar
notion of bags and instances in MIL, we defined bags as ECG segments
and instances as heartbeats in this study. instances in one bag. The loss function adopted in this study is defined
Here, the training dataset is expressed as S = by:
{(Xi , Yi ), i = 1, 2, 3, 4, …, n }, in which Xi denotes the ith ECG 1∑ n
(4)
(1− Yi )
segment and Yi is the corresponding label. Yi equals to 1 when at least L(θ) = − Y ∧Yi (1 − Yi∧ )
n i=1 i
one aberrant heartbeat appears in a segment, and 0 when all the
heartbeats in the segment are healthy. Bag Xi is composed of instances Through the backpropagation, parameters θ of the model would be
Xik (k = 1, 2, 3, 4, …, m), where m equals to the number of instances, updated so as to minimize the loss function.
and Yik is considered as the label of Xik . Hence, the correlation between

Fig. 4. CNN model structure. The front part is five CONV blocks: conv1_64, conv2_128, conv3_256, conv4_512 and conv5_512. The kernel size of convolutional layers
and max pooling layers are 1 × 9 and 1 × 3. Each output of the convolutional layer is activated by rectification non-linearity (ReLU) [32], and then undergoes the
batch normalization (BN) transformation [33]. The amount of neuros in three fully connected layers are 2048, 1024 and 1. Dropout [34] is applied between fully
connected layers to prevent overfitting.

4
Y. Tong et al.
Fig. 5. ROC curves of MINN and VGG-16 tested on four datasets of MIT-BIH. The symbol ‘*’ indicates the three models with highest AUC values. Variables false
positive rate (FPR) and true positive rate equal to TP/(TP + FN) and FP/(FP + TN), where TP, TN, FP and FN are defined as true positives, true negatives, false
positives and false negatives, respectively.
2.6. CNN architecture
From invention of LeNet, VGG, ResNet, etc. initially to commonly
used U-NET, GAN, etc. today, one of the stationary structures in those
networks is fitting convolutional layers. In recent years, as convolutional
layers gain an edge among image processing, they have also been used
widely in the domains of disposing one-dimensional time sequences
such as ECG [23–27] and electroencephalogram (EEG) [28–30].
Here, we created a model based on CNN, making it adjust to the
analysis of one-dimensional ECG data, meanwhile combined it with MIL
to design a model with the function of locating anomalous heartbeats in
long ECG signals. The architecture of the model is displayed in Fig. 4.
The heartbeats Xik in an ECG segment Xi are concatenated in parallel to
serve as the input of the model, and the input shape is m × s, in which m
denotes the quantity of the heartbeats and s denotes the number of
sampling points included in each heartbeat. Inspired by the convolu­
tional modules used in VGG net [31], we built similar structure in the
CONV blocks of MINN. Interactively superimposed convolutional and
pooling layers comprise the CONV blocks of MINN responsible for
extracting features from signals. Through regulating the parameters, we
found that increasing the size of convolutional kernel appropriately is
conducive to the capture of detailed information of ECG signals and the
improvement of classification results, thereby the larger kernel of 1 × 9
is chose to perform convolutional operation with generated feature
maps. Separate submodules are attached to the terminal of CONV
blocks, and the amount of them would vary according to the number of
instances contained by the bag. These submodules form different
branches to process instances independently in parallel, and each of
them is made up of three fully connected layers, executing the function
5
Biomedical Signal Processing and Control 68 (2021) 102674
of selecting features and making the related features activated when the
classification is ascertained. The last fully connected layer uses sigmoid
function to generate instance-level predictions Yˆik . After the concurrent
submodules is a MIL layer, which could aggregate the Yˆik through
function f(∙) defined above. The previously detached instances are
merged back into bags after the MIL layer, which could be expressed as
( ) 1
Yˆi = f Yˆik = f(σ (h(xik , θ) ) ), where σ(v) = 1− exp(− v), and h(xik , θ) are
the values output by the last dense layer of the corresponding submodule
which is responsible for processing instance k of bag i. Logically, the
better Yˆi prediction outcomes achieved, the probabilities of capturing
abnormal heartbeats precisely are higher. Equivalently, the selection of
heartbeats would influence ECG segments recognition, thereby, the
output of Yˆi and Yˆik constrain each other during the training process.
3. Results
3.1. Experiment A
The main purpose of this experiment was to gain a well-trained
MINN model and test it using MIT-BIH dataset. The experiment was
implemented on NVIDIA GeForce GTX 1080 Ti with 11G Memory, using
keras 2.1.2 and python 3.6.5.
3.1.1. Evaluation on bag-level prediction
One of the major premises of locating abnormal heartbeats well in a
length of ECG is assuring that the accuracy of classifying ECG segments,
namely the performance of MINN in bag-level predictions is exceptional.
The binary classification effect of MINN was evaluated in this section.
Y. Tong et al. Biomedical Signal Processing and Control 68 (2021) 102674

3.1.2. Evaluation on instance-level prediction

In this section instance-level prediction, which embodies the capa­
bility of capturing abnormal heartbeats in ECG segments, is evaluated.
Firstly, RMSE (root mean square error) were calculated based on Yik and
Yˆik (Fig. 7). The results reveal that for one specific test dataset, there is a
gap in the deviation between Yik and Yˆik generated by different

Fig. 6. AUC (95 % CI) values of MINN with different MIL functions.

Here we compared the performance of MINN with different MIL func­

tions. In addition, we also exploited the original VGG-16 as a compari­
son model. The CONV blocks of MINN was designed based on the
structure of VGG-16, and the transformation we made here is that BN
layers were added to the CONV blocks and the kernels were resized. All
Fig. 7. RMSE measured by MINN with different MIL functions.
the models were trained 200 epochs with mini batch size equal to 32,
and the hyper-parameters configurations were as follows: Adam opti­
mizer [35], setting β1 = 0.9, β2 = 0.999 and learning rate to 0.001. The
learning rate was reduced by a factor of 10 whenever the training loss
did not elevate for three successive epochs.
When the model finished training, receiver operating characteristic
(ROC) curves were constructed based on the test results Yˆi , simulta­
neously, areas under the curve (AUC) were calculated. The test results
are presented in Fig. 5. Delong test [36] was used to calculate AUC with
95 % CI (Fig. 6) and P-value. We can see that MINN with ISR, maxpool
and noisy-or global functions gained the supreme AUC values among all
those five test datasets. Through setting significance level to 0.05 we
could draw these conclusions: (i) For each dataset, the performance of
MINN using ISR, maxpool and noisy-or functions all surpass the original
VGG-16 model (P < 6 × 10− 5 ). That means appropriate network ar­
chitecture could contribute to the enhancement of detection. Meanwhile
compared to VGG-16, MINN not only has the ability to distinguish be­
tween abnormal and normal ECG segments, but also the ability to find
internal abnormal heartbeats simultaneously. (ii) In general, LSE func­
tion does not significantly improve, or even reduces the predictive
ability of the model. Besides, as recommended in [37,38], we adopted
lower r value in LSE and found that the prediction results get better with
r increasing. (iii) The difference between MINN using functions of ISR,
maxpool and noisy-or is not obvious (P > 0.05). (iv) As the length of ECG
signal increases, MINN’s ability to classify them is on a downward trend.
Based on the results of AUC, we chose ISR, maxpool and noisy-or
functions that reacted better in distinguishing problematic ECG seg­
ments to calculate their sensitivity (Sen) and specificity (Spec) in pre­
dicting Yi . The results are summarized in Table 3. We can see MINN
could achieve balanced results between Sen and Spec, which means our
Fig. 8. Violin plots of Yik
ˆ
predicted by MINN with ISR, maxpool and noisy-
proposed model has equal capability to distinguish normal and
or functions.
abnormal ECG segments.

Table 3
Performance of MINN tested on four datasets and comparison among three global functions.
Method dataset A dataset B dataset C dataset D

Sen Spec Sen Spec Sen Spec Sen Spec

MINN_ISR 0.9835 0.9857 0.9593 0.9839 0.9792 0.9759 0.9676 0.9811

MINN_maxpool 0.9864 0.9913 0.9779 0.9860 0.9641 0.9703 0.9705 0.9892
MINN_noisy-or 0.9864 0.9876 0.9822 0.9882 0.9618 0.9685 0.9809 0.9865

6
Y. Tong et al. Biomedical Signal Processing and Control 68 (2021) 102674

Table 4
Performance of MINN on instance-level and comparison among three global functions.
Method dataset A dataset B dataset C dataset D

Sen Spec Sen Spec Sen Spec Sen Spec

MINN_ISR 0.9212 0.9926 0.8830 0.9931 0.8308 0.9925 0.7832 0.9862

MINN_maxpool 0.9552 0.9954 0.9051 0.9950 0.8485 0.9912 0.8292 0.9893
MINN_noisy-or 0.9473 0.9956 0.8933 0.9939 0.8187 0.9914 0.8271 0.9832

Fig. 9. Visualization of instance-level predictions Yik

ˆ
. (A) ~ (D) depict the probability maps of ECG segments of different lengths. The symbols "N", "L", etc. are
ground truth.

functions. ISR, maxpool and noisy-or could make MINN achieve lower
RMSE on instance-level. We depict the distribution of Yˆik produced by
those three MIL functions through violin plots (Fig. 8). We can clearly
see that the Yˆik values have higher density in 0 and 1 position, meaning
the distribution of it is concentrated. But there are also some obvious
discrete points in the figure, which represent the false positive and false
negative instances emerging in the prediction process.
Table 4 shows Sen and Spec metrics of MINN in predicting Yˆik .
Compared to its performance in bag-level, we find that for the same test
dataset, Sen reduced large, while Spec did not fluctuate much. It means
that although the model could distinguish ECG segments well, the
abnormal heartbeats in them still could not be located very precisely.
The occurrence of the above situation is related to the principle of MIL,
where Yˆi is judged as 0 only if all the Yˆik in the corresponding bag Xi are
judged as 0, and Yˆi is judged as 1, provided any Yˆik in the corresponding
bag Xi is judged as 1. Thus, excellent Spec obtained at bag-level portends
high Spec at instance-level. Whereas, high Sen in bag-level just means
that at least one abnormal heartbeat in an ECG segment will be picked
out exactly, and not every abnormal one will be detected. Fig. 10. ROC curves of MINN and VGG-16 tested on CMUH dataset.
For each test sample, we visualize the last dense layer of MINN which
outputs Yˆik , and then map the probabilities back to the original data. As 3.2. Experiment B
shown in Fig. 9, it illustrates instance-level probability map calculated
by the model. The maps indicate the proposed model has the ability to The main purpose of this experiment was to evaluate the general­
locate abnormal heartbeats, which is meaningful in auxiliary diagnosis. ization ability of MINN, and the operation was testing the trained model

7
Y. Tong et al. Biomedical Signal Processing and Control 68 (2021) 102674

Table 5
Sensitivity and specificity for classifying ECG segments.
Method Sen
MINN_ISR 0.9763
MINN_maxpool 0.9467
MINN_noisy-or 0.9290
Table 6
Sensitivity and specificity for locating abnormal heartbeats.
Method Sen
MINN_ISR 0.8844
MINN_maxpool 0.7591
MINN_noisy-or 0.8013
on a brand new dataset, CMUH dataset, which had never been
encountered by our model before. This behavior could make the test
results a more realistic reflection of clinical practice effect of our
algorithm.
3.2.1. Evaluation on bag-level prediction
According to models’ prediction of Yˆi , the ROC curve was drawn in
Fig. 10. We can achieve that although MINN has lower performance on
CMUH dataset, a dataset collected from a new database and not covered
by the previous training process, MINN with ISR function still acquired a
maximum AUC of 0.9846 (95 % CI, 0.9708− 0.9985). And MINN using
maxpool and noisy-or functions could also achieve AUC values of 0.9477
(95 % CI, 0.9243− 0.9712) and 0.9520 (95 % CI, 0.9281− 0.9758).
Table 5 lists test results of Sen and Spec. Compared to the test on MIT-
BIH dataset, metric Sen remains relatively stable while the Spec
declined considerably, indicating the probability of MINN mistaking
normal for abnormal segments is high. However, for a completely new
dataset, which is quite different from MIT-BIH datasets in aspects of
country, races and acquisition device, the model with ISR function could
still achieve Sen of 0.9763. That proves the model handles the problem
of false negatives well, which is substantial in clinical diagnosis.
3.2.2. Evaluation on instance-level prediction
Table 6 is the exhibition of evaluation metrics on instance-level.
Fig. 11 shows the effect of MINN in locating abnormal heartbeats.
4. Discussion
Spec Clinically, for the diagnosis of ECG, the doctors would refer to pa­
rameters calculated by computers. That information would assist in
0.8288
0.6507 making final judgments, thus, designing auxiliary diagnosis systems,
0.8082 considering which useful messages should be provided by systems to
better support doctors to interpret ECG is essential, especially in low and
middle-income regions. Those regions are limited by medical resources,
and usually do not have seasoned doctors, resulting in more than 75 % of
relevant cardiovascular diseases deaths [39]. Based on aforementioned
situation, we aim at creating an ECG analysis system to better assist
Spec
physicians, especially those lacking experiences. The previous ECG
0.8662 auxiliary diagnostic items generally inclined to directly give ECG types,
0.8551
0.9039
which may interfere with the judgements of physicians. Considering that
shortage, our proposed model would give probability of each heartbeat
being abnormal in the form of heat map, that information could serve as
a reference in diagnosis. For the weakly supervised learning mode MIL
adopted by our study, which has been widely used in many aspects of
computer aided diagnosis [40–43], this method could avert people from
consuming plenty of time to make tags for training dataset. Compared to
those models trained under supervised learning method, the superior
thing about our model is that it could realize heartbeat detection
without labeling any heartbeat in the training datasets.
In the part of model evaluation, we used data derived from databases
built on distinct countries and races to test model. This can largely
simulate a scenario, where the model would be really applied to clinical
ECG diagnosis. The test results revealed that the generalization property
of MINN is outstanding, which makes the model a potential and pro­
spective method to apply in clinical. However, there are also some
problems and limitations in our experiments. In clinical diagnosis, the
false negative events is worth noticing. Thereby, according to the test
results of MIT-BIH dataset, we calculated how many problematic ECG
segments were mistaken for healthy ones by MINN, namely the number
of false negative data. Based on the theory of MIL, the reason an ECG
segment was considered as normal was that the model thought every
heartbeat in the segment was normal. Here, for false negative segments,
we counted the types of abnormal heartbeats and the amount of corre­
sponding types. The results are shown in Fig. 12. It can be discovered
that types’ A’,’ F’,’ a’ and’ j’ are easily regarded as normal type. It may
be due to (1) the disparity between the non-sinus P wave in type’ A’
(atrial premature contraction) and sinus P wave in type’ N’ is not

Fig. 11. Illustration of heat maps of ECG segments. The symbols "N", "L", etc. in the figure are ground truth.

8
Y. Tong et al.
Fig. 12. Graphical representation of false negative heartbeats.
Fig. 13. Illustration of normal ECG data in both datasets.
apparent, which is easy to be ignored; (2) types’ F’ (fusion of ventricular
and normal),’ a’ (aberrated atrial premature) and’ j’ (junctional escape),
those types of heartbeats involved in the training dataset were relatively
small, which prevented the model from learning their characteristics
well.
For test results of CMUH dataset, we can conclude that MINN is not
good at recognizing normal class in dataset CMUH as it is in dataset MIT-
BIH. We review the samples of normal class in both datasets and show
9
Biomedical Signal Processing and Control 68 (2021) 102674
part of them in Fig. 13. Through the observation of ECG signals, we
consider the inferior Spec may be contributed by the circumstance that
the morphology of normal waveform in two datasets exists discrepancy.
Especially, the CMUH database did not participate in the training pro­
cess, this may cause the model not recognizing the normal ECG of CMUH
database.
5. Conclusion
In this study, we proposed a novel and practical function to enrich
ECG analysis system, which is integrating the strengths of MIL with the
classification capability of CNN to locate abnormal portions of ECG. The
model aims at optimizing the auxiliary diagnosis system to offer valu­
able information to physicians rather than replace them. The test results
show the model can achieve the function of locating abnormal heart­
beats while classifying ECG signals. However, one of the limitations in
our study is that we cannot verify the generalization ability of our model
on more standard databases and real-time databases at present. Due to
the lack of ECG databases with heartbeat annotations, we will increase
cooperation with hospitals in future to collect more real-time data with
marked heartbeats to train the model, improving the robustness of it.
Another limitation is that the model can only be applied to the ECG
segments with a fixed number of heartbeats, and cannot perform
heartbeat analysis on ECG segments with uneven segmentation. Future
work should make the model capable of detecting ECG signals with any
length.
Y. Tong et al.
CRediT authorship contribution statement
Yanni Tong: Methodology, Software, Writing - original draft. Yinan
Sun: Methodology, Writing - review & editing. Peng Zhou: Software.
Yang Shen: Software. Hua Jiang: Validation. Xianzheng Sha: Super­
vision, Writing - review & editing. Shijie Chang: Writing - review &
editing.
Acknowledgement
This work was supported in part by the National Science Fund of
Liaoning under Grant 2018-64, in part by the Big Data Research for
Health Science of China Medical University under Grant Key Project 6.
Declaration of Competing Interest
The authors declare that they have no known competing financial
interests or personal relationships that could have appeared to influence
the work reported in this paper.
References
[1] M. Guanglong, W. Xiangqing, Y. Junsheng, ECG signal classification algorithm
based on fusion features, J. Phys. Conf. Ser. 1207 (2019), https://doi.org/10.1088/
1742-6596/1207/1/012003.
[2] M.Z. Poh, Y.C. Poh, P.H. Chan, C.K. Wong, L. Pun, W.W.C. Leung, Y.F. Wong, M.M.
Y. Wong, D.W.S. Chu, C.W. Siu, Diagnostic assessment of a deep learning system
for detecting atrial fibrillation in pulse waveforms, Heart (2018), https://doi.org/
10.1136/heartjnl-2018-313147.
[3] M. Hammad, S. Zhang, K. Wang, A novel two-dimensional ECG feature extraction
and classification algorithm based on convolution neural network for human
authentication, Future Gener. Comput. Syst. 101 (2019) 180–196, https://doi.org/
10.1016/j.future.2019.06.008.
[4] Y. Xia, N. Wulan, K. Wang, H. Zhang, Detecting atrial fibrillation by deep
convolutional neural networks, Comput. Biol. Med. 93 (2018) 84–92, https://doi.
org/10.1016/j.compbiomed.2017.12.007.
[5] X. Zhai, C. Tin, Automated ECG classification using dual heartbeat coupling based
on convolutional neural network, IEEE Access 6 (2018) 27465–27472, https://doi.
org/10.1109/ACCESS.2018.2833841.
[6] A.Y. Hannun, P. Rajpurkar, M. Haghpanahi, G.H. Tison, C. Bourn, M.P. Turakhia,
A.Y. Ng, Publisher correction: cardiologist-level arrhythmia detection and
classification in ambulatory electrocardiograms using a deep neural network, Nat.
Med. 25 (1) (2019) 65–69, https://doi.org/10.1038/s41591-018-0268-3. Nat.
Med. 25 (2019) 530. https://doi.org/10.1038/s41591-019-0359-9.
[7] Y. Li, Y. Pang, J. Wang, X. Li, Patient-specific ECG classification by deeper CNN
from generic to dedicated, Neurocomputing 314 (2018) 336–346, https://doi.org/
10.1016/j.neucom.2018.06.068.
[8] Y. Xu, J.Y. Zhu, E. Chang, Z. Tu, Multiple clustered instance learning for
histopathology cancer image classification, segmentation and clustering, Proc.
IEEE Comput. Soc. Conf. Comput. Vis. Pattern Recognit. (2012) 964–971, https://
doi.org/10.1109/CVPR.2012.6247772.
[9] Z. Jia, X. Huang, E.I. Chang, Y. Xu, Constrained deep weak supervision for
histopathology image segmentation, IEEE Trans. Med. Imaging 36 (2017)
2376–2388, https://doi.org/10.1109/TMI.2017.2724070.
[10] Y. Xu, T. Mo, Q. Feng, P. Zhong, M. Lai, E.I. Chang, Deep learning of feature
representation with multiple instance learning for medical image analysis, ICASSP,
IEEE Int. Conf. Acoust. Speech Signal Process. - Proc. (2014) 1626–1630, https://
doi.org/10.1109/ICASSP.2014.6853873.
[11] A. Vezhnevets, J.M. Buhmann, Towards weakly supervised semantic segmentation
by means of multiple instance and multitask learning, Proc. IEEE Comput. Soc.
Conf. Comput. Vis. Pattern Recognit. (2010) 3249–3256, https://doi.org/10.1109/
CVPR.2010.5540060.
[12] P.O. Pinheiro, R. Collobert, From image-level to pixel-level labeling with
convolutional networks, Proc. IEEE Comput. Soc. Conf. Comput. Vis. Pattern
Recognit. (2015) 1713–1721, https://doi.org/10.1109/CVPR.2015.7298780.
[13] D. Pathak, P. Krahenbuhl, T. Darrell, Constrained convolutional neural networks
for weakly supervised segmentation, Proc. IEEE Int. Conf. Comput. Vis. (2015)
1796–1804, https://doi.org/10.1109/ICCV.2015.209.
[14] J. Feng, Z.H. Zhou, Deep MIML network, 31st AAAI Conf. Artif. Intell. AAAI 2017
(2017) 1884–1890.
[15] G.B. Moody, R.G. Mark, The impact of the MIT-BIH arrhythmia database, IEEE Eng.
Med. Biol. Mag. 20 (2001) 45–50, https://doi.org/10.1109/51.932724.
[16] M. Arif, I.A. Malagore, F.A. Afsar, Detection and localization of myocardial
infarction using K-nearest neighbor classifier, J. Med. Syst. 36 (2012) 279–289,
https://doi.org/10.1007/s10916-010-9474-3.
[17] U.R. Acharya, H. Fujita, O.S. Lih, Y. Hagiwara, J.H. Tan, M. Adam, Automated
detection of arrhythmias using different intervals of tachycardia ECG segments
10
Biomedical Signal Processing and Control 68 (2021) 102674
with convolutional neural network, Inf. Sci. (Ny). 405 (2017) 81–90, https://doi.
org/10.1016/j.ins.2017.04.012.
[18] S. Kiranyaz, T. Ince, M. Gabbouj, Real-time patient-specific ECG classification by 1-
D convolutional neural networks, IEEE Trans. Biomed. Eng. (2016), https://doi.
org/10.1109/TBME.2015.2468589.
[19] S.L. Oh, E.Y.K. Ng, R.S. Tan, U.R. Acharya, Automated diagnosis of arrhythmia
using combination of CNN and LSTM techniques with variable length heart beats,
Comput. Biol. Med. (2018), https://doi.org/10.1016/j.compbiomed.2018.06.002.
[20] T.G. Dietterich, R.H. Lathrop, T. Lozano-Pérez, Solving the multiple instance
problem with axis-parallel rectangles, Artif. Intell. 89 (1997) 31–71, https://doi.
org/10.1016/s0004-3702(96)00034-3.
[21] P. Viola, J.C. Platt, C. Zhang, Multiple Instance boosting for object detection, Adv.
Neural Inf. Process. Syst. (2005) 1417–1424.
[22] Y. Xu, J.Y. Zhu, E.I.C. Chang, M. Lai, Z. Tu, Weakly supervised histopathology
cancer image segmentation and classification, Med. Image Anal. 18 (2014)
591–604, https://doi.org/10.1016/j.media.2014.01.010.
[23] Y. Zhao, J. Xiong, Y. Hou, M. Zhu, Y. Lu, Y. Xu, J. Teliewubai, W. Liu, X. Xu, X. Li,
Z. Liu, W. Peng, X. Zhao, Y. Zhang, Y. Xu, Early detection of ST-segment elevated
myocardial infarction by artificial intelligence with 12-lead electrocardiogram, Int.
J. Cardiol. 317 (2020) 223–230, https://doi.org/10.1016/j.ijcard.2020.04.089.
[24] S.M. Mathews, C. Kambhamettu, K.E. Barner, A novel application of deep learning
for single-lead ECG classification, Comput. Biol. Med. 99 (2018) 53–62, https://
doi.org/10.1016/j.compbiomed.2018.05.013.
[25] S. Hong, Y. Zhou, J. Shang, C. Xiao, J. Sun, Opportunities and challenges of deep
learning methods for electrocardiogram data: a systematic review, Comput. Biol.
Med. 122 (2020), 103801, https://doi.org/10.1016/j.compbiomed.2020.103801.
[26] Z. Yao, Z. Zhu, Y. Chen, Atrial fibrillation detection by multi-scale convolutional
neural networks, 20th Int. Conf. Inf. Fusion, Fusion 2017 - Proc (2017), https://
doi.org/10.23919/ICIF.2017.8009782.
[27] U.R. Acharya, H. Fujita, S.L. Oh, Y. Hagiwara, J.H. Tan, M. Adam, Application of
deep convolutional neural network for automated detection of myocardial
infarction using ECG signals, Inf. Sci. (Ny). 415–416 (2017) 190–198, https://doi.
org/10.1016/j.ins.2017.06.027.
[28] R. Schirrmeister, L. Gemein, K. Eggensperger, F. Hutter, T. Ball, Deep learning with
convolutional neural networks for decoding and visualization of EEG pathology,
2017 IEEE Signal Process. Med. Biol. Symp. SPMB 2017 - Proc. (2017), https://doi.
org/10.1109/SPMB.2017.8257015.
[29] R.T. Schirrmeister, J.T. Springenberg, L.D.J. Fiederer, M. Glasstetter,
K. Eggensperger, M. Tangermann, F. Hutter, W. Burgard, T. Ball, Deep learning
with convolutional neural networks for EEG decoding and visualization, Hum.
Brain Mapp. (2017), https://doi.org/10.1002/hbm.23730.
[30] Y.R. Tabar, U. Halici, A novel deep learning approach for classification of EEG
motor imagery signals, J. Neural Eng. (2017), https://doi.org/10.1088/1741-
2560/14/1/016003.
[31] K. Simonyan, A. Zisserman, Very deep convolutional networks for large-scale
image recognition, 3rd Int. Conf. Learn. Represent. ICLR 2015 - Conf. Track Proc.
(2015).
[32] X. Glorot, A. Bordes, Y. Bengio, Deep sparse rectifier neural networks, J. Mach.
Learn. Res. 15 (2011) 315–323.
[33] S. Ioffe, C. Szegedy, Batch normalization: accelerating deep network training by
reducing internal covariate shift, 32nd Int. Conf. Mach. Learn. ICML 2015 (2015)
448–456.
[34] N. Srivastava, G. Hinton, A. Krizhevsky, I. Sutskever, R. Salakhutdinov, Dropout: A
simple way to prevent neural networks from overfitting, J. Mach. Learn. Res. 15
(2014) 1929–1958.
[35] D.P. Kingma, J.L. Ba, Adam: a method for stochastic optimization, 3rd Int. Conf.
Learn. Represent. ICLR 2015 - Conf. Track Proc. (2015) 1–15.
[36] E. DeLong, D. DeLong, D. Clarke-Pearson, Comparing the areas under two or more
correlated receiver operating characteristic curves: a nonparametric, Approach,
Biometrics 44 (1988) 837–845, https://doi.org/10.2307/2531595.
[37] J. Ramon, L. De Raedt, Multi instance neural networks, ICML Work. (2000) 53–60.
[38] O.Z. Kraus, J.L. Ba, B.J. Frey, Classifying and segmenting microscopy images with
deep multiple instance learning, Bioinformatics 32 (2016) i52–i59, https://doi.
org/10.1093/bioinformatics/btw252.
[39] A.H. Ribeiro, M.H. Ribeiro, G.M.M. Paixão, D.M. Oliveira, P.R. Gomes, J.
A. Canazart, M.P.S. Ferreira, C.R. Andersson, P.W. Macfarlane, M. Wagner, T.
B. Schön, A.L.P. Ribeiro, Automatic diagnosis of the 12-lead ECG using a deep
neural network, Nat. Commun. 11 (2020) 1–9, https://doi.org/10.1038/s41467-
020-15432-4.
[40] M.M. Dundar, S. Badve, V.C. Raykar, R.K. Jain, O. Sertel, M.N. Gurcan, A multiple
instance learning approach toward optimal classification of pathology slides, Proc.
- Int. Conf. Pattern Recognit. (2010) 2732–2735, https://doi.org/10.1109/
ICPR.2010.669.
[41] L. Sun, Y. Lu, K. Yang, S. Li, ECG analysis using multiple instance learning for
myocardial infarction detection, IEEE Trans. Biomed. Eng. 59 (2012) 3348–3356,
https://doi.org/10.1109/TBME.2012.2213597.
[42] D. Wang, A. Khosla, R. Gargeya, H. Irshad, A.H. Beck, Deep Learning for
Identifying Metastatic Breast Cancer, 2016, pp. 1–6. http://arxiv.org/abs/1
606.05718.
[43] Y. Xu, Y. Li, Z. Shen, Z. Wu, T. Gao, Y. Fan, M. Lai, E.I.C. Chang, Parallel multiple
instance learning for extremely large histopathology image analysis, BMC
Bioinformatics 18 (2017) 1–15, https://doi.org/10.1186/s12859-017-1768-8.