crystals
Article
Protein Crystal Instance Segmentation Based on Mask R-CNN
Jiangping Qin 1,2 , Yan Zhang 2 , Huan Zhou 3 , Feng Yu 3 , Bo Sun 3, * and Qisheng Wang 1,3, *






Citation: Qin, J.; Zhang, Y.; Zhou, H.;
Yu, F.; Sun, B.; Wang, Q. Protein
Crystal Instance Segmentation Based
on Mask R-CNN. Crystals 2021, 11,
157. https://doi.org/10.3390/cryst
11020157
Academic Editor: Rocco Caliandro
Received: 8 January 2021
Accepted: 1 February 2021
Published: 4 February 2021
Publisher’s Note: MDPI stays neutral
with regard to jurisdictional claims in
published maps and institutional affil-
iations.
Copyright: © 2021 by the authors.
Licensee MDPI, Basel, Switzerland.
This article is an open access article
distributed under the terms and
conditions of the Creative Commons
Attribution (CC BY) license (https://
creativecommons.org/licenses/by/
4.0/).
Crystals 2021, 11, 157. https://doi.org/10.3390/cryst11020157
1 Shanghai Institute of Applied Physics, Chinese Academy of Sciences, Shanghai 201210, China;
qinjiangping@stu.shmtu.edu.cn
2 Logistics Engineering College, Shanghai Maritime University, Shanghai 201306, China;
zhangyan@shmtu.edu.cn
3 Shanghai Advanced Research Institute, Chinese Academy of Sciences, Shanghai 201210, China;
zhouhuan@sari.ac.cn (H.Z.); yufeng@sari.ac.cn (F.Y.)
* Correspondence: sunb@sari.ac.cn (B.S.); wangqs@sari.ac.cn (Q.W.)
Abstract: Protein crystallization is the bottleneck in macromolecular crystallography, and crystal
recognition is a very important step in the experiment. To improve the recognition accuracy by
image classification algorithms further, the Mask R-CNN model is introduced for the detection of
protein crystals in this paper. Because the protein crystal image is greatly affected by backlight and
precipitate, the contrast limit adaptive histogram equalization (CLAHE) is applied with Mask R-
CNN. Meanwhile, the Transfer Learning method is used to optimize the parameters in Mask R-CNN.
Through the comparison experiments between this combined algorithm and the original algorithm,
it shows that the improved algorithm can effectively improve the accuracy of segmentation.
Keywords: protein crystal; Mask R-CNN; instance segmentation; transfer learning
1. Introduction
Protein crystallography is an important subject for studying structure biology. The
three-dimensional structural characterization of biological macromolecules is very important
in order to understand their mechanism of action. The crystallography method is widely
used in the drug discovery also, especially in the fragment-based drug screening [1–3].
At present, 169,436 protein structures have been deposited in the Protein Data Bank
(PDB), and more than 88% of them are resolved by the X-ray crystal diffraction method
(http://www1.rcsb.org (accessed on 8 January 2021)). To crystallize the protein, in most
situations, it is still necessary for researchers to observe the samples through a microscope
and to determine whether the crystallization process is completed. The observation would
cost time and labor. Therefore, to design an automated system for protein crystallography,
from protein purification to crystal growth, becomes an urgent requirement in the field of
life sciences [4–7].
Some work has been done to classify the images of protein crystals [8,9]. Because
they relied on a special imaging system, which was relatively rare at that time, it could not
achieve good classification results. The image of protein crystal is seriously affected by
the performance of backlight and the focusing of the microscope. Therefore, the quality of
the classification algorithm based on a traditional machine learning method was totally
dependent on the design of a feature vector, and the classification task could not be
well realized. Bruno et al. [10] proposed a classification algorithm based on the deep
convolutional neural network to classify protein crystallization results in 2018, which can
achieve about 94% of the classification effect. However, it is only able to show whether
there are crystals in the droplet. If we wanted to know where crystals are in the droplets
from image, the classification task would not be able to solve this problem. Meanwhile,
some commercial devices have been developed to satisfy the requirement from academic
centers and pharmaceutical companies. The fully automatic crystallization imaging system,
https://www.mdpi.com/journal/crystals
Crystals 2021, 11, x FOR PEER REVIEW 2 of 9

and pharmaceutical companies. The fully automatic crystallization imaging system,

Crystals 2021, 11, 157 Rockimager1500, was designed by the Formulatrix [11] company. However, it2was of 8 only

able to analyze the experimental drop and not crystals in the drop.
Other optical methods also have been adopted for more trials. To reduce the influ-
ence of optical scattering,
Rockimager1500, was designedthe second-order nonlinear
by the Formulatrix optical imaging
[11] company. However, method was used
it was only
toable to analyze
identify the experimental
protein crystals [7]. drop and not harmonic
The second crystals in the drop.
signal that was generated by light
Other optical
and materials wasmethods
used to also have
search been adopted
crystals. for more
Meanwhile, it trials.
was ableTo reduce
to help the influence
observe smaller
of optical scattering, the second-order nonlinear optical imaging method
crystals because a second harmonic generation (SHG) signal can frequently be observed was used to
identify protein crystals [7]. The second harmonic signal that was generated by light and
from structures that are approximately the same size as or even smaller than the lateral
materials was used to search crystals. Meanwhile, it was able to help observe smaller
resolution of the microscope. However, the method is more suitable for drug crystals with
crystals because a second harmonic generation (SHG) signal can frequently be observed
chiral. Therefore,that
from structures theare
scope of application
approximately the is limited.
same Compared
size as with the
or even smaller thanabove mentioned
the lateral
studies, none
resolution of microscope.
of the them could However,
well meetthethe requirements
method of researchers.
is more suitable Image segmenta-
for drug crystals with
tion can determine pixels which belong to the object or background in
chiral. Therefore, the scope of application is limited. Compared with the above mentionedthe image. There-
fore, thisnone
studies, paperof attempts
them could towell
use meet
the instance segmentation
the requirements algorithm
of researchers. to better
Image identify pro-
segmentation
tein
can crystals
determinein pixels
the drop.
which belong to the object or background in the image. Therefore,
this Mask
paper attempts
R-CNN was to useproposed
the instance
by segmentation
the Facebookalgorithm
Researcher to better
KM identify
He [12],protein
which inte-
crystals in the drop.
grates target detection and instance segmentation. In this paper, based on the Mask R-
CNNMask R-CNN was proposed by the Facebook Researcher KM He [12], which integrates
method, the collected protein crystal image is marked as a suitable format that is in
target detection and instance segmentation. In this paper, based on the Mask R-CNN
accordance with the format of the Microsoft Common Objects in Context (MS COCO) da-
method, the collected protein crystal image is marked as a suitable format that is in
taset for network
accordance training.
with the formatThe self-built
of the protein
Microsoft crystal
Common dataset
Objects inwas trained
Context (MSusing
COCO)the pre-
trained
dataset for network training. The self-built protein crystal dataset was trained using the was
network weight in the way of data transfer, and the Mask R-CNN network
fine-tuned.
pre-trained network weight in the way of data transfer, and the Mask R-CNN network was
fine-tuned.
2. Algorithm Design
2. Algorithm Design
2.1. Network Introduction
2.1. Network Introduction
As protein crystal images are highly affected by the light and lens focus during col-
As protein crystal images are highly affected by the light and lens focus during
lection process,
collection process,a pre-processing
a pre-processingmodule
module was addedbefore
was added beforethe
theframe
frameof of Mask
Mask R-CNN to
R-CNN
process thethe
to process input
inputimages,
images,which couldbetter
which could betterhighlight
highlight thethe features
features of protein
of protein crystals
crystals in in
the
theimage.
image. The improvedMask
The improved MaskR-CNN
R-CNN structure
structure is shown
is shown in Figure
in Figure 1. 1.

Figure
Figure 1.
1. The networkstructure
The network structureofof improved
improved Mask
Mask R-CNN.
R-CNN.

The output
The output ofof Mask
MaskR-CNN
R-CNNisisdivided
divided into three
into parts:
three the the
parts: prediction box regression,
prediction box regression,
the image classification, and the mask branch. Among them, the prediction
the image classification, and the mask branch. Among them, the prediction box regression box regression
and the
and the image
image classification
classificationbelong
belongtoto thethe
target detection
target part,part,
detection while the mask
while branch
the mask branch
belongs to the instance segmentation part.
belongs to the instance segmentation part.
In Mask R-CNN structure, the protein crystal image is input into the network, and
In Mask R-CNN structure, the protein crystal image is input into the network, and
then different feature maps are output by means of a series of convolution and pooling in
then different
feature pyramidfeature maps
networks are output
(FPN). by means
After that, different offeature
a seriesmaps
of convolution
are delivered and
intopooling
the in
feature pyramid networks (FPN). After that, different feature maps are
region proposal networks (RPN) so as to extract the region of interest (ROI). Then the ROI delivered into the
region
is input proposal
to the ROInetworks
Align to(RPN)
performso pixel
as to extract
correctiontheon region of interest
the feature map (ROI). Then the ROI
for subsequent
istarget
inputclassification
to the ROIand Align to perform
bounding pixel correction
box regression. on the
In the mask feature
branch, the map forimages
original subsequent
target classification and bounding box regression. In the mask branch, the original images
are cropped using the corrected bounding box, and then the images in ROI are performed
by cropped
are mask prediction.
using theTherefore,
correctedthe object inbox,
bounding the bounding
and then the box images
belongsin toROI
the two-class
are performed
byclassification problem Therefore,
mask prediction. (0: background,
the 1: object).
object in This can avoid inter-class
the bounding competition
box belongs and
to the two-class
classification problem (0: background, 1: object). This can avoid inter-class competition
 and the final result belongs to instance segmentation. The total loss function of Mask R-
CNN is defined as
Crystals 2021, 11, 157 3 of 8
𝐿 =𝐿 +𝐿 +𝐿 , (1)

where 𝐿 is classification loss; 𝐿 is regression loss of bounding box; 𝐿 is se-

mantic segmentation loss.
the final result belongs to instance segmentation. The total loss function of Mask R-CNN is
defined as
2.2. Pre-Processing Module Ltotal = Lcls + Lbox + Lmask , (1)
It is Lnecessary
where to use image enhancement technology to enhance the contrast of pro-
cls is classification loss; Lbox is regression loss of bounding box; Lmask is semantic
tein crystal images
segmentation loss. and to highlight the features of protein crystals in the image. Histo-
gram equalization (HE) is a common contrast enhancement method in gray space. Firstly,
the2.2. Pre-Processing
frequency Module
of each pixel level is counted by means of histogram equalization. Then,
It is necessary to
cumulative distribution functionuse image enhancement
(CDF) is used technology to enhancefrequency
in the acquired the contrastasofaprotein
result that
thecrystal
pixelsimages and toimage
of original highlight
are the features
mapped toofnew
protein crystals
pixels in the
through image.
CDF, andHistogram
this transfor-
equalization
mation process(HE) is aisnonlinear
a commontransformation.
contrast enhancement Because methodCDFinisgray space. Firstly,
a monotonic the
increasing
frequency of each pixel level is counted by means of histogram equalization. Then, cu-
function, after transformation, the brighter areas in the original image are still brighter.
mulative distribution function (CDF) is used in the acquired frequency as a result that the
Histogram equalization is a nonlinear mapping method which is performed over the en-
pixels of original image are mapped to new pixels through CDF, and this transformation
tireprocess
gray image instead transformation.
is a nonlinear of focusing on the local CDF
Because features of the image.
is a monotonic Although
increasing the pixels
function,
areafter
stretched in the more
transformation, the concentrated
brighter areas in area
theoforiginal
the gray image
image are to make
still the dynamic
brighter. Histogram range
expand,
equalization is a nonlinear mapping method which is performed over the entire gray image that
noise information may be enhanced by HE in images, and for those images
contain
insteadobviously
of focusing brighter or darker
on the local features areas,
of theitimage.
often fails to achieve
Although a significant
the pixels are stretched enhance-
in
mentthe effect.
more concentrated area of the gray image to make the dynamic range expand, noise
information
In order may be enhanced
to better deal with by HE in images,
local features, and
thisforpaper
those images
uses the that contain limited
contrast obviously adap-
brighter or darker areas, it often fails to achieve a significant enhancement
tive histogram equalization (CLAHE) algorithm to preprocess images [13]. The input im- effect.
ages are In divided
order to better
into 𝑚 deal
× with localinfeatures,
𝑛 areas CLAHE, thisand
paper usesareas
these the contrast limited adaptive
are processed separately.
histogram equalization (CLAHE) algorithm to preprocess images [13]. The input images
Firstly, the gray histogram of each area is calculated. Next, the parts of the histogram
are divided into m × n areas in CLAHE, and these areas are processed separately. Firstly,
above the threshold are cropped and then are accumulated. Then, the accumulated result
the gray histogram of each area is calculated. Next, the parts of the histogram above
is averagely
the threshold distributed
are cropped to each
and pixel
then arelevel. The slope ofThen,
accumulated. the cumulative
the accumulated distribution
result isfunc-
tion could be effectively limited by a cropping operation. Enhancement
averagely distributed to each pixel level. The slope of the cumulative distribution function of neighborhood
noise
couldaround the pixel
be effectively is mainly
limited caused by
by a cropping the slope
operation. of transformation
Enhancement function,noise
of neighborhood and the
noise thatthe
around is around the pixel
pixel is mainly is proportional
caused by the slope of totransformation
the cumulative distribution
function, and the function
noise of
thethat is around theTherefore,
neighborhood. pixel is proportional
once the slope to the ofcumulative distribution
the cumulative functionfunction
distribution of the is
neighborhood.
limited, the noiseTherefore,
could beonce the slopelimited.
effectively of the cumulative
Then, the distribution
limited gray function
histogramis limited,
is equal-
the noise could be effectively limited. Then, the limited gray histogram
ized to obtain the pixel mapping relationship. The pixel information of the edge between is equalized to
obtain the pixel mapping relationship. The pixel information of the
regions is discontinuous. As a result, a block effect occurs. Therefore, bilinear interpola- edge between regions
is discontinuous. As a result, a block effect occurs. Therefore, bilinear interpolation is used
tion is used to fix the block effect in images. Meanwhile, the bilinear interpolation can also
to fix the block effect in images. Meanwhile, the bilinear interpolation can also improve the
improve the computational efficiency. The effect of CLAHE is shown in Figure 2.
computational efficiency. The effect of CLAHE is shown in Figure 2.

Figure 2. Comparison before and after processed by contrast limit adaptive histogram equalization (CLAHE). (b) is the
Figure 2. Comparison before and after processed by contrast limit adaptive histogram equalization (CLAHE). (b) is the
image processed by CLAHE, and it can be clearly seen that the features of the crystal edge in (b) are more obvious than
image processed by CLAHE, and it can be clearly seen that the features of the crystal edge in (b) are more obvious than in
in (a).
(a).
Crystals 2021, 11, x FOR PEER REVIEW
Crystals 2021, 11, 157 4 of 8

2.3. FPN Module

2.3. FPN Module
FPN is a multi-scale feature fusion network structure which was proposed b
team FPN
of is
KM a multi-scale
He in 2017 feature
[14].fusion
FPN isnetwork structure
different fromwhich was proposed
the traditional by the
image team struc
pyramid
of KM He in 2017 [14]. FPN is different from the traditional image pyramid structure. It is
It is divided into three parts: bottom-up, top-down, and horizontal-connection. The s
divided into three parts: bottom-up, top-down, and horizontal-connection. The structure is
ture isinshown
shown in Figure 3.
Figure 3.

Figure
Figure 3. 3. Feature
Feature pyramid
pyramid networks
networks (FPN) (FPN) structure.
structure.

ResNet101
ResNet101 was wasadopted as a feature
adopted as aextraction
featurenetwork
extraction for obtaining
network different feature
for obtaining diff
maps
feature
h [ C 1 , C 2 , C3 , C4 , C5 ] in the bottom-up structure. The
mapsi 𝐶 , 𝐶 , 𝐶 , 𝐶 , 𝐶 in the bottom-up structure. The scale scale relative to the original image
relative to
is 12 , 14 , 18 , 16
1 1
, 32 . Up-sampling operation is continuously performed from the C5 to C3
original image is , , , , . Up-sampling operation is continuously performed
layers in the top-down structure. The up-sampling is performed by means of nearest
the 𝐶 to
neighbor 𝐶 layersininthe
up-sampling thetop-down
top-down structure.
structure, and the Thepurpose
up-sampling is performed
of up-sampling is to by m
double the scale of the upper-layer feature map. Through the horizontal connection,purpose
of nearest neighbor up-sampling in the top-down structure, and the the o
sampling is
up-sampled to double
high-level the can
features scale be of thewith
fused upper-layer
the low-level feature map.
features, Through
which the horiz
can better
integrate the semantic information and location information,
connection, the up-sampled high-level features can be fused with the low-level feat and can also use different
scale
which featurescan bettermore effectively.
integrateAfterthe semantic 3 × 3 convolution
fusion, the information andkernel
locationwill be used to
information, and
process the fused features in order to eliminate the aliasing effect of up-sampling.
also use different scale features more effectively. After fusion, the 3 × 3 convol
Finally, the [ P2 , P3 , P4 , P5 , P6 ] which are generated in FPN are sent to the RPN for
kernel
generating will thebe usedproposal,
region to processand the fused proposal
the region featuresare in performed
order to eliminate the aliasing eff
for target detection
up-sampling.
in RPN. There are three priori boxes with aspect ratios [2 : 1, 1 : 1, 1 : 2] generated on each
Finally,
pixel from different 𝑃 , 𝑃 ,maps
the feature 𝑃 , 𝑃 in, 𝑃RPN.which are generated
The scale of the prioriin FPN
box are sent
increases to scale
as the the RPN for
of the feature map decreases. At the
erating the region proposal, and the region proposal same time, [ P2 , P3 , P4 , P5 ] are sent to Fast RCNN
are performed for target and detecti
are combined with the region proposal that is output by RPN
RPN. There are three priori boxes with aspect ratios 2: 1,1: 1,1: 2 generated on each to perform regression and
classification on the detection frame and the recognized object. The RPN loss function is
from different feature maps in RPN. The scale of the priori box increases as the sca
the feature map decreases. 1At the same time, 𝑃1 , 𝑃 , 𝑃 ∗, 𝑃 are∗sent to Fast RCNN an
L({ pi }, {ti }) =
combined with the regionNproposal
cls
∑ L ( p , p∗ ) + λ
i cls i i
that is N
output
∑i pi Lreg (ti , ti ). (2)
reg by RPN to perform regression and
sification on the detection frame and the recognized object. The RPN loss function is
2.4. ROI Align Module
The ROI Align method was proposed in Mask R-CNN. Because it does not perform the
𝐿 𝑝 of, coordinates
quantization and rounding 𝑡 = ∑ 𝐿 ROI
of the 𝑝∗ +the
𝑝 , area, ∑ 𝑝∗of𝐿mis-alignment
𝜆 problem 𝑡 , 𝑡∗ .
between the feature map and the original image in ROI pooling was solved by ROI Align.

2.4. ROI Align Module

The ROI Align method was proposed in Mask R-CNN. Because it does not per
Crystals 2021, 11, x FOR PEER REVIEW

Crystals 2021, 11, 157 5 of 8

Align. The structure of ROI Align is shown in Figure 4. The region with dotted line r
sents
The the generated
structure of ROI Align feature
is shown maps, and4. the
in Figure The rectangle
region with region surrounded
dotted line by a solid
represents the
represents
generated the maps,
feature ROI that
and has been adjusted.
the rectangle The ROI isbydivided
region surrounded into
a solid line 5 × 5 cells.
represents the If the
ROI that has been adjusted. The ROI is divided into 5 × 5 cells. If the number
ber of samples in each cell is 4, each cell will be averaged divided into four bins, an of samples
in each cell is 4, each cell will be averaged divided into four bins, and the center of each
center of each bin is the sampling point. Since the coordinates of the ROI are floating-
bin is the sampling point. Since the coordinates of the ROI are floating-point numbers,
numbers, the coordinates of the sampling points are usually also floating-point num
the coordinates of the sampling points are usually also floating-point numbers. Therefore,
Therefore,
bilinear bilinearisinterpolation
interpolation adopt for eachissampling
adopt for each
point sampling
pixel, as shown point pixel,
by the asinshown b
arrow
arrow3. in
Figure Figure
This 3. This
operation can beoperation can be
used to obtain theused
pixel to obtain
value of thethe pixel value
sampling of the sam
point, and
point,
then fourand then four
sampling pointssampling points
are performed maxare performed
pooling on eachmax
cell. pooling onROI
Finally, the each cell. Finall
Align
output are obtained.
ROI Align output are obtained.

Figure
Figure 4. ROI
4. ROI Align
Align structure.
structure.

3. Results and Analysis

3. Results and Analysis
3.1. Experiment Platform
3.1.The
Experiment Platform for the experiment platform is based on Windows 10. The
software environment
experiment framework is keras 2.2.4 and
The software environment for tensorflow 1.13. The
the experiment CPU is AMD
platform R5 3600.
is based The
on Windows 10
memory is 16G, and the graphics processing unit (GPU) is NVIDIA RTX2060.
experiment framework is keras 2.2.4 and tensorflow 1.13. The CPU is AMD R5 3600 In order to
effectively utilize the GPU resources, the scale of the original image is adjusted to 512 × 512
memory is 16G, and the graphics processing unit (GPU) is NVIDIA RTX2060. In ord
before training. The area with no image is filled with black edges, and then, adjusted
effectively
images utilize
are input thenetwork
into the GPU forresources,
training. the scale of the original image is adjust
512 × 512 before training. The area with no image is filled with black edges, and
3.2. Experiment
adjusted Dataset
images are input into the network for training.
The COCO dataset is a dataset which is provided by the Microsoft company. It can
be used for image Dataset
3.2. Experiment segmentation or target detection. In this paper, the weight model is
obtained by pre-training on this dataset, and the crystal images are downloaded from Ma-
The COCOof dataset
chine Recognition is a dataset
Crystallization Outcomeswhich is provided
(MARCO) by the Microsoft company.
(https://marco.ccr.buffalo.edu/
be usedonfor
(accessed image 2021)).
8 January segmentation
Because theor MARCO
target detection. In this
dataset is only usedpaper, the weight mo
for classification,
the ground truth of downloaded crystal images is annotated by colleagues
obtained by pre-training on this dataset, and the crystal images are downloaded with the back- from
ground of protein crystallography. The Labelme software (https://github.com/wkentaro/
chine Recognition of Crystallization Outcomes (MARCO) (https://marco.cc
labelme/tree/v3.11.2 (accessed on 8 January 2021), version is 3.16.2) is used to annotate
falo.edu/). Because the MARCO dataset is only used for classification, the ground tru
the crystals as masks in the image, and according to the format of the COCO dataset, these
downloaded
annotated images crystal imagesasisa crystal
are designed annotated by Corresponding
dataset. colleagues with jsonthe
files,background
yaml files, of pr
crystallography. The Labelme software (https://github.com/w
and mask files are generated in the crystal dataset. The labeled mask image is shown in
taro/labelme/tree/v3.11.2,
Figure 5. version is 3.16.2) is used to annotate the crystals as masks
image, and according to the format of the COCO dataset, these annotated images ar
signed as a crystal dataset. Corresponding json files, yaml files, and mask files are g
ated in the crystal dataset. The labeled mask image is shown in Figure 5.
Crystals 2021, 11, 157 6 of 8
Crystals 2021, 11, x FOR PEER REVIEW 6 of 9

Figure 5. Four labeled mask images.

Figure 5. Four labeled mask images.

3.3. Experiment Results and Analysis

3.3. Experiment Results and Analysis
There are two important evaluation indicators for the performance of the classifica-
There are two important evaluation indicators for the performance of the classification
tion problem. One is precision, which is used to evaluate how many objects are correctly
problem. One is precision, which is used to evaluate how many objects are correctly
identified in the result of classification. The other is Recall, which is used to evaluate how
identified in the result of classification. The other is Recall, which is used to evaluate
many positive examples are predicted correctly in the total positive samples. The calcula-
how many positive examples are predicted correctly in the total positive samples. The
tion formulas for Precision and Recall are (3) and (4), respectively.
calculation formulas for Precision and Recall are (3) and (4), respectively.

𝑃 = TP , (3)
P= , (3)
TP + FP
TP
R =𝑅 = ,, (4)
(4)
TP + FN
where TP means that the positive class is predicted to be positive; FP means that the
where TP means that the positive class is predicted to be positive; FP means that the neg-
negative class is predicted to be positive; FN means that the positive class is predicted to
ative class is predicted to be positive; FN means that the positive class is predicted to be
be negative.
negative.
For the target detection network, there is a very important concept, intersection over
For the target detection network, there is a very important concept, intersection over
union (IOU). The degree of overlap of two regions is expressed by IOU. When it is adopted
union (IOU). The degree of overlap of two regions is expressed by IOU. When it is adopted
to test the accuracy of the network prediction, IOU expresses the overlap between the
to test the accuracy of the network prediction, IOU expresses the overlap between the
prediction box and the labeled box. The calculation formula is as follows:
prediction box and the labeled box. The calculation formula is as follows:
A∩B
IOU = ∩ . (5)
IOU =A ∪ B. (5)
∪

Firstly, the result of experiment is evaluated by mAP (IOU = 0.50) in this paper, and
Firstly,
10 images arethe result randomly
selected of experiment
fromisthe
evaluated by set
validation mAP (IOU = 0.50)
to calculate mAPin values.
this paper, and
Secondly,
10 images are selected randomly from the validation set to calculate mAP
100 images are randomly selected from the validation set to calculate mAP (IOU = 0.65). values. Sec-
ondly, 100 images
According to mAPare randomly
values, selected
precision from the prediction
of network validation set
cantobecalculate
verifiedmAP
after(IOU
adding=
0.65).
the According
CLAHE to mAPThe
algorithm. values, precision
results of network
are shown prediction
in Table 1. We can can
seebethat
verified after add-of
the precision
ing the CLAHE
network algorithm.
prediction The results
is improved are shown
by means in Table
of adding 1. We can see
a preprocessing that the precision
module.
of network prediction is improved by means of adding a preprocessing module.
Table 1. The mAP values of network prediction before and after adding CLAHE.
Table 1. The mAP values of network prediction before and after adding CLAHE.
ResNet101 CLAHE-ResNet101 IOU
ResNet101 CLAHE-ResNet101 IOU
10 images 0.668 0.703 0.50
10 images 0.668 0.703 0.50
100 images 0.302 0.430 0.65
100 images 0.302 0.430 0.65

The instance segmentation results of the dataset are shown in Figure 6. Even with
many precipitations, most protein crystals are identified by the network after adding the
CLAHE. The results of instance segmentation are more conforming with the shape of
protein crystals.
Crystals 2021, 11, x FOR PEER REVIEW 7 of 9

Crystals 2021, 11, 157
The instance segmentation results of the dataset are shown in Figure 6. Even with
many precipitations, most protein crystals are identified by the network after adding7 the
of 8
CLAHE. The results of instance segmentation are more conforming with the shape of pro-
tein crystals.

Figure
Figure 6.
6. Result
Result of
of protein
protein crystal
crystal instance
instance segmentation.
segmentation. The
The cases
cases of
of instance
instance segmentation
segmentation without
without CLAHE
CLAHE are
are listed
listed in
in
the first row. The cases of instance segmentation with CLAHE are listed in the second row.
the first row. The cases of instance segmentation with CLAHE are listed in the second row.

4.
4. Conclusions
Conclusions
Mask R-CNNisisintroduced
Mask R-CNN introducedinin thisthis paper
paper andand the CLAHE
the CLAHE algorithm
algorithm is as
is tried tried as an
an image
image pre-processing module. The two parts are combined to realize
pre-processing module. The two parts are combined to realize the instance segmentationthe instance segmen-
tation
task oftask of protein
protein crystal.crystal.
From theFrom the perspective
perspective of mAP of mAP quantitative
quantitative and qualitative
and qualitative analysis,
analysis, the test accuracy of the network improved by 42%,
the test accuracy of the network improved by 42%, from 30.2% to 43.0% after from 30.2% to 43.0%
additionafter
of
addition
CLAHE in IOU = 0.65. Even with just 10 images and an IOU of 0.5, the test accuracy of test
of CLAHE in IOU = 0.65. Even with just 10 images and an IOU of 0.5, the the
accuracy of the
network also networkby
improved also
5%improved
(from 66.8%by 5% (from 66.8%
to 70.3%). to 70.3%).
It is proved thatItthe
is proved that the
performance of
performance
the network can of the network can
be improved be improved
by image by image
preprocessing, evenpreprocessing,
when the model even when has
structure the
model
not beenstructure
changed has
toonot been changed too much.
much.
However,
However, according to
according to the
the results
results ofof instance
instance segmentation,
segmentation, the the spots outside the
spots outside the
droplet
droplet that are not crystals may be misidentified as crystals by the network. Therefore,
that are not crystals may be misidentified as crystals by the network. Therefore,
the
the accuracy
accuracy ofof network
network prediction
prediction will be be improved
improved in subsequent
subsequent experiments from two
aspects:
aspects: increasing
increasing the
the segmentation
segmentation accuracy
accuracy or or preprocessing
preprocessing the droplet. At the same
time,
time, the
the following
following work
work will
will also reduce the amount of computation and parameters in
order
order to improve the network computing speed.

Author Contributions: Data curation,

curation, J.Q.;
J.Q.; funding
funding acquisition,
acquisition, Q.W.;
Q.W.; investigation,
investigation, J.Q.;
J.Q.; method-
method-
ology,
ology, J.Q., Y.Z., and B.S.; resources,
resources, Q.W.;
Q.W.; supervision,
supervision, Y.Z.,
Y.Z., H.Z.,
H.Z., F.Y.,
F.Y., and B.S.; validation,
validation, J.Q.;
J.Q.;
writing—original
writing—original draft,
draft, J.Q.;
J.Q.; writing—review
writing—review & & editing,
editing, B.S.
B.S. and
and Q.W.
Q.W. All authors have read and
agreed
agreed to the published version of the manuscript.
Funding:
Funding: This
This research
research was
was funded
funded by a grant from the National Key Research and Development
Program
Program of
of China
China (Grant
(Grant No.
No. 2017YFA0504901).
2017YFA0504901).
Acknowledgments:
Acknowledgments: We We thank
thank the staff of
the staff of beamline
beamline BL17U1
BL17U1 and
and the
the experimental
experimental auxiliary
auxiliary system
system
at Shanghai Synchrotron Radiation Facility (SSRF), beamlines BL18U1 at National Facility
at Shanghai Synchrotron Radiation Facility (SSRF), beamlines BL18U1 at National Facility forfor Pro-
Protein
tein Sciences (NFPS) for images support and technology guidance.
Sciences (NFPS) for images support and technology guidance.
Conflicts of Interest:
Conflicts of Interest: The
The authors
authors declare no conflict
declare no conflict of
of interest.
interest.

References
1. Blundell, T.L.
T.L. Protein
Protein crystallography
crystallographyand
anddrug
drugdiscovery:
discovery:Recollections
Recollectionsofofknowledge
knowledgeexchange
exchangebetween
betweenacademia
academia and
and indus-
industry.
try.
IUCrJIUCrJ
2017,2017, 4, 308–321,
4, 308–321. doi:10.1107/s2052252517009241.
[CrossRef] [PubMed]
2. Spiliopoulou, M.;M.; Valmas,
Valmas, A.; Triandafillidis, D.-P.;
D.-P.; Kosinas, C.; Fitch, A.N.; Karavassili, F.; Margiolaki, I. Applications of X-ray
Screening. Crystals 2020, 10, 54.
Powder Diffraction in Protein Crystallography and Drug Screening. 54, [CrossRef]
doi:10.3390/cryst10020054.
3. Brink, A.; Helliwell, J. Why is interoperability between the two fields of chemical crystallography and protein crystallography so
difficult? IUCrJ 2019, 6, 788–793. [CrossRef] [PubMed]
4. Theveneau, P.; Baker, P.; Barrett, R.; Beteva, A.; Bowler, M.W.; Carpentier, P.; Caserotto, H.; Sanctis, D.; Dobias, F.; Flot,
D.; et al. The Upgrade Programme for the Structural Biology beamlines at the European Synchrotron Radiation Facility—High
throughput sample evaluation and automation. In Proceedings of the 11th International Conference on Synchrotron Radiation
Instrumentation, Lyon, France, 9–13 July 2012. [CrossRef]
Crystals 2021, 11, 157 8 of 8

5. Ng, J.T.; Dekker, C.; Reardon, P.; Von Delft, F. Lessons from ten years of crystallization experiments at the SGC. Acta Crystallogr.
Sect. D Struct. Biol. 2016, 72, 224–235. [CrossRef] [PubMed]
6. Zheng, B.; Tice, J.D.; Roach, L.S.; Ismagilov, R.F. A Droplet-Based, Composite PDMS/Glass Capillary Microfluidic System for
Evaluating Protein Crystallization Conditions by Microbatch and Vapor-Diffusion Methods with On-Chip X-ray Diffraction.
Angew. Chem. Int. Ed. 2004, 43, 2508–2511. [CrossRef]
7. Kissick, D.J.; Wanapun, D.; Simpson, G.J. Second-Order Nonlinear Optical Imaging of Chiral Crystals. Annu. Rev. Anal. Chem.
2011, 4, 419–437. [CrossRef] [PubMed]
8. Spraggon, G.; Lesley, S.A.; Kreusch, A.; Priestle, J.P. Computational analysis of crystallization trials. Acta Crystallogr. Sect. D Biol.
Crystallogr. 2002, 58, 1915–1923. [CrossRef] [PubMed]
9. Snell, E.H.; Luft, J.R.; Potter, S.A.; Lauricella, A.M.; Gulde, S.M.; Malkowski, M.G.; Koszelak-Rosenblum, M.; Said, M.I.; Smith,
J.L.; Veatch, C.K.; et al. Establishing a training set through the visual analysis of crystallization trials. Part I: ∼150 000 images.
Acta Crystallogr. Sect. D Biol. Crystallogr. 2008, 64, 1123–1130. [CrossRef] [PubMed]
10. Bruno, A.E.; Charbonneau, P.; Newman, J.; Snell, E.H.; So, D.R.; Vanhoucke, V.; Watkins, C.J.; Williams, S.; Wilson, J.C.
Classification of crystallization outcomes using deep convolutional neural networks. PLoS ONE 2018, 13, e0198883. [CrossRef]
[PubMed]
11. Jones, H.G.; Wrapp, D.; Gilman, M.S.A.; Battles, M.B.; Wang, N.; Sacerdote, S.; Chuang, G.-Y.; Kwong, P.D.; McLellan, J.S. Iterative
screen optimization maximizes the efficiency of macromolecular crystallization. Acta Crystallogr. Sect. F Struct. Biol. Commun.
2019, 75, 123–131. [CrossRef] [PubMed]
12. He, K.; Gkioxari, G.; Dollar, P.; Girshick, R.B. Mask R-CNN. IEEE Trans. Pattern Anal. Mach. Intell. 2020, 42, 386–397. [CrossRef]
[PubMed]
13. Reza, A.M. Realization of the Contrast Limited Adaptive Histogram Equalization (CLAHE) for Real-Time Image Enhancement. J.
VLSI Signal Process. Syst. Signal Image Video Technol. 2004, 38, 35–44. [CrossRef]
14. Lin, T.Y.; Dollár, P.; Girshick, R.; He, K.; Hariharan, B.; Belongie, S. Feature Pyramid Networks for Object Detection. In Proceedings
of the IEEE Conference on Computer Vision and Pattern Recognition 2016, Las Vegas, NV, USA, 27–30 June 2016; pp. 937–944.