See discussions, stats, and author profiles for this publication at: https://www.researchgate.

net/publication/346415448

Semi-Automatic Mask Generating for Electronics Component Inspection

Article in IEEE Transactions on Components, Packaging, and Manufacturing Technology · October 2020
DOI: 10.1109/TCPMT.2020.3033837

CITATIONS READS
4 111

4 authors, including:

Hao Wu Xiangrong Xu
Anhui University of Technology Anhui University of Technology
26 PUBLICATIONS 251 CITATIONS 70 PUBLICATIONS 306 CITATIONS

SEE PROFILE SEE PROFILE

All content following this page was uploaded by Hao Wu on 22 April 2021.

The user has requested enhancement of the downloaded file.

> REPLACE THIS LINE WITH YOUR PAPER IDENTIFICATION NUMBER (DOUBLE-CLICK HERE TO EDIT) <
Automatic Mask Generating for Electronics
Component Inspection
Hao Wu *, Wenbin Gao, Xiangrong Xu, and Sixiang Xu
Abstract—This paper proposes a new method for automatically
generating the image mask required for training MASK R-CNN.
Since manual labeling is very time-consuming and laborious to
obtain the image mask, we propose a very simple and fast method
based on Graph cut to obtain Image Mask method, which consists
of two stages: The first stage implements Graph cut-based image
segmentation, which outputs pixel-level segmentation results and
then obtains Mask of the input image through image transform;
In the second stage, the Mask R-CNN-based surface defect
detection method is implemented, which includes three branches
of different tasks, namely, the boundary box regression and
positioning branch, the boundary box classification branch, and
the segmentation branch, to realize the function of locating,
classifying and segmenting defects at the same time. The
experimental results verify the effectiveness of our proposed
method. On the premise of ensuring the detection accuracy of the
Mask R-CNN method, we can quickly and easily obtain the Mask
required to train the Mask R-CNN.
Index Terms— Mask R-CNN, Deep learning, Defect detection,
Solder joint inspection
I. INTRODUCTION
I N the manufacturing process of electronic components,
defect detection based on automatic optical inspection is very
important. The defects caused by the assembly of electronic
components into the circuit board often affect the function of
the product and reduce the economic value of the product, so
machine vision-based defect detection technology is becoming
more and more widely used in the SMT industry. In recent
years, with the rise of deep learning technology, more and more
new technologies based on convolutional neural networks have
been added to machine vision detection [1-4].
The common defects of electronic components that may
occur during the installation on the PCB are as follows: wrong
component, component shift, tombstones, missing component.
The wrong component means that the component is installed in
a mismatched position. The component shift means that the
component has moved or rotated in the horizontal welding area.
The tombstone indicates that the component is separated from
This paragraph of the first footnote will contain the date on which you
submitted your paper for review. It will also contain support information,
including sponsor and financial support acknowledgment.
This work was supported in part by National Natural Science Foundation of
China ( No. 51505002, 51605004 ) ,Anhui Provincial Natural Science
Foundation(1808085QE162)
Hao Wu is with Anhui Province Key Laboratory of Special Heavy Load
Robot, the Anhui University of Technology, China, 243032 (e-mail:
hao.wu@ahut.edu.cn)* Corresponding author.
1
the mother board in the height direction, so there is no contact
with the welding area, and the missing component refers to the
absence of components in the welding area. Figure 1 shows the
comparison between these types of defects (a)-(d) and a good
component (e).
Figure 1 different type of electronic components
In the past years, there has been a lot of research on vision
inspection. These methods can be divided into traditional image
processing methods, machine learning methods, and deep
learning methods [5]. Traditional image processing methods
use wavelet transform [6], Fourier transform[7], principal
component analysis [8] and template matching. Machine
learning methods generally first extract features such as
grayscale, color, and geometry of components, and then input
the features into the classifier for classification. Commonly
used machine learning classifiers include support vector
machines [9] and various traditional neural networks (self-
organizing map[10], back-propagation, learning vector
quantization[11], etc.).
In recent years, deep learning techniques based on deep
neural networks have achieved very excellent results in
computer vision, face recognition, image classification and
recognition, and object detection. In image recognition and
object detection, a large number of CNN-based studies are used,
and there are also some researches using CNN in the field of
component detection [12], Cai[13] proposed a novel cascaded
convolutional neural network, three CNNs with different
network parameters form the proposed cascaded CNN. A CNN
is used to adaptively learn the region of interest (ROI) of SMT
solder joint images. Then, feed the learned ROI and the entire
solder joint image to the other two CNNs respectively. Finally,
the examination results are obtained through the cascaded CNN
learned.
Wenbin Gao is with the Anhui University of Technology, China, 243032 (e-
mail:wenbingao@126.com).Xiangrong Xu is with the the Anhui University of
Technology, China, 243032 (e-mail:xuxr88@qq.com). Sixiang Xu is with the
the Anhui University of Technology, China, 243032 (e-mail:
857393428@qq.com)
> REPLACE THIS LINE WITH YOUR PAPER IDENTIFICATION NUMBER (DOUBLE-CLICK HERE TO EDIT) <
In recent years, more advanced object detection methods
Faster R-CNN [14] and instance segmentation method Mask R-
CNN [15] have been proposed. The key component of them is
the RPN (Region Proposal Network). The core idea of RPN is
to densely sample the entire input image through many
overlapping bounding boxes of different shapes and sizes, and
then train the network to generate multiple object proposals
(also called Region of Interest (RoI), this design allows RPN to
smoothly search for features at different scales, RPN consists
of a convolutional neural network, input feature images, and
output the probability of bounding boxes and contained objects.
Faster R-CNN refines and classifies RoI through additional
fully connected layers on the basis of RPN, while Mask R-CNN
further improves Faster R-CNN by dividing instances with
additional convolutional layers in RoI, The specific structure of
the Mask R-CNN method has two main parts. The first part is
the Regional Proposal Network (RPN). After ROIAlign
alignment, they all enter the second part, namely the object
detection and segmentation mask prediction network.
Because Mask R-CNN has achieved good results in the
classification of object instances (obtained the best paper of the
International Conference on Computer Vision (ICCV)), it also
has research used Mask R-CNN to classify, segment and locate
components in solder component defects detection, Hao et
al[16] proposed a component defect detection method based on
Mask R-CNN. The proposed Mask R-CNN contains three
branches with different tasks, namely, boundary box regression
and positioning branch, boundary box classifying branch, and
segmentation branch to complete the tasks of positioning,
classifying and segmenting components. The regional proposal
network used in the framework can accurately align the object
boundary and obtain the location and classify the object, and
can accurately segment the defects through multiple
convolution layers and deconvolution and segmentation masks
in the segmentation branch. Experimental results prove that the
proposed method can accurately classify, locate and segment
solder components.
However, when using the above-mentioned Mask R-CNN-
based component detection method for training, it is very time-
consuming and laborious to label and obtain the target image
Mask required for training (such as using LabelMe labeling
tool to get the Masks, http://LabelMe .csail.mit.edu/Release3.0/
[17] ), we propose a new simple and fast method for
automatically generating Mask to train Mask R-CNN. The
proposed method consists of two stages to automatically
generate the image Mask. The first stage is to generate Mask
based on Graph Cut, and the second stage is based on Mask R-
CNN object detection network, that is, the Mask output from
Graph Cut, the original input image and the attached label
information are used for training, and then test samples using
the trained model.
This paper is organized as follows: the sectionⅡwill discuss
the proposed method in detail, followed by the implementation
and experimental results as well as the discussions and analysis.
II. SYSTEM OVERVIEW
The proposed method includes two parts: The Mask
segmentation module and Mask R-CNN classification module.
2
The pipeline of our proposed automatically mask generating
method for electronic component inspection is shown in figure
2. The original images are acquired by automated optical
inspection [9] with the size of 200×100 ×3 pixels. The original
input image is segmented by the graph cut method and then is
transformed to Masks. Secondly, the classification module
using Mask R-CNN method to position, classify and segment
the original image with label information and Masks.
Figure 2. The pipeline of the proposed detection method for
component inspection. (a)Original image, (b) Component
segment, (c)Segmented Mask, (d)Detection result
A. Mask Segmentation Module
In graph cut, the problem of image segmentation can be
described as binary (foreground and background) label
combination optimization process for each pixel in the image.
And the core idea of the Graph cut model[18] is to construct an
energy function, and then through the weighted graph mapping
and the application of network flow theory, the global
optimization solution of the labeling problem is transformed
into the corresponding maximum flow/minimum cut problem.
Image segmentation can be regarded as a problem of pixel
labeling. The label of the target is set to 1, and the label of the
background is set to 0. This process can minimize the energy
function by minimizing graph cuts. It is obvious that the cut that
occurs at the boundary between the target and the background
is what we want (equivalent to cutting the place where the
background and the target are connected in the image, then it is
equivalent to dividing it). At the same time, the energy should
be minimal at this time. Suppose the label of the entire image
(the label of each pixel) is L = {l1, l2 ,.., lp}, where p is the total
number of pixels in the image, li is 0 (background) or 1 (target).
Assuming that the image segmentation is L, the image energy
can be expressed as:
𝐸(𝐿) = aR(L) + B(L) (1)
Among them, R (L) is the regional term, B (L) is the
boundary term, and a is an important factor between the
regional term and the boundary term, which determines their
impact on energy. If a is 0, then only the boundary factors are
considered. E (L) represents the weight, that is, the loss
function, also called the energy function. The goal of graph cut
is to optimize the energy function to minimize its value.
In addition, because Graph Cuts needs to have two terminal
nodes "S" and "T", respectively representing the initial target
area and background area, so need to manually specify the
> REPLACE THIS LINE WITH YOUR PAPER IDENTIFICATION NUMBER (DOUBLE-CLICK HERE TO EDIT) <
initial s vertices and t vertices. For HCI, we can directly click
these two areas with the mouse. One of the areas represents a
pixel position on the desired segmentation target, and the other
is a pixel position on the background. With the definition of the
energy function and the initial terminal vertex, we can use graph
cut theory to continuously iterate and segment the target.
B. Mask R-CNN Classification Module
The Mask R-CNN can effectively detect the object in the
image, and can generate high quality instance segmentation
results. Compared with single detection, it is a more elaborate
target detection task. At the same time, due to the model
provided pixel-level annotations during training, and the target
object and background are classified with more information, so
the results of the algorithm can be effectively improved.
The Mask R-CNN is based on the improvement of Faster R-
CNN [19] and belongs to a two-stage method detector. As
shown in Figure 3, the first stage is called the Regional Proposal
Network (RPN), and its purpose is to propose regions of
interest. The second stage extracts feature of the proposed
region of interest in the first stage and processes them into three
branches, one for classification, one for bounding box
regression, and one for generating binary segmentation mask.
The features of the two stages are extracted by the underlying
backbone network and can be shared to use.
Figure 3 Mask R-CNN structure diagram
The detail structure of Mask R-CNN method will be
described as follows:
Backbone network: Use the ResNet Network [20] to extract
the features of the input image to get feature map of the whole
picture.
Regional Proposal Network (RPN): Using the sliding
window strategy, the original image is divided into anchor, all
anchor are classified and regressed with bounded box, and the
region of interest (Rol) with a higher score is extracted to
3
perform non-maximum suppression, then enter into ROI Align
layer for feature extraction.
Region of interest alignment (ROI Align layer): The ROI
Align layer extract the features of the corresponding ROI area
selected in RPN in the whole image from the backbone
network, get a fixed-size feature map of each ROI.
Classification branch and regression branch: further
classification and regression of bounding box ROI after RPN
selection; the classification branch uses the Somax activation
function.
Segmentation mask branch: use fully convolutional network
to predict the segmentation mask
The loss function L of the entire network is defined as the
sum of classification, regression, and segmentation mask
branch loss:
L = Lcls + L box + L mask
(2)
The classification branch loss function Lcls is defined as the
cross entropy function, and the bounding box regression branch
loss function is the SmoothL1 loss function, and the mask
branch loss function is the average binary cross-entropy loss.
III. EXPERIMENTS
In this section, our proposed method is evaluated on the
images of electronic component of PCB, the dataset and
experiment setup is described firstly, then the segmentation
results using graph cut method are presented. Finally, the
classification results with Mask R-CNN are reported with
comparing methods.
A. Experimental Setup
The dataset of electronic component images is collected from
a factory using automated optical inspection equipment. All of
the components are evaluated by an expert, and labeled with
region and category. For the segmentation task, all the samples
have their own label image (Masks), which is an image and had
the same size as the original image. It is shown in Figure 2(b)
as before, the black pixels in the image had the value 0, which
represented the background region, and the central color map
represented the electronic component object.
The experimental system was developed on Python3.6 and
Tensorflow deep learning platforms with GPU. For the training
of the Mask R-CNN, the network was initialized and trained on
the Microsoft COCO dataset, then the transfer learning method
use the learning information on COCO dataset to improve the
accuracy on electronic component dataset. The Mask R-CNN’s
head layer is trained for 10 epochs with a learning rate of 0.0001
firstly, and then the whole layers are trained for 30 epochs with
a learning rate of 0.00001.
In order to evaluate our proposed method performance on
electronic component detection, we use the mean of average
precision (mAP) [21] as a metric, and the intersection over
union (IoU) metric,which measure the overlap ratio between the
detection results and the ground truth, IoU was defined as:
DetectionResult ⋂ GroundTruth
𝐼𝑜𝑈 =
DetectionResult ⋃ GroundTruth
> REPLACE THIS LINE WITH YOUR PAPER IDENTIFICATION NUMBER (DOUBLE-CLICK HERE TO EDIT) <
B. Performance of Graph Cut
To evaluate the performance of the segmentation results
based on Graph cut, we compare our method with other
traditional segmentation methods, which include Otsu
threshold [22]and FCN (fully convolutional network)
method[23]. The component region is the object area; these
samples consist of different types of component mounted on the
PCB.
Figure 4 shows the segmentation results. These include a
good component sample (Figure 4 (a1)), defective component
samples such as, wrong component (Figure 4 (a2)), component
shift (Figure 4 (a3)), tombstones (Figure 4 (a4)), Missing
component (figure 4 (a5)).
Figure 4 segmentation results. (a1) a good component sample.
(a2)- (a5) different types of defect component samples; (b1)-
(b5) segmentation results with the Otsu threshold method; (c1)-
(c5) results with the FCN method; (d1)- (d5) results with the
Graph Cut method.
It can be seen from Figure 4 that the Otsu threshold method
is not very suitable for the segmentation of components. Due to
the highlight of the solder pad area, the Otsu method will always
highlight it during the segmentation process. As shown in
Figure 4 (b1) -(b5). When the electronic component is doped
with many highlight pixels in the solder pad area (the white
pixels after segment), the segmentations fails in this task using
the Otsu method.
For the FCN method and our proposed Graph Cut method,
they have relatively close effects on most sample segmentation.
However, FCN also needs to be trained first with the labeled
4
component images to obtain the segmentation effect here. The
FCN segmentation results are shown in Figure 4 (c1-c5), while
Graph Cut only requires users to simply draw a line and other
operations to segment the component part, which is shown in
Figure 4 (d1-d5). Therefore, the Graph Cut method is the
simplest and most convenient method to obtain the image
Mask.
C. Performance of Classification
To quantitatively evaluate our proposed classification
method performance based on Mask R-CNN, we also compare
it with traditional classification methods including the support
vector machine, principal component analysis and random
forest ensemble learning, and with the aforementioned related
research using Mask R-CNN method.
(1) Eginsoder [24]: This is a classic method based on
principal component analysis, which includes feature value
extraction, selection and classification.
(2) Support vector machine [9]: This is a linear classification
model. Its basic model is a linear classifier with the largest
interval defined in the feature space.
(3) Ensemble learning[25]: Based on the random forest
algorithm, the geometry and color features extracted from the
components are classified.
(4) Mask R-CNN with LabelMe [15]: first label the image
with LabelMe , then train Mask R-CNN to obtain the model,
and test the sample.
(5) Our proposed method: first use the Graph Cut proposed
in this article to label and segment the image, then train the
Mask R-CNN to obtain the model, and test the sample.
According to the above method, five experiments for
component detection were carried out.
Table І shows the experimental results. It can be proved that
the method we proposed not only obtains the highest accuracy
rate with the Mask R-CNN method used in the paper in [15],
but also the mean of average precision (mAP) index of the
segmentation is 98.9% higher, the reason may be that the
annotation images used in the literature are artificially
mandatory annotation ( Use LabelMe labeling software), and
the method proposed in this article is an interactive labeling
method, which combines the features of the image itself with
the human labeling operation, so it can obtain better
segmentation results.
TABLE I COMPARING RESULT OF THE RECOGNITION ACCURACY AND OBJECT
DETECTION WITH DIFFERENT PREVIOUS METHODS
Method Accuracy mAPbbox
Eginsoder [24] 96.43% ---
Support vector machine [9] 96.43% ---
Ensemble learning [25] 89.29% ---
Mask R-CNN with LabelMe [15] 100% 97.4%
Our proposed method 100% 98.9%
For component shift defection, there are comparing results
between our proposed method and Mask R-CNN with
LabelMe. Figure 7(a) shows the original component shift defect
sample, figure 7(b) shows the Mask R-CNN with LabelMe
detection method in [15] classify the component ambiguity with
two overlapping bounding boxes, the red dotted-line box with a
mAPbbox score of 0.934, the green dotted-line box with a
> REPLACE THIS LINE WITH YOUR PAPER IDENTIFICATION NUMBER (DOUBLE-CLICK HERE TO EDIT) <
mAPbbox score 0.738. Figure 7(c) shows the detection result
with our proposed method, which detect the component
correctly with a mAPbbox score of 0.995.
Figure 5. Component segmentation results in component
shift defect. (a)original component shift sample, (b) ambiguity
segmentation results, (c) correct segmentation results using our
proposed method.
IV. CONCLUSIONS
When MASK R-CNN method is used to locate, classify and
segment electronic components, it is very time-consuming and
laborious to label and obtain the image Mask required for Mask
R-CNN training, so we propose a new method of automatically
generating image Mask to train the Mask R-CNN, which is
based on the Graph cut method.
Experiments results verify the effectiveness of our proposed
method. Not only can we quickly and easily obtain the Mask
required for Mask R-CNN training, but also ensure the
performance of Mask R-CNN positioning, classification and
segmentation. The method we proposed can be applied to
automatic optical inspection equipment, and can also be easily
extended to other fields that require image annotation during
training.
ACKNOWLEDGMENT
This research was supported in part by the National Natural
Science Foundation of China (No. 51605004) and the Anhui
Provincial Natural Science Foundation (1808085QE162). The
authors would also like to thank Dr Jan Paul Siebert, School of
Computing Science, University of Glasgow, for his friendly
hosting and instruction during the authors’ visiting research in
UK. This support is greatly acknowledged.
5
REFERENCES
[1] He D, Xu K, Wang D. Design of multi-scale receptive field convolutional
neural network for surface inspection of hot rolled steels[J]. Image and
Vision Computing, 2019, 89: 12-20.
[2] Wen S, Chen Z, Li C. Vision-based surface inspection system for bearing
rollers using convolutional neural networks[J]. Applied Sciences, 2018,
8(12): 2565.
[3] Sun J, Wang P, Luo Y K, et al. Surface defects detection based on adaptive
multiscale image collection and convolutional neural networks[J]. IEEE
Transactions on Instrumentation and Measurement, 2019, 68(12): 4787-
4797.
[4] Wang Y, Liu M, Zheng P, et al. A smart surface inspection system using
faster R-CNN in cloud-edge computing environment[J]. Advanced
Engineering Informatics, 2020, 43: 101037.
[5] Tao X, Zhang D, Ma W, et al. Automatic metallic surface defect detection
and recognition with convolutional neural networks[J]. Applied Sciences,
2018, 8(9): 1575.
[6] Tsai D M, Hsiao B. Automatic surface inspection using wavelet
reconstruction[J]. Pattern Recognition, 2001, 34(6): 1285-1305.
[7] Tsai D M, Hsieh C Y. Automated surface inspection for directional
textures[J]. Image and Vision computing, 1999, 18(1): 49-62.
[8] Chen S H, Perng D B. Directional textures auto-inspection using principal
component analysis[J]. The International Journal of Advanced
Manufacturing Technology, 2011, 55(9-12): 1099-1110.
[9] Wu H, Zhang X, Xie H, et al. Classification of solder joint using feature
selection based on Bayes and support vector machine[J]. IEEE
Transactions on Components, Packaging and Manufacturing Technology,
2013, 3(3): 516-522.
[10] Tsai T N. Development of a soldering quality classifier system using a
hybrid data mining approach[J]. expert systems with applications, 2012,
39(5): 5727-5738.
[11] Acciani G, Brunetti G, Fornarelli G. Application of neural networks in
optical inspection and classification of solder joints in surface mount
technology[J]. IEEE Transactions on industrial informatics, 2006, 2(3):
200-209.
[12] Wei P, Liu C, Liu M, et al. CNN-based reference comparison method for
classifying bare PCB defects[J]. The Journal of Engineering, 2018,
2018(16): 1528-1533.
[13] Cai N, Cen G, Wu J, et al. SMT solder joint inspection via a novel
cascaded convolutional neural network[J]. IEEE Transactions on
Components, Packaging and Manufacturing Technology, 2018, 8(4): 670-
677.
[14] Ren S, He K, Girshick R, et al. Faster R-CNN: towards real-time object
detection with region proposal networks[J]. IEEE Transactions on Pattern
Analysis & Machine Intelligence, 2017 (6): 1137-1149.
[15] He K, Gkioxari G, Dollár P, et al. Mask r-cnn[C]//Computer Vision
(ICCV), 2017 IEEE International Conference on. IEEE, 2017: 2980-2988.
[16] Wu H, Gao W, Xu X. Solder joint recognition using Mask R-CNN
method[J]. IEEE Transactions on Components, Packaging and
Manufacturing Technology, 2020, 10(3): 525-530.
[17] Russell B C , Torralba A , Murphy K P , et al. LabelMe : A Database and
Web-Based Tool for Image Annotation[J]. International Journal of
Computer Vision, 2008, 77(1-3).
[18] Boykov Y Y, Jolly M P. Interactive graph cuts for optimal boundary &
region segmentation of objects in ND images[C]//Proceedings eighth
IEEE international conference on computer vision. ICCV 2001. IEEE,
2001, 1: 105-112.
[19] Girshick R. Fast r-cnn[C]//Proceedings of the IEEE international
conference on computer vision. 2015: 1440-1448.
[20] He K, Zhang X, Ren S, et al. Deep residual learning for image
recognition[C]//Proceedings of the IEEE conference on computer vision
and pattern recognition. 2016: 770-778.
[21] C. Manning, P. Raghavan, and H. Schtze, Introduction to Information
Retrieval, vol. 39. Cambridge University Press.
[22] Otsu N. A threshold selection method from gray-level histograms[J].
IEEE transactions on systems, man, and cybernetics, 1979, 9(1): 62-66.
[23] Long J, Shelhamer E, Darrell T. Fully convolutional networks for
semantic segmentation[C]//Proceedings of the IEEE conference on
computer vision and pattern recognition. 2015: 3431-3440.
[24] Wu, Hao, and Xiangrong Xu. "Solder joint inspection using eigensolder
features." Soldering & Surface Mount Technology30.4 (2018): 227-232.
[25] Wu H. Solder joint defect classification based on ensemble learning[J].
Soldering & Surface Mount Technology, 2017, 29(3): 164-170.
> REPLACE THIS LINE WITH YOUR PAPER IDENTIFICATION NUMBER (DOUBLE-CLICK HERE TO EDIT) < 6

Hao Wu received the B.S. degree in mechanical engineering

from Central South University, Changsha, China, in 2008, and
the Ph.D. degrees in mechanical
engineering and automation from South
China University of Technology,
Guangzhou, China, in 2013. From 2013 to
2014, he was an assistant professor at
Guangzhou institute of energy
conversion, Chinese academy of sciences,
where his work focused on machine
vision research and development. Currently, he is a Lecturer of
mechanical engineering at the Anhui University of Technology,
China. His research interests include automated visual
inspection, machine vision and pattern recognition

Wenbin Gao, Ph.D, He is currently an associate professor in the

School of Mechanical Engineering, Anhui University of
Technology, Ma’anshan, Anhui, 243002, China. His main
research direction is robotics.

Xiangrong Xu, Ph.D, He is currently a professor in the School

of Mechanical Engineering, Anhui University of Technology,
Ma’anshan, Anhui, 243002, China. His main research direction
is the principle and method of robot dynamics and trajectory
planning control .

Sixiang Xu, Ph.D, He is currently a professor in the School of

Mechanical Engineering, Anhui University of Technology,
Ma’anshan, Anhui, 243002, China. His main research direction
is robot vision and control.

View publication stats