The 7th Annual IEEE International Conference on

Cyber Technology in Automation, Control and Intelligent Systems

July 31-August 4, 2017, Hawaii, USA

Automated Visual Inspection of Metallic Parts

Jiang Liu, Yaonan Li, Caijin Pan, Heping Chen, and Ning Xi

parts have complex textures which increase the complexity



Abstract— The copper alloy casting parts are widely used

in the manufacturing of the taps, pipes, motor stators, crafts, and
so on. It is required to recognize the casting parts which have the
surface blisters during the on-line automatic grinding. In this
paper, we propose a high-speed automatic visual inspection
system to detect the surface blisters in real time. The casting part
is mounted on the manipulator that can automatically adjust the
posture of the casting part in the visual inspection system. The
images of casting parts are captured by a single high speed
industrial camera, then the feature analysis is applied to these
images to identify the blisters. The contour and intensity are
extracted to describe the surface blister features which are used
to train the recognition model. In the experiments, copper alloy
taps are inspected after grinding. A KUKA robotic arm is used to
move a part to different inspection surfaces, and an Ethernet
camera is used to capture the images. It takes about 35ms for
processing one captured frame. For the whole casting taps, it
takes less than 10s to scan the whole surface. The results show
that the failure rate is nearly 0% and false alarm rate is under
2% for each surface blister.
I. INTRODUCTION
The blisters are the gas bubbles or the sand holes that
emerge when the casting parts are produced. This surface
blister could destroy the casting parts’ appearance that is the
most significant quality feature for some kind of the casting
production. Especially for the cooper alloy casting parts
which are used for the water taps that have extreme
smoothness. During the casting parts grinding, it is required to
pick out the parts with the surface blister in the early grinding
stage. [1] In most of the factory, there are skilled workers
to find the surface blisters by the human vision. For human
inspection, there is no strict standard to describe the blisters so
that the individual difference between the workers could cause
the nonuniform quality. This situation could be amplified
when the human gets fatigued. In another case, for the large
volume production, where the production lines could be
automatically done by the robots, that is inappropriate
to inspect the surface quality by human.
Using the machine vision instead of human vision, the
performance of the inspection should be uniform, and it can be
integrated into the automatic production line for saving the
labor. But the surface properties of the cooper alloy casting
parts are disadvantageous for machine vision inspection.
Firstly, it is the high optical reflection rate of the cooper alloy
which could lead the passive optical inspection system failure.
For example, the image could be saturated when the projected
light patterns are strongly reflected. Secondly, the casting
*Resrach supported by Shenzhen Academy of Robotics, Shenzhen, China.
Jiang Liu, Yaonan Li, and Heping Chen are with Shenzhen Academy of
Robotics (phone: 0755-26951258; fax: 0755-26951258;
e-mail:15940525709@qq.com, swfang@szarobots.com,
h46incon@gmail.com, chen@szarobots.com).
Ning Xi is with Emerging Technologies Institute, The University of Hong
Kong (e-mail: xining@hku.hk)
and the difficulty of the image processing algorithm,
especially if the casting parts haven not been ground properly.
[2] Thirdly, because the blisters are different in size, depth,
shape, and location, the adaption of the inspection system is
required.
In this paper, we describe an automatic visual inspection
system which could detect the surface blisters on the cooper
alloy casting parts. In the system, an arch-like light source are
introduced to fit the curve surface, in case that the casting
parts usually have many complex shape and surface. The
blisters are clearly shown after projecting a
co mp leme ntary light to a part surface. [3] We propose an
image analysis method which can segment the local
discontinuity from the image. Then the contours and
i n t e n s i t y of these segments are used to match with the
blisters feature model, which is trained at the first stage by
using the blister sample images. In the offline processing, the
proposed algorithm could analyses one frame within 35 ms. In
the experiments, the water taps casting parts mounted on an
KUKA robotic arm are change the posture as the given path,
and the visual inspection system are working as the same time.
The results show that the failure rate is nearly 0% and the
false alarm rate is under 2% for each surface blisters. It is
proved that the proposed method can be used for surface
blisters inspection for cooper alloy casting parts.
II. VISUAL INSPECTION SYSTEM
A. Surface defects of the copper alloy casting castings
The defect features of the surface of copper alloy casting
part can be divided into scratches and sand holes. The
scratches are randomly distributed on the surfaces of copper
alloy casting parts, which is produced during castings This
defect does not belong to a surface defect. [4]
scratche
Figure 1. Scratches on the casting surface.
Blister is due to the metal in the casting process due to the
bubble caused by the hollow defect, this defect will directly
affect copper alloy casting product quality, so accurate to find
sand holes can improve the speed of grinding.

978-1-5386-0490-8/17/$31.00 © 2017 IEEE 1158


Figure 2. Copper on the casting surface.
B. Surface defect Image Detection method
Image preprocessing is the first step of visual inspection.
The image processing improves the detection speed because
the surface of the casting image does not have excessive sharp
edges. We need to do mean filter first. [5] The pixel bits that
set the center point coordinates are x, y , and the gray value
g x, y after processing is:
1
g x, y
N
¦
f M
f x, y N
Where M is a template; f x, y is a bit of pixel value in the
reference image and N is for the number of pixels in the
image. This algorithm increases the volume of smoothing in a
homogeneous region, and decreases the smoothness in sharp
variations.
Edge detection is the second step of visual work, Image
Edge is the most basic feature of image. [6] Edges can be
divided into two kinds of characteristics: one called step edge,
the gray level of its two pixels is significantly different; the
first is called the roof-shaped edge, and its gray values from
increasing to decreasing the inflection point.
Here because of the image features acquired, we adopt the
Sobel operator, and the Sobel operator can set the following
function:
g x, y ¬ª f i  1, j  1  2 f i  1, j  f i  1, j  1 ¼º
 ª¬ f i  1, j  1  2 f i  1, j  1 º¼
+ ª¬ f i  1, j  1  2 f i, j  1  f i  1, j  1 º¼
 ª¬ f i  1, j  1  2 f i, j  1  f i  1, j  1 º¼
The Sobel operator has two, one is to detect the horizontal
along the operator:
ª1 2 1º
«0 0 0» (2-3)
« »
«¬ 1 2 1»¼
The other is the operator to detect vertical leveling:
ª 1 0 1 º
« 2 0 2 »
« »
«¬ 1 0 1»¼
ª 1 1 1º ª 1 0 1º
Sobel operator and «« 0 0 0 »» , «« 1 0 1»» two operators
«¬ 1 1 1 »¼ «¬ 1 1 1»¼
the effect is better than the location of pixels with privileges.
The Sobel operator can smooth the noise, and can extract the
edge information accurately, but the precision is not high.
The traditional ROI algorithm is to intercept the acquired
image according to the area of interest, and to design different
areas of interest for different images. Thus, ignoring the
information of interest outside the area, this method,
inefficient, inflexible.
Figure 3. Automatically interested areas.
(2-1)
Figure 4. Traditional areas of interest.
Compared with traditional visual Roi acquisition method,
efficient Intelligent ROI algorithm can adapt to different
products provided by the image. [7] Here the CFAR algorithm
is used, the process is:
Step1. Global thresholds for fetching
According to the image histogram adaptive to determine Tg .
Set I for the pixel intensity value, at the given target pixel
confidence level is 1  M o'clock, Tg is determined by the
(2-2)
following:
P ^ I ! Tg ` 1  M (2-5)
The P represents the probability value, M  >0,1@ is an
empirical value, and expresses the proportion of clutter in the
whole image. In the overall image, the value of M is nearly 1. If
the image cumulative distribution function (CDF) is F by the
image histogram to be detected, the formula (2-5) can be
rewritten as:
1  F Tg 1  M (2-6)
It is convenient to statistic the histogram of the whole image
(2-4)
by the above-type Tg .
Step2. Index matrices
Set the image size of N u M , at position i, j pixel
intensity of I i , j , then point i, j of the index value defined as
Vi , j , can be obtained:
1159
 ­1, I i , j ! Tg H1
Vi , j
® (2-7)
¯0, other !
The index matrix of the image is: I0 TI (2-17)
V ^Vi , j 1 d i d N ;1 d i d M ` (2-8) 
Step3. The solution of CFAR detection of local H0
threshold TI In this image, a point-by-spot adaptive determination of TI
As shown in (3-4), a square hollow window with a is then judged to achieve target detection.
pixel-centered test is selected, the inner side length of the
square should be larger than the expected target size to ensure C. Target Pixel Clustering
that the region used to estimate the clutter statistic features In the image, an area of interest contains multiple resolution
contains all the pixels to be measured and improves the units, due to the defect in the concave range, in the obtained
accuracy of the statistical attribute. binary graph of the same target pixels cannot form a connected
For the remaining N c  D stray pixels in the clutter area of area, so the binary graph of the detection of further target pixel
the sliding window, using the distribution model, the clustering is necessary.
expression is: %LQDUL]DWLRQ $UHDFXOOLQJ
n n * n  D I n 1
f I ~ G0 D , r, n
I , D , J , n, I ! 0 ˄2-9˅
n D
J D * n * D J  nI
The I is the pixel intensity value, n is the equivalent
5HJLRQDOFDOLEUDWLRQ 
viewport, D is the shape parameter, J is the scale parameter. &OXVWHULQJ
(GJHDUHDFXOOLQJ
For the remaining N c  D stray pixels using moment

<
estimation, the expression is:
nE I 2 ,VVHDUFKLQJ
7DNHDQ\WZRDUHDV
Dˆ 1  (2-10) ILQLVKHG
nE I 2  n  1 E 2 I
1
Jˆ aˆ  1 E I (2-11)
The false alarm probability of the given detection p f D , the 0D[LPXP!'PD[" 0HUJH$UHD

corresponding CFAR detection local threshold TI can be

obtained by the following: <
T1
1 pfD ³ 0
f I dI (2-12) Figure 5. Characteristic clustering after binary value.
The length of the area of interest is L and the width is W .
In the case of single-view, G divisions degenerate into E c
0
I
The image resolution is A . [9] Because the pixels in the
distributions: image are not necessarily scattering energy in the target size,
DJ D the area of the target area detected by CFAR is S upper bound
f ZI I 1D
, D , J , I ! 0 (2-13)
J I S max :
By (2-10) and (2-11), the parameters are estimated as S d Smax L uW (2-18)
follows: Au A
E I2 The relationship between two pixels i and j in the same
D 1 
ˆ (2-14) target area satisfies the following relationship:
E I 2  2E 2 I
d i, j d d max L2  W 2 (2-19)
Jˆ aˆ  1 E I (2-15) A
Similarly, given the probability of false alarm p f D , by Therefore, for the instrumented binary graph. can be used
(3-6) the process of target pixel clustering.
(2-12), the local threshold of the CFAR detection is:
TI Jˆ p1/f DDˆ  1 (2-16) D. Target Pixel Clustering
In the Windows 10 operating system, I use the Visual C++
Therefore, to the current sliding window of the test pixel, its
development platform, using the object-oriented program
strength is I 0 , do the following binary assumption H1 The method, the software development of surface defect detection
current test pixel is the target pixel H 0 current test pixels are system is completed. This program adopts modularization to
simplify the programming process on the basis of realizing
stray pixels. [8] The binary hypothesis criterion is
function. The main detection functions are described below:

The first step, we need a threshold function to deal with the

graph, according to the gray level image of the different gray

1160
values, the input image is layered, the image gray value
between the maximum gray level and the smallest gray area,
will be separated, through the local variables output.
The second step, through the National Brigade function, to
the threshold function extracted from the region, through the
feature parameters of the area, width, length and other
characteristics of the filter, the feature value is not in the region
of the part of the filter out, the result region through the
variable output. [10]
The third step, the image extraction function, extracts the
input image in the area of interest, the extracted part of the
image through the image reduction variable output.
In the fourth step, the edge processing of the extracted area
is done by the expansion function and the corrosion function,
the combination of expansion function and corrosion function
is mainly used to filter a disturbing point of image
The process is as follows:
GLODWLRQBFLUFOH
WKUHVKROG
HURVLRQBFLUFOH
WKUHVKROG
UHGXFHBGRPDLQ
GLODWLRQBFLUFOH
VHOHFWBVKDSH
HURVLRQBFLUFOH
Figure 6. Detection algorithm Flow.
III. ROBOTICS POLISHING SYSTEM
Faucet castings are irregular, so the use of six-axis robots to
do grinding work, which is conducive to artificial grinding
from the harsh, high-risk work environment release, and
increased production capacity. For different shapes of castings,
robots can be covered with the grinding work.
When the castings are visually examined, the castings are
required to pose a different posture, and the camera remains
relatively stationary under the desired posture. This is
conducive to visual imaging, to avoid the generation of image
virtualization. The robot can control the castings in the CCD
plane, and make it easy to gather the images of the castings in a
full range of visual systems.
Robots have multiple communication protocols, which can
be exchanged through real-time communication and visual
system, so as to realize intelligent robot. [11] When detecting,
the outer surface of the castings can be divided into 1 images,
i.e., a single image can completely inspect all the surfaces of
the castings. Based on 1 images, can determine each image
corresponds to the position, the number of workstations to
arrive 1, and the workstation number through the
communication protocol to the robot, the robot can be based
on the signal of the visual system to make corresponding to
each station signal movement and posture, and at the arrival of
the position feedback to the visual system to form a
closed-loop When the visual system discovers that the casting
surface has problems, by detecting the defect characteristics to
1161
determine whether the defect can be eliminated by twice
grinding, if it can be eliminated, the defective station
information to the robot to carry out twice grinding, the robot
can automatically generate a new secondary process grinding
path based on defective workstation. [12] If the defect is not
enough to be eliminated through secondary processing, the
robot places the castings in the defective product area. This is
more efficient than traditional manual testing.
H[WUDFWLQJDUWLIDFWV
SROLVKLQJE\VWDWLRQ
YLVXDOLQVSHFWLRQ
,VWKHUH
EOLVWHU"
<
VHFRQGDU\SROLVKLQJ
SROLVKWZLFH" EDVHGRQGHIHFWLYH
SRVLWLRQ
1 1
SROLVKLQJILQLVK
SXWLWLQTXDOLI\LQJ
DUHD
Figure 7. The workflow of visual system and robot.
IV. THE COMBINATION OF VISUAL INSPECTION AND
ROBOTICS
A conventional faucet polishing process is selected. To
validate the reliability of this system. The robot adopts the
KUKA KR20-3 model robot, its performance can satisfy the
demand of the tap grinding.
In the visual inspection system, a good photo scheme is very
important. The suitable camera, lens and light source can
guarantee the image collection. [13] The visual system selects
Basler Industrial camera acA1300-60gm, mesh
communication, frame rate 60 frames matched Omputar 16mm
megapixel industrial lens, model M1614-MP2. High reactive
power resolution can be useful for more accurate analysis of
captured images. [14] The visual light source adopts the white
arc led light source, the high angle way, the position change
does not affect the imaging effect as much as possible
increases the prominent object flaw characteristic. [15] For
tap-specific cylindrical surfaces and curved surfaces, a wide
range of high contrast can be collected for the desired image
data.

Figure 8. Light source and camera system
The visual system can adjust the grinding speed by the
frequency converter of the abrasive belt machine, and carry on
intelligent control to the secondary grinding. [16] The defect
size and the degree of SAG information obtained by the visual
system calculate the rotational frequency of the motor of the
sanding belt. This avoids the second grinding and excessive
cuts in the castings.
Figure 9. Frequency conversion Motor grinding system.
The faucet is divided into 9 stations, and the image acquired
at 9 stations can determine if the faucet is defective. [17]
Figure 10. Workpiece Grinding Area Division.
Figure X shows some experimental results. [18] The
combination of vision and robots can completely replace the
traditional artificial detection.
1162
Figure 11. Experimental process and results.
V. CONCLUSION
The defect detection of metal castings is reformed in the
casting processing industry, and it is important to the release of
labor force and the promotion of the production machine.
Through the simple and efficient visual detection algorithm,
the defects of the workpiece surface are detected quickly, thus
avoiding unnecessary rework. The validity of the method is
verified by experiments, and the results show that the method
is feasible, and the technology can be practically put into the
production of the factory.
ACKNOWLEDGMENT
The authors would like to thank Shenzhen Academy of
robotics, Shenzhen, China for supporting the research.
REFERENCES
[1] Batchelor B, Hill D, Hodgson H. Automated visual inspection[J]. 1985.
[2] Vernon D. Machine vision-Automated visual inspection and robot
vision[J]. NASA STI/Recon Technical Report A, 1991, 92.
[3] Newman T S, Jain A K. A survey of automated visual inspection[J].
Computer vision and image understanding, 1995, 61(2): 231-262.
[4] Nazarov A, Thierry D. Application of Volta potential mapping to
determine metal surface defects[J]. Electrochimica Acta, 2007, 52(27):
7689-7696.
[5] Vedaldi A, Fulkerson B. VLFeat: An open and portable library of
computer vision algorithms[C]//Proceedings of the 18th ACM
international conference on Multimedia. ACM, 2010: 1469-1472.
[6] Szeliski R. Computer vision: algorithms and applications[M]. Springer
Science & Business Media, 2010.
[7] Steger C, Ulrich M, Wiedemann C. Machine vision algorithms and
applications[J]. Illumination, 2008, 5(2.1): 1.
[8] Milgram P, Zhai S, Drascic D, et al. Applications of augmented reality
for human-robot communication[C]//Intelligent Robots and Systems'
93, IROS'93. Proceedings of the 1993 IEEE/RSJ International
Conference on. IEEE, 1993, 3: 1467-1472.
[9] Wang J. On sign-board based inter-robot communication in distributed
robotic systems[C]//Robotics and Automation, 1994. Proceedings.,
1994 IEEE International Conference on. IEEE, 1994: 1045-1050.
[10] Bachelder I A, Waxman A M. Mobile robot visual mapping and
localization: A view-based neurocomputational architecture that
emulates hippocampal place learning[J]. Neural networks, 1994, 7(6):
1083-1099.
[11] LaValle S M. Rapidly-exploring random trees: A new tool for path
planning[J]. 1998.
[12] Ge S S, Cui Y J. New potential functions for mobile robot path
planning[J]. IEEE Transactions on robotics and automation, 2000,
16(5): 615-620.
[13] Li J, Yang M M. Metal polishing composition: U.S. Patent
6,274,063[P]. 2001-8-14.
[14] Rea W V. Metal polishing composition and process: U.S. Patent
4,475,981[P]. 1984-10-9.
[15] Hanley J A, McNeil B J. The meaning and use of the area under a
receiver operating characteristic (ROC) curve[J]. Radiology, 1982,
143(1): 29-36.
[16] Burgard W, Moors M, Stachniss C, et al. Coordinated multi-robot
exploration[J]. IEEE Transactions on robotics, 2005, 21(3): 376-386.
[17] Li Z, Hsu P, Sastry S. Grasping and coordinated manipulation by a
multifingered robot hand[J]. The International Journal of Robotics
Research, 1989, 8(4): 33-50.
[18] Takeuchi Y, Asakawa N, Ge D. Automation of polishing work by an
industrial robot: System of polishing robot[J]. JSME international
journal. Ser. C, Dynamics, control, robotics, design and manufacturing,
1993, 36(4): 556-561.