SS symmetry

Article
Blind Deblurring of Saturated Images Based on
Optimization and Deep Learning for Dynamic Visual
Inspection on the Assembly Line
Bodi Wang 1 , Guixiong Liu 1, * and Junfang Wu 2,3
1 School of Mechanical and Automotive Engineering, South China University of Technology,
Guangzhou 510640, China; merobbiewong@mail.scut.edu.cn
2 School of Physics, South China University of Technology, Guangzhou 510640, China; wujf@scut.edu.cn
3 Guangdong Key Laboratory of Modern Geometry and Mechanics Metrology Technology,
Guangzhou 510405, China
* Correspondence: megxliu@scut.edu.cn




Received: 14 April 2019; Accepted: 13 May 2019; Published: 16 May 2019


Abstract: Image deblurring can improve visual quality and mitigates motion blur for dynamic visual
inspection. We propose a method to deblur saturated images for dynamic visual inspection by
applying blur kernel estimation and deconvolution modeling. The blur kernel is estimated in a
transform domain, whereas the deconvolution model is decoupled into deblurring and denoising
stages via variable splitting. Deblurring predicts the mask specifying saturated pixels, which are then
discarded, and denoising is learned via the fast and flexible denoising network (FFDNet) convolutional
neural network (CNN) at a wide range of noise levels. Hence, the proposed deconvolution model
provides the benefits of both model optimization and deep learning. Experiments demonstrate that
the proposed method suitably restores visual quality and outperforms existing approaches with good
score improvements.

Keywords: visual inspection; image deblurring; blur kernel; deconvolution; deep learning

1. Introduction
Visual inspection has been extensively investigated and is a well-established approach for a myriad
of engineering applications. In dynamic scenes, visual inspection helps to improve the efficiency
of monitoring tasks, but low visual quality may undermine its effectiveness [1–4]. For instance,
in manufacturing environments, the object under inspection moves along the assembly line during
exposure time, resulting in motion blur. Consequently, latent images are degraded by effects such as
smears on sharp edges and loss of semantic information. Hardware-based approaches can be applied
to compensate for visual quality. However, such methods are costly and require manual adjustments,
and the devices often do not provide setup flexibility [5]. Alternatively, image deblurring can be
employed, as it reverses degradation and restores the latent image from a blurred one.
Image deblurring can be either blind or non-blind, where the former type operates with unknown
kernel. Therefore, blind image deblurring is equivalent to inferring two functions from their product,
being an ill-posed problem in general. Recently, blind image deblurring methods have relied on
the image prior-based regularization of objective functions to guide deconvolution and alternately
estimate the blur kernel and intermediate latent images [6,7]. Once the blur kernel is obtained, blind
image deblurring reduces to its non-blind counterpart. Unlike conventional blind image deblurring,
the challenges of deblurring for dynamic visual inspection are threefold:

Symmetry 2019, 11, 678; doi:10.3390/sym11050678 www.mdpi.com/journal/symmetry

Symmetry 2019, 11, 678 2 of 18

1. Dynamic visual inspection inherently poses a blind image deblurring problem that should include
blur kernel estimation and deconvolution modeling. Still, linear motion of inspected objects in
structured environments (e.g., assembly lines) allows to parameterize the blur kernel.
2. Conventional model optimization relies on a handcrafted prior, which is often sophisticated and
non-convex. Moreover, the tradeoff between computation time and recovered visual quality
should be considered, as image deblurring for dynamic visual inspection requires fast optimization.
3. In the case considered for this study, the sheet metal of objects for dynamic visual inspection
might intensely reflect light, resulting in a blurred image with saturated pixels, which affect the
goodness of fit in data fidelity. Consequently, deblurring based on linear degradation models
might fail due to the appearance of severe ringing artifacts.

We focus on image deblurring to improve the visual quality and mitigate motion blur for dynamic
visual inspection. Degradation of dynamic visual inspection is mainly attributed to linear motion blur,
whose kernel exhibits a sinc-shaped ripple in the transform domain and hence can be characterized by
the blur angle and blur length [8]. Considering a fixed heading of the inspected object with respect to
the camera, the blur angle can be considered as a known parameter, and the blur length is estimated
according to extracted features in the transform domain. Then, variable splitting can be applied to the
deconvolution model for its decoupling into data fidelity and regularization terms. To maximize the
restored visual quality, the data fidelity term is associated with the mask specifying inliers and outliers
via variational expectation–maximization (VEM) [9], and the regularization term is learned via the fast
and flexible denoising convolutional neural network (CNN), or fast and flexible denoising network
(FFDNet) [10], at a wide range of noise levels. Different from measures that consider an original clear
image as ground truth, we evaluate the deblurring performance from the structural similarity index
(SSIM) and feature similarity index (FSIM) between the blurred and re-blurred images, as no ground
truth is available in dynamic visual inspection.
The remainder of this paper is organized as follows. Section 2 presents a survey of related work
on blur kernel estimation, deconvolution modeling, and outlier handling. The degradation model and
blur kernel model for dynamic visual inspection are presented in Section 3. In Section 4, we introduce
the proposed blind deblurring method of saturated images for dynamic visual inspection and report its
validation both quantitatively and qualitatively in Section 5. Finally, we draw conclusions in Section 6.

2. Related Work
As we split blind image deblurring into blur kernel estimation and deconvolution modeling,
we performed a review of these two topics. Given the need for high restored visual quality during
dynamic visual inspection, we also reviewed methods for handling outliers.

2.1. Blur Kernel Estimation

Blind image deblurring depends on blur kernel estimation. As both the blur kernel and
intermediate latent image are unknown, an image prior is utilized to constrain the solution space
and enable blind image deblurring. The image prior describes the nature of an image and can adopt
approaches such as heavy-tailed distribution [5,11] and L0 regularization [12,13] or channel prior [14,15].
These methods model the blur kernel as a projection transform and estimate it in a coarse-to-fine
process. This process, however, is not suitable for the linear motion blur appearing during dynamic
visual inspection [16]. In fact, linear motion blur in the transform domain is identifiable without an
image prior and determined by the blur angle and blur length. Thus, blur kernel estimation in linear
motion relies on specific features from the transform domain [8,17].
Dash and Majhi [18] found the blur angle according to the frequency response of a blurred image
filtered by a 2D Gabor mask, and a radial basis function network retrieved the relationship between
the Fourier coefficients and blur length. Oliveira et al. [16] fit a third-order polynomial to approximate
the modified Radon transform and assumed that the blur angle maximizes its mean squared error.
Symmetry 2019, 11, 678 3 of 18

Then, they computed the Radon transform along the motion direction to estimate the blur length.
Lu et al. [6] found that a restored image has its most sparse representation at the true blur angle,
and in their approach, they captured sinc-shaped components in the Fourier domain to measure the
blur length. Yan and Shao [19] classified the blur type on the discriminative feature space of a deep
neural network and used a general regression neural network to estimate the blur kernel parameters.
Lu et al. [20] mined the linear motion blur kernel features in CNNs and mapped them into the blur
angle and blur length using a support vector machine.
To collapse 2D data into one dimension (i.e., to align the motion direction with the horizon), several
methods rotate the blurred image, its derivative, or its axis using the estimated blur angle [6,8,18].
However, rotation-induced interpolation causes out-of-range pixels, thus hindering the subsequent
blur length estimation [21]. In addition, rotation might lead to error propagation given the dependence
between blur length and blur angle during estimation [16,17].

2.2. Deconvolution Modeling

Deconvolution relies on the estimated blur kernel for performing non-blind image deblurring.
The Richardson–Lucy (RL) [22] and Wiener [23] deconvolutions are simple and effective methods
but suffer from oversmoothed edges and ringing artifacts. Model optimization methods rely
on image prior that is mainly used to characterize local smoothness (e.g., total variation (TV)
norm [24,25], hyper-Laplacian (HL) prior [26,27]) and nonlocal self-similarity (e.g., iterative
decoupled deblurring-block matching and 3D filtering (IDD-BM3D) [28], nonlocally centralized sparse
representation (NCSR) [29], multi scale-weighted nuclear norm minimization (MS-WNNM) [30]) of
images. However, optimization is often time consuming, and the solution can reach a local minimum
for image prior-based regularization, which may not be sufficiently strong [31].
Recently, deep learning has been applied to low-level vision tasks [32–34]. Xu et al. [35] used
the singular value decomposition of a 2D pseudoinverse kernel to initialize the weights of a network,
which should be retrained for different blur kernels. To handle different blur kernels in a unified
framework, Ren et al. [36] used a neural network motivated by the generalized low-rank approximation
of a set of blur kernels, and similarly exploited the separable structure of blur kernels to initialize the
network instances. Zhang et al. [37] integrated model optimization and deep learning via half quadratic
splitting and then applied a CNN denoiser to tackle the problem of denoising. Zhang et al. [38] trained
a fully CNN in the gradient domain to learn the image prior and employed network cascading with a
deconvolution module to iteratively deblur images. In [37,38], the handcrafted prior was avoided,
and a flexible approach to handling different vision problems was proposed. The authors addressed
deblurring by learning a direct mapping from the degraded image to the ground truth, but the learned
prior was used for denoising, whereas the blur kernel went almost unused in the network.
In general, deep learning retrieves a mapping function obtained from supervised learning and
requires paired data (e.g., blurred image and its ground truth) for training. However, the ground truth
is often unavailable in practice, thus limiting its applicability to blind natural image deblurring [39].
Overall, optimization and deep learning can be seen as complementary approaches. In fact, model
optimization can be applied to a variety of vision tasks without requiring ground truth, whereas deep
learning omits handcrafted priors and provides fast testing speed for deblurring images [40].

2.3. Outlier Handling

The complicated digital imaging pipeline influences the final visual quality, which is subject
to external and internal factors, including illumination, exposure, camera lens, and sensors.
Therefore, outliers produced by saturated pixels, non-Gaussian noise, camera nonlinear response,
among others, tend to further degrade a blurred image, producing considerable ringing artifacts during
deconvolution [41]. For instance, the sheet metal of an object under inspection can be very reflective
and saturate pixels. Thus, suitable image deblurring is essential for dynamic visual inspection of these
types of objects.
Symmetry 2019, 11, x FOR PEER REVIEW 4 of 18
Symmetry 2019, 11, 678 4 of 18
Whyte et al. [42] embedded a nonlinear degradation model into the RL deconvolution to discard
saturated pixels by assigning low weights to them. Cho et al. [9] computed the mask specifying inliers
Whyte et al. [42] embedded a nonlinear degradation model into the RL deconvolution to discard
and outliers during expectation and restored the image using hyper-Laplacian prior during
saturated pixels by assigning low weights to them. Cho et al. [9] computed the mask specifying
maximization in the expectation–maximization algorithm. Hu et al. [43] incorporated the mask
inliers and outliers during expectation and restored the image using hyper-Laplacian prior during
proposed in [9] to modify the RL deconvolution and combined the approaches in [42] and [9].
maximization in the expectation–maximization algorithm. Hu et al. [43] incorporated the mask
Moreover, the approaches in [35,36] added three convolutional layers at the end of their respective
proposed in [9] to modify the RL deconvolution and combined the approaches in [42] and [9].
networks to reject outliers.
Moreover, the approaches in [35,36] added three convolutional layers at the end of their respective
For dynamic visual inspection, we propose a novel method for blind image deblurring
networks to reject outliers.
comprising blur kernel estimation and deconvolution modeling. Features in the transform domain
For dynamic visual inspection, we propose a novel method for blind image deblurring comprising
were first selected to effectively estimate the blur kernel parameters assuming linear motion, thus
blur kernel estimation and deconvolution modeling. Features in the transform domain were first
avoiding rotation and error propagation. Then, the CNN-learned prior was embedded into the
selected to effectively estimate the blur kernel parameters assuming linear motion, thus avoiding
conventional objective function, where deconvolution was split into deblurring and denoising stages.
rotation and error propagation. Then, the CNN-learned prior was embedded into the conventional
Saturated pixels from metallic reflection were handled during deblurring. Instead of a handcrafted
objective function, where deconvolution was split into deblurring and denoising stages. Saturated pixels
prior, we used a prior learned via CNN for denoising at a wide range of noise levels. Experimental
from metallic reflection were handled during deblurring. Instead of a handcrafted prior, we used a
results demonstrate that the proposed method can improve dynamic visual inspection and retrieves
prior learned via CNN for denoising at a wide range of noise levels. Experimental results demonstrate
higher SSIM and FSIM than existing approaches.
that the proposed method can improve dynamic visual inspection and retrieves higher SSIM and FSIM
than
3. existingof
Overview approaches.
Deblurring for Dynamic Visual Inspection
3. Overview of Deblurring for Dynamic Visual Inspection
3.1. Dynamic Visual Inspection System
3.1. Dynamic
Figure 1 Visual Inspection
illustrates System visual inspection system considered in this study. Four LED
the dynamic
stripsFigure
are located at the the
1 illustrates sides of the object
dynamic visualunder inspection
inspection systemand intersectin
considered with
thisthe axisFour
study. of the imaging
LED strips
plane. The object
are located underofinspection
at the sides the object is surrounded
under inspectionby and
threeintersect
cameraswith
installed at of
the axis thethe
sides and on
imaging top
plane.
of the assembly line and pointing to the object. The camera lens and light sources
The object under inspection is surrounded by three cameras installed at the sides and on top of the are intended to
guarantee highand
assembly line visual quality.
pointing to In
thethe imaging
object. The pipeline for dynamic
camera lens and lightvisual inspection,
sources the object
are intended moves
to guarantee
with
high visual quality. In the imaging pipeline for dynamic visual inspection, the object movesedges.
respect to the cameras, leading to motion blur, especially appearing as smears on sharp with
In turn,to
respect blurred imagesleading
the cameras, may lead to unreliable
to motion inspection
blur, especially results. Thus,
appearing we aimed
as smears to develop
on sharp edges. Inblind
turn,
image
blurred deblurring
images may for dynamic visual inspection
lead to unreliable inspectionto restore
results.a suitable
Thus, welatent image
aimed and mitigate
to develop blindmotion
image
blur.
deblurring for dynamic visual inspection to restore a suitable latent image and mitigate motion blur.

(a) (b)

Figure
Figure 1. Diagram
Diagram of
of dynamic
dynamic visual
visual inspection
inspection system:
system: (a)
(a) top
top and
and (b)
(b) front
front view.
view.

3.2. Blur
3.2. Blur Mitigation
Mitigation
Based on
Based on aa spatially
spatially invariant
invariant blur
blur kernel,
kernel, we
we formulated
formulated the
the blur
blur model
model as
as the
the latent
latent image
image
convolved with
convolved withthe
theblur kernel
blur plus
kernel additive
plus whitewhite
additive Gaussian noise. noise.
Gaussian As convolution is a linear
As convolution is operator,
a linear
we express blurring as follows:
operator, we express blurring as follows:
b = H f + η, (1)
Symmetry 2019, 11, 678 5 of 18

where b, f , and η are vectors representing the blurred image, latent image, and noise, respectively,
and matrix H represents the blur kernel.
Blind image deblurring restores the latent image from its blurred version provided an accurate
blur kernel estimate is available. The distance between the latent and restored images reflects the
deblurring performance.
During the exposure time, the object under inspection moves along the assembly line, resulting in
linear motion blur, which can be modeled as follows:
p y
1
x2 + y2 ≤ L2 , x = tan θ
(
H(x, y) = L , (2)
0 otherwise

where x and y denote the pixel indices in the horizontal and vertical directions, respectively.
Hence, the linear-motion blur kernel is described by two parameters, namely, blur angle θ and
blur length L. The blur angle indicates the motion direction of the object with respect to the camera,
and the blur length characterizes the distance that the object moves over the camera in pixels (px).
To guarantee high image quality for dynamic visual inspection, we estimated the blur kernel in
the transform domain of the blurred image and then deblurred the image considering the presence of
outliers based on the estimated kernel.

4. Proposed Method
In this section, we detail our method of blind image deblurring for dynamic visual inspection,
including blur kernel estimation and deconvolution modeling. The blur kernel is estimated via features
selected from the transform domain, and the deconvolution iteratively performs deblurring and
denoising, where the former aims to reject outliers, and the latter is based on a CNN.

4.1. Blur Kernel Estimation

In our system (Figure 1), the heading of the object with respect to the cameras is fixed, and hence
the blur angle is assumed to be known in advance (either 0◦ or 90◦ ). Thus, the blur length is the
only parameter to estimate for obtaining the blur kernel. First, we preprocessed the blurred image
to select sharp edges and then estimated the blur length based on feature points from the resulting
autocorrelation function.

4.1.1. Sharp Edge Selection

We performed preprocessing to enhance sharp edge extraction and the subsequent blur kernel
estimation. To this end, we filtered the blurred image with the Sobel detector to efficiently determine
weighted edges for mitigating smear:
 
 1 2 1 
 
∇b = b ⊗  0 0 0 ,
 (3)

−1 −2 −1


where ∇b is the preprocessed image, and ⊗ denotes the convolution operator. The Sobel detector and
its transpose detect vertical and horizontal edges, respectively.
Figure 2 shows examples of preprocessing for dynamic visual inspection, where the sharp edges
in the blurred images are enhanced. The selected sharp edge has a scale corresponding to the blur
length we aimed to estimate.
Symmetry 2019, 11, 678 6 of 18
Symmetry 2019, 11, x FOR PEER REVIEW 6 of 18

(a) (b)
Figure 2. Examples
Figure visual inspection.
Examples of preprocessing for dynamic visual inspection. The examples in (a) and (b) show
the blurred image (left) and the result
result of
of preprocessing
preprocessing (right).
(right).

4.1.2. Blur Length

4.1.2. Blur Length Estimation
Estimation
Once
Once aa sharp
sharp edge
edge was
was selected,
selected, we
we estimated
estimated thethe blur
blur length
length based
based on
on the
the autocorrelation
autocorrelation
function of the preprocessed blurred image as follows:
function of the preprocessed blurred image as follows:
Φ= =  −−11
F F T −1 {
F F (T∇(b∇b
) ), , }
n o
Φ (4)
(4)

where FFT and FFT–1 –1

denote the Fourier transform and its inverse, respectively, and Φ is the
where FFT and FFT−1 denote the Fourier transform and its inverse, respectively, and Φ is the
autocorrelation function with the same size as the blurred image. To visualize function Φ , we
autocorrelation function with the same size as the blurred image. To visualize function Φ, we plotted
plotted its columns according to the row in a plane, where the x-axis scale ranged from 1 to the
its columns according to the row in a plane, where the x-axis scale ranged from 1 to the number of
number of rows in Φ . This visualization results in conjugate pairs of valleys that retrieve sinc-
rows in Φ. This visualization results in conjugate pairs of valleys that retrieve sinc-shaped curves in
shaped curves in the autocorrelation function, where the lowest pair corresponds to the blur length.
the autocorrelation function, where the lowest pair corresponds to the blur length.
For instance, the images with sharp edges in Figure 2 retrieve the visualization of the
For instance, the images with sharp edges in Figure 2 retrieve the visualization of the autocorrelation
autocorrelation function shown in Figure 3. A conjugate pair of valleys is clear in each plot, whereas
function shown in Figure 3. A conjugate pair of valleys is clear in each plot, whereas other characteristics
other characteristics might be related to noise and outliers.
might be related to noise and outliers.

(a) (b)
3. Autocorrelation
Figure 3. Autocorrelationfunction
functionofofpreprocessed
preprocessed images
images in in Figure
Figure 2. The
2. The functions
functions plotted
plotted in (a)inand
(a)
(b)
andcorrespond to thetorespective
(b) correspond images
the respective in Figure
images 2. 2.
in Figure

Given
Given the known blur
the known angle of
blur angle 90◦ for
of 90° for dynamic
dynamic visual
visual inspection,
inspection, blur
blur length
length estimation
estimation is
is based
based
on 1D data and does not need additional operations. Specifically, let V11 ( x11 , z11 ) and V22 ( x22 , z22 ) be
on 1D data and does not need additional operations. Specifically, let V1 (x1 , z1 ) and V2 (x2 , z2 ) be the
two lowest conjugate valleys, where x, z respectively denote the horizontal pixel indices and the
the two lowest conjugate valleys, where x , z respectively denote the horizontal pixel indices and
magnitude of its autocorrelation function. The half interval between these valleys corresponds to the
the magnitude
blur of its autocorrelation function. The half interval between these valleys corresponds to
length as follows:
the blur length as follows: |x − x2 |
L= 1 . (5)
x11 − 2x 22
L= . (5)
2
Symmetry 2019, 11, 678 7 of 18

4.2. Deconvolution Modeling

In principle, deconvolution can reverse blurred image degradation after blur kernel estimation
through non-blind deblurring. Conventional optimization relies on image prior-based regularization,
but either very strong or weak regularization can undermine the deblurring performance. Furthermore,
such optimization is often time consuming because of the sophisticated and non-convex regularization.
In contrast, deep learning can overcome these problems as it does not require handcrafted priors and
provides fast testing speed.

4.2.1. Deconvolution Model Splitting

Conventional optimization is formulated via an objective function that includes data fidelity
and regularization terms. The data fidelity term is modeled based on the noise distribution, and the
regularization term constrains the solution space. From a Bayesian perspective, the best restored image
is that optimizing the following objective function:

λ
fˆ = argmin kb − H f k22 + Θ( f ), (6)
f 2

where the first term is the data fidelity weighted by λ and assuming Gaussian noise, and Θ( f )
represents the image prior-based regularization.
Regularization is often non-convex and hence hinders optimization. To relax the model with
non-convex regularization, we adopted variable splitting to decouple the objective function into
individual subproblems. Among several variable splitting approaches, we used half-quadratic
splitting [44] given its faster convergence than methods such as alternating direction method of
multipliers (ADMM) [45] and primal-dual (PD) [46]. Hence, the deconvolution model is split
as follows:
λ
argmin kb − H f k22 + Θ(l) + βk f − lk22 , (7)
f ,l 2

where l is a slack variable approximately equal to f and β is a weight. Equation (7) can be alternately
solved via the following iterative process:

l̂t+1 = argminβk ft − lk22 + Θ(l), (8)

l

λ
fˆt+1 = argmin kb − H f k22 + βk f − lt k22 . (9)
f 2

Equation (8) represents a deconvolution model, whose degradation is the identity matrix,
and corresponds to Gaussian denoising. In addition, Equation (9) is convex and can be efficiently
solved in the frequency domain of the Fourier transform. Therefore, half-quadratic splitting turns the
complicated and non-convex deconvolution model into easier problems of denoising and deblurring
given by Equations (8) and (9), respectively.

4.2.2. Outlier Handling

Deblurring in Equation (9) can be efficiently solved due to the convex L2 norm, but outliers
induced by the digital imaging pipeline can affect the goodness of fit in the data fidelity term and
undermine the deblurring performance. Moreover, the sheet metal of the object under inspection
considered in this study can intensely reflect light, thus saturating pixels in the images. Figure 4 shows
a restoration example using the RL deconvolution [22] for dynamic visual inspection. The restored
image contains severe ringing artifacts, especially at its salient edges, because saturated pixels behave
as outliers, undermining the performance of approaches based on linear degradation [41].
Symmetry 2019, 11, 678 8 of 18
Symmetry 2019, 11, x FOR PEER REVIEW 8 of 18

(a) (b)

Figure 4. Deblurring
Figure 4. Deblurringwith with resulting
resulting ringing
ringing artifacts.
artifacts. (a) Blurred
(a) Blurred image
image from from visual
dynamic dynamic visual
inspection
inspection
system andsystem
(b) its and (b) its restoration
restoration via the Richardson–Lucy
via the Richardson–Lucy (RL) deconvolution.
(RL) deconvolution.

To handle
To handle saturated
saturated pixels,
pixels,wewefirst
firstclassified
classifiedimage
imageintensities into
intensities saturated
into and
saturated unsaturated
and by
unsaturated
using
by the binary
using mask in
the binary [9], where
mask in [9],saturated pixels were
where saturated rejected
pixels via rejected
were low weights
via during deconvolution.
low weights during
To compute its expectation,
deconvolution. To compute itstheexpectation,
likelihood term associatedterm
the likelihood mask m is with
withassociated formulated
mask masisfollows:
formulated
as follows: 
N (b|Hl, σ ) if m = 1


p(b|m, H, l ) =  , (10)
 
p( b m , H , l ) = 

( )
C b H l , σ ififmm==1 0
, (10)
 C if m = 0
where N is a Gaussian distribution with standard deviation σ and C is the constant defined as the
inverse 
where is adynamic
of the Gaussian distribution
range. Then, weightwith standard
Wt in thedeviationmask is givenσ and C is
by the the constant defined as
following:
the inverse of the dynamic range. Then, weight W t in the mask is given by the following:
E[mt ]
Wt = , (11)
E  m2σt 2
Wt = , (11)
 N (b|Hl ,σ2)σp 2
t u
 N (b|Hlt ,σ )pu +Cps if Hlt ∈ DR


0 ( b H lt , σ ) p u
E[mt] =  , (12)
 otherwise

if H lt ∈ D R
E  m t  =   ( b H lt , σ ) p u + C p s , (12)
where pu and ps are the probabilities  of a pixel being unsaturated and saturated, respectively, and t is
 0 otherw ise
the iteration indices.
where pu and
Finally, ps are γthe
by letting λ/2β and including
= probabilities of a pixel Wbeing
t into Equation
unsaturated (9),and
we obtain the following:
saturated, respectively, and
t is the iteration indices.
fˆ = argmin λW t
2 kb − H f k22 + βk f − lt k22
Finally, by letting γ = λ 2 βt+1and including
f W t into Equation (9), we obtain the following:
 −1   . (13)
= γWt HT H + I γWt HT b + lt
λWt 2 2
fˆt + 1 = arg min b − H f 2 + β f − lt 2
Hence, E[mt ] predicts the mask specifyingf 2unsaturated and saturated . (13)
pixels in the Bayesian setting.
( ) ( )
−1
Saturated pixels are not apt for the=linear T
H+I
γWt Hdegradation γWmodel,
t
H T
b + thereby
lt
affecting the goodness of fit in
the data fidelity term. By associating the data fidelity term to the mask, saturated pixels are rejected
Hence, E  mt  predicts
during deconvolution, andthe mask can
outliers specifying unsaturated
be handled and saturated pixels in the Bayesian
during deblurring.
setting. Saturated pixels are not apt for the linear degradation model, thereby affecting the goodness
4.2.3.
of FFDNet
fit in the data Denoising
fidelity term. By associating the data fidelity term to the mask, saturated pixels are
rejected during deconvolution,
As Equation (8) correspondsandtooutliers
Gaussiancan be handled
denoising, weduring deblurring.
can rewrite it as follows:

4.2.3. FFDNet Denoising  s −2

1  1 
l̂t+1 = argmin   k ft − lk22 + Θ(l). (14)
As Equation (8) corresponds to Gaussian
l 2 denoising,
2β  we can rewrite it as follows:
−2
1  1Gaussian
 level 1/2β, and Θ(l) denotes the
p
Hence, denoising processes
ˆl =image ft with noise
ft − l 2 + Θ ( l ) .
2
arg min   (14)
t +1
image prior. There are two kinds of image
l 2  2 βthat
priors  are
 widely used in deconvolution—one based
on local smoothness and the other characterizing nonlocal self-similarity. These priors can retrieve
Hence, denoising processes image ft with Gaussian noise level 1 2β , and Θ ( l ) denotes the
image prior. There are two kinds of image priors that are widely used in deconvolution—one based
Symmetry 2019, 11, 678 9 of 18

suitable results, but they might not be strong enough for complex structures and textures. Furthermore,
Symmetry 2019, 11, x FOR PEER REVIEW 9 of 18
regardless of the time-consuming optimization, these priors must deal with excessive smoothing and
block-like artifacts.
on local smoothness and the other characterizing nonlocal self-similarity. These priors can retrieve
Recently, CNNs have
suitable results, shown
but they promising
might not be results
strong to efficiently
enough learn image
for complex priors and
structures giventextures.
the parallel
Furthermore,
computation regardless
ability of theprocessing
of graphics time-consuming
units,optimization,
and refinedthese priors must
techniques deal and
to train with design
excessiveCNNs
smoothing and block-like artifacts.
facilitate their application for image restoration [38]. Thus, we use a CNN-based prior for denoising.
Many CNN Recently, CNNs
priors are have with
trained shown promising
paired data results to efficiently
at a fixed learn
noise level, thusimage priors
lacking given theFor
flexibility. parallel
dynamic
computation ability of graphics processing units, and refined techniques to train and design CNNs
visual inspection, the CNN prior should be determined at different noise levels and even handle the
facilitate their application for image restoration [38]. Thus, we use a CNN-based prior for denoising.
possibility of spatially variant noise. FFDNet can model a noise-level map as input to match varying
Many CNN priors are trained with paired data at a fixed noise level, thus lacking flexibility. For
noisedynamic
levels, thus generalizing
visual single
inspection, the CNN CNN priors
prior should[10]. Table 1 listsatthe
be determined settings
different of FFDNet
noise levels andused
even in this
study.handle
The CNNs of grayscale
the possibility and color
of spatially images
variant have
noise. 15 andcan
FFDNet 12 convolutional
model a noise-levellayers,map respectively.
as input to Each
layer match
basically consists
varying noiseoflevels,
convolutions (Conv), rectified
thus generalizing single CNNlinear units
priors (ReLU),
[10]. Table and batch
1 lists normalization
the settings of
(BN).FFDNet
The grayscale
used in thisandstudy.
color The
images
CNNsemploy 64 and
of grayscale 96color
and convolution
images have kernels,
15 and respectively, and zero
12 convolutional
layers,
padding respectively.
maintains fixedEach
thelayer
size basically
of featureconsists
maps.of convolutions (Conv), rectified linear units (ReLU),
and batch normalization (BN). The grayscale and color images employ 64 and 96 convolution kernels,
respectively,
Tableand
1. zero
Fast padding maintains
and flexible fixed
denoising the size(FFDNet)
network of feature maps. used in this study.
settings
Table 1. Fast and flexible denoising network (FFDNet) settings used in this study.
Grayscale Image Color Image
Grayscale Image
15 convolutional layers Color Image layers
12 convolutional
15 convolutional layers
Downsampling 12 convolutional
+ noise-level map layers
(Conv 3 × 3 + ReLU) × Downsampling
1 layer + noise-level
(Conv 3 × 3 + ReLU) × 1 layer
map
(Conv 3 × 3 + BN
(Conv 3 ×+3 ReLU)
+ ReLU) × ×131layers
layer (Conv33×× 33 +BN
(Conv + ReLU)
+ ReLU) × 1 layer
× 10 layers
(Conv 3×3×
(Conv 3) ×+1ReLU)
+ BN layer × 13 layers (Conv 3 × 3(Conv × 3) × ×1 10
+BN +3ReLU) layer
layers
(Conv 3 × 3) × 1 layer Upscaling (Conv 3 × 3) × 1 layer
Upscaling
Figure 5 illustrates the proposed blind
Figure 5 illustrates the proposed blind deblurring deblurring of saturated
of saturated images images for dynamic
for dynamic visual visual
inspection.
inspection. After blur kernel estimation, a predicted mask was employed during
After blur kernel estimation, a predicted mask was employed during deblurring to discard saturated deblurring to
discard saturated pixels and prevent ringing artifacts. Instead of specifying a handcrafted prior, we
pixels and prevent ringing artifacts. Instead of specifying a handcrafted prior, we trained the FFDNet
trained the FFDNet at a wide range of noise levels, and the image prior was parameterized as weights
at a wide range of noise levels, and the image prior was parameterized as weights in the network for
in the network for denoising. Alternate deblurring and denoising were iteratively applied to fully
denoising.
deblur Alternate
the image. deblurring and denoising were iteratively applied to fully deblur the image.

Figure
Figure 5. Diagram
5. Diagram of proposed
of proposed blinddeblurring
blind deblurring of
of saturated
saturatedimages
imagesforfor
dynamic
dynamicvisual inspection.
visual inspection.
Deblurring
Deblurring handles
handles saturated
saturated pixels,
pixels, and denoising
and denoising proceeds
proceeds using
using imageimage
priorprior obtained
obtained fromfrom
FFDNet.
FFDNet. Conv, convolution; ReLU, rectified linear unit; BN, batch normalization.
Conv, convolution; ReLU, rectified linear unit; BN, batch normalization.
Symmetry 2019, 11, x FOR PEER REVIEW 10 of 18

Symmetry 2019, 11, 678 10 of 18

Note that the proposed method can be extended to different vision tasks by setting matrix H .
For instance, besides deblurring or denoising by setting H as the blur kernel or identity matrix,
respectively,
Note thatsuper-resolution can be achieved
the proposed method for H being
can be extended a subsampling
to different matrix
vision tasks [47]. matrix H.
by setting
For instance, besides deblurring or denoising by setting H as the blur kernel or identity matrix,
5. Experimental
respectively, Evaluation can be achieved for H being a subsampling matrix [47].
super-resolution
We conducted experiments to validate the proposed blind deblurring method using the machine
5. Experimental Evaluation
vision-based assembly line inspection system from Lite-On (Taipei, Taiwan) considering the objects
underWeinspection
conducted to be computer cases
experiments Lenovothe
to validate T280 G3, ADLINK
proposed blindEOS-1300, and
deblurring Neousys
method NUVO-
using the
2400. Wevision-based
machine evaluated the proposed
assembly method through
line inspection experiments
system from on blurTaiwan)
Lite-On (Taipei, kernel considering
estimation and
the
deconvolution
objects modeling.to be computer cases Lenovo T280 G3, ADLINK EOS-1300, and Neousys
under inspection
NUVO-2400. We evaluated the proposed method through experiments on blur kernel estimation and
5.1. Parameter Settings
deconvolution modeling.
We set standard
5.1. Parameter Settings deviation
σ in Equations (10)–(12) to 5/255 and pu , ps to 0.9 and 0.1,
respectively. Only λ and β were the unspecified parameters in Equation (13). Note that λ
We set standard deviation σ in Equations (10)–(12) to 5/255 and pu , ps to 0.9 and 0.1, respectively.
depends on the noise level of the blurred image and remains fixed during deconvolution, whereas
Only λ and β were the unspecified parameters in Equation (13). Note that λ depends on the noise level of
β regularizes the FFDNet-learned prior, which can be implicitly determined by its noise level. The
the blurred image and remains fixed during deconvolution, whereas β regularizes the FFDNet-learned
p
noise which
prior, level ofcan
thebeprior, 1 2β
implicitly , exponentially
determined by its decays from 25/255
noise level. to the
The noise interval
level of the [1/255,
prior, 10/255]
1/2β,
exponentially decays from 25/255 to the interval [1/255, 10/255] according to the number
according to the number of iterations set to 30. Based on the original training set of FFDNet, we of iterations
set to 30. Based
randomly on the
selected 10%original training and
of the samples set ofincreased
FFDNet,pixel
we randomly selected
intensities by 30%10% of the samples
to simulate and
saturation.
increased
To prevent pixel intensities
adverse byon
effects 30% to simulate
the L2 normsaturation.
by outliers,Towe
prevent adverse
modified the effects on the Lof
loss function 2 norm by
FFDNet
outliers, we modified the loss function of FFDNet using the L1 norm as follows:
using the L1 norm as follows:
N
1X 0 1
L(ϕ) =1 N kF (x , M ; ϕ) − x k1 ,
 ( φ ) = N i =1  ( xii , M ii ; φ ) − xi0i 1,
(15)
(15)
N i=1 1

0
where represents
whereϕφrepresents the FFDNet
the FFDNet weights,
weights,  mapping
F is the is the mapping
function,function,
and x andand x and
x0 denote denote
thexdegraded
the degraded
image image truth,
and its ground and its ground truth,
respectively, respectively,
N is the N is the
number of samples number
used of samples
for training used
per batch and for
M
trainingthe
denotes pernoise-level
batch and map
M of denotes the noise-level map of the FFDNet.
the FFDNet.

5.2.
5.2. Blur
Blur Kernel
Kernel Estimation
Estimation
As
As the ground truth
the ground truthofofthe
theblur
blurkernel
kernelremains
remains unknown
unknown during
during dynamic
dynamic visual
visual inspection,
inspection, we
we adopted the classical RL deconvolution [22] to deblur images based on the estimated
adopted the classical RL deconvolution [22] to deblur images based on the estimated blur kernel and blur kernel
and validate
validate the the estimation
estimation detailed
detailed in in Section
Section 4.1.
4.1. Figure6 6show
Figure showexamples
examplesof of blurred
blurred computer
computer case
case
images
images obtained
obtained from
from dynamic
dynamic visual
visual inspection
inspection andand their
their restorations.
restorations. Figure
Figure 77 shows
shows magnified
magnified
views ◦
views ofof these
these images.
images. The
The estimated blur lengths
estimated blur lengths areare 28,
28, 20,
20, 22,
22, and
and 27
27 px
px at
at fixed
fixed blur
blur angle
angle (90
(90°))
for
for the
the images
images in in Figure
Figure 6,
6, respectively.
respectively. Excluding
Excluding the the ringing
ringing artifacts
artifacts around
around the the saturated
saturated pixels,
pixels,
motion blur is mostly removed, thus visually validating the proposed blur kernel
motion blur is mostly removed, thus visually validating the proposed blur kernel estimation. estimation.

(a) (b)

Figure 6. Cont.
Symmetry 2019, 11, 678 11 of 18
Symmetry 2019, 11, x FOR PEER REVIEW 11 of 18

Symmetry 2019, 11, x FOR PEER REVIEW 11 of 18

(c) (d)
(c) (d)
Figure 6.
6. Blind
Figure Blind image
6. Blindimage
deblurring
image deblurring
forfor
deblurring for
dynamic
dynamic
dynamic
visual inspection
visual
visual inspection
inspection
based on estimated
based based
blur blur
on estimated
on estimated
kernel.
blur kernel.
The
Thekernel.
estimated
The estimatedblurblur
estimated lengths
blur areare
lengths
lengths (a)
(a)28(a)
are 28px,28(b)
px, px,20
(b) px,
(b)
20 (c)
20(c)
px, 22(c)
px,
22 px,
px,22and(d)(d)
px,
and 2727
and px.
(d)
px. 27The
The px. blurred images
Theimages
blurred blurred and their
images
and their and
restored versions
theirrestored
restored areare
versions
versions shown
are side
shown
shown sidebybyside.
side by side.
side.

Figure 7. Magnified views of examples in Figure 6. The blurred images and their restored versions
Figure 7. Magnified views of examples in Figure 6. The blurred images and their restored versions are
are shown side by side.
shown side
Figure by side. views of examples in Figure 6. The blurred images and their restored versions
7. Magnified
are Deconvolution
5.3. shown side byModeling
side.
5.3. Deconvolution Modeling
Although the proposed blur kernel estimation improves image quality, ringing artifacts
5.3. Deconvolution
Although the Modeling
proposed blur kernel estimation improves image quality, ringing artifacts attributed
attributed to saturated pixels persist. Hence, we further validated the proposed deconvolution model
to saturated
Although
and
pixels
compared theit to persist.
proposed Hence,
blur kernel
existing approaches,
wenamely,
further
estimation validated
RL [22],improves
the proposed
IDD-BM3Dimage [28], NCSR
deconvolution
quality,
[29], HL ringing model
artifacts
[26], fully
and compared
attributed it
to saturated
convolutional to existing
neural pixels approaches,
persist.
network namely,
Hence,[37],
(FCNN) we further RL [22], IDD-BM3D
validated the
image restoration [28], NCSR
proposed deconvolution
convolutional [29],
neural networkmodelHL [26],
fullycompared
and convolutional
(IRCNN) [38], and
it to neural
VEM [9],
existing network
for dynamic
approaches, (FCNN) [37],
visual
namely, image
[22], restoration
inspection.
RL convolutional
For a fair comparison,
IDD-BM3D [28], NCSR we [29], neural
fine-tuned
HL [26],network
the fully
(IRCNN) [38],
parameters and
from VEM
those [9], for
approaches dynamicand visual
used the inspection.
pretrained
convolutional neural network (FCNN) [37], image restoration convolutional neural network For
models a fair
for comparison,
FCNN and IRCNN.we fine-tuned the
parameters Figure
from 8 visually
those compares
approaches the
and restored
used the images
pretrainedusing the
models
(IRCNN) [38], and VEM [9], for dynamic visual inspection. For a fair comparison, we fine-tuned the proposed
for FCNN and similar
and IRCNN.methods.
Regarding
Figure 8from visual quality,
visually compares IDD-BM3D
the restoredand NCSR
images slightly
usingoutperform
the proposed theand
RL deconvolution,
similar methods. but those
parameters those approaches and used the pretrained models for FCNN and IRCNN.Regarding
two
visualFiguremethods
quality, collect similar image blocks nonlocally, and small structures, especially text, are not
8 IDD-BM3D
visually comparesand NCSR theslightly
restoredoutperform
images the RL deconvolution,
using the proposed and
suitably restored where block-like artifacts occur. HL uses a hyper-Laplacian prior to model a heavy-
but those
similartwo methods
methods.
collect similar
Regarding image
visual blocks
quality, nonlocally,
IDD-BM3D andand smallslightly
NCSR structures, especially
outperform thetext,
RL are not suitablybut
deconvolution, restored
tailed distribution, but its performance reduces in the presence of saturated pixels, leading to severe those
where block-like
two ringing
methods artifacts
collectFCNN occur.
similar HL uses a hyper-Laplacian prior to model a heavy-tailed distribution,
artifacts. andimage
IRCNNblocks are deep nonlocally,
learning methodsand small structures,
that fail especially
to effectively text, are not
handle outliers.
but its
suitably performance
restored
Moreover, FCNNwhere reduces
needs
in theartifacts
block-like
retraining
presence
according
of
occur. saturated
to theHLnoise apixels,
useslevel, leading to severe
hyper-Laplacian
and residual blur prior toringing
remains model artifacts.
over athe heavy-
FCNN and
tailedtext. IRCNN
distribution,
VEM is tailored are deep
but its learning
forperformance methods
handling saturated reduces that fail
in the
pixels, to effectively
thuspresence handle
providingofa saturated outliers. Moreover,
pixels,visual
highly improved leading FCNN
to severe
quality
needs
ringing retraining
in spite of a few
artifacts. according
visible
FCNN and to the
particles.
IRCNN noise
Comparedlevel,
are deepwith and residual
VEM, methods
learning blur
the proposedremains
thatmethod over the text.
with CNN-based
fail to effectively VEM
handle is
priortailored
outliers.
for handling
further FCNN
Moreover, saturated
denoises andpixels,
needs smooths thus
retraining theproviding
restored
according a to
image,highly improved
thusnoise
the reducing
level, visual
noise
andand quality in
highlighting
residual blurspite of details.
fine
remains a few visible
over the
particles. Measures
Compared such as
with the
VEM,peak signal-to-noise
the proposed ratio
method and mean
with squared
CNN-based
text. VEM is tailored for handling saturated pixels, thus providing a highly improved visual quality error cannot
prior be
further applied
denoises to and
dynamic restored
smooths visual inspection, as noreducing
ground truth is available as reference. Ondetails.
the other hand, measures
in spite ofthea few visible image, thus
particles. Comparednoise with andVEM, highlighting
the proposed fine method with CNN-based prior
without
Measures reference are not suited for comparison with human perception [48]. Thus, we combined
further denoisessuch andas the peak
smooths thesignal-to-noise
restored image,ratio thusand mean noise
reducing squared
reference-based measures with subjective visual quality evaluations to determine the deblurring
anderror cannot
highlighting be applied
fine details.to
dynamic visual
Measures such inspection, as
as the peak no ground truth
signal-to-noise is available
ratio whose as
and mean reference.
squared On the
error other
cannot hand, measures
be applied to
performance. Specifically, we first excluded the results subjective visual quality was obviously
without
dynamic reference
visual are not suited
inspection, as no for comparison
ground truth is with human
available as perception
reference. On [48].
the Thus,
other we combined
hand, measures
low, and for the rest of the results, we applied blurring to the restored images, that is, we subjected
reference-based measures with subjective visual quality evaluations
without reference are not suited for comparison with human perception [48]. Thus, we combined to determine the deblurring
performance. Specifically,
reference-based measureswe withfirstsubjective
excluded visualthe results whose
quality subjectiveto
evaluations visual qualitythe
determine wasdeblurring
obviously
performance. Specifically, we first excluded the results whose subjective visual quality was obviously
low, and for the rest of the results, we applied blurring to the restored images, that is, we subjected
Symmetry 2019, 11, 678 12 of 18

low, and for the rest of the results, we applied blurring to the restored images, that is, we subjected
the restored image
Symmetry 2019, fˆ toPEER
11, x FOR the estimated
REVIEW blur kernel Ĥ to synthesize blurred image b̂ (b̂ = Ĥ 12 fˆ) offrom
18 the
original blurred image b. Then, we adopted two reference-based measures, namely, SSIM [49] and
FSIMthe[50],
restored image f̂ the
to determine to the estimated
difference blur kernel
between b̂ andĤ btofor
synthesize
obtaining blurred
the deblurring ˆ=H ˆ fˆ )
image b̂ ( bperformance,
whilefrom
the the
conventional measures
original blurred imagewere b . Then,
basedwe onadopted
the restored image fˆ and its
two reference-based truth f , SSIM
ground namely,
measures, where f is
unavailable
[49] andin our [50],
FSIM case.to determine the difference between b̂ and b for obtaining the deblurring
Table 2 lists
performance, while the quantitative
the conventionalresults from comparing
measures were basedrestorations using
on the restored RL, IRCNN,
image f̂ and itsVEM,
ground and the
proposed
truth method.
f , where The f isresults from in
unavailable IDDBM3D,
our case. NCSR, HL, and FCNN are excluded from the table
given their low subjective visual quality.
Table 2 lists the quantitative results from It can be seen that
comparing the proposed
restorations using RL,method
IRCNN, outperforms
VEM, and the
existing approaches, especially for the images in Figure 8b,c
the proposed method. The results from IDDBM3D, NCSR, HL, and FCNN are excluded from which contain severe pixel saturation.
the
The increase
table given in their
scores low ranges fromvisual
subjective 0.09%quality.
(FSIMItofcan restored
be seenimage using
that the our method
proposed method versus that using
outperforms
VEMthe inexisting
Figureapproaches,
8b) to 19.62% (SSIM
especially foroftherestored
images inimage
Figureusing our contain
8b,c which methodsevere
versus that
pixel using RL in
saturation.
The8c).
Figure increase in scores ranges from 0.09% (FSIM of restored image using our method versus that using
VEM in Figure 8b)
For simplicity, thetoblur19.62%
kernel(SSIM of restored
is assumed toimage using our
be spatially method However,
invariant. versus thatthe using
blurRLkernelin is
Figure 8c).
in fact spatially variant, because the background remains static during exposure time. Including the
For simplicity, the blur kernel is assumed to be spatially invariant. However, the blur kernel is
proposed method, the restoration in Figure 8d using the evaluated deblurring methods contains severe
in fact spatially variant, because the background remains static during exposure time. Including the
ringing artifacts over the background.
proposed method, the restoration in Figure 8d using the evaluated deblurring methods contains
As theringing
severe RL deconvolution
artifacts over does not involve regularization, it might not suitably reflect the inherent
the background.
nature ofAs thethe image for dynamic visual
RL deconvolution does not inspection. Consequently,
involve regularization, it less
mightsevere ringingreflect
not suitably artifacts the occur
overinherent
the background
nature ofinthe its image
restoration, and it scores
for dynamic visual higher on the
inspection. employed measures
Consequently, less severeat ringing
the expense
of large noise.
artifacts occur over the background in its restoration, and it scores higher on the employed measures
at the expense
Table 3 reports of large noise. run times (in seconds) on the restorations of Figure 8 (856 × 856).
the average
RL, IRCNN, TableVEM,3 reports
andthetheaverage
proposed run times
method (in seconds)
were run onon
thearestorations of Figure
Nvidia GeForce GTX 8 (856 × 856).
1080 (Santa RL,Clara,
IRCNN, VEM, and the proposed method were run on a Nvidia
CA, USA). One can see that the proposed method avoids the time-consuming optimization by GeForce GTX 1080 (Santa Clara, CA,
USA). One can see that the proposed method avoids the time-consuming optimization by
incorporating the FFDNet-learned prior, thus delivering a faster speed than VEM. Compared with
incorporating the FFDNet-learned prior, thus delivering a faster speed than VEM. Compared with
IRCNN, the proposed method sacrifices a little speed for a better deblurring performance. The simple
IRCNN, the proposed method sacrifices a little speed for a better deblurring performance. The simple
and and
efficient
efficientRL RLdeconvolution
deconvolutionachieves achieves aa very competitivespeed,
very competitive speed,butbut it does
it does not make
not make a good a good
compromise between speed and performance, as is shown
compromise between speed and performance, as is shown in Figure 8 and Table 2. in Figure 8 and Table 2.

Figure 8. Cont.
Symmetry 2019, 11, 678 13 of 18
Symmetry 2019, 11, x FOR PEER REVIEW 13 of 18

(a)

(b)

Figure 8. Cont.
Symmetry 2019, 11, 678 14 of 18
Symmetry 2019, 11, x FOR PEER REVIEW 14 of 18

(c)

(d)

Figure8.8. Restorations
Figure Restorations including
includingmagnified
magnified views of the
views of blurred images
the blurred in Figure
images 6 for dynamic
in Figure visual
6 for dynamic
inspection
visual using using
inspection different methods
different where the
methods estimated
where blur lengths
the estimated blurare (a) 28 px,
lengths are(b)
(a)20
28px,
px,(c)(b)
2220
px,px,
(c)(d)
2227
px,px.
(d)In27the
px.left
Into
theright
left and top-down
to right order, deblurring
and top-down was performed
order, deblurring by applying
was performed byRL [22],
applying
RL iterative decoupled
[22], iterative deblurring-block
decoupled matching
deblurring-block and and
matching 3D filtering (IDD-BM3D)
3D filtering (IDD-BM3D) [28], nonlocally
[28], nonlocally
centralized sparse representation (NCSR) [29], hyper-Laplacian (HL) [26],
centralized sparse representation (NCSR) [29], hyper-Laplacian (HL) [26], fully convolutional fully convolutional neural
neural
network (FCNN) [37], image restoration convolutional neural network
network (FCNN) [37], image restoration convolutional neural network (IRCNN) [38], variational (IRCNN) [38], variational
expectation–maximization(VEM)
expectation–maximization (VEM) [9],
[9], and
and our
our proposed
proposed method.
method.
Symmetry 2019, 11, 678 15 of 18

Table 2. Score comparisons of the structural similarity index (SSIM) and feature similarity index (FSIM)
on the restorations of Figure 8.

Figure 8 (a) (b) (c) (d) (a) (b) (c) (d)

Measures SSIM FSIM
RL [22] 0.6932 0.6467 0.6126 0.7337 0.7586 0.7675 0.7384 0.8140
IRCNN [38] 0.7208 0.7178 0.7300 0.7030 0.7587 0.7685 0.7685 0.7390
VEM [9] 0.7094 0.7279 0.7198 0.6911 0.7384 0.7970 0.7864 0.7063
Our method 0.7263 0.7417 0.7328 0.7055 0.7407 0.7977 0.7889 0.7093

Table 3. Average run time (in seconds) on the restorations of Figure 8.

Methods RL [22] IRCNN [38] VEM [9] Our Method

Figure 8 (856 × 856) 2.03 6.83 18.95 7.26

6. Conclusions
We propose a method to perform blind deblurring of saturated images for dynamic visual
inspection. Given the assembly line application, in the proposed method, we considered the
blur angle of the linear motion as known and fixed, and only estimated the blur length from
the autocorrelation function of selected sharp edges. Then, we employed variable splitting and
included an FFDNet-learned prior into the objective function for optimization. Furthermore, saturated
pixels were rejected via a binary mask associated with the data fidelity term, whereas a noise-level
map ensured that the prior was applicable under a wide range of noise levels. The proposed method
omitted saturated pixels and learned the image prior adaptively, effectively integrating optimization
and deep learning. For deblurring performance evaluation, instead of using an original clear image as
ground truth, we synthesized a blurred image from the restored one. Then, we determined the SSIM
and FSIM, which indicated that the proposed method in general outperforms existing approaches,
with improvements in the measures of 0.09–19.62%. As image degradation during dynamic visual
inspection on an assembly line corresponds to linear motion with a static background, the blur kernel
is spatially variant. Hence, in future work, we will focus on spatially variant blur kernel estimation
to further improve the deblurring performance. In addition, as data fidelity is decoupled to handle
saturated pixels, the fast testing speed of deep learning is not fully exploited, and further developments
could improve the proposed method to deblur images in real time.

Author Contributions: B.W. and G.L. conceived the study. B.W. performed the experiments and wrote the paper
with J.W. who also analyzed the results.
Funding: This research was funded by the Guangzhou Science and Technology Plan Project (201802030006)
and the Open Project Program of Guangdong Key Laboratory of Modern Geometry and Mechanics Metrology
Technology (SCMKF201801).
Acknowledgments: The authors would like to thank Lenovo and Dell, Inc. for providing several kinds of
computer cases. They also thank Lite-On for building up and allowing them to use the visual inspection equipment
on its computer case assembly line. The authors also thank Editage (www.editage.cn) for English language editing.
Conflicts of Interest: The authors declare no conflict of interest.

References
1. Si, L.; Wang, Z.; Xu, R.; Tan, C.; Liu, X.; Xu, J. Image enhancement for surveillance video of coal mining face
based on single-scale retinex algorithm combined with bilateral filtering. Symmetry 2017, 9, 93. [CrossRef]
2. Kang, J.S.; Kim, C.S.; Lee, Y.W.; Cho, S.W.; Park, K.R. Age estimation robust to optical and motion blurring
by deep residual CNN. Symmetry 2018, 10, 108. [CrossRef]
3. Pandey, A.; Gregory, J.W. Iterative blind deconvolution algorithm for deblurring a single PSP/TSP image of
rotating surfaces. Sensors 2018, 18, 3075. [CrossRef] [PubMed]
Symmetry 2019, 11, 678 16 of 18

4. Wang, R.; Ma, G.; Qin, Q.; Shi, Q.; Huang, J. Blind UAV images deblurring based on discriminative networks.
Sensors 2018, 18, 2874. [CrossRef] [PubMed]
5. Fergus, R.; Singh, B.; Hertzmann, A.; Roweis, S.T.; Freeman, W.T. Removing camera shake from a single
photograph. ACM Trans. Graph. 2006, 25, 787–794. [CrossRef]
6. Lu, Q.; Zhou, W.; Fang, L.; Li, H. Robust blur kernel estimation for license plate images from fast moving
vehicles. IEEE Trans. Image Process. 2016, 25, 2311–2323. [CrossRef]
7. Xu, X.; Pan, J.; Zhang, Y.J.; Yang, M.H. Motion blur kernel estimation via deep learning. IEEE Trans. Image
Process. 2018, 27, 194–205. [CrossRef] [PubMed]
8. Moghaddam, M.E.; Jamzad, M. Motion blur identification in noisy images using fuzzy sets. In Proceedings
of the IEEE International Symposium on Signal Processing and Information Technology, Athens, Greece,
18–21 December 2005; pp. 862–866.
9. Cho, S.; Wang, J.; Lee, S. Handling outliers in non-blind image deconvolution. In Proceedings of the IEEE
International Conference on Computer Vision, Barcelona, Spain, 6–13 November 2011; pp. 495–502.
10. Zhang, K.; Zuo, W.; Zhang, L. FFDNet: Toward a fast and flexible solution for CNN-based image denoising.
IEEE Trans. Image Process. 2018, 27, 4608–4622. [CrossRef] [PubMed]
11. Shan, Q.; Jia, J.; Agarwala, A. High-quality motion deblurring from a single image. ACM Trans. Graph. 2008,
27, 73. [CrossRef]
12. Xu, L.; Zheng, S.; Jia, J. Unnatural L0 sparse representation for natural image deblurring. In Proceedings of
the IEEE Conference on Computer Vision and Pattern Recognition, Portland, OR, USA, 23–28 June 2013;
pp. 1107–1114.
13. Pan, J.; Hu, Z.; Su, Z.; Yang, M.H. Deblurring text images via L0 -regularized intensity and gradient prior.
In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Columbus, OH, USA,
23–28 June 2014; pp. 2901–2908.
14. Pan, J.; Sun, D.; Pfister, H.; Yang, M.H. Blind image deblurring using dark channel prior. In Proceedings
of the IEEE Conference on Computer Vision and Pattern Recognition, Seattle, WA, USA, 27–30 June 2016;
pp. 1628–1636.
15. Yan, Y.; Ren, W.; Guo, Y.; Wang, R.; Cao, X. Image deblurring via extreme channels prior. In Proceedings of
the IEEE Conference on Computer Vision and Pattern Recognition, Honolulu, HI, USA, 21–26 July 2017;
pp. 6978–6986.
16. Oliveira, J.P.; Figueiredo, M.A.T.; Bioucas-Dias, J.M. Parametric blur estimation for blind restoration of
natural images: Linear motion and out-of-focus. IEEE Trans. Image Process. 2014, 23, 466–477. [CrossRef]
[PubMed]
17. Moghaddam, M.E.; Jamzad, M. Motion blur identification in noisy images using mathematical models and
statistical measures. Pattern Recognit. 2007, 40, 1946–1957. [CrossRef]
18. Dash, R.; Majhi, B. Motion blur parameters estimation for image restoration. Optik 2014, 125, 1634–1640.
[CrossRef]
19. Yan, R.; Shao, L. Blind image blur estimation via deep learning. IEEE Trans. Image Process. 2016, 25, 1910–1921.
20. Lu, Y.; Xie, F.; Jiang, Z. Kernel Estimation for Motion Blur Removal Using Deep Convolutional Neural
Network. In Proceedings of the IEEE International Conference on Computational Photography, Beijing,
China, 17–20 September 2017; pp. 3755–3759.
21. Moghaddam, M.E.; Jamzad, M. Linear motion blur parameter estimation in noisy images using fuzzy sets
and power spectrum. Eur. J. Adv. Signal Process. 2006, 2007, 068985. [CrossRef]
22. Richardson, W.H. Bayesian-based iterative method of image restoration. J. Opt. Soc. Am. 1972, 62, 55–59.
[CrossRef]
23. Wiener, N. Extrapolation, Interpolation, and Smoothing of Stationary Time Series: With Engineering Applications;
MIT Press: Cambridge, MA, USA, 1964; Volume 113, pp. 1043–1054.
24. Oliveira, J.P.; Bioucas-Dias, J.M.; Figueiredo, M.A.T. Adaptive total variation image deblurring:
A majorization-minimization approach. Signal Process. 2009, 89, 1683–1693. [CrossRef]
25. Zuo, W.; Lin, Z. A generalized accelerated proximal gradient approach for total-variation-based image
restoration. IEEE Trans. Image Process. 2011, 20, 2748–2759. [CrossRef]
26. Krishnan, D.; Fergus, R. Fast image deconvolution using hyper-laplacian priors. In Proceedings of
the Advances in Neural Information Processing Systems, Vancouver, BC, Canada, 7–10 December 2009;
pp. 1033–1041.
Symmetry 2019, 11, 678 17 of 18

27. Zuo, W.; Meng, D.; Zhang, L.; Feng, X.; Zhang, D. A generalized iterated shrinkage algorithm for non-convex
sparse coding. In Proceedings of the IEEE International Conference on Computer Vision, Sydney, Australia,
1–8 December 2013; pp. 217–224.
28. Danielyan, A.; Katkovnik, V.; Egiazarian, K. BM3D frames and variational image deblurring. IEEE Trans.
Image Process. 2012, 21, 1715–1728. [CrossRef] [PubMed]
29. Dong, W.; Zhang, L.; Shi, G.; Li, X. Nonlocally centralized sparse representation for image restoration.
IEEE Trans. Image Process. 2013, 22, 1618–1628. [CrossRef]
30. Yair, N.; Michaeli, T. Multi-scale weighted nuclear norm image restoration. In Proceedings of the IEEE
Conference on Computer Vision and Pattern Recognition, Salt Lake City, UT, USA, 18–23 June 2018;
pp. 3165–3174.
31. Libin, S.; Sunghyun, C.; Jue, W.; Hays, J. Edge-based blur kernel estimation using patch priors. In Proceedings
of the IEEE International Conference on Computational Photography, Cambridge, MA, USA, 19–21 April
2013; pp. 1–8.
32. Eigen, D.; Krishnan, D.; Fergus, R. Restoring an image taken through a window covered with dirt or rain.
In Proceedings of the IEEE International Conference on Computer Vision, Sydney, Australia, 1–8 December
2013; pp. 633–640.
33. Schuler, C.J.; Burger, H.C.; Harmeling, S.; Schoelkopf, B. A machine learning approach for non-blind image
deconvolution. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition,
Portland, OR, USA, 23–28 June 2013; pp. 1067–1074.
34. Kim, J.; Lee, J.K.; Lee, K.M. Accurate image super-resolution using very deep convolutional networks.
In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Seattle, WA, USA,
27–30 June 2016; pp. 1646–1654.
35. Xu, L.; Ren, J.S.J.; Liu, C.; Jia, J. Deep convolutional neural network for image deconvolution. In Proceedings
of the Advances in Neural Information Processing Systems, Montreal, QC, Canada, 8–13 December 2014;
pp. 1790–1798.
36. Ren, W.; Zhang, J.; Ma, L.; Pan, J.; Cao, X.; Zuo, W.; Liu, W.; Yang, M.H. Deep non-blind deconvolution via
generalized low-rank approximation. In Proceedings of the Advances in Neural Information Processing
Systems, Montreal, QC, Canada, 2–8 December 2018; pp. 295–305.
37. Zhang, J.; Pan, J.; Lai, W.S.; Lau, R.W.H.; Yang, M.H. Learning fully convolutional networks for iterative
non-blind deconvolution. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition,
Honolulu, HI, USA, 21–26 July 2017; pp. 6969–6977.
38. Zhang, K.; Zuo, W.; Gu, S.; Zhang, L. Learning deep CNN denoiser prior for image restoration. In Proceedings
of the IEEE Conference on Computer Vision and Pattern Recognition, Honolulu, HI, USA, 21–26 July 2017;
pp. 2808–2817.
39. Liu, Y.; Wang, J.; Cho, S.; Finkelstein, A.; Rusinkiewicz, S. A no-reference metric for evaluating the quality of
motion deblurring. ACM Trans. Graph. 2013, 32, 175. [CrossRef]
40. Zhang, L.; Zuo, W. Image restoration: From sparse and low-rank priors to deep priors. IEEE Signal Process.
Mag. 2017, 34, 172–179. [CrossRef]
41. Dong, J.; Pan, J.; Su, Z.; Yang, M.H. Blind image deblurring with outlier handling. In Proceedings of the IEEE
International Conference on Computer Vision, Venice, Italy, 22–29 October 2017; pp. 2497–2505.
42. Whyte, O.; Sivic, J.; Zisserman, A. Deblurring shaken and partially saturated images. Int. J. Comput. Vis.
2014, 110, 185–201. [CrossRef]
43. Hu, Z.; Cho, S.; Wang, J.; Yang, M.H. Deblurring low-light images with light streaks. In Proceedings of
the IEEE Conference on Computer Vision and Pattern Recognition, Columbus, OH, USA, 23–28 June 2014;
pp. 3382–3389.
44. Geman, D.; Yang, C. Nonlinear image recovery with half-quadratic regularization. IEEE Trans. Image Process.
1995, 4, 932–946. [CrossRef]
45. Boyd, S.; Parikh, N.; Chu, E.; Peleato, B.; Eckstein, J. Distributed optimization and statistical learning via the
alternating direction method of multipliers. Found. Trends Mach. Learn. 2010, 3, 1–122. [CrossRef]
46. Chambolle, A.; Pock, T. A first-order primal-dual algorithm for convex problems with applications to imaging.
J. Math. Imaging Vis. 2011, 40, 120–145. [CrossRef]
47. Xiao, L.; Heide, F.; Heidrich, W.; Scholkopf, B.; Hirsch, M. Discriminative transfer learning for general image
restoration. IEEE Trans. Image Process. 2018, 27, 4091–4104. [CrossRef]
Symmetry 2019, 11, 678 18 of 18

48. Lai, W.S.; Huang, J.B.; Hu, Z.; Ahuja, N.; Yang, M.H. A comparative study for single image blind deblurring.
In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Seattle, WA, USA,
27–30 June 2016; pp. 1701–1709.
49. Wang, Z.; Bovik, A.C.; Sheikh, H.R.; Simoncelli, E.P. Image quality assessment: From error visibility to
structural similarity. IEEE Trans. Image Process. 2004, 13, 600–612. [CrossRef] [PubMed]
50. Zhang, L.; Zhang, L.; Mou, X.; Zhang, D. FSIM: A feature similarity index for image quality assessment.
IEEE Trans. Image Process. 2011, 20, 2378–2386. [CrossRef]

© 2019 by the authors. Licensee MDPI, Basel, Switzerland. This article is an open access
article distributed under the terms and conditions of the Creative Commons Attribution
(CC BY) license (http://creativecommons.org/licenses/by/4.0/).