Proceedings of the IEEE International Conference on Multimedia and Expo (ICME) 2017
AN ACCURATE DEEP CONVOLUTIONAL NEURAL NETWORKS MODEL FOR
NO-REFERENCE IMAGE QUALITY ASSESSMENT
Bahetiyaer Bare, Ke Li, Bo Yan
School of Computer Science, Shanghai Key Laboratory of Intelligent Information Processing,
Fudan University,Shanghai 201203, China
byan@fudan.edu.cn
ABSTRACT
The goal of image quality assessment (IQA) is to use com-
putational models to measure the consistency between im-
age quality and subjective evaluations. In recent years, con-
volutional neural networks (CNNs) have been widely used
in image processing community and have achieved perfor-
mance leaps than non CNNs-based methods. In this work,
we describe an accurate deep CNNs model for no-reference
IQA. Taking image patches as input, our deep CNNs model
achieves an end-to-end method without any handcrafted fea-
tures and pre-processing procedures that are employed by pre-
vious no-reference IQA methods. The proposed model con-
sists of six convolutional layers, two fully connected layers,
one max pooling layer and two sum layers. The experimental
results verify that our model outperforms the state-of-the-art
no-reference IQA methods and most of the full-reference IQA
metrics.
Index Terms— Image quality assessment, Deep learning,
Convolutional neural networks
1. INTRODUCTION
With the development of social networks and the increasing
number of imaging devices, an enormous amount of visu-
al data is making its way to consumers. Digital images are
subject to a wide variety of distortions during acquisition,
processing, compression, storage, transmission and reproduc-
tion, any of which may result in a degradation of visual qual-
ity. Thus a perceptual evaluation process is needed for digital
images. While human subjective judgments of images are the
most reliable assessment, these are time consuming and diffi-
cult to obtain. Thus image quality assessment (IQA) methods
are used to automatically predict the visual quality of images.
IQA methods can be used to optimize image processing algo-
rithms and can be used to benchmark image processing sys-
tems and algorithms. Therefore, IQA plays a very important
role in image processing community.
This work is supported in part by the National Key Research and De-
velopment Plan (Grant No. 2016YFC0801005), and NSFC (Grant No.:
61522202; 61370158).
978-1-5090-6067-2/17/$31.00 2017
c IEEE
978-1-5090-6067-2/17/$31.00 ©2017 IEEE
1356
10-14 July 2017
Objective quality assessment methods can be categorized
into three groups based on whether and how reference im-
ages are used. These three groups are : full-reference IQA
methods, reduced-reference IQA methods, and no-reference
IQA methods. Taking full information of original image as
reference, full-reference IQA methods perform better than
other type of methods. Among various various kind of full-
reference IQA methods, structural similarity [1] (SSIM) in-
dex is a new standard for image processing applications. It
has better quality prediction results than classic methods like
PSNR or MSE due to taking structural similarity as an im-
portant factor. In [2], Zhang et al. proposed an feature simi-
larity (FSIM) index based on the fact that human visual sys-
tem (HVS) understands an image mainly according to its low-
level features. According to the experiment results, FSIM
achieves the state-of-the-art performances on various image
quality databases. It predicts image quality very similar to the
HVS.
Although full-reference IQA provides a useful and effec-
tive way to evaluate quality differences, in many applications
the reference image is not available. So, no-reference IQA is
required. Because humans often can not judge an distorted
image without reference image, it is very challenging from
a computational perspective. No-reference measures can di-
rectly quantify image degradations by exploiting features that
are discriminant for image degradations. Most successful ap-
proaches are Natural Scene Statistics (NSS) based methods.
Early NSS based methods extracted features in transforma-
tion domains via DCT transform [3] or wavelet transform [4].
However, it is very slow to extract features from transforma-
tion domain. BRISQUE [5] extracted features from the spatial
domain, which leads to a significant reduction in computation
time. Very different from NSS based methods, CORNIA [6]
demonstrates that it is possible to learn discriminant image
features directly from the raw image pixels, instead of using
handcrafted features.
In recent years, with the explosion of CNNs based meth-
ods in computer vision and image processing tasks, some
CNNs based no-reference IQA methods are emerged. A-
mong them, Kang et al. [7] proposed a CNNs model for
ICME 2017
no-reference IQA and achieved very similar results to full-
reference IQA methods. Inspired by [7], Liang et al. [8] pro-
posed an IQA method using similar scene as reference and
implement full-reference and no-reference IQA methods in
the paper. Their no-reference IQA method named CNN-NR-
d achieved the state-of-the-art performances in two popular
IQA databases. However, these methods have limitations.
The shallow network structure and the training strategy limit
their stability.
In this paper, in order to address the limitations of previ-
ous CNNs based no-reference IQA methods, we proposed an
accurate deep CNNs model for no-reference IQA. The main
contributions of our work lie in three aspects: i) Inspired by
[9], we proposed a deep CNNs model for no-reference IQA ,
which has better stability and performances; ii) we achieved
an end-to-end IQA method without pre-processing, which is
used by previous CNNs based methods; iii) by training the
proposed model on image patches, which are employed FSIM
[2] value as label, we achieved state-of-the-art performances.
The rest of the paper is organized as follows. In Section 2,
we introduce related works. Then we describe our proposed
model in Section 3. In Section 4, we present experimental
results. Finally, we draw conclusions in Section 5.
2. RELATED WORK
No-reference IQA methods aims to predict the quality of the
distorted images without reference images. A large amount of
study has been conducted on to address the no-reference IQA
problem. The vast majority of no-reference IQA algorithm-
s attempt to detect specific types of distortion such as blur-
ring, blocking, ringing, or various forms of noise. In our re-
search, we focus on more general-purpose no-reference IQA
algorithms which do not attempt to detect specific types of
distortions. This type of methods typically reformulate the
IQA problem into a classification and regression problem in
which the regressors/classifiers are trained using specific fea-
tures. The relevant features are either discovered via machine
learning or specified by using natural-scene statistics [10].
In [11], Tang et al. presented a learning-based no-
reference IQA method named LBIQ, which measures various
low-level quality features derived from natural-scene and tex-
ture statistics. LBIQ estimates quality via a regression based
combination of the features. In [12, 13], Ye and Doermann
presented the CBIQI and CBIQ-II algorithms which operate
based on visual codebooks.
Another popular approach to no-reference IQA is to use
natural scene statistics (NSS). The main idea in this approach
is that natural images demonstrate certain statistical regular-
ities that can be affected in the presence of distortion. This
kind of methods always require training, where the regression
model is used to map the measured features to an associated
quality score. In [14], Moorthy and Bovik presented the BIQI
algorithm which estimates quality based on statistical features
1357
extracted using the 9/7 DWT. In [4], Moorthy and Bovik pre-
sented the DIIVINE algorithm, which improves upon BIQI by
using a steerable pyramid transform with two scales and six
orientations. In [15, 3], Saad et al. presented the BLIINDS-I
and BLIINDS-II algorithms which estimate quality based on
DCT statistics. In [5], Mittal et al. presented the BRISQUE
algorithm, a fast no-reference IQA algorithm, which employs
statistics measured in the spatial domain.
Since 2012, CNNs have been widely applied in com-
puter vision fields with the success of deep CNNs in image
classification. Successful deep CNNs models such as deep
residual network [9] have been continuously proposed and
achieved outstanding performances. In [7], Kang et al. pro-
posed convolutional neural networks for no-reference IQA
and achieved performance leaps than classic methods. In [8],
Liang et al proposed an IQA method using similar scene as
reference, which is also achieved by using dual-path convo-
lutional neural network. For comparison, they also achieved
full-reference and no-reference IQA methods using same net-
work architecture.
Although CNNs-based IQA methods achieved outstand-
ing performances, they have limitations in three aspects: first,
they take the subjective evaluation value of the entire image
as a label of small patches. Thus all the image patches which
are taken from same image have a same label. Although this
work for uniformly distributed noises, it makes the trained
model has poor generalization ability; second, the input im-
ages of these two methods go through a preprocessing, so
these two methods is not end-to-end method. We believe that
convolutional neural networks can learn features from raw
images without preprocessing; third, the network they used
is shallow, some revolutionary techniques like deep residual
network [9] motivates us to use deeper network architecture.
3. IQA WITH PROPOSED MODEL
In most of the IQA cases, there is no reference image provid-
ed for distorted images. Although there existed some very ex-
cellent no-reference IQA methods, the performance of these
methods is limited by the lack of large scale image quality as-
sessment database and shallow network architecture. There-
fore, we propose a deep CNNs model for no-reference IQA
in this section. The model can accurately predict the quality
of distorted images and has well generalization ability. The
proposed model is detailed in the following subsections.
3.1. Deep CNNs Model for No-Reference IQA
We present an accurate deep CNNs model for no-reference
IQA problem. For an input image, we split it to multiple
32 ˆ 32 small patches without overlaps at first. Then, the
proposed model predict quality score for each small patch.
At the end, the final quality score of the input image is the
average sum of the quality score of small patches. The net-
 9[SRG_KX

/TV[Z OSGMKVGZINKY )UT\UR[ZOUTGRRG_KX )UT\UR[ZOUTGRRG_KX )UT\UR[ZOUTGRRG_KX

1KXTKR VGJJOTM  1KXTKR VGJJOTM  1KXTKR VGJJOTM 

VUUROTMRG_KX
YO`K YZXOJK 

RGHKR

,)
RUYY

9[SRG_KX )UT\UR[ZOUTGRRG_KX )UT\UR[ZOUTGRRG_KX )UT\UR[ZOUTGRRG_KX

1KXTKR VGJJOTM  1KXTKR VGJJOTM  1KXTKR VGJJOTM 
,)

Fig. 1. The network architecture of our method. Our network consists of six convolutional layers, one max pooling layer, two
sum layers and two fully connected (FC) layers. The output of the second FC layer is the predicted quality score of the input
image patch. The overall quality score of the input image is the average sum of the predicted score of small patches.

work architecture of our model is shown in Fig. 1. As shown
in this figure, our model consists of six convolutional layer-
s with taking rectified linear unit (ReLU) [16] as activation
function, two fully connected layers and a max pooling layer.
Its worth noting that we add two sum layers to our network
inspired by deep residual network [9].
3.2. Layers
The network configuration is listed in Table 1. Our model
has six convolutional layers to extract local features. Each
convolutional layer takes ReLU [16] as activation function.
Denote Cj as the feature map of the j th layer, Wj and Bj as
the weight and bias of the j th layer, then the local information
is extracted into deeper layers by Eq.(1), where * denotes the
calculation of convolution. We set the bias Bj to zero in our
model.
Cj`1 “ maxp0, Wj ˚ Cj ` Bj q (1)
In order to reduce the complexity and computation cost,
we add pooling layers to our model. We employed max pool-
ing with 2 ˆ 2 window size. So after pooling layer, the feature
map size is reduced to half. The max pooling is applied as
Eq.(2), where R is the pooling region of corresponding fea-
ture map.
Cj`1 “ max Cj (2)
R
scent we add two sum layers to our method. As revealed in
[9], sum layer is very effective to train deep CNNs model. The
first sum layer adds the outputs of second and the third convo-
lutional layers and send them to the pooling layer. Similarly,
the second sum layer adds the outputs of the 5th and 6th con-
volutional layers and send them to first fully connected layer.
The second fully connected layer outputs the predicted val-
ue. The image score is predicted by minimizing the following
Euclidean loss,
min ||f pX; W q ´ Y || (3)
W
where X and Y denote the input image patch and its label
respectively and f pX; W q be the predicted score of X with
network weights W .
3.3. Learning
Stochastic gradient decent (SGD) and backpropagation are
used to solve the parameters W in Eq.(3) that minimizes the
loss between predicted score and ground truth. In particular,
weights of the convolutional or fully connected layers can be
updated as Eq.(4) [8]
BL
Δi`1 “ m ¨ Δi ´ η BW j
i
(4)
j
Wi`1 “ Wij ` Δi`1 ´ ληWij

Our model is a deep CNNs model with six convolutional where m is the momentum factors, η is the learning rate,
layers. In order to easily converge and prevent gradient de- j is index of the layer and Δi`1 is the gradient increment

1358
 Reference image score=0.849, DMOS=44.84 score=0.425, DMOS=72.49

score=0.841, DMOS=48.07 score=0.725, DMOS=56.7 score=0.917, DMOS=25.73

Fig. 2. An example of predicted quality of our method. We show the performance of our model with five different kind of
distortions at different degree. The score of each image is predicted by our model (1 is the best quality and 0 is the worst). For
comparison, we put the DMOS value for each distorted image (higher DMOS represent lower perceptual quality).

4. EXPERIMENT
Table 1. Configuration of our deep CNNs model

Layer name Padding Filter size Stride Output size

In this section, we demonstrates experiments to validate the
effectiveness of the proposed model. We apply deep learning
input 0 32 ˆ 32 ˆ 3
toolbox Matconvnet [17] to train the deep CNNs model for
conv1/relu 0 5ˆ5 1 28 ˆ 28 ˆ 50 no-reference IQA. The LIVE [18] database is used to train
conv2/relu 0 3ˆ3 1 26 ˆ 26 ˆ 50 our model. At the training stage, we first extract 32 ˆ 32
conv3/relu 1 3ˆ3 1 26 ˆ 26 ˆ 50 patches with stride 32 from the images in LIVE database. S-
sum1 26 ˆ 26 ˆ 50
ince different image patches have different quality values, we
compute FSIM [2] value for them and take FSIM value as la-
max pooling 2ˆ2 2 13 ˆ 13 ˆ 50
bel. Then we train our model with different learning rates.
conv4/relu 0 3ˆ3 1 11 ˆ 11 ˆ 25 Learning rate is changed to 1{10 of the previous one at the
conv5/relu 0 3ˆ3 1 9 ˆ 9 ˆ 25 interval of five epoch. It begins with 0.1 and stops at 0.00001.
conv6/relu 1 3ˆ3 1 9 ˆ 9 ˆ 25 After reaches 0.00001, it becomes a fixed value. Finally, our
deep CNNs model is obtained by training after 80 epoches.
sum2 9 ˆ 9 ˆ 25
The experimental results are demonstrated in the following
fc1 1024 subsections.
fc2 1

for training iteration i. λ is the weight decay factor. Mo-
mentum factor and weight decay factor were fixed to 0.9 and
0.0005 respectively in our model training. Learning rate is set
to different values ranges from 0.01 to 0.00001 at different e-
poches. Experiment details will be further explained in next
section.
4.1. Datasets
The following two datasets are used in our experiments.
(1) LIVE [18]: A total of 779 distorted images with five
different distortions - JP2k compression (JP2K), JPEG com-
pression (JPEG), White Noise (WN), Gaussian blur (BLUR)
and Fast Fading (FF) at 7-8 degradation levels derived from
29 reference images. Each image has a differential Mean
Opinion Scores (DMOS), which are in the range [0,100].

1359
 Table 2. SROCC and LCC scores for testing on LIVE

Full-reference methods No-reference methods

SSIM PSNR FR-DCNN BRISQUE CORNIA CNN-NR CNN-NR-d Ours
LCC 0.945 0.875 0.977 0.942 0.935 0.953 0.968 0.974
SROCC 0.948 0.876 0.975 0.940 0.942 0.956 0.967 0.971

Higher DMOS indicates lower quality.

Table 3. SROCC and LCC scores for testing on TID2008
(2) TID2008 [19]: 1700 distorted images with 17 different
distortions derived from 25 reference images at 4 degradation
levels. In our experiments, we only consider the four common BRISQUE CORNIA CNN-NR CNN-NR-d Ours
distortions that are shared by the LIVE dataset, i.e. JP2k, SROCC 0.882 0.890 0.920 0.921 0.957
JPEG, WN and BLUR. Each image is associated with a Mean LCC 0.892 0.880 0.903 0.920 0.939
Opinion Score (MOS) in the range [0, 9]. Contrary to DMOS,
higher MOS indicates higher quality.
4.4. Cross Dataset Test

4.2. Evaluation In order to test the generalization ability of our method, we

We deploy Pearsons (linear) correlation coefficient (LCC) and
Spearman Rank Order Correlation Coefficient (SROCC) to e-
valuate the performance of IQA algorithms. LCC measures
the linear dependence between two quantities and SROCC
measures how well one quantity can be described as a mono-
tonic function of another quantity.
4.3. Evaluation on LIVE
In order to test the correlation between our method and hu-
man opinions, we conducted test on LIVE [18] dataset. S-
ince our approach is totally trained on the LIVE dataset
by setting FSIM [2] values as label of image patches, we
can test our method on LIVE. We compare our method
with two widely used full-reference method, one deep CNNs
based full-reference method, and state-of-the-art no-reference
IQA methods. They are: PSNR, SSIM [1], FR-DCNN [8],
BRISQUE [5], CORNIA [6], CNN-NR [7], and CNN-NR-
d [8]. Experiment may he different from other no-reference
IQA methods since they used 80% data for training and the
remaining 20% for testing. Because they randomly select
80% data for training and iterate train-test process up to 1000
times, the experimental results are very similar to test all the
images of the datasets. So it is fair to compare our method
with other no-reference IQA methods.
The experimental results are listed in Table 2. As shown
in this table, our method outperforms state-of-the-art no-
reference methods in SROCC value and LCC value compar-
isons. When compared with full-reference IQA methods, our
method generates very similar results to the state-of-the-art
deep CNNs based full-reference method. It is worth noting
that our method outperforms two widely used full-reference
IQA methods.
trained proposed method on the whole LIVE [18] dataset and
conducted cross dataset test on the TID2008 [19] dataset.
This kind of test is followed by the protocol of previous no-
reference IQA methods [5, 6, 7, 8]. Only four types of dis-
tortions that are shared by LIVE and TID2008 are examined
in this experiment. Because our method is trained by tak-
ing FSIM [2] value as the label of the image patches, we have
no mapping process like other CNNs-based no-reference IQA
methods [7, 8].
The experimental results are listed in Table 3. We also
compare performance of our method with the state-of-the-
art no-reference IQA methods: BRISQUE [5], CORNIA [6],
CNN-NR [7], and CNN-NR-d [8]. As can be seen from this
table, our method achieves the best performance on TID2008
dataset among the no-reference IQA methods. Because our
method is more deeper than other CNNs-based no-reference
IQA methods [7, 8] and learning different scores for differ-
ent small patches that taken from same images, our method
has more generalization ability than compared state-of-the-art
methods.
4.5. Discussion
Our model achieves good performance in two well-known
databases. By taking FSIM value as label of image patches
and applying deeper CNNs model, we achieve better results
than previous no-refrence IQA methods. Deeper CNNs mod-
el and larger size of image patches will further improve the
performance of CNNs based methods. In Fig.2, we show the
predicted score of our model to same images suffered from
different distortions at different level. As we can observed
from this figure, our method can predict image quality, which
is consistent with the HVS.

1360
 5. CONCLUSION
In this paper, we proposed a highly accurate deep CNNs mod-
el for no-reference image quality assessment. We achieved
an end-to-end no-reference image quality assessment method
without any pre-processing and post-processing procedures.
By taking the large scale of image patches as the training set
and taking its FSIM value as the label, our model can achieve
promising quality prediction results. We used the contents in
LIVE dataset to train and test our method. In order to ver-
ify stability of our method, we also conducted cross dataset
test on TID2008 dataset. The experimental results confirm
a performance leap relative to ten compared state-of-the-art
methods.
6. REFERENCES
[1] Zhou Wang, Alan C Bovik, Hamid R Sheikh, and Eero P
Simoncelli, “Image quality assessment: from error vis-
ibility to structural similarity,” IEEE transactions on
image processing, vol. 13, no. 4, pp. 600–612, 2004.
[2] Lin Zhang, Lei Zhang, Xuanqin Mou, and David Zhang,
“Fsim: a feature similarity index for image quality as-
sessment,” IEEE transactions on Image Processing, vol.
20, no. 8, pp. 2378–2386, 2011.
[3] Michele A Saad, Alan C Bovik, and Christophe Charri-
er, “Blind image quality assessment: A natural scene s-
tatistics approach in the dct domain,” IEEE Transaction-
s on Image Processing, vol. 21, no. 8, pp. 3339–3352,
2012.
[4] Anush Krishna Moorthy and Alan Conrad Bovik,
“Blind image quality assessment: From natural scene
statistics to perceptual quality,” IEEE Transactions on
Image Processing, vol. 20, no. 12, pp. 3350–3364, 2011.
[5] Anish Mittal, Anush Krishna Moorthy, and Alan Con-
rad Bovik, “No-reference image quality assessment in
the spatial domain,” IEEE Transactions on Image Pro-
cessing, vol. 21, no. 12, pp. 4695–4708, 2012.
[6] Peng Ye, Jayant Kumar, Le Kang, and David Doerman-
n, “Unsupervised feature learning framework for no-
reference image quality assessment,” in Computer Vi-
sion and Pattern Recognition (CVPR), 2012 IEEE Con-
ference on. IEEE, 2012, pp. 1098–1105.
[7] Le Kang, Peng Ye, Yi Li, and David Doermann, “Con-
volutional neural networks for no-reference image qual-
ity assessment,” in Proceedings of the IEEE Conference
on Computer Vision and Pattern Recognition, 2014, pp.
1733–1740.
[8] Yudong Liang, Jinjun Wang, Xingyu Wan, Yihong
Gong, and Nanning Zheng, “Image quality assessment
1361
using similar scene as reference,” in European Confer-
ence on Computer Vision. Springer, 2016, pp. 3–18.
[9] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian
Sun, “Deep residual learning for image recognition,”
arXiv preprint arXiv:1512.03385, 2015.
[10] Damon M Chandler, “Seven challenges in image quality
assessment: past, present, and future research,” ISRN
Signal Processing, vol. 2013, 2013.
[11] Huixuan Tang, Neel Joshi, and Ashish Kapoor, “Learn-
ing a blind measure of perceptual image quality,” in
Computer Vision and Pattern Recognition (CVPR), 2011
IEEE Conference on. IEEE, 2011, pp. 305–312.
[12] Peng Ye and David Doermann, “No-reference image
quality assessment using visual codebooks,” in IEEE
International Conference on Image Processing (ICIP),
pp. 3150–3153, 2011.
[13] Peng Ye and David Doermann, “No-reference im-
age quality assessment using visual codebooks,” IEEE
Transactions on Image Processing, vol. 21, no. 7, pp.
3129–3138, 2012.
[14] Anush Krishna Moorthy and Alan Conrad Bovik, “A
two-step framework for constructing blind image quali-
ty indices,” IEEE Signal Processing Letters, vol. 17, no.
5, pp. 513–516, 2010.
[15] Michele A Saad, Alan C Bovik, and Christophe Char-
rier, “A dct statistics-based blind image quality index,”
IEEE Signal Processing Letters, vol. 17, no. 6, pp. 583–
586, 2010.
[16] Vinod Nair and Geoffrey E. Hinton, “Rectified lin-
ear units improve restricted boltzmann machines vinod
nair,” in International Conference on Machine Learn-
ing, 2010, pp. 807–814.
[17] Andrea Vedaldi and Karel Lenc, “Matconvnet: Convo-
lutional neural networks for matlab,” in Proceedings of
the 23rd ACM international conference on Multimedia.
ACM, 2015, pp. 689–692.
[18] Hamid R Sheikh, Zhou Wang, Lawrence Cor-
mack, and Alan C Bovik, “Live image qual-
ity assessment database release 2. online,
http://live.ece.utexas.edu/research/quality,” 2005.
[19] Nikolay Ponomarenko, Vladimir Lukin, Alexander Ze-
lensky, Karen Egiazarian, M Carli, and F Battisti,
“Tid2008-a database for evaluation of full-reference vi-
sual quality assessment metrics,” Advances of Modern
Radioelectronics, vol. 10, no. 4, pp. 30–45, 2009.