2016 Second International Conference on Web Research (ICWR)
A Novel Approach for Website Aesthetic
Evaluation based on Convolutional Neural Networks
Masoud Ganj Khani 1, Mohammad Reza Mazinani2
1 1,2
AmirKabir University of Technology
2
Malek-Ashtar University of Technology
Tehran, Iran
1 2
ganjkhany@aut.ac.ir
2
mazinany@gmail.com
Abstract—In this paper we propose a website aesthetic
evaluation method. For achieving better performance, we have
applied convolutional neural networks, which are one of the
methods of deep learning research area. Using deep learning
and convolutional neural networks for feature representation
is one of the main tips that makes difference between our work
and previous ones. Our system takes a screenshot of the
website as input, and finally reports it is a good or bad website
based on users’ country or not. For evaluation process, we
represent the website screenshot using MemNet convolutional
neural network. Then we decrease the extracted features
dimension using principal component analysis algorithm.
Finally, we classify them using a SVM classifier, which trained,
based on users’ ratings. Furthermore, aesthetics evaluation in
this research is language independent. It means the website’s
language is not important and our method works for all
languages.
Index Terms— Aesthetics, Website, Deep Learning,
Convolutional Neural Network.
I. INTRODUCTION
Today, the majority of our needs met through internet.
Things like shopping, reading news, and online gaming are
only part of the websites’ application. Thus, in general
websites classified in groups, such as, news, scientific,
marketing, etc. There are many sites in each group. This
number added each day and even every hour. One crucial
point that makes the users to be attracted to a website is
aesthetic. Aesthetic is the first thing that a user after visiting
the website is unconsciously affected by[1]. This effect
makes the user next visits to the sites that they are desirable
for him or her. In other words, when a group of websites
offer similar services, users tend to be among the sites that
they are more visually attractive than others[1]. That is why
today's web designers spend a lot of time on graphical
elements and images used on the website and they design
them with great care.
However, what is really causes beauty for the site is in
the eyes of the audience. Attractive and high quality images,
graphic elements, and animations can make the website
beautiful. Nevertheless, in fact this is not the case. Because
we know websites that take advantage of this feature, but
978-1-5090-2166-6/16/$31.00 ©2016 IEEE
48
1
Mohsen Fayyaz, 2Mojtaba Hoseini
Malek-Ashtar University of Technology
Tehran, Iran
1
mohsen.fayyaz89@gmail.com
mojtabahoseini@aut.ac.ir
users do not tend to refer to them, and they have little traffic
[2] [3]. In other words, aesthetic is more a quality parameter
than a quantitative parameter to measure. Another view
examines the effect of complexity on websites. Some
researchers believe that simplicity makes beauty. But as was
said, it is not all the reason of beauty [4].
In this article, we seek to understand the beauty with a
general way. It seems that the use of numerical quantities,
such as the extraction of some of the parameters of the site is
a part of providence. For example, using some quantitative
parameters such as the number of graphical elements,
density, and size can only be a guess about the beauty of a
website. Guess that may be not true in many cases and does
not lead to results. There is the possibility of wrong
parameter selection for representing the beauty of the site. It
is also possible that some errors occur during calculation of
parameters. Generality keeps us away from such mistakes.
Using a general strategy can have a better performance on
website aesthetic evaluation. However, as our approach is in
general way, we do not use the numerical parameters.
Our approach is also language-independent and can be
used for all sites in any language. This feature makes this
algorithm to be able to be applied in different countries. One
of the major challenge of the traditional website aesthetic
evaluation systems, such as, [5]is deciding what features to
be extracted from the website, to help the system in the
evaluation process. In other word to evaluate the aesthetic of
websites, some features are needed. These features should be
able to represent the websites based on their aesthetic. This is
a traditional problem in machine learning research. For
solving such problem, a wide variety of algorithms, methods,
and field of research have been introduced. One of the most
promising introduced fields of research is deep learning. It is
a new area of machine learning research that solves the
problem of choosing best hand crafted features by learning
the best features for representing the raw data.
To alleviate the problem of choosing best features, we
have used Convolutional Neural Networks (CNN), which are
one of the best feature extraction methods in deep learning.
Our system takes the website screenshot as input. Then
represents it using a convolutional neural network. Finally,
we classify the represented data using a Support Vector
Machine (SVM) to find out that the website is good or not.
We have trained our classifier based on the opinion of the
dataset users. For better evaluation of the websites’ aesthetic,
we consider the country of the users. Therefore, we have
evaluated each website based on the target society.
Our main contributions are:
• Applying deep learning and convolutional neural
networks for feature representation of input
screenshots of websites. Our main difference with
pictorial aesthetic evaluation systems, such as, [6] is
that we are evaluating a website aesthetic which
composes of texts, images, ribbons, changing parts,
etc. ,which each of them can have an enormous
effect on user opinion about the whole website,
while they are evaluating images of natural scenes,
see sides, objects, animals, etc. which most of them
doesn’t compose of different parts with different
subjects, such as, texts, movies, .etc.
• We evaluate the websites’ screenshots, in which all
the elements of a website are considered together;
like what happens in human mind when watching a
scene. While in other works, such as, [2], [7], each
element of website are evaluated separately.
• Using a public dataset which has been used for
website aesthetic evaluation with a different point of
view, such as, statistical demographic analysis in [2],
[7].
• Considering the home country of the user whom
websites we would evaluate.
The rest of the paper is organized as follows: we survey
related works in this field of research in section II. Our
proposed method is introduced in section III that explains
how we applied deep learning in our work. In section IV,
experimental results have been described. Finally, in section
V we have concluded the paper and proposed some
suggestions for future work.
II. RELATED WORK
According to [1] for measuring how much user interested
in a website, the first feeling after entering site is very
important. In a short time, the user decides to stay in this site
or surf elsewhere. According to [8] this short time is only 50
milliseconds. We should manage this time. In this, paper the
effect of complexity on aesthetics reviewed by voting.
In consonance with [1] “color” has a special role in
website aesthetics. Some papers like [9] show that each color
has special effects on the viewer. Some colors are warm and
other ones are cold and so on. In addition to color, “balance”
is also important. As stated in [10] when a website was
balanced, users sense equilibrium psychologically. Balance
means equal visual weight distribution horizontally and
vertically. In other word, it means that html elements
distributed equally inside the page.
In [2] authors have employed some metrics, such as,
balance, hue, and saturation of colors, symmetry of elements,
number of images, and other visual statistical metrics. This
49
paper compares rate of people to websites and the predicted
rate extracted from visual metrics. It concluded that
colorfulness and visual complexity are the main parameters
in aesthetics and ratings. As stated in [3] also vertical
symmetry is an important parameter in aesthetic website
design.
As reported by [4] using separated elements, intelligent
titles and images has a good impression on attracting viewer.
The information in these sites visually separated from each
other and classified in groups. In the case it takes, about five
seconds to user find what he is looking for. The visual
complexity of a website depends on the way of present,
dispersion and density of the elements. In [4] authors
Showed the relation between aesthetic and complexity. It
uses a part of top hundred Alexa’s website with voting to
show that relation. The parameters in this examination are
menus, images, words, links, and part of page that has no
sharing space with other parts. Complexity means using
many parameters in the website. It shows that if a website
was complex, it was not beautiful.
In agreement with [11] number of text sections, number
of links, page dimension, graphic elements, number of
colors, and complexity of reading are more important in high
rated websites. This paper also uses voting for gathering
information about aesthetics. This voting was on web experts
who pay attention to content, structure, navigation, graphical
design, interact with the user, and self-experiences.
In [12] authors determine the average of people’s scores
about a website as a criterion for website aesthetic.
In practice, some papers, such as, [5], emphasized that
online shopping website environment and its aesthetics have
an effect on attracting customers. In [13] it is shown that the
aesthetic increase the usability, hedonic, and utilitarian value.
These three values are the most important parameters in
shopping website. In addition, the paper uses a questionnaire
for gathering the information about 10 shopping websites.
Conforming to [13] high quality images also can attract
customers and improve sales.
III. METHODOLOGY
As mentioned above, for evaluation of the aesthetic of
websites, we need features that can separate the websites
based on their aesthetic when they are described by such
features. The problem of choosing the best features for
representing the raw data by is a challenging problem in
machine learning research. Deep learning is a new area of
machine learning research that provides us the ability to
solve such problems by learning the best features for
representation of the raw data. Not only deep learning
provides us the power of learning the best features to extract,
but also the ability to control the dimension of new
represented data. In other word, we achieve both the power
of controlling amount of learned features robustness and the
power of controlling representation data reduction amount
simultaneously. Deep learning based algorithms and
methods are growing fast in a wide range of subjects, such
as, [6], [14], [15], [16] which shows the power of deep
learning.
As mentioned above deep learning has the power of
extracting the best features from our raw data. This key point
of deep learning motivates us for applying deep learning
methods for websites aesthetic evaluation problem. One of
the state of the art deep learning tools for feature extraction
is Convolutional Neural Network [17] which will be referred
to by, CNN. This motivates us to apply the CNNs for
aesthetic evaluation problems. The basic idea is that, given a
screenshot from the website page, which we want to
evaluate, we use a CNN to represent it based on the CNN
learned features, then we dimensionally reduce the extracted
features to achieve better performance in classification,
finally we classify websites using a support vector machine
(SVM). The flowchart of our proposed method is shown in
Fig. 1.
Website screenshot
Feature Representation
using CNN
Feature Dimensionality
Reduction using PCA
Classification using MLP
Fig. 1. Our proposed method consist of three main steps:
Feature representation using CNN, feature dimension reduction, and
SVM classification. In the following paragraphs, these stages
are explained.
A. Feature representation using CNN
In this step, we represent the input raw data using CNN.
There are a wide variety of CNN architectures which have
the state of the art performances,[17] [18] [19] The CNN
architecture we used is based on AlexNet CNN [17].The
training of our CNN is like MemNet [20] that uses pre-
trained form of Hybrid-CNN [21], trained on both Places
dataset [22] and ILSVRC 2012 dataset [17], and fine-tuned
on LaMem dataset [20]. As it can be seen in Fig. 2, The
CNN we used for feature extraction consists of five
convolutional layers, three max pooling layers, and two full-
connected layers (fc6 and fc7). The input dimensions of the
CNN are 227×227×3 (an RGB image with size of 227×227).
In the first convolution layer, a convolution operation with
50
kernel size of 11×11 and stride of 4 performs which outputs
96 feature maps with size of 55×55. Then a max pooling,
pools features to decrease avoid the impact of rotation of the
input image to final features. In second convolution layer,
convolution has been done with kernel size of 5×5, stride of
one, which outputs 256 feature maps with size of 27×27. The
other convolutional layers operate like previous and what is
seen in the Fig. 2. Finally, after five convolutional layers
there are two full-connected layers for abstracting the
features and decreasing the features dimension to 4096. For
extracting the features, we have used layers fc6 and fc7 of
the CNN to evaluate them and choose the best one that is
discussed in next section. Both fc6 and fc7 layers have the
dimension of 4096.
B. Feature dimension reduction
As mentioned above, representing features using CNN
gives us a feature vector with dimension of 4096, which is
computationally expensive for classification step. To solve
the problem, we have used principal component analysis
(PCA) algorithm to reduce the dimension of features
C. SVM classification
Our feature dimension was 128 for each websites. After
obtaining features for image we used Support Vector
Machine [23], with Gaussian radial basis function kernel, as
classifier for training and testing step. The classifier takes
features vector of size 128 for each website as input and
classify them into two categories: good or bad.
IV. EXPERIMENTAL RESULT
In this section, we first introduce websites screenshots
dataset, then our result for all the dataset and for each
country presented and discussed separately.
A. Websites screenshot dataset
We use dataset presented by Intelligent Interactive
Systems Group at Harvard University, which has been used
in [2], [7]. Dataset consist of 430 screenshots captured from
websites’ homepages. All image resolution was 1024×768.
350 websites were English websites, 60 websites obtained
from other languages and 20 websites selected from Webby
Award Candidate [2]. They selected websites to represent
wide variety of genres but unfortunately structure of most
sites were very similar which results in low variety of users’
rating. For evaluating the collected websites’ aesthetic, they
designed an online test that people around the world could
rate complexity, colorfulness and visual appeal of webpages.
Each webpages display 500ms for users so sites contents did
not affected their impression. Data collected from volunteer
between June 2012 and August 2013. Each participant rate
30 random websites two times to ensure that they rated the
sites accurately.
 Fig. 1. Architecture of pretrained CNN
Because the aesthetic of each site varied by opinion of
people, mean rating for not all sites were variant too much.
Fig. 3 shows histogram of sites aesthetic rate.
Fig. 2. Histogram of sites aesthetic rate. Websites with rate zero didn’t
have any rate and excluded from the dataset.
B. Feature representation using CNN
We downsized each image to 227×227 and used all three
categories together. To extracting best features from CNN,
we compare output of layer fc6 and fc7 from pretrained
CNN. For each of layer output we could extract 2048 feature
per sample. Then we trained a SVM classifier for each.
Based on result for all samples and all rates presented in
Table I we choose feature extracted from fc7 because of
lower test error. For implementing CNN and feature
extraction step, we used Caffe deep learning framework [24].
We run our proposed system on a system with a simple
NVIDA GT620 GPU with CUDA compute capability of 2.1.
51
TABLE I. CLASSIFICATION ACCURACY FOR FEATURE OF LAYER FC6 AND
FC7
Mean
Feature extracted Feature extracted
participant
from fc6 from fc7
number
All country 1757 48.78 34.15
C. Feature dimension reduction
Features dimension from fc7 reduced by Principal
Component Analyzing to 128 dimensions. We tested 16, 64,
128 and 256 features as Principal Component. Choosing
lower Principal Components tend to under fitting and large
train error and choosing large Principal Components tend to
over fitting (small train error but large test error). Based on
our experiments and analyses best-reduced size is 128.
D. Classification Result
There was no rating for some sites so we excluded them
from dataset and 418 sites remained. We use 90 percent of
the dataset for training and 10 percent for testing. We train
our SVM classifier for each country separately. Because
each participant did not rate all the websites, some websites
did not have enough rates so we only use countries that have
more than 40 rates for most of websites. At the end, we
calculated rates for all countries. For SVM classification, we
used SVM in MATLAB 2013b. We used Radial basis
function (RBF) for SVM kernel. RBF kernel fits training
data very well and by using big Gaussian parameter we
could have generality so the classifier could separate test
data.
To compare our model to previous work [2] we used
their two features (colorfulness and visual complexity) and
trained SVM classifier for that feature. Then we tested their
classification robustness.
 Table II shows classification error per country for towards understanding aesthetic judgments,” Int. J. Hum. Comput.
deciding good website from bad. In last row of table, we Stud., vol. 70, no. 11, pp. 794–811, Nov. 2012.
show results for trained classifier based on all users' rates. [2] K. Reinecke, T. Yeh, L. Miratrix, R. Mardiko, Y. Zhao,
Results show that our simple model has better J. Liu, and K. Z. Gajos, “Predicting users’ first impressions of
performance for some country and for total people rate. We website aesthetics with a quantification of perceived visual
complexity and colorfulness,” Proc. SIGCHI Conf. Hum. Factors
have the same result in compare to two features, which are
Comput. Syst. - CHI ’13, pp. 2049–2058, 2013.
handcraft. Also, our CNN is pretrained on another
dataset[20] which has another goal with different images. [3] A. N. Tuch, J. A. Bargas-Avila, and K. Opwis,
“Symmetry and aesthetics in website design: It’s a man’s
TABLE II. CLASSIFICATION ACCURACY BY COUNTRY
business,” Comput. Human Behav., vol. 26, no. 6, pp. 1831–1837,
2010.
[4] E. Michailidou, S. Harper, and S. Bechhofer, “Visual
Color and complexity and aesthetic perception of web pages,” in Proceedings
Our model
Mean participant Complexity
Country
number feature Test
Test error of the 26th annual ACM international conference on Design of
(%) communication, 2008, pp. 215–224.
error (%)
[5] Z. DONG, N. Univeristy, A. T. Submitted, F. O. R. The,
D. Of, and M. Of, “Website aesthetics: Does it matter? A study on
United States 727 34.15 31.71
effects of perception of website aesthetics, usability and content
quality on online purchase intention,” 2007.
Germany 50 51.22 39.02
[6] X. Lu, Z. Lin, H. Jin, J. Yang, and J. Z. Wang, “RAPID:
Rating Pictorial Aesthetics using Deep Learning,” in Proceedings
United Kingdom 270 26.83 34.15 of the ACM International Conference on Multimedia - MM ’14,
2014, pp. 457–466.
Canada 101 34.15 46.34 [7] K. Reinecke, A. Arbor, and K. Z. Gajos, “Quantifying
Visual Preferences Around the World,” Chi, pp. 11–20, 2014.
Finland 40 36.59 41.46
[8] G. Lindgaard, G. Fernandes, C. Dudek, J. Brown, and J.
Lindgaard, Gitte and Fernandes, Gary and Dudek, Cathy and
All 1757 34.15 34.15 Brown, “Attention web designers: You have 50 milliseconds to
make a good first impression!,” Behav. Inf. Technol., vol. 25, no.
V. CONCLUSION 2, pp. 115–126, 2006.
[9] H.-Y. Lim, “The Effect of Color in Web Page Design,”
In this paper, we proposed a website aesthetic evaluation Complement, 2003.
method based on CNNs. Using deep learning and CNNs for
[10] G. Lindgaard, “Does emotional appeal determine
feature representation is one of the main tips that made our perceived usability of web sites,” in Proceedings of CybErg: the
system comparable to the previous ones. We showed that a second international cyberspace conference on ergonomics, 1999,
pretrained CNN feature extraction ability that has been pp. 202–211.
trained on another dataset is comparable to the traditional [11] M. Y. Ivory, U. C. Berkeley, R. R. Sinha, and M. a
handcraft feature extraction methods. Furthermore, CNNs Hearst, “Empirically Validated Web Page Design Metrics,” Proc.
have the options of fine-tuning that brings them the ability of SIGCHI Conf. Hum. Factors Comput. Syst., pp. 53–60, 2001.
learning better and better features to achieve the goal we [12] R. Datta and J. Z. Wang, “Algorithmic inferencing of
have chosen for them. Another contribution of this work is aesthetics and emotion in natural images: An exposition,” 2008
considering the target users’ home country for evaluating the 15th IEEE Int. Conf. Image Process., pp. 105–108, 2008.
websites aesthetic especially for them. [13] Z. Jiang, L. Qiu, C. Yi, B. Choi, and D. Zhang, “An
As future work, a better dataset could be collected with investigation of the effects of website aesthetics and usability on
more samples with more variation in aesthetic rating of online shoppers’ purchase intention,” 16th Am. Conf. Inf. Syst.
users. Another work could be done in CNN fine-tuning. The 2010, AMCIS 2010, vol. 7, pp. 5397–5407, 2010.
CNN used in this work is pretrained on both Places dataset [14] M. Fayyaz, M. H. Saffar, M. Sabokrou, M. Hoseini, and
[22] and ILSVRC 2012 dataset [17], and fine-tuned on M. Fathy, “Online signature verification based on feature
LaMem dataset [20]. It can be fine-tuned on a websites’ representation,” in 2015 The International Symposium on Artificial
screenshots dataset to achieve better feature extraction Intelligence and Signal Processing (AISP), 2015, pp. 211–216.
accuracy. Another improvement can be done in classification [15] M. Sabokrou, M. Fathy, M. Hoseini, and R. Klette,
step. By collecting a better dataset with more aesthetic “Real-time anomaly detection and localization in crowded scenes,”
in 2015 IEEE Conference on Computer Vision and Pattern
variant websites, classification can have more classes for
Recognition Workshops (CVPRW), 2015, pp. 56–62.
better evaluation, or it could be converted to regression for
[16] M. Fayyaz, M. Hajizadeh-Saffar, M. Sabokrou, M.
exact rating.
Hoseini, and M. Fathy, “A novel approach for Finger Vein
VI. REFERENCES verification based on self-taught learning,” 2015 9th Iranian
Conference on Machine Vision and Image Processing (MVIP). pp.
[1] A. N. Tuch, E. E. Presslaber, M. Stöcklin, K. Opwis, and 88–91, 2015.
J. a. Bargas-Avila, “The role of visual complexity and
prototypicality regarding first impression of websites: Working

52
[17] A. Krizhevsky, I. Sutskever, and G. Hinton, “Imagenet
classification with deep convolutional neural networks,” Adv.
neural …, pp. 1–9, 2012.
[18] C. Szegedy, W. Liu, Y. Jia, P. Sermanet, S. Reed, D.
Anguelov, D. Erhan, V. Vanhoucke, and A. Rabinovich, “Going
deeper with convolutions,” arXiv Prepr. arXiv1409.4842, 2014.
[19] Y. Lecun, L. Bottou, Y. Bengio, and P. Haffner,
“Gradient-based learning applied to document recognition,” Proc.
IEEE, vol. 86, no. 11, pp. 2278–2324, 1998.
[20] A. O. Aditya Khosla, Akhil S. Raju, Antonio Torralba,
A. Khosla, A. S. Raju, A. Torralba, and A. Oliva, “Understanding
and Predicting Image Memorability at a Large Scale,” IEEE Int.
Conf. Comput. Vis., pp. 2390–2398, 2015.
[21] X.-X. Niu and C. Y. Suen, “A novel hybrid CNN–SVM
classifier for recognizing handwritten digits,” Pattern Recognit.,
vol. 45, no. 4, pp. 1318–1325, 2012.
[22] B. Zhou, A. Lapedriza, J. Xiao, A. Torralba, and A.
Oliva, “Learning deep features for scene recognition using places
database,” in Advances in Neural Information Processing Systems,
2014, pp. 487–495.
[23] C. Cortes and V. Vapnik, “Support-vector networks,”
Mach. Learn., vol. 20, no. 3, pp. 273–297, Sep. 1995.
[24] Y. Jia, E. Shelhamer, J. Donahue, S. Karayev, J. Long,
R. Girshick, S. Guadarrama, and T. Darrell, “Caffe: Convolutional
architecture for fast feature embedding,” in Proceedings of the
ACM International Conference on Multimedia, 2014, pp. 675–678.

53