2012 4th International Conference on Intelligent Human-Machine Systems and Cybernetics
A Web Crawler Detection Algorithm Based on Web Page Member List
Weigang Guo, Yong Zhong, Jianqin Xie
School of Electronic and Information Engineering
Foshan University
Foshan, Guangdong Province, China
1049621@qq.com
Abstract—Following the widely use of search engines, the
impact Web crawlers have on the Web sites should not be
ignored. After analyzing the navigational patterns of Web
crawlers from Web logs, a new algorithm based on Web page
member list is proposed. The algorithm constructs one
member list for every Web page and one show table for every
visitor. The experiment shows that the new algorithm can
detect the unknown crawlers and unfriendly crawlers who do
not obey the Standard for Robot Exclusion.
Keywords- Search engine; Web crawler detection; Web page
member list; Web log
I. INTRODUCTION
A Web crawler(also called Web robot or Web spider) is a
program that automatically traverses the Web's hypertext
structure by retrieving a document, and recursively retrieving
all documents that are referenced. Web crawlers are often
used as resource discovery and retrieval tools for Web search
engines such as Google, Baidu, etc.
But the crawler's automatic visits toward the Web sites
also cause many problems. First, considering the business
secret, many E-commerce Web sites do not hope the
unauthorized crawlers retrieve information from their Web
site. Second, many E-commerce Web sites need to analyze
the visitors’ browsing behavior, but such analysis can be
severely distorted by the presence of Web crawlers [1,2].
Third, many government Web sites also do not hope their
information collected and indexed by the crawlers for some
reason. Fourth, poorly-designed crawlers often consume lots
of network and server resources, affecting the visit of normal
customers. So, it is necessary for the Web site managers to
detect Web crawlers from all the visitors, and take proper
measures to redirect the Web crawlers or stop responding to
HTTP requests coming from the unauthorized crawlers.
The commonly used Web detection method is to set up a
database of known crawlers [3], and compare the IP address
and User Agent fields of the HTTP request message against
the known crawlers. But this method can detect the well-
known crawlers only. There are three simple techniques to
detect the unknown crawlers from Web logs:
(1) According to the SRE (Standard for Robot Exclusion)
[4], whenever a crawler visits a Web site, it should first
request a file called robots.txt. So, by examining the user
sessions generated from Web logs, new crawlers that follow
978-0-7695-4721-3/12 $26.00 © 2012 IEEE 189
DOI 10.1109/IHMSC.2012.54
the SRE can be found. However, the standard is voluntary,
and many crawlers do not obey it.
(2) Most crawlers do not assign any value to the
“referrer” field in their HTTP re-quest messages. And, in the
Web log, the “referrer” field is empty (=“-”). So, if the user
sessions have large number of requests with empty referrer
fields, the visitor is a “suspicious” crawler. But, as Web
browsers can sometimes generate HTTP messages with
empty referrer values, this method is also not reliable.
(3) When checking the validity of the hyperlink structure,
most crawler use HEAD request method to reduce the
burden on Web servers. Therefore, one can examine user
sessions with lots of HEAD requests to discover potential
crawlers. Similarly, as Web browsers can sometimes
generate HEAD request type, this method is also not reliable.
To solve the problem, Pang-Ning Tan and Vipin Kumar
[1] adopted C4.5 decision tree algorithm to classify the
crawlers visit and human visit based on the characteristics of
the crawlers’ access pattern. Their method can effectively
detect the unknown crawlers. But it is a bit complicated. In
this paper, after analyzing the access patterns of Web
crawlers, we propose a new simple but effective algorithm to
detect Web crawlers based on Web page member list.
II. THE DIFFERENCES BETWEEN CRAWLERS
VISIT AND HUMAN VISIT
There are great differences between crawlers visit and
human visit.
(1) when a person inputs a URL address in the browser
(eg. Microsoft Internet Explorer), the browser will send out
the HTTP request to the target server. According to the
HTTP protocol [5], the server will check whether it has the
document specified by the URL after it has received the
request. If it does have, it sends out that document.
Otherwise, it gives out an error message. And, the browser
will parse the document after receiving the document sent by
the server. If it is a single document, such as a picture etc.,
the browser shows it directly. If it is an HTML document,
then the browser will analyze the embedded and linked
objects in this document (such as image files, animation
files, scripts files, cascading style sheet files and frames,
etc.), and then continuously and automatically send out the
HTTP requests to the server until all the embedded objects
have been requested. On the server side, the server sends out
all the requested documents in order after receiving the
client’s requests. When the browser has received all the
embedded objects, it “assembles” them and then generates a
complete Web page from the human point of view. So, one
request of a person may generate several records in Web
server logs, and all the embedded objects “show” in the log.
(2) The crawler is different. Usually, after getting a URL
(assuming it is an html document) from the URL list which
is waiting for being visited, the crawler sends HTTP request
to the target server. The crawler also analyzes the embedded
objects and hyperlinks within the received HTML document
after receiving the server's response, and then adds the
embedded hyperlinks to the waiting list according to the its
visiting rules. Here, the method of treating embedded objects
(such as image files, animation files, scripts files, cascading
style sheet files and frames, etc.) may be different
respectively. Some search engine crawlers add the URLs of
these objects to the waiting list as well, while some give
them up directly or modify the links of the objects in the
HTML document rather than requested them straightly. But
one thing is same that the crawlers don’t send the requests
for the embedded objects to the server at once. Therefore one
request of crawler visit only leaves one piece of record in the
server logs, which exactly represents the request of the
crawler.
III. CONSTRUCTION OF WEB PAGE MEMBER LIST
Definition 1: A Web page is a collection of information,
consisting of one or more Web resources, intended to be
rendered simultaneously, and identified by a single URL.
More specifically, a Web page consists of a Web resource
with zero, one, or more embedded Web resources intended to
be rendered as a single unit, and referred to by the URL of
the one Web resource which is not embedded [6].
Definition 2: The member list of a Web page is defined
by a three-tuple:
t=<webpage, memeberset, n>
where, webpage is the URL of the Web page, memberset
is the aggregate of all the URLs of the embedded objects and
is represented as {m1,m2,Ă,mn}ˈmi˄i=1,Ă,n˅is the
URL of the embedded objects, n is the number of the
members.
The embedded objects mi include: 1) Multimedia files
(images, sounds, animations, etc) defined by the SRC
attribute of IMGǃBGSOUNDǃEMBEDǃOBJECT tags
of HTML. 2) Frames defined by the SRC attribute of
FRAME ੠ IFRAME tags of HTML. 3) Cascading style
sheet files linked by the HREF attribute of LINK tag of
HTML. 4) Scripts files linked by the SRC attribute of
SCRIPT tag of HTML. 5) Java applet class files linked by
the CODE attribute of APPLET tag of HTML.
For example, if one Web page sample.htm is consisted of
two frames: list.htm, welcome.htm, and every frame has one
picture named littledog.jpg, bigdog.jpg respectively, and
there is a cascading style sheet file style.css in welcome.htm,
then, supposing all the files are in the root directory of the
Web site, the member list of sample.htm is:
<sample.htm, {list.htm, welcome.htm, littledog.jpg,
bigdog.jpg, style.css}, 5>
The number of multimedia files’ member is 0.
190
We developed an HTML analyzer to analysis the HTML
tags and its attributes and generate the Web page member list
for every file in the Web site. All the member lists consists
the member list set of the Web site.
IV. THE DETECTION ALGORITHM
Step1: data preprocessing. Sort the Web logs by IP field,
agent field and time field as the first, second and third
keywords, and then treat the records with same IP and agent
field as one visitor’s visiting records and assign a label uid.
Each record of the log files can be processed as:
R=<uid, url, time>
Where uid is the label of every different visitor, url is the
requested URL resources, and time is the request time. So,
the visiting records set of the user is represented as:
S = <uid, {(url1,time1),…,(urlk,timek)}>
Where, k is the total number of visiting records.
Step2: Constructing a ShowTable(see Table I) which
records actual attendance of the members of the visited Web
pages for every visitor. In the ShowTable, url is URL of the
Web page, NumberOfMember is the number of member of
the Web page and is obtained from the Web site’s Web page
member list, ShowNumber is the total number of members
who appear in his visiting records set.
The algorithm of computing ShowNumber is as follows:
for each ręS do
if the URL type of r is multimedia files(images, sounds)
then ShowNumber := 0 ;
else
for each member of r do
{ judge whether this member appear in its close
succeeding sequence;
if it does appear
then { ShowNumber := ShowNumber + 1;
delete this member’s record}
next member;
end for
next r;
end for
TABLE I. SHOWTABLE: RECORDS ACTUAL ATTENDANCE OF THE
MEMBERS OF THE VISITED WEB PAGES FOR EVERY VISITOR.
url NumberOfMember ShowNumber
index.htm 6 0
introduction.htm 1 0
myideas.htm 4 0
bigdog.jpg 0 0
Ă … …
Where, the close succeeding sequence of r means all the
visiting records behind r in a certain time interval in the
visiting records set S. The interval can be from 0 to 30
seconds. The reasons is that, if the visitor is a person, the
browser usually requests the embedded objects in 0~5
seconds, and the request intervals will not be over 30
seconds, otherwise the visitor will be impatient and give up
or exit. For crawler visits, according to its retrieving strategy,
if the crawler wants to request the embedded objects, the TABLE III. THE RESULT CRAWLERS DETECTED FROM THE LOGS
time interval is usually large than 30 seconds.
IP address Agent Visiting robots.txt
Step 3: To judge whether one uid is a crawler, the simple or not
method is to check the ShowNumber field of its ShowTable.
If all the corresponding ShowNumber fields of its visited 64.68.82.135 Googlebot yes
URLs equal 0, we could think the uid is a crawler. 202.108.249.130 BaiduSpider yes

V. EXPERIMENTS 192.160.51.70 LinkScan/11.0+Unix no

Our Experiments were performed on Foshan University 210.72.21.199 HTML_GET_APP yes

server logs (http://202.192.168.175) collected from July 1st 138.15.164.14 Lachesis no
to July 31st, 2009. The structure of the Web site is simple,
and Web pages of this site contain only text and small 138.15.164.37 - no
amount of images, sounds, frames, Java applets, JavaScript 138.15.164.25 - no
files, cascading style sheet files. The member lists of the
Web pages of top two layers of the Web site are shown in TABLE IV. THE SHOWTABLE OF GOOGLEBOT AND BAIDUSPIDER
Table II.
The Web logs of July 1st to July 31st, 2009 contain a URL NumberOfMember ShowNumber
total of 30328 records. Using the detection algorithm /index.htm 6 0
proposed in this paper, we got five crawlers from the Web
logs, which are shown in Table III. The ShowTables of /media/index.htm 1 0
Googlebot (IP:64.68.82.135) and BaiduSpider /search/index.htm 3 0
(IP:202.108.249.130) of their visited Web pages of top two
/myideas/index.htm 2 0
layers are shown in Table IV. Table V and Table VI show
the ShowTable of anonymous agent(-) with IP address /students/index.htm 2 0
138.15.164.37 and 138.15.164.25. We can see that the /cheng/index.htm 1 0
crawler from 138.15.164.37 only retrieves image files while
the crawler from 138.15.164.25 only retrieves MP3 music /ella/index.htm 2 0
files. And, these two IP address are maybe in a same local /flash/index.htm 3 0
area network with IP 138.15.164.14(its agent is Lachesis).
The reasonable explanation should be that IP 138.15.164.14 TABLE V. THE SHOWTABLE OF IP: 138.15.164.37
only retrieves HTML documents and analyzes image file
sources and music file sources. After having got the URLs, URL NumberOfMember ShowNumber
then IP 138.15.164.37 and 138.15.164.25 crawls image files
/media/images/dian2.jpg 0 0
and music files respectively. These three IP should be
owned by one same organization or company. /media/images/diehe3.jpg 0 0
/media/images/diehe3.jpg 0 0
TABLE II. THE MEMBER LISTS OF THE WEB PAGES OF THE TOP TWO
LAYERS OF THE WEB SITE (HTTP://202.192.168.175) /media/images/diehe3.jpg 0 0

URL Number of Type of members /cheng/041059/03.gif 0 0

members /Ella/Shopping/image001.gif 0 0
/index.htm 6 image files: 3 ……
sound files:1
CSS files: 1
JavaScript files:1 TABLE VI. THE SHOWTABLE OF IP: 138.15.164.25
/media/index.htm 1 image files: 1 URL NumberOfMember ShowNumber
/search/index.htm 3 image files: 1 /mp3/whitetree.mp3 0 0
frame files:2
/mp3/songtoremember.mp3 0 0
/myideas/index.htm 2 image files: 1
CSS files: 1 /mp3/Sunflowers.mp3 0 0
/students/index.htm 2 image files: 2 /mp3/ThoseFlowers.mp3 0 0
/cheng/index.htm 1 image files: 1 ……
/ella/index.htm 2 image files: 1
Java Applet files:1 VI. CONCLUSIONS
/mp3/index.htm 3 image files: 2 Our detection algorithm is simple, but it is effective and
MP3 files:1 has a high accurateness, and need only a few records to
detect whether the visitor is a crawler. The weakness of the

191
algorithm is that if the Web pages of the Web site only
contain plain text and no images, sounds, the algorithm
maybe regards a human visitor as a crawler. And if a person
uses very simple browser or set his browser not displaying
images and not playing sounds, the algorithm may mistake a
human visit for a crawler visit. For future work, we would
like to take into account objects and hyperlinks of the Web
page to detect the search engine crawlers more effectively.
REFERENCES
[1] Pang-Ning Tan, Vipin Kumar. “Discovery of Web robot sessions
based on their navigational patterns, ” Data Mining and Knowledge
Discovery, Vol. 6, pp.9-35, January 2002.
[2] Omer Duskin Dror G. Feitelson. “Distinguishing humans from robots
in Web search logs: Preliminary Results Using Query Rates and
Intervals,” Workshop on Web Search Click Data ’09 Barcelona,
Spain,2009.
[3] The Web robots database. http://www.robotstxt.org/wc/active.html
[4] Robots exclusion. http://www.robotstxt.org/wc/exclusion.html
[5] Hypertext transfer protocol- HTTP/1.1. http://www.w3.org
[6] Web characterization terminology & Definitions Sheet.
http://www.w3.org /1999/05/WCA-terms/01/

192