Fuzzy Based Approach to URL Assignment in
Dynamic Web Crawler
Raghav Sharma
Computer Science & Engineering Department
PEC University of Technology
Chandigarh, India
rghvsharma4@gmail.com
Abstract— WWW is a huge collection of unorganized
documents. Web Crawler is the process used by the search
engines to build the database from this unorganized web. The
crawler which interacts with millions of web pages has to be
made efficient in order to make a search engine powerful. This
necessitates parallelization of web crawlers to enhance the
download rate due to the fast increasing size of the web. The
paper review different parallel web crawling techniques in the
literature. The paper proposes an approach for URL assignment
in dynamic parallel web crawler using fuzzy logic which
addresses two important aspects of a crawler: first one to create
crawling framework with load balancing among parallel
crawlers. The second aspect is to make crawling process fast by
using parallel crawlers with efficient network access.
Keywords— Static Parallel Crawler, Dynamic Parallel
Crawler, Fuzzy Logic.
I. INTRODUCTION
A crawler is a program that downloads and stores Web
pages, often for a Web search engine. A crawler plays a vital
role in data mining algorithms in many fields of research e.g.
mining of twitter data for opinion mining or finding the success
ratio in project funding sites like Kickstart [1, 2]. Generally, a
web crawler starts its work from a single seed URL keeping it
into a queue Q0, where it keeps all the URLs to be extracted.
From there, it extracts a URL based on some ordering and
downloads that page, extracts any URL in the downloaded
page and put them in the same queue. It repeats this function
until it is stopped.
The major difficulty for a single process web crawler is that
crawling the web may take months and in the mean time a
number of web pages may have changed and thus, not useful
for the end users. So, to minimize the download time, search
engines execute multiple crawlers simultaneously known as
parallel web crawlers.
An appropriate architecture of a parallel crawler demands
the overlapping of the downloaded web pages by different
parallel agents to be negligible. Further, the coverage rate of
the web should not be compromised within each parallel agents
range. Next, the quality of web crawled should not be less than
a single centralized crawler [4]. To achieve all these,
communication overhead should be taken into account to
Rajesh Bhatia
Computer Science & Engineering Department
PEC University of Technology
Chandigarh, India
rbhatiapatiala@gmail.com
achieve a tradeoff among the objectives to build an optimized
crawler.
Besides these challenges, the advantages of parallel crawler
over single process crawler are [4]:
 Scalability: With millions of pages been added to the
web daily, it’s almost impossible to crawl the web by a
single process crawler.
 Network Load Dispersion: With parallel crawlers, we
can disperse the load to multiple regions rather than
overloading one local network.
 Network Load Reduction: Allowing parallel agents to
crawl specific local data (of the same country or region
of that of crawler), the pages will have to go through
local network, thereby reducing the network load.
Further, to reduce the overlapping of the downloaded pages
by parallel crawlers, the parallel agents need to coordinate. On
that basis, the parallel crawler can be implemented in three
ways [4]:
 Static Parallel Crawler: In this, the web is partitioned
by some logic and each crawler knows its own partition
to crawl. So, there is no need of the central coordinator.
 Dynamic Parallel Crawler: In this, there is a central
coordinator which assigns the URL to different parallel
agents based on some logic i.e. the web is partitioned
by the central coordinator at the run time.
 Independent Parallel Crawler: Here, there is no
coordination among the parallel agents. Each parallel
agent continues crawling from its own seed URL. So,
the overlap can be significant in this case unless the
domain of crawl agents is limited and entirely different
for each crawl agent.
A. Static Parallel Crawler
As discussed, there is no need of the central coordinator for
the static parallel crawler. Instead, we need a good partitioning
method to partition the web before the crawling. A number of
partitioning scheme have been proposed, as follow [5]:
 URL Hash Based: In this scheme, the page is sent to a
parallel agent based on the hash value of the URL. So,
 in between a crawl, a parallel agent may not be able to
crawl URL of the same site due to different hash value
leading to interpartition link.
 Site Hash Based: In this scheme, the hash value is
calculated only on the site name of the URL. So, the
URLs of the same site will be crawled by the same
parallel agents, resulting in less interpartition links and
further, less communication bandwidth.
 Hierarchical: Here, partitioning is done on the basis of
issues like country, language or the type of URL
extension.
One concern in the literature of Static Parallel Crawler is the
mode of job division among the parallel agents. There are
different modes of job division like firewall, crossover, and
exchange [5]. Under the first mode, the parallel agents crawl
pages in its partition only, neglecting the interpartition links.
Under the second mode, the parallel agents primarily crawls
only same partition links and if there are no more links left to
crawl in the same partition, then it moves to interpartition links
to crawl them. Under the third mode, parallel agents
communicate through message exchanges whenever they
encounter an interpartition link to increase the coverage and
decrease overlap.
The drawbacks of the static parallel crawler are as follow:
 Scalability: In order to reduce the overlap and increase
the coverage, there should be N! connections for URL
transfer to appropriate parallel agent, in case the
number of parallel crawlers are N.
 Quality of web pages crawled: Here, each parallel
agent is unaware of the web crawled by other agents.
So, they don’t have the global image of the web
crawled and thus, the decision of URL selection is
entirely based on the subset of web crawled, which is
nothing but the web crawled by the parallel crawl
agent.
B. Dynamic Parallel Crawler
As discussed, there is a central coordinator to manage the
assignment of URLs to different crawler agents in case of
dynamic parallel crawler. The architecture of dynamic web
crawler is as follow:
Figure 1: Architecture of Dynamic Parallel Crawler
The dynamic parallel crawler starts its working from the
central coordinator as depicted by the figure 1. Each crawl
agent behaves as a separate single process crawler receiving
the seed URL for its domain from the central coordinator. It
then downloads the page from the web and extracts the URL
links from the downloaded page and sends them to the central
coordinator for assignment, in case it is outside the domain of
the crawl agent. The domain of each parallel agent is
implementation specific. Further, the dynamic parallel crawler
has a number of advantages which are explained as follow:
 Crawling Decision: Static parallel crawler suffers from
poor crawling decision i.e. which web page to crawl
next. This is because no crawling agent has complete
view of the web crawled. But in dynamic parallel
crawler, the central coordinator has the global image of
the web and the decision about the URL selection and
assignment is taken by the central coordinator, not the
crawl agents [6].
 Scalability : In case of dynamic parallel crawlers only
N connections are needed with the central coordinator
for URL assignments, in case the number of crawl
agents are N. If a new crawl agent is added to the
system, only one socket connection will be required
between the crawl agent and the central coordinator.
 Minimizing Web Server Load: One important aspect of
a web crawler is that it should not overload a server
with its requests. It has been observed that a web page
contains a number of links to the pages of same web
server. The crawl agent sends the URL links to the
central coordinator which sends the most important
unvisited link to the crawl agents. It is not possible that
all the pages of same server would be always
important. This leads to the decreased load on a single
web server.
Dynamic web crawler design poses a number of challenges
too which need to be addressed here:
 Which distribution algorithm should be used for URL
assignment?
 How to distribute jobs to different crawlers based on
their health i.e. the crawler selection for the URL to
optimize load balancing?
 How to manage the already crawled pages to avoid
replication of pages in database?
The main objective of this paper is the URL assignment
strategies which is one of the important functionalities of the
dynamic parallel crawler.
The paper is organized as follow: Related work in the
dynamic URL assignment is explained in section II. Section III
describes proposed fuzzy technique for URL assignment.
Section IV describes the fuzzy phase of the technique
including the benefits of the proposed architecture. Section V
concludes the paper.
 II. RELATED WORK
A. Hash Based Approach for URL Assignment
The basic approach for the hash based URL assignment is
using the key value of each URL to determine its crawl agent
for parsing. [7] proposed the architecture for URL assignment
by transforming the URL into a set of numerical information
which represent coordinates of 3D space vector(x,y,z). Using
such transformation of URL, a number of values can be
generated from a single URL. [7] used URI 3986 standard
definition to split URL.
Uniform Resource Identifier (URI) is a string of characters
used to locate the host name in the internet. In [7], the URL
strings have been defined in 3 ways i.e. scheme, domain and
path, query and fragment. These elements are the coordinate
function in vector space. If URL is
“http://www.abc.com/index.php?q=1#session”, it corresponds
to coordinates as:
URI Parts Substring Coordinate
Scheme & http://www.ab X=3487
Domain c.com
Path /index.php? Y=2241
Query & q=1#session Z=744
Fragment
Table 1: Coordinates of URI parts
In this way, the URI structure is transformed into 3D
vector space over which a fuzzy clustering technique can be
applied for the assignment of particular URL to the specified
crawl agent. The advantage of this scheme is that it is easy to
implement but it does not reflect the locality structure of the
links.
B. Domain Specific Approach for URL Assignment
One of the proposed approach for the dynamic partitioning
of the web is based on the domain of the crawl agents which is
influenced by the fact that web pages are more likely to link to
pages that are relevant to domain of the same page [8]. So,
after retrieving a web page, the crawl agent has to perform
analysis to predict its relevancy to one of the many domains of
all crawl agents. Thus, breaking down a domain into sub-
domains can add up new crawl agents increasing the
scalability of the system [8].
The domain oriented partitioning approach requires the
initial seed URLs of various domains to be gathered which can
be represented by some hub pages which consists of primarily
links to different pages highly relevant to various domains.
Once the web page is downloaded, it is fed to parser,
classifier and dispatcher module. The role of the parser
module is to extract various HTML components of the page to
extract the list of new unvisited URLs from the page specified
in the href attribute of the anchor tag. Later, the classifier
module analyses the domain of the web page and adds it to the
associated repository of its domain. It also tags the URL of the
downloaded page with its domain in the same database which
is used to store the unvisited URLs.
Finally, the URL dispatcher sends the discovered
URLs to the central coordinator performing following steps
[8]:
 It restores the relative address of hyperlinks to
absolute address which is important as a number of
documents can refer to same URL.
 It tries to predict the domain of the discovered URLs
by the help of tagged URLs which can serve as the
source of the newly discovered URLs reflecting the
hyperlinked behavior of the web i.e. web pages are
most likely to link to pages of same domain.
 It can check for the duplication of the discovered
pages with the pages of the same domain partition
pool which have been visited.
B. Virtualized URL Assignment
In this approach, virtualization concept is used for the
working of parallel crawlers. Here, multiple cores of multicore
processors are treated as virtual machines which interact with
each other through a shared region or memory using VMCI
(Virtual Machine Communication Interface). These virtual
machines are also treated as clusters and URLs belonging to
same clusters are served by virtual machines of corresponding
clusters [9]. Initially, there is a need of an injector module to
provide the seed URLs which are used by clustering module
for cluster formation. The clustering module calculates the
hash value of the URLs and by using the URI; the cluster is
identified to which the URL belongs. Then, the URL is
assigned to a virtual machine depending upon the threshold
value of that machine which is decided as per the availability
of the virtual machine.
III. PROPOSED FUZZY BASED APPROACH FOR URL
ASSIGNMENT
After doing systematic literature survey of parallel crawlers
and the material associated with the issue of assignment of
URL problems, we can safely say that limited work has been
done to exploit the usage of Fuzzy Logic in parallel crawlers.
In this work, we propose a parallel architecture, where the
systems can be geographically distributed. As shown in figure
2, the system has three main components: a fuzzy logic
controller, a URL distributor, and some parallel agents.
Figure 2: Architecture of a Parallel Web Crawler
 a) Fuzzy Logic Controller
The main task of this component is of load optimization
among the parallel agents by monitoring the health of parallel
agents in regular intervals. The approach will be discussed
deeply in section IV.
b) URL Distributor
The URL distributor selects a set of URLs from the
database and by the use of fuzzy logic controller, it distributes
them to different parallel agents optimizing the load. It
connects crawlers using HTTP connection and receives,
analyzes and store their result after aggregation into the central
repository.
c) Parallel Agents
Each parallel agent works like a single centralized crawler
starting from a single URL or a set of URLs and analyze every
page specified by each URL. The link extractor analyses the
page to identify the links and make a list of them. The page
downloader takes a new URL from this list and downloads the
page from the internet if it’s not there in the local cache. It then
sends this page for analysis to link extractor again. The parallel
agents keep track of their health based on some attributes
depicted in figure 2 which will be discussed in the next section.
Further, they send the list of URLs with the corresponding
health to the central coordinator after regular time intervals
(crawl session) for the fuzzy logic controller to distribute the
load according to the health of the machine.
IV. FUZZY PHASE
In current context, the use of fuzzy logic can be exploited
for doing multiple tasks including:
a) Assignment of the URL to the priority queue for
crawling sessions.
b) Assignment of the URL to the machine or core used
for invoking crawling sessions in the parallel crawlers.
The fuzzy logic controller is responsible for the
implementation of fuzzy logic for doing above tasks. The
technique takes the following assumptions as shown in the
table in figure 2:
 “A” be the Last Average Page Size in kb downloaded
by a machine “m”.
 “Td” be the Last Average Time in seconds taken by a
machine “m” to download “p” Pages.
 “Tdd” be the Last Average Time in seconds taken by
a machine “m” to save the “p” pages to the disk “d”.
 “Tdns” be the Last Average Time in seconds taken by
a machine “m” for DNS resolution.
 “AL” = {normal, small, large} be the set of Lingustic
Variable set describing the “A” having discrete range
of values.
 “TdL” = {less, more} be the set of Lingustic Variable
set describing the “Td” having discrete range of values.
 “TddL” = {less, more} be the set of Linguistic
Variable set describing the “Tdd” having discrete range
of values.
 “TdnsL” = {less, more} be the set of Linguistic
Variable set describing the “Tdns” having discrete
range of values.
So, the Input-Set is {AL,TdL,TddL,TdnsL} and the
Output-Set is {Mc1,Mc2,Mc3} where Mc1,Mc2,Mc3 are the
parallel crawling agents.
Since, fuzzy logic is a form of many-valued logic, it deals
with the reasoning i.e. approximate rather than fixed [10] and
exact, therefore, this process will undergo steps which include:
a) Determination of input variables explained above.
b) Process of fuzzification already explained above in
terms of linguistic variables and defining the values of
fuzzy set with their range. This is done with help of
membership functions which will have particular
shape based on the distribution of the input variables
which may be triangle, for instance.
c) For each set of input values, run inference engine,
which means what rules can guide or set of rules that
activates(Antecedents) the fuzzy controller
(Modification of Consequents ) to assign the URLs to
the queue “q” (Accumulation) of a particular machine
“m”, for instance. Inferences can be derived as:
 “If AveragePageSize is “small” and “TdL” is not
“more”, then Machine is Machine1”.
 “If AveragePageSize is “large” and “Tdd” is
“more”, then Machine is Machine2”.
d) Defuzzification (Converting the fuzzy outputs into
Crisp values by calculating centroid or maxima for
example) and finally the output, which in our case will
be the machine index to which url will be assigned to.
The process of defuzzification is infact the process of
producing a quantifiable result in fuzzy logic, given
fuzzy sets and corresponding membership degrees.
e) Evaluate performance of the crawling agents based on
the defuzzified output.
Merits of this approach in URL assignment problem of the
parallel crawler are:
 It supports the load balancing property of the parallel
crawler agents which is required to maintain
equilibrium.
 It takes into account the external features like the time
taken for DNS resolution which includes the network
congestion at a particular span of time.
 The health of the crawl agents is monitored at regular
time intervals through which the system can be scaled
 up all time in case there is unbalanced state due to
uncontrolled behavior of the web, thus increasing the
robustness of the system.
CONCLUSION
In this paper, we have reviewed some basic concepts of
parallel web crawler along with the implementation of parallel
crawler in static and dynamic way. The static parallel crawlers,
as discussed are simple to build but have a number of
drawbacks which are overcome by the dynamic behavior of
parallel crawler but with the difficulty of implementation of its
modules. Though, very less work has been focused on the
dynamic parallel crawler in the literature, this paper discusses
the different architectures of the important phase of dynamic
parallel crawler i.e. how to distribute URLs from the URL
frontier to the various concurrently executing crawling process
threads which is an orthogonal problem(URL Assignment
Problem). Finally, a new approach for URL assignment to
crawl agents based on their health monitoring is proposed
through fuzzy logic which provides with a number of
advantages.
References
[1] Etter, Vincent, Matthias Grossglauser, and Patrick Thiran, “Launch hard
or go home!: predicting the success of kickstarter campaigns.”
Proceedings of the first ACM conference on Online social networks.
ACM, 2013.
[2] Pak, Alexander, and Patrick Paroubek, “Twitter as a Corpus for
Sentiment Analysis and Opinion Mining.” LREC, 2010.
[3] Divakar Yadav, AK Sharma, Sonia, Jorge Marato, “An approach to
design incremental parallel web crawler.” Journal of Theoretical and
Applied Information Technology Vol 43,2012.
[4] Garcia-Molina and Junghoo Chu, “Parallel Crawlers.” Proceedings of
the 11th international conference on World Wide Web, 2002.
[5] Fatemeh, Ali Sehmat, “ An architecture for a focused trend Parallel Web
crawler with the application of clickstream analysis.” Information
Sciences Vol 184 Elsevier,2011.
[6] Debajyoti Mukhopadhyay, Sajal Mukherje, Soumya Ghosh, Saheli Kar,
Young-Chon Kim, “Architecture of A Scalable Dynamic Parallel
WebCrawler with High Speed Downloadable Capability for a Web
Search Engine.” 6th International Workshop on MSPT Proceedings,
2006.
[7] A.Guerriero, F. Ragni, C. Martines, “A dynamic URL assignment
method for parallel web crawler.” Computational Intelligence for
Measurement Systems and Applications (CIMSA), IEEE International
Conference on IEEE, 2010.
[8] Gupta, Sonali, Komal Bhatia, and Pikakshi Manchanda, “WebParF: A
Web Partitioning Framework for Parallel Crawler.” International Journal
on Computer Science and Engineering, Aug 2013.
[9] Bhaginath, Wani Rohit, Sandip Shingade, and Mahesh Shirole,
“Virtualized dynamic URL assignment web crawling model.” Advances
in Engineering and Technology Research (ICAETR), 2014 International
Conference on. IEEE, 2014.
[10] Rondeau, L., R. Ruelas, L. Levrat, and M. Lamotte, “A defuzzification
method respecting the fuzzification.” Fuzzy sets and systems 86, no. 3
(1997): 311-320.
[11] Y. Wan, H. Tong, “URL Assignment Algorithm of Crawler in Distributed
System Based on Hash. “ IEEE International Conference on
Networking, Sensing and Control, ICNSC 2008, Hainan, China, 6-8
April 2008. pages 1632-1635, IEEE, 2008.
[12] Debajyoti Mukhopadhyay, Sajal Mukherje, Soumya Ghosh, Saheli Kar,
Young-Chon Kim, “Architecture of A Scalable Dynamic Parallel
WebCrawler with High Speed Downloadable Capability for a Web
Search Engine.” 6th International Workshop on MSPT Proceedings,
2006.
[13] Huang, Qiuyan, Qingzhong Li, and Zhongmin Yan, “A Novel URL
Assignment Model Based on Multi-objective Decision Making
Method.” In Web Information Systems and Applications Conference
(WISA), 2012 Ninth, pp. 31-34. IEEE, 2012.
[14] Marin, Mauricio, Rodrigo Paredes, and Carolina Bonacic. “High-
performance priority queues for parallel crawlers.” Proceedings of the
10th ACM workshop on Web information and data management. ACM,
2008.
[15] Y. Wan, H. Tong, “URL Assignment Algorithm of Crawler in Distributed
System Based on Hash.” IEEE International Conference on Networking,
Sensing and Control, ICNSC 2008, Hainan, China, 6-8 April 2008.
pages 1632-1635, IEEE, 2008.