10th International Conference on Information Technology
Change Detection in Web Pages
Divakar Yadav A.K.Sharma
JIIT University, Noida (India) YMCA, Faridabad (India)
dsy99@rediffmail.com ashokkale2@rediffmail.com
Abstract
A large amount of new information is posted
on the Web every day. We can take the example of the
news portals, which keep on changing not only each
and every day but also within each and every hour.
Now, which information or data is of how much
importance depends upon the perception of the specific
user. The Internet and the World Wide Web have
enabled a publishing explosion of useful online
information, which has produced the unfortunate side
effect of information overload: it is increasingly
difficult for individuals to keep abreast of fresh
information.
In this paper, we describe an approach for building a
system for efficiently monitoring changes to Web
documents. We discuss the mechanism that our
proposed algorithm uses to discover and detect
changes to the Web pages efficiently. Our solution for
searching new information from the web page by
tracking the changes in web document’s structure has
been discussed. In the methodology section, we present
the algorithm and technique useful for detecting web
pages that are changed, extracting changes from
different versions of a web page, and evaluating the
significance of web changes. Our algorithm for
extracting web changes consists of three steps:
document tree construction, document tree encoding
and tree matching (based upon the concept of R.M.S.
value of the content), for the detection of two types of
changes basically -- structural changes and content
changes. It has linear time complexity and extracts
effectively the changed content from different versions
of a web page.
1. Introduction
The World Wide Web (the Web), as one of the
most popular applications on the Internet, continues to
grow at an astounding speed. Not only the size of static
web pages increases approximately 15% per month,
the number of dynamic pages generated by programs
has been growing exponentially. The rapid growth of
the Web has affected the ways in which fresh
information is delivered and disseminated.
0-7695-3068-0/07 $25.00 © 2007 IEEE
DOI 10.1109/ICIT.2007.37
J.P.Gupta
JIIT University, Noida (India
jp.gupta@jiit.ac.in
Instead of having users tracking when to visit web
pages of interest and identifying what and how the
page of interest has been changed, the information
change monitoring service is becoming increasingly
popular, thus enabling information to be delivered
while they are still fresh. Several tools are available to
assist users in tracking changes in web pages of
interests. Examples of tools are Timely Web and URLy
Warning [16, 17]. In this type of scenario, it becomes
natural that these tools are getting the popularity.
The Web provides access to a wide variety of
information but much of this information is fluid; it
changes, moves, and occasionally disappears.
Bookmarks, paths over Web pages, and catalogs like
Yahoo! are examples of page collections that can
become out-of-date as continuous changes are made to
their components. Maintaining these collections
requires that they should be updated continuously.
Tools to help in this maintenance require an
understanding of what changes are important and what
changes are not.
So keeping all these aspects of web in mind, in this
paper we are proposing a scheme to detect the changes
in the web documents. In section 3.1, we identify and
discuss the types of changes that can take place in the
web pages. Section 3.2 and 3.3 discuss the proposed
solution for the problems identified in the previous
section. Here we make use of the tree structure of the
web documents and traverse down the tree in order to
detect the changes in the structure as well as in the
content of the web document. Instead of traveling
down the whole tree we make use of the levels in the
tree structure in order to detect the addition or deletion
of the node at a specific level or levels. In order to
detect the change in the content we make use of the
R.M.S. value of the ASCII values of the characters of
the text extracted from the web document. They can be
in the form of the paragraphs, headers or any other
form of text.
2. Related work
Now the question which arises in our minds as soon as
changes in the web pages becomes the centre of
concern is what makes retrieving and managing web
changes an effective method for retrieving new
259
265
information from the web? Some of the studies show
that although the Web is growing and changing fast,
the absolute amount of changed content on existing
web pages during a short period is significantly smaller
than the total amount of content in the Web.
Two recent studies as mentioned below show the need
for management of changes in web pages.
around 50%) of web pages remain completely
unchanged during the entire period they studied. To
measure the degree of change, they compute the
between different versions of web
documents. They show that many of the pages that do
change undergo only minor changes in their content:
even after a whole year, 50% of the changed pages are
less than 5% different from their initial version.
Fretterly [19] performed a large crawl that
downloaded 151 million HTML pages. They then
attempted to fetch each of these 151 million HTML
pages ten more times over a span of ten weeks during
Dec. 2002 to Mar. 2003. For each version of each
document, they compute the checksum and shingles to
measure the degree of change. The degree of change is
categorized into 6 groups: complete change (no
common singles), large change (less than 30%
common shingles), medium change (30%-70%
common shingles), small change (70%-99% common
shingles), no text change (100% common shingles),
and no change (same checksum). Experiments show
that about 76% of all pages fall into the groups of no
text change and no change. The percentage for the
group of small change is around 16% while the
percentage for groups of complete change and large
change is only 3%. The above results are very
supportive to our studies. They suggest that
incremental method may be very effective in updating
web indexes, and that searching for new information
appearing on the web by retrieving the changes will
require a small amount of data processing as compared
to the huge size of the Web.
It has been discussed in [6] how to estimate the
change frequency of a web page by revisiting the page
periodically. A study [1] has been done on how often a
crawler should visit a page when it knows how often
the page changes. Their initial experiment tries to
answer the following questions about the evolving
web:
1) How often does a web page change?
2) What is the lifespan of a page?
3) How long does it take for 50% of the web to
change?
4) Can we des cribe changes of web pages by a
mathematical model?
References [1] and [2] experimentally study how
often web pages change. Reference [1] and [5] studies
the relationship between the desirability of a page and
its lifespan. Some papers investigate page changes to
266
260
Ntoulas [18] collected a historical database
for the web by downloading 154 popular Web sites
(e.g., acm.org, hp.com and oreilly.com) every week
from October 2002 until October 2003, for a total of 51
weeks. The average number of web pages downloaded
weekly was 4.4 million. The experiments show that a
significant fraction (
shingles of each document and measure the difference
of shingles
improve web caching policies. Comparing with the
research in active databases, the WebCQ [11] system
differs in three ways: First, the WebCQ system targets
at monitoring and tracking changes to arbitrary web
pages. Second, WebCQ monitors data provided by the
content providers on remote servers, and WebCQ
monitoring and tracking service requires neither the
control over the data it monitors nor the structural
information about the data it is tracking.
A study [8] by Rocco, Butler, Luis has
contributed to this field by developing a mechanism for
efficient storage and processing of Web documents.
The Page Digest design encourages a clean separation
of the structural elements of Web documents from their
content. Its encoding transformation produces many of
the advantages of traditional string digest schemes yet
remains invertible without introducing significant
additional cost or complexity. Using the Page Digest
encoding can provide at least an order of magnitude
speedup when traversing a Web document as compared
to using a standard Document Object Model
implementation. The experiments show that change
detection using Page Digest operates in linear time,
offering 75% improvement in execution performance
compared with existing systems. In addition, the Page
Digest encoding can reduce the tag name redundancy
found in Web documents, allowing 30% to 50%
reduction in document size. They have used depth first
traversal to parse the document tree.
Various online tools to detect and notify web
page changes are available on the internet. some of this
are like timelyWeb [16] and URLY Warning[17].
There has been a lot of research going on in this field
as this has been a newer topic in the field of hypertext.
Researchers are trying to increase the efficiencies of
the change detection algorithms in order to increase the
efficiencies of the web crawler.
3. Proposed Methods
3.1 Classification of changes:
Broadly speaking these changes can be classified
into 4 major categories: -
1. Content/Semantic changes refer to modifications
of the page contents from the reader’s point of view.
For example, a page created for a soccer tournament
might be continuously updated as the tournament
progresses. After the tournament has ended, the page
might change to a presentation about the tournament
results and sports injuries.
2. Presentation/Cosmetic changes are changes related
to the document representation that do not reflect
changes in the topic presented in the document. For
instance, changes to HTML tags can modify the
appearance of a Web page while it otherwise remains
the same.
3. Structural changes refer to the underlying
connection of the document to other documents. As an
example, consider changes in the link destinations of a
―Weekly Hot Links Web page. While this page might
be conceptually the same, the fact that the destination
of the links has changed might be relevant, even if the
text of the links has not. Structural changes are also
important to detect, as they often might not be visually
perceptible.
4. Behavioral changes refer to modifications to the
active components of a document. For Web pages,
this includes scripts, plug-ins and applets. The
consequences of these changes are harder to predict,
especially since many pages hide the script code in
other files.
Fig.1 Types of changes in the web pages
Structural changes can be in the HTML or XML
document (structured or semi structured) of a specific
web page, these changes can be in the form of updating
content (addition of a new node to the tree structure,
deletion of nodes from the structure), image or link
insertion /deletion etc. Techniques like page checksum,
digital fingerprinting, page mirroring etc. are used for
detecting content or semantic changes. We are using an
improved version of page digest [8] to detect the
structural changes in the web page structure i.e. in the
HTML tags structure. The suggested algorithm is
based on level order search which is another form of
breadth first search. We move in the tree level by level.
The basic modules, which constitute the application
are-
• Document tree construction (It takes a HTML file
as an input file, parses the tree with the help of a
parser (as discussed later), identifies those opening
tags as nodes of the trees, and maintaining the
parent child relationships constructs the tree of the
given web page.) ;
267
261
• Parser level order child enumeration (tree
traversal/parsing by level order);
• Tree mapping (finding the combined r.m.s value
of the tags and content for change
detection/extraction); and
• Change detector and presenter.
Our extractor is very effective for real web data in
practice and has linear scalability. Change detector
works basically for two semantic phases—content
changes and the structured changes. We can further
divide this into 3 views-
1. HTML view
2. Content view
3. Tree view
The HTML view shows and compares the old and the
new version of the web page e.g. any news portal. It
extracts the HTML source code of those pages and the
changed portion is displayed in the source code only.
The content view deals with the changed portions of
the contents of the web page versions. It includes the
text, paragraphs, headers; etc. There is R.M.S value
calculator which calculates the R.M.S. value of the
text or the paragraph, or heading. The tree view
traverses the document tree structure and presents the
changes by comparing the number of nodes and by
comparing the array structure of the HTML page. e.g.
if the number of node on traversing the old and the new
versions of the tree are different that means that there
is new node added or deleted in the page structure. In
all we need some fact or figure (i.e. signature) on the
basis of which we can distinguish the content of the
node of the web page.
In this section, we study the types of problems that are
emerging in detecting the new or changed information
of the web pages and for each problem, we present a
practical solution and analyze its efficiency. The
problems and solutions are:
1) How can changes in web pages be detected
effectively? First we classify the problem into two
broad categories- the structural changes and the content
based changes. We detect changes using a hypertext
document tree encoding of the document. The first
module renders an ordered tree structure of the input
HTML web page, counts the number of nodes i.e. node
count if it is different that means the structure of the
web page has changed or a new node has been added
or deleted. Further case may arise where there are
equal number of nodes after comparing the older and
the newer version of the HTML web page tree
structure, but the order of the tags have been changed
or a nested structure has been added. In order to detect
these types of changes parsing plays an important role,
which we have executed after studying different
existing techniques based on data structures. The
technique, which we have used, incorporates level
order ordered tree traversal, which has its advantage of 6) TAG NAME: This store the tag name.
traversing the tree level by level.
2) How can changes between different versions of web
documents be presented/extracted effectively? We
present an efficient and effective algorithm for
comparing different versions of web documents. The
change extractor based on this algorithm includes three
phases: document tree construction, level order child
enumeration (tree traversal/parsing by level order), and
tree mapping (finding the combined R.M.S. value of
the tags and content for change detection/extraction).
Our extractor is very effective for real web data in
practice and has linear scalability.
3) How should a change both in structure or content be
detected and evaluated? We have constructed a
specialized set of arrays for HTML pages which show
the relationships among the tree nodes, i.e. a child or a
parent. By analyzing the content of the arrays, we can
easily determine the difference between the older
version of the web page and the newer version. There
are different cases in a HTML tree structure where the Fig. 3 HTML Tree structure of web page
order of the tags plays an important role.
Addition /Deletion of node (Tag) in the Tree
Suppose a new node is introduced into the tree i.e. a
3.2 Change Detection: new tag is added into the html code as described
below.
Method for structural change detection: -We consider
the following sample HTML web page whose structure
is to be detected for the changes using the proposed
algorithm.

Fig 4. Initial structure (before addition)

Fig. 2 An example web page

There may be two types of changes in web pages : Fig 5. Final structure (after addition of TR)
(1) Addition/deletion of a node containing tag of the
HTML page. The following s the change in the structure
(2) Modification in the tag or tag value or change in
content. LEVEL INTIALLY LATER
The structure of every node of the tree representing the LEVEL 1 1 1
web page shall contain the following information: LEVEL 2 1 1
1) ID: This index stores the unique id representing LEVEL 3 1 2
each node; LEVEL 4 1 1
2) CHILD: This index stores the information of child.
3) PARENT: This index stores the information of the Initially:
parent; LEVEL = {1 2 3 4 }
4) LEVEL: This index stores the level, where the ID = {1 2 3 4 }
node exists; CHILD = {1 1 1 NULL }
5) CONTENT VALUE : This index stores the RMS LEVEL_ARRY = {1 1 1 1 }
value of sum of character of the content;

268
262
Later:
LEVEL = {1 2 3 3 4}
ID = {1 2 3 5 4}
CHILD = {1 2 1 NULL NULL}
LEVEL_ARRY={1 1 2 1 1}
Now, by comparing the two sets we get the idea that
the modification has been done at LEVEL 3. In order
to find the modification at a given level, we use the
level order traversal The algorithm for level order
traversal with breadth first search will give us the
location where the change has taken place.
Now, the second case is when the modification has
been done in the value or content of the text. Example:
Initially <td style="TEXT-ALIGN: left">
Structure of node <td > is as given below;
Fig 6. Initial content of the arrays
Calculation of R.M.S. value before change:
Cont_value= l + e +f +t
= 108, 101,102,116 = 106.9170239.
Now, <td style="TEXT-ALIGN: right">
Structure of node <td >is:
Fig 7. Final array structure
Calculation of R.M.S. value after the change:
Cont_value= r + i +g +h +t = 108.5375511
For anagrams and text where numbers of similar
characters are same but at different positions e.g.
NAME and MANE, the R.M.S. Value shall be same.
So, in order to avoid confusion we propose a solution
in which we multiply the position of the character with
its ASCII value and then we follow:
We have taken an example of anagram NAME AND
MANE. In first word position of N=0, A=1, M=2, E=3
and in second it is M=0, A=1, N=2, E=3.Since we are
multiplying this with the ASCII value of each
character, so the R.M.S. value will be different. The
formula for R.M.S. value shall be:
269
263
4. Results
By using the above techniques, we have been able to
detect the structural and content-based changes
successfully. Though many researches in the past have
been conducted to detect the changes in the pages, so
in comparison to others our technique gives a linear
performance since we have used level order traversal
which indeed saves our time by avoiding traversal of
the whole tree. Rather our technique aims at traversing
only the changed portion of the tree thus saving time.
We conducted our tests on some of the practical web
pages and have been able to extract the text
successfully and present the changes between the two
proposed the concept of page quality, which is closely
related versions of the web page i.e. old and new
versions of web pages.
5. Conclusion and Future Work
We presented a study of how to model web
changes and how to detect them in web pages which
are volatile that means which change almost each and
every day. The proposed algorithm extract changes
between different versions of web pages. We have
been able to make out structural as well as content-
based changes by developing an efficient application in
java which scores on other techniques on the basis of
simplicity and understandability. Though level order
traversal is another form of the breadth first traversal,
so it may incorporate some of the drawbacks of the
breadth first traversal . First, we found that detecting
changes using HTTP Meta data can successfully
reduce network traffic. Second, existing algorithms for
extracting changes using tree edit distance have high
computational cost, which is inappropriate for large-
scale search engine development. We proposed a new
algorithm, which reduces the cost to linear using both
tree encoding and level-by-level tree matching.
Tracking changes can successfully retrieve the
summary but not the complete content of the newly
created page. A good future direction is to integrate
both changed content and newly created content into a
unified search index. Such index can have a higher
coverage of new information on the web. A further
thought is that whether updates on popular pages
which are more likely to be updated by web authors are
of high quality for retrieval. Cho [1] proposed the
concept of page quality, which is closely, related to the
popularity metrics of web pages. In our prospect ion, if
such quality metric can be used to evaluate the updates
of web pages, then people may be able to develop
solutions to improve the quality, as well as freshness,
of web index for retrieval. We leave it as a good future
direction that how web change detection on the basis of
change frequency, quality, popularity, etc. can be used
in a unified framework for web index synchronization
problem.
References:
[1] Junghoo Cho, Hector Garcia-Molina, Department of
Computer Science, Stanford, The Evolution of the Web and
Implications for an Incremental Crawler, CA 94305,
December 2, 1999
[2] Jenny Edwards Kevin McCurley John Tomlin, An
Adaptive Model for Optimizing Performance of an
Incremental Web Crawler, 2000
[3] Junghoo Cho and Hector Garcia-Molina, Synchronizing a
database to improve freshness, submitted for publication,
1999.
http://www-db.stanford.edu/~cho/papers/cho-synch.ps.
[4] Junghoo Cho, Hector Garcia-Molina, and Lawrence,
Efficient crawling through URL ordering, Page In
Proceedings of the 7th World-Wide Web Conference, 1998.
[5] Junghoo Cho, University of California, Los Angeles,
California a Hector Garcia-Molina, Stanford University,
Stanford, California. Effective Page Refresh Policies for Web
Crawlers,
[6] Junghoo Cho, University of California, Los Angeles and
Hector Garcia- Molina, Stanford University, Estimating
Frequency of Change,
Department of Computer Science, Texas A&M, College
Station, TX, 77843-3112, US,2001
[13] Zubin Dalal, Suvendu Dash, Pratik Dave, Luis
Francisco-Revilla, Richard Furuta, Unmil Karadkar, Frank
Shipman, Managing Distributed Collections: Evaluating
Webpage Changes, Movement, and Replacement,
Department of Computer Science and Center for the Study of
Digital Libraries, Texas A&M University, and College
Station, TX 77843-3112..JCDL’04, June 7–11, 2004,
Tucson, Arizona, USA.Copyright 2004 ACM
[14] Latifur Khan, Lei Wang and Yan Rao, Change
Detection of XML Documents Using Signatures, Department
of Computer Science, University of Texas at Dallas,
Richardson, TX 75083-0688
[15] Shuohao Zhang, Curtis Dyreson, and Richard T.
Snodgrass Schema- Less, Semantics-Based Change
Detection for XML Detection, Washington State University,
Pullman, Washington, U.S.A. WISE 2004, LNCS 3306, pp.
279–290,© Springer-Verlag Berlin Heidelberg 2004
[16] Timely Web tool, www.timelyweb.com/index.html
[17] URLyWarningwww.bleepingcomputer.com
[18] A. Ntoulas, J. Cho, and C. Olston. What’s new on the
web? The evolution of the web from a search engine
perspective. In Proc. 13th International World Wide Web
Conference, 2004.
[19] D. Fretterly, M. Manasse, M. Najork, and J. Wiener. A
large-scale study of the evolution of web pages. In Proc. 12th
International World Wide Web Conference, 2003.

[7] Arvind Arasu, Junghoo Cho, Hector Garcia-Molina,

Searching the Web, Stanford University, 1999

[8] Daniel Rocco, David Buttler, Ling Liu, Page Digest for
Large-Scale Web Services Georgia Institute of Technology,
College of Computing, Atlanta, GA 30332, U.S.A.,
Proceedings of the IEEE International Conference on E-
Commerce (CEC‘03), 2003

[9] David Buttler, Daniel Rocco, Ling Liu, Efficient Web

Change Monitoring with Page Digest, 2001

[10] Buda Rahardjo, Roland H.C. Yap, Automatic

Information Extraction from Web Pages, , School of
Computing, National University of Singapore, Republic of
Singapore,2001

[11] Ling Liu Carlton Pu, Wei Tang, WebCQ – Detecting

and Delivering Information Changes on the Web, 2000

[12] Luis Francisco-Revilla, Frank M. Shipman III, Richard

Furuta, Unmil Karadkar, Avital Arora,Perception of Content,
Structure, and Presentation Changes in Web-based
Hypertext, , Center for the Study of Digital Libraries and

270
264