See discussions, stats, and author profiles for this publication at: https://www.researchgate.

net/publication/314457405

Artificial Intelligence for Information Retrieval

Chapter · January 2008

DOI: 10.4018/9781599048499.ch023

CITATIONS READS
5 10,312

1 author:

Thomas Mandl
Universität Hildesheim
324 PUBLICATIONS 2,337 CITATIONS

SEE PROFILE

All content following this page was uploaded by Thomas Mandl on 11 March 2019.

The user has requested enhancement of the downloaded file.

Encyclopedia of
Artificial Intelligence

Juan Ramón Rabuñal Dopico

University of A Coruña, Spain

Julián Dorado de la Calle

University of A Coruña, Spain

Alejandro Pazos Sierra

University of A Coruña, Spain

Information Sci
Hershey • New York
Director of Editorial Content: Kristin Klinger
Managing Development Editor: Kristin Roth
Development Editorial Assistant: Julia Mosemann, Rebecca Beistline
Senior Managing Editor: Jennifer Neidig
Managing Editor: Jamie Snavely
Assistant Managing Editor: Carole Coulson
Typesetter: Jennifer Neidig, Amanda Appicello, Cindy Consonery
Cover Design: Lisa Tosheff
Printed at: Yurchak Printing Inc.

Published in the United States of America by

Information Science Reference (an imprint of IGI Global)
701 E. Chocolate Avenue, Suite 200
Hershey PA 17033
Tel: 717-533-8845
Fax: 717-533-8661
E-mail: cust@igi-global.com
Web site: http://www.igi-global.com/reference

and in the United Kingdom by

Information Science Reference (an imprint of IGI Global)
3 Henrietta Street
Covent Garden
London WC2E 8LU
Tel: 44 20 7240 0856
Fax: 44 20 7379 0609
Web site: http://www.eurospanbookstore.com

Copyright © 2009 by IGI Global. All rights reserved. No part of this publication may be reproduced, stored or distributed in any form or by any
means, electronic or mechanical, including photocopying, without written permission from the publisher.
Product or company names used in this set are for identification purposes only. Inclusion of the names of the products or companies does not indicate
a claim of ownership by IGI Global of the trademark or registered trademark.

Library of Congress Cataloging-in-Publication Data

Encyclopedia of artificial intelligence / Juan Ramon Rabunal Dopico, Julian Dorado de la Calle, and Alejandro Pazos Sierra, editors.
p. cm.
Includes bibliographical references and index.
Summary: "This book is a comprehensive and in-depth reference to the most recent developments in the field covering theoretical developments, tech-
niques, technologies, among others"--Provided by publisher.
ISBN 978-1-59904-849-9 (hardcover) -- ISBN 978-1-59904-850-5 (ebook)
1. Artificial intelligence--Encyclopedias. I. Rabunal, Juan Ramon, 1973- II. Dorado, Julian, 1970- III. Pazos Sierra, Alejandro.
Q334.2.E63 2008
006.303--dc22
2008027245

British Cataloguing in Publication Data

A Cataloguing in Publication record for this book is available from the British Library.

All work contributed to this encyclopedia set is new, previously-unpublished material. The views expressed in this encyclopedia set are those of the
authors, but not necessarily of the publisher.

If a library purchased a print copy of this publication, please go to http://www.igi-global.com/agreement for information on activating
the library's complimentary electronic access to this publication.

Artificial Intelligence for Information Retrieval
Thomas Mandl
University of Hildesheim, Germany
INTRODUCTION
This article describes the most prominent approaches
to apply artificial intelligence technologies to infor-
mation retrieval (IR). Information retrieval is a
key technology for knowledge management. It deals
with the search for information and the representation,
storage and organization of knowledge. Information
retrieval is concerned with search processes in which a
user needs to identify a subset of information which is
relevant for his information need within a large amount
of knowledge. The information seeker formulates a
query trying to describe his information need. The query
is compared to document representations which were
extracted during an indexing phase. The representations
of documents and queries are typically matched by a
similarity function such as the Cosine. The most similar
documents are presented to the users who can evaluate
the relevance with respect to their problem (Belkin,
2000). The problem to properly represent documents
and to match imprecise representations has soon led to
the application of techniques developed within Artificial
Intelligence to information retrieval.
BACKGROUND
In the early days of computer science, information
retrieval (IR) and artificial intelligence (AI) developed
in parallel. In the 1980s, they started to cooperate and
the term intelligent information retrieval was coined
for AI applications in IR. In the 1990s, information
retrieval has seen a shift from set based Boolean
retrieval models to ranking systems like the vector
space model and probabilistic approaches. These
approximate reasoning systems opened the door for
more intelligent value added components. The large
amount of text documents available in professional
databases and on the internet has led to a demand for
intelligent methods in text retrieval and to considerable
research in this area. The need for better preprocessing
to extract more knowledge from data has become an
Copyright © 2009, IGI Global, distributing in print or electronic forms without written permission of IGI Global is prohibited.

A
important way to improve systems. Off the shelf ap-
proaches promise worse results than systems adapted
to users, domain and information needs. Today, most
techniques developed in AI have been applied to re-
trieval systems with more or less success. When data
from users is available, systems use often machine
learning to optimize their results.
Artificial Intelligence Methods in
Information Retrieval
Artificial intelligence methods are employed throughout
the standard information retrieval process and for
novel value added services. The first section gives a
brief overview of information retrieval. The subsequent
sections are organized along the steps in the retrieval
process and give examples for applications.
Information Retrieval
Information retrieval deals with the storage and
representation of knowledge and the retrieval of
information relevant for a specific user problem. The
information seeker formulates a query trying to de-
scribe his information need. The query is compared
to document representations. The representations
of documents and queries are typically matched by a
similarity function such as the Cosine or the Dice coef-
ficient. The most similar documents are presented to
the users who can evaluate the relevance with respect
to their problem.
Indexing usually consists of the several phases.
After word segmentation, stopwords are removed.
These common words like articles or prepositions
contain little meaning by themselves and are ignored
in the document representation. Second, word forms
are transformed into their basic form, the stem. During
the stemming phase, e.g. houses would be transformed
into house. For the document representation, different
word forms are usually not necessary. The importance
of a word for a document can be different. Some words
better describe the content of a document than others.

This weight is determined by the frequency of a stem
within the text of a document (Savoy, 2003).
In multimedia retrieval, the context is essential
for the selection of a form of query and document
representation. Different media representations may
be matched against each other or transformations may
become necessary (e.g. to match terms against pictures
or spoken language utterances against documents in
written text).
As information retrieval needs to deal with
vague knowledge, exact processing methods are
not appropriate. Vague retrieval models like the
probabilistic model are more suitable. Within these
models, terms are provided with weights corresponding
to their importance for a document. These weights
mirror different levels of relevance.
The result of current information retrieval systems
are usually sorted lists of documents where the top
results are more likely to be relevant according to the
system. In some approaches, the user can judge the
documents returned to him and tell the systems which
ones are relevant for him. The system then resorts
the result set. Documents which contain many of the
words present in the relevant documents are ranked
higher. This relevance feedback process is known to
greatly improve the performance. Relevance feedback
is also an interesting application for machine learning.
Based on a human decisions, the optimization step can
be modeled with several approaches, e.g. with rough
sets (Singh & Dey 2005). In Web environments, a click
is often interpreted as an implicit positive relevance
judgment (Joachims & Radlinski, 2007).
Advanced Representation Models
In order to represent documents in natural language, the
content of these documents needs to be analyzed. This
is a hard task for computer systems. Robust semantic
analysis for large text collections or even multime-
dia objects has yet to be developed. Therefore, text
documents are represented by natural language terms
mostly without syntactic or semantic context. This is
often referred to as the bag-of-words approach. These
keywords or terms can only imperfectly represent an
object because their context and relations to other
terms are lost.
However, great progress has been made and systems
for semantic analysis are getting competitive. Advanced
syntactic and semantic parsing for robust processing

Artificial Intelligence for Information Retrieval
of mass data has been derived from computational
linguistics (Hartrumpf, 2006).
For application and domain specific knowledge,
another approach is taken to improve the representation
of documents. The representation scheme is enriched
by exploiting knowledge about concepts of the domain
(Lin & Demner-Fushman, 2006).
Match Between Query and Document
Once the representation has been derived, a crucial
aspect of an information retrieval system is the
similarity calculation between query and document
representation. Most systems use mathematical simi-
larity functions such as the Cosine. The decision for
a specific function is based on heuristics or empirical
evaluations. Several approaches use machine learning
for long term optimization of the matching between
term and document. E.g. one approach applies genetic
algorithm to adapt a weighting function to a collection
(Almeida et al., 2007).
Neural networks have been applied widely in IR.
Several network architectures have been applied for
retrieval tasks, most often the so-called spreading activa-
tion networks are used. Spreading activation networks
are simple Hopfield-style networks, however, they do
not use the learning rule of Hopfield networks. They
typically consist of two layers representing terms and
documents. The weights of connections between the
layers are bi-directional and initially set according to
the results of the traditional indexing and weighting
algorithms (Belkin, 2000). The neurons corresponding
to the terms of the user’s query are activated in the term
layer and activation spreads along the weights into
the document layer and back. Activation represents
relevance or interest and reaches potentially relevant
terms and documents. The most highly activated docu-
ments are presented to the user as result. A closer look
at the models reveals that they very much resemble
the traditional vector space model of Information
Retrieval (Mandl, 2000). It is not until after the second
step that associative nature of the spreading activation
process leads to results different from a vector space
model. The spreading activation networks successfully
tested with mass data do not take advantage of this
associative property. In some systems the process is
halted after only one step from the term layer into the
document layer, whereas others make one more step
Artificial Intelligence for Information Retrieval
back to the term layer to facilitate learning (Kwok &
Grunfeld, 1996).
Queries in information retrieval systems are
usually short and contain few words. Longer queries
have a higher probability to achieve good results. As a
consequence, systems try to add good terms to a query
entered by a user. Several techniques have been applied.
Either these terms are taken from top ranked documents
or terms similar to the original ones are used. Another
technique is to use terms from documents from the same
category. For this task, classification algorithms from
machine learning are used (Sebastiani, 2002).
Link analysis applies well known measures from
bibliometric analysis to the Web. The number links
pointing to a Web page is used as an indicator for its
quality (Borodin et al., 2005). PageRank assigns an
authority value to each Web page which is primarily a
function of its back links. Additionally, it assumes that
links from pages with high authority should be weighed
higher and should result in a higher authority for the
receiving page. To account for the different values
each page has to distribute, the algorithm is carried
out iteratively until the result converges (Borodin et
al., 2005). Machine Learning approaches complement
link analysis. Decisions of humans about the quality
of Web pages are used to determine design features of
these pages which are good indicators of their quality.
Machine learning models are applied to determine the
quality of pages not judged yet (Mandl, 2006, Marti
& Hearst, 2002).
Learning from users has been an important strategy
to improve systems. In addition to the content, artificial
intelligence methods have been used to improve the
user interface.
Value Added Components for User
Interfaces
Several Researchers have implemented information
retrieval systems based on the Kohonen self organiz-
ing map (SOM), a neural network model for unsuper-
vised classification. They provide an associative user
interface where neighborhood of documents expresses
a semantic relation. Implementations for large collec-
tions can be tested on the internet (Kohonen, 1998).
The SOM consists of a usually two-dimensional grid
of neurons, each associated with a weight vector. Input
documents are classified according to the similarity
between the input pattern and the weight vectors, and,
the algorithm adapts the weights of the winning neuron
and its neighbor. In that way, neighboring clusters have A
a high similarity.
The information retrieval applications of SOMs
classify documents and assign the dominant term as
name for the cluster. For real world large scale col-
lections, one two-dimensional grid is not sufficient. It
would be either too big or each node would contain
too many documents consequently. Neither would be
helpful for users, therefore, a layered architecture is
adopted. The highest layer consists of nodes which
represent clusters of documents. The documents of
these nodes are again analyzed by a SOM. For the
user, the system consists of several two-dimensional
maps of terms where similar terms are close to each
other. After choosing one node, he may reach another
two-dimensional SOM.
The information retrieval paradigm for the SOM is
browsing and navigating between layers of maps. The
SOM seems to be a very natural visualization. However,
the SOM approach has some serious drawbacks.
• The interface for interacting with several layers
of maps makes the system difficult to browse.
• Users of large text collections need primarily
search mechanisms which the SOM itself does
not offer.
• The similarity of the document collection is
reduced to two dimensions omitting many po-
tentially interesting aspects.
• The SOM unfolds its advantages for human-
computer-interaction better for a small number
of documents. A very encouraging application
would be the clustering of the result set. The
neurons would fit on one screen, the number of
terms would be limited and therefore, the reduc-
tion to two dimensions would not omit so many
aspects.
User Classification and Personalization
Adaptive information retrieval approaches intend to
tailor the results of a system to one user and his inter-
ests and preferences. The most popular representation
scheme relies on the representation scheme used in
information retrieval where a document-term-matrix
stores the importance or weight of each term for each
document. When a term appears in a document, this
weight should be different form zero. User interest can


also be stored like a document. Then the interest is a
vector of terms. These terms can be ones that a user
has entered or selected in a user interface or which the
system has extracted from documents for which the
user has shown interest by viewing or downloading
them (Agichtein et al., 2006).
An example for such a system is UCAIR which
can be installed as a browser plugin. UCAIR relies
on a standard web search engine to obtain a search
result and a primary ranking. This ranking is now
being modified by re-ranking the documents based
on implicit feedback and a stored user interest profile
(Shen et al., 2005).
Most systems use this method of storing the user
interest in a term vector. However, this method has
several drawbacks. The interest profile may not be stable
and the user may have a variety of diverging interests
for work and leisure which are mixed in one profile.
Advanced individualization techniques personal-
ize the underlying system functions. The results of
empirical studies have shown that relevance feedback
is an effective technique to improve retrieval quality.
Learning methods for information retrieval need to
extend the range of relevance feedback effects beyond
the modification of the query in order to achieve long-
term adaptation to the subjective point of view of the
user. The mere change of the query often results in
improved quality; however, the information is lost after
the current session.
Some systems change the document representation
according to the relevance feedback information. In
a vector space metaphor, the relevant documents are
moved toward the query representation. This approach
also comprises some problems. Because only a fraction
of the documents are affected by the modifications, the
basic data from the indexing process is changed to a
somewhat heterogeneous state. The original indexing
result is not available anymore.
Certainly, this technique is inadequate for fusion
approaches where several retrieval methods are com-
bined. In this case, several basic representations would
need to be changed according to the influence of the
corresponding methods on the relevant documents.
The indexes are usually heterogeneous, which is often
considered an advantage of fusion approaches. A high
computational overload would be the consequence.
The MIMOR (Multiple Indexing and Method-Object
Relations) approach does not rely on changes to the
document or the query representation when processing

Artificial Intelligence for Information Retrieval
relevance feedback information for personalization.
Instead, it focuses on the central aspect of a retrieval
function, the calculation of the similarity between docu-
ment and query. Like other fusion methods, MIMOR
accepts the result of individual retrieval systems like
from a black box. These results are fused by a linear
combination which is stored during many sessions. The
weights for the systems experience a change through
learning. They adapt according to relevance feedback
information provided by users and create a long-term
model for future use. That way, MIMOR learns which
systems were successful in the past (Mandl & Womser-
Hacker, 2004).
FUTURE TRENDS
Information retrieval systems are applied in more and
more complex and diverse environments. Searching
e-mail, social computing collections and other specific
domains pose new challenges which lead to innovative
systems. These retrieval applications require thorough
and user oriented evaluation. New evaluation measures
and standardized test collections are necessary to
achieve reliable evaluation results.
In user adaptation, recommendation systems are an
important trend for future improvement. Recommenda-
tion systems need to be seen in the context of social
computing applications. System developers face the
growth of user generated content which allows new
reasoning methods.
New application like question answering relying on
more intelligent processing can be expected to gain more
market share in the near future (Hartrumpf, 2006)
CONCLUSION
Knowledge management is of main importance for
the information society. Documents written in natural
language contain an important share of the knowl-
edge available. Consequently, retrieval is crucial for
the success of knowledge management systems. AI
technologies have been widely applied in retrieval
systems. Exploiting knowledge more efficiently is a
major research field. In addition, user oriented value
added systems require intelligent processing and ma-
chine learning in many forms.
An important future trend for AI methods in IR will
be the context specific adaptation of retrieval methods.
Artificial Intelligence for Information Retrieval
Machine learning can be applied to find optimized
functions for collections or queries.
REFERENCES
Agichtein, E., Brill, E., Dumais, S., & Ragno, R. (2006).
Learning user interaction models for predicting web
search result preferences. Annual International ACM
Conference on Research and Development in Informa-
tion Retrieval (SIGIR) Seattle. ACM Press. 3-10.
de Almeida, H.M., Gonçalves, M.A, Cristo, M., &
Calado, P. (2007). A combined component approach
for finding collection-adapted ranking functions based
on genetic programming. Annual International ACM
Conference on Research and Development in Infor-
mation Retrieval (SIGIR) Amsterdam. ACM Press.
399-406.
Belkin, R. (2000). Finding out about: a Cognitive Per-
spective on Search Engine Technology and the WWW.
Cambridge et al.: Cambridge University Press.
Brusilovsky, P., Kobsa, A. & Nejdl, W. (eds.) (2007).
The adaptive Web: Methods and strategies of Web
personalization. Heidelberg: Springer.
Boughanem, M., & Soulé-Dupuy, C. (1998). Mercure
at trec6. In Voorhees, E. & Harman, D. (eds.). The
sixth text retrieval conf (TREC-6). NIST Special Publ
500-240. Gaithersburg, MY.
Borodin, A., Roberts, G., Rosenthal, J. & Tsaparas, P.
(2005). Link analysis ranking: algorithms, theory, and
experiments. ACM Transactions on Internet Technology
(TOIT) 5(1), 231-297.
Hartrumpf, S. (2006). Extending Knowledge and
Deepening Linguistic Processing for the Question
Answering System InSicht. Accessing Multilingual
Information Repositories, 6th Workshop of the Cross-
Language Evalution Forum, CLEF. [LNCS 4022]
Springer. 361-369
Ivory M. & Hearst, M. (2002). Statistical Profiles of
Highly-Rated Sites. ACM CHI Conference on Hu-
man Factors in Computing Systems. ACM Press.
367-374.
Joachims, T., & Radlinski, F. (2007). Search engines
that learn from implicit feedback. IEEE Computer
40(8), 34-40.
Kohonen T. (1998). Self-organization of very large
document collections: state of the art. In Proceedings A
8th Intl Conf Artificial Neural Networks. Springer, 1.
65-74.
Kwok, K., & Grunfeld, L. (1996). TREC-4 ad-hoc,
routing retrieval and filtering experiments using PIRCS.
In: Harman Donna (ed.). The fourth Text Retrieval
Conference (TREC-4). NIST Special Publ 500-236.
Gaithersburg, MY.
Lin, J., & Demner-Fushman, D. (2006). The role of
knowledge in conceptual retrieval: a study in the domain
of clinical medicine. In Annual International ACM
Conference on Research and Development in Informa-
tion Retrieval (SIGIR) Seattle. ACM Press. 99-106
Mandl, T. (2000). Tolerant Information Retrieval with
Backpropagation Networks. Neural Computing & Ap-
plications 9(4), 280-289.
Mandl, T. (2006). Implementation and evaluation of a
quality based search engine. In Proceedings of the 17th
ACM Conference on Hypertext and Hypermedia (HT
‘06) Odense, Denmark. ACM Press. 73-84.
Mandl, T., & Womser-Hacker, C. (2004). A Framework
for long-term Learning of Topical User Preferences in
Information Retrieval. New Library World, 105(5/6)
184-195.
Savoy, J. (2003). Cross-language information retrieval:
experiments based on CLEF 2000 corporaSebastiani, F.
(2002). Machine Learning in Automated Text Catego-
rization. ACM Computing Surveys 34(1), 1-47.
Shen, X., Tan, B. & Zhai, C. (2005). Context-sensitive
information retrieval using implicit feedback. Annual
International ACM Conference on Research and De-
velopment in Information Retrieval (SIGIR), Seattle,
WA. ACM Press. 43-50.
Singh, S., & Dey Lipika (2005). A rough-fuzzy docu-
ment grading system for customized text information
retrieval. Information Processing & Management
41(2), 195-216.
Zhang, D., Chen, X. & Lee, W. (2005). Text clas-
sification with kernels on the multinomial manifold.
Annual International ACM Conference on Research
and Development in Information Retrieval (SIGIR)
Seattle. ACM Press. 266-273.


KEy TERMS
Adaptation: Adaptation is a process of modification
based on input or observation. An information system
should adapt itself to the specific needs of individual
users in order to produce optimized results.
Indexing: Indexing means the assignment of terms
(words) which represent a document in an index. In-
dexing can be carried out manually or automatically.
Automatic indexing requires the elimination of stop-
words and stemming.
Information Retrieval: Information retrieval is
concerned with the representation and knowledge and
subsequent search for relevant information within these
knowledge sources. Information retrieval provides the
technology behind search engines.
Link Analysis: The links between pages on the web
are a large knowledge source which is exploited by link
analysis algorithms for many ends. Many algorithms

View publication stats
Artificial Intelligence for Information Retrieval
similar to PageRank determine a quality or authority
score based on the number of in-coming links of a
page. Furthermore, link analysis is applied to identify
thematically similar pages, web communities and other
social structures.
Recommendation Systems: Actions or content is
suggested to the user based on past experience collected
from other users. Very often, documents are recom-
mended based on similarity profiles between users.
Term Expansion: Terms not present in the original
query to an information retrieval system entered by the
user are added automatically. The expanded query is
then sent to the system again.
Weighting: Weighting determines the importance
of a term for a document. Weights are calculated using
many different formulas which consider the frequency
of each term in a document and in the collection as
well as the length of the document and the average or
maximum length of any document in the collection.