Professional Documents
Culture Documents
Word Sense Disambiguation - FINAL by Workineh Tesema Gudisa
Word Sense Disambiguation - FINAL by Workineh Tesema Gudisa
Word Sense Disambiguation - FINAL by Workineh Tesema Gudisa
INSTITUTE OF TECHNOLOGY
DEPARTMENT OF INFORMATION TECHNOLOGY
Jimma University
Jimma
May 22, 2014
ii
MASTER OF SCIENCE
IN
INFORMATION TECHNOLOGY
2014 A.Y
Acknowledgment
I would like to thanks gratefully and acknowledge my advisor Mr. Debela Tesfaye for all his
guidance at every step of the way, for patiently listening to me even during uncountable hours,
ii
for teaching me how to do seminar, for imparting so much valuable knowledge and for all his
encouragement and words of kindness. Any noteworthy aspects of this seminar are entirely
because of him; all faults are in spite of him.
Table of Content
Table of Contents
Page
Acknowledgment.............................................................................................................................ii
Acronyms........................................................................................................................................iv
ii
Operational Definitions...................................................................................................................v
Abstract...........................................................................................................................................vi
1. Introduction..................................................................................................................................1
2. Literature Review........................................................................................................................4
2.1. Introduction...........................................................................................................................4
2.2. Related Works.......................................................................................................................6
3. Methodology...............................................................................................................................9
3.1. Algorithm..............................................................................................................................9
3.2. Evaluation...........................................................................................................................10
4. Research Gap Identification ......................................................................................................11
5. Conclusion ................................................................................................................................11
6. Future Work...............................................................................................................................12
Reference............................................................................................................................13
Appendix A........................................................................................................................14
Acronyms
CBC: Cluster By Committee
IDLE: Interactive DeveLopment Environment
MT: Machine Translation
ii
Operational Definitions
Word Sense Disambiguation: Word sense disambiguation is the process of
automatically figuring out the intended meaning of such a word when used in a sentence
ii
ii
Abstract
All human languages have words that can mean different things in different contexts. Word sense
disambiguation is the process of automatically figuring out the intended meaning of such a word
when used in a sentence. This seminar paper deals with word sense disambiguation (WSD).
WSD has obvious relationships to other fields such as lexical semantics, whose main endeavour
is to define, analyze, and ultimately understand the relationships between word, meaning,
and context.
After making intensive review on the related works we picked a paper that follow corpus based
statistical approach. More specifically, we extracted lexical features of the target word to learn if
there is any ambiguity or not and if the word is ambiguous we applied clustering algorithm to
disambiguate the word. The method relied on the text data extracted from Wikipedia. we have
implemented and evaluated the algorithm and obtained a promising result comparable to the
result reported by the original work. We assumed that the evaluation result difference observed
might be resulted from employing different test and training set.
1.Introduction
The goal of natural language processing is to teach the computer or machine, human language.
Natural language processing here refers to the use and ability of systems to process sentences in
a natural language such as English, rather than in a specialized artificial computer language such
as C++. Occasionally the phrase natural language is used not for actual languages as we are used
in ordinary, such as our actual use of English to communicate in everyday life, but for a more
restricted subset of such a human language, one purged of constructions and ambiguities that
computers could not sort out. Hence one writer states that human languages allow anomalies that
natural languages cannot allow. There may be a need for such a language, but a natural language
restricted in this way is artificial, not natural. In today computation the issue of word sense is
creating great problem in dealing with natural language processing.
Ambiguity is crucial problem in natural language processing.
14
Word sense disambiguation (WSD) can also be defined as the process of identifying the correct
sense or meaning of a word in particular context. Human beings are especially good at this. For
example, given the sentence a financial institution that accepts deposits and channels the money
into lending activities, it is immediately apparent to us that bank here refers to the financial
institution, whereas given the bank sloping land (especially the slope beside a body of water) we
know that bank here means the sloping land(mouth of the river). Making computers to
understand natural languages is challenged by many problems, word sense ambiguity being one
of them and it is very difficult for computers to achieve this same feat. Although computers are
best at following fixed rules, it is impossible to create a set of simple rules that would accurately
disambiguate any word in any context. This difficulty stems from the fact that natural languages
themselves seldom follow hard and fast rules.
One of the first problems encountered by any natural language processing system is that of
lexical ambiguity, be it syntactic or semantic. The problem of resolving semantic ambiguity is
generally known as word sense disambiguation and has proved to be more difficult than syntactic
disambiguation. The problem is that words often have more than one meaning, sometimes fairly
similar and sometimes completely different. The meaning of a word in a particular usage can
only be determined by examining its context. This is, in general, a trivial task for the human
language processing system. However, the task has proved to be difficult for computer and some
have believed that it would never be solved. However, there have been several advances in word
sense disambiguation and we are now at a stage where lexical ambiguity in text can be resolved
with a reasonable degree of accuracy.
There are no decisive ways of identifying where one sense of a word ends and the next begins,
and this is at the core of what makes word sense disambiguation hard. Another problem is the
challenge in deciding what the senses are.
Since WSD involves the association of a given word in a text or discourse with a definition or
meaning which is distinguishable from other meanings potentially attributable to that word, it
usually involves two steps (Ide and Veronis ,1998). The first step is to determine all the different
senses for every word relevant to the text or discourse under consideration, i.e., to choose a sense
inventory, e.g., from the lists of senses in everyday dictionaries, from the synonyms in a
thesaurus, or from the translations in a translation dictionary. The second step involves a means
14
to assign the appropriate sense to each occurrence of a word in context. All disambiguation
works involves matching the context of an instance of the word to be disambiguated either with
information from external knowledge sources or with contexts of previously disambiguated
instances of the word. For both of these sources we need preprocessing or knowledge extraction
procedures representing the information as context features. For some disambiguation tasks,
there are already well-known procedures such as morphosyntactic disambiguation and therefore
WSD has largely focused on distinguishing senses among homographs belonging to the same
syntactic category.
Natural language processing (NLP) involves resolution of various types of ambiguity. Lexical
ambiguity is one of these ambiguity types, and occurs when a single word (lexical form) is
associated with multiple senses or meanings. For applications which are sensitive to semantic
denotation, or more precisely lexical semantics, this ambiguity type can pose a major obstacle.
WSD has obvious relationships to other fields such as lexical semantics, whose main endeavor is
to define, analyze, and ultimately understand the relationships between word, meaning, and
context. But even though word meaning is at the heart of the problem, WSD has never really
found a home in lexical semantics. It could be that lexical semantics has always been more
concerned with representational issues (Lyons 1995) and models of word meaning and polysemy
so far too complex for WSD (Cruse 1986; Ravin and Leacock 2000). And so, the obvious
procedural or computational nature of WSD paired with its early invocation in the context of
machine translation (Weaver 1949) has allied it more closely with language technology and thus
computational linguistics. In fact, WSD has more in common with modern lexicography, with its
intuitive premise that word uses group into coherent semantic units and its empirical corpus
based approaches, than with lexical semantics (Wilks et al. 1993).
Automatic word sense disambiguation can play an important role in the field of machine
translation. Accurate word sense disambiguation can also lead to better results in information
retrieval. For example given a query that consists of the words financial bank we would like to
return documents that contains information about financial and their call and not about the slope
beside a body of water.
Generally, the goal of natural language processing is to help computers learn human languages.
Natural Language Processing can bring powerful enhancements enabling the computers to
14
communicate with humans using natural languages. As one of the efforts, the main focus of this
seminar is to resolve ambiguity in natural language. The systems are trained to perform the task
of word sense disambiguation.
2. Literature Review
2. 1 Introduction
Natural Language Processing is the branch of Artificial language focused on developing systems
that allow computers to communicate with people using everyday language. It is also called
Computational Linguistics which concerns how computational methods can aid the
understanding of human language. The primary aim of natural language processing is to make
communication between humans and computers. However, during the communication there is a
problem of ambiguity. Among these, word sense ambiguity. As explained in the introductory
part, the resolution of such ambiguity, is commonly termed as word sense disambiguation
(WSD) (Jurafsky & Martin, 1999).
The trouble with word sense disambiguation is word senses. There are no decisive ways of
identifying where one sense of a word ends and the next begins. This makes the definition of the
WSD task problematic. Word meaning is in principle infinitely variable and context sensitive. It
does not divide up easily into distinct sub-meanings or senses. Lexicographers frequently
discover in corpus data loose and overlapping word meanings, and standard or conventional
meanings extended, modulated, and exploited in a bewildering variety of ways (Kilgarriff 1997;
Hanks 2000).
WSD approaches can be categorized in to several classes based on various criteria. In this paper
we organized them as knowledge intensive or shallow approaches (the once relying on free text
or corpora) based on the level and type of information employed. Further classification can also
made as supervised, unsupervised or hybrid based on the level of human involvement.
The knowledge intensive approaches usually employee information obtained from WordNet,
dictionaries, thesauri, or concept hierarchies and other corpuses tagged with sense of the
constituent words (Wanton and Llavori, 2012; Banerjee, 2002; Ide &Veronis, 1998; Ellen
14
(1993); Ying Liu (2002) Ying Liu in his work titled Using WordNet to Disambiguate Word
Senses argued that WordNet can be used to extract the target noun by visiting synonym set in
WordNet. To this end Ying Liu (2002) counted the frequency of synonym sets visit to select the
most visit one as the sense of the target word in the given context.
The main drawback of such approaches is that the resources required to annotate the words are
hand-built (by humans) and are therefore expensive to acquire and maintain. This inevitably
leads to knowledge acquisition bottlenecks when attempting to handle larger amounts of text,
new domains or even new languages.
In corpus based approaches, the information required for the disambiguation is gained from the
training on a given corpus. As mentioned above, this approach can be further classified into two
subclasses based on the level of human involvement in training the algorithm on the corpus as:
supervised and unsupervised learning approaches.
In supervised learning approaches, a sense disambiguation algorithm is learned from a
representative set of labeled instances drawn from the same distribution as the test set to be used.
This is a straightforward application of the supervised learning approach to creating a classifier.
In such approaches, a learning system is presented with a training set consisting of featureencoded inputs along with their appropriate label, or category. The output of the system is a
classifier system capable of assigning labels to new feature-encoded inputs (Jurafsky &
Martin,1999).
In contrast to the supervised approaches, the unsupervised approaches in sense disambiguation
eschew the use of sense tagged data of any kind during training. In these approaches, featurevector representations of unlabeled instances are taken as input and are then grouped into clusters
according to a similarity metric. These clusters can then be represented as the average of their
constituent feature-vectors, and labeled by hand with known word senses. Unseen featureencoded instances can be classified by assigning the word sense from the cluster to which they
are closest according to the similarity metric (Jurafsky & Martin,1999).
14
simpler baselines. However, since Wikipedia is a free online encyclopedia, anybody who have
Internet access can edit or delete the data which results in reduction in the quality of the corpus
affecting the word sense disambiguation system performance.
Some works demonstrated that word senses can be disambiguated using the information obtained
from electronic dictionary (Ide &Veronis, 1998; Ellen (1993). The work proved that the correct
sense of a given word can be learned using word definitions extracted from electronic dictionary.
Ide &Veronis (1998) work combines the characteristics of both dictionary (WordNet) and
structured semantic network which provide definitions for the different senses of words to learn
the senses. The basic assumption behind the approach is that words having multiple senses will
have distinct features and multiple definitions which can be learned from electronic dictionary.
Ellen (1993) also relied on the information obtained from WordNet for WSD and pointed out that
the more common words are tend to be appear in the larger synonym sets in WordNet. It is
precisely those nouns that actually get used in documents and are most likely to have many
senses and synonyms. The basic problem addressed in Ellen (1993) study is to detect polysems
and synonyms. According to the study, the of polysems and synonyms are dealt with assigning
different senses of a word to different concept identifiers and assigning the same concept
identifier to synonyms. Ellen concluded that the algorithm used is not sufficient to reliably select
the correct sense of a noun from the set of sense disambiguation in WordNet. Despite the
achievement, the algorithim is only limited to disambiguating nouns.
Banerjee(2002) also used dictionary definition and the information extracted from the famous
electroninc database, WordNet. Accordingly ,given a target word to be disambiguated and a few
surrounding words that provide the context for the word, they have used the definitional glosses
of these words and extracted words related to them through various relations defined in
WordNet. Since the algorithm is dependent on the glosses of the words in the context window,
words that do not occur in any synonym set in WordNet are ignored. This rules out function
words like the, of, an, etc and also most proper nouns. If the words exist to the left or right of the
target word are not enough, they add additional words from the other direction. This is an
attempt to provide roughly the same amount of data for every target word.
Some promising works like Wanton and Llavori(2012)
disambiguate word senses. Their approach not only able to disambiguate all words in a sentence
14
but also reveals the implicit relationships (not asserted in WordNet) existing among these word
senses and the clustering algorithm is used to contextually group word senses according to their
representations.
14
3. Methodology
The approach we followed in this work is unsupervised corpus-based methods of word sense
discrimination and do not rely on external knowledge sources such as machine readable
dictionaries, concept hierarchies or sense-tagged text. Our justification for selecting the approach
is based on the fact that unsupervised approach are extremely cheap to implement in terms of
resource requirement. The counter part of the unsupervised approach namely the supervised
approach relays on huge and expensive resources like WordNet which are very hard to acquire,
modify and develop it for new languages. It is also quite obvious that such resources are
developed only for few languages in the word. The method followed in this seminar work is
therefore corpus based method which doesnt require the expensive resources. The algorithm
first gathers all the contexts of a given word and then cluster them by using Cluster BY
Committee (CBC) algorithm.
number of clusters produced is considered as the number of the senses of the target word.
CBC takes a word type as input, and finds clusters of words that represent each sense of the word
(Lin and Pantel 2002). The clusters will be made up of synonyms or words that are related to the
discovered senses. For example, if chair is input, CBC produces two sets: the first including
president, while the second includes house chair that used in the house.
14
The premise behind the approach is that words having multiple senses or ambiguous words have
different distinct groups of contexts. Usually such contexts are expressed in terms of words.
Therefore one can identify senses of a given word by retrieving all the words occurring in all the
contexts of the words and group the words depending on the contexts. Lets consider our bank
example again to illustrate the idea. The term bank has at least 2 contexts: financial institution vs
portion of land. The first sense is expressed with a certain set of words like finance, money,
deposit, interest, loan etc where as the second sense is expressed by completely different sets of
words which include land, river, water, sand etc.. One possible way of disambiguating the senses
of a given word can therefore be achieved by identifying such distinct sets of words which can
be obtained by extracting words co-occurring with the term followed by clustering them. As
depicted in the above table (table 1), the approach extract all the contexts of a given word. To
this end we relied on the co-occurrence information to retrieve the contexts. Accordingly all the
words appearing with the target word in a fixed sized window of words is considered as context.
The extracted words are grouped and labeled. Each group is considered as a sense of the target
word.
3.2. Evaluation
We used a portion of Wikipedia text to train and evaluate the algorithm. The system is therefore
made to identify the sense of certain sets of word from Wikipedia. To this end, we deliberately
selected 10 words 6 ambiguous and the remaining 4 unambiguous words and provided to the
system. We then manually evaluated the result produced by the system. The task is basically
clustering of the 10 words in to the correct classes: ambiguous or unambiguous. Our strategy to
evaluate the performance is therefore counting the number of words assigned to the correct class
ambiguous or unambiguous class. Accordingly out of the 6 ambiguous words 3 of them are
correctly classified as ambiguous giving the performance level of 51%. Similarly out of the 4
unambiguous words 3 of them are correctly classified as unambiguous giving the performance
level of 49%. The average performance level is therefore 50%.
14
One can also evaluate the performance of the algorithm by counting the number of senses
produced for a given word and the correctness of the label produced by the system which is
simply out of the scope of our current work
5. Conclusion
In computational linguistics, word-sense disambiguation (WSD) is an open problem of natural
language processing, which governs the process of identifying which sense of a word (i.e.
meaning) is used in a sentence, when the word has multiple meanings. The solution to this
problem impacts other computer-related writing, such as discourse, improving relevance of
search engines, anaphora resolution, coherence, inference.
The role of this seminar also to solve word-sense disambiguation (WSD) in natural language.
algorithm used in this work is known as Cluster By Committee(CBC). We relied Wikipedia
corpus to train the algorithm. The system take input from user and return the senses of the word
as an output. We have evaluated the system and obtained a promising result.
14
6. Future Work
Word sense is young especially in local language like Ethiopians language a lot of works are yet
to be done in the future.
languages.
Reference
14
14
APPENDIX A
- 2.7 version
14