Unit 3

Introduction to Semantic Analysis

Semantic Analysis is a subfield of Natural Language Processing (NLP) that
attempts to understand the meaning of Natural Language. Understanding
Natural Language might seem a straightforward process to us as humans.
However, due to the vast complexity and subjectivity involved in human
language, interpreting it is quite a complicated task for machines. Semantic
Analysis of Natural Language captures the meaning of the given text while
taking into account context, logical structuring of sentences and grammar
roles.

Parts of Semantic Analysis

Semantic Analysis of Natural Language can be classified into two broad
parts:
1. Lexical Semantic Analysis: Lexical Semantic Analysis involves
understanding the meaning of each word of the text individually. It basically
refers to fetching the dictionary meaning that a word in the text is deputed to
carry.
2. Compositional Semantics Analysis: Although knowing the meaning of
each word of the text is essential, it is not sufficient to completely understand
the meaning of the text.
For example, consider the following two sentences:
• Sentence 1: Students love GeeksforGeeks.
• Sentence 2: GeeksforGeeks loves Students.
Although both these sentences 1 and 2 use the same set of root words
{student, love, geeksforgeeks}, they convey entirely different meanings.
Hence, under Compositional Semantics Analysis, we try to understand how
combinations of individual words form the meaning of the text.

Tasks involved in Semantic Analysis

In order to understand the meaning of a sentence, the following are the
major processes involved in Semantic Analysis:
1. Word Sense Disambiguation
2. Relationship Extraction

Word Sense Disambiguation:

In Natural Language, the meaning of a word may vary as per its usage in
sentences and the context of the text. Word Sense Disambiguation involves
interpreting the meaning of a word based upon the context of its occurrence
in a text.
For example, the word ‘Bark’ may mean ‘the sound made by a dog’ or ‘the
outermost layer of a tree.’
Likewise, the word ‘rock’ may mean ‘a stone‘ or ‘a genre of music‘ – hence,
the accurate meaning of the word is highly dependent upon its context and
usage in the text.
Thus, the ability of a machine to overcome the ambiguity involved in
identifying the meaning of a word based on its usage and context is called
Word Sense Disambiguation.

Relationship Extraction:

Another important task involved in Semantic Analysis is Relationship

Extracting. It involves firstly identifying various entities present in the
sentence and then extracting the relationships between those entities.
For example, consider the following sentence:
Semantic Analysis is a topic of NLP which is explained on the GeeksforGeeks
blog. The entities involved in this text, along with their relationships, are
shown below.

Entities

Relationships
Elements of Semantic Analysis
Some of the critical elements of Semantic Analysis that must be scrutinized
and taken into account while processing Natural Language are:
• Hyponymy: Hyponymys refers to a term that is an instance of a
generic term. They can be understood by taking class-object as an
analogy. For example: ‘Color‘ is a hypernymy while ‘grey‘, ‘blue‘,
‘red‘, etc, are its hyponyms.
• Homonymy: Homonymy refers to two or more lexical terms with
the same spellings but completely distinct in meaning. For
example: ‘Rose‘ might mean ‘the past form of rise‘ or ‘a flower‘, –
same spelling but different meanings; hence, ‘rose‘ is a homonymy.
• Synonymy: When two or more lexical terms that might be spelt
distinctly have the same or similar meaning, they are called
Synonymy. For example: (Job, Occupation), (Large, Big), (Stop,
Halt).
• Antonymy: Antonymy refers to a pair of lexical terms that have
contrasting meanings – they are symmetric to a semantic axis. For
example: (Day, Night), (Hot, Cold), (Large, Small).
• Polysemy: Polysemy refers to lexical terms that have the same
spelling but multiple closely related meanings. It differs from
homonymy because the meanings of the terms need not be closely
related in the case of homonymy. For example: ‘man‘ may mean
‘the human species‘ or ‘a male human‘ or ‘an adult male human‘ –
since all these different meanings bear a close association, the
lexical term ‘man‘ is a polysemy.
• Meronomy: Meronomy refers to a relationship wherein one lexical
term is a constituent of some larger entity. For example: ‘Wheel‘ is
a meronym of ‘Automobile‘
Meaning Representation
While, as humans, it is pretty simple for us to understand the meaning of
textual information, it is not so in the case of machines. Thus, machines tend
to represent the text in specific formats in order to interpret its meaning.
This formal structure that is used to understand the meaning of a text is
called meaning representation.

Basic Units of Semantic System:

In order to accomplish Meaning Representation in Semantic Analysis, it is
vital to understand the building units of such representations. The basic
units of semantic systems are explained below:
1. Entity: An entity refers to a particular unit or individual in specific
such as a person or a location. For example GeeksforGeeks, Delhi,
etc.
2. Concept: A Concept may be understood as a generalization of
entities. It refers to a broad class of individual units. For example
Learning Portals, City, Students.
3. Relations: Relations help establish relationships between various
entities and concepts. For example: ‘GeeksforGeeks is a Learning
Portal’, ‘Delhi is a City.’, etc.
4. Predicate: Predicates represent the verb structures of the
sentences.
In Meaning Representation, we employ these basic units to represent
textual information.

Approaches to Meaning Representations:

Now that we are familiar with the basic understanding of Meaning

Representations, here are some of the most popular approaches to meaning
representation:
1. First-order predicate logic (FOPL)
2. Semantic Nets
3. Frames
4. Conceptual dependency (CD)
5. Rule-based architecture
6. Case Grammar
7. Conceptual Graphs
Semantic Analysis Techniques
Based upon the end goal one is trying to accomplish, Semantic Analysis can
be used in various ways. Two of the most common Semantic Analysis
techniques are:

Text Classification
In-Text Classification, our aim is to label the text according to the insights
we intend to gain from the textual data.
For example:
 • In Sentiment Analysis, we try to label the text with the prominent
emotion they convey. It is highly beneficial when analyzing
customer reviews for improvement.
• In Topic Classification, we try to categories our text into some
predefined categories. For example: Identifying whether a research
paper is of Physics, Chemistry or Maths
• In Intent Classification, we try to determine the intent behind a
text message. For example: Identifying whether an e-mail received
at customer care service is a query, complaint or request.

Text Extraction

In-Text Extraction, we aim at obtaining specific information from our text.

For Example,
• In Keyword Extraction, we try to obtain the essential words that
define the entire document.
• In Entity Extraction, we try to obtain all the entities involved in a
document.
Significance of Semantics Analysis
Semantics Analysis is a crucial part of Natural Language Processing (NLP).
In the ever-expanding era of textual information, it is important for
organizations to draw insights from such data to fuel businesses. Semantic
Analysis helps machines interpret the meaning of texts and extract useful
information, thus providing invaluable data while reducing manual efforts.
Besides, Semantics Analysis is also widely employed to facilitate the
processes of automated answering systems such as chatbots – that answer
user queries without any human interventions.

Lexical semantics
Lexical semantics is the study of the meaning of words and how they relate
to each other. It focuses on understanding the meaning of individual words,
their relationships with other words, and the way in which they contribute
to the overall meaning of sentences and discourse. The field of lexical
semantics explores various aspects of word meaning, including:
 1. Word Sense: Many words have multiple senses or meanings. Lexical
semantics aims to identify and describe these different senses and
understand how they are used in different contexts. For example, the word
"bank" can refer to a financial institution, a riverbank, or a slope.
2. Word Relations: Lexical semantics investigates the relationships between
words, such as synonymy (similar meaning), antonymy (opposite meaning),
hyponymy (hierarchical relation), and meronymy (part-whole relation).
These relationships help us understand how words are organized in
semantic networks.
3. Polysemy: Polysemy refers to the phenomenon where a single word has
multiple related meanings. For example, the word "foot" can refer to the
body part, a unit of measurement, or the bottom part of something. Lexical
semantics examines how polysemous words acquire and maintain their
related senses.
4. Word Formation: Lexical semantics also explores how words are formed,
including processes such as derivation, compounding, and inflection. It
investigates how these processes affect the meaning of words and how
new words are created.
5. Idioms and Collocations: Idioms are fixed expressions whose meaning
cannot be inferred from the individual words within them. Lexical semantics
studies idiomatic expressions and collocations (word combinations that
commonly occur together) to understand how the meaning of these phrases
is derived from the meanings of their constituent words.
6. Word Ambiguity: As mentioned earlier, ambiguity is an important aspect
of lexical semantics. It examines the different types of ambiguity that can
occur due to multiple word senses, syntactic structures, or contextual
factors. The study of lexical semantics has practical applications in various
fields, including natural language processing, computational linguistics,
lexicography, and language teaching. Understanding the subtle nuances
and relationships between words helps improve language understanding
and generation systems, as well as language learning and communication in
general.
Ambiguity
Ambiguity refers to a situation where a word, phrase, sentence, or even an
entire discourse can be interpreted in more than one way, leading to
uncertainty or confusion regarding its intended meaning. Ambiguity is a
common phenomenon in natural language due to the complexity and
flexibility of human communication. It can arise from various sources,
including lexical choices, syntax, semantics, and pragmatics.

There are different types of ambiguity:

1. Lexical Ambiguity: Lexical ambiguity occurs when a word has multiple

meanings. For example, the word "bank" can refer to a financial institution
or the side of a river. The specific meaning depends on the context in which
the word is used.

2. Structural Ambiguity: Structural ambiguity arises when a sentence or

phrase has more than one possible interpretation due to its syntactic
structure. For example, consider the sentence "I saw the man with the
telescope." It can be interpreted as either "I used a telescope to see the
man" or "I saw the man who had a telescope."

3. Semantic Ambiguity: Semantic ambiguity arises when the meaning of a

word or phrase is unclear due to its inherent vagueness or multiple
interpretations. For example, the phrase "Time flies like an arrow" can be
understood in different ways, such as "Time passes quickly, similar to the
flight of an arrow" or "Different types of flies are attracted to arrows."

4. Pragmatic Ambiguity: Pragmatic ambiguity arises from the context or

situation in which language is used. It occurs when the intended meaning
relies on shared knowledge, implicatures, or conversational implicatures.
For example, the statement "It's warm in here" can be interpreted as a
factual observation or as a request to adjust the temperature.
Ambiguity can pose challenges in communication and comprehension, as it
requires the listener or reader to disambiguate the intended meaning based
on context, background knowledge, and linguistic cues. In some cases,
ambiguity can be deliberately used for rhetorical effect, humor, or artistic
purposes. However, in other situations, ambiguity can lead to
miscommunication or misunderstanding. Resolving ambiguity often involves
considering contextual clues, pragmatic inferences, and applying knowledge
of language conventions.

We understand that words have different meanings based on the context of its usage
in the sentence. If we talk about human languages, then they are ambiguous too
because many words can be interpreted in multiple ways depending upon the
context of their occurrence.
Word sense disambiguation, in natural language processing (NLP), may be defined
as the ability to determine which meaning of word is activated by the use of word in
a particular context. Lexical ambiguity, syntactic or semantic, is one of the very first
problem that any NLP system faces. Part-of-speech (POS) taggers with high level of
accuracy can solve Word’s syntactic ambiguity. On the other hand, the problem of
resolving semantic ambiguity is called WSD (word sense disambiguation). Resolving
semantic ambiguity is harder than resolving syntactic ambiguity.
For example, consider the two examples of the distinct sense that exist for the
word “bass” −
• I can hear bass sound.
• He likes to eat grilled bass.
The occurrence of the word bass clearly denotes the distinct meaning. In first
sentence, it means frequency and in second, it means fish. Hence, if it would be
disambiguated by WSD then the correct meaning to the above sentences can be
assigned as follows −
• I can hear bass/frequency sound.
• He likes to eat grilled bass/fish.

Evaluation of WSD
The evaluation of WSD requires the following two inputs −
A Dictionary
The very first input for evaluation of WSD is dictionary, which is used to specify the
senses to be disambiguated.

Test Corpus
Another input required by WSD is the high-annotated test corpus that has the target
or correct-senses. The test corpora can be of two types &minsu;
• Lexical sample − This kind of corpora is used in the system, where it is
required to disambiguate a small sample of words.
• All-words − This kind of corpora is used in the system, where it is
expected to disambiguate all the words in a piece of running text.

Approaches and Methods to Word Sense Disambiguation

(WSD)
Approaches and methods to WSD are classified according to the source of
knowledge used in word disambiguation.
Let us now see the four conventional methods to WSD −

Dictionary-based or Knowledge-based Methods

As the name suggests, for disambiguation, these methods primarily rely on
dictionaries, treasures and lexical knowledge base. They do not use corpora
evidences for disambiguation. The Lesk method is the seminal dictionary-based
method introduced by Michael Lesk in 1986. The Lesk definition, on which the Lesk
algorithm is based is “measure overlap between sense definitions for all words in
context”. However, in 2000, Kilgarriff and Rosensweig gave the simplified Lesk
definition as “measure overlap between sense definitions of word and current
context”, which further means identify the correct sense for one word at a time. Here
the current context is the set of words in surrounding sentence or paragraph.

Supervised Methods
For disambiguation, machine learning methods make use of sense-annotated corpora
to train. These methods assume that the context can provide enough evidence on its
own to disambiguate the sense. In these methods, the words knowledge and
reasoning are deemed unnecessary. The context is represented as a set of “features”
of the words. It includes the information about the surrounding words also. Support
vector machine and memory-based learning are the most successful supervised
learning approaches to WSD. These methods rely on substantial amount of manually
sense-tagged corpora, which is very expensive to create.
Semi-supervised Methods
Due to the lack of training corpus, most of the word sense disambiguation algorithms
use semi-supervised learning methods. It is because semi-supervised methods use
both labelled as well as unlabeled data. These methods require very small amount
of annotated text and large amount of plain unannotated text. The technique that is
used by semisupervised methods is bootstrapping from seed data.

Unsupervised Methods
These methods assume that similar senses occur in similar context. That is why the
senses can be induced from text by clustering word occurrences by using some
measure of similarity of the context. This task is called word sense induction or
discrimination. Unsupervised methods have great potential to overcome the
knowledge acquisition bottleneck due to non-dependency on manual efforts.

Applications of Word Sense Disambiguation (WSD)

Word sense disambiguation (WSD) is applied in almost every application of
language technology.
Let us now see the scope of WSD −

Machine Translation
Machine translation or MT is the most obvious application of WSD. In MT, Lexical
choice for the words that have distinct translations for different senses, is done by
WSD. The senses in MT are represented as words in the target language. Most of the
machine translation systems do not use explicit WSD module.

Information Retrieval (IR)

Information retrieval (IR) may be defined as a software program that deals with the
organization, storage, retrieval and evaluation of information from document
repositories particularly textual information. The system basically assists users in
finding the information they required but it does not explicitly return the answers of
the questions. WSD is used to resolve the ambiguities of the queries provided to IR
system. As like MT, current IR systems do not explicitly use WSD module and they
rely on the concept that user would type enough context in the query to only retrieve
relevant documents.

Text Mining and Information Extraction (IE)

In most of the applications, WSD is necessary to do accurate analysis of text. For
example, WSD helps intelligent gathering system to do flagging of the correct
words. For example, medical intelligent system might need flagging of “illegal drugs”
rather than “medical drugs”

Lexicography
WSD and lexicography can work together in loop because modern lexicography is
corpus based. With lexicography, WSD provides rough empirical sense groupings as
well as statistically significant contextual indicators of sense.

Difficulties in Word Sense Disambiguation (WSD)

Followings are some difficulties faced by word sense disambiguation (WSD) −

Differences between dictionaries

The major problem of WSD is to decide the sense of the word because different
senses can be very closely related. Even different dictionaries and thesauruses can
provide different divisions of words into senses.

Different algorithms for different applications

Another problem of WSD is that completely different algorithm might be needed for
different applications. For example, in machine translation, it takes the form of target
word selection; and in information retrieval, a sense inventory is not required.

Inter-judge variance
Another problem of WSD is that WSD systems are generally tested by having their
results on a task compared against the task of human beings. This is called the
problem of inter judge variance.

Word-sense discreteness
Another difficulty in WSD is that words cannot be easily divided into discrete sub
meanings.

The most difficult problem of AI is to process the natural language by computers or

in other words natural language processing is the most difficult problem of artificial
intelligence. If we talk about the major problems in NLP, then one of the major
problems in NLP is discourse processing − building theories and models of how
utterances stick together to form coherent discourse. Actually, the language always
consists of collocated, structured and coherent groups of sentences rather than
isolated and unrelated sentences like movies. These coherent groups of sentences
are referred to as discourse.
Concept of Coherence
Coherence and discourse structure are interconnected in many ways. Coherence,
along with property of good text, is used to evaluate the output quality of natural
language generation system. The question that arises here is what does it mean for
a text to be coherent? Suppose we collected one sentence from every page of the
newspaper, then will it be a discourse? Of-course, not. It is because these sentences
do not exhibit coherence. The coherent discourse must possess the following
properties −

Coherence relation between utterances

The discourse would be coherent if it has meaningful connections between its
utterances. This property is called coherence relation. For example, some sort of
explanation must be there to justify the connection between utterances.

Relationship between entities

Another property that makes a discourse coherent is that there must be a certain kind
of relationship with the entities. Such kind of coherence is called entity-based
coherence.

Discourse structure
An important question regarding discourse is what kind of structure the discourse
must have. The answer to this question depends upon the segmentation we applied
on discourse. Discourse segmentations may be defined as determining the types of
structures for large discourse. It is quite difficult to implement discourse
segmentation, but it is very important for information retrieval, text summarization
and information extraction kind of applications.

Algorithms for Discourse Segmentation

In this section, we will learn about the algorithms for discourse segmentation. The
algorithms are described below −

Unsupervised Discourse Segmentation

The class of unsupervised discourse segmentation is often represented as linear
segmentation. We can understand the task of linear segmentation with the help of
an example. In the example, there is a task of segmenting the text into multi-
paragraph units; the units represent the passage of the original text. These
algorithms are dependent on cohesion that may be defined as the use of certain
linguistic devices to tie the textual units together. On the other hand, lexicon cohesion
is the cohesion that is indicated by the relationship between two or more words in
two units like the use of synonyms.

Supervised Discourse Segmentation

The earlier method does not have any hand-labeled segment boundaries. On the
other hand, supervised discourse segmentation needs to have boundary-labeled
training data. It is very easy to acquire the same. In supervised discourse
segmentation, discourse marker or cue words play an important role. Discourse
marker or cue word is a word or phrase that functions to signal discourse structure.
These discourse markers are domain-specific.

Text Coherence
Lexical repetition is a way to find the structure in a discourse, but it does not satisfy
the requirement of being coherent discourse. To achieve the coherent discourse, we
must focus on coherence relations in specific. As we know that coherence relation
defines the possible connection between utterances in a discourse. Hebb has
proposed such kind of relations as follows −
We are taking two terms S0 and S1 to represent the meaning of the two related
sentences −

Result
It infers that the state asserted by term S0 could cause the state asserted by S1. For
example, two statements show the relationship result: Ram was caught in the fire.
His skin burned.

Explanation
It infers that the state asserted by S1 could cause the state asserted by S0. For
example, two statements show the relationship − Ram fought with Shyam’s friend.
He was drunk.

Parallel
It infers p(a1,a2,…) from assertion of S0 and p(b1,b2,…) from assertion S1. Here ai and
bi are similar for all i. For example, two statements are parallel − Ram wanted car.
Shyam wanted money.

Elaboration
It infers the same proposition P from both the assertions − S0 and S1 For example,
two statements show the relation elaboration: Ram was from Chandigarh. Shyam
was from Kerala.
Occasion
It happens when a change of state can be inferred from the assertion of S0, final state
of which can be inferred from S1 and vice-versa. For example, the two statements
show the relation occasion: Ram picked up the book. He gave it to Shyam.

Building Hierarchical Discourse Structure

The coherence of entire discourse can also be considered by hierarchical structure
between coherence relations. For example, the following passage can be
represented as hierarchical structure −
• S1 − Ram went to the bank to deposit money.
• S2 − He then took a train to Shyam’s cloth shop.
• S3 − He wanted to buy some clothes.
• S4 − He do not have new clothes for party.
• S5 − He also wanted to talk to Shyam regarding his health

Reference Resolution
Interpretation of the sentences from any discourse is another important task and to
achieve this we need to know who or what entity is being talked about. Here,
interpretation reference is the key element. Reference may be defined as the
linguistic expression to denote an entity or individual. For example, in the
passage, Ram, the manager of ABC bank, saw his friend Shyam at a shop. He went
to meet him, the linguistic expressions like Ram, His, He are reference.
On the same note, reference resolution may be defined as the task of determining
what entities are referred to by which linguistic expression.

Terminology Used in Reference Resolution

We use the following terminologies in reference resolution −
• Referring expression − The natural language expression that is used to
perform reference is called a referring expression. For example, the
passage used above is a referring expression.
• Referent − It is the entity that is referred. For example, in the last given
example Ram is a referent.
• Corefer − When two expressions are used to refer to the same entity,
they are called corefers. For example, Ram and he are corefers.
• Antecedent − The term has the license to use another term. For
example, Ram is the antecedent of the reference he.
• Anaphora & Anaphoric − It may be defined as the reference to an entity
that has been previously introduced into the sentence. And, the referring
expression is called anaphoric.
• Discourse model − The model that contains the representations of the
entities that have been referred to in the discourse and the relationship
they are engaged in.

Types of Referring Expressions

Let us now see the different types of referring expressions. The five types of referring
expressions are described below −

Indefinite Noun Phrases

Such kind of reference represents the entities that are new to the hearer into the
discourse context. For example − in the sentence Ram had gone around one day to
bring him some food − some is an indefinite reference.

Definite Noun Phrases

Opposite to above, such kind of reference represents the entities that are not new or
identifiable to the hearer into the discourse context. For example, in the sentence - I
used to read The Times of India – The Times of India is a definite reference.
Pronouns
It is a form of definite reference. For example, Ram laughed as loud as he could. The
word he represents pronoun referring expression.

Demonstratives
These demonstrate and behave differently than simple definite pronouns. For
example, this and that are demonstrative pronouns.

Names
It is the simplest type of referring expression. It can be the name of a person,
organization and location also. For example, in the above examples, Ram is the name-
refereeing expression.

Reference Resolution Tasks

The two reference resolution tasks are described below.

Coreference Resolution
It is the task of finding referring expressions in a text that refer to the same entity. In
simple words, it is the task of finding corefer expressions. A set of coreferring
expressions are called coreference chain. For example - He, Chief Manager and His -
these are referring expressions in the first passage given as example.

Constraint on Coreference Resolution

In English, the main problem for coreference resolution is the pronoun it. The reason
behind this is that the pronoun it has many uses. For example, it can refer much like
he and she. The pronoun it also refers to the things that do not refer to specific things.
For example, It’s raining. It is really good.

Pronominal Anaphora Resolution

Unlike the coreference resolution, pronominal anaphora resolution may be defined as
the task of finding the antecedent for a single pronoun. For example, the pronoun is
his and the task of pronominal anaphora resolution is to find the word Ram because
Ram is the antecedent.
UNIT 4
Extracting relations from text is a key task in natural language processing and
information extraction. It involves identifying and extracting structured information
about relationships between entities mentioned in the text. These relationships can
represent various types of connections, such as associations, interactions,
dependencies, or affiliations.
Here are the general steps involved in extracting relations from text:
1. Named Entity Recognition (NER): The first step is to identify and extract named
entities from the text. Named entities can be entities like persons, organizations,
locations, dates, or other specific terms of interest. NER techniques use machine
learning or rule-based approaches to identify and classify these entities.

2. Entity Linking/Resolution: After identifying named entities, entity linking or

resolution is performed to disambiguate and link the entities to specific knowledge
bases or unique identifiers. This step helps in identifying the correct references for
entities, especially when multiple entities have the same name.

3. Relation Extraction: Once the entities are identified and linked, the next step is to
extract the relationships between them. Relation extraction techniques involve
analyzing the syntactic and semantic patterns in the text to identify the relevant
linguistic cues that indicate a relationship. These cues can include verbs, prepositions,
adjectives, or specific patterns of words.

- Rule-based Approaches: Rule-based systems define patterns or rules to identify

relationships based on specific syntactic or lexical patterns. These patterns can be
defined manually or derived from linguistic resources.

- Supervised Machine Learning: Supervised learning approaches utilize labeled

training data, where human annotators mark the relationships between entities in a
given context. Machine learning algorithms are trained on this data to learn patterns
and features that indicate relationships. The trained models can then predict
relationships in new, unseen text.

- Unsupervised and Semi-Supervised Approaches: Unsupervised and semi-

supervised methods aim to discover relationships without relying on labeled training
data. They utilize statistical techniques, clustering algorithms, or co-occurrence
patterns to identify patterns or associations between entities.

4. Relationship Classification: Once relations are extracted, they can be further

classified into specific types or categories. For example, in a medical text, relations
between drugs and side effects can be classified into categories like "causes,"
"treats," or "has side effect." Classification can be done using machine learning
algorithms trained on labeled data or through rule-based approaches.

5. Post-processing and Validation: After extracting and classifying relations, post-

processing steps can be performed to refine the results. This may involve removing
duplicate or irrelevant relations, resolving conflicts or inconsistencies, and validating
the extracted relations against domain-specific knowledge or external resources.

Extracting relations from text has applications in various fields, including knowledge
graph construction, question answering, recommendation systems, and information
retrieval. It enables the extraction of structured knowledge from unstructured text,
facilitating further analysis and knowledge discovery.

From word sequences to dependency paths

From word sequences to dependency paths, the shift represents a more structured
and syntactic approach to understanding the relationships between words in a
sentence. While word sequences provide basic linear information about the order of
words, dependency paths capture the hierarchical and grammatical connections
between words.

Dependency parsing is the key technique that enables the transition from word
sequences to dependency paths. It involves analyzing the grammatical structure of a
sentence and establishing the syntactic relationships between words. The result is a
dependency tree, which represents the hierarchical structure of the sentence and the
dependencies (links) between words.

In a dependency tree, each word is represented as a node, and the directed links
between the nodes indicate the syntactic relationships. These relationships typically
include dependencies such as subject, object, modifier, or conjunction. The tree
structure allows us to navigate from one word to another through the links, forming
a dependency path.

Dependency paths provide a more precise representation of how words are

connected within a sentence. Instead of considering only the immediate neighbors of
a word, dependency paths consider all the words in the path between two entities,
capturing the syntactic roles and dependencies that connect them.

By extracting and analyzing dependency paths, we can uncover valuable linguistic

information. Dependency paths help identify syntactic patterns, dependencies, and
semantic relationships between entities. They allow for more accurate and context-
aware relation extraction, where the relations between entities are determined not
only by their proximity but also by the syntactic structure and the specific
grammatical roles they play in the sentence.

The shift from word sequences to dependency paths enhances the understanding of
language, enabling more sophisticated natural language processing tasks such as
information extraction, question answering, sentiment analysis, and text
summarization. Dependency-based approaches provide a deeper insight into the
syntactic and semantic connections between words, leading to more robust and
accurate analysis of textual data.

Subsequence kernels:
Subsequence kernels are a type of kernel function used in machine learning to
compare and measure the similarity between sequences, such as sentences or
phrases. They have been successfully applied to relation extraction tasks to capture
the relationship between entities in a sentence based on the subsequences of words
that occur between them.

In the context of relation extraction, the goal is to identify and classify the relationship
between two entities mentioned in a sentence. Subsequence kernels offer a way to
represent and compare the sequences of words between the entities, allowing for
effective relation extraction.
Subsequence kernels:
Here is an overview of how subsequence kernels can be used for relation extraction:

1. Entity Recognition: The first step is to identify and recognize the entities of interest
in the sentence. Named entity recognition (NER) techniques can be employed to
identify entity mentions, such as person names, organizations, or locations.

2. Subsequence Extraction: Once the entities are identified, the subsequences of

words between the entities are extracted. These subsequences capture the linguistic
context and information that may indicate the relationship between the entities.

- The subsequences can be defined as the words that occur between the starting
and ending positions of the two entities in the sentence.

- The length of the subsequences can vary, ranging from a few words to the entire
sentence or a fixed window of words around the entities.

3. Feature Representation: To compare and measure the similarity between different

subsequences, feature representations are created. Subsequence features can be
based on various linguistic properties, such as the presence of certain words, part-
of-speech tags, syntactic dependencies, or word embeddings.

4. Subsequence Kernel Computation: Subsequence kernels compute the similarity

between pairs of subsequences. They measure the similarity based on the shared
features or patterns in the subsequences.

- Different types of subsequence kernels can be used, such as the linear kernel,
string kernel, or graph kernel. Each kernel employs specific techniques to compare
the subsequences, such as string matching, subgraph matching, or statistical
measures.
 - The kernel computation results in a similarity matrix or vector that represents the
pairwise similarity between the subsequences.

5. Relation Classification: The similarity matrix or vector obtained from the

subsequence kernel can be used as input to a machine learning algorithm for relation
classification. This can be done using techniques such as support vector machines
(SVMs), convolutional neural networks (CNNs), or recurrent neural networks (RNNs).

- The machine learning model learns to classify the relationships between the
entities based on the similarity scores and other relevant features.

Subsequence kernels provide a flexible and powerful approach for relation extraction
by capturing the contextual information between entities. They enable the
comparison and similarity measurement of subsequences, allowing for effective
modeling of the relationship between entities in a sentence. By incorporating
subsequence kernels into relation extraction pipelines, accurate and robust relation
classification can be achieved, facilitating various downstream applications in natural
language processing and information extraction.

A Dependency-Path Kernel for Relation Extraction and Experimental Evaluation:

The paper titled "A Dependency-Path Kernel for Relation Extraction and
Experimental Evaluation" introduces a novel kernel method for relation extraction
using dependency paths. The authors propose a kernel function that captures the
syntactic dependencies between entities in a sentence and utilizes this information
to classify the relationship between them.

The key contributions and experimental evaluation of the proposed dependency-

path kernel are as follows:

1. Dependency-Path Kernel: The paper introduces a kernel function that measures

the similarity between pairs of dependency paths. Dependency paths are sequences
of words and their corresponding dependency relations that connect two entities in
a sentence. The kernel function computes the similarity between these paths based
on their shared substructures and patterns.
2. Feature Extraction: The authors extract features from the dependency paths to
represent their structural and semantic properties. These features include the words,
POS tags, and dependency relations along the path. By incorporating these features,
the kernel can capture both syntactic and semantic information.

3. Relation Classification: The extracted features and the dependency-path kernel are
used as input to a relation classification algorithm. The authors employ support
vector machines (SVMs) to train and classify the relations between entities. The
SVMs learn the patterns and relationships in the data and can predict the relationship
type for unseen entity pairs.

4. Experimental Evaluation: The proposed approach is evaluated on standard

benchmark datasets for relation extraction, such as ACE (Automatic Content
Extraction) and SemEval. The performance of the dependency-path kernel is
compared with other state-of-the-art methods, including bag-of-words, syntactic,
and handcrafted feature-based approaches.
- The evaluation metrics include precision, recall, and F1-score, which measure the
accuracy and effectiveness of relation extraction.
- The experimental results demonstrate that the dependency-path kernel
outperforms other methods, achieving higher accuracy and better relation
classification performance.
- The paper also provides detailed analysis and comparisons of different features
and kernel variations, highlighting the effectiveness of the proposed approach.
The paper's findings highlight the importance of utilizing syntactic dependencies
captured by dependency paths for relation extraction. The proposed dependency-
path kernel shows promising results in accurately identifying and classifying
relations between entities in sentences. It contributes to the field of relation
extraction by leveraging structural information and providing a more robust and
accurate approach to capturing the syntactic dependencies between entities.
Domain Knowledge and Knowledge Roles
Domain knowledge refers to the information, concepts, and expertise specific to a
particular field or subject area. It encompasses the understanding of key concepts,
principles, facts, and relationships within that domain. Domain knowledge plays a
crucial role in various applications, including problem-solving, decision-making, and
knowledge-based systems.

In the context of the paper "Mining Diagnostic Text Reports by Learning to Annotate
Knowledge Roles," domain knowledge refers to the specialized knowledge related
to the healthcare domain, particularly in the context of diagnostic information. It
involves understanding the terminology, diseases, symptoms, treatments, and other
relevant aspects specific to the healthcare field.

Knowledge roles, on the other hand, represent specific types or categories of

information within a given domain. They define the roles or functions that different
pieces of information play in the overall knowledge representation. In the context of
the paper, knowledge roles refer to the distinct categories of diagnostic information
present in text reports. These roles may include the disease or condition being
diagnosed, the symptoms experienced by the patient, the prescribed treatments, or
any other relevant details that contribute to understanding the diagnosis and
treatment process.

Annotating knowledge roles involves identifying and labeling specific portions of the
text reports that correspond to these roles. This annotation process helps in
structuring and organizing the diagnostic information, making it easier to extract and
utilize. By learning to annotate knowledge roles through machine learning
techniques, the proposed approach in the paper enables automated extraction of
valuable diagnostic information from text reports, facilitating further analysis and
decision support in the healthcare domain.
Overall, domain knowledge and knowledge roles are essential components in
extracting and understanding specialized information within a particular domain.
They enable effective information retrieval, analysis, and decision-making by
leveraging the expertise and structured representation of knowledge in that specific
field.
The paper titled "Mining Diagnostic Text Reports by Learning to Annotate
Knowledge Roles" presents a novel approach for mining diagnostic information from
text reports. The authors propose a technique that learns to annotate knowledge
roles in the reports, enabling the extraction of specific information related to
diagnoses, symptoms, treatments, and other relevant aspects.

The introduction of the paper outlines the motivation behind the research and
provides an overview of the proposed method. Here is a summary of the introduction:

1. Motivation: The authors emphasize the importance of mining diagnostic

information from text reports in the healthcare domain. Diagnostic reports contain
valuable knowledge about patients' conditions, diseases, and treatments, but
extracting this information automatically is a challenging task. The paper addresses
this challenge by introducing a learning-based approach to annotate knowledge
roles in diagnostic reports.

2. Knowledge Roles: The concept of knowledge roles refers to specific types of

information present in the text reports. Examples of knowledge roles in the medical
domain include the disease or condition being diagnosed, the symptoms exhibited by
the patient, the prescribed treatments, and any additional relevant details.
Annotating these knowledge roles in the reports enables structured extraction and
organization of diagnostic information.

3. Learning to Annotate: The proposed approach involves training a machine learning

model to automatically annotate knowledge roles in diagnostic reports. The model
learns from annotated training data, where human annotators mark the relevant
knowledge roles in the reports. By leveraging this labeled data, the model learns to
identify and annotate the same roles in new, unseen reports.

4. Information Extraction: Once the knowledge roles are annotated in the reports,
information extraction techniques can be applied to extract specific information of
interest. This involves identifying and capturing the relevant details related to
diagnoses, symptoms, treatments, and other aspects. The extracted information can
then be used for various purposes, such as clinical decision support, research
analysis, or building medical knowledge bases.
The introduction sets the stage for the proposed approach, highlighting the
importance of mining diagnostic information from text reports and the challenges
involved. It introduces the concept of knowledge roles and outlines the learning-
based approach to annotate these roles in the reports. By learning to annotate
knowledge roles, the proposed method enables effective information extraction from
diagnostic text reports, ultimately contributing to improved healthcare analysis and
decision-making.

Frame Semantics and Semantic Role Labelling

Frame semantics is a linguistic theory that aims to represent the meaning of words
and sentences based on the underlying conceptual frames or scenarios they evoke.
Frames are mental structures that organize our knowledge and understanding of the
world, capturing the knowledge, beliefs, and expectations associated with specific
situations or events. Frame semantics focuses on identifying and describing these
frames and their associated roles, known as semantic roles.

Semantic role labeling (SRL) is a natural language processing task that involves
identifying and labeling the semantic roles played by different constituents (words
or phrases) in a sentence. Semantic roles capture the underlying relationships and
functions of these constituents in relation to a predicate or event described in the
sentence.

Here is an overview of frame semantics and semantic role labeling:

1. Frame Semantics: Frame semantics views language understanding as the

activation of specific mental frames that represent different conceptual scenarios.
Frames consist of a set of semantic roles, which describe the participants and their
roles within a frame. For example, the frame "Buying" may have roles such as
"Buyer," "Seller," "Item," and "Price."

2. Lexical Units: Words or expressions associated with specific frames are called
lexical units. Lexical units are linked to frames based on their typical usage and the
conceptual associations they evoke. For example, the word "buy" is a lexical unit
associated with the "Buying" frame.
3. Semantic Role Labeling: Semantic role labeling aims to identify and classify the
semantic roles played by the constituents in a sentence. It involves analyzing the
syntactic structure of the sentence and mapping the constituents to their
corresponding semantic roles within a specific frame. For instance, in the sentence
"John bought a book for $10," "John" would be labeled as the "Buyer," "book" as the
"Item," and "$10" as the "Price" within the "Buying" frame.

4. Machine Learning Approaches: SRL can be performed using machine learning

techniques, particularly supervised learning algorithms. Annotated training data is
used to train models that can automatically assign semantic role labels to sentence
constituents. Features such as syntactic parse trees, part-of-speech tags, and
contextual information are typically used to inform the labeling decisions.

5. Applications: SRL has numerous applications in natural language processing,

including question answering, information extraction, machine translation, and
dialogue systems. It helps in understanding the semantic structure of sentences,
extracting important information, and facilitating deeper language understanding.

Frame semantics and semantic role labelling provide a rich framework for
representing and analysing the meaning of language. By identifying the conceptual
frames and assigning semantic roles to sentence constituents, these approaches
enable a deeper understanding of how language relates to our knowledge and
experiences of the world.

Learning to Annotate Cases with Knowledge Roles and Evaluations

Learning to annotate cases with knowledge roles and evaluations refers to the
process of training a machine learning model to automatically assign specific roles
and evaluations to cases or instances within a given domain. This approach enables
the structured annotation of cases based on predefined categories or criteria,
facilitating subsequent analysis and decision-making.
Here is an overview of the process of learning to annotate cases with knowledge
roles and evaluations:

1. Definition of Knowledge Roles and Evaluations: The first step is to define the
knowledge roles and evaluations that are relevant to the specific domain or problem
at hand. Knowledge roles represent the different types or categories of information
that need to be identified and labelled within each case. Evaluations, on the other
hand, capture assessments or judgments associated with the cases.

2. Annotated Training Data: To train a machine learning model, annotated training

data is needed. Human annotators review a set of cases and assign the appropriate
knowledge roles and evaluations to each case based on the predefined categories.
This annotated dataset serves as the training ground for the machine learning model.

3. Feature Extraction: Features need to be extracted from the cases to provide input
for the machine learning model. These features can include textual information,
metadata, contextual information, or any other relevant attributes of the cases. The
goal is to capture the key characteristics that are indicative of the knowledge roles
and evaluations.

4. Model Training: Machine learning algorithms are employed to train a model using
the annotated training data and the extracted features. The choice of algorithm can
vary depending on the specific task and the available data. Techniques such as
supervised learning, deep learning, or ensemble methods can be used to train the
model.

5. Model Evaluation: Once the model is trained, it needs to be evaluated to assess its
performance. Annotated test data, separate from the training data, is used to
evaluate the model's ability to correctly assign knowledge roles and evaluations to
cases. Evaluation metrics such as precision, recall, F1-score, or accuracy are
commonly used to measure the performance of the model.

6. Iterative Refinement: Based on the evaluation results, the model can be further
refined and improved. This may involve adjusting the feature set, experimenting with
different algorithms or parameter settings, or collecting additional annotated data to
enhance the model's performance.
By learning to annotate cases with knowledge roles and evaluations, machine
learning models can automatically assign structured labels to cases, enabling
efficient analysis, decision-making, and knowledge extraction. This approach can be
applied in various domains, including healthcare, finance, legal, customer service, and
more, where structured information and evaluations are critical for effective decision
support.
Unit V
Automatic document separation
Automatic document separation, also known as document classification or document
clustering, refers to the process of automatically categorizing or separating a
collection of documents into different groups or classes based on their content,
characteristics, or other relevant features. It is a common task in information retrieval
and document management systems to organize and classify large document
collections efficiently.

Here is an overview of the process of automatic document separation:

1. Data Collection: The first step is to gather a collection of documents that need to
be separated or classified. These documents can be in various formats such as text
files, PDFs, web pages, or emails.

2. Feature Extraction: To analyze and categorize the documents, relevant features

need to be extracted. Features can include words, phrases, topics, metadata, or any
other attributes that capture the characteristics of the documents. Common
techniques for feature extraction include term frequency-inverse document
frequency (TF-IDF), word embeddings, or topic modelling.

3. Training Data Preparation: To train a machine learning model, a labelled dataset is

required. This dataset consists of a set of documents with pre-assigned classes or
categories. Human annotators assign the appropriate class labels to each document
based on its content or other criteria.

4. Model Training: Machine learning algorithms are used to train a model on the
labelled dataset. Various algorithms can be employed, such as Naive Bayes, Support
Vector Machines (SVM), Random Forests, or Neural Networks. The model learns to
recognize patterns and relationships between the extracted features and the
corresponding document classes.

5. Model Evaluation: The trained model needs to be evaluated to assess its

performance. Annotated test data, separate from the training data, is used to
evaluate how accurately the model can classify or separate the documents into the
correct classes. Evaluation metrics such as precision, recall, F1-score, or accuracy are
commonly used to measure the model's performance.
6. Document Separation: Once the model is trained and evaluated, it can be applied
to new, unseen documents for automatic separation or classification. The model
assigns a class label to each document based on its content and the learned patterns
from the training data.

7. Iterative Improvement: The performance of the model can be iteratively improved

by refining the feature set, experimenting with different algorithms or parameter
settings, or collecting additional labelled data for training.

Automatic document separation has numerous practical applications, such as

organizing large document collections, information retrieval, spam detection,
sentiment analysis, or content filtering. By automatically categorizing documents, it
enables efficient document management, retrieval, and analysis, saving time and
effort in dealing with large volumes of textual data.

A Combination of Probabilistic Classification and Finite-State Sequence:

A combination of probabilistic classification and finite-state sequence models refers
to the integration of these two approaches to tackle complex tasks involving
sequential data. It combines the strengths of probabilistic classification models,
which capture the statistical relationships between input features and output labels,
with the expressiveness and modelling capabilities of finite-state sequence models,
which can handle sequential patterns and dependencies.

Probabilistic Classification:
Probabilistic classification models, such as Naive Bayes, Logistic Regression, or
Random Forests, are based on statistical principles and learn the probabilistic
relationship between input features and output labels. These models estimate the
probability of each class label given the input features and make predictions based
on these probabilities. They are effective for tasks where features can independently
contribute to the prediction or when the interactions between features are relatively
simple.
Finite-State Sequence Models:
Finite-state sequence models, such as Hidden Markov Models (HMMs) or Conditional
Random Fields (CRFs), are specifically designed to handle sequential data. They
model the dependencies between input features and output labels by considering
the entire sequence of observations. These models can capture complex patterns and
dependencies, taking into account the context and ordering of the data. Finite-state
sequence models are commonly used in tasks such as part-of-speech tagging, named
entity recognition, or speech recognition.

Combining Probabilistic Classification and Finite-State Sequence Models:

The combination of these two approaches can be beneficial when dealing with tasks
that involve both individual feature contributions and sequential dependencies.
Here's an overview of how they can be combined:

1. Feature Extraction: Relevant features are extracted from the input data,
considering both individual observations and their contextual information.

2. Probabilistic Classification: Probabilistic classification models are trained to

capture the statistical relationships between the input features and output labels,
focusing on individual feature contributions. These models estimate the conditional
probabilities of the output labels given the input features.

3.Finite-State Sequence Modeling: Finite-state sequence models are utilized to

model the sequential patterns and dependencies in the data. They take into account
the context and ordering of the observations, capturing the dependencies between
consecutive features and labels.
4.Integration: The output probabilities from the probabilistic classification model and
the predictions from the finite-state sequence model are combined using techniques
such as voting, weighted averaging, or stacking. This integration process incorporates
both the individual feature contributions and the sequential dependencies into the
final predictions.
By combining probabilistic classification and finite-state sequence models, the
resulting model can benefit from the complementary strengths of each approach. It
can effectively capture both the statistical relationships between features and labels
and the sequential patterns and dependencies in the data, leading to improved
performance on tasks that require modeling both aspects simultaneously.
Modelling: Introduction

In the context of Natural Language Processing (NLP), modeling refers to the process
of creating computational representations or models that can understand, generate,
or process human language. NLP models aim to capture the complexities of language
and enable computers to perform tasks such as language translation, sentiment
analysis, text classification, question answering, and more. Here's an introduction to
modeling in NLP:

1. Text Representation: One of the fundamental aspects of NLP modeling is

representing text in a way that can be processed by computational algorithms. This
involves transforming raw text into numerical or symbolic representations that
capture the semantic and syntactic information of the text. Common techniques for
text representation include bag-of-words, word embeddings (e.g., Word2Vec,
GloVe), and contextual embeddings (e.g., BERT, GPT).

2. Task Definition: Clearly define the NLP task you want to solve. It could be text
classification, named entity recognition, sentiment analysis, machine translation,
language generation, or any other language-related problem. Understanding the
task and its objectives is crucial for selecting the appropriate modeling approach.

3. Pretrained Models: NLP has seen significant advancements with the availability of
large-scale pretrained language models. These models are trained on massive
amounts of text data to learn general language representations. Pretrained models,
such as BERT, GPT, and Transformer-based models, can be fine-tuned for specific
downstream tasks. They provide a powerful starting point for many NLP applications,
allowing you to leverage their contextual understanding of language.

4. Neural Network Architectures: Deep learning and neural networks have

revolutionized NLP modeling. Various neural network architectures have been
developed specifically for NLP tasks. Convolutional Neural Networks (CNNs) and
Recurrent Neural Networks (RNNs), including Long Short-Term Memory (LSTM) and
Gated Recurrent Units (GRUs), have been widely used for tasks like text classification
and sequence labeling. Transformers, with their attention mechanisms, have proven
effective for tasks involving long-range dependencies, such as machine translation
and language generation.

5. Training and Evaluation: Train your NLP model using labeled data specific to your
task. This typically involves an iterative process of feeding input data into the model,
computing predictions, comparing them with the ground truth labels, and updating
the model's parameters to minimize the prediction errors. Evaluation metrics like
accuracy, precision, recall, F1 score.

Document separation as a sequence mapping problem

Document separation as a sequence mapping problem in NLP refers to the task of

accurately splitting concatenated documents into individual documents. It involves
identifying the boundaries or delimiters between different documents within a single
sequence of text. This problem is particularly relevant in scenarios where multiple
documents are combined together, such as in email threads, legal documents, or
news articles. The goal is to develop models or techniques that can automatically
detect and separate these individual documents.

The document separation problem can be framed as a sequence mapping task, where
the input is a sequence of concatenated documents and the output is a sequence of
segmented or separated documents. This problem is challenging due to the absence
of explicit markers or indicators that denote the boundaries between documents. The
models need to learn and generalize patterns from the data to accurately identify the
document boundaries.

Approaches to tackle document separation as a sequence mapping problem in NLP

involve leveraging techniques such as machine learning and deep learning. Some
common approaches include:

1. Rule-based Methods: These methods involve designing specific rules or heuristics

based on patterns or characteristics of the concatenated documents. For example,
identifying common header or footer patterns, document formatting, or specific
keywords that indicate the beginning or end of a document.
2. Statistical Methods: Statistical techniques, such as probabilistic models or hidden
Markov models, can be employed to identify the document boundaries. These
methods learn statistical patterns from the training data and use them to predict the
boundaries in unseen documents.

3. Sequence Labeling: Sequence labeling approaches involve treating document

separation as a sequence labeling task, where each position in the sequence is
assigned a label indicating whether it is the beginning, inside, or end of a document.
Models like Conditional Random Fields (CRF) or Recurrent Neural Networks (RNNs)
can be utilized for sequence labeling.

4. Transformer-based Models: Transformer-based models, such as the popular BERT

(Bidirectional Encoder Representations from Transformers), can be fine-tuned for
document separation. These models have demonstrated strong performance in
various NLP tasks and can be adapted to the document separation problem by
introducing appropriate modifications and training procedures.

Document separation as a sequence mapping problem in NLP has practical

applications in various domains, including information extraction, document analysis,
content indexing, and text understanding. Solving this problem can improve the
efficiency and accuracy of downstream NLP tasks that require individual document
processing.

To address this problem, research often involves collecting or creating annotated

datasets of concatenated documents and their corresponding segmented documents.
These datasets are used for training and evaluating the performance of different
models and approaches. The evaluation metrics typically include precision, recall, and
F1 score to assess the model's ability to correctly identify document boundaries.

Advancements in document separation techniques can significantly impact document

processing pipelines, information retrieval systems, and data analysis workflows,
making it easier to handle and extract information from large volumes of
concatenated documents efficiently and accurately.
Data preparation

Data preparation is a crucial step in Natural Language Processing (NLP) that involves
transforming raw text data into a format suitable for NLP models and algorithms. It
encompasses various preprocessing and cleaning steps to enhance the quality and
usability of the data. Here are some common data preparation techniques in NLP:

1. Text Cleaning:

- Removing special characters, punctuation marks, and numerical digits that are not
relevant to the analysis.

- Converting text to lowercase or uppercase to ensure consistency in text

representations.

- Handling contractions and expanding abbreviations (e.g., converting "can't" to

"cannot").

- Removing HTML tags, URLs, or other web-related artifacts.

- Eliminating non-ASCII characters or converting them to their ASCII counterparts.

2. Tokenization:

- Splitting text into individual words, phrases, or tokens.

- Handling sentence tokenization to separate text into sentences.

- Dealing with language-specific challenges, such as compound words or

languages without clear word boundaries.

3. Stop Word Removal:

- Removing common words that do not contribute much to the overall meaning of
the text, such as "a," "an," "the," etc.

- Customizing the list of stop words based on the specific domain or task.
4. Lemmatization and Stemming:

- Lemmatization: Reducing words to their base or canonical form (lemma) to handle

different inflections. For example, converting "running," "runs," and "ran" to the base
form "run."

- Stemming: Reducing words to their root form by removing prefixes or suffixes.

For example, converting "running" and "ran" to "run."

5. Removal of Irrelevant or Noisy Text:

- Filtering out irrelevant or noisy text, such as system-generated messages,

advertisements, or non-textual elements.

- Identifying and removing duplicate or near-duplicate documents.

6. Handling Spelling Errors and Abbreviations:

- Correcting common spelling errors or standardizing abbreviations for consistency.

- Expanding acronyms or abbreviations to their full forms for better understanding.

7. Handling Imbalanced Data:

- Addressing class imbalance issues, especially in classification tasks, by

oversampling the minority class or undersampling the majority class.

8. Data Splitting:

- Splitting the data into training, validation, and testing sets for model development,
evaluation, and fine-tuning.

- Ensuring a representative distribution of data across different sets to avoid bias.

9. Encoding and Vectorization:

- Converting text data into numerical representations that can be processed by

machine learning models.

- Techniques include one-hot encoding, count vectorization, TF-IDF (Term

Frequency-Inverse Document Frequency) vectorization, and word embeddings like
Word2Vec or GloVe.

10. Handling Missing Data:

- Dealing with missing values in the text data by either imputing them or removing
the corresponding instances or documents.

The specific data preparation steps in NLP may vary depending on the task, domain,
and specific requirements of the project. It is important to carefully analyze the data,
understand the characteristics and challenges, and apply appropriate preprocessing
techniques to ensure the data is ready for subsequent analysis, model training, or
other NLP tasks.

Evolving Explanatory Novel Patterns for Semantically Based Text Mining

Related Work

Typical approaches to text mining and knowledge discovery from texts are based
on simple bag-of-words (BOW) representations of texts which make it easy to
analyse them but restrict the kind of discovered knowledge [2]. Furthermore, the
discoveries rely on patterns in the form of numerical associations between concepts
(i.e., these terms will be later referred to as target concepts) from the documents,
which fails to provide explanations of, for example, why these terms show a strong
connection. Consequently, no deeper knowledge or evaluation of the discovered
knowledge is considered and so the techniques become merely “adaptations” of
traditional DM methods with an unproven effectiveness from a user viewpoint.
Traditional approaches to KDT share many characteristics with classical DM but
they also differ in many ways: many classical DM algorithms [19, 6], are irrelevant
or illsuited for textual applications as they rely on the structuring of data and the
availability of large amounts of structured information [7, 18, 27]. Many KDT
techniques inherit traditional DM methods and keyword-based representation
which are insufficient to cope with the rich information contained in natural-
language text. In addition, it is still unclear how to rate the novelty and/or
interestingness of the knowledge discovered from texts.

A semantically guided model for effective text mining

A semantically guided model for effective text mining refers to a model or approach
that leverages semantic information to enhance the performance and efficiency of
text mining tasks. This type of model goes beyond traditional methods that rely
solely on surface-level patterns and incorporates semantic understanding and
context into the text mining process. Here's an overview of the key components and
considerations in building a semantically guided model for effective text mining:

1. Semantic Representation:

- Utilize techniques such as word embeddings, semantic vectors, or pre-trained

language models (e.g., BERT, ELMO, GPT) to capture semantic information from the
input text.

- These representations encode contextual and semantic relationships between

words, enabling the model to understand the meaning and nuances of the text.

2. Semantic Similarity and Relation Extraction:

- Apply methods for measuring semantic similarity or relatedness between words,

sentences, or documents.

- Extract and leverage semantic relations between entities or concepts within the
text.

- These techniques help identify relevant information and enable the model to
make more accurate predictions or categorizations.
3. Named Entity Recognition and Entity Linking:

- Incorporate named entity recognition (NER) techniques to identify and classify

entities such as names, organizations, locations, or dates in the text.

- Perform entity linking to connect recognized entities with external knowledge

bases or ontologies, enriching the semantic understanding of the text.

4. Topic Modeling and Document Clustering:

- Employ topic modeling algorithms (e.g., Latent Dirichlet Allocation, Non-

negative Matrix Factorization) to extract latent topics from the text.

- Perform document clustering to group similar documents based on their

semantic content.

- These approaches enable the discovery of hidden themes, facilitate document

organization, and support information retrieval.

5. Sentiment Analysis and Opinion Mining:

- Integrate sentiment analysis techniques to determine the sentiment or polarity

expressed in the text.

- Extract opinions, attitudes, or subjective information from the text using opinion
mining methods.

- These capabilities help understand the sentiment of users or customers, identify

trends, and extract valuable insights.

6. Ontologies and Knowledge Graphs:

- Incorporate domain-specific ontologies or knowledge graphs to provide

structured semantic knowledge and relationships.

- Use ontologies to define and categorize concepts, entities, and relationships

relevant to the specific text mining task.

- Knowledge graphs facilitate semantic querying, inferencing, and reasoning,

enhancing the accuracy and depth of text mining results.
7. Evaluation and Benchmarking:

- Define appropriate evaluation metrics for assessing the performance of the

semantically guided model.

- Benchmark the model against existing text mining approaches to demonstrate

its effectiveness and superiority.

- Consider metrics such as precision, recall, F1 score, or domain-specific metrics

that align with the text mining task.

By incorporating semantic understanding into the text mining process, a

semantically guided model can achieve more accurate categorization, information
extraction, and knowledge discovery from textual data. It enables a deeper
understanding of the content, context, and relationships within the text, leading to
more effective text mining outcomes in various domains such as information
retrieval, recommendation systems, sentiment analysis, and content analysis.

Information Retrieval (IR) Model

Mathematically, models are used in many scientific areas having objective to
understand some phenomenon in the real world. A model of information retrieval
predicts and explains what a user will find in relevance to the given query. IR model
is basically a pattern that defines the above-mentioned aspects of retrieval procedure
and consists of the following −
• A model for documents.
• A model for queries.
• A matching function that compares queries to documents.
Mathematically, a retrieval model consists of −
D − Representation for documents.
R − Representation for queries.
F − The modeling framework for D, Q along with relationship between them.
R (q,di) − A similarity function which orders the documents with respect to the query.
It is also called ranking.

Types of Information Retrieval (IR) Model

An information model (IR) model can be classified into the following three models −
Classical IR Model
It is the simplest and easy to implement IR model. This model is based on
mathematical knowledge that was easily recognized and understood as well.
Boolean, Vector and Probabilistic are the three classical IR models.

Non-Classical IR Model
It is completely opposite to classical IR model. Such kind of IR models are based on
principles other than similarity, probability, Boolean operations. Information logic
model, situation theory model and interaction models are the examples of non-
classical IR model.

Alternative IR Model
It is the enhancement of classical IR model making use of some specific techniques
from some other fields. Cluster model, fuzzy model and latent semantic indexing (LSI)
models are the example of alternative IR model.

Design features of Information retrieval (IR)

systems
Let us now learn about the design features of IR systems −

Inverted Index
The primary data structure of most of the IR systems is in the form of inverted index.
We can define an inverted index as a data structure that list, for every word, all
documents that contain it and frequency of the occurrences in document. It makes it
easy to search for ‘hits’ of a query word.

Stop Word Elimination

Stop words are those high frequency words that are deemed unlikely to be useful for
searching. They have less semantic weights. All such kind of words are in a list called
stop list. For example, articles “a”, “an”, “the” and prepositions like “in”, “of”, “for”, “at”
etc. are the examples of stop words. The size of the inverted index can be significantly
reduced by stop list. As per Zipf’s law, a stop list covering a few dozen words reduces
the size of inverted index by almost half. On the other hand, sometimes the
elimination of stop word may cause elimination of the term that is useful for
searching. For example, if we eliminate the alphabet “A” from “Vitamin A” then it
would have no significance.
Stemming
Stemming, the simplified form of morphological analysis, is the heuristic process of
extracting the base form of words by chopping off the ends of words. For example,
the words laughing, laughs, laughed would be stemmed to the root word laugh.
In our subsequent sections, we will discuss about some important and useful IR
models.

In the context of information retrieval, classical and non-classical models can be

understood in terms of the underlying approaches or techniques used to retrieve
relevant information from a collection of documents. Let's explore how these models
are applied in information retrieval:

1. Classical Models in Information Retrieval:

Classical models in information retrieval refer to traditional approaches that are

based on well-established principles and assumptions. Some common classical
models include:

- Boolean Model: The Boolean model treats information retrieval as a binary

classification problem, where queries and documents are represented using Boolean
terms and operators (AND, OR, NOT). It retrieves documents that match the query
based on exact term matches.

- Vector Space Model: The vector space model represents queries and documents
as vectors in a multi-dimensional space. It measures the similarity between the query
vector and document vectors using techniques like cosine similarity and ranks
documents based on their proximity to the query.

- Probabilistic Model: The probabilistic model, such as the Binary Independence

Retrieval (BIR) model or the Okapi BM25 model, uses probabilistic principles to
estimate the relevance of documents given a query. It considers factors like term
frequency, document length, and term weights to rank documents.
2. Non-classical Models in Information Retrieval:

Non-classical models in information retrieval deviate from the traditional

assumptions and principles of classical models. These models often incorporate
advanced techniques or alternative paradigms to improve retrieval effectiveness.
Some examples of non-classical models include:

- Latent Semantic Indexing (LSI) and Latent Dirichlet Allocation (LDA): These
models leverage statistical techniques to capture latent semantic relationships
among terms and documents. They aim to overcome the limitations of term-based
matching and incorporate the underlying semantic structure of the collection.

- Neural Network Models: Non-classical models in information retrieval may utilize

neural network architectures such as Convolutional Neural Networks (CNNs) or
Recurrent Neural Networks (RNNs). These models can learn complex patterns and
capture semantic relationships in a more nuanced way than traditional models.

- Learning to Rank Models: These models employ machine learning algorithms to

train ranking models based on various features and relevance signals. They learn
from past user interactions and feedback to improve the ranking of search results.

The choice between classical and non-classical models in information retrieval

depends on factors such as the specific retrieval task, the available data, and the
desired level of sophistication or accuracy. Classical models are often simple and
interpretable, making them suitable for certain scenarios. Non-classical models, on
the other hand, offer the potential for capturing more complex relationships and
semantic understanding, but they may require more computational resources and
larger amounts of training data.

It's worth noting that the boundaries between classical and non-classical models in
information retrieval can be blurry, and there is often a continuum of approaches with
varying degrees of classical and non-classical characteristics. Researchers and
practitioners continuously explore new techniques and hybrid models to improve
retrieval performance and adapt to evolving information needs.
Lexical Resources

Lexical resources play a crucial role in natural language processing (NLP) tasks by
providing structured and organized information about words, their meanings,
relationships, and linguistic properties. Here are some commonly used lexical
resources in NLP:

1. WordNet:

- WordNet is a lexical database that provides a comprehensive inventory of words

in the English language.

- It organizes words into sets of synonyms called synsets, where each synset
represents a distinct concept or meaning.

- WordNet also captures relationships between words such as hypernymy (is-a

relationship), hyponymy (specific instances), meronymy (part-whole relationships),
and antonymy.

- It is widely used for tasks like word sense disambiguation, semantic similarity
measurement, and synonym expansion.

2. FrameNet:

- FrameNet is a lexical database that focuses on the semantic frames of words and
the relationships between frames.

- It represents words in terms of the frames they evoke, which are abstract
structures representing a situation, event, or concept.

- FrameNet captures the lexical units (words or phrases) associated with each
frame and describes the roles and semantic annotations associated with the units.

- It is useful for tasks like semantic role labeling, information extraction, and
semantic analysis of texts.

3. Stemmers:
 - Stemmers are algorithms or tools used to reduce words to their base or root form,
called the stem or lemma.

- Stemming helps in normalizing words and reducing inflectional variations.

- Common stemmers include the Porter stemmer, Snowball stemmer, and

Lancaster stemmer, each with its own rules and heuristics.

- Stemmers are used in information retrieval, text mining, and indexing to enhance
the retrieval of relevant information.

4. POS Tagger:

- A part-of-speech (POS) tagger assigns grammatical tags to words in a sentence,

indicating their syntactic category (e.g., noun, verb, adjective, etc.).

- POS tagging helps in syntactic analysis, grammar-based parsing, and semantic

analysis of text.

- POS taggers are trained using annotated corpora and statistical models or rule-
based approaches.

- Popular POS tagging tools include NLTK (Natural Language Toolkit), Stanford
POS Tagger, and spaCy.

5. Research Corpora:

- Research corpora are large collections of text or speech data that are annotated
or curated for specific research purposes.

- They provide valuable resources for training and evaluating NLP models and
algorithms.

- Research corpora may include text from various domains and genres, such as
news articles, books, web pages, and social media posts.

- Some widely used research corpora include the Penn Treebank, CoNLL corpora,
Wikipedia dumps, and social media corpora.
These lexical resources and tools form the foundation for various NLP tasks, including
information retrieval, text classification, named entity recognition, sentiment analysis,
and machine translation. By leveraging these resources, NLP systems can better
understand and process natural language, enabling more accurate and meaningful
analysis of textual data.

iSTART

iSTART (Intelligent Strategy Training for Active Reading and Thinking) is an

advanced educational software system developed to improve reading
comprehension and critical thinking skills. It is designed to assist learners in
understanding and analyzing text effectively. iSTART combines techniques from
cognitive science, artificial intelligence, and natural language processing to provide
personalized instruction and support.

The primary goal of iSTART is to enhance reading comprehension by helping learners

develop effective reading strategies and metacognitive skills. It achieves this through
various interactive activities and feedback mechanisms. Here are some key features
and components of iSTART:

1. Pre-reading Activities:

- iSTART includes pre-reading activities that activate learners' prior knowledge and
build a conceptual framework for understanding the text.

- These activities aim to engage learners and provide them with relevant
background information related to the topic of the text.

2. Reading and Strategic Processing:

- During the reading phase, learners interact with the text while employing
strategic processing techniques.

- iSTART guides learners to use active reading strategies, such as highlighting

important information, generating questions, making inferences, and summarizing
key points.
 - The system provides prompts and suggestions to assist learners in applying these
strategies effectively.

3. Metacognitive Reflection:

- iSTART encourages learners to reflect on their reading process and develop

metacognitive awareness.

- It prompts learners to monitor their comprehension, identify difficulties or

confusion, and take appropriate actions to resolve comprehension problems.

- Learners are encouraged to use self-explanation techniques to articulate their

understanding of the text and identify areas where further clarification or elaboration
is needed.

4. Intelligent Tutoring and Feedback:

- iSTART uses intelligent tutoring techniques to provide personalized instruction

and feedback.

- The system analyzes learners' interactions, responses, and performance to adapt

its instruction and provide tailored feedback.

- Learners receive immediate feedback on their strategic choices, comprehension

monitoring, and comprehension outcomes to facilitate their learning process.

5. Assessment and Progress Tracking:

- iSTART incorporates assessment measures to evaluate learners' reading

comprehension and strategic processing skills.

- The system tracks learners' progress over time, providing both learners and
instructors with insights into their development and areas for improvement.

iSTART has been used in various educational contexts, including classrooms,

tutoring, and self-paced learning environments. It aims to support learners in
developing effective reading habits, metacognitive skills, and critical thinking
abilities. By providing personalized instruction and guidance, iSTART helps learners
become more engaged, active readers capable of comprehending complex texts and
thinking critically about the information presented.