Chapter 1 : NLP – Introduction

Adama Science and Technology University

School of Electrical Engineering and Computing
Department of CSE
Dr. Mesfin Abebe Haile (2022)
 Outline

 Introduction to NLP
 What is NLP?
 Aspects of Language Processing
 Goal of NLP
 History of NLP
 Application of NLP
 Open Problems
 Knowledge Sources
 Computational Morphology

01/02/23 2
 What is Natural Language
Processing ?

 Natural Language Processing is one of the subfields of Artificial

Intelligence:
 Natural Language Processing (NLP)
 Knowledge Representation
 Automated Reasoning
Disciplines of AI
 Machine Learning
 Computer Vision
 Robotics

01/02/23 3
 What is Natural Language
Processing ?

 Develop computers that can understand human (“natural”)

language and speak human language.

 Computers would be much easier to use with Natural

Language Interface.

 Can we “teach” them to understand human language?

 Many approaches to NLP are based on Machine Learning.

01/02/23 4
 What is Natural Language
Processing ?
 NLP is a field of computer science, artificial intelligence and
computational linguistics concerned with the interactions
between computers and human (natural) languages, and, in
particular, concerned with programming computers to fruitfully
process large natural language corpora*.

 Sub-field of Artificial Intelligence, but very interdisciplinary.

 Computer science, human-computer interaction (HCI), linguistics,
cognitive psychology, speech signal processing, …

 *A corpus is a collection of written or spoken texts.

 With the use of computers, it is possible to compile large
amounts of authentic written and spoken language.
01/02/23 5
 Aspects of Language Processing

 “Natural” languages:
 Geez, Amharic, Oromifa, Tigrigna,, English, Mandarin, French,
Swahili, Arabic, …

 NOT Java, C++, Perl, … (Programming Languages)

01/02/23 6
 Aspects of Language Processing

 Traditionally, work in NLP has tended to view the process of

language analysis as being decomposable into a number of stages,
mirroring the theoretical linguistic distinctions drawn between
SYNTAX, SEMANTICS, and PRAGMATICS.
 The simple view is that the sentences of a text are first analyzed in terms
of their syntax.
 This provides an order and structure that is more amenable to an
analysis in terms of semantics, or literal meaning; and

 This is followed by a stage of pragmatic analysis whereby the meaning

of the utterance or text in context is determined.
 This last stage is often seen as being concerned with DISCOURSE,
whereas the previous two are generally concerned with sentential
matters.
01/02/23 7
 Aspects of Language Processing

Figure 1. The Stages of Analysis in Processing Natural Language

 The tripartite distinction into
syntax, semantics, and pragmatics
only serves at best as a starting
point when to consider the
processing of real NL texts.

 A finer-grained decomposition of
the process is useful when taken
into account the current state of
the art in combination with the
need to deal with real language
data as reflected in Figure.

01/02/23 8
 Aspects of Language Processing

 Word, lexicon: lexical analysis:

 Morphology, word segmentation;
 In lexical analysis, tokenization is the process of breaking a stream of text
up into words, phrases, symbols, or other meaningful elements called
tokens. The list of tokens becomes input for further processing such as
parsing or text mining.
 Tokenization is useful both in linguistics (where it is a form of text
segmentation), and in computer science, where it forms part of lexical
analysis.
 Tokenization is the task of chopping it up into pieces, called tokens ,
perhaps at the same time throwing away certain characters, such as
punctuation.
 Example:
Input: Friends, Romans, Countrymen, lend me your ears;
Output:
01/02/23 9
 Aspects of Language Processing

 Syntax:
 Sentence structure, phrase, grammar, …

 Semantics:
 Meaning,
 Execute commands

 Discourse analysis:
 Meaning of a text,
 Relationship between sentences (e.g. anaphora)

01/02/23 10
 Aspects of Language Processing

 Morphology: What is a word?

 奧林匹克運動會（希臘語： Ολυμπιακοί Αγώνες ， 簡稱奧運會或 奧運）是國際奧林
匹克委員會主辦的包含多種體育運動項目的國際 性運動會，每四年舉行一次。•
‫“ = آ‬to her houses”

 Lexicography: What does each word mean?

 He plays bass guitar.
 That bass was delicious!

 Syntax: How do the words relate to each other?

 The dog bit the man. ≠ The man bit the dog.
 But in Russian: человек собаку съел = человек съел собаку
 man ate a dog = man ate a dog

01/02/23 11
 Aspects of Language Processing

 Semantics: How can we infer meaning from sentences?

 I saw the man on the hill with the telescope.
 The ipod is so small! ☺
 The monitor is so small!

 Discourse: How about across many sentences?

 President Obama met with President-Elect Trump today at the
White House. He welcomed him, and showed him around.
 Who is “he”? Who is “him”? How would a computer figure that
out?

01/02/23 12
 Aspects of Language Processing

 Syntax
 Lemmatization:
Lemmatization usually refers to doing things properly with the use of a
vocabulary and morphological analysis of words, normally aiming to
remove inflectional endings only and to return the base or dictionary form
of a word, which is known as the lemma.
 Morphological segmentation:
 Separate words into individual morphemes and identify the class of the
morphemes.
 The difficulty of this task depends greatly on the complexity of the
morphology (i.e. the structure of words) of the language being considered.

01/02/23 13
 Aspects of Language Processing

 Syntax …
 Part-of-speech tagging:
 Example, "book" can be a noun ("the book on the table") or verb ("to
book a flight")

 Sentence breaking (also known as sentence boundary disambiguation):

 Given a chunk of text, find the sentence boundaries.
 Sentence boundaries are often marked by periods or other punctuation
marks, but these same characters can serve other purposes (e.g.
marking abbreviations).

01/02/23 14
 Aspects of Language Processing

 Syntax …
 Stemming
 Stemming usually refers to a crude heuristic process that chops off the ends of
words in the hope of achieving this goal correctly most of the time, and often
includes the removal of derivational affixes.
 Word segmentation
 Separate a chunk of continuous text into separate words.
 For a language like English, this is fairly trivial, since words are usually separated
by spaces.
 However, some written languages like Chinese, Japanese and Thai do not mark
word boundaries in such a fashion, and in those languages text segmentation is a
significant task requiring knowledge of the vocabulary and morphology of words
in the language.

01/02/23 15
 Aspects of Language Processing

 Semantics (Individual Assignment One : Defined the terms)

 Lexical semantics
 Machine translation
 Named entity recognition
 Natural language generation
 Natural language understanding
 Optical character recognition
 Question answering
 Recognizing Textual entailment
 Relationship extraction
 Sentiment analysis
 Topic segmentation
 Word sense disambiguation

01/02/23 16
 Aspects of Language Processing

 Discourse :
 Automatic summarization
 Coreference resolution
 Given a sentence or larger chunk of text, determine which words ("mentions") refer to the same
objects ("entities"). Anaphora resolution is a specific example of this task, and is specifically
concerned with matching up pronouns with the nouns or names to which they refer.
 The more general task of coreference resolution also includes identifying so-called "bridging
relationships" involving referring expressions.
 For example, in a sentence such as "He entered John's house through the front door", "the front
door" is a referring expression and the bridging relationship to be identified is the fact that the
door being referred to is the front door of John's house (rather than of some other structure that
might also be referred to).
 Discourse analysis:

01/02/23 17
 Aspects of Language Processing

 Speech Processing
 Speech recognition
 Speech segmentation
 Given a sound clip of a person or people speaking, separate
it into words.
 A subtask of speech recognition and typically grouped with
it.

 Text-to-speech (Speech Synthesis)

01/02/23 18
Goal of Natural Language Processing

 Ultimate goal: Natural human-to-computer communication.

 The goal of natural language processing (NLP) is to design and build
computer systems that are able to analyze natural languages like Geez,
Amharic, German or English, and that generate their outputs in a natural
language, too.
 In natural language understanding, the objective is to extract the meaning
of an input sentence or an input text. Usually, the meaning is represented in
a suitable formal representation language so that it can be processed by a
computer.
 The goal in text classification is to assign a text document to one out of
several text classes.
 Example: for newspaper articles, such classes are sports reports, finances, and
politics.

01/02/23 19
 History of Natural Language
Processing
 1950s
 Early MT: word translation + re-ordering.
 Chomsky’s Generative grammar.
 Bar-Hill’s argument.
 1960-80s
 Applications:
 BASEBALL: use NL interface to search in a database on baseball
games.
 LUNAR: NL interface to search in Lunar.
 ELIZA: simulation of conversation with a psychoanalyst.
 SHREDLU: use NL to manipulate block world.
 Message understanding: understand a newspaper article on terrorism.
 Machine translation.
01/02/23 20
 History of Natural Language
Processing
 1960-80s
 Methods
 ATN (augmented transition networks): extended context-free
grammar
 Case grammar (agent, object, etc.)
 DCG – Definite Clause Grammar
 Dependency grammar: an element depends on another
 1990s-now
 Statistical methods
 Speech recognition
 MT systems
 Question-answering
 etc…
01/02/23 21
 History of Natural Language
Processing

 Traditional NLP approaches: symbolic, grammar, …

 More recent approaches: statistical
 For some applications: statistical approaches are better
(tagging, speech recognition, …)

 For some others, traditional approaches are better (MT)

 Trend: combine statistics with rules (grammar)

 E.g. Probabilistic Context Free Grammar (PCFG)
 Consider some grammatical connections in statistical approaches.

01/02/23 22
 History of Natural Language
Processing
 Classical symbolic methods:
 Morphological analyzer

 Parser (syntactic analysis)

 Semantic analysis (transform into a logical form, semantic

network, etc.)

 Discourse analysis

 Pragmatic analysis

01/02/23 23
 History of Natural Language
Processing
 Empirical and Statistical Approaches
 Corpus Creation

 Treebank Annotation

 Fundamental Statistical Techniques

 Part-of-Speech Tagging

 Statistical Parsing

 Etc…
01/02/23 24
 NLP Applications
 Intelligent computer systems
 NLU interfaces to databases
 Computer aided instruction
 Intelligent Web searching
 Data mining
 Machine translation
 Speech recognition
 Natural language generation
 Question answering
 Text classification
 etc
01/02/23
 Information Retrieval,
 Information retrieval (IR) deals
with the representation, storage,
organization of, and access to
information items. Given a query
the goal is to extract a subset of
documents from a large data
collection that satisfies a user's
information need. Besides written
texts the database may also contain
multimedia documents, e.g. audio
and video data.
 Query-answering
 Text categorization
 Summarization
 Data extraction etc… 25
 NLP Applications

 Speech Synthesis
 Text to Speech:

01/02/23 26
 Open Problems in NLP

 Challenges in natural language processing frequently involve:

 Natural language understanding,

 Natural language generation (frequently from formal, machine-

readable logical forms),

 Connecting language and machine perception,

 Managing human-computer dialog systems, or

 Some combination thereof…

01/02/23 27
 Open Problems in NLP

 Ambiguity:

 Lexical/morphological: change (V,N), training (V,N), even

(ADJ, ADV) …
 Syntactic: Helicopter powered by human flies.

 Semantic: He saw a man on the hill with a telescope.

 Discourse: anaphora, …

01/02/23 28
 Open Problems in NLP

 Classical solution:
 Using a later analysis to solve ambiguity of an earlier step.
 Eg. He gives him the change:
 (change as verb does not work for parsing)
 He changes the place:
 (change as noun does not work for parsing)

 However: He saw a man on the hill with a telescope.

 Correct multiple parsings,
 Correct semantic interpretations -> semantic ambiguity,
 Use contextual information to disambiguate (does a sentence in the
text mention that “He” holds a telescope?)

01/02/23 29
 Knowledge Sources

 When using NLP for a new domain, one also needs to answer
what text source should be used for extracting content.
 Of course, not any arbitrary text source is applicable.
 In order to qualify as a source, the text type needs to meet the
following two criteria:
 Firstly, the text type needs to contain sufficient domain
knowledge.
 In other words, if we choose a text type that only infrequently
contains content regarding a given domain, then we are not very
likely to extract any significant amount of knowledge.
 In the past, most research in NLP has been carried out on news
corpora. The topic that is predominant on this text type are issues
out of the domain. Consequently, this text type would be of little
value for knowledge extraction.
01/02/23 30
 Knowledge Sources

 In order to qualify as a source, the text type needs to meet the

following two criteria:…
 Secondly, the text type should not only contain knowledge about
the domain that is already widely available in structured format
(such as databases)

 Otherwise, there would hardly be any point in extracting

knowledge from those texts as it would already be available.

01/02/23 31
 Computational Morphology

 What is it?
 Morphology: the study/knowledge of structure/form.
• In this case: of words,
• How words are created, structured, analyzed
• Morpheme: basic meaningful unit of language.

 Computational morphology: developing/using computer

applications that involve morphology.

01/02/23 32
 Computational Morphology

 Computational applications:
 Analysis: parse/break a word into its constituent morphemes.

 Generation: create/generate a word from its constituent

morpheme.

01/02/23 33
 Computational Morphology

 Morphological processes:
 Affixation: prefix, suffix, infix
 Interleaving (KaTaB, uKTaB)
 Cliticization (isn’t, s’appelle)
 Internal change: (sing/sang, goose/geese)
 Suppletion (irregularity): (aller/ir, be/am)
 Stress placement: implant, import, contest
 Tone placement: dà vs. dá ( will spank vs. spanked)
 Reduplication
 Full: iji/ijiiji
 Partial: lakad/lalakad

01/02/23 34
 Computational Morphology

 Areas of focus in morphology:

 Derivational
 do+able, adjourn+ment, depos+ition, un+lock, teach+er

 Inflectional
 dog+s, sneez+ed

 Compounding
 overkill, BYU intramural track star

 Cliticization
 I’m, she’ll, they’ve, o’clock

01/02/23 35
 Computational Morphology

 Computational morphology
 Processing morphological structure via computer (parsing,
generation)
 Traditional approach:
 ad-hoc methods,
 Cut-and-paste algorithms,
 Dictionary lookup,
 Inadequate for highly inflected languages.

 Even statistical approaches are often un-useful,

 Two-level approach w/finite-state techniques,
 Machine learning is making inroads,
 Sequence labeling, morpheme boundary detection.

01/02/23 36
 Question & Answer

01/02/23 37
 Thank You !!!

01/02/23 38