UNIT 3 Language Modelling

What is Language Modelling?

Language modeling is a fundamental task in natural language processing (NLP) that involves
predicting the next word in a sequence of words. It's essentially a way for a machine learning
model to learn the structure and probabilities of sequences of words in a language.
A language model learns to predict the probability of a word given its context, which can be a
single word or a sequence of words. The model assigns higher probabilities to words that are
more likely to occur next in the sequence according to the language's syntax, semantics, and
context.
For example, given the sentence "The cat is on the", a language model would predict that the
next word is more likely to be "mat" than "moon", because "mat" is a more common word to
follow "on the" than "moon".
Language models can be used for various NLP tasks such as:
1. Machine Translation: Predicting the next word or phrase in the target language.
2. Speech Recognition: Transcribing spoken language into text.
3. Text Generation: Generating coherent and contextually relevant text.
4. Spell Checking and Auto-correction: Suggesting corrections based on context.
5. Sentiment Analysis: Understanding the sentiment of a given text.
6. Question Answering: Generating answers based on questions.

Probabilistic language modelling -

Probabilistic language modeling is a method used to estimate the probability of a sequence of
words occurring in a language. The goal is to model the probability distribution of word
sequences, so the model can predict the likelihood of observing a particular sequence of
words.
In probabilistic language modeling:
1. Word Sequences: The language model considers sequences of words. It tries to
predict the probability of a word given the previous words in the sequence.
2. Probability Estimation: The model estimates the probability of each word in the
sequence given its context (the previous words). Mathematically, this is represented as
𝑃(𝑤𝑖∣𝑤1,𝑤2,...,𝑤𝑖−1) where 𝑤𝑖 is the current word and 𝑤1,𝑤2,...,𝑤𝑖−1 are the
preceding words.
3. Markov Assumption: In many cases, the probability of a word depends only on a
finite history of preceding words. This assumption simplifies the model. For example,
a bigram model assumes that the probability of a word only depends on the previous
word (i.e., 𝑃(𝑤𝑖∣𝑤𝑖−1). Similarly, a trigram model considers the probability of a word
given the two previous words.
Examples of probabilistic language models include n-gram models (like bigram and trigram
models) and more sophisticated models like recurrent neural networks (RNNs) and
transformers.

Markov Models –
A Markov model is a stochastic model used to model randomly changing systems where it is
assumed that the future state depends only on the current state and not on the sequence of
events that preceded it.
Markov Property:
The Markov property states that the future state of a system depends only on its current state
and is independent of its past states.
Types of Markov Models in Language Modeling:
1. Bigram Model (First-order Markov Model):
 It assumes that the probability of a word depends only on the preceding word.
 Example: 𝑃(𝑤𝑖∣𝑤𝑖−1)
2. Trigram Model (Second-order Markov Model):
 It considers the two preceding words to estimate the probability of a word.
 Example: 𝑃(𝑤𝑖∣𝑤𝑖−1,𝑤𝑖−2)
3. Higher-order Markov Models:
 These models consider more than two preceding words, up to 𝑁N words.
 Example: 𝑃(𝑤𝑖∣𝑤𝑖−1,𝑤𝑖−2,...,𝑤𝑖−𝑁+1)
Advantages of Markov Models in Language Modeling:
 Simplicity: Markov models are relatively simple and easy to understand.
 Efficiency: They require less memory and computational resources compared to more
complex models.
 Interpretability: It's straightforward to interpret the predictions and understand why a
certain prediction was made.
 Flexibility: Markov models can be extended to higher orders to capture more
complex dependencies if needed.
Limitations of Markov Models:
 Limited Context: Markov models assume that the future state depends only on a
fixed-size window of previous states. This can lead to limited modeling of long-range
dependencies.
  Data Sparsity: As the order of the model increases, the amount of data required for
accurate estimation grows exponentially. This can lead to sparse data problems,
especially for high-order models.
 Fixed Window Size: Markov models have a fixed window size for context, which
may not be sufficient for capturing complex linguistic phenomena.

Generative models of language

Generative models of language are statistical models that learn the joint probability
distribution of word sequences in a language. These models are capable of generating new
sequences of words that resemble natural language. In other words, they can generate text by
sampling from the learned probability distribution.
Here's a detailed explanation of generative models of language:
1. Basic Concept:
Generative models aim to learn the underlying probability distribution of word sequences in a
language. This means that given a sequence of words 𝑤1,𝑤2,...,𝑤𝑛 a generative model learns
𝑃(𝑤1,𝑤2,...,𝑤𝑛), the probability of that specific sequence occurring.
By learning this distribution, generative models can generate new sequences of words that are
likely to be seen in the training data.
2. Approaches:
There are several approaches to building generative models of language:
 Hidden Markov Models (HMMs): HMMs model sequences of observed
events (words) through a series of hidden states. Each state emits a probability
distribution over possible words, and transitions between states are governed
by transition probabilities.
 N-gram Models: N-gram models are a simpler form of generative language
models that predict the probability of a word based on the previous 𝑛n words.
For example, a bigram model predicts the probability of a word given only its
previous word.
 Neural Language Models: Neural networks, particularly recurrent neural
networks (RNNs), long short-term memory (LSTM) networks, and more
recently, Transformer-based models, are used to learn the probability
distribution over sequences of words. These models capture complex
dependencies between words and can generate text that is often more fluent
and contextually appropriate.

Log-Linear Models –
Log-linear models, also known as log-linear classifiers or maximum entropy models, are a
type of statistical model used for classification tasks, including language modeling. These
models are particularly useful when dealing with large and sparse feature spaces. Log-linear
models are widely used in natural language processing for tasks such as part-of-speech
tagging, named entity recognition, and machine translation.
Basics of Log-linear Models:
1. Basic Idea:
 Log-linear models aim to estimate the conditional probability 𝑃(𝑦∣𝑥)of a label
𝑦 given input features 𝑥.
 They model this probability using a linear combination of features, where the
combination is transformed using the exponential function to ensure non-
negativity and normalization.

2. Fe
atures:
 Features 𝑓𝑖(𝑥) are functions that capture relevant properties of the input 𝑥 for
predicting the label 𝑦.
 These features can be binary indicators, counts, or any other representation
that characterizes the input.
3. Parameters:
 𝜆𝑖 are the parameters of the model, also known as weights or coefficients.
 These parameters are learned from training data using optimization algorithms
such as gradient descent.
4. Normalization Factor:

Advantages of Log-linear Models:

1. Flexibility:
 Log-linear models can incorporate various types of features and capture
complex relationships between features and labels.
2. Interpretability:
  The model parameters 𝜆𝑖λi provide insight into the importance of each feature
for prediction.
3. Generality:
 Log-linear models can handle both classification and regression tasks.
4. Robustness:
 They are less prone to overfitting, especially when using regularization
techniques.
Applications of Log-linear Models in NLP:
1. Part-of-Speech Tagging:
 Assigning the correct part-of-speech tag to each word in a sentence.
2. Named Entity Recognition:
 Identifying and classifying entities such as names of persons, organizations,
and locations in text.
3. Text Classification:
 Classifying documents into predefined categories or topics.
4. Machine Translation:
 Predicting the target language sentence given the source language sentence.

Graph-based models -
Graph-based models in natural language processing (NLP) represent language as a graph,
where nodes represent words or entities, and edges represent relationships between them.
Graph-based models are particularly effective for tasks that require capturing long-range
dependencies and understanding the relationships between words or entities.
Here's a detailed explanation of graph-based models in NLP:
1. Basic Concept:
In graph-based models, language is represented as a graph, where:
 Nodes: Nodes represent words, entities, or concepts in the language.
 Edges: Edges represent relationships or dependencies between nodes.
This graph structure allows capturing various linguistic phenomena such as semantic
relationships, syntactic dependencies, and even contextual information.
2. Types of Graph-based Models:
There are several types of graph-based models used in NLP, including:
  Semantic Graphs: These graphs represent the semantic relationships between
words or concepts. Examples include WordNet, ConceptNet, and knowledge
graphs like DBpedia or Wikidata.
 Syntactic Dependency Graphs: These graphs capture syntactic relationships
between words in a sentence. Each word is a node, and edges represent
dependencies such as subject-verb or verb-object relationships.
 Contextualized Graphs: These graphs capture contextual relationships
between words in a text. Contextual information can include word embeddings
or representations learned from large text corpora.
 Knowledge Graphs: These are large graphs representing factual knowledge
about the world. They are often used to enhance language understanding by
providing external knowledge to NLP models.
3. Applications:
Graph-based models find applications in various NLP tasks, including:
 Semantic Similarity: Determining the semantic similarity between words or
sentences by analyzing their positions in a semantic graph.
 Named Entity Recognition: Identifying named entities in text by leveraging
knowledge graphs or semantic graphs.
 Relation Extraction: Extracting relationships between entities in text by
analyzing the structure of syntactic or semantic dependency graphs.
 Question Answering: Answering questions by traversing knowledge graphs
or syntactic/semantic graphs to find relevant information.
 Summarization and Text Generation: Generating summaries or coherent
text by leveraging contextualized graph representations.

What are N-Gram Models ?

A word n-gram language model is a purely statistical model of language. It is based on an
assumption that the probability of the next word in a sequence depends only on a fixed size
window of previous words. If only one previous word was considered, it was called a bigram
model; if two words, a trigram model; if n − 1 words, an n-gram model.[2] Special tokens
were introduced to denote the start(<s>) and end(</s>) of a sentence and .Here's a detailed
explanation of N-gram models:
1. Basic Concept:
In N-gram models, the probability of a word wi occurring given its preceding context
𝑤𝑖−1,𝑤𝑖−2,...,𝑤𝑖−𝑛+1 is estimated. The notation 𝑛 represents the size of the context window,
which is typically referred to as the "order" of the N-gram.
For example, a bigram (2-gram) model considers the probability of a word given only its
previous word:
𝑃(𝑤𝑖∣𝑤𝑖−1)P(wi∣wi−1)
A trigram (3-gram) model considers the probability of a word given the two previous words:
𝑃(𝑤𝑖∣𝑤𝑖−2,𝑤𝑖−1)P(wi∣wi−2,wi−1)
In general, an 𝑛n-gram model considers the probability of a word given the previous 𝑛−1n−1
words.
2. Estimation:
N-gram models estimate the probabilities based on the frequency of occurrences in a training
corpus. For example, in a bigram model, the probability 𝑃(𝑤𝑖∣𝑤𝑖−1) can be estimated as:
𝑃(𝑤𝑖∣𝑤𝑖−1)=Count(𝑤𝑖−1,𝑤𝑖)/Count(𝑤𝑖−1)
Here, Count(𝑤𝑖−1,𝑤𝑖) represents the number of times the word sequence 𝑤𝑖−1,𝑤𝑖 occurs,
and Count(𝑤𝑖−1) represents the number of times the word 𝑤𝑖−1 occurs in the corpus.
3. Applications:
N-gram models are widely used in various NLP tasks, including:
 Language Modeling: Predicting the next word in a sequence given the
previous words.
 Spell Checking: Predicting the correct word given the context, useful in
autocorrect systems.
 Speech Recognition: Estimating the likelihood of word sequences in spoken
language.
 Machine Translation: Modeling the probability of word sequences in source
and target languages.
 Text Generation: Generating coherent and contextually relevant text.
4. Advantages:
 N-gram models are computationally efficient and easy to implement.
 They capture local dependencies between adjacent words effectively.
 They perform well on tasks where short-range context is important.
5. Challenges:
 N-gram models suffer from the curse of dimensionality when dealing with
large vocabularies and high-order N-grams, leading to sparse data and
overfitting.
 They are limited in capturing long-range dependencies and understanding
context beyond the immediate past.
6. Smoothing:
To handle zero probabilities for unseen N-grams in the training data, smoothing techniques
such as Laplace smoothing, Good-Turing smoothing, or Katz smoothing are often used to
adjust the probabilities.
N-Gram models - https://aiml.com/what-is-an-n-gram-model/
https://medium.com/@ompramod9921/exploring-n-grams-the-building-blocks-of-natural-
language-understanding-58e259cd4e1c

Word Embeddings –
Word embeddings are dense vector representations of words in a continuous vector space
where the similarity between words is captured by the proximity of their vectors. These
embeddings are learned from large text corpora using neural network-based techniques, such
as Word2Vec, GloVe, or FastText. Word embeddings have become a fundamental component
of natural language processing (NLP) models, allowing them to better understand and
represent the semantic relationships between words.
Here's a detailed explanation of word embeddings:
Basic Concept:
Word embeddings represent words as high-dimensional vectors in a continuous vector space,
typically with several hundred dimensions. Each word is mapped to a unique vector, and
similar words are expected to have similar vector representations. These vectors are learned
in such a way that they capture semantic and syntactic relationships between words.
Challenges:
Out-of-Vocabulary Words: Word embeddings may not represent rare or out-of-vocabulary
words well, which can lead to information loss.
Domain Specificity: Pre-trained word embeddings may not capture domain-specific
semantics effectively. Fine-tuning or training embeddings on domain-specific data may be
necessary.

Bag-of-Words -
The bag-of-words (BoW) model is a simple and commonly used technique in natural
language processing (NLP) for representing text data. In this model, a document is
represented as a "bag" (multiset) of words, disregarding grammar and word order but keeping
track of word frequency. Each unique word in the document is treated as a feature, and its
frequency in the document becomes its value.
Steps in Creating a Bag-of-Words Representation:
Tokenization:
The text is split into individual words or tokens. Punctuation and other non-alphanumeric
characters are often removed or treated as separate tokens.
Building the Vocabulary:
From the tokenized text, a vocabulary is created containing all unique words across the entire
corpus.
Vectorization:
Each document is represented as a vector, with each element representing the count of a word
from the vocabulary in the document. Alternatively, binary values can be used to indicate the
presence or absence of each word.
Example: Consider the following two sentences:
Sentence 1: "The cat sat on the mat."
Sentence 2: "The dog ate the bone."
Vocabulary:
The vocabulary for these sentences would be: {"The", "cat", "sat", "on", "the", "mat", "dog",
"ate", "bone"}.
Bag-of-Words Representation:
Sentence 1: [1, 1, 1, 1, 1, 1, 0, 0, 0]
Sentence 2: [1, 0, 0, 0, 2, 0, 1, 1, 1]
Applications:
Document Classification: BoW representations are commonly used in text classification
tasks such as sentiment analysis, spam detection, and topic classification.
Information Retrieval: BoW representations are used in information retrieval systems to
match user queries with relevant documents.
Topic Modeling: BoW representations serve as input for topic modeling algorithms like
Latent Dirichlet Allocation (LDA).
Text Mining: BoW representations are used for mining text data to discover patterns or
extract useful information.

TFIDF –
TF-IDF (Term Frequency-Inverse Document Frequency) is a statistical measure used to
evaluate the importance of a word in a document relative to a corpus of documents. While it's
not technically a word embedding technique, it's often used in combination with word
embeddings to enhance the representation of text data. TF-IDF provides a numerical
representation of the importance of each word in a document within a corpus.
Here's a detailed explanation of TF-IDF:
1. Term Frequency (TF):
Definition: Term frequency measures the frequency of a word within a document. It indicates
how often a word appears in a document relative to the total number of words in that
document.
TF(𝑤,𝑑)=Count(𝑤,𝑑)Total words in 𝑑TF(w,d)=Total words in dCount(w,d)
Where:
Count(𝑤,𝑑) is the number of times word 𝑤w appears in document 𝑑d.
Total words in d is the total number of words in document 𝑑.
2. Inverse Document Frequency (IDF):
Definition: Inverse document frequency measures the rarity of a word across the corpus. It
penalizes common words and gives higher weights to rare words that are more informative.
IDF(𝑤,𝐷)=log⁡(Total documents in 𝐷/Number of documents containing 𝑤)
Where:
Total documents in D is the total number of documents in the corpus 𝐷D.
Number of documents containing w is the number of documents in which word 𝑤wappears at
least once.
3. TF-IDF Score:
Definition: The TF-IDF score combines the term frequency and inverse document frequency
to calculate the importance of a word in a document relative to the corpus.
TF-IDF(𝑤,𝑑,𝐷)=TF(𝑤,𝑑)×IDF(𝑤,𝐷)

Word2Vec –
Word2Vec is a widely used method in natural language processing (NLP) that allows words
to be represented as vectors in a continuous vector space. Word2Vec is an effort to map words
to high-dimensional vectors to capture the semantic relationships between words, developed
by researchers at Google. Words with similar meanings should have similar vector
representations, according to the main principle of Word2Vec. Word2Vec utilizes two
architectures:
 CBOW (Continuous Bag of Words): The CBOW model predicts the current word
given context words within a specific window. The input layer contains the context
words and the output layer contains the current word. The hidden layer contains the
dimensions we want to represent the current word present at the output layer.
Skip Gram : Skip gram predicts the surrounding context words within specific window
given current word. The input layer contains the current word and the output layer contains
the context words. The hidden layer contains the number of dimensions in which we want to
represent current word present at the input layer.

What is Topic Modelling ?

Topic modeling is a statistical modeling technique used in Natural Language Processing
(NLP) to identify abstract topics or themes within a collection of documents. It's particularly
useful for large collections of text data where understanding the underlying themes can be
challenging.
Objective:
The main goal of topic modeling is to discover the latent topics present in a collection of
documents without any prior labeling or supervision.
It helps in understanding the main themes, trends, and patterns in the text data.
Latent Variables:
In topic modeling, topics are latent variables, meaning they are not directly observable but
inferred from the observed data (documents).
Each document is assumed to be generated from a mixture of topics, and each topic is
characterized by a distribution over words.
Latent Dirichlet Allocation (LDA) –
LDA (Latent Dirichlet Allocation) model is an unsupervised machine learning algorithm
used for topic modelling . It is a statistical method used for uncovering the underlying themes
or topics in a collection of documents. It assumes that each document is a mixture of various
topics, and each topic is a distribution of words. LDA aims to learn these topics from the data
by iteratively assigning words to topics and adjusting the topic distributions until a coherent
structure is found. The algorithm does not consider the order of words within documents and
operates based on a 'bag of words' approach. Through statistical inference techniques, LDA
uncovers the hidden topic distributions within the dataset, providing insights into the thematic
content of the documents.

Working of LDA –
Step1: It assigns a random topic to each word.
Step2: It iterates to each word ‘w’ for each document and tries to adjust current topic-word
assignment with a new assignment. A new topic ‘k’ is being assigned to the word ‘w’ with
probability ‘P’ which is product of 2 probabilities; P1 and P2. So for every topic assigned to a
word, there are 2 probabilities calculated.
Step3 p(topic t ∣ document d): Probability of topic 𝑡 given document 𝑑. This reflects how
much document 𝑑 talks about topic 𝑡.
𝑝(𝑤𝑜𝑟𝑑 𝑤 ∣ 𝑡𝑜𝑝𝑖𝑐 𝑡): Probability of word 𝑤 given topic 𝑡. This reflects how likely word 𝑤 is
to be generated from topic 𝑡.
LDA computes p1*p2, based upon which it finds optimal topic ‘k’ for each word ‘w’.
Step4: This is being repeated many times until a steady stage is achieved where doc topic and
topic term distributions are fairly good. This is where LDA converges.
Advantages of LDA
One of the main advantages of LDA is that it is an unsupervised learning technique, meaning
that you do not need to provide any labels or categories for your documents. LDA can
automatically infer the topics from the data, and assign each document a probability of
belonging to each topic.
LDA produces topics that are interpretable to humans.Each topic is represented as a
distribution over words, making it easy to understand the main themes associated with each
topic.
Another advantage of LDA is that it is a flexible and adaptable method, that can be applied to
different types of text data, such as news articles, social media posts, reviews, or books. You
can also customize the number of topics, the hyperparameters, and the evaluation metrics
according to your needs and preferences.

Disadvantages of LDA
One of the main disadvantages of LDA is that it can produce ambiguous or incoherent topics,
especially if the data is noisy, sparse, or heterogeneous.
LDA relies on the assumption that the words in each topic are related and meaningful, but
this may not always be the case in reality. For example, some words may have multiple
meanings, some topics may overlap or be too broad, and some documents may contain
multiple or unrelated topics.
Another disadvantage of LDA is that it can be computationally expensive and time-
consuming, especially if the data is large, the number of topics is high, or the model is
complex. LDA requires multiple iterations and optimization steps to estimate the topic
distributions, which can take a lot of resources and memory.

Latent Semantic Analysis (LSA) -

Latent Semantic Analysis (LSA) is a technique used for topic modeling and dimensionality
reduction in Natural Language Processing (NLP). It's an unsupervised learning method that
analyzes relationships between a set of documents and the terms they contain to identify the
underlying structure in the text.
Term-Document Matrix:
LSA begins with a term-document matrix 𝐴A, where rows represent terms and columns
represent documents.
Each element 𝑎𝑖𝑗aij of 𝐴A represents the frequency of term 𝑖i in document 𝑗j, or some other
measure of term occurrence like TF-IDF scores.
Singular Value Decomposition (SVD):
LSA employs Singular Value Decomposition to reduce the dimensionality of the term-
document matrix.
SVD decomposes the term-document matrix 𝐴 into three matrices: 𝑈, Σ, and 𝑉𝑇.
Mathematically: 𝐴=𝑈Σ𝑉𝑇
𝑈 and 𝑉 are orthogonal matrices, and Σ is a diagonal matrix of singular values.
𝑈 contains the left singular vectors, 𝑉𝑇 contains the right singular vectors, and Σ contains the
singular values.
Dimensionality Reduction:
LSA retains only the top 𝑘k singular values and their corresponding singular vectors to reduce
the dimensionality.
The reduced matrices are denoted as 𝑈𝑘, Σ𝑘, and 𝑉𝑘𝑇.
𝑈𝑘 contains the 𝑘 most important left singular vectors, 𝑉𝑘Tcontains the 𝑘 most important
right singular vectors, and Σ𝑘 contains the top 𝑘×𝑘 submatrix of singular values.
Latent Semantic Space:
The reduced matrices 𝑈𝑘, Σ𝑘, and 𝑉𝑘𝑇 represent the latent semantic space.
Documents and terms are now represented as vectors in this lower-dimensional space.
Each document 𝑑d is represented as 𝑑d in 𝑉𝑘𝑇, and each term 𝑡 is represented as 𝑡 in 𝑈𝑘.
Topic Modeling:
LSA interprets the columns of 𝑈𝑘Uk as topics and the rows of 𝑉𝑘𝑇VkT as topic loadings for
each document.
Each topic 𝑘k is represented as a vector in the term space, where each element represents the
importance of the corresponding term in that topic.
Mathematically, the topic-term matrix is given by: 𝑇=𝑈𝑘⋅Σ𝑘T=Uk⋅Σk
Each document 𝑑d is represented as a vector in the topic space, where each element
represents the strength of the document's association with that topic.
Mathematically, the document-topic matrix is given by: 𝐷=Σ𝑘⋅𝑉𝑘𝑇D=Σk⋅VkT
Similarity Measurement:
LSA measures the similarity between documents or between a query and documents using
cosine similarity.
Cosine similarity measures the cosine of the angle between two vectors, providing a measure
of their similarity.
For example, the cosine similarity between two documents 𝑑1d1 and 𝑑2d2 is given by:
sim(𝑑1,𝑑2)=𝑑1⋅𝑑2∥𝑑1∥∥𝑑2∥sim(d1,d2)=∥d1∥∥d2∥d1⋅d2

BERT (Bidirectional Encoder Representations from

Transformers) -
BERT (Bidirectional Encoder Representations from Transformers) revolutionized Natural
Language Processing (NLP) by introducing contextualized word representations. Unlike
traditional word embeddings that assign a fixed vector to each word, BERT generates
dynamic embeddings that capture the meaning of words based on their context within a
sentence.
Here are key points about BERT's contextualized representations:
1. Contextual Understanding: BERT captures the meaning of words by considering
their surrounding context in the sentence. This allows BERT to understand nuances
and polysemy, where the meaning of a word changes depending on its context.
2. Transformer Architecture: BERT is based on the Transformer architecture, which
enables it to capture long-range dependencies in text efficiently. It consists of multiple
layers of self-attention mechanisms and feedforward neural networks.
3. Bidirectionality: BERT is bidirectional, meaning it considers both left and right
contexts of each word. This allows it to capture dependencies and relationships in
both directions, enhancing its contextual understanding.
4. Pre-training: BERT is pre-trained on large text corpora using two unsupervised tasks:
Masked Language Model (MLM) and Next Sentence Prediction (NSP). This pre-
training process enables BERT to learn rich and informative representations of words.
5. Fine-tuning: BERT's pre-trained weights can be fine-tuned on specific downstream
tasks, such as text classification, named entity recognition, and question answering.
Fine-tuning allows BERT to adapt its representations to the nuances of the task at
hand, improving performance.
6. Applications: BERT's contextualized representations have been applied to various
NLP tasks, including sentiment analysis, machine translation, text summarization, and
more. Its versatility and effectiveness make it a cornerstone in modern NLP research
and applications.