Université Hassan II de Casablanca

École Normale Supérieure de l'Enseignement Technique de Mohammedia

1
Traitement Automatique du Langage
Naturel (NLP)
Introduction

Zakariae EN-NAIMANI

2023/2024
 CBOW : pseudo-code
 Fonction Entraîner_CBOW(corpus, vector_size, window, learning_rate, nombre_iterations)

# Initialisation des vecteurs de mots

 Initialiser les vecteurs de mots aléatoirement
 Pour iter de 1 à nombre_iterations

– Pour chaque phrase dans le corpus

• Pour chaque mot de la phrase

# Récupérer les mots de contexte

 context_words = Mots de la fenêtre autour du mot cible

# Calcul de la moyenne des vecteurs de contexte

 average_context_vector = Moyenne des vecteurs des mots de contexte

# Prédiction du mot cible

 predicted_word_vector = Calculer la prédiction en utilisant le modèle CBOW

# Calcul de l'erreur de prédiction

 error = Différence entre le vecteur réel du mot cible et la prédiction

# Mise à jour des vecteurs de mots

 Mise à jour des vecteurs des mots de contexte et du mot cible en utilisant l'erreur Fin Pour Fin Fonction

Traitement Automatique du Langage Naturel Pr. Z. EN-NAIMANI 1-35

  Ashok believes that the weather operates as a discrete Markov chain,
wherein the chain there are only two states whether the weather is
Rainy or it is sunny. The condition of the weather cannot be observed by
Ashok, here the conditions of the weather are hidden from Ashok. On
each day, there is a certain chance that Bob will perform one activity
from the set of the following activities {“jog”, “work”,” clean”}, which
are depending on the weather. Since Rahul tells Ashok that what he has
done, those are the observations. The entire system is that of a hidden
Markov model (HMM).
 Here we can say that the parameter of HMM is known to Ashok because
he has general information about the weather and he also knows what
Rahul likes to do on average.
 So let’s consider a day where Rahul called Ashok and told him that he has
cleaned his residence. In that scenario, Ashok will have a belief that there
are more chances of a rainy day and we can say that belief Ashok has is
the start probability of HMM let’s say which is like the following.
Traitement Automatique du Langage Naturel Pr. Z. EN-NAIMANI 1-40
  The states and observation are: states = ('Rainy', 'Sunny') observations =
('walk', 'shop', 'clean')
 And the start probability is:
 start_probability = {'Rainy': 0.6, 'Sunny': 0.4}
 Now the distribution of the probability has the weightage more on the
rainy day stateside so we can say there will be more chances for a day to
being rainy again and the probabilities for next day weather states are as
following
 transition_probability = { 'Rainy' : {'Rainy': 0.7, 'Sunny': 0.3}, 'Sunny' :
{'Rainy': 0.4, 'Sunny': 0.6}, }
 From the above we can say the changes in the probability for a day is
transition probabilities and according to the transition probability the
emitted results for the probability of work that Rahul will perform is

Traitement Automatique du Langage Naturel Pr. Z. EN-NAIMANI 1-41

  emission_probability = { 'Rainy' : {'jog': 0.1, 'work': 0.4, 'clean': 0.5},
'Sunny' : {'jog': 0.6, 'work: 0.3, 'clean': 0.1}, }
 This probability can be considered as the emission probability. Using the
emission probability Ashok can predict the states of the weather or using
the transition probabilities Ashok can predict the work which Rahul is
going to perform the next day.
 Below image shown the HMM process for making probabilities

Traitement Automatique du Langage Naturel Pr. Z. EN-NAIMANI 1-42

 Hidden Markov Models in NLP

 So here from the above intuition and the example we can understand
how we can use this probabilistic model to make a prediction. Now let’s
just discuss the applications where it can be used.
 From the above application of HMM, we can understand that the
applications where the HMM can be used have sequential data like time
series data, audio, and video data, and text data or NLP data. In this
article, our main focus is on those applications of NLP where we can use
the HMM for better performance of the model, and here in the above-
given list, we can see that one of the applications of the HMM is that we
can use it in the Part-of-Speech tagging. Next in the article, we will see
how we can use the HMM for POS-tagging.

Traitement Automatique du Langage Naturel Pr. Z. EN-NAIMANI 1-43

Traitement Automatique du Langage Naturel Pr. Z. EN-NAIMANI 1-44
Traitement Automatique du Langage Naturel Pr. Z. EN-NAIMANI 1-45
 Notations

Traitement Automatique du Langage Naturel Pr. Z. EN-NAIMANI 1-46

Traitement Automatique du Langage Naturel Pr. Z. EN-NAIMANI 1-47
 Comment un Hmm engendre une séquence?

Traitement Automatique du Langage Naturel Pr. Z. EN-NAIMANI 1-48

 HMM comme règles de classification
de séquences
 Les trois problèmes des HMM

Traitement Automatique du Langage Naturel Pr. Z. EN-NAIMANI 1-49

 L’évaluation de la probabilité d’observation

 L’évaluation directe

Traitement Automatique du Langage Naturel Pr. Z. EN-NAIMANI 1-50

 L’évaluation par les fonctions
forward-backward

Traitement Automatique du Langage Naturel Pr. Z. EN-NAIMANI 1-51

 Calcule de la fonction forward

Traitement Automatique du Langage Naturel Pr. Z. EN-NAIMANI 1-52

 Calcul de la fonction backward

Traitement Automatique du Langage Naturel Pr. Z. EN-NAIMANI 1-53

 Probabilité d’observation

Traitement Automatique du Langage Naturel Pr. Z. EN-NAIMANI 1-54

 Le calcul du chemin optimal :
l'algorithme de Viterbi

Traitement Automatique du Langage Naturel Pr. Z. EN-NAIMANI 1-55

 Algorithme de Viterbi

Traitement Automatique du Langage Naturel Pr. Z. EN-NAIMANI 1-56

 Apprentissage de HMM

Traitement Automatique du Langage Naturel Pr. Z. EN-NAIMANI 1-57

Traitement Automatique du Langage Naturel Pr. Z. EN-NAIMANI 1-58
Traitement Automatique du Langage Naturel Pr. Z. EN-NAIMANI 1-59
Traitement Automatique du Langage Naturel Pr. Z. EN-NAIMANI 1-60
 What is POS-tagging?

 We have learned in our school timings that the part of speech indicates
the function of any word, like what it means in any sentence. There are
commonly nine parts of speeches; noun, pronoun, verb, adverb, article,
adjective, preposition, conjunction, interjection, and a word need to be
fit into the proper part of speech to make sense in the sentence.
 POS tagging is a very useful part of text preprocessing in NLP as we
know that NLP is a task where we make a machine able to communicate
with a human or with a different machine. So it becomes compulsory for
a machine to understand the part of speech.
 Classifying words in their part of speech and providing their labels
according to their part of speech is called part of speech tagging or POS
tagging OR POST. Hence the set of labels/tags is called a tagset. In the
article, we have seen how we can implement the part of speech at a
beginning level using the NLTK where the tagsets package of NLTK was
helping us to provide the part of speech tag to our documents.

Traitement Automatique du Langage Naturel Pr. Z. EN-NAIMANI 1-61

 POS Tagging With Hidden Markov Model

 We can say that in the case of HMM is a stochastic technique for POS
tagging. Let’s take an example to make it more clear how HMM helps in
selecting an accurate POS tag for a sentence.
 As we have seen in the example of the HMM process in POS tagging the
transition probability is the likelihood of any sequence for example what
are the chances for a noun word to come after any modal and a modal
after a verb and a verb after a noun.
 Let’s take the sentence “Rahul will eat food” where Rahul is a noun, will
is a modal, eat is a verb and food is also a noun, so the probability for a
word to be in a particular class of part of speech is called the Emission
probability.

Traitement Automatique du Langage Naturel Pr. Z. EN-NAIMANI 1-62

  Let’s take a look at how we can calculate these two probabilities for a set
of sentences:
 Mary Jane can see will
 The spot will see Mary
 Will Jane spot Mary?
 Mary will pat Spot
 The below table is a counting tableau for the words with their part of
speech type

Traitement Automatique du Langage Naturel Pr. Z. EN-NAIMANI 1-63

  Let’s divide each word’s appearance by the total number of every part of
speech in the set of sentences.

Traitement Automatique du Langage Naturel Pr. Z. EN-NAIMANI 1-64

  Here in the table, we can see the emission probabilities of every word.
 Now as we have discussed that the transition probability is the
probability of the sequences we can define a table for the above set of
sentences according to the sequence of part of speech.

Traitement Automatique du Langage Naturel Pr. Z. EN-NAIMANI 1-65

Traitement Automatique du Langage Naturel Pr. Z. EN-NAIMANI 1-66
Traitement Automatique du Langage Naturel Pr. Z. EN-NAIMANI 1-67
Traitement Automatique du Langage Naturel Pr. Z. EN-NAIMANI 1-68
Traitement Automatique du Langage Naturel Pr. Z. EN-NAIMANI 1-69