NLP Worksheet

Text processing, bag of words, tf-idf activity

Suppose you have obtained these information and you would like to analyse it. Let’s start by making it ready for the
computer!

Corpus
Document 1: We can use health chatbots for treating stress.
Document 2: We can use NLP to create chatbots and we will be making health chatbots now!
Document 3: Health Chatbots cannot replace human counsellors now. Yay >< !! @1nteLA!4Y

Step 1: Sentence Segmentation

No. Sentence

1 We can use health chatbots for treating stress.

2 We can use NLP to create chatbots and we will be making health chatbots now!
3 Health Chatbots cannot replace human counsellors now. Yay! >#< @!@!nLtAY

Step 2: Tokenization
Separate your sentences into tokens. How many tokens do you have?

Tokens

We, can, use, health, chatbots, for, treating, stress, ., We, can, use, NLP, to, create, chatbots,
and, we, will, be, making, health, chatbots, now, !, Health, Chatbots, cannot, replace, human,
counsellors, now, ., Yay, !, >#<, @!@!nLtAY

26
Number of tokens: ________
Step 3: Remove stopwords, special characters, numbers
List out the stopwords, special characters, and numbers that you want to remove!

Stopwords, special characters, and numbers

Remove stopwords, special characters, numbers List out the stopwords, special characters,
and numbers that you want to remove! Stopwords, special characters, and numbers

we, can, use, for, to, and, we, will, be, now, ., !, >#<, @!@!nLtAY

Step 4: Converting text to a common case

Which text do you need to modify? What is the modified form?

Modified form

Converting text to a common case Which text do you need to modify? What is the modified
form? Modified form

health chatbots, nlp, create chatbots, making health chatbots, health chatbots cannot replace
human counsellors, yay

Step 5: Stemming
List out the stem words.

Stem words

Stemming List out the stem words. Stem words

health, chatbot, nlp, creat, chatbot, make, health, chatbot, health, chatbot, cannot, replac,
human, counsellor, yay

Step 6: Lemmatization
List out the root words/ lemma.

Lemma

Lemmatization List out the root words/ lemma. Lemma

health, chatbot, nlp, create, chatbot, make, health, chatbot, health, chatbot, can, replace,
human, counsellor, yay
Final data
List out the final, processed data.

Processed data

Congratulations, you’ve managed to process the data!

Bag of words
Step 1: Collect data and process it
For this exercise, we can use the sentences without processing it so that it is easier for us to read the sentences.

No. Sentence

1 We can use health chatbots for treating stress

2 We can use NLP to create chatbots and we will be making health chatbots now

3 Health chatbots cannot replace human counsellors now

Step 2: Create dictionary

Make a list of all the different words in the text.

Dictionary

dictionary = [‘we’, ‘can’, ‘use’, ‘health’, ‘chatbots’, ‘for’, ‘treating’, ‘stress’, ‘nlp’, ‘to’, ‘create’,
‘and’, ‘will’, ‘be’, ‘making’, ‘now’, ‘cannot’, ‘replace’, ‘human’, ‘counsellors’]

Step 3:Create document vectors

Use the next page to create your document vector!
document_vectors = [ [1, 1, 1, 1, 1, 1, 1, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0],# sentence 1
[1, 1, 1, 1, 1, 0, 0, 0, 1, 1, 1, 1, 1, 1, 1, 1, 0, 0, 0, 0], # sentence 2
[0, 0, 0, 1, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 1, 1, 1] # sentence 3 ]

Each vector has the same length as the dictionary, and each element corresponds to the
count of a word in the sentence. For example, the first element of the first vector is 1, which
means that the word ‘we’ appears once in the first sentence. The last element of the last vector
is 1, which means that the word ‘counsellors’ appears once in the last sentence.
Tf-idf
You’ve obtained your bag of words. Now let’s continue with the tf-idf!

Step 1 - 3: Count the number of documents where the word appears at least once & write that
number down next to the word in your vocabulary to get your document frequency. Draw your
own table for this!

Word Document
Example 1 Document
of a document frequency:2 Document 3
we 1 1 0
can 1 1 0
use
aman1 1 0and Anil are stressed went to a therapist download health chatbot
health 1 1 1
chatbots 1 1 1
for 21 0 0 1 2 1 1 2 2 2 1 1 1 1
treating 1 0 0
stress 1 0 0
nlp 0 1 0
Your
to 0document
10 frequency:
create 0 1 0
and 0 1 0
will 0 1 0
be 0 1 0
making 0 1 0
now 0 1 1
cannot 0 0 1
replace 0 0 1
human 0 0 1
counsellors 0 0 1

Step 4: Get your inverse document frequency.

Word Document
Example 1 Document
of an inverse document 2 Document 3
frequency:
we 0.025 0.016 0
can 0.025 0.016 0
use 0.025and
aman 0.016are
anil 0
stressed went to a therapist download health chatbot
health 0 0 0
chatbots0 0 0
for 0.068 0 0
treating
3/2 0.068
3/1 3/2 0 3/1 0
3/1 3/2 3/2 3/2 3/1 3/1 3/1 3/1
stress 0.068 0 0
nlp 0 0.043 0
to 0 0.043 0
Your inverse
create 0 document
0.043frequency:
0
and 0 0.043 0
will 0 0.043 0
be 0 0.043 0
making 0 0.043 0
now 0 0.016 0.016
Step 5: Get your tf-idf
Example of a tf-idf:

After log operation:

Your tf-idf: