See discussions, stats, and author profiles for this publication at: https://www.researchgate.

net/publication/308371781

An introduction to Twitter Data Analysis in Python

Working Paper · September 2016

DOI: 10.13140/RG.2.2.12803.30243

CITATIONS READS

2 2,445

1 author:

Vivek Wisdom
Deloitte & Touche Llp
2 PUBLICATIONS   2 CITATIONS   

SEE PROFILE

Some of the authors of this publication are also working on these related projects:

Data Analysis with Twitter View project

DeepGAN View project

All content following this page was uploaded by Vivek Wisdom on 21 September 2016.

The user has requested enhancement of the downloaded file.

 An introduction to Twitter Data Analysis in Python
Vivek Wisdom, Rajat Gupta
Artigence Inc
{Vivek, Rajat}@artigence.in

ABSTRACT 2.1. Registering App

In this introductory paper, we explain the process of
storing, preparing and analyzing twitter streaming data,
then we examine the methods and tools available in python
programming language to visualize the analyzed data. we
believe that using social networks and microblogs for
efficient analysis of massive real-time data about the
product, services, and global events is crucial for better
decisions. Twitter’s popularity as an information source has
led to the development of applications and research in
various domains. Humanitarian Assistance and Disaster
Relief is one domain where information from Twitter is used
to provide situational awareness to a crisis situation.
Managerial decisions, stock prediction[4], improving traffic
prediction[1] are some example of the use cases getting
favorites among researchers.
Keywords—Twitter; Data Analysis; Python;
1. INTRODUCTION
Social networks and microblogging sites have become
the unparalleled source of unstructured data. This data is
enormous in quantity and also in terms of the useful
information they can provide if we process them effectively.
This is due to the nature of microblogs on which people post
real-time messages about their opinions on a variety of
topics, discuss current issues, complain, and express their
sentiment for products they use in daily life. In fact, many
companies have started analyzing such massive amount of
important data to get a sense of general sentiment for their
product and/or services. Many times proactive companies
study user reactions and reply to the user on social
microblogs. This process provides on spot solution to make
the user experience better but at large scale its very painful
and time-consuming task. So the challenge here is to build
solutions which analyze the sources of data coming from
various microblogging and social networks to make
important long-term product design and service
implementation decisions.
In this paper, we take one such social network called
Twitter to analyze and visualize various important metrics
related to an event, product or service.
2. COLLECTING DATA
To be able to access Twitter data programmatically we
need to create and register an app on twitter developers
website for authentication and thereafter we can access data
by using Twitter API.
To register the twitter app, we need to create a new app
https://apps.twitter.com/. On registering the app we will
receive consumer_key and consumer_secret_key. Next, From
the configuration page of the app, we will get access_token
and access_token_secret, which will be used to get access to
twitter on behalf of our application. We must keep these
authentication tokens private as they can be misused. Best
practice is to create a separate config file and keep these
tokens.
2.2. Accessing Data
Twitter provides REST API’s to connect with their
service. We will use one python library to access the twitter
REST API’s called Tweepy. It provides wrapper methods to
easily access twitter REST API.
To install Tweepy we can use below command.
pip install tweepy
In order to authorize our app to access Twitter on our
behalf, we need to use the OAuth interface. Below code will
use tweepy OAuthHandler method and our configuration
tokens to provide access to twitter. [5]
import tweepy
from tweepy import OAuthHandler
consumer_key = 'YOUR-CONSUMER-KEY'
consumer_secret = 'YOUR-CONSUMER-
SECRET'
access_token = 'YOUR-ACCESS-TOKEN'
access_secret = 'YOUR-ACCESS-SECRET'
auth=OAuthHandler(consumer_key,consu
mer_secret)
auth.set_access_token(access_token,a
ccess_secret)
api = tweepy.API(auth)
(Python code to get twitter access)
 The api variable is now our entry point for most of the
operations we can perform with Twitter.
2.3. Storing Data
Now we will access all tweet data from personal profile
and store that is JSON file to use that for our analysis steps.
Tweepy library provides simple cursor interface to iterate
through all the tweets and store them in JSON file.
We will use below code to store our user timeline tweet
data to my_tweets.json file. [5]
def process_or_store(tweet):
with open(‘data/my_tweets.json','a')
as mt:
mt.write(json.dumps(tweet)+’\n')
for tweet in tweepy.Cursor
(api.user_timeline).items():
process_or_store(tweet._json)
(Python code to read user timeline)
3. PREPARING DATA
Before we begin to analyze the twitter data, it's important
to understand the structure of the tweet as well as pre-process
the data to remove non-useful terms called stop-words.
Preprocessing of data in data analysis is the very important
step. Preprocessing is in the simple term means to take in the
data and prepare the data for optimal output considering our
requirement.
So, to preprocess the tweet data we need to understand
the structure of the single tweet and analyze its different
parts.
3.1.The Anatomy of the Tweet
Tweets are short messages, restricted to 140 characters in
length. Due to the nature of this microblogging service
(quick and short messages), people use acronyms, make
spelling mistakes, use emoticons and other characters that
express special meanings. Following is a brief terminology
associated with tweets. Emoticons: These are facial
expressions pictorially represented using punctuation and
letters; they express the user’s mood. Target: Users of Twitter
use the “@” symbol to refer to other users on the microblog.
Referring to other users in this manner automatically alerts
them. Hashtags: Users usually use hashtags to mark topics.
This is primarily done to increase the visibility of their
tweets.
A single tweet contains a lot of information related to
users, the text of the tweet, created date of the tweet, the
location of the tweet and many more fields. We will use
some of the fields to complete the analysis.
The key fields of a single tweet are following:
text: text of the tweet,
lang: acronym of the tweet language like ‘en’,
created_date: date of creation of the tweet,
favorite_count: number of favorites of the tweet,
retweet_count: retweets of the tweet,
place, geolocation: location information, if available,
user: the full profile of user,
entities: list of entities like url’s, @mentions, #hashtags
We can imagine how these data already allow for some
interesting analysis: we can check who is most favorited/
retweeted, who’s discussing with who, what are the most
popular hashtags and so on. But, the most important field we
are looking for is the content of the tweet which is
represented by field text. Next step is to tokenize the tweet
text fields.
3.2.Tokenizing the Tweet
Tokenization is one of the most basic, yet most important,
steps in text analysis. The purpose of tokenization is to split a
stream of text into smaller units called tokens, usually words
or phrases. We will use python NLTK library to tokenize the
tweets. Even NLTK library needs some preprocessing steps
to correctly tokenize @mentions and #hashtags. We use
regular expressions to provide exceptions for mentions and
hashtags.
Tokenization prepares the text for next step, which is to
removing stop-words like ‘the’, ‘or’, ‘to’, ‘and’ etc.
3.3.Removing Stop-Words
Stop-word removal is one important step that should be
considered during the pre-processing stages. Stop-words are
most popular and common words of any language. While
their use in the language is crucial, they don’t usually convey
a particular meaning, especially if taken out of context. This
is the case of articles, conjunctions, some adverbs, etc. which
are commonly called stop-words. Some libraries provide
default stop-words for different languages. NLTK library
provides default stop-words for English language.
An array of custom stop words included in this analysis
are as below:
[‘The', 'what', 'What', 'You',
'Your', 'A', 'new', 'https', 'Hi',
'We', 'My', 'Now', 'please', ‘get’]
As we can see in the above array of stop words, they
don’t add any information in data analysis where we are
finding term frequencies.These words are heavily used in
English language and we might these examples as the most
occurred terms.
Given the nature of our data and our tokenization, we
should also be careful with all the punctuation marks and
with terms like RT (used for re-tweets) and via (used to
mention the original author of an article or a re-tweet), which
are not in the default stop-word list.
4. ANALYZING THE DATA
After the text data is preprocessed and tokenized we can
now proceed with different analysis objectives.
4.1.Term Frequencies
Counting frequencies of a term in twitter data analysis is
one of the simplest steps. By this, we can analyze for a
particular user that what he frequently tweets about. One use
case of term frequencies is that advertisement companies can
provide targeted ads based on the user's term frequencies. It
will have more possibility that user clicks or visit the
promoted website.
Below code can be an example of counting all the
frequencies of all the terms.
terms_only = [term for term in
preprocess (tweet['text']) if term
not in stop and not
term.startswith(('@', ‘#'))]
In the above code snippet we are listing out all the terms
which are in preprocessed tweet text if that term is not in
stop-words array stop it doesn’t starts with @ or #. We can
use python collection called counter() to count the
occurrences of the terms and list them aside to their count.
Here are the 20 most used terms on my personal user
timeline. [5]
[('India', 67), ('world', 38),
('work', 30), ('make', 30), ('©',
27), ('Photography', 27), ('future',
26), ('Nice', 24), ('visit', 24),
('going', 24), ('time', 24),
('This', 24), ('twitter', 23),
('great', 22), ('Do', 22), ('win',
22), ('final', 22), ("India's", 21),
(‘PM’, 20), (‘next’, 20)]
If we see above result, this is also not perfect tokenizer
we have used, as © copyright text is still present in the
tokens. But with a number of iterations or by using powerful
stop words library we can develop tokenizer close to perfect.
4.2.Bigram Terms
Bigram terms are those terms who occur together
frequently. NLTK library provides a method to find bigrams
in tokenized data. The bigrams() method from NLTK will
take a list of tokens and produce a list of tuples using
adjacent tokens.
Bigrams provide more meaning than a single text, they
can be crucial in finding useful information.Below code can
generate bigrams from a list of term frequencies:
terms_bigrams=bigrams(terms_only)
So passing our tokenized data after removing stop-words
and some other commonly known words, we get following
10 most frequently used bigrams:
[(('Photography', '©'), 27),
(('Vision', '2020'), 12), (('Vivek',
'Wisdom'), 12), (('India',
'Vision'), 12), (('transfer',
'handle'), 12), (("Google's",
'CEO'), 8), (('Sundar', 'Pichai'),
8), (('Alibaba', 'Singles'), 8),
(('really', 'deserve'), 8),
(('Sultanpur', 'really'), 8)]
So it clearly shows to include most frequently used terms
on my timeline.
4.3.Most Used Hashtags
Hashtags arena of the most frequently used features of
twitter. They used to represent most recent happening in the
world. Using hashtags effectively we can find many useful
pieces of information like a number of tweets of particular
hashtags, which is generally used to compare the twitter
battle between teams in most sports.
In current dataset i.e. my_tweets.json we will try to find
the most used #hashtags. Below sample code can generate all
the hashtags used in the tweet dataset:
terms_hash= [term for term in
preprocess(tweet[‘text']) if
term.startswith('#')]
Below are most used hashtags on my user timeline tweet
data:
[('#ibmsmartcamp', 32), ('#', 12),
('#MakeItHappen', 12), ('#IndvsPak',
10), ('#ShangriLaExperience', 6),
('#EarthDay', 6), ('#Scala', 5),
('#MissingOut', 6),('#CRUD', 5),
(‘#AllEnglandSSP', 8),
('#IBMSmartCamp', 8),
('#SpotTheMissing', 6),
('#entrepreneur', 6),
('#SuperGenes', 5),
('#BillionsinChange', 5)]
If we notice there is exactly one ‘#’ is present in my
tweets for 12 times, which has to be ignored as it doesn’t
convey any information. We can correct those mistakes using
iterative or by implying more sophisticated tokenizers.
4.4.Most Used Mentions
Mentions in tweets are used to refer someone who is
present on the twitter using ‘@username’. They are also used
to reply someone are to notify multiple people about an
information. Analyzing mentions, we can find most frequent
contacts of a user and then we can suggest them of new
followers based on the followers of frequent contacts.
In current dataset i.e. my_tweets.json we will try to find
the most used @mentions. To find all the mentions in the
tweet dataset we can search for all the words which are
starting with ‘@ Symbol’, as in below code:
terms_mentions = [term for term in
preprocess(tweet[‘text']) if
term.startswith('@')]
Below are most used mentions on my user timeline tweet
data:
[('@narendramodi', 58), ('@Piclogy',
27), ('@google', 24), ('@Artigence',
24), ('@Inc', 16), ('@xprize', 16),
('@businessinsider', 14),
('@pranavmistry', 12),
('@IndianDiplomacy', 12),
('@elonmusk', 12),
('@richardbranson', 10), ('@buffer',
10), ('@singularityhub', 10),
('@YouTube', 10), ('@ICC', 10)]
5. ANALYZING STREAMING DATA 5.4.Most Used Mentions
Analyzing streaming data is very important as it allows Out of 10000 live tweets below are the most used
us to make real-time decisions on the basis of real-time data. m e n t i o n s f o r # D e a d l i n e D a y. A s w e c a n s e e
Streaming data analysis can be large in terms of data being ‘@SkySportsNewsHQ’ is the most active twitter account for
generated so analysis of streaming data requires heavy this dataset.
computation resources.
[('@SkySportsNewsHQ', 445),
In this paper, we have analyzed #DeadlineDay hashtag. ('@TransferRelated', 262),
#DeadlineDay is the transfer window period during the year ('@TransferSources', 249),
in which a football club can transfer players from other ('@bbcsport_david', 225), ('@br_uk',
countries into their playing staff. Such a transfer is completed 175), ('@ChelseaFC', 145),
by registering the player into the new club through FIFA. ('@SkyFootball', 145),
In streaming data analysis we will use the different ('@SamNasri19', 142),
analysis use cases for data which is being generated live. ('@JackWilshere', 139), ('@ManCity',
136), ('@LeeClayton_', 136),
5.1.Term Occurrences ('@NCLairport', 135),
('@DeadlineDayLive', 132),
For #DeadlineDay streaming data, we have analyzed ('@3FLnQe', 131), ('@CPFC', 127)]
below most used term occurrences. This shows that David
Luiz is tweeted most of time in approximately 5000 live
tweets. 6. VISUALIZATIONS
Visualizations helps represent the analyzed data to make
[('David', 578), ('Luiz', 517), decisions effectively. Here for #DeadlineDay we will display
('loan', 433), ('Heathrow', 380), map of the users tweeting about deadline day.
('spotted', 378), ('Airport', 360),
('Roma', 264), ('Chelsea', 245),
('Arsenal', 243), ( ', 228),
('City', 214), ('Wilshere', 210),
('move', 204), ('season-long', 188),
('deal', 177)]

5.2.Bigrams Terms
Bigrams in this #DeadlineDay data show that David Luiz
terms has been used most frequently together as its complete
name of the footballer from Brazil.

[(('David', 'Luiz'), 561),

(('Heathrow', 'Airport'), 390),
(('Luiz', 'spotted'), 388),
(('spotted', 'Heathrow'), 386),
(('season-long', 'loan'), 220), (Image) Display of Tweet locations for #DeadlineDay
(('Airport', 'https://t.co/
fbU4drhfRB'), 151), (('Samir', As we can see in the above map red dots represent
'Nasri'), 145), (('think', 'might'), #DeadlineDay is most used hashtag in UK and Europe.
104), (('Quite', 'welcoming'), 104), To display the map we have used the geoJSON library to
(('Jack', 'Wilshere'), 104), tag the different tweet locations on map provided by Leaflet
(('party', 'someone'), 104), map library.
(('someone', 'Do'), 104), (('Do',
'think'), 104), (('welcoming', 7. CONCLUSION
'party'), 104)]
In this paper, we started with very basics of Twitter data
5.3.Most Used Hashtags analysis. We explained for twitter app authentication using
OAuth and Tweepy. Then we explained steps to collect
Out of 6500 live tweets below are the most used hashtags historical data as well as streaming data. We then
for #DeadlineDay. preprocessed the data using tokenizers.
[('#DeadlineDay', 5163), ('#MUFC', In the final step, we tried to execute a number of use
233), ('#CFC', 151), ('#AFC', 143), cases to analyze the stored data. We represented results of
('#deadlineday', 116), ('#SSNHQ', analyzing most used terms for a data set, most used hashtags,
106), ('#', 105), ('#CPFC', 94), most used mentions of user accounts on twitter and we also
('#LFC', 81), ('#TIMYHelloGreece', represented the bigrams i.e. two terms used frequently in our
75), ('#GirlsInTheHouse3', 72), dataset.
('#FelizMiercoles', 62), ('#UCL', Then we created a map visualization of user tweeting for
58), ('#PSG', 48)] a particular hashtag #DeadlineDay and we found that it has
been most actively tweeted from Europe.
 This paper is introductory in nature and hence deals with
basics of twitter data analysis using python. In future work,
we will try to represent more advanced data analysis patterns
like sentiment analysis and decision making with more
accurate results.
REFERENCES

1. J He, W Shen, P Divakaruni, L Wynter, R Lawrence, “Improving

Traffic Prediction with Tweet Semantics”, Proceedings of the Twenty-
Third International Joint Conference on Artificial Intelligence, pp.
1387–1393, August 3-9 2013.
2. A. Agarwal, B. Xie, I Vovsha, O. Rambow, R. Passonneau “Sentiment
Analysis of Twitter Data” In the proceedings of Workshop on
Language in Social Media, ACL, 2011
3. S Kumar, F Morstatter, H Liu, “Twitter Data Analytics” Springer
Book 2013
4. A Mittal, A Goel, “Stock Prediction Using Twitter Sentiment
Analysis”, Stanford University, 2011
5. “D Ediger, K Jiang, J Riedy, D. A. Bader “Massive Social Network
Analysis: Mining Twitter for Social Good”, 39th International
Conference on Parallel Processing 2010, pp. 583-593
6. https://github.com/vivekwisdom/TwitterAnalysisApp, Code
repository of the sample application

View publication stats