Probabilistic Topic Models

Probabilistic Topic Models for
Text Mining
ChengXiang Zhai ( 翟成祥 )
Department of Computer Science
Graduate School of Library & Information Science
Institute for Genomic Biology, Statistics
University of Illinois, Urbana-Champaign
http://www-faculty.cs.uiuc.edu/~czhai, czhai@cs.uiuc.edu
2008 © ChengXiang Zhai China-US-France Summer School, Lotus Hill Inst., 2008 1
What Is Text Mining?
“The objective of Text Mining is to exploit information
contained in textual documents in various ways, including
…discovery of patterns and trends in data, associations
among entities, predictive rules, etc.” (Grobelnik et al.,
2001)
“Another way to view text data mining is as a process of

exploratory data analysis that leads to heretofore
unknown information, or to answers for questions for
which the answer is not currently known.” (Hearst, 1999)
(Slide from Rebecca Hwa’s “Intro to Text Mining”)
Two Different Views of Text Mining
• Data Mining View: Explore patterns in textual data

– Find latent topics
Shallow mining
– Find topical trends
– Find outliers and other hidden patterns
• Natural Language Processing View: Make

inferences based on partial understanding natural
language text
– Information extraction Deep mining
– Question answering
Applications of Text Mining
• Direct applications: Go beyond search to find knowledge

– Question-driven (Bioinformatics, Business Intelligence, etc):
We have specific questions; how can we exploit data mining to
answer the questions?
– Data-driven (WWW, literature, email, customer reviews, etc):
We have a lot of data; what can we do with it?
• Indirect applications
– Assist information access (e.g., discover latent topics to better
summarize search results)
– Assist information organization (e.g., discover hidden
structures)
Text Mining Methods
• Data Mining Style: View text as high dimensional data
– Frequent pattern finding
– Association analysis
– Outlier detection
• Information Retrieval Style: Fine granularity topical analysis
– Topic extraction
– Exploit term weighting and text similarity measures
• Natural Language Processing Style: Information Extraction
– Entity extraction
– Relation extraction
– Sentiment analysis
– Question answering
• Machine Learning Style: Unsupervised or semi-supervised learning
– Mixture models
Topic of this lecture
– Dimension reduction
Outline
• The Basic Topic Models:
– Probabilistic Latent Semantic Analysis (PLSA) [Hofmann
99]
– Latent Dirichlet Allocation (LDA) [Blei et al. 02]
• Extensions
– Contextual Probabilistic Latent Semantic Analysis
(CPLSA) [Mei & Zhai 06]
Basic Topic Model: PLSA
PLSA: Motivation
What did people say in their blog articles about “Hurricane Katrina”?
Query = “Hurricane Katrina”
Results:
Government Response New Orleans Oil Price Praying and Blessing Aid and Donation Personal
bush 0.071 city 0.063 price 0.077 god 0.141 donate 0.120 i 0.405
president 0.061 orleans 0.054 oil 0.064 pray 0.047 relief 0.076 my 0.116
federal 0.051 new 0.034 gas 0.045 prayer 0.041 red 0.070 me 0.060
government 0.047 louisiana 0.023 increase 0.020 love 0.030 cross 0.065 am 0.029
fema 0.047 flood 0.022 product 0.020 life 0.025 help 0.050 think 0.015
administrate 0.023 evacuate 0.021 fuel 0.018 bless 0.025 victim 0.036 feel 0.012
response 0.020 storm 0.017 company 0.018 lord 0.017 organize 0.022 know 0.011
brown 0.019 resident 0.016 energy 0.017 jesus 0.016 effort 0.020 something 0.007
blame 0.017 center 0.016 market 0.016 will 0.013 fund 0.019 guess 0.007
governor 0.014 rescue 0.012 gasoline 0.012 faith 0.012 volunteer 0.019 myself 0.006
Probabilistic Latent Semantic
Analysis/Indexing (PLSA/PLSI) [Hofmann 99]
• Mix k multinomial distributions to generate a document

• Each document has a potentially different set of mixing
weights which captures the topic coverage
• When generating words in a document, each word may be
generated using a DIFFERENT multinomial distribution
(this is in contrast with the document clustering model
where, once a multinomial distribution is chosen, all the
words in a document would be generated using the same
model)
• We may add a background distribution to “attract”
background words
PLSA as a Mixture Model
k
pd ( w)  B p ( w |  B )  (1  B )  d , j p ( w |  j )
j 1
k
log p (d )   c( w, d ) log[B p ( w |  B )  (1  B )  d , j p ( w |  j )]
wV j 1
Document d
warning 0.3
?
Topic 1 system 0.2.. d,1
? 1 “Generating” word w
in doc d in the collection
aid 0.1
? 2
Topic 2 ?
donation 0.05 d,2 1 - B
support 0.02
? ..
d, k W
… k
statistics 0.2
?
loss 0.1? B
Topic k dead 0.05? .. B
is 0.05
?
Background B ?
the 0.04 Parameters:
a 0.03
? .. B=noise-level (manually set)
’s and ’s are estimated with Maximum Likelihood
How to Estimate j: EM Algorithm
the 0.2
Known a 0.1
Background we 0.01 Observed Doc(s)
p(w | B) to 0.02
…
Unknown … Suppose,
topic model text =? we know
mining =? ML
p(w|1)=? association =? the identity Estimator
word =? of each
“Text mining” …
word ...
…
Unknown …
topic model information =?
p(w|2)=? retrieval =?
query =?
document =?
“information
…
retrieval”
How the Algorithm Works
c(w,d)(1 - p(zd,w = B))p(zd,w=j)
c(w,d)p(zd,w = B) πd1,1 πd1,2
c(w, d) ( P(θ1|d1) ) ( P(θ2|d1) )
aid 7
d1 price 5
Initial value
oil 6
πd2,1 πd2,2
aid 8 ( P(θ1|d2) ) ( P(θ2|d2) )
d2 price 7
oil 5 Initial value
P(w| θ) Topic 1 Topic 2

Initializing
Iteration
Iteration
Iteration
1:
2:
1:
2:πd,E
3,
M
j Step:
and
4,
Step:
5,P(w|
…
split
re-θj)
aid estimate
word Until
counts
with πconverging
d, j and
random withvalues
different
P(w| θj) by
price Initial value topics (by
adding andcomputing
normalizing z’ the
s)12
oil splitted word counts
Parameter Estimation
E-Step:
Word w in doc d is generated Application of Bayes rule
- from cluster j
- from background
 d( n, )j p ( n ) ( w |  j )
p( zd ,w  j ) 

k
j '1
 d( n, )j ' p ( n ) ( w |  j ' )
B p ( w |  B )
p( zd ,w  B) 
B p( w |  B )  (1  B ) j 1  d( n, )j p ( n ) ( w |  j )
k
M-Step:  d( n, j 1) 
 wV
c( w, d )(1  p ( z d , w  B)) p ( z d , w  j )
Re-estimate  j' wV

c( w, d )(1  p ( z d , w  B)) p ( z d , w  j ' )
- mixing weights  
m
c( w, d )(1  p ( z  B)) p ( z  j )
i 1 d Ci d ,w d ,w
- cluster LM p ( n 1)
(w |  j )
   c(w' , d )(1  p( z  B)) p( z  j )
m
w 'V i 1 d Ci d , w' d , w'
Sum over all docs Fractional counts contributing to

- using cluster j in generating d
(in multiple collections)
- generating w from cluster j
m = 1 if one collection
PLSA with Prior Knowledge
• There are different ways of choosing aspects (topics)

– Google = Google News + Google Map + Google scholar, …
– Google = Google US + Google France + Google China, …
• Users have some domain knowledge in mind, e.g.,

– We expect to see “retrieval models” as a topic in IR.
– We want to show the aspects of “history” and “statistics” for
Youtube
• A flexible way to incorporate such knowledge as priors of

PLSA model
• In Bayesian, it’s your “belief” on the topic distributions
14
Adding Prior
*  arg max p ( ) p ( Data |  )

Most likely 
Document d
warning 0.3
Topic 1 system 0.2.. d,1
1 “Generating” word w
in doc d in the collection
aid 0.1 2
Topic 2 donation 0.05 d,2 1 - B
support 0.02 ..
d, k W
… k
statistics 0.2
loss 0.1 B
Topic k dead 0.05 .. B
is 0.05
Background B the 0.04 Parameters:
a 0.03 .. B=noise-level (manually set)
’s and ’s are estimated with Maximum Likelihood
15
Adding Prior as Pseudo Counts
Observed Doc(s)
the 0.2
Known a 0.1
Background we 0.01
p(w | B) to 0.02
…
MAP
Unknown … Suppose, Estimator
topic model text =? we know
mining =?
p(w|1)=? association =? the identity
word =? of each
“Text mining” …
word ... Pseudo Doc
…
Unknown …
topic model information =?
p(w|2)=? retrieval =?
query =? Size = μ
document =? text
“information
…
retrieval”
16
mining
Maximum A Posterior (MAP) Estimation
 d( n, )j p ( n ) ( w |  j )
p( zd ,w  j ) 

k
j '1
 (n) (n)
d , j' p (w |  j ' )
B p ( w |  B )
p( zd ,w  B) 
B p( w |  B )  (1  B ) j 1  d( n, )j p ( n ) ( w |  j )
k
 d( n, j 1) 
 wV
c( w, d )(1  p ( z d , w  B )) p ( z d , w  j )
Pseudo counts of w from prior ’
j' wV
c( w, d )(1  p ( z d , w  B )) p ( z d , w  j ' )
 
m
c( w, d )(1  p ( z  B )) p ( z  j ) +p(w|’ )
i 1 d Ci d ,w d ,w
p ( n 1)
(w |  )  j
   c(w' , d )(1  p( z  B)) p( z  j ) +

j m
w 'V i 1 d Ci d , w' d , w'
Sum of all pseudo counts
What if =0? What if =+?
17
Basic Topic Model: LDA
The following slides about LDA are taken from Michael C. Mozer’s course lecture
http://www.cs.colorado.edu/~mozer/courses/ProbabilisticModels/
LDA: Motivation
– “Documents have no generative probabilistic semantics”
•i.e., document is just a symbol
– Model has many parameters
•linear in number of documents
•need heuristic methods to prevent overfitting
– Cannot generalize to new documents
2008 © ChengXiang Zhai China-US-France Summer School, Lotus Hill Inst., 2008
Unigram Model
N
p (w )   p ( wn )
n 1
Mixture of Unigrams
N
p (w )   p ( z ) p ( wn | z )
z n 1
Topic Model / Probabilistic LSI
p (d , wn )  p (d ) p ( wn | z ) p ( z | d )
z
•d is a localist representation of (trained) documents

•LDA provides a distributed representation
LDA
•Vocabulary of |V| words
•Document is a collection of words from vocabulary.
•N words in document
•w = (w1, ..., wN)
•Latent topics
•random variable z, with values 1, ..., k
•Like topic model, document is generated by sampling a

topic from a mixture and then sampling a word from a
mixture.
•But topic model assumes a fixed mixture of topics (multinomial
distribution) for each document.
•LDA assumes a random mixture of topics (Dirichlet distribution) for each
topic.
Generative Model
•“Plates” indicate looping structure

•Outer plate replicated for each document
•Inner plate replicated for each word
•Same conditional distributions apply for each replicate
•Document probability
Fancier Version
(ik1 i )  1
p (  )  k 1  k
 1 k 1
i 1 ( i )
Inference
p ( , z | w , ,  )  p ( , z , w | ,  )
p ( w | ,  )
N
p ( , z, w  ,  )  p (  ) p ( z n  ) p ( wn z n ,  )
n 1
 N  k
p (w  ,  )   p (  )  p ( z n  ) p ( wn z n ,  ) d 

 n 1 z n 
Inference
•In general, this formula is intractable:
 N  k
p (w  ,  )   p (  )  p ( z n  ) p ( wn z n ,  ) d 
 n 1 z n 
•Expanded version: 1 if wn is the j'th vocab word
(i  i )  k  i 1  N k V wnj 
p (w |  ,  )      i   ( i  ij ) d
i ( i )  i 1  n1 i 1 j 1 
Variational Approximation
•Computing log likelihood and introducing Jensen's inequality:
log(E[x]) >= E[log(x)]
•Find variational distribution q such that the above equation is

computable.
– q parameterized by γ and φn
– Maximize bound with respect to γ and φn to obtain best approximation
to p(w | α, β)
– Lead to variational EM algorithm
•Sampling algorithms (e.g., Gibbs sampling) are also common

Data Sets
C. Elegans Community abstracts

 5,225 abstracts
 28,414 unique terms
TREC AP corpus (subset)
 16,333 newswire articles
 23,075 unique terms
Held-out data – 10%

Removed terms
50 stop words, words appearing once
C. Elegans
Note: fold in hack for pLSI to allow it to handle novel documents.

Involves refitting p(z|dnew) parameters -> sort of a cheat
AP
Summary: PLSA vs. LDA
• LDA adds a Dirichlet distribution on top of PLSA to
regularize the model
• Estimation of LDA is more complicated than PLSA
• LDA is a generative model, while PLSA isn’t
• PLSA is more likely to over-fit the data than LDA
• Which one to use?
– If you need generalization capacity, LDA
– If you want to mine topics from a collection, PLSA may
be better (we want overfitting!)
Extension of PLSA:
Contextual Probabilistic Latent
Semantic Analysis (CPLSA)
A General Introduction to EM
Data: X (observed) + H(hidden) Parameter: 
“Incomplete” likelihood: L( )= log p(X| )

“Complete” likelihood: Lc( )= log p(X,H| )
EM tries to iteratively maximize the incomplete likelihood:
Starting with an initial guess (0),
1. E-step: compute the expectation of the complete likelihood

Q( ; ( n 1) )  E [ Lc ( ) | X ]   p( H  hi | X , ( n 1) ) log P( X , hi )
( n1)
hi
2. M-step: compute (n) by maximizing the Q-function

 ( n )  arg max Q( ; ( n 1) )  arg max  p( H  hi | X , ( n 1) ) log P( X , hi )
hi
Convergence Guarantee
Goal: maximizing “Incomplete” likelihood: L( )= log p(X| )
I.e., choosing (n), so that L((n))-L((n-1))0
Note that, since p(X,H| ) =p(H|X, ) P(X| ) , L() =Lc() -log p(H|X, )
L((n))-L((n-1)) = Lc((n))-Lc( (n-1))+log [p(H|X,  (n-1) )/p(H|X, (n))]
Taking expectation w.r.t. p(H|X, (n-1)),
L((n))-L((n-1)) = Q((n);  (n-1))-Q( (n-1);  (n-1)) + D(p(H|X,  (n-1))||p(H|X,  (n)))

Doesn’t contain H
EM chooses (n) to maximize Q KL-divergence, always non-negative
Therefore, L((n))  L((n-1))!
Another way of looking at EM
Likelihood p(X| )
L((n-1)) + Q(; (n-1)) -Q( (n-1);  (n-1) ) + D(p(H|X,  (n-1) )||p(H|X,  ))
L((n-1)) + Q(; (n-1)) -Q( (n-1);  (n-1) )
next guess
current
guess
Lower bound
(Q function)

E-step = computing the lower bound
M-step = maximizing the lower bound
Why Contextual PLSA?
Motivating Example:
Comparing Product Reviews
IBM Laptop APPLE Laptop DELL Laptop
Reviews Reviews Reviews
Common Themes “IBM” specific “APPLE” specific “DELL” specific
Battery Life Long, 4-3 hrs Medium, 3-2 hrs Short, 2-1 hrs
Hard disk Large, 80-100 GB Small, 5-10 GB Medium, 20-50 GB
Speed Slow, 100-200 Mhz Very Fast, 3-4 Ghz Moderate, 1-2 Ghz
Unsupervised discovery of common topics and their variations

Motivating Example:
Comparing News about Similar Topics
Vietnam War Afghan War Iraq War
Common Themes “Vietnam” specific “Afghan” specific “Iraq” specific
United nations … … …
Death of people … … …
… … … …
Unsupervised discovery of common topics and their variations

Motivating Example:
Discovering Topical Trends in Literature
Theme Strength
Time
1980 1990 1998 2003
TF-IDF Retrieval Language Model
IR Applications Text Categorization
Unsupervised discovery of topics and their temporal variations
Motivating Example:
Analyzing Spatial Topic Patterns
• How do blog writers in different states respond to

topics such as “oil price increase during Hurricane
Karina”?
• Unsupervised discovery of topics and their variations
in different locations
Motivating Example:
Sentiment Summary
Unsupervised/Semi-supervised discovery of topics and

different sentiments of the topics
Research Questions
• Can we model all these problems generally?
• Can we solve these problems with a unified
approach?
• How can we bring human into the loop?
Contextual Text Mining
• Given collections of text with contextual information (meta-
data)
• Discover themes/subtopics/topics (interesting word clusters)
• Compute variations of themes over contexts
• Applications:
– Summarizing search results
– Federation of text information
– Opinion analysis
– Social network analysis
– Business intelligence
– ..
Context Features of Text (Meta-data)
Weblog Article
communities
Author
source
Time Location
Author’s Occupation
Context = Partitioning of Text
papers written
1998 in 1998
1999 Papers about Web

…… ……
2005
2006 papers written by

authors in US
WWW SIGIR ACL KDD SIGMOD
Themes/Topics
Theme 1 government 0.3 [ Criticism of government response to the
hurricane primarily consisted of criticism of its
response 0.2..
response to the approach of the storm and its
donate 0.1
Theme 2 relief 0.05 aftermath, specifically in the delayed
help 0.02 .. response ] to the [ flooding of New Orleans. …
… 80% of the 1.3 million residents of the greater
city 0.2 New Orleans metropolitan area evacuated ] …
new 0.1 [ Over seventy countries pledged monetary
Theme k
orleans 0.05 .. donations or other assistance]. …
Is 0.05
Background B the 0.04 • Uses of themes:
a 0.03 .. – Summarize topics/subtopics
– Navigate in a document space
– Retrieve documents
– Segment documents
– …
View of Themes:
Context-Specific Version of Views
Context: After 1998 (Language models)
vector
space
TF-IDF
Okapi Theme 2: vector
retrieve LSI
Theme 1: feedback Rocchio
model retrieval
judge Feedback
weighting
relevance feedback
Retrieval documen expansion term
Model t query pseudo
query
language mixture
model model
smoothing estimate
query EM
generation feedback
pseudo
Context: Before 1998 (Traditional models)
Coverage of Themes:
Distribution over Themes
Oil Price
Government Criticism of government response to the
Response hurricane primarily consisted of criticism of its
Aid and response to … The total shut-in oil production
donation
Background from the Gulf of Mexico … approximately 24% of
the annual production and the shut-in gas
production … Over seventy countries pledged
Context: Texas monetary donations or other assistance. …
Oil Price
Government
Response
Aid and
• Theme coverage can depend on
context
donation
Background
Context: Louisiana
General Tasks of Contextual Text Mining
• Theme Extraction: Extract the global salient themes

– Common information shared over all contexts
• View Comparison: Compare a theme from different views
– Analyze the content variation of themes over contexts
• Coverage Comparison: Compare the theme coverage of different
contexts
– Reveal how closely a theme is associated to a context
• Others:
– Causal analysis
– Correlation analysis
A General Solution: CPLSA
• CPLAS = Contextual Probabilistic Latent Semantic Analysis
• An extension of PLSA model ([Hofmann 99]) by
– Introducing context variables
– Modeling views of topics
– Modeling coverage variations of topics
• Process of contextual text mining

– Instantiation of CPLSA (context, views, coverage)
– Fit the model to text data (EM algorithm)
– Compute probabilistic topic patterns
“Generation” Process of CPLSA
View1 View2 View3 Choose a theme
Themes Draw a word from i
Criticism of government
government 0.3 response togovernment
the hurricane
government
response 0.2.. Document
primarily consisted of
response
criticism of its response
context:
to … The total shut-in oil
donate 0.1 Time = from
production July the
2005
Gulf
relief 0.05 Location
of Mexico …donate= Texas
approximately
donation 24% ofAuthor = xxx
help 0.02 .. aid
the annual
help
Occup. = and
production Sociologist
the shut-
in gas
Ageproduction
Group = … 45+Over
city 0.2
seventy countries pledged
New new 0.1 …Orleans
monetary donations or
Orleans orleans 0.05 .. new …
other assistance.
Texas July sociolo

2005 gist
Choose a view
Theme
coverages …… Choose a
: Coverage
Texas July 2005 document
Probabilistic Model
• To generate a document D with context feature set C:
– Choose a view vi according to the view distribution p (vi | D, C )
– Choose a coverage кj according to the coverage distribution
p ( j | D, C )
– Choose a theme  il according to the coverage кj

– Generate a word using  il
– The likelihood of the document collection is:
n m k
log p(D)    c(w, D) log( p(v | D, C ) p(
( D ,C )D wV i 1
i
j 1
j | D, C ) p(l |  j ) p( w |  il ))
l 1
Parameter Estimation: EM Algorithm
• Interesting patterns:
 c( w, D) j 1 l 1 p ( z w,i , j ,l  1)
m k
– Theme content p ( t 1)

(v | D, C ) 
wV
  c( w, D) j '1 l '1 p ( z w,i ', j ',l '  1)

i n m k
variation for each i '1 wV
p ( t ) (vi | D, C ) p (t ) ( j | D, C ) p ( t ) (l |  j ) p ( t ) ( w |  il )
view: p ( z w,i , j ,l  1) 
 p ( t ) (vi ' | D, C ) j '1 p (t ) ( j ' | D, C )l '1 p (t ) (l ' |  j ' ) p (t ) ( w |  i 'l ' )
p( w |  il )
n m k
i '1
 c( w, D)i 1 l 1 p( z w,i , j ,l  1)
n k
– Theme strength p (t 1) ( j | D, C )  wV
  c( w, D)i '1 l '1 p ( z w,i ', j ',l '  1)

m n k
variation for each j '1 wV
context   c( w, D)i 1 p ( z w,i , j ,l  1)

n
p (l |  j )
( D ,C )D wV
p ( t 1)
(l |  j )
   c( w, D)i '1 p ( z w,i ', j ,l '  1)
l n
l '1 ( D ,C )D w 'V
• Prior from a user can p ( t 1) ( w |  il )

 ( D ,C )D
c( w, D) j 1 p ( z w,i , j ,l  1)
m
be incorporated using   w 'V ( D ,C )D

c( w' , D) j '1 p ( z w',i , j ',l  1)
m
MAP estimation
Regularization of the Model
• Why?
– Generality high complexity (inefficient, multiple local maxima)
– Real applications have domain constraints/knowledge
• Two useful simplifications:

– Fixed-Coverage: Only analyze the content variation of themes (e.g.,
author-topic analysis, cross-collection comparative analysis )
– Fixed-View: Only analyze the coverage variation of themes (e.g.,
spatiotemporal theme analysis)
• In general
– Impose priors on model parameters
– Support the whole spectrum from unsupervised to supervised
learning
Interpretation of Topics
term 0.1599 term 0.1599 term 0.1599
Statistical relevance
weight
0.0752
0.0660
relevance
weight
0.0752
0.0660
relevance
weight
0.0752
0.0660
topic models feedback 0.0372
independence 0.0311
feedback 0.0372
independence 0.0311
feedback 0.0372
independence 0.0311
model 0.0310 model 0.0310 model 0.0310
frequent 0.0233 frequent 0.0233 frequent 0.0233
probabilistic 0.0188 probabilistic 0.0188 probabilistic 0.0188
document 0.0173 document 0.0173 document 0.0173
… … …
Multinomial topic models
Collection (Context) Coverage; Discrimination

Relevance Score Re-ranking
clustering algorithm;
distance measure;
database system, clustering algorithm, …
NLP Chunker r tree, functional dependency, iceberg
Ngram stat. cube, concurrency control, Ranked List
index structure … of Labels
Candidate label pool
Relevance: the Zero-Order Score
• Intuition: prefer phrases covering high probability words
Clustering
Good Label (l1):

dimensional “clustering
p (l |  ) algorithm”
p (l )
Latent algorithm …
Topic 
birch
shape
Bad Label (l2):
… “body shape”
p(w| body
)
Relevance: the First-Order Score
• Intuition: prefer phrases with similar context (distribution)
Clustering
Clustering Clustering
dimension
dimension dimension
Good Label (l1): Bad Label (l2):
“clustering “hash join”
Topic  …
partition partition algorithm”
algorithm
algorithm algorithm
… … join
Score (l,  )
C: SIGMOD
hash hash Proceedings
hash   p ( w |  ) PMI ( w, l | C )
P(w|) P(w|l1) P(w|l2) w
D(||l1) < D(||l2)
Sample Results
• Comparative text mining
• Spatiotemporal pattern mining
• Sentiment summary
• Event impact analysis
• Temporal author-topic analysis
Comparing News Articles
Iraq War (30 articles) vs. Afghan War (26 articles)
The common theme indicates that “United Nations” is involved in both wars
Cluster 1 Cluster 2 Cluster 3

Common united 0.042 killed 0.035 …
nations 0.04 month 0.032
Theme … deaths 0.023
…
Iraq n 0.03 troops 0.016 …
Weapons 0.024 hoon 0.015
Theme Inspections 0.023 sanches 0.012
… …
Northern 0.04 taleban 0.026 …
alliance 0.04 rumsfeld 0.02
Afghan kabul 0.03 hotel 0.012
taleban 0.025 front 0.011
Theme aid 0.02 …
…
Collection-specific themes indicate different roles of “United Nations” in the two wars
Comparing Laptop Reviews
Top words serve as “labels” for common themes

(e.g., [sound, speakers], [battery, hours], [cd,drive])
These word distributions can be used to segment text and

add hyperlinks between documents
Spatiotemporal Patterns in Blog Articles
• Query= “Hurricane Katrina”
• Topics in the results:
Government Response New Orleans Oil Price Praying and Blessing Aid and Donation Personal
bush 0.071 city 0.063 price 0.077 god 0.141 donate 0.120 i 0.405
president 0.061 orleans 0.054 oil 0.064 pray 0.047 relief 0.076 my 0.116
federal 0.051 new 0.034 gas 0.045 prayer 0.041 red 0.070 me 0.060
government 0.047 louisiana 0.023 increase 0.020 love 0.030 cross 0.065 am 0.029
fema 0.047 flood 0.022 product 0.020 life 0.025 help 0.050 think 0.015
administrate 0.023 evacuate 0.021 fuel 0.018 bless 0.025 victim 0.036 feel 0.012
response 0.020 storm 0.017 company 0.018 lord 0.017 organize 0.022 know 0.011
brown 0.019 resident 0.016 energy 0.017 jesus 0.016 effort 0.020 something 0.007
blame 0.017 center 0.016 market 0.016 will 0.013 fund 0.019 guess 0.007
governor 0.014 rescue 0.012 gasoline 0.012 faith 0.012 volunteer 0.019 myself 0.006
• Spatiotemporal patterns
Theme Life Cycles for Hurricane
Katrina
Oil Price
price 0.0772
oil 0.0643
New Orleans gas 0.0454
increase 0.0210
product 0.0203
fuel 0.0188
company 0.0182
…
city 0.0634
orleans 0.0541
new 0.0342
louisiana 0.0235
flood 0.0227
evacuate 0.0211
storm 0.0177
…
Theme Snapshots for Hurricane
Katrina
Week2: The discussion moves towards the north and west
Week1: The theme is the strongest along the Gulf of Mexico Week3: The theme distributes more uniformly over the states
Week4: The theme is again strong along the east coast and the Gulf of Mexico
Week5: The theme fades out in most states
Theme Life Cycles: KDD
gene 0.0173
0.02 expressions 0.0096
Biology Data
0.018
probability 0.0081
W eb Information
microarray 0.0038
0.016 Time Series
Normalized Strength of Theme
Classification
…
0.014
Association Rule marketing 0.0087
0.012
Clustering customer 0.0086
0.01 Bussiness model 0.0079
0.008 business 0.0048
0.006
…
0.004
rules 0.0142
association 0.0064
0.002
support 0.0053
0 …
1999 2000 2001 2002 2003 2004
Time (year)
Global Themes life cycles of KDD Abstracts
Theme Evolution Graph: KDD
1999 2000 2001 2002 2003 2004 T
web 0.009
SVM 0.007 classifica – mixture 0.005
criteria 0.007 tion 0.007 random 0.006
classifica – features0.006 cluster 0.006
tion 0.006 topic 0.005 clustering 0.005
linear 0.005 … variables 0.005
… … topic 0.010
mixture 0.008
decision 0.006 LDA 0.006
tree 0.006 … semantic
classifier 0.005 Classifica 0.005
class 0.005 - tion 0.015 Informa …
Bayes 0.005 text 0.013 - tion 0.012
… unlabeled 0.012 web 0.010
document 0.008 social 0.008
labeled 0.008 retrieval 0.007
learning 0.007 distance 0.005
…
… … networks 0.004
…
…
Blog Sentiment Summary
(query=“Da Vinci Code”)
Neutral Positive Negative
... Ron Howards selection of Tom Hanks stars in the But the movie might get
Tom Hanks to play Robert movie,who can be mad at delayed, and even killed off if
Langdon. that? he loses.
Directed by: Ron Howard Tom Hanks, who is my protesting ... will lose your faith
Facet 1:
Writing credits: Akiva favorite movie star act by ... watching the movie.
Movie
Goldsman ... the leading role.
After watching the movie I Anybody is interested in ... so sick of people making
went online and some it? such a big deal about a
research on ... FICTION book and movie.
I remembered when i first Awesome book. ... so sick of people making

read the book, I finished the such a big deal about a
book in two days. FICTION book and movie.
Facet 2:
Book I’m reading “Da Vinci Code” So still a good book to This controversy book cause
now. past time. lots conflict in west society.
…
Results: Sentiment Dynamics
Facet: the book “ the da vinci Facet: religious beliefs ( Bursts

code”. ( Bursts during the during the movie, Neg >
movie, Pos > Neg ) Pos )
Event Impact Analysis: IR Research
xml 0.0678
vector 0.0514 email 0.0197
concept 0.0298 model 0.0191
extend 0.0297 collect 0.0187
Theme: retrieval model 0.0291 judgment 0.0102 SIGIR papers
models space 0.0236 rank 0.0097
boolean 0.0151 subtopic 0.0079
function 0.0123 …
term 0.1599 feedback 0.0077
relevance 0.0752 … Publication of the paper “A language
weight 0.0660 1992 modeling approach to information retrieval”
feedback 0.0372
independence 0.0311
year
model 0.0310 Starting of the TREC conferences
1998
frequent 0.0233 model 0.1687
probabilistic 0.0188 language 0.0753
probabilist 0.0778 estimate 0.0520
document 0.0173
model 0.0432 parameter 0.0281
…
logic 0.0404 distribution 0.0268
ir 0.0338 probable 0.0205
boolean 0.0281 smooth 0.0198
algebra 0.0200 markov 0.0137
estimate 0.0119 likelihood 0.0059
weight 0.0111 …
…
Temporal-Author-Topic Analysis
close 0.0805 index 0.0440
project 0.0444 pattern 0.0720 graph 0.0343
itemset 0.0433 sequential 0.0462 web 0.0307
intertransaction 0.0397 min_support 0.0353 gspan 0.0273
support 0.0264 threshold 0.0207 substructure 0.0201
associate 0.0258 top-k 0.0176 gindex 0.0164
frequent 0.0181 fp-tree 0.0102 bide 0.0115
closet 0.0176 Author … xml 0.0109
prefixspan 0.0170 Jiawei Han …
…
Author A
Global theme:
frequent patterns time
2000
pattern 0.1107 Author B research 0.0551
frequent 0.0406 Rakesh Agrawal next 0.0308
frequent-pattern 0.039 transaction 0.0308
sequential 0.0360 panel 0.0275
pattern-growth 0.0203 technical 0.0275
constraint 0.0184 article 0.0258
push 0.0138 revolution 0.0154
… innovate 0.0154
…
Modeling Topical Communities
(Mei et al. 08)
Community 1:
Information Retrieval
Community 2:
Data Mining
Community 3:
Machine Learning
71
Other Extensions (LDA Extensions)
• Many extensions of LDA, mostly done by David

Blei, Andrew McCallum and their co-authors
• Some examples:
– Hierarchical topic models [Blei et al. 03]
– Modeling annotated data [Blei & Jordan 03]
– Dynamic topic models [Blei & Lafferty 06]
– Pachinko allocation [Li & McCallum 06])
• Also, some specific context extension of PLSA,
e.g., author-topic model [Steyvers et al. 04]
Future Research Directions
• Topic models for text mining
– Evaluation of topic models
– Improve the efficiency of estimation and inferences
– Incorporate linguistic knowledge
– Applications in new domains and for new tasks
• Text mining in general

– Combination of NLP-style and DM-style mining algorithms
– Integrated mining of text (unstructured) and unstructured data
(e.g., Text OLAP)
– Interactive mining:
• Incorporate user constraints and support iterative mining
• Design and implement mining languages
Lecture 5: Key Points
• Topic models coupled with topic labeling are quite
useful for extracting and modeling subtopics in text
• Adding context variables significantly increases a
topic model’s capacity of performing text mining
– Enable interpretation of topics in context
– Accommodate variation analysis and correlation
analysis of topics over context
• User’s preferences and domain knowledge can be

added as prior or soft constraint
Readings
• PLSA:
– http://www.cs.brown.edu/~th/papers/Hofmann-UAI99.pdf
• LDA:
– http://www.cs.princeton.edu/~blei/papers/BleiNgJordan2003.pdf
– Many recent extensions, mostly done by David Blei and Andrew
McCallums
• CPLSA:
– http://sifaka.cs.uiuc.edu/czhai/pub/kdd06-mix.pdf
– http://sifaka.cs.uiuc.edu/czhai/pub/www08-net.pdf
Discussion
• Topic models for mining multimedia data
– Simultaneous modeling of text and images
• Cross-media analysis
– Text provides context to analyze images and vice
versa
Course Summary
Integrated Multimedia Data AnalysisScope of the course
-Mutual reinforcement (e.g., text images)
Information Retrieval
-Simultaneous mining of text + images +video…
1. Evaluation
Multimedia Data Retrieval models/framework
Text Data
2. User modeling Evaluation Feedback
3. Ranking Contextual topic models
4. Computer Vision
Learning with little Natural Language Processing
supervision
Machine Learning
Looking forward to collaborations…

Statistics
Thank You!

Probabilistic Topic Models

Uploaded by

Document Information

Original Title

Copyright

Available Formats

Share this document

Share or Embed Document

Sharing Options

Did you find this document useful?

Is this content inappropriate?

Copyright:

Available Formats

Probabilistic Topic Models

Uploaded by

Copyright:

Available Formats

Probabilistic Topic Models for

“Another way to view text data mining is as a process of

(Slide from Rebecca Hwa’s “Intro to Text Mining”)

• Data Mining View: Explore patterns in textual data

• Natural Language Processing View: Make

• Direct applications: Go beyond search to find knowledge

– Latent Dirichlet Allocation (LDA) [Blei et al. 02]

Query = “Hurricane Katrina”

• Mix k multinomial distributions to generate a document

P(w| θ) Topic 1 Topic 2

Re-estimate  j' wV

Sum over all docs Fractional counts contributing to

• There are different ways of choosing aspects (topics)

• Users have some domain knowledge in mind, e.g.,

• A flexible way to incorporate such knowledge as priors of

   c(w' , d )(1  p( z  B)) p( z  j ) +

Sum of all pseudo counts

What if =0? What if =+?

•d is a localist representation of (trained) documents

•Like topic model, document is generated by sampling a

•“Plates” indicate looping structure

•Expanded version: 1 if wn is the j'th vocab word

•Find variational distribution q such that the above equation is

•Sampling algorithms (e.g., Gibbs sampling) are also common

C. Elegans Community abstracts

Held-out data – 10%

Note: fold in hack for pLSI to allow it to handle novel documents.

Data: X (observed) + H(hidden) Parameter: 

“Incomplete” likelihood: L( )= log p(X| )

Starting with an initial guess (0),

1. E-step: compute the expectation of the complete likelihood

2. M-step: compute (n) by maximizing the Q-function

Taking expectation w.r.t. p(H|X, (n-1)),

L((n))-L((n-1)) = Q((n);  (n-1))-Q( (n-1);  (n-1)) + D(p(H|X,  (n-1))||p(H|X,  (n)))

EM chooses (n) to maximize Q KL-divergence, always non-negative

Therefore, L((n))  L((n-1))!

L((n-1)) + Q(; (n-1)) -Q( (n-1);  (n-1) )

Common Themes “IBM” specific “APPLE” specific “DELL” specific

Hard disk Large, 80-100 GB Small, 5-10 GB Medium, 20-50 GB

Unsupervised discovery of common topics and their variations

Common Themes “Vietnam” specific “Afghan” specific “Iraq” specific

Unsupervised discovery of common topics and their variations

Unsupervised discovery of topics and their temporal variations

• How do blog writers in different states respond to

Unsupervised/Semi-supervised discovery of topics and

1999 Papers about Web

2006 papers written by

WWW SIGIR ACL KDD SIGMOD

Context: Before 1998 (Traditional models)

• Theme Extraction: Extract the global salient themes

• Process of contextual text mining

Texas July sociolo

– Choose a theme  il according to the coverage кj

– Theme content p ( t 1)

  c( w, D) j '1 l '1 p ( z w,i ', j ',l '  1)

variation for each i '1 wV

– Theme strength p (t 1) ( j | D, C )  wV

  c( w, D)i '1 l '1 p ( z w,i ', j ',l '  1)

variation for each j '1 wV

context   c( w, D)i 1 p ( z w,i , j ,l  1)

• Prior from a user can p ( t 1) ( w |  il )

be incorporated using   w 'V ( D ,C )D

• Two useful simplifications: