Professional Documents
Culture Documents
MSIM 111 Session 5 (IBM Watson by Armen Pischdotchian)
MSIM 111 Session 5 (IBM Watson by Armen Pischdotchian)
Agenda
Linear Regression
Logistcal Regression
NLP terminology
10
11
12
Copyright 2010, Association for the Advancement of Artificial Intelligence. All rights reserved. ISSN 0738-4602
13
14
15
ral
Isaac Newton
Wilhelm Tempel
HMS Paramour
[0.12 0
Christiaan Huygens
Halleys Comet
[0.33 0
Edmond Halley
Pink Panther
Evidence
Retrieval
Models
Models
Models
Models
Models
Models
2.0 0.40]
Peter Sellers
16
Te
mp
o
Le
xic
Primary
Search
Ta
xo
no
mi
c
Sp
ati
al
Question
Analysis
al
Related Content
(Structured & Unstructured)
6.3 0.83]
1)
2)
3)
Merging &
Ranking
Evidence
Scoring
2015 IBM Corporaton
Primary
Search
Candidate
Answer
Generation
Answer Contextual
Contextual
Answer Answer
Scoring
Contextual
Answer
AnswerScoring
Scoring
Answer
Scoring Scoring
Scoring
Evidence
Retrieval
Trained
Models
Question
Question
Analysis
Search
Scoring
Scoring
Scoring
Final
Merging
Ranking
Answer, Confidence,
Evidence
17
18
19
Primary
Search
Question
Question
Analysis
20
Search
Primary
Search
Question
Question
Analysis
21
Search
Primary
Search
Candidate
Answer
Generation
Questio
n
Question
Analysis
22
Search
23
Hypotheses
Textual
Alignment
Term and
nGram
Matching
...
Logical
Form
Analysis
Evidence
Features
Question
Question
/Topic
Analysis
Hypothesis
Generation
Final
Merging
& Ranking
Hypothesis &
Evidence Scoring
Trained
Models
Answer,
Confidence
Evidence
24
AnswerIdf scorer
Context Independent scorer
Uses concept referred to as Inverse Document Frequency
Rato of total documents versus documents containing target
text
Target text = candidate answer text
Large corpus (e.g., Wikipedia)
Lucene formula
Log scale
10,000 documents
Answer text appears in only 10 documents
Log (10,000 / 10) = Log (1,000) = 3
25
26
Question
Analysis
27
Search
Scoring
Scoring
Contextual
Contextual
Answer
Contextual
Answer
Scoring
Answer
Scoring
Scoring
Barack Obama .95
George W. Bush .80
Harvard Law School .05
Illinois.10
Scoring
Primary
Search
Candidate
Answer
Generation
Answer Contextual
Contextual
Answer Answer
Scoring
Contextual
Answer
AnswerScoring
Scoring
Answer
Scoring Scoring
Scoring
Trained
Models
Questio
n
Question
Analysis
Search
Scoring
Scoring
Scoring
Final
Merging
Ranking
Answer, Confidence,
Evidence
28
29
30
Answer
Scoring
Contextual Answer
Scoring
Confidence
Evidence
Retrieval
Barack Obama
0.90
0.90
George W. Bush
0.90
0.80
.65
0.10
0.05
.05
Illinois
0.15
0.10
.10
.95
31
32
33