» Turing Test

Eli M. Dow <emdow@us.ibm.com:>

 Overview:
» Turing Test Overview
» Watson and the Test
» Questions
© 2011 IBM Corporation
 What is the Turing Test?

» About Dr. Turing

» The Imitation Game
» The Turing Test
© 2011 IBM Corporation
In order to determine if Watson
might pass the Turing Test, we
should first define what it is...

© 2011 IBM Corporation

The test was first proposed by the
very famous computer scientist,
Alan Turing Ph.D.

© 2011 IBM Corporation

•
© 2011 IBM Corporation
 Turing's Contributions:


Turing Machine Definition


The Church-Turing Thesis


The Halting Problem


The Turing Test
© 2011 IBM Corporation
 Turing Machine Definition:

- A Turing machine is a theoretical

device that manipulates symbols on a
strip of tape according to a table of
rules. This computer can provably
compute anything that any modern
computer can do with a sufficiently
fast tape reader.
© 2011 IBM Corporation
 The Church-Turing Thesis

Stated Informally – thesis states that if

some method (algorithm) exists to carry
out a calculation, then the same
calculation can also be carried out by a
Turing machine (as well as by a
recursively-definable function, and by a λ-
function).
© 2011 IBM Corporation
 The Halting Problem

- Given a description of a computer

program, decide whether the program
finishes running or continues to run
forever. This is equivalent to the problem
of deciding, given a program and an input,
whether the program will eventually halt
when run with that input, or will run
forever.
© 2011 IBM Corporation
 The Turing Test
During his later years, Turing examined
problems outside of pure mathematics and
moved on to more abstract research.

Among that work, he addressed the problem of

artificial intelligence and proposed an
experiment in a Published in a Paper Entitled:
"Computing machinery and intelligence"
(Mind, October 1950)
© 2011 IBM Corporation
This proposed experiment became known as
the Turing test.

It was the first legitimate attempt to define a

standard for a machine to be called
"intelligent".

© 2011 IBM Corporation

Idea - A computer could be said to "think" if a
human interrogator could not tell it apart,
through conversation, from a human being.

© 2011 IBM Corporation

Provocative Indeed

© 2011 IBM Corporation

Forward Thinking In the Age of the Univac...

© 2011 IBM Corporation

“Instead of trying to produce a programme to
simulate the adult mind, why not rather try to
produce one which simulates the child's? If this
were then subjected to an appropriate course
of education one would obtain the adult brain.”
- ALAN TURING, "Computing Machinery and
Intelligence”

© 2011 IBM Corporation

"The cocktail party has a simple function in modern society.
It's basic purpose is to pay off social debts." -

© 2011 IBM Corporation

The Turing Imitation Game

• 1 Host
• 2 Contestants
• Fabulous Cash Prizes*
© 2011 IBM Corporation
 The Imitation Game:
We need a Judge/Host

© 2011 IBM Corporation

 The Imitation Game:
We need a Judge/Host

© 2011 IBM Corporation

The Imitation Game:
Male Contestant Y

© 2011 IBM Corporation

The Imitation Game:
Female Contestant X

© 2011 IBM Corporation

 Fabulous Cash Prizes*!

[*] The IBM lawyers would like me to state that there is no actual
fabulous cash prize for winning the imitation game or attending this
talk.

© 2011 IBM Corporation

© 2011 IBM Corporation
 The Curtain

The interviewer asks questions of X and Y through an intermediary (so that

no auditory or vocal judgment can be made on the questioned).

© 2011 IBM Corporation

© 2011 IBM Corporation
The Imitation Game is not the same as what is now known as the Turing
Test. The revised version of the Turing Test sets Y as a computer and X as
a woman. Again, A and B must answer the interviewer’s as if they were
both women.

© 2011 IBM Corporation

 The 3rd & Final modification to the Imitation Game (1952)

In this version a jury asks questions of a person and a computer through a

teletype interface, and the role of the computer attempts to convince a
significant proportion of the jury believe that said machine is really a human.
© 2011 IBM Corporation
Turing believed that by approximately the
year 2000:

“it will be possible to programme

computers, with a storage capacity of
about 109, to make them play the
imitation game so well that an average
interrogator will not have more than 70
percent chance of making the right
identification after five minutes of
questioning.”
© 2011 IBM Corporation
 The Turing Test is the
Standard Litmus Test for
referring to a machine as
“Intelligent”

© 2011 IBM Corporation

•Watson and the Turing Test

» Shortcomings of the
Turing Test Definition
» How Watson Relates
to the Turing Test
© 2011 IBM Corporation
“There are no simple language tricks
that would enable a computer to
pass a well-designed Turing test. A
computer would need to actually
master human levels of
understanding to pass this
threshold.” - Ray Kurzweil

© 2011 IBM Corporation

J: How do you define life?
A: Life is the opposite of death.

J: So death is the absence of life?

A: Take it easy. I try not to think about such things.

J: Why not?
A: Ambiguous.

J: Yes!
J: You seem fun.
A: You think I am fun.
A: : -) Awesome.

J: How old are you?

A: I am 9 years old.

J: Nine? I don't believe you :).

A: : -) I don't have any reason to deceive you.

J: Yes, you do, you're a woman.

© 2011 IBM Corporation
Undefined Questions About the Turing Test:


Does all of human knowledge exist in language?

How qualified does the human judge need to be?

How do you interact with the participants?

How long should the sessions be?

What does a significant portion of the Jury mean?

© 2011 IBM Corporation

 Does all of human knowledge
exist in language?

Is mastery over linguistics

sufficient to describe intelligence?

© 2011 IBM Corporation

How qualified does the judge
need to be?

How many judges should there

be?

What constitutes a significant

portion of convinced judges?
© 2011 IBM Corporation
How does a judge interact
with the participants?

© 2011 IBM Corporation

 What about a Jeopardy variation of the
Imitation Game?

Rather than have a computer attempt to

convince a substantial portion of a panel of
judges that it is human by answering any
question conversationally, we can restrict the
questions have to be similar to what you would
expect on the quiz TV show Jeopardy!
© 2011 IBM Corporation
Could Watson Pass our
Jeopardy Imitation
Game?

© 2011 IBM Corporation

Does Watson pass the
Turing Test?

© 2011 IBM Corporation

© 2011 IBM Corporation
© 2011 IBM Corporation
Concluding Remarks

© 2011 IBM Corporation

Questions ?

© 2011 IBM Corporation

Thank You
 IBM Watson:
Watson and the Turing Test
Eli M. Dow
<emdow@us.ibm.com>

© 2011 IBM Corporation