Computer Science 4

Theory of Computation

Dr. Bernhard Putz

UBT 2008

1
 Why Study Automata Theory?
Finite automata are a useful model for
important kinds of hardware and software:

• Software for designing and checking digital

circuits.
• Lexical analyzer of compilers.
• Finding words and patterns in large bodies of
text, e.g. in web pages.
• Verification of systems with finite number of
states, e.g. communication protocols.
2
Why Study Automata Theory? (2)
The study of Finite Automata and Formal
Languages are intimately connected.
Methods for specifying formal languages
are very important in many areas of CS,
e.g.:

• Context Free Grammars are very useful when

designing software that processes data with
recursive structure, like the parser in a compiler.
• Regular Expressions are very useful for
specifying lexical aspects of programming
languages and search patterns. 3
Why Study Automata Theory? (3)
Automata are essential for the study of
the limits of computation. Two issues:

• What can a computer do at all?

(Decidability)

• What can a computer do efficiently?

(Intractability)
4
 5
...

Theoretical Computer Science

Computer-Aided
Automata Theory, Formal Languages,

Verification
Computability, Complexity …
Quantum
computing
Applications

Supervisory
control
Pattern
recognition
circuits
Comm.
protocols
Prog. languages
Compiler
 Aims of the Course
• To familiarize you with key Computer Science
concepts in central areas like
- Automata Theory
- Formal Languages
- Models of Computation
- Complexity Theory
• To equip you with tools with wide applicability
in the fields of CS and EE, e.g. for
- Complier Construction
- Text Processing
- XML
6
 Fundamental Theme
• What are the capabilities and
limitations of computers and computer
programs?
– What can we do with
computers/programs?
– Are there things we cannot do with
computers/programs?

7
 Studying the Theme
• How do we prove something CAN be done
by SOME program?

• How do we prove something CANNOT be

done by ANY program?

8
Example: The Halting Problem (1)
Consider the following program. Does it
terminate for all values of n  1?

while (n > 1) {
if even(n) {
n = n / 2;
} else {
n = n * 3 + 1;
}
} 9
Example: The Halting Problem (2)
Not as easy to answer as it might first seem.
Say we start with n = 7, for example:
7, 22, 11, 34, 17, 52, 26, 13, 40, 20, 10, 5,
16, 8, 4, 2, 1

In fact, for all numbers that have been tried

(a lot!), it does terminate . . .

. . . but in general?
10
Example: The Halting Problem (3)
Then the following important undecidability
result should perhaps not come as a total
surprise:

It is impossible to write a program that

decides if another, arbitrary, program
terminates (halts) or not.

What might be surprising is that it is possible

to prove such a result. This was first done
by the British mathematician Alan Turing.
11
 Our focus

Computability
Automata

Complexity

12
 Topics
1.  Finite
automata, Regular languages, Regular
grammars: deterministic vs. nondeterministic,
one-way vs. two-way finite automata,
minimization, pumping lemma for regular sets,
closure properties.
2.  Pushdown automata, Context-free languages,
Context-free grammars: deterministic vs.
nondeterministic, one-way vs. two-way PDAs,
reversal bounded PDAs, linear grammars, counter
machines, pumping lemma for CFLs, Chomsky
normal form, Greibach normal form, closure
properties. 13
 Topics (cont’d)
3. Linearbounded automata, Context-
sensitive languages, Context-sensitive
grammars.

4. Turing machines, Recursively

enumerable sets, Type 0 grammars:
variants of Turing machines, halting
problem, undecidability, Post
correspondence problem, valid and
invalid computations of TMs. 14
 Topics (cont’d)
5. Basic recursive function theory
6. Basic complexity theory: Various
resource bounded complexity classes,
including NLOGSPACE, P, NP,
PSPACE, EXPTIME, and many more.
reducibility, completeness.
7. Advanced topics: Tree Automata,
quantum automata, probabilistic
automata, interactive proof systems,
oracle computations, cryptography. 15
 Languages
The terms language and word are used in
a strict technical sense in this course:
A language is a set of words.
A word is a sequence (or string) of
symbols.
 (or ) denotes the empty word, the
sequence of zero symbols.

16
 Symbols and Alphabets
• What is a symbol, then?
• Anything, but it has to come from an
alphabet which is a finite set.
• A common (and important) instance is
= {0, 1}.
 , the empty word, is never an symbol
of an alphabet.

17
Computation

CPU memory

18
temporary memory

input memory
CPU
output memory

Program memory

19
 3
Example: f ( x)  x

temporary memory

input memory
CPU
output memory
Program memory
compute xx
2
compute x x 20
 3
f ( x)  x

temporary memory
input memory
x2
CPU
output memory
Program memory
compute xx
2
compute x x 21
 3
temporary memory f ( x)  x
z  2*2  4
f ( x)  z * 2  8
input memory
x2
CPU
output memory
Program memory
compute xx
2
compute x x 22
 3
temporary memory f ( x)  x
z  2*2  4
f ( x)  z * 2  8
input memory
x2
CPU
f ( x)  8
Program memory output memory
compute xx
2
compute x x 23
 Automaton
temporary memory

Automaton
input memory
CPU
output memory

Program memory

24
 Different Kinds of Automata
Automata are distinguished by the temporary memory

• Finite Automata: no temporary memory

• Pushdown Automata: stack

• Turing Machines: random access memory

25
 Finite Automaton

temporary memory

input memory
Finite
Automaton
output memory

Example: Vending Machines

(small computing power) 26
 Pushdown Automaton
Stack Push, Pop

input memory
Pushdown
Automaton
output memory

Example: Compilers for Programming Languages

(medium computing power) 27
 Turing Machine

Random Access Memory

input memory
Turing
Machine
output memory

Examples: Any Algorithm

(highest computing power) 28
 Power of Automata

Finite Pushdown Turing

Automata Automata Machine

Less power More power

Solve more
computational problems
29
Mathematical Preliminaries

30
 Mathematical Preliminaries

• Sets
• Functions
• Relations
• Graphs
• Proof Techniques

31
 SETS
A set is a collection of elements
A  {1, 2, 3}
B  {train, bus, bicycle, airplane}

We write
1 A
ship  B
32
 Set Representations

C = { a, b, c, d, e, f, g, h, i, j, k }

C = { a, b, …, k } finite set

S = { 2, 4, 6, … } infinite set

S = { j : j > 0, and j = 2k for some k>0 }

S = { j : j is nonnegative and even }

33
A = { 1, 2, 3, 4, 5 }

U
6 A
2 3 8
1
7 4 5
9
10

Universal Set: all possible elements

U = { 1 , … , 10 }
34
 Set Operations
A = { 1, 2, 3 } B = { 2, 3, 4, 5}
A B
• Union
A U B = { 1, 2, 3, 4, 5 }
• Intersection
U
A B = { 2, 3 }
• Difference
A-B={1}
A-B
B - A = { 4, 5 }
35
• Complement
Universal set = {1, …, 7}
A = { 1, 2, 3 } A = { 4, 5, 6, 7}

4
A
A 3 6
1
2
5 7

A=A
36
{ even integers } = { odd integers }

Integers

1 odd
even 5
2 6
0
4
3 7

37
DeMorgan’s Laws

U
AUB=A B

U
A B=AUB

38
 Empty, Null Set:
={}

SU =S
U
S = = Universal Set

S- =S

-S=

39
 Subset
A = { 1, 2, 3} B = { 1, 2, 3, 4, 5 }
A B

U
Proper Subset: A B

U
B
A

40
 Disjoint Sets
A = { 1, 2, 3 } B = { 5, 6}

U
A B=

A B

41
 Set Cardinality
• For finite sets
A = { 2, 5, 7 }

|A| = 3

42
 Powersets
A powerset is a set of sets

S = { a, b, c }

Powerset of S = the set of all the subsets of S

2S = { , {a}, {b}, {c}, {a, b}, {a, c}, {b, c}, {a, b, c} }

Observation: | 2S | = 2|S| ( 8 = 23 )

43
 Cartesian Product
A = { 2, 4 } B = { 2, 3, 5 }

A X B = { (2, 2), (2, 3), (2, 5), ( 4, 2), (4, 3), (4, 5) }

|A X B| = |A| |B|

Generalizes to more than two sets

AXBX…XZ 44
 FUNCTIONS
domain range
A B
f(1) = a a
1
2 b
3 c

f : A -> B
If A = domain
then f is a total function
otherwise f is a partial function
45
 RELATIONS
R = {(x1, y1), (x2, y2), (x3, y3), …}

xi R yi

e. g. if R = ‘>’: 2 > 1, 3 > 2, 3 > 1

In relations xi can be repeated

46
 Equivalence Relations
• Reflexive: xRx
• Symmetric: xRy yRx
• Transitive: x R y and y R z xRz

Example: R = ‘=‘
•x=x
•x=y y=x
• x = y and y = z x=z
47
 Equivalence Classes
For equivalence relation R
equivalence class of x = {y : x R y}

Example:
R = { (1, 1), (2, 2), (1, 2), (2, 1),
(3, 3), (4, 4), (3, 4), (4, 3) }

Equivalence class of 1 = {1, 2}

Equivalence class of 3 = {3, 4} 48
 GRAPHS
A directed graph
e
b
node
a d
edge c
• Nodes (Vertices)
V = { a, b, c, d, e }
• Edges
E = { (a,b), (b,c), (b,e),(c,a), (c,e), (d,c), (e,b), (e,d)
49 }
 Labeled Graph
2
6 e
b 2
1 3
a 6 d
5
c

50
 Walk

e
b
a d

Walk is a sequence of adjacent edges

(e, d), (d, c), (c, a)

51
 Path

e
b
a d

Path is a walk where no edge is repeated

Simple path: no node is repeated

52
 Cycle

base e
b
3
a 1 d
2
c

Cycle: a walk from a node (base) to itself

Simple cycle: only the base node is repeated

53
 Euler Tour
8 base
7 e
b 1
4 6
a 5 2 d
3
c

A cycle that contains each edge once

54
 Hamiltonian Cycle
5 base
e
b 1
4
a 2 d
3
c

A simple cycle that contains all nodes

55
 Trees
root

parent

leaf

child

Trees have no cycles

56
 root
Level 0

Level 1
leaf Height 3

Level 2

Level 3
57
Binary Trees

58
 PROOF TECHNIQUES

• Proof by induction

• Proof by contradiction

59
 Induction

We have statements P1, P2, P3, …

If we know
• for some b that P1, P2, …, Pb are true
• for any k >= b that
P1, P2, …, Pk imply Pk+1
Then
Every Pi is true
60
 Proof by Induction
• Inductive basis
Find P1, P2, …, Pb which are true

• Inductive hypothesis
Let’s assume P1, P2, …, Pk are true,
for any k >= b

• Inductive step
61
Show that Pk+1 is true
 Example
Theorem: A binary tree of height n
has at most 2n leaves.

Proof by induction:
let L(i) be the number of leaves at level i

L(0) = 1
L(1) = 2
L(2) = 4

L(3) = 8 62
We want to show: L(i) <= 2i

• Inductive basis
L(0) = 1 (the root node)

• Inductive hypothesis
Let’s assume L(i) <= 2i for all i = 0, 1, …, k

• Induction step
we need to show that L(k + 1) <= 2k+1 63
 Induction Step

Level
k

k+1

From Inductive hypothesis: L(k) <= 2k

64
 Induction Step

Level
k L(k) <= 2k

k+1

L(k+1) <= 2 * L(k) <= 2 * 2k = 2k+1

65
 Remark
Recursion is another thing

Example of recursive function:

f(n) = f(n-1) + f(n-2)

f(0) = 1, f(1) = 1

66
 Proof by Contradiction

We want to prove that a statement P is true

• we assume that P is false

• then we arrive at an incorrect conclusion
• therefore, statement P must be true

67
 Example
Theorem: 2 is not rational

Proof:
Assume by contradiction that it is rational
2 = n/m
n and m have no common factors

We will show that this is impossible

68
 2 = n/m 2 m 2 = n2

n is even
Therefore, n2 is even
n=2k

m is even
2 m2 = 4k2 m2 = 2k2
m=2p

Thus, m and n have common factor 2

Contradiction! 69