Professional Documents
Culture Documents
Computational Learning Theory (COLT) : Goals
Computational Learning Theory (COLT) : Goals
1
Machine Learning, Chapter 7 CSE 574, Spring 2004
• Cryptography
• Ek where k specifies the key
• Even if values of Ek are known for polynomially many
dynamically chosen inputs
• Computationally infeasible to deduce an algorithm for Ek
or an approximation to it
2
Machine Learning, Chapter 7 CSE 574, Spring 2004
3
Machine Learning, Chapter 7 CSE 574, Spring 2004
Goal of COLT
4
Machine Learning, Chapter 7 CSE 574, Spring 2004
5
Machine Learning, Chapter 7 CSE 574, Spring 2004
6
Machine Learning, Chapter 7 CSE 574, Spring 2004
7
Machine Learning, Chapter 7 CSE 574, Spring 2004
8
Machine Learning, Chapter 7 CSE 574, Spring 2004
Error of a Hypothesis
9
Machine Learning, Chapter 7 CSE 574, Spring 2004
Error of hypothesis h
Instance Space X
c
- -
h
+
+
Where c -
and h disagree
11
Machine Learning, Chapter 7 CSE 574, Spring 2004
PAC Learnable
12
Machine Learning, Chapter 7 CSE 574, Spring 2004
13
Machine Learning, Chapter 7 CSE 574, Spring 2004
14
Machine Learning, Chapter 7 CSE 574, Spring 2004
• Sample Complexity
• No. of training examples required
• Growth with problem size
• Bound on no. of training samples needed for
consistent learners– learners that perfectly fit the
training data
15
Machine Learning, Chapter 7 CSE 574, Spring 2004
Version Space
Hypothesis Space H
. . .
.
. .
16
Machine Learning, Chapter 7 CSE 574, Spring 2004
Version Space
17
Machine Learning, Chapter 7 CSE 574, Spring 2004
Version Space
Hypothesis h
Hypothesis Space H
.
. .
.
VSH,D . .
Hypothesis Space H
.
error=.1 . .
. r = .2 error=.2 error=.3
r=0 r = .4
error=.3
r = .1 VSH,D . .
error=.1
r=0 error=.2
r = .3
19
Machine Learning, Chapter 7 CSE 574, Spring 2004
Hypothesis Space H
. . ε = 0.21
error=.1
r = .2 .
. error=.2
r=0
error=.3
r = .4
error=.3
VSH,D
r = .1
. error=.1
r=0 .
error=.2
r = .3
20
Machine Learning, Chapter 7 CSE 574, Spring 2004
• Theorem:
• If the hypothesis space H is finite
• D is a sequence of m > 1 independent random samples of
concept c
• Then for any 0 < ε <1
• Probability of not ε-exhausted (with respect to c) is less than
or equal to
− εm
|H|e
• Bounds the probability that m training samples will fail
to eliminate all “bad” hypotheses
21
Machine Learning, Chapter 7 CSE 574, Spring 2004
Probability of failure
− εm
|H|e ≤δ is below some desired level
Re arranging
1
m ≥ (ln | H | + ln(1 / δ )
ε
• Provides general bound on the no. of training
samples
• sufficient for any consistent learner to learn any target
concept in H for
• any desired values of δ and ε
22
Machine Learning, Chapter 7 CSE 574, Spring 2004
1
m ≥ 2 (ln | H | + ln(1 / δ )
2ε
23
Machine Learning, Chapter 7 CSE 574, Spring 2004
• Sample complexity
1
m≥ (n ln 3 + ln(1 / δ )
ε
24
Machine Learning, Chapter 7 CSE 574, Spring 2004
1
m≥ (nk ln 3 + ln(1 / δ )
ε
25