Introduction to Artificial Intelligence, Autumn Semester 2011

Nick Hawes, School of Computer Science, University of Birmingham

Deadline: IAI Lab, December 6h 2011

Exercise 3
Machine Learning
Question 1

Calculate the entropy of the following probability distributions.

v1 v2 v3

P(V) 0.5 0.25 0.25

P(v)* log2(P(v)) -0.5 -0.5 -0.5

H(V) 1.5

v1 v2 v3 v4

P(V) 0.01 0.4 0.1 0.49

P(v)* log2(P(v)) -0.06644 -0.52877 -0.33219 -0.50428

H(V) 1.43168431871459

Question 2

The first split should be on Time as this has the highest information gain (0.45 vs 0.35). To
calculate this, first calculate the entropy of the entire set.

P(Snack = chocolate) = 5/8

P(Snack = fruit) = 3/8

H(Snack)
= - (5/8 * log2(5/8)) + (3/8 * log2(3/8))
= - (5/8 * -0.6781) + (3/8 * -1.4150)
= - (-0.4238 + -0.5306)
= 0.9544

Split on Energy

Energy = tired
P(chocolate) = 3/3
H(Snacktired) = 0
Energy = awake
P(chocolate) = 2/5
P(fruit) = 3/5
H(Snackawake)
= - (2/5 * log2(2/5)) + (3/5 * log2(3/5))
= - (2/5 * -1.3219) + (3/5 * -0.737)
= - (-0.5288 + -0.4422)
= 0.971

Remainder(Energy) = 3/8 * 0 + 5/8 * 0.971 = 0.6068

IG(Energy) = 0.9544 – 0.6068 = 0.3476

Split on Time

Time = morning
P(chocolate) = 2/4
P(fruit) = 2/4
H(Snackmorning) = 1

Time = afternoon
P(chocolate) = 3/3
H(Snackafternoon) = 0

Time = evening
P(fruit) = 1/1
H(Snackevening) = 0

Remainder(Time) = 4/8 * 1 + 3/8 * 0 + 1/8 * 0 = 0.5

IG(Time) = 0.9544 – 0.5 = 0.4544

Question 3

Batch 1
w0 w1 alpha
40 5 0.00001
x y h y-h y-h*x
84 71 460 -389 -32676
31 43 195 -152 -4712
75 53 415 -362 -27150
108 59 580 -521 -56268
161 66 845 -779 -125419
222 107 1150 -1043 -231546
sums: -3246 -477771
* alpha -0.03246 -4.77771
w0,w1 39.96754 0.22229
 Batch 2
w0 w1 alpha
39.96754 0.22229 0.00001
x y h y-h
84 71 58.6399 12.3601
31 43 46.85853 -3.85853
75 53 56.63929 -3.63929
108 59 63.97486 -4.97486
161 66 75.75623 -9.75623
222 107 89.31592 17.68408

all within 20, so stop

Question 4

The hard part getting started is remembering all the extra bits. The equation for defining
the straight line for the perceptron learning rule is

w0x0 + w1x1 + w2x2 = 0

x0 is always 1, so it can effectively be ignored. w2 must start at -1 to allow x2 to be moved

from the right hand side of the equation to the left hand side. The result of this is that after
learning the line must be calculated by rearranging the above formula, not using the w0
and w1 direction as before, i.e. the x2 can be found using the following:

- (w0 + w1x1) / w2 = x2

One pass through the data needs to consider each example 3 times, one time for each
input (x0 to x2). The weights should be updated each time, such that the weights may
chance for presentations of the same example. Once you have done a complete pass
through the data, you can either recalculate all the predictions to get the error rate, or
read them off the graph.
One pass through the data should look like this. The green cells show the weights that are
have been updated as a result of the line above. The peach (?) cells indicate which input
is being used for the index for that row.

1st pass through training data

alpha
0.1
x0 x1 x2 class w0 w1 w2 x*w h cl - h
1 1 3 1 8 1 -1 6 1 0
1 1 3 1 8 1 -1 6 1 0
1 1 3 1 8 1 -1 6 1 0
1 2 2 1 8 1 -1 8 1 0
1 2 2 1 8 1 -1 8 1 0
1 2 2 1 8 1 -1 8 1 0
1 1 7 0 8 1 -1 2 1 -1
1 1 7 0 7.9 1 -1 1.9 1 -1
1 1 7 0 7.9 0.9 -1 1.8 1 -1
1 2 5 0 7.9 0.9 -1.7 1.2 1 -1
1 2 5 0 7.8 0.9 -1.7 1.1 1 -1
1 2 5 0 7.8 0.7 -1.7 0.7 1 -1
final weights: 7.8 0.7 -2.2

New hypothesis
x0 x1 x2 class sep h
1 1 3 1 1.9 1
1 2 2 1 4.8 1
1 1 7 0 -6.9 0 0% error
1 2 5 0 -1.8 0

Plotting the initial line should be straightforward from the first principles of the straight line
equation. Plotting the learnt line is harder. If we plug the new weights into the equation
above we get.
x2 = - (7.8 + 0.7x1) / -2.2

We can then choose 2 values of x1, use the equation to calculate the matching x2 values,
then use these to plot the line.

x1 = 0, x2 = 3.5
x1 = 3, x2 = 4.5
 Line from initial weights
12

10

0
0 1 2 3

Line from learnt weights

12

10

0
0 1 2 3