MNG A LP

THUT TON LAN TRUYN NGC

Trnh by: Trn Thanh Tm
1
THUT TON LAN TRUYN NGC
Vector gradient ca mt phng sai s.
Ch ra ng gim dc nht v v tr hin ti , t
quyt nh di chuyn t c gi tr cc tiu ton
cc.
Kh khn ln bc di chuyn.

2
NGNG SIGMOID
(y) =
1
1 + e
-y
d(y)
dy
=
(y) . (1 - (y))
.
.
.

x
1

x
2

x
n

w
1

w
2

w
n

x
1
= 1
w
0

net = w
i
x
i

1
o = o(net) =
1 + e
-net

3
THUT TON LAN TRUYN NGC
Hm li:
E(w) =
deD

keoutputs
(t
kd
o
kd
)
2


ouputs: tp tt c cc n-ron u ra ca mng
t
kd
, o
kd
: l gi tr ch v gi tr u ra tng ng ca n-ron
th k ng vi mu hc d

4
THUT TON LAN TRUYN NGC
BackPropagation (training_examples, , n
in
, n
out
, n
hidden
)
in
n
in
hidden
n
hidden
out
n
out
To mng truyn thng gm:
n
in
u vo,
n
hidden
n-ron tng n,
n
out
n-ron u ra.
5
THUT TON LAN TRUYN NGC
BackPropagation (training_examples, , n
in
, n
out
, n
hidden
)
in
n
in
hidden
n
hidden
out
n
out
Khi to b trng cho mng vi gi tr nh
w
hi
w
oh
6
THUT TON LAN TRUYN NGC
BackPropagation (training_examples, , n
in
, n
out
, n
hidden
)
in
n
in
hidden
n
hidden
out
n
out
Trong khi <iu kin kt thc cha tha> lm:
Vi mi cp (x, t) trong khng gian mu hun luyn thc hin:
w
hi
w
oh
1. a gi tr u vo x qua mng v tnh u
ra o
u
ng vi mi n-ron u trong mng.

o
h
o
k
7
THUT TON LAN TRUYN NGC
BackPropagation (training_examples, , n
in
, n
out
, n
hidden
)
in
n
in
hidden
n
hidden
out
n
out
w
hi
w
oh
2. Vi mi n-ron u ra k, ta tnh gi tr li o
k

o
k
= o
k
(1 o
k
)(t
k
o
k
)

o
h
o
k
o
k
Trong khi <iu kin kt thc cha tha> lm:
Vi mi cp (x, t) trong khng gian mu hun luyn thc hin:
8
THUT TON LAN TRUYN NGC
BackPropagation (training_examples, , n
in
, n
out
, n
hidden
)
in
n
in
hidden
n
hidden
out
n
out
w
hi
w
oh
3. Vi mi n-ron tng n h, ta tnh gi tr li o
h

o
h
= o
h
(1 o
h
)
koutputs
w
kh
o
k
o
h
o
k
o
k
o
h
Trong khi <iu kin kt thc cha tha> lm:
Vi mi cp (x, t) trong khng gian mu hun luyn thc hin:
9
THUT TON LAN TRUYN NGC
BackPropagation (training_examples, , n
in
, n
out
, n
hidden
)
in
n
in
hidden
n
hidden
out
n
out
w
hi
w
oh
4. Cp nht li trng s c trong mng w
ji
w
ji
w
ji
+ w
ji

w
ji
= o
j
x
ji

o
h
o
k
o
k
o
h
Trong khi <iu kin kt thc cha tha> lm:
Vi mi cp (x, t) trong khng gian mu hun luyn thc hin:
10
THUT TON LAN TRUYN NGC
V d:
H
1
O
1
0.2
I
2
I
1
I
3
H
2
O
2
0.7
-0.1
-1.2
0.4
1.2
1.1
3.1
0.1
1.17
1
0
Input:
x = (10, 30, 20)
Target: t = (1, 0)
H s hc: q = 0.1
11
THUT TON LAN TRUYN NGC
V d (tt):
12
1. a gi tr u vo x qua mng v tnh u ra o
u
ng vi
mi n-ron u trong mng.

H
1
:

net
H1
= 10 * 0.2 + 30 * (-0.1) + 20 * 0.4 = 7

o
H1
= o(net
H1
) = 0.9990

H
2
:

net
H2
= 10 * 0.7 + 30 * (-1.2) + 20 * 1.2 = -5

o
H2
= o(net
H2
) = 0.0067

O
1
:

net
O1
= 0.9990 * 1.1 + 0.0067

* 0.1 = 1.0996

o
O1
= o(net
O1
) = 0.7501

O
2
:

net
O2
= 0.9990 * 3.1 + 0.0067

* 1.17 = 3.1047

o
O2
= o(net
O2
) = 0.9571

1
o = o(net) =
1 + e
-net

Vi net = Ew
ji
x
ji
THUT TON LAN TRUYN NGC
V d (tt):
13
2. Vi mi n-ron u ra k, ta tnh gi tr li o
k
o
k
= o
k
(1 o
k
)(t
k
o
k
)

3. Vi mi n-ron tng n h, ta tnh gi tr li o
h

o
h
= o
h
(1 o
h
)
koutputs
w
kh
o
k
o
O1
= o
O1
(1 - o
O1
)(t
O1
- o
O1
) = 0.750 (1 0.750)(1 0.750) = 0.0469

o
O2
= o
O2
(1 o
O2
)(t
O2
o
O2
) = 0.957 (1 0.957)(0 0.957) = - 0.0394

o
H1
= o
H1
(1 - o
H1
)[(w
11
* o
O1
) + (w
21
* o
O2
)]
= 0.999(1 0.999)[(1.1 * 0.0469) + (3.1 * (-0.0394))]
= - 0.0000705
o
H2
= o
H2
(1 o
H2
)[(w
12
* o
O1
) + (w
22
* o
O2
)]
= 0.0067(1 0.0067)[(0.1 * 0.0469) + (1.17 * (-0.0394))]
= - 0.000275
THUT TON LAN TRUYN NGC
V d (tt):
14
4. Cp nht li trng s c trong mng w
ji
w
ji
w
ji
+ w
ji

w
ji
= o
j
x
ji

N-ron
n
N-ron
u ra
q o
O
o
H
= x
ji
= o
O
x
ji
W c W mi
H
1
O
1
0.1 0.0469

0.999 0.000469 1.1 1.100469
H
1
O
2
0.1 - 0.0394 0.999 -0.00394 3.1 3.09606
H
2
O
1
0.1 0.0469

0.0067 0.0000314 0.1 0.1000314
H
2
O
2
0.1 - 0.0394 0.0067 -0.0000264 1.17 1.1699736
THUT TON LAN TRUYN NGC
V d (tt):
15
4. Cp nht li trng s c trong mng w
ji
u
vo
N-ron
n
q o
H
x
I
= o
O
x
ji
W c W mi
I
1
H
1
0.1 -0.0000705 10 -0.0000705 0.2 0.1999295
I
1
H
2
0.1 -0.000275 10 -0.000275 0.7 0.699725
I
2
H
1
0.1 -0.0000705 30 -0.0002115 -0.1 -0.1000705
I
2
H
2
0.1 -0.000275 30 -0.000825 -1.2 -1.200825
I
3
H
1
0.1 -0.0000705 20 -0.000141 0.4 0.399859
I
3
H
2
0.1 -0.000275 20 -0.00055 1.2 1.19945
Cp nht trng tng cng

Aw
ji
(n) = o
i
x
ji
+ oAw
ji
(n-1)

n: l ln lp th n
0 o< 1: l hng s tng cng (momentum)
THUT TON LAN TRUYN NGC
16
nh gi thut ton lan truyn
ngc (tt)
Vn hi t v cc tr cc b
Mng hi t ti nhng im c th
cc tiu ha cc b khng gian li
Vn ca cc tr cc b
Cc tiu li c th ng vi trng ny
nhng khng ng vi trng khc
S vng lp hc phi ln xut
hin nhiu cc tr cc b (gn vi cc
tr ton cc) th mng mi c th biu
din c nhng hm phc tp
nh gi thut ton lan truyn
ngc (tt)
PP hn ch vn cc tr cc b
Cp nht trng tng cng (Adding
momentum)
Dng thut ton Gradient gim
ngu nhin (Stochastic gradient
descent)
Cng mt d liu hc, dng nhiu
mng cng loi c b trng c
khi to khc nhau hc, kt qu
u ra ly trung bnh cc mng
nh gi thut ton lan truyn
ngc (tt)
iu kindngvqukhp
iukindng: Li E <ngng
Nhngnu qunhth qukhp
xyra.
nh gi thut ton lan truyn
ngc (tt)
Gii php khc phc qu khp
Km hm b trng (weight decay):
gim trng s mt lng nh sau
mi vng lp hc nhm lm trn b
mt quyt nh.
nh gi cho (cross-validation)
nh gi cho k ln (k-fold cross-
validation)