Professional Documents
Culture Documents
lec2 - 딥러닝 기초
lec2 - 딥러닝 기초
lec2 - 딥러닝 기초
1. Linear Classifier
2. Multilayer Neural Networks
3. Convolutional Neural Networks
g (x) = 0
g (x ) is linear → decision surface is hyperplane
출처: Pattern Classification, Richard O. Duda, Peter E. Hart, and David G. Stork, John Wiley & Sons, Inc. 2001
g (x) = wt x + ω0
w : weight vector
ω0 : threshold weight
출처: Pattern Classification, Richard O. Duda, Peter E. Hart, and David G. Stork, John Wiley & Sons, Inc. 2001
출처: Pattern Classification, Richard O. Duda, Peter E. Hart, and David G. Stork, John Wiley & Sons, Inc. 2001
출처: Pattern Classification, Richard O. Duda, Peter E. Hart, and David G. Stork, John Wiley & Sons, Inc. 2001
5 Autonomous Intelligent Systems (AIS) LAB
Linear Classifier
Two-Category Linear Separable Case
yi → ω1 if at yi > 0
yi → ω2 if at yi < 0
separating vector or
solution vector
출처: Pattern Classification, Richard O. Duda, Peter E. Hart, and David G. Stork, John Wiley & Sons, Inc. 2001
6 Autonomous Intelligent Systems (AIS) LAB
Linear Classifier
Gradient Descent Procedures
a * = min J (a )
solution to the set of linear
inequalities at yi > 0 a
출처: Pattern Classification, Richard O. Duda, Peter E. Hart, and David G. Stork, John Wiley & Sons, Inc. 2001
7 Autonomous Intelligent Systems (AIS) LAB
Linear Classifier
Minimizing the Perceptron Criterion Function
a * = min J (a )
J(a) 결정 solution to the set of linear
inequalities a yi > 0
t
a
The Perceptron Criterion Function
J p (a) = J p (a; y1, K, yn ) = - at y ( )
y∈Y
= - y update rule : a k + 1 = a k − η k ( ) ( ) (- y)
∂J p
∇J p = ( ) ( )
j y∈Y
∂a y∈Yk
출처: Pattern Classification, Richard O. Duda, Peter E. Hart, and David G. Stork, John Wiley & Sons, Inc. 2001
8 Autonomous Intelligent Systems (AIS) LAB
Linear Classifier
Minimizing the Perceptron Criterion Function
Algorithm 3. (Batch Perception)
1 begin initialize a,η(⋅) , criterion θ , k ← 0
2 do k ← k + 1
3 a ← a + η(k ) y Use all samples
y ∈Yk
for parameters update
4 until η(k ) y < θ
y ∈Yk
5 return a
6 end
Algorithm 4. (Fixed-Increment Single-Sample Perceptron)
1 begin initialize a, k ← 0
2 do k ← (k + 1) mod n Use one sample
3 if yk is misclassified by a then a ← a + yk for parameters update
4 until all patterns properly classified
5 return
6 end
출처: Pattern Classification, Richard O. Duda, Peter E. Hart, and David G. Stork, John Wiley & Sons, Inc. 2001
9 Autonomous Intelligent Systems (AIS) LAB
Multilayer Neural Networks
• Classifiers: input units connected by modifiable weights to
output units
• With clever choice of nonlinear func3ons → decision region
leading to minimum error → difficulty in choosing the
appropriate nonlinear functions
• ex) choose a complete basis set such as all polynomials
→ too many free parameters to de determined from a
limited number of training patterns
• ex) use prior knowledge relevant to the classification
problem → guide our choice of nonlinearity
출처: Pattern Classification, Richard O. Duda, Peter E. Hart, and David G. Stork, John Wiley & Sons, Inc. 2001
10 Autonomous Intelligent Systems (AIS) LAB
Multilayer Neural Networks
출처: Pattern Classification, Richard O. Duda, Peter E. Hart, and David G. Stork, John Wiley & Sons, Inc. 2001
11 Autonomous Intelligent Systems (AIS) LAB
Multilayer Neural Networks
Multilayer Neural Networks: Feedforward Operation and
Classification
d d
net j = xi ω ji + ω j 0 =
i =1
i ji j x
x ω
i =0
≡ wt
출처: Pattern Classification, Richard O. Duda, Peter E. Hart, and David G. Stork, John Wiley & Sons, Inc. 2001
12 Autonomous Intelligent Systems (AIS) LAB
Multilayer Neural Networks
Multilayer Neural Networks: Feedforward Operation and
Classification
출처: Pattern Classification, Richard O. Duda, Peter E. Hart, and David G. Stork, John Wiley & Sons, Inc. 2001
13 Autonomous Intelligent Systems (AIS) LAB
Multilayer Neural Networks
Multilayer Neural Networks: Feedforward Operation and
Classification
y j = f (net j )
1 if net ≥ 0
f (net ) ≡ Sgn(net ) ≡
- 1 if net < 0
activation function
()
f ⋅ → " nonlinearity" of a unit :
serve as a ϕ function
nH nH
net k = y j ωkj + ωk 0 =
j =1
j kj k y
y ω
j =0
= wt
z k = f (net k )
출처: Pattern Classification, Richard O. Duda, Peter E. Hart, and David G. Stork, John Wiley & Sons, Inc. 2001
14 Autonomous Intelligent Systems (AIS) LAB
Multilayer Neural Networks
Multilayer Neural Networks: Expressive Power
출처: Pattern Classification, Richard O. Duda, Peter E. Hart, and David G. Stork, John Wiley & Sons, Inc. 2001
15 Autonomous Intelligent Systems (AIS) LAB
Multilayer Neural Networks
Multilayer Neural Networks: Expressive Power
nH d
g k (x) ≡ z k = f w kjf ω ji xi + ω j 0 + ωk 0
j =1 i =1
Kolmogorov : any continuous function g (x)
defined on the unit hypercube I n (I = [0,1] and n ≥ 2 )
can be expressed in the form
2n +1
d
g (x) = Ξ j Ψij (xi )
j =1 i =1
출처: Pattern Classification, Richard O. Duda, Peter E. Hart, and David G. Stork, John Wiley & Sons, Inc. 2001
16 Autonomous Intelligent Systems (AIS) LAB
Multilayer Neural Networks
Multilayer Neural Networks: Expressive Power
출처: Pattern Classification, Richard O. Duda, Peter E. Hart, and David G. Stork, John Wiley & Sons, Inc. 2001
17 Autonomous Intelligent Systems (AIS) LAB
Multilayer Neural Networks
Multilayer Neural Networks: Expressive Power
출처: Pattern Classification, Richard O. Duda, Peter E. Hart, and David G. Stork, John Wiley & Sons, Inc. 2001
18 Autonomous Intelligent Systems (AIS) LAB
Multilayer Neural Networks
Multilayer Neural Networks: Backpropagation
Algorithm
• The simplest and most general method for supervised
training of multilayer neural networks
• Natural extension of the LMS algorithm for linear systems
2 k =1 2
t k : desired output
z k : actual output
1) weights are initialized with random values
∂J ∂J
2) ∆w = −η or ∆w pg = −η
∂w ∂w pq
3) w(m + 1) = w(m ) + ∆w(m )
출처: Pattern Classification, Richard O. Duda, Peter E. Hart, and David G. Stork, John Wiley & Sons, Inc. 2001
19 Autonomous Intelligent Systems (AIS) LAB
Multilayer Neural Networks
Multilayer Neural Networks: Backpropagation
Algorithm
출처: Pattern Classification, Richard O. Duda, Peter E. Hart, and David G. Stork, John Wiley & Sons, Inc. 2001
∆w kj = ηδ k y j = η (t k − z k )f ′(net k )y j
출처: Pattern Classification, Richard O. Duda, Peter E. Hart, and David G. Stork, John Wiley & Sons, Inc. 2001
21 Autonomous Intelligent Systems (AIS) LAB
Multilayer Neural Networks
Multilayer Neural Networks: Backpropagation
Algorithm
input-to-hidden
∂J ∂J ∂y j ∂net j
=
∂w ji ∂y j ∂net j ∂w ji
∂J ∂ 1 c 2 ∂z k
( ) ( )
c
= k t − z k = − t − z
∂y j ∂y j ∂y j
k k
2 k =1 k =1
∂z k ∂net k
= − (t k − z k ) = − (t k − z k )f ′(net k )w kj
c c
k =1 ∂net k ∂y j k =1
δ j ≡ f ′(net j )w kj δ k
c
k =1
c
∆w ji = ηxi δ j = η w kj δ k f ′(net j )xi
k =1
출처: Pattern Classification, Richard O. Duda, Peter E. Hart, and David G. Stork, John Wiley & Sons, Inc. 2001
22 Autonomous Intelligent Systems (AIS) LAB
Multilayer Neural Networks
Multilayer Neural Networks: Training Protocols
Three most useful training protocols: stochastic, batch,
and on-line
• Stochastic training: patterns are chosen randomly from
the training data set, network weights are updated for
each pattern representation
• Batch training: all patterns are presented to the network
before learning takes place
• On-line training: each pattern is presented once and
only once, no use of memory for storing patterns
출처: Pattern Classification, Richard O. Duda, Peter E. Hart, and David G. Stork, John Wiley & Sons, Inc. 2001
23 Autonomous Intelligent Systems (AIS) LAB
Multilayer Neural Networks
출처: Pattern Classification, Richard O. Duda, Peter E. Hart, and David G. Stork, John Wiley & Sons, Inc. 2001
24 Autonomous Intelligent Systems (AIS) LAB
Multilayer Neural Networks: Toy Problem(1)
Output
-1 -1 -1
-1 1 1
1 -1 1
1 1 -1
parameters: ( , , ,( , , ,( , ,
z: network output
t: label/ground truth
parameters: ( , , ,
( , , ,( , ,
parameters: ( , , ,
( , , ,( , ,
parameters: ( , , ,
( , , ,( , ,
Getting parameters
(1) random initialization
(2) update parameters
( )
parameters: ( , , ,( , , ,( , ,
(1) forward
(2) Backward: