Professional Documents
Culture Documents
Chapter 5 - Vector Calculus File
Chapter 5 - Vector Calculus File
Contents
• ex11-12, bt11-12
Differentiation of Univariate Functions
Partial Differentiation and Gradients
Gradients of Matrices
Backpropagation
Higher-Order Derivatives
Linearization and Multivariate Taylor Series
x f f(x) g (gf)(x)
f
• = 2(x3 + 2y) (x3 + 2y) = 4(x3 + 2y)
y y
The gradient of f is [6x2(x3 + 2y) 4(x3 + 2y)]
f3
fm
Jacobian of f: J 23
𝑛
fi(x) = 𝑗=1 aij xj
fi
= aij ∇xf = A
xj
4×2×3 tensor
4×2×3 tensor
Kpq
forward phase
a(0) z(1), a(1) z(2), a(2) … C
𝜕𝐶 𝜕𝐶 𝜕𝐶
…
𝜕 (1) 𝜕 (2) 𝜕 (𝑁)
backward phase
𝜕𝐶 𝜕𝐶 𝜕𝒂(𝑁) 𝜕𝒂(𝑁−1)
=
𝜕𝑾(𝑁−1) 𝜕𝒂(𝑁) 𝜕𝒂(𝑁−1) 𝜕𝑾(𝑁−1) 𝜕𝐶 𝜕𝐶 𝜕𝒂(𝑁)
=
𝜕𝑾(𝑁) 𝜕𝒂(𝑁) 𝜕𝑾(𝑁)
𝜕𝐶 𝜕𝐶 𝜕𝒂(𝑁) 𝜕𝒂(𝑁−1)
=
𝜕𝒃(𝑁−1) 𝜕𝒂(𝑁) 𝜕𝒂(𝑁−1) 𝜕𝒃(𝑁−1) 𝜕𝐶 𝜕𝐶 𝜕𝒂(𝑁)
=
𝜕𝒃(𝑁) 𝜕𝒂(𝑁) 𝜕𝒃(𝑁)
𝜕𝐶 𝜕𝐶 𝜕𝒂(𝑁) 𝜕𝒂(𝑁−1) 𝜕𝒂(𝑁−2)
= Benefit of backpropagation:
𝜕𝑾(𝑁−2) 𝜕𝒂(𝑁) 𝜕𝒂(𝑁−1) 𝜕𝒂(𝑁−2) 𝜕𝑾(𝑁−2)
Reused terms outside the box
𝜕𝐶 𝜕𝐶 𝜕𝒂(𝑁) 𝜕𝒂(𝑁−1) 𝜕𝒂(𝑁−2)
=
𝜕𝒃(𝑁−2) 𝜕𝒂(𝑁) 𝜕𝒂(𝑁−1) 𝜕𝒂(𝑁−2) 𝜕𝒃(𝑁−2)
Consider a function f : 2
of two variables x, y.
Second order partial derivatives:
2 f 2 f Ex.
f: 2 , f(x, y) = x3y – 3xy2 + 5y,
x 2 xy 𝜕𝑓 𝜕𝑓
= 3x2y – 3y2, = x3 – 6xy +5
𝜕𝑥 𝜕𝑦
2 f 2 f 2
𝜕𝑓 2
𝜕𝑓
= 3𝑥2 − 6𝑦,
2 = 6𝑥𝑦,
yx y 2
𝜕𝑥 𝜕𝑥𝜕𝑦
2
𝜕𝑓 2 𝜕2𝑓
= 3𝑥 − 6𝑦, 2 = −6𝑥
𝜕𝑦𝜕𝑥 𝜕𝑦
n f
is the nth partial derivative of f with respect to x
x n
x x xn x2
...
xn 2
n 1
Dimension: n n
x1 f1
Gradient x2 f2 Hessian
x3
m n matrix m (n n) tensor
Dimension: 2 3
Dimension: 2 (3 3)
Approximation problems
where
*
* *
[i].[j].[k]
3[i,j,k]