Professional Documents
Culture Documents
Mathematics For Machine Learning Multivariate Calculus Formula Sheet
Mathematics For Machine Learning Multivariate Calculus Formula Sheet
Mathematics For Machine Learning Multivariate Calculus Formula Sheet
Multivariate Calculus
Formula sheet
Dr Samuel J. Cooper
Prof. David Dye
Dr A. Freddie Page
- Power Rule:
Derivative structures
Given f (x) = axb ,
Given f = f (x, y, z)
then f 0 (x) = abx(b−1)
- Product Rule:
- Jacobian:
Given A(x) = f (x)g(x),
∂f ∂f ∂f
then A0 (x) = f 0 (x)g(x) + f (x)g 0 (x) Jf = , ,
∂x ∂y ∂z
- Chain Rule:
Given h = h(p) and p = p(m), - Hessian:
dh dh dp
then = × ∂2f ∂2f ∂2f
dm dp dm
∂x2 ∂x∂y ∂x∂z
Hf = ∂2f ∂2f ∂2f
- Total derivative:
∂y∂x ∂y 2 ∂y∂z
For the function f (x, y, z, ...), where each variable
∂2f ∂2f ∂2f
is a function of parameter t, the total derivative is ∂z∂x ∂z∂y ∂z 2
df ∂f dx ∂f dy ∂f dz
= + + + ...
dt ∂x dt ∂y dt ∂z dt
1
Neural networks - Grad :
- Activation function: ∂f
∂x
ex − e−x
σ(x) = tanh(x) = ∇f = ∂f
ex + e−x ∂y
d 1 4
(σ(x)) = 2 = ∂f
dx cosh (x) (e + e−x )2
x ∂z
- Directional Gradient:
Taylor Series
∇f.r̂
- Univariate:
- Gradient Descent:
1
f (x) = f (c) + f (c)(x − c) + f 00 (c)(x − c)2 + ...
0
2 sn+1 = sn − γ∇f
∞ (n)
X f (c)
= (x − c)n
n! - Lagrange Multipliers λ:
n=0
- Multivariate: ∇f = λ∇g
criterion: ∇χ2 = 0
Optimization and Vector Calculus
- Newton-Raphson:
anext = acur − γ∇χ2
n
f (xi ) X (yi − y(xi ; ak )) ∂y
xi+1 = xi − = acur + γ
f 0 (xi ) σi ∂ak
i