MACHINE LEARNING - FOUNDATIONS

REVISION (WEEK 2)

IIT Madras Online Degree

Topics

• Continuity
• Differentiability
• Linear Approximation
• Higher order approximations
• Multivariate Linear Approximation
• Directional Derivative

2
Continuity

A function 𝑓(𝑥) is continuous at 𝑥 = 𝑎 if:

𝑓(𝑎) = lim− 𝑓(𝑥) = lim+ 𝑓(𝑥)

𝑥→𝑎 𝑥→𝑎

3
Differentiability

A function 𝑓(𝑥) is differentiable at 𝑥 = 𝑎 if:

𝑓(𝑥) − 𝑓(𝑎) 𝑓(𝑥) − 𝑓(𝑎)

𝑓 ′ (𝑎) = lim− = lim+
𝑥→𝑎 𝑥−𝑎 𝑥→𝑎 𝑥−𝑎

4
Linear Approximation

The linear approximation 𝐿(𝑥) of a function 𝑓(𝑥) at point 𝑎 is given by:

′
𝐿(𝑥) = 𝑓(𝑎) + 𝑓 (𝑎)(𝑥 − 𝑎)

5
Linear Approximation

The linear approximation 𝐿(𝑥) of a function 𝑓(𝑥) at point 𝑎 is given by:

′
𝐿(𝑥) = 𝑓(𝑎) + 𝑓 (𝑎)(𝑥 − 𝑎)

This is indeed the equation of a tangent line:

𝑦 − 𝑦1 = 𝑚(𝑥 − 𝑥1 )
𝑦 = 𝑦1 + 𝑚(𝑥 − 𝑥1 )

5
Linear Approximation

The linear approximation 𝐿(𝑥) of a function 𝑓(𝑥) at point 𝑎 is given by:

′
𝐿(𝑥) = 𝑓(𝑎) + 𝑓 (𝑎)(𝑥 − 𝑎)

This is indeed the equation of a tangent line:

𝑦 − 𝑦1 = 𝑚(𝑥 − 𝑥1 )
𝑦 = 𝑦1 + 𝑚(𝑥 − 𝑥1 )

If 𝑥1 = 𝑎, 𝑦1 = 𝑓(𝑎) and 𝑚 = 𝑓 ′ (𝑎), we get,

𝑦 = 𝑓(𝑎) + 𝑓 ′ (𝑎)(𝑥 − 𝑎)

5
Higher order approximations

Linear Approximation
′
𝐿(𝑥) = 𝑓(𝑎) + 𝑓 (𝑎)(𝑥 − 𝑎)

6
Higher order approximations

Linear Approximation
′
𝐿(𝑥) = 𝑓(𝑎) + 𝑓 (𝑎)(𝑥 − 𝑎)

Quadratic Approximation
″
′ 𝑓 (𝑎)
𝐿(𝑥) = 𝑓(𝑎) + 𝑓 (𝑎)(𝑥 − 𝑎) + (𝑥 − 𝑎)2
2

6
Higher order approximations

Linear Approximation
′
𝐿(𝑥) = 𝑓(𝑎) + 𝑓 (𝑎)(𝑥 − 𝑎)

Quadratic Approximation
″
′ 𝑓 (𝑎)
𝐿(𝑥) = 𝑓(𝑎) + 𝑓 (𝑎)(𝑥 − 𝑎) + (𝑥 − 𝑎)2
2

Higher-order Approximations
𝑓 (2) (𝑎)
𝐿(𝑥) = 𝑓(𝑎) + 𝑓 (1) (𝑎)(𝑥 − 𝑎) + (𝑥 − 𝑎)2 +
2
𝑓 (3) (𝑎) 𝑓 (4) (𝑎)
+ (𝑥 − 𝑎)3 + (𝑥 − 𝑎)4 ...
3⋅2 4⋅3⋅2

6
Multivariate linear approximation: Linear approximation of functions involving
multiple variables

The linear approximation of a function 𝑓 of two variables 𝑥 and 𝑦 in the neighborhood of

(𝑎, 𝑏) is:
𝜕𝑓 𝜕𝑓
𝐿(𝑥, 𝑦) = 𝑓(𝑎, 𝑏) + (𝑎, 𝑏)(𝑥 − 𝑎) + (𝑎, 𝑏)(𝑦 − 𝑏)
𝜕𝑥 𝜕𝑦

7
DIRECTIONAL DERIVATIVES
Directional Derivative

𝜕𝑓
• 𝑓𝑥 (𝑥, 𝑦) = 𝜕𝑥 (𝑥, 𝑦)

9
Directional Derivative

𝜕𝑓
• 𝑓𝑥 (𝑥, 𝑦) = 𝜕𝑥 (𝑥, 𝑦) = Rate of change of 𝑓 as we vary 𝑥 (keeping 𝑦 fixed).

9
Directional Derivative

𝜕𝑓
• 𝑓𝑥 (𝑥, 𝑦) = 𝜕𝑥 (𝑥, 𝑦) = Rate of change of 𝑓 as we vary 𝑥 (keeping 𝑦 fixed).
𝜕𝑓
• 𝑓𝑦 (𝑥, 𝑦) = 𝜕𝑦 (𝑥, 𝑦)

9
Directional Derivative

𝜕𝑓
• 𝑓𝑥 (𝑥, 𝑦) = 𝜕𝑥 (𝑥, 𝑦) = Rate of change of 𝑓 as we vary 𝑥 (keeping 𝑦 fixed).
𝜕𝑓
• 𝑓𝑦 (𝑥, 𝑦) = 𝜕𝑦 (𝑥, 𝑦) = Rate of change of 𝑓 as we vary 𝑦 (keeping 𝑥 fixed).

9
Directional Derivative

𝜕𝑓
• 𝑓𝑥 (𝑥, 𝑦) = 𝜕𝑥 (𝑥, 𝑦) = Rate of change of 𝑓 as we vary 𝑥 (keeping 𝑦 fixed).
𝜕𝑓
• 𝑓𝑦 (𝑥, 𝑦) = 𝜕𝑦 (𝑥, 𝑦) = Rate of change of 𝑓 as we vary 𝑦 (keeping 𝑥 fixed).
• Directional derivative of 𝑓(𝑥, 𝑦)

9
Directional Derivative

𝜕𝑓
• 𝑓𝑥 (𝑥, 𝑦) = 𝜕𝑥 (𝑥, 𝑦) = Rate of change of 𝑓 as we vary 𝑥 (keeping 𝑦 fixed).
𝜕𝑓
• 𝑓𝑦 (𝑥, 𝑦) = 𝜕𝑦 (𝑥, 𝑦) = Rate of change of 𝑓 as we vary 𝑦 (keeping 𝑥 fixed).
• Directional derivative of 𝑓(𝑥, 𝑦) = Rate of change of 𝑓 if we allow both 𝑥 and 𝑦 to
change simultaneously (in some direction (𝑢)).

9
Directional Derivative

𝜕𝑓
• 𝑓𝑥 (𝑥, 𝑦) = 𝜕𝑥 (𝑥, 𝑦) = Rate of change of 𝑓 as we vary 𝑥 (keeping 𝑦 fixed).
𝜕𝑓
• 𝑓𝑦 (𝑥, 𝑦) = 𝜕𝑦 (𝑥, 𝑦) = Rate of change of 𝑓 as we vary 𝑦 (keeping 𝑥 fixed).
• Directional derivative of 𝑓(𝑥, 𝑦) = Rate of change of 𝑓 if we allow both 𝑥 and 𝑦 to
change simultaneously (in some direction (𝑢)).

𝐷𝑢⃗⃗⃗ ⃗ 𝑓(𝑥, 𝑦) = ∇𝑓 ⋅ 𝑢
𝜕𝑓 𝜕𝑓
=[ , ] ⋅ [𝑢1 , 𝑢2 ]
𝜕𝑥 𝜕𝑦
𝜕𝑓 𝜕𝑓
= 𝑢1 + 𝑢2
𝜕𝑥 𝜕𝑦

9
Directional Derivative

𝜕𝑓
• 𝑓𝑥 (𝑥, 𝑦) = 𝜕𝑥 (𝑥, 𝑦) = Rate of change of 𝑓 as we vary 𝑥 (keeping 𝑦 fixed).
𝜕𝑓
• 𝑓𝑦 (𝑥, 𝑦) = 𝜕𝑦 (𝑥, 𝑦) = Rate of change of 𝑓 as we vary 𝑦 (keeping 𝑥 fixed).
• Directional derivative of 𝑓(𝑥, 𝑦) = Rate of change of 𝑓 if we allow both 𝑥 and 𝑦 to
change simultaneously (in some direction (𝑢)).

𝐷𝑢⃗⃗⃗ ⃗ 𝑓(𝑥, 𝑦) = ∇𝑓 ⋅ 𝑢
𝜕𝑓 𝜕𝑓
=[ , ] ⋅ [𝑢1 , 𝑢2 ]
𝜕𝑥 𝜕𝑦
𝜕𝑓 𝜕𝑓
= 𝑢1 + 𝑢2
𝜕𝑥 𝜕𝑦

Directional derivative can be considered to be a weighted sum of partial derivatives.

9
Thank you.