Professional Documents
Culture Documents
7 Newton Raphson Method
7 Newton Raphson Method
7 Newton Raphson Method
• Better performance than the Steepest descent method due to the use of first and second derivative.
• However this happens when the initial guess is nearer to the minima.
• Question: Can you combine any other method with Newton-Raphson so that its performance can
be improved.
Newton Raphson Method
𝑇 𝑘 𝑇 𝑘
𝜙′ 𝛼 = 𝛻𝑓 𝑥𝑘 + 𝑘
𝛼𝑑 𝑑 ⇒ 𝜙′ 0 = 𝛻𝑓 𝑥 𝑘 𝑑
𝑘 𝑇 𝑘 −1
= −𝛻𝑓 𝑥 𝐹 𝑥 𝛻𝑓 𝑥 𝑘 < 0
𝒌 −𝟏
for 𝑭 𝒙 > 𝟎 and 𝛁𝒇 𝒙𝒌 ≠ 𝟎
• This means that the slope of 𝜙(𝛼) at 0 is negative => The function is decreasing.
• Hence it is possible to find an 𝛼 ∈ 0, 𝛼ത for which 𝜙 𝛼 < 𝜙(0)
• Which means 𝑓 𝑥 𝑘 + 𝛼𝑑𝑘 < 𝑓(𝑥 𝑘 )
𝒌 −𝟏
• Hence 𝑭 𝒙 > 𝟎 and 𝛁𝒇 𝒙𝒌 ≠ 𝟎 are necessary criteria for convergence.
Newton Raphson Method
𝑘 𝑇 𝑘 −1
−𝛻𝑓 𝑥 𝐹 𝑥 𝛻𝑓 𝑥𝑘 < 0
𝒌 −𝟏
for 𝑭 𝒙 < 𝟎 and 𝛁𝒇 𝒙𝒌 ≠ 𝟎
• This means that the slope of 𝜙(𝛼) at 0 is negative => The function is decreasing.
• Hence it is possible to find an 𝛼 ∈ 0, 𝛼ത for which
𝒌 −𝟏
• Which means Hence 𝑭 𝒙 > 𝟎 and 𝛁𝒇 𝒙𝒌 ≠ 𝟎 are necessary criteria for
convergence.
Newton Raphson Method
𝑘+1 𝑘 𝑘 −1
𝑥 = 𝑥 − 𝛼𝐹 𝑥 𝛻𝑓 𝑥 𝑘
• Some disadvantages:
𝑘 −1
• The 𝐹 𝑥 matrix should be invertible.
• For a large n it becomes computationally expensive.
• We need to start at a sufficiently close range.
Levenberg Marquardt Modification
−1
𝑥 𝑘+1 = 𝑥 𝑘 − 𝛼𝐹 𝑥 𝑘 𝛻𝑓 𝑥 𝑘
𝑘 −1
• Disadvantage: The 𝐹 𝑥 matrix should be invertible.
−𝟏
Solution: 𝒙𝒌+𝟏 = 𝒙𝒌 − 𝜶 𝑭 𝐱 𝐤 + 𝝁𝒌 𝑰 𝜵𝒇 𝒙𝒌
• Where 𝜇𝑘 ≥ 0.
• If 𝜇𝑘 → 0, it approaches Newton’s Method
• If 𝜇𝑘 → ∞, it approaches Steepest descent.
Example
𝑓 𝑥1 , 𝑥2 , 𝑥3 , 𝑥4 = 𝑥1 + 10𝑥2 2 + 5 𝑥3 − 𝑥4 2 + 𝑥2 − 2𝑥3 4 + 10 𝑥1 − 𝑥4 4
𝑓 𝑥1 , 𝑥2 , 𝑥3 , 𝑥4 = 𝑥1 + 10𝑥2 2 + 5 𝑥3 − 𝑥4 2 + 𝑥2 − 2𝑥3 4 + 10 𝑥1 − 𝑥4 4
1. Find Gradient
Example
𝑓 𝑥1 , 𝑥2 , 𝑥3 , 𝑥4 = 𝑥1 + 10𝑥2 2 + 5 𝑥3 − 𝑥4 2 + 𝑥2 − 2𝑥3 4 + 10 𝑥1 − 𝑥4 4
𝑓 𝑥1 , 𝑥2 , 𝑥3 , 𝑥4 = 𝑥1 + 10𝑥2 2 + 5 𝑥3 − 𝑥4 2 + 𝑥2 − 2𝑥3 4 + 10 𝑥1 − 𝑥4 4
3. Start with 𝑥0 = 3, −1,0,1 and do iteration 1 and repeat until 𝛻𝑓 𝑥 = 0 or stopping criterion.
Newton’s Method for Curve Fitting
𝑇
𝑥0 = 𝑥1 0 , 𝑥2 0 , 𝑥3 0 , 𝑥4 0
𝑝 𝑥 = 𝑥1 𝑡 3 + 𝑥2 𝑡 2 + 𝑥3 𝑡 + 𝑥4
• Define the error between the actual data and the estimated data as
𝑟𝑖 𝑥 = 𝑦𝑖 − 𝑝𝑖 𝑥 = 𝑦𝑖 − 𝑥1 𝑡𝑖3 − 𝑥2 𝑡𝑖2 − 𝑥3 𝑡𝑖 − 𝑥4
Newton’s Method for Curve Fitting
𝑚
2
𝑓 𝑥 = 𝑟𝑖 𝑥
𝑖=1
• The optimization problem is now 𝑚
𝑚𝑖𝑛𝑖𝑚𝑖𝑧𝑒 𝑟𝑖 𝑥 2
𝑖=1
• Define 𝑟 = 𝑟1 𝑟2 … 𝑟𝑚 𝑇 and 𝑓 𝑥 = 𝑟 𝑇 𝑟 = σ𝑚
𝑖=1 𝑟𝑖 𝑥
2.
Newton’s Method for Curve Fitting
• Hessian
𝜕 𝜕𝑓
𝐹 𝑥 =
𝜕𝑥𝑘 𝜕𝑥𝑗
𝑚
𝜕 𝜕𝑟𝑖
= 2𝑟𝑖
𝜕𝑥𝑘 𝜕𝑥𝑗
𝑖=1
𝑚
𝜕𝑟𝑖 𝜕𝑟𝑖 𝜕 2 𝑟𝑖
=2 + 𝑟𝑖
𝜕𝑥𝑘 𝜕𝑥𝑗 𝜕𝑥𝑘 𝜕𝑥𝑗
𝑖=1
= 2( 𝐽𝑇 𝐽 + 𝑆)
Newton’s Method for Curve Fitting
𝜕 2 𝑟𝑖
• Here, S = S(x) = 𝑟𝑖 and it can be ignored as its contribution becomes negligible.
𝜕𝑥𝑘 𝜕𝑥𝑗
𝑥𝑘+1 = 𝑥𝑘 − 𝐽𝑇 𝐽 + 𝜇𝑘 𝐼 −1 𝐽𝑇 𝑟
Levenberg Marquardt Algorithm
Newton’s Method for Curve Fitting: Dimensions
𝑇 −1 𝑇
𝑥𝑘+1 𝟒×𝟏 = 𝑥𝑘(𝟒×𝟏) − 𝐽𝟒×𝟏𝟎𝟎 𝐽𝟏𝟎𝟎×𝟒 + 𝜇𝑘 𝐼𝟒×𝟒 𝟒×𝟒 𝐽𝟒×𝟏𝟎𝟎 𝑟𝟏𝟎𝟎×𝟏