Review: basic notions in nonlinear programming

Convexity and optimality

Global minimizer: x S such that f (x ) f (x ) x S R n Local minimizer: x S such that, for some > 0, f (x ) f (x ) x S B (x ; )

f(x)
Local Local

Global

Theorem. For convex functions on convex sets holds that each local minimizer is a global minimizer
Example: min f (x ) such that
x

Ax b , where f (x ) = c x (linear program) or T f (x ) = + c T x + 1 2 x Qx , Q positive semidenite (quadratic program)

September 18, 2007 (1 : 7)

Review: basic notions in nonlinear programming

Iterative algorithms

Problem: min f (x )
x S

Many optimization algorithms are of the type 1. Specify an initial guess x0 2. For k = 0, 1, . . .
2.1 If xk optimal stop 2.2 Determine a search direction pk and a step k and set 2.3 xk +1 = xk + k pk

September 18, 2007

(2 : 7)

Review: basic notions in nonlinear programming

Convergence rate
Denition
n The sequence { xk } k =1 R converges to x with rate p and rate constant C if xk +1 x lim = C. k xk x p

Linear: p = 1 and 0 < C < 1. The error is multiplied by C each iteration. Quadratic: p = 2. Roughly a doubling of the correct digits each iteration. Superlinear: p = 1, C = 0. Faster than linear. Includes quadratic convergence but also intermediate rates.

September 18, 2007

(3 : 7)

Review: basic notions in nonlinear programming

Taylor series with remainder term

Let f : R n R be of class C 2 (twice continuously dierentiable). Then, x , y R n , 1 f (y ) = f (x ) + (y x )T f (x ) + (y x )T 2 f ( )(y x ) 2 where = y + (1 )x for some [0, 1]. The gradient of f : f f T f = ,..., x1 xn The Hessian of f : 2 f = 2 f x n x1
September 18, 2007

2f 2 x1 . . .

... .. .

...

2f x1 xn . . . 2f , 2 xn
(4 : 7)

Review: basic notions in nonlinear programming

Positive denite matrices

An n-by-n real matrix A is positive semidenite if v T Av 0 It is positive denite if v T Av > 0 0 = v R n v R n

A positive denite matrix is nonsingular Matrix A is positive denite if and only if matrix A1 is positive denite A symmetric matrix A is positive denite if and only if All eigenvalues of A are strictly positive A = LLT with L lower triangular and Lii > 0 (Cholesky factorization).
September 18, 2007 (5 : 7)

Review: basic notions in nonlinear programming

Unconstrained minimization: necessary and sucient conditions

First-order necessary condition: Assume f : R n R has a local minimum at x = x and that f is dierentiable at x = x . Then f (x ) = 0. Second-order necessary condition: Assume f : R n R has a local minimum at x = x and that f is of class C 2 . Then 2 f (x ) is positive semidenite. Second-order suciency condition: Assume f : R n R is of class C 2 , f (x ) = 0 and 2 f (x ) is positive denite then x is a local minimizer.

September 18, 2007

(6 : 7)

Review: basic notions in nonlinear programming

Quadratic functions

Let : R n R be dened by 1 (s ) = + g T s + s T Hs , 2 where R , g and s are n-vectors, and H an n-by-n matrix.

Theorem
If H is symmetric and positive denite, then the solution to Hs = g is the unique minimizer of .

September 18, 2007

(7 : 7)