GME-507/ME-407

Engineering Optimization

Hamid Torab, PhD, PE

 The Design Process
 Problem Statement: A description of the design problem that
includes the goal and all the requirements.

 Design Variables: what the designer has control over

 Design Parameters: what the designer has no control over

• i.e. cost of gas, steel

 Feasible Domain: the collection of all acceptable designs

 Constraints: limits on or relations between variables.

 Optimization
 Engineering Optimization: The use of optimization
techniques to minimize or maximize a design objective.

 When maximizing or minimizing a specific goal in design,

the objective function 𝑓(𝑥) (or cost function) must be
identified
 e.g. minimize cost or weight, maximize profit or efficiency

 Steps:
• Convert the problem statement into a mathematical model (modeling)

• Try to optimize the goal of this design

 Part I

Mathematical Modeling and

Basic Optimization
 Modeling
 In this course, we work with Mathematical Modeling: A
description of a system using mathematical concepts and
language.
 This means converting the problem statement into mathematical
relations between the goal of the optimization, the design
variables, and the design parameters

 This includes converting all the constraints into a mathematical

format as well.
 General Format
 After modeling is complete, all optimization problems will
have a format such as: Vector form:

 Minimize 𝑓(𝑥) subject to (s.t.) 𝑥1

𝑥2
• ℎ𝑖 𝑥 = 0 (𝑖 = 1,2, … , 𝑛) 𝑥= :
:
• ℎ𝑖 𝑥 an equality constraint 𝑥𝑛

• 𝑔𝑗 (𝑥) ≤ 0 (𝑗 = 1,2, … , 𝑚) 𝑥 is called a vector of design

variables.
• 𝑔𝑗 (𝑥) an inequality constraint

 All optimization problems can be thought of as minimization. A maximization

problem can be converted to a minimization problem by minimizing the negative
of that function.

• max 𝑓 = min(−𝑓)

 All underlined letters (variables) are vectors

 Example 1: Modeling
 A company is asked to redesign the heat exchanger tubes with
length (𝑙), radius (𝑟). The smallest allowable radius is 0.5𝑐𝑚. All the
tubes are of the same size. The total cross section of all the tubes is
less than 2000𝑐𝑚2 . Find the maximum surface area 𝑆.
 Solution:

Maximize 𝑆 = 2π𝑟𝑙𝑛

 Subject to: 𝑟 ≥ 0.5 → 0.5 − 𝑟 ≤ 0

 𝑛π𝑟 2 ≤ 2000 → 𝑛π𝑟 2 − 2000 ≤ 0

 (2-D Problem, n,r)

 Example 2: Modeling
 Design a beer mug that will hold max amount of beer.
– Height and Radius must be no more than 20 𝑐𝑚

– Radius must be at least 5 𝑐𝑚

– Surface area excluding the bottom must be no more than 900 𝑐𝑚2

 Solution:

Maximize 𝑉 = π𝑟 2 h

 Subject to: h ≤ 20

 5 ≤ 𝑟 ≤ 20
2𝜋𝑟ℎ ≤ 900
 Example 3: Modeling
 Design a circular tank with minimum cost.
 1. V (volume) = 250𝑚3

 2. Fabrication costs from sheet metal = 400/𝑚2

𝐷
 3. Tank has to fit in a shed with the restriction 𝐻 ≤ 10 − 2

 Solution:

𝜋D2
Minimize 𝐶 = 400 2𝜋𝑅𝐻 + 2𝜋𝑅 2 = 400(𝜋𝐷𝐻 + )
2

𝐷2 𝐷2
Subject to: 𝜋 𝐻 = 250 → 𝜋 𝐻 − 250 = 0
4 4

𝐷 𝐷
 H ≤ 10 − → 𝐻 + − 10 ≤ 0
2 2
 Review from Calculus
 1-Dimensional (1-D) Problems with no constraints (Minimize /Maximize
𝑓(𝑥) subject to no constraints)
 Min/Max occurs at Stationary Points: where the function “stops” increasing or
decreasing (first derivative is equal to zero)

• Necessary Condition:

𝑑𝑓
• Requirement for a stationary point is =0
𝑑𝑥

• Sufficient Condition:

𝑑2 𝑓
• For a minimum, >0
𝑑𝑥 2

𝑑2 𝑓
• For a maximum, <0
𝑑𝑥 2

𝑑2 𝑓
• Otherwise if = 0, the point is called an inflection point
𝑑𝑥 2
 1-D Taylor’s Expansion
 A 1-D function can be approximated at a point 𝑥 ∗ using Taylor’s
Expansion.

𝑑𝑓 𝑥 ∗ 1 𝑑2𝑓 𝑥∗
𝑓 𝑥 ≈𝑓 𝑥∗ + 𝑥− 𝑥∗ + 𝑥 − 𝑥∗ 2 +𝑅
𝑑𝑥 2 𝑑𝑥 2

Linear Expansion Quadratic Form

2nd Order Expansion

 The error for the 2nd order approximation is less than the error for the
linear approximation as you get further from 𝑥 ∗
Example 4: 1-D Taylor’s Expansion
 Find the linear approximation of 𝑓 𝑥 = 5𝑥 2 + 10 at x∗= 2
 Solution:

𝑑𝑓 𝑥 ∗
𝑓 𝑥 ≈ 𝑓 𝑥∗ + 𝑥 − 𝑥∗
𝑑𝑥
2
= 5 2 + 10 + 10 2 𝑥 − 2

= 20𝑥 − 10
 f(x) and its lineaer approximation at X=2
60

50

40

30

20
f

Linear approximation at x=2

f(x)
10

0
0 0.5 1 1.5 2 2.5 3 3.5
-10

-20
x
 Example 5: 1-D Taylor’s Expansion
 Find the linear approximation of 𝑓 𝑥 = sin 𝑥 at x = 0
 Solution:

𝑑𝑓 𝑥 ∗
𝑓 𝑥 ≈ 𝑓 𝑥∗ + 𝑥 − 𝑥∗
𝑑𝑥
sin 𝑥 ≈ sin 0 + cos 0 𝑥 − 0

=𝑥
 Example 6: 1-D Taylor’s Expansion
 Find the linear and 2nd order approximation of 𝑓 𝑥 = 𝑐𝑜𝑠 𝑥 at x = 0

 Solution:

∗
𝑑𝑓 𝑥 ∗
𝑓 𝑥 ≈𝑓 𝑥 + 𝑥 − 𝑥∗
𝑑𝑥
c𝑜𝑠 𝑥 ≈ 𝑐𝑜𝑠 0 + [−𝑠𝑖𝑛 0 𝑥 − 0 ] = 1

∗
𝑑𝑓 𝑥 ∗ ∗
1 𝑑2𝑓 𝑥 ∗
𝑓 𝑥 ≈𝑓 𝑥 + 𝑥−𝑥 + 𝑥 − 𝑥∗ 2
𝑑𝑥 2 𝑑𝑥 2
1 2
1
cos 𝑥 ≈ 1 + [− cos 0 ] 𝑥 − 0 = 1 − 𝑥2
2 2
Example 7: Taylor’s Expansion
 𝑓 𝑥 = 2𝑥 3 + 5𝑥 2 + 2𝑥 + 1
Find the 1st and 2nd order approximation of this function around 𝑥 = 2

 Solution:
𝑑𝑓(𝑥) 𝑑 2 𝑓(𝑥)
= 6𝑥 2 + 10𝑥 + 2, = 12𝑥 + 10
𝑑𝑥 𝑑𝑥 2

First-order:
𝑑𝑓
𝑓 𝑥 ≈𝑓 2 + 2 𝑥−2
𝑑𝑥
𝑓 𝑥 ≈ 41 + 46(𝑥 − 2)
Second-order:

𝑑𝑓 1 𝑑 2 𝑓(2) 2
𝑓(𝑥) ≈ 𝑓 2 + 2 𝑥−2 + 𝑥−2
𝑑𝑥 2 𝑑𝑥 2
1 2
= 41 + 46 𝑥 − 2 + 12 2 + 10 𝑥 − 2
2
 Taylor’s Expansion (cont.)
 At stationary point, the linear portion is equal to zero, since
the first derivative is zero.
 𝑓 𝑥 = 𝑓(𝑥 ∗ ) at 𝑥 ∗ for linear approximation

 Since x − x ∗ 2 is always a positive number in the quadratic

portion, the second derivative must be less than zero for a
maximum (decrease on both sides of the maximum point) and
greater than zero for a minimum (increase on both sides of the
minimum point).
 Gradient of a Function
 The gradient
 A vector that shows the direction of greatest increase of the function at a
particular point.

 Perpendicular to the tangent of the function at that point (if the function is a line,
its gradient at any point is perpendicular to it).

 Gradient of a multi-dimensional function (n variables) 𝑓(𝑥) is:

𝜕𝑓
𝜕𝑥1
𝜕𝑓
𝜕𝑥2
𝛻𝑓 𝑥 = : 𝛻 = "𝑑𝑒𝑙"
:
𝜕𝑓
𝜕𝑥𝑛
 Example 8: Gradient
 Find the gradient of the following function at (3,3).
2 2
𝑓 𝑥 = 𝑥1 − 2 + 𝑥2 − 2
 Solution:
• This is the equation of a circle: 𝑅2 = 𝑥1 − 𝑥1𝑐 2
+ 𝑥2 − 𝑥2𝑐 2

• 𝑥1 and 𝑥2 are variables, 𝑥1𝑐 and 𝑥2𝑐 are constants and the coordinates of the center of the circle
(x1c,x2c)

𝑥1 = 3
 Let (point 1) → (3,3)
𝑥2 = 3
𝑓 3,3 = 2, so radius of the circle r = 2

2 𝑥1 − 2
𝛻𝑓 =
2(𝑥2 − 2)

2
𝛻𝑓 3,3 =
2
The vector starts at the given point and ends 2 units up and 2 units over from that point.
 Example 9: Gradient
 Find and plot the gradient of the following function at (1,2).
𝑓 𝑥1 , 𝑥2 = 𝑥12 + 𝑥22 + 3𝑥1 𝑥2
 Solution:
𝜕𝑓
= 2𝑥1 + 3𝑥2
𝜕𝑥1

𝜕𝑓
= 2𝑥2 + 3𝑥1
𝜕𝑥2

2𝑥1 + 3𝑥2
𝛻𝑓 =
2𝑥2 + 3𝑥1

8
𝛻𝑓 1,2 =
7
 Example 10: Gradient
𝜋
 Find and plot the gradient of the following function at ( , 𝜋).
2

𝑓 𝑥1 , 𝑥2 = 𝑠𝑖𝑛𝑥1 + 𝑠𝑖𝑛𝑥2
 Solution:
𝜕𝑓
= 𝑐𝑜𝑠𝑥1
𝜕𝑥1

𝜕𝑓
= 𝑐𝑜𝑠𝑥2
𝜕𝑥2
𝑐𝑜𝑠𝑥1
𝛻𝑓 = 𝑐𝑜𝑠𝑥
2

𝜋 0
𝛻𝑓 ,𝜋 =
2 −1
 Example 11: Gradient
 Find and plot the gradient of the following function at (3,4).
𝑓 𝑥1 , 𝑥2 = 𝑥1 − 2 2 + 𝑥2 − 3 2

 Solution:
𝜕𝑓
= 2(𝑥1 − 2)
𝜕𝑥1

𝜕𝑓
= 2(𝑥2 − 3)
𝜕𝑥2

2(𝑥1 − 2)
𝛻𝑓 =
2(𝑥2 − 3)

2
𝛻𝑓 3,4 =
2
 Graphical Optimization
Steps:
Plot the constraints and identify the Feasible
Domain (collection of all acceptable solutions)
Plot the contours of the objective functions
Determine the direction of increase or decrease of
the function
Identify the optimal solution by inspection
Example 12: Graphical Optimization

Minimize 𝑓 x1 , x2 = 𝑥1 − 5 2 + 𝑥2 − 5 2

s.t. 𝑥1 , 𝑥2 ≥ 0

𝑥1 + 𝑥2 ≤ 3
The optimal solution occurs at the point (1.5,1.5) with a value
of 2(1.5 – 5)2 = 24.5
 Example 13: Graphical Optimization

 Maximize 𝑓 x1 , x2 = 6x1 + 9x2

s.t. 𝑥1 , 𝑥2 ≥ 0

𝑥1 + 𝑥2 ≤ 5
 Solution:

Assume 𝑓 = 9

9 = 6𝑥1 + 9𝑥2

𝑖𝑓 𝑥1 = 0 → 𝑥2 = 1

𝑖𝑓 𝑥2 = 0 → 𝑥1 = 1.5
𝑥 ∗ = (0,5)
𝑥 ∗ = (0,5)
 Example 14: Graphical Optimization
 Using the graphical optimization, determine the solution of the
following problem.
2 2
Minimize 𝑓 𝑥, 𝑦 = 𝑥 − 1 + 𝑦−1
s.t. 𝑥 + 𝑦 ≥ 4
𝑥 + 2𝑦 ≤ 8
𝑥, 𝑦 ≥ 0
 Solution:
– 𝑥+𝑦 =4

– 𝑎𝑡 𝑥 = 0 → 𝑦 = 4

– 𝑎𝑡 𝑦 = 0 → 𝑥 = 4

– 𝑥 + 2𝑦 = 8

– 𝑎𝑡 𝑥 = 0 → 𝑦 = 4

– 𝑎𝑡 𝑦 = 0 → 𝑥 = 8
– Plot the contours of 𝑓, starting with 𝑓 = 1 and increase the value of 𝑓 until the
optimal point is found.
𝑥 ∗ = 2,2 𝑓∗ = 1 + 1 = 2
𝑥 ∗ = 2,2 𝑓∗ = 1 + 1 = 2
 Example 15: Graphical Optimization

2 2
 Maximize 𝑓 𝑥1 , 𝑥2 = 𝑥1 − 2 + 𝑥2 − 3
 s.t. 𝑥1 ≤ 10

𝑥2 ≤ 5

𝑥1 + 𝑥2 ≤ 4

𝑥1 , 𝑥2 ≥ 0
Example 16: Graphical Optimization

 Maximize 𝑓 = 2x + 3y
 s.t. 𝑥≤5

 𝑦≤3

 𝑥, 𝑦 ≥ 0
Example 17: Graphical Optimization

2 2
 Minimize 𝑓 = 𝑥 − 5 + 𝑦−6
 s.t. 𝑥+𝑦 ≤4

 𝑥, 𝑦 ≥ 0
Example 18: Graphical Optimization

 Minimize 𝑓 = 5𝑥 + 2𝑦
 s.t. 𝑥+𝑦 ≤4

 𝑥, 𝑦 ≥ 0
Example 19: Graphical Optimization

 Maximize 𝑓 = 𝑥 + 2𝑦
 s.t. 𝑥 + 𝑦 ≤ 10

 4≤𝑥≤8

 0≤𝑦≤6
 Example 20: Graphical Optimization

 Minimize 𝑓(𝑥, 𝑦) = 𝑥 + 𝑦
 s.t. 𝑥−1 2 + 𝑦−1 2 ≤4

 𝑥, 𝑦 ≥ 0
 Example 21: Graphical Optimization

2 2
 Minimize 𝑓 𝑥, 𝑦 = 𝑥 − 4 + 𝑦−4
 s.t. 𝑥 + 𝑦 ≤ 4

 𝑥 + 2𝑦 ≤ 6
𝑥, 𝑦 ≥ 0
 Taylor’s Expansion for a Multi-
Dimensional Problem
𝑇 1 𝑇
𝑓 𝑥 ≈𝑓 𝑥∗ + 𝛻𝑓 𝑥 ∗ 𝑥− 𝑥∗ + 𝑥− 𝑥 ∗ 𝐻 𝑥 − 𝑥∗ + 𝑅
2

 T = transpose, interchanging columns and rows of a matrix

 [H] = Hessian Matrix, square matrix of second order partial
derivatives of the function (calculated at 𝑥 ∗ )
• 2-D 3-D
𝜕2 𝑓 𝜕2 𝑓 𝜕2 𝑓
𝜕2 𝑓 𝜕2 𝑓 𝜕𝑥12 𝜕𝑥1 𝜕𝑥2 𝜕𝑥1 𝜕𝑥3
𝜕𝑥12 𝜕𝑥1 𝜕𝑥2 𝜕2 𝑓 𝜕2 𝑓 𝜕2 𝑓
𝜕2 𝑓 𝜕2 𝑓 𝜕𝑥2 𝜕𝑥1 𝜕𝑥22 𝜕𝑥2 𝜕𝑥3
𝜕𝑥2 𝜕𝑥1 𝜕𝑥22 𝜕2 𝑓 𝜕2 𝑓 𝜕2 𝑓
𝜕𝑥3 𝜕𝑥1 𝜕𝑥3 𝜕𝑥2 𝜕𝑥32
 R = remainder
 𝑥 − 𝑥 ∗ can also be written as 𝑑, a vector of difference, 𝑑 = 𝑥 − 𝑥 ∗
 Example 22 Taylor’s Expansion
 𝑓 𝑥, 𝑦, 𝑧 = 3𝑥2 + 2𝑥𝑦 2 + 6𝑦 3 + 9𝑧 3 + 3𝑥𝑧 + 9𝑦 2 𝑧
 Find 𝛻𝑓 𝑥 ∗ , 𝑥 ∗ = 1,3,2 and [𝐻]
 Solution:

6𝑥 + 2𝑦 2 + 3𝑧
𝛻𝑓 = 4𝑥𝑦 + 18𝑦 2 + 18𝑦𝑧
27𝑧 2 + 3𝑥 + 9𝑦 2

30
∗
𝛻𝑓 𝑥 ≈ 282
192

6 4𝑦 3
𝐻 = 4𝑦 4𝑥 + 36𝑦 + 18𝑧 18𝑦
3 18𝑦 54𝑧
 Example 23: Taylor’s Expansion
2 2
 𝑓 𝑥 = 𝑥1 − 3 + 𝑥2 − 2
Find 1st order approximation at 𝑓(4,4)
 Solution:
𝜕𝑓
𝜕𝑥1 2 𝑥1 − 3
𝛻𝑓 = =
𝜕𝑓 2 𝑥2 − 2
𝜕𝑥2
2 4−3 2
𝛻𝑓 𝑥 ∗ = =
2 4−2 4
𝑥1 4 𝑥 −4
𝑥 − 𝑥∗ = 𝑥 − = 1
2 4 𝑥2 − 4
𝑥1 − 4
𝑓 𝑥 ≈ 𝑓 𝑥 ∗ + 𝛻𝑓 𝑥 ∗ 𝑇
𝑥 − 𝑥 ∗ = 5 + [2 4]
𝑥2 − 4
= 5 + 2 𝑥1 − 4 + 4 𝑥2 − 4 = 5 + 2𝑥1 − 8 + 4𝑥2 − 16
= 2𝑥1 + 4𝑥2 − 19
 Example 24: Taylor’s Expansion
 𝑓(𝑥, 𝑦) = 5𝑥2𝑦 + 3𝑥3 + 2𝑥𝑦 + 4𝑦2𝑥

 Find 1st order approximation at 𝑓(2,3)

 Solution:

10𝑥𝑦 + 9𝑥 2 + 2𝑦 + 4𝑦 2 138
𝛻𝑓 = =
5𝑥 2 + 2𝑥 + 8𝑥𝑦 72

𝑥−2
𝑓 𝑥 ≈ 168 + 138 72
𝑦−3

𝑓 𝑥 ≈ 168 + 138 𝑥 − 2 + 72 𝑦 − 3
= 138𝑥 + 72𝑦 − 324
 Example 24 (cont.)
 Now, find 2nd order approximation at 𝑓(2,3)
 Solution:
10𝑦 + 18𝑥 10𝑥 + 8𝑦 + 2 66 46
• 𝐻 = =
10𝑥 + 8𝑦 + 2 8𝑥 46 16

1 66 46 𝑥 − 2
• 𝑄𝑢𝑎𝑑𝑟𝑎𝑡𝑖𝑐 𝑝𝑜𝑟𝑡𝑖𝑜𝑛 = 𝑥−2 𝑦−3
2 46 16 𝑦 − 3

1 66 𝑥 − 2 + 46(𝑦 − 3)
= 𝑥 − 2 , (𝑦 − 3
2 46 𝑥 − 2 + 16(𝑦 − 3)

1 2 2
= 66 𝑥 − 2 + 46 𝑥 − 2 𝑦 − 3 + 46 𝑥 − 2 𝑦 − 3 + 16 𝑦 − 3
2
2 2
= 33 𝑥 − 2 + 46 𝑥 − 2 𝑦 − 3 + 8 𝑦 − 3
• Summing the 1st order approximation, the 2nd order approximation:
2 2
138𝑥 + 72𝑦 − 324 + 33 𝑥 − 2 + 46 𝑥 − 2 𝑦 − 3 + 8 𝑦 − 3
 Example 25: Taylor’s Expansion
 𝑓 𝑥, 𝑦 = 5𝑥 3 + 2𝑥𝑦 + 9𝑥𝑦 2 + 3𝑦 2
Find the quadratic approximation of this function at (1,1)
 Solution:
𝑓 1,1 = 5 + 2 + 9 + 3 = 19

15𝑥 2 + 2𝑦 + 9𝑦 2 26
𝛻𝑓 = → 𝛻𝑓 1,1 =
2𝑥 + 18𝑥𝑦 + 6𝑦 26

30𝑥 2 + 18𝑦 30 20
𝐻 = →
2 + 18𝑦 18𝑥 + 6 20 24

𝑇 1 𝑇
𝑓 𝑥 ≈ 𝑓 𝑥 ∗ + 𝛻𝑓 𝑥 ∗ 𝑥 − 𝑥∗ + 𝑥 − 𝑥∗ 𝐻 𝑥 − 𝑥∗
2

𝑥−1 1 20 𝑥 − 1
𝑓 𝑥, 𝑦 ≈ 19 + 26 26 𝑦 − 1 + [𝑥 − 1 𝑦 − 1] 30
2 20 24 𝑦 − 1
= 19 + 26 𝑥 − 1 + 26 𝑦 − 1

1 2 2
+ 30 𝑥 − 1 + 20 𝑥 − 1 𝑦 − 1 + 20 𝑦 − 1 𝑥 − 1 + 24 𝑦 − 1
2
 Example 26: Stationary Points
 𝑓 𝑥 = 5𝑥 2 − 20𝑥 + 1
Find the stationary points for the function and determine the
natural of these points (max or min)

 Solution:
𝑑𝑓
=0
𝑑𝑥

𝑑𝑓
= 10𝑥 − 20 = 0 → 𝑥 = 2
𝑑𝑥

𝑑2 𝑓
= 10 𝑝𝑜𝑠𝑖𝑡𝑖𝑣𝑒 → 𝑚𝑖𝑛
𝑑𝑥 2
 Multi-Dimensional Functions
 Necessary condition for stationary points is 𝛻𝑓 = 0

 Sufficient condition is that [𝐻] matrix is positive definite for a minimum and
negative definite for a maximum

 How to determine if a matrix is positive definite, negative definite, or indefinite

 Method 1:

• [H] is positive definite if the determinant of the upper left 1x1 matrix, upper left 2x2
matrix, and so on, plus the matrix itself, is positive (all the leading principal minors
are positive).

• [H] is negative definite if the determinant of the principal minors alternate between
negative and positive, starting with negative (the 1x1 matrix) (odd matrices have
negative determinant, even ones have positive determinant).

• Otherwise, the result is indeterminate

 Multi-Dimensional Functions
(cont.)
 Method 2: Using Eigenvalues (𝜆)
• Find λ through the relationship:

𝐻 −λ 𝐼 =0

1 0 0
𝐼 = 𝑡ℎ𝑒 𝑖𝑑𝑒𝑛𝑡𝑖𝑡𝑦 𝑚𝑎𝑡𝑟𝑖𝑥 = 0 1 0
0 0 1

• If all λ’s are positive, 𝐻 is positive definite, if all λ’s are negative, 𝐻 is negative definite,
otherwise, the matrix is indefinite.
 Example 27: Hessian Matrix

 Determine the nature of the following Hessian Matrix

5 2 5>0 positive definite

𝐻 = 30 − 4 = 26 > 0 minimum stationary point
2 6

−5 6 −5 < 0
𝐻 = indeterminate
6 2 −10 − 36 = −46 < 0

−4 3 −4 < 0 negative definite

𝐻 = maximum stationary point
1 −2 8−3=5>0
Examples 28: Hessian Matrix (Eigen values)

 Find the Eigen values of the following matrix

5 2
2 6
 Solution:
5 2 1 0 5−λ 2
𝐻 −λ 𝐼 = −λ = =0
2 6 0 1 2 6−λ

5−λ 6−λ −4=0

λ2 − 11λ + 26 = 0

11 ± (112 − 4(1)(26)
𝜆= = 5.5 ± 2.1
2

Both λ’s positive, so Hessian is positive definite.

Examples 29: Hessian Matrix (Eigen values)

 Find the Eigen values of the following matrix

2 1
1 3
 Solution:
2 1 1 0 2−λ 1
𝐻 −λ 𝐼 = −λ = =0
1 3 0 1 1 3−λ

𝑑𝑒𝑡 = 2 − λ 3 − λ − 1 = 0

λ2 − 5λ + 5 = 0

5 ± (5)2 − 4(1)(5)
𝜆= = 2.5 ± 1.12
2

Both λ’s positive, so Hessian is positive definite.

 Example 30: Stationary Points
 𝑓 𝑥, 𝑦 = 5𝑥 2 − 2𝑥𝑦 + 9𝑦 2
Find the stationary points for the function

 Solution:
𝜕𝑓
= 10𝑥 + 2𝑦 = 0
𝜕𝑥

𝜕𝑓
= 2𝑥 + 18𝑦 = 0
𝜕𝑦

𝑥=0
(𝑜𝑛𝑒 𝑠𝑡𝑎𝑡𝑖𝑜𝑛𝑎𝑟𝑦 𝑝𝑜𝑖𝑛𝑡)
𝑦=0
 Example 31: Stationary Points
 Find the stationary points for the function and determine if they are
minimum, maximum, or inflection points?
𝑓 𝑥 = 5𝑥12 + 3𝑥22 − 2𝑥1 𝑥2

 Solution:
𝛻𝑓 = 0

𝜕𝑓
= 10𝑥1 − 2𝑥2 = 0
𝜕𝑥

𝜕𝑓
= 6𝑥2 − 2𝑥1 = 0
𝜕𝑦

𝑥1 = 0
𝑥2 = 0

10 −2 10 > 0
𝐻 =
−2 6 60 − 4 = 56 > 0

H is positive definite, there fore (0,0) is a minimum.

 Quadratic Form
 The quadratic form is a function that is made of 2nd order
components (the sum of the powers in every term equals to 2)
 2-D: 𝑎𝑥 2 + 𝑏𝑦 2 + 𝑐𝑥𝑦 (a, b, and c are constants; x, y, and z are variables)

 3-D: 𝑎𝑥 2 + 𝑏𝑦 2 + 𝑐𝑧 2 + 𝑑𝑥𝑦 + 𝑒𝑥𝑧 + 𝑓𝑦𝑧 (a, b, c, d, e, and f are constants;

x, y, and z are variables)
• The nature of a quadratic form is its sign in a region

• The nature of a quadratic form can be found by separating the function into
the product of vectors and a matrix as shown below:

1
𝑎 𝑐 𝑥
2
• 2-D: 𝑥 𝑦 1 𝑦
𝑐 𝑏
2
 Quadratic Form (cont.)

1 1
𝑎 𝑑 𝑒
2 2 𝑥  Note: A symmetric matrix must be used,
1 1
• 3-D: 𝑥 𝑦 𝑧 𝑑 𝑏 𝑓 𝑦 which only occurs in these cases
2 2
1 1 𝑧
𝑒 𝑓 𝑐
2 2

• If the center matrix is positive definite, the function is positive everywhere,

if negative definite, the function is negative everywhere

 A quadratic form is definite if it has only one sign over the entire range

of acceptable variables, otherwise it is indefinite.

 Thus, Hessian rules apply to these matrices

 Example 32: Quadratic Form
 Find the nature of this function.

𝑓 𝑥, 𝑦 = 5𝑥 2 + 4𝑥𝑦 + 6𝑦 2
 Solution:

• To find the nature of the quadratic function, we have to find the 𝐻 ,

• if 𝐻 is positive definite, function is positive everywhere,

• if 𝐻 is negative definite, function is negative everywhere,

• If 𝐻 is indeterminate, function has different signs on different domains.

10𝑥 + 4𝑦
𝛻𝑓 = =0
4𝑥 + 12𝑦

10 4
𝐻 =
4 12
ℎ1 = 10 > 0

ℎ2 = 10 12 − 4 4 = 104 > 0

𝐻 is positive definite, function is positive everywhere.

 Example 33: Quadratic Functions

 What is the nature of the quadratic function:

𝑓 𝑥1 , 𝑥2 = 𝑥12 + 2𝑥22 + 3𝑥1 𝑥2
 Solution:
𝑇 1 1 𝑥1
𝑥 (𝑥) → 𝑥1 , 𝑥2
2 2 𝑥2

1 1.5 ℎ1 = 2 > 0
𝐻 = ℎ2 = 2 − 2.25 = 0.25 < 0
1.5 2
Indefinite

The sign for 𝑓 at various locations may be positive or negative

 Example 34: Quadratic Functions

 What is the nature of the quadratic function:

𝑓 𝑥1 , 𝑥2 = 2𝑥12 + 5𝑥22 − 2𝑥1 𝑥2
 Solution:
𝑇 2 −1 𝑥1
𝑥 (𝑥) → 𝑥1 , 𝑥2
−1 5 𝑥2

2 −1 ℎ1 = 2 > 0
𝐻 =
−1 5 ℎ2 = 10 − 1 = 9 > 0

Definite.

Function is positive everywhere.

 Example 35: Quadratic Form
 What is the nature of the quadratic form:
𝑓 𝑥, 𝑦 = 3𝑥 2 + 6𝑥𝑦 + 𝑦 2

 Solution:
𝑇 3 3 𝑥
𝑥 𝑥 → [𝑥 𝑦]
3 1 𝑦
3 3 ℎ1 = 3 > 0
𝐻 =
3 1 ℎ2 = 3 − 9 = −6 < 0

Indeterminate
 Example 36: Quadratic Form
 What is the nature of the quadratic form:

𝑓 𝑥, 𝑦, 𝑧 = 6𝑥 2 + 3𝑥𝑦 + 2𝑥𝑧 + 3𝑦 2 + 𝑦𝑧 + 𝑧 2
 Solution:
3
6 1
2
𝑇 3 𝑥
1 𝑦
𝑥 (𝑥) → 𝑥 𝑦 𝑧 3
2 2 𝑧
1
1 1
2

3 ℎ1 = 6 > 0
6 1
2 3 3
3 1 ℎ2 = 6 3 − = 15.75 > 0
𝐻 = 3 2 2
2 2
1 1 3
1 1 3 3 1 3
2 ℎ3 = 6 2
−
3
2 2 +1 2
=6 3−4 −2
1 3 3 1
−2 +1
3
− 3 = 16.5 − 1.5 −
1 2 1 2 4
1 1 1 1
2 2
2.25 = 12.75 > 0

Positive Definite
Example 37: Quadratic Functions
 What is the nature of the quadratic function:
𝑓 𝑥1 , 𝑥2 = 2𝑥12 + 𝑥1 𝑥2 + 2𝑥22 + 3𝑥32 − 2𝑥1 𝑥3
 Solution:

𝑇
2 0.5 −1 𝑥1
𝑥 (𝑥) → 𝑥1 𝑥2 𝑥3 0.5 2 0 𝑥2
−1 0 3 𝑥3

ℎ1 = 2 > 0
2 0.5 −1 ℎ2 = 4 − 0.25 = 3.75 > 0
𝐻 = 0.5 2 0 ℎ3 = 2
2 0
− 0.5
0.5 0
+ −1
0.5 2
−1 0 3 0 3 −1 3 −1 0
= 2 6 − 0 − (0.5) 1.5 − 0 + (−1) 0 + 2 = 12 − 0.75 − 2 = 9.25 > 0

Definite.
Function is positive everywhere.
 Example 38: Quadratic Form
 Find the stationary point and it’s nature for the given function.

𝑓 𝑥, 𝑦 = 2𝑥𝑦 + 5𝑥 2 + 6𝑦 2
 Solution:

• 𝛻𝑓 = 0 𝑎𝑡 𝑠𝑡𝑎𝑡𝑖𝑜𝑛𝑎𝑟𝑦 𝑝𝑜𝑖𝑛𝑡

2𝑦 + 10𝑥
𝛻𝑓 = =0
2𝑥 + 12𝑦

2𝑥 + 12𝑦 = 0

2𝑦 + 10𝑥 = 0

𝑥=0
→
𝑦=0

𝑇ℎ𝑒 𝑠𝑡𝑎𝑡𝑖𝑜𝑛𝑎𝑟𝑦 𝑝𝑜𝑖𝑛𝑡 𝑖𝑠 (0,0)

• To find the nature of the stationary point, we must find the 𝐻

10 2
𝐻 =
2 12
ℎ1 = 10 > 0 ℎ2 = 10 12 − 2 2 = 116 > 0

𝐻 𝑖𝑠 𝑝𝑜𝑠𝑖𝑡𝑖𝑣𝑒 𝑑𝑒𝑓𝑖𝑛𝑖𝑡𝑒 → 0,0 𝑖𝑠 𝑚𝑖𝑛𝑖𝑚𝑢𝑛

 Example 39: Quadratic Form
 Find the nature of this function.
𝑓 𝑥, 𝑦 = 3𝑥 2 + 6𝑥𝑦 + 𝑦 2
 Solution:

6𝑥 + 6𝑦
𝛻𝑓 = =0
6𝑥 + 2𝑦

6 6
𝐻 =
6 2
ℎ1 = 6 > 0
ℎ2 = 6 2 − 6 6 = −24 < 0
𝐻 is indeterminate, function has different signs.
Optimization of Constrained Problems

 Form a new function called the Lagrangian (L)

𝐿 = 𝑓 𝑥 + 𝑣ℎ(𝑥)

 𝑣 = "𝑛𝑢“, the Lagrange Multiplier (treated as a new variable)

 𝛻𝐿 = 0 for stationary points

 Example 40: Lagrangian
2
 Minimize 𝑓 = 𝑥 − 4 + 𝑦 − 4 2 , s.t. ℎ 𝑥, 𝑦 = 𝑥 + 𝑦 − 4
 Solution:
• First we solve by plot.
𝑥 ∗ = 2, 𝑦∗ = 2
𝑓∗ = 2 − 4 2
+ 2−4 2
=8
• Then we solve using Lagrange multiplier.
𝐿 = 𝑓 𝑥, 𝑦 + 𝑣ℎ 𝑥, 𝑦
2 2
𝐿 = 𝑥−4 + 𝑦−4 +𝑣 𝑥+𝑦−4
𝛻𝐿 = 0

𝜕𝐿
=2 𝑥−4 +𝑣 =0
𝜕𝑥

𝜕𝐿
=2 𝑦−4 +𝑣 =0
𝜕𝑦

𝜕𝐿
=𝑥+𝑦−4=0
𝜕𝑣
𝑥 ∗ = 2, 𝑦 ∗ = 2, 𝑣=4
𝑓∗ = 2 − 4 2
+ 2−4 2
=8

−4 1
• 𝛻𝑓 2,2 = 𝛻ℎ 2,2 =
−4 1

• 𝛻𝑓 𝑥 ∗ = −𝑣𝛻ℎ(𝑥 ∗ )
 Example 41: Lagrangian

 Minimize 𝑓(𝑥, 𝑦) = 𝑥 − 5 2 + 𝑦−5 2

s.t. 𝑥 + 2𝑦 − 4 = 0
2𝑥 + 𝑦 − 4 = 0
 Solution:
From the plot
𝑥 ∗ = 1.33, 𝑦 ∗ = 1.33
𝑓 ∗ = 26.94
• Using Lagrange multiplier.
𝐿 = 𝑓 𝑥, 𝑦 + 𝑣1 ℎ1 𝑥, 𝑦 + 𝑣2 ℎ2 (𝑥, 𝑦)
2 2
𝐿 = 𝑥−5 + 𝑦−5 + 𝑣1 𝑥 + 2𝑦 − 4 + 𝑣2 (2𝑥 + 𝑦 − 4)
𝛻𝐿 = 0

𝜕𝐿
= 2 𝑥 − 5 + 𝑣1 + 2𝑣2 = 0
𝜕𝑥

𝜕𝐿
= 2 𝑦 − 5 + 2𝑣1 + 𝑣2 = 0
𝜕𝑦

𝜕𝐿
= 𝑥 + 2𝑦 − 4 = 0
𝜕𝑣1

𝜕𝐿
= 2𝑥 + 𝑦 − 4 = 0
𝜕𝑣2

𝑥 ∗ = 1.33, 𝑦 ∗ = 1.33, 𝑣1 = 2.46, 𝑣2 = 2.44

𝑓 ∗ = 1.33 − 4 2
+ 1.33 − 4 2
= 26.94
 Example 42: Lagrangian
2
 Minimize 𝑓(𝑥, 𝑦) = 𝑥 − 4 + 𝑦 − 4 2,
s.t. 𝑥 + 𝑦 = 2
 Solution:
• Convert to standard format: ℎ 𝑥, 𝑦 = 𝑥 + 𝑦 − 2 = 0
• L = (x – 4)2 + (y – 4)2 + ν(x + y – 2)

2 x−4 +v
• 𝛻𝐿 = 2 y − 4 + v = 0
x+y−2

• 3 equations, 3 unknowns, 𝑥 = 1, 𝑦 = 1, 𝑣 = 6
• 𝑓∗ = 1 − 4 2
+ 1−4 2
= 18
• The optimal point (found graphically) is (1,1)
−6 1
• 𝛻𝑓 1,1 = 𝛻ℎ 1,1 =
−6 1

• 𝛻𝑓 𝑥 ∗ = −𝑣𝛻ℎ(𝑥 ∗ )

• For a problem with two equality constraints,

𝛻𝑓 𝑥 ∗ = −𝑣1 𝛻ℎ1 𝑥 ∗ − 𝑣2 𝛻ℎ2 (𝑥 ∗ )
 Example 43: Lagrangian
 Minimize the surface area of a cylindrical storage tank with fixed volume of V

 Solution 1:
minimize: SA = 2π𝑅𝐻 + 2π𝑅 2 → 𝑓 𝑅 = 𝑅𝐻 + 𝑅 2

s.t. V = π𝑅 2 𝐻 → π𝑅 2 𝐻 − 𝑉 = 0

𝑉
𝐻=
𝜋𝑅 2
𝑉
𝑓 𝑅 = + 𝑅2
π𝑅
𝑑𝑓 𝑉
= − 2 + 2𝑅 = 0
𝑑𝑅 π𝑅

1
𝑉 3
𝑅∗ =
2π

1 3
𝑉 𝑉 3 22
𝐻∗ = 2
= 1
π𝑅
π3

𝑑2 𝑓 2𝑉 2𝑉
=2+ 3 =2+ =6>0
𝑑𝑅 2 π𝑅 𝑉
π 2π

𝑅 ∗ , 𝐻 ∗ 𝑖𝑠 𝑎 𝑚𝑖𝑛𝑖𝑚𝑢𝑚
 Solution 2 (Lagrangian):
minimize: 𝑓 𝑅, 𝐻 = 𝑅𝐻 + 𝑅2

s.t. ℎ 𝑅, 𝐻 = π𝑅2 𝐻 − 𝑉 = 0
ϕ(𝑅, 𝐻, 𝑣) = 𝑓(𝑅, 𝐻) + 𝑣ℎ(𝑅, 𝐻)

𝜕ϕ
=0 𝜕𝑓 𝜕ℎ
𝜕𝑅 +𝑣 =0
𝜕ϕ 𝜕𝑅 𝜕𝑅
𝛻ϕ = 0 → = 0 → 𝜕𝑓 𝜕ℎ
𝜕𝐻 +𝑣 =0
𝜕ϕ 𝜕𝐻 𝜕𝐻
=0 ℎ(𝑅, 𝐻) = 0
𝜕𝑣

𝜕ϕ
= 𝐻 + 2𝑅 + 𝑣 2π𝑅𝐻 = 0
𝜕𝑅
𝜕ϕ
= 𝑅 + 𝑣(π𝑅2 ) = 0
𝜕𝐻
𝜕ϕ
= π𝑅2 𝐻 − 𝑉 = 0
𝜕𝑣
Three equations, three unknowns.
 What is the Lagrange Multiplier?
 Minimize 𝑓(𝑥, 𝑦) s.t. ℎ 𝑥, 𝑦 = 0
𝑑𝑦 𝑑ϕ
𝑦 = ϕ 𝑥 → =
𝑑𝑥 𝑑𝑥

𝑑ℎ 𝜕ℎ 𝜕ℎ 𝑑ϕ 𝑑ϕ 𝜕ℎ 𝜕ℎ
 = + ∙ =0 → =− /
𝑑𝑥 𝜕𝑥 𝜕𝑦 𝑑𝑥 𝑑𝑥 𝜕𝑥 𝜕𝑦

𝑑𝑓 𝜕𝑓 𝜕𝑓 𝑑𝑦 𝜕𝑓 𝜕𝑓 𝑑ϕ
 = + ∙ = + ∙
𝑑𝑥 𝜕𝑥 𝜕𝑦 𝑑𝑥 𝜕𝑥 𝜕𝑦 𝑑𝑥

𝜕𝑓 𝜕𝑓 𝜕ℎ 𝜕ℎ 𝜕𝑓 𝜕ℎ 𝜕𝑓 𝜕ℎ
• = + − / = + − /
𝜕𝑥 𝜕𝑦 𝜕𝑥 𝜕𝑦 𝜕𝑥 𝜕𝑥 𝜕𝑦 𝜕𝑦

𝜕𝑓 𝜕ℎ
• → 𝑣=− /
𝜕𝑦 𝜕𝑦
 Inequality Constraints
 Standard Format for All Cases:
minimize 𝑓(𝑥)

s.t. 𝑔𝑗 (𝑥) ≤ 0
 Add a slack variable 𝒔𝟐 to each 𝑔(𝑥) ≤ 0 to convert it to an equality
constraint, use 𝑢 as the new Lagrange multiplier

• 𝑔 𝑥 + 𝑠2 = 0

 Now form Lagrangian: ℒ = 𝑓 𝑥 + 𝑢(𝑔 𝑥 + 𝑠 2 )

 For each inequality constraint 𝑔𝑗 , we add two new variables 𝑢𝑗 and𝑠𝑗

 𝑢 must be non-negative (≥ 0)
 𝑠 2 must be non-negative (≥ 0)
Inequality Constraints (cont.)
 Switching Condition:
𝜕𝐿
• = 2𝑢𝑠 = 0
𝜕𝑆

• 𝑠 = 0 𝑎𝑛𝑑/𝑜𝑟 𝑢 = 0

• All possible cases must be checked for the switching condition.

• This is generally the best place to start a problem from.

Example 44: Inequality Constraints
2 2
 minimize 𝑓 𝑥, 𝑦 = 𝑥 − 4 + 𝑦−4

s.t. 𝑥 + 𝑦 − 4 ≤ 0
 Solution:
2 2
• ℒ = 𝑥−4 + 𝑦−4 + 𝑢(𝑥 + 𝑦 − 4 + 𝑠 2 )

𝜕ℒ
=0
𝜕𝑥
𝜕ℒ 2 𝑥−4 +𝑢=0
=0 2 𝑦−4 +𝑢 =0
𝜕𝑦
• 𝛻ℒ = 0 → →
𝜕ℒ
=0 𝑥 + 𝑦 − 4 + 𝑠2 = 0
𝜕𝑢
𝜕ℒ
2𝑢𝑠 = 0
=0
𝜕𝑠
 Example 44 (cont.)
• Case 1: 𝑢 = 0
𝑥=𝑦=4
4 + 4 − 4 + 𝑠2 = 0
𝑠 2 = −4 not acceptable

• Case 2: 𝑠 = 0
𝑥∗ = 𝑦∗ = 2
𝑢=4>0 (𝑎𝑐𝑐𝑒𝑝𝑡𝑎𝑏𝑙𝑒)
 Active Constraints
 Active Constraint: an inequality constraint that becomes an
equality at the optimum point
 S must be equal to 0 for a constraint to be active

 If a constraint is known to be inactive, it can be left out of the problem

statement, as it plays no role in the outcome of the problem

 If 𝑠𝑗2 > 0, constraint j is inactive and 𝑢𝑗 = 0 (because of the switching

condition 𝑢𝑗 𝑠𝑗 = 0)
K-K-T (Karush-Khun-Tucker) Conditions

 All problems must first be converted to standard format for rules to apply:
 Minimize 𝑓(𝑥)

s.t. ℎ𝑖 𝑥 = 0 𝑖 = 1,2, … , 𝑛

𝑔𝑗 𝑥 ≤ 0 𝑗 = 1,2, … , 𝑚

 Find 𝛻ℒ = 0 using Lagrange Multipliers 𝑣, 𝑢 and 𝑠 as the additional

variables
 At the optimal point, the K-K-T conditions must be met (necessary conditions)
• Always start at the switching condition for problems involving inequality constraints to
evaluate various cases.
 Example 45: KKT Condition
 Minimize 𝑓 𝑥, 𝑦 = 4𝑥12 + 3𝑥22 − 5𝑥1 𝑥2 − 8

 s.t. 𝑥1 + 𝑥2 ≤ 4
 Solution:
• Convert: 𝑥1 + 𝑥2 − 4 + 𝑠 2 = 0

• ℒ = 4x12 + 3x22 − 5x1 x2 − 8 + u x1 + x2 − 4 + s 2

• K.K.T. Condition:

𝜕ℒ
• = 8𝑥1 − 5𝑥2 + 𝑢 = 0
𝜕𝑥1

𝜕ℒ
• = 6x2 − 5x1 + u = 0
𝜕𝑥2

𝜕ℒ
• = 𝑥1 + 𝑥2 − 4 + 𝑠 2 = 0
𝜕𝑢

𝜕ℒ
• = 2𝑢𝑠 = 0 (𝑠𝑤𝑖𝑡𝑐ℎ𝑖𝑛𝑔 𝑐𝑜𝑛𝑑𝑖𝑡𝑖𝑜𝑛)
𝜕𝑠
• Always start from a switching condition

• Case 1:

• 𝑢=0

8𝑥1 − 5𝑥2 = 0 𝑥1 = 0
• →
6x2 − 5x1 = 0 𝑥2 = 0

• 𝑥1 + 𝑥2 − 4 + 𝑠 2 = 0 → 𝑠 2 = 4

• Acceptable

• It’s not active because 𝑠 ≠ 0

• Case 2:

• 𝑠=0

8𝑥1 − 5𝑥2 + 𝑢 = 0 𝑥1 = 1.833

• 6x2 − 5x1 + u = 0 → 𝑥2 = 2.166
𝑥1 + 𝑥2 − 4 = 0 𝑢 = −3.833
• 𝑢<0

• Not acceptable
 Example 46: KKT Condition
 Minimize 𝑓 𝑥1 , 𝑥2 = 2𝑥1 + 3𝑥2 − 𝑥13 − 2𝑥22
 s.t. 𝑥1 + 3𝑥2 ≤ 6
5𝑥1 + 2𝑥2 ≤ 10
 Solution:
• Convert: 𝑥1 + 3𝑥2 − 6 ≤ 0
• 5𝑥1 + 2𝑥2 − 10 ≤ 0
• ℒ = 2𝑥1 + 3𝑥2 − x13 − 2x22 + u1 x1 + 3x2 − 6 + 𝑠12 + 𝑢2 (5𝑥1 + 2𝑥2 − 10 + 𝑠22 )
• K.K.T. Condition:

𝜕ℒ
• 𝜕𝑥1
= 2 − 3𝑥12 + 𝑢1 + 5𝑢2 = 0

𝜕ℒ
• 𝜕𝑥2
= 3 − 4x2 + 3u1 + 2u2 = 0

𝜕ℒ
• = 𝑥1 + 3𝑥2 − 6 + 𝑠12 = 0
𝜕𝑢1

𝜕ℒ
• = 5𝑥1 + 2𝑥2 − 10 + 𝑠22 = 0
𝜕𝑢2

𝜕ℒ
• 𝜕𝑠1
= 2𝑢1 𝑠1 = 0

𝜕ℒ
• 𝜕𝑠2
= 2𝑢2 𝑠2 = 0
• Case 1:

• 𝑢1 = 0 , 𝑢2 = 0

2 2
2 − 3𝑥12 = 0 𝑥1 = (𝑛𝑒𝑔𝑙𝑒𝑐𝑡 − )
• → 3 3
3 − 4x2 = 0 3
𝑥2 =
4

• 𝑥1 + 3𝑥2 − 6 + 𝑠12 = 0 → 𝑠12 = 2.934 > 0

• 5𝑥1 + 2𝑥2 − 10 + 𝑠22 = 0 → 𝑠22 = 4.42 > 0

• Acceptable

• Both constraints are inactive.

• Other Cases:

• Case 2: Case 3: Case 4:

• 𝑢1 = 0 , 𝑠2 = 0 𝑢2 = 0 , 𝑠1 = 0 𝑠1 = 0 , 𝑠2 = 0
 Homework
 Maximize 𝑓 𝑥1 , 𝑥2 = 𝑥 2 + 2𝑦 2 − 𝑥𝑦 − 4

 s.t. h=𝑥+𝑦 =4

𝑔 = 𝑥2 − 𝑦 ≥ 5
 Solution:
• Convert:

• Minimize 𝑓 𝑥, 𝑦 = −𝑥 2 − 2𝑦 2 + 𝑥𝑦 + 4

• s.t. h=𝑥+𝑦−4=0

• g = −x 2 + 𝑦 + 5 ≤ 0

• ℒ = 𝑓 𝑥, 𝑦 + 𝑣ℎ 𝑥, 𝑦 + 𝑢(𝑔 𝑥, 𝑦 + 𝑠 2 )

• Complete
 Global Minimum
 K-K-T Conditions only provide the local minimum.

 We want to be able to identify if a local minimum is a global

minimum or not.
 Convex Set: A set is convex if any two points within the domain or on
the border are attached by a straight line; that line must be entirely
within the feasible domain

convex not convex

 Global Minimum (cont.)
 A feasible domain is known to be convex if:
 All of the equality constraints are linear

 All of the inequality constraints are convex functions (when they are
in standard format)

 Convex Functions: A function with a positive-definite 𝐻

 𝑓 𝑡𝑥1 + 1 − 𝑡 𝑥2 ≤ 𝑡𝑓 𝑥1 + 1 − 𝑡 𝑓(𝑥2 ) for a 1-D function

 Connect any two points, the portion of the function is completely

below the line.

 A linear function is convex.

x1 x x 2
 If a convex function is defined over a convex set, the K-K-
T Condition would provide a global minimum
 Example 47: Convexity
 Is the set defined by the following constraints convex?
ℎ1 = 3𝑥1 − 5𝑥2 − 1 = 0
ℎ2 = 𝑥12 − 6𝑥2 + 9 = 0
𝑔1 = 𝑥12 + 𝑥22 − 6 ≤ 0
𝑔2 = 𝑥12 − 𝑥22 − 5 ≤ 0
 No, because ℎ2 is nonlinear.

 Is the set defined by the following constraints convex?

ℎ1 = 5𝑥1 − 2𝑥2 − 1 = 0
ℎ2 = 9𝑥1 + 5𝑥2 − 6 = 0
10𝑥1 10 0
𝛻𝑔1 = → 𝐻 = 10 > 0, 20 > 0 𝑝𝑜𝑠. −𝑑𝑒𝑓
2𝑥2
𝑔1 = 5𝑥12 + 𝑥22 −2≤0 4𝑥1
0 2
4 0
𝛻𝑔2 = → 𝐻 = 4 > 0, −8 < 0 𝑖𝑛𝑑𝑒𝑡𝑒𝑟𝑚𝑖𝑛𝑎𝑡𝑒
−2𝑥2 0 −2
𝑔2 = 2𝑥12 − 𝑥22 −6≤0
 This is not a convex set, since 𝑔2 is not pos.-def.
 Example 48: Convexity
 Is the following function convex everywhere? (why?)
𝑓 𝑥, 𝑦 = 3𝑥 2 + 4𝑦 4

 Solution:

𝜕𝑓
𝜕𝑥 6𝑥
 𝛻𝑓 𝑥, 𝑦 = =
𝜕𝑓 16𝑦 3
𝜕𝑦
6>0
6 0 6 48𝑦 = 288𝑦 2 ≥ 0
2
 𝐻 =
0 48𝑦 2

 𝐻 will be at least semi pos.-def. Thus, the function will be convex

everywhere.
 Example 49: Global Minimum
 Without solving the following problem, determine if it has a global minimum
(explain why).
Minimize 𝑓 𝑥, 𝑦 = 4𝑥 2 + 6𝑦 2 − 2𝑥𝑦

s.t. 𝑔1 = 2𝑥 + 3𝑦 − 5 = 0

𝑔2 = 2𝑥 2 + 𝑦 2 − 5 ≤ 0

 Solution:

8𝑥 − 2𝑦
 𝛻𝑓 𝑥, 𝑦 =
12𝑦 − 2𝑥
8>0
8 −2
8 12 − (−2)(−2) = 92 ≥ 0
 𝐻 = pos.-def
−2 12
• Function is convex.

 𝑔1 is convex because it is a linear function

4>0
4𝑥 4 2 −0=8≥0
4 0
 𝛻𝑔2 = 𝐻 = pos.-def
2𝑦 0 2
• Convex feasible domain

• The function has a global minimum since it is a convex function defined over a convex set.
 Sensitivity Analysis
 Minimize 𝑓(𝑥)
s.t. ℎ𝑖 𝑥 = 0 𝑖 = 1,2, … 𝑛

 Set the right side of the equality (0) equal to 𝑏𝑖 instead

s.t. 𝑔𝑗 𝑥 = 0 𝑗 = 1,2, … 𝑚

 Set the right side of the inequality (0) equal to 𝑒𝑗 instead

 First solve the problem to find 𝑥 ∗ and 𝑓 ∗ ,

 Then:

𝜕𝑓∗
• = −𝑣𝑖
𝜕𝑏𝑖

𝜕𝑓∗
• = −𝑢𝑗
𝜕𝑒𝑗

𝑛 𝑚
• δ𝑓 ∗ = − 𝑖=1 𝑣𝑖 𝑏𝑖 − 𝑗=1 𝑢𝑗 𝑒𝑗
 Examples 50: Sensitivity Analysis
 Minimize 𝑓 𝑥, 𝑦 = 𝑥 − 4 2 + 𝑦−4 2

s.t. ℎ =𝑥+𝑦−2=0

𝑔 =𝑥+𝑦−4≤0

Assume it is found that 𝑣 = 6 and 𝑢 = 0. What is the change in the

optimal value if the RHS of ℎ is increased to 0.3 and the RHS of g is
decreased to −0.2?
 Solution:

 δ𝑓 ∗ = − 0.3 6 − −0.2 0 = −1.8

 Example 51: Sensitivity Analysis
2 2
 Minimize 𝑓 𝑥, 𝑦 = 𝑥 − 1 + 𝑦−1

s.t. 𝑔1 = 4 − 𝑥 − 𝑦 ≤ 0

𝑔2 = 𝑥 + 2𝑦 − 8 ≤ 0

𝑥, 𝑦 ≥ 0

Assume it is found that u1 = 2 and 𝑢2 = 0. What is the change in the

optimal value if the RHS of 𝑔1 is increased to 0.1 and the RHS of 𝑔2 is
decreased to −0.2?
 Solution:
𝑛 𝑚
 δ𝑓 ∗ = − 𝑖=1 𝑣𝑖 𝑏𝑖 − 𝑗=1 𝑢𝑗 𝑒𝑗

 δ𝑓 ∗ = −0 − 2 0.1 − 0 −0.2 = −0.2