Issues

EE 477: Optimization Techniques

• What is optimization?

• What real life situations give rise to optimization problems?

Handout #1 • When is it easy to optimize?

• What are we trying to optimize?

Introduction • What can cause problems when we try to optimize?

• What methods can we use to optimize?

Prof. Mohamed Zribi

1 EE477 Dr. Mohamed Zribi 2

What is Optimization?

• Optimization is derived from the Latin word “optimus”,

the best.

What is Optimization? • Optimization characterizes the activities involved to

find “the best”.

• People have been “optimizing” forever, but the roots

for modern day optimization can be traced to the
Second World War.

EE477 Dr. Mohamed Zribi 3 EE477 Dr. Mohamed Zribi 4

 What is Optimization? What is Optimization?

• Optimization: The act of obtaining the best result Optimization is defined as the process of finding
under the given circumstances.
the conditions that give the minimum or
• Design, construction and maintenance of
maximum value of a function, where the function
engineering systems involve decision making both
at the managerial and the technological level represents the effort required or the desired

• Goals of such decisions : benefit.

– to minimize the effort required or
– to maximize the desired benefit

EE477 Dr. Mohamed Zribi 5 EE477 Dr. Mohamed Zribi 6

What is Optimization? What is Optimization?

• Optimization is the act of obtaining the best result under given • operational research (OR is an interdisciplinary branch of
circumstances. mathematics which uses methods like:

– mathematical modeling
• Optimization can be defined as the process of finding the conditions
– statistics
that give the maximum or minimum of a function.
– algorithms to arrive at optimal or good decisions in complex problems
which are concerned with optimizing the maxima (profit, faster assembly
• The optimum seeking methods are also known as mathematical line, greater crop yield, higher bandwidth, etc) or minima (cost loss,
lowering of risk, etc) of some objective function.
programming techniques and are generally studied as a part of
operations research.
• The eventual intention behind using operations research is to elicit a
best possible solution to a problem mathematically, which improves or
• Operations research is a branch of mathematics concerned with the optimizes the performance of the system.
application of scientific methods and techniques to decision making
problems and with establishing the best or optimal solutions.

EE477 Dr. Mohamed Zribi 7 EE477 Dr. Mohamed Zribi 8

 Historical Development

• Existence of optimization methods can be traced to the days

of Newton, Lagrange, and Cauchy.

• Development of differential calculus methods of

Historical Development optimization was possible because of the contributions of
Newton and Leibnitz to calculus.

• Foundations of calculus of variations, dealing with the

minimizations of functions, were laid by Bernoulli Euler,
Lagrange, and Weistrass

EE477 Dr. Mohamed Zribi 9 EE477 Dr. Mohamed Zribi 10

Historical Development Historical Development

• The method of optimization for constrained problems, • Isaac Newton (1642-1727)

(The development of differential calculus
which involve the inclusion of unknown multipliers, methods of optimization)
became known by the name of its inventor, Lagrange.
• Joseph-Louis Lagrange (1736-1813)
(Calculus of variations, minimization of functionals,
method of optimization for constrained problems)
• Cauchy made the first application of the steepest descent
• Augustin-Louis Cauchy (1789-1857)
method to solve unconstrained optimization problems. (Solution by direct substitution, steepest
descent method for unconstrained optimization)

EE477 Dr. Mohamed Zribi 11 EE477 Dr. Mohamed Zribi 12

 Historical Development Historical Development

• Leonhard Euler (1707-1783) • George Bernard Dantzig (1914-2005)

(Calculus of variations, minimization of
functionals)
• Gottfried Leibnitz (1646-1716)
(Differential calculus methods
of optimization)
(Linear programming and Simplex method (1947))
• Richard Bellman (1920-1984)
(Principle of optimality in dynamic
programming problems)
• Harold William Kuhn (1925-)
(Necessary and sufficient conditions for the optimal solution of
programming problems, game theory)

EE477 Dr. Mohamed Zribi 13 EE477 Dr. Mohamed Zribi 14

Historical Development

• Albert William Tucker (1905-1995)

(Necessary and sufficient conditions
for the optimal solution of programming Engineering Applications
problems, nonlinear programming, game
theory: his PhD student
was John Nash) of Optimization
• Von Neumann (1903-1957)
(game theory)

EE477 Dr. Mohamed Zribi 15 EE477 Dr. Mohamed Zribi 16

 Engineering Applications of
Engineering Applications of Optimization
Optimization
• Design of structural units in construction, machinery, and in
• Architecture
space vehicles. • Nutrition
• Maximizing benefit/minimizing product costs in various • Electrical circuits
• Networks
manufacturing and construction processes.
• Optimal Control
• Optimal path finding in road networks/freight handling • Economics
processes. • Transportation
• Scheduling
• Optimal production planning, controlling and scheduling.
• etc.
• Optimal Allocation of resources or services among several
activities to maximize the benefit.
EE477 Dr. Mohamed Zribi 17 EE477 Dr. Mohamed Zribi 18

What do we optimize?

• A real function of n variables

f ( x1 , x2 , , xn )
What do we optimize? with or without constrains

EE477 Dr. Mohamed Zribi 19 EE477 Dr. Mohamed Zribi 20

 Unconstrained Optimization Optimization with Constraints
min f ( x, y ) x2 2 y2
x 0

or
min f ( x, y ) x2 2 y2 min f ( x, y ) x2 2 y2
2 x 5, y 1

or
min f ( x, y ) x2 2 y2
x y 2

EE477 Dr. Mohamed Zribi 21 EE477 Dr. Mohamed Zribi 22

What do we optimize? What do we optimize?

Optimization is a process by which a desired solution
(max/min) of the problem can be found while satisfying all its
constraint or bounded conditions.

Optimum solution is found while

satisfying its constraint (derivative must
be zero at optimum).

Optimization problem could be linear or non-linear.

Nonlinear optimization is accomplished by numerical ‘Search
Methods’.
Search methods are used iteratively before a solution is achieved. Global versus local optimization.
The search procedure is termed as algorithm.
EE477 Dr. Mohamed Zribi 23 EE477 Dr. Mohamed Zribi 24
 What do we optimize? What do we optimize?
Linear problem – solved by Simplex or Graphical methods.
The solution of the linear problem lies on boundaries of the feasible
region.

Solution of linear problem Three dimensional solution of non-

linear problem
Local point is equal to global point if the function is convex.
Non-linear problem solution lies within and on the boundaries of the
EE477 Dr. Mohamed Zribi 25
feasible region. EE477 Dr. Mohamed Zribi 26

What do we optimize? max-min for R

Single Objective function f(x)
• Maximization • What is special about a local max or a local min of a
• Minimization
Design Variables, xi , i=0,1,2,3….. function f (x)?
Constraints
• Inequality at local max or local min f’(x)=0
• Equality
Optimal points f”(x) > 0 if local min
• Local minima/maxima points: A point or Solution x* is at local point
if there is no other x in its Neighborhood less than x* f”(x) < 0 if local max
• Global minima/maxima points: A point or Solution x** is at global
point if there is no other x in entire search space less than x**

EE477 Dr. Mohamed Zribi 27 EE477 Dr. Mohamed Zribi 28

 Max-min for R3 Max-min for R3
• Second Derivative Test

• Local min, local max, saddle point

• Gradient of f – vector (df/dx, df/dy, df/dz)

direction of fastest increase of f

• Global min/max vs. local min/max

EE477 Dr. Mohamed Zribi 29 EE477 Dr. Mohamed Zribi 30

Search Methods

• Minimize function
f ( x, y ) 0.5( x y ) 2 2

11 x, y 11 Classification of
Minimize function
f ( x, y ) cos( x) cos( y )
Optimization Methods
4 x, y 4

EE477 Dr. Mohamed Zribi 31 EE477 Dr. Mohamed Zribi 32

Classification of Optimization Methods Classifications of Optimization Methods
• Single variable optimization
• Multi-variable optimization
• Linear optimization
• Nonlinear optimization
• Constrained optimization
• Non-constrained optimization
• Single objective optimization
• Multi-objective optimization

EE477 Dr. Mohamed Zribi 33 EE477 Dr. Mohamed Zribi 34

Classification of Optimization Methods

Optimization and
Computers

EE477 Dr. Mohamed Zribi 35 EE477 Dr. Mohamed Zribi 36

 Optimization and Computers The Computer and Linear Programming
It is generally accepted that without computers, optimization would not be what Consider the case of Linear Programming:
• The first large scale computer became a practical reality in 1946 at the
it is today. University of Pennsylvania.

• Earlier mathematical models (such as calculus, Lagrange multipliers) relied

on sophistication of technique to solve the problem classes for which they
were suited.
• Methods of mathematical optimization (e.g., Linear Programming) rely far
less on mathematical sophistication than they do on an unusual “adaptibility
to the mode of solution inherent in the modern digital computer”.
• Particularly striking is the simplicity of these methods of mathematics
coupled with their iterative processes (i.e., the repeated performance of a
relatively simple set of operations)
• This was just one year before the development of simplex.
• The simplex method for linear programming consists only of a few steps and
these steps require only the most basic mathematical operations which a
computer is well suited to handle.
• However, these steps must be repeated over and over before one finally obtains
an answer.
• The first successful computer solution of a LP problem was in January 1952 on
the National Bureau of Standards SEAC computer.

EE477 Dr. Mohamed Zribi 37 EE477 Dr. Mohamed Zribi 38

Remark:
This has (in principle) nothing to do with modern day
You will often hear the phrase “programming” as in: computer programming.

Mathematical programming,
Linear programming, In the early days, a set of values which represented a
solution to a problem was referred to as a “program”.
Nonlinear programming,
Integer programming, However, nowadays you program (software) to find
a program!
mixed integer programming, etc.

EE477 Dr. Mohamed Zribi 39 EE477 Dr. Mohamed Zribi 40