Reference books

(1) Introduction to Operations Research by Hamdy A. Taha

Latest 10th edition Pearsons publication
(2) Introduction to Operations Research by Hillier and
Liberman Tata Mcgraw hill Latest 9th edition
(3) Operations Research, Algorithms and Applications by
Wyane L. Winston 4th edition Duxbury publication
(4) Introduction to mathematical programing 4th edition
Wyane L. Winston
(5) Introduction to Optimal Design by Jasbir S. Arora Latest 4th
edition Elsevier
 Optimization - Introduction
 Optimization is the act of obtaining the best result under given circumstances.
 Optimization can be defined as the process of finding the conditions that give the
maximum or minimum of a function.
 The optimum seeking methods are also known as mathematical programming techniques
and are generally studied as a part of operations research.
 Operations research is a branch of mathematics concerned with the application of scientific
methods and techniques to decision making problems and with establishing the best or
optimal solutions.
An optimization problem or a mathematical program or a mathematical
programming problem.
Minimize f(x) f(x): Objective function

subject to h(x)=c x: Decision/Design variables

g(x)> b h(x)=c: Equality constraint
g(x)> b: Inequality constraint
x*: solution
 Optimization - Introduction
Classification of Optimization Problems
Continuous Optimization Optimization Under Uncertainty Robust
Unconstrained Optimization Optimization
Bound Constrained Optimization Stochastic Programming
Global Optimization Stochastic Algorithms
Linear Programming
Network Flow Problems Systems of Equations
Nondifferentiable Optimization Data Fitting/Robust Estimation
Nonlinear Programming Nonlinear Equations
Optimization of Dynamic Systems Nonlinear Least Squares
Quadratic Constrained Quadratic Programming
Quadratic Programming Complementarity Constraints
Discrete and Integer Optimization Game Theory
Combinatorial Optimization
Traveling Salesman Problem Multiobjective Optimization
Integer Programming
Mixed Integer Linear Programming
Mixed Integer Nonlinear Programming
 Optimization - Introduction
Types of optimization problems - Classification can be based on different criteria
• (5) Permissable values of the variables
(1) Based on presence of constraints
 Unconstrained optimization problems – Integer programming problems
 Constrained optimization problems – Continuous variable, Real valued
(2) Based on no. of variables programming problems
 Single variable optimization problems • (6) Based on Nature of Objective
(one dimensional) function and constraints (Nature of the
 Multi variable optimization problems equations involved)
(multi dimensional)  Linear programming problems
(3) Based on no. of Objectives  Non linear programming problems
 Single objective optimization problems
Non linear programming problems
 Multi objective optimization problems
 Linearly constrained N-L programming
(4) Based on Nature of variables (Certainty) problems
 Deterministic optimization problems  Quadratic programming problems
 Stochastic optimization problems  Geometric programming problems
 Optimization - Introduction
Integer Programming Problem
• If some or all of the design variables x1,x2,..,xn of an optimization problem are
restricted to take on only integer (or discrete) values, the problem is called an
integer programming problem.
• If all the design variables are permitted to take any real value, the optimization
problem is called a real-valued programming problem.
Stochastic Programming Problem
• A stochastic programming problem is an optimization problem in which some or
all of the parameters (design variables and/or preassigned parameters) are
probabilistic (nondeterministic or stochastic).
• In other words, stochastic programming deals with the solution of the optimization
problems in which some of the variables are described by probability distributions.
 Optimization - Introduction
Optimal Control Problem
• An optimal control (OC) problem is a mathematical programming problem
involving a number of stages, where each stage evolves from the preceding
stage in a prescribed manner.

• It is usually described by two types of variables: the control (design) and the
state variables. The control variables define the system and govern the evolution
of the system from one stage to the next, and the state variables describe the
behaviour or status of the system in any stage.
• The problem is to find a set of control or design variables such that the total
objective function (also known as the performance index) over all stages is
minimized subject to a set of constraints on the control and state variables.
 Optimization - Introduction
Single variable optimization
• Useful in finding the optimum solutions of continuous and differentiable functions
• These methods are analytical and make use of the techniques of differential
calculus in locating the optimum points.
• Since some of the practical problems involve objective functions that are not
continuous and/or differentiable, the classical optimization techniques have limited
scope in practical applications.
For a continuous and differentiable function f(x)
A stationary point x* is a point at which the slope of the
function is zero, i.e. f ʹ(x) = 0 at x = x*,
 Optimization - Introduction
Optimization terminology

 Representation of optimum points.

 (a) The unbounded domain and
function (no global optimum).

 (b) The bounded domain and function

(global minimum and maximum
exist).
 Optimization - Introduction
Types of minima
weak strong
strong
local local
local
f(x) minimum
minimum strong minimum
global
minimum

feasible region x

• A function f (x) is said to have a global or absolute minimum at x* if f (x*) ≤ f (x) for all
x, and not just for all x close to x*, in the domain over which f (x) is defined.
• Similarly, a point x* will be a global maximum of f (x) if f (x*) ≥ f (x) for all x in the
domain.
 Optimization - Introduction
Global vs. local optima
Example: Solve the following:
Minimize f(x)=x2
Solution: f’(x)=2x=0x*=0.
This solution is a local optimum.
It is also the global optimum.
Example: Solve the following:
Minimize f(x)=x3-17x2+80x-100
Solution: f’(x)=3x2-34x+80=0
Solving the above results in x=3.33 and x=8.
Issue#1: Which is the best solution?
Issue#2: Is the best solution the global solution?
Issue#1: Which is the best
solution?
x=8
Issue#2: Is the best
solution the global
solution?
No! It is unbounded.
 Optimization - Introduction
Convex functions
Definition #2: A function f(x) is convex if a
Definition #1: A function f(x) is line drawn between any two points on the
convex in an interval if its second function remains on or above the function in
derivative is positive on that the interval between the two points.
interval.
Is a linear function convex?
Example: f(x)=x2 is convex since
f’(x)=2x, f’’(x)=2>0

Answer is “yes” since a line

drawn between any two
The second derivative test is points on the function
sufficient but not necessary. remains on the function.
 Optimization - Introduction
Convex Sets
A set C is convex if a line segment
between any two points in C lies in C.

Ex: Which of the below are convex sets?

The set on the left is convex. The

set on the right is not.
 Optimization - Introduction
Convex Polyhedron and Convex Polytope:
A convex polyhedron is a set of points common to one or more half spaces
A convex polyhedron that is bounded is called a convex polytope
 Optimization - Introduction
Convexity & global vs. local optima
When minimizing a function, if we want to be sure that we can get a global solution
via differentiation, we need to impose some requirements on our objective function.
We will also need to impose some requirements on the feasible set S (set of possible
values the solution x* may take).
Min f(x) min f ( x)
subject to subject to
h(x)=c
g(x)> b
Feasible set
xS
Definition: If f(x) is a convex function, and if S is a convex set, then
the above problem is a convex programming problem.
Definition: If f(x) is not a convex function, or if S is not a convex set,
then the above problem is a non-convex programming problem.
 Optimization - Introduction MATHEMATICAL
PROGRAMMING
Convex vs. nonconvex programming problems
The desirable quality of a convex Convex
programming problem is that any locally
optimal solution is also a globally optimal We address convex
programming
solution. If we have a method of problems in
finding a locally optimal solution, that addressing linear
programming.
method also finds for us the globally
optimum solution.
The undesirable quality of a non-convex Non-convex
programming problem is that any We will also, later,
address a special form of
method which finds a locally optimal non-convex programming
solution does not necessarily find the problems called integer
programs.
globally optimum solution.
 Optimization - Introduction
(multivariate) Multivariable optimization problems (Multi
dimensions)
• How big N (Dimensions)can be?
– problem sizes can vary from a handful of parameters to many
– consider examples for N=2, so that cost function surfaces can be
visualized.
 Optimization - Introduction
Contour maps and 3-D illustrations Definition: A contour map is a 2-dimensional
plane, i.e., a coordinate system in 2
Example: Draw the 3-D surface for variables, say, x1, x2, that illustrates curves
f ( x1, x2 )  2
x1  2
x2 (contours) of constant functional value f(x1,
x2).
Example: the contour map for

f ( x1, x2 )  2
x1  2
x2

Height is f(x)

Each contour of fixed value f Contours

is the projection onto the x1-
x2 plane of a horizontal slice
made of the 3-D figure at a
value f above the x1-x2 plane.
 Contour plot
Optimization - Introduction
3

2.5

1.5 0.1
0.1
0.2 0.3 0.2
1 0.4
0.5
0.5 0.
6

0.1

0.4

0.7
0.3

0.4
0

0.1
7
0.8 0. 5

0. 3
0.2
-0.5 0.

0.
5 0.

2
0.6
-1 0.4
0.3
0.1 0.2
-1.5 0.1

-2
-2 -1.5 -1 -0.5 0 0.5 1 1.5 2
 Optimization - Introduction
 Global optimum vs Local optimum
3

2.5 2
2 4
2

6
1.5

2
1 4
local 0 2 2
minimum 0.5
-2

-0.5 0
2
-1 2 -2 0
0
-4
-1.5

-6
-2
-2 -1.5 -1 -0.5 0 0.5 1 1.5 2

local
minimum
Optimization - Introduction
Optimization - Introduction