"Optimization is essentially the art, science and mathematics of choosing the best among a

given set of finite or infinite alternatives. A new impetus to the development of optimization
began with dramatic development of linear programming in the late 1940's. Convexity also
Brics
brought in a complete new feature to optimization"

Optimization Theory - A Modern face of Applied Mathematics

By Joydeep Dutta

1. Optimization Theory: A Historical Rundown

Optimization is essentially the art, science and
mathematics of choosing the best among a given
set of finite or infinite alternatives. Though
currently optimization is an interdisciplinary
subject cutting through the boundaries of
mathematics, economics, engineering, natural
sciences, and many other fields of human
endeavour it had its root in antiquity. In fact the
oldest story that goes with the history of
optimization concerns an ancient princess
named Dido. She was fleeing from the
prosecution of her brother and a piece of land on
the Mediterranean coast caught her fancy. She
made a deal with the local leader. She requested
him to cut a bull's hide into thin strips and and tie
them up and enclose as much land as one can
with it. In modern day language the problem
mathematically is as follows - Among all closed
curves of a given length find the one that
encloses maximum area. This is called the
Isoperimetric problem. This problem is now
mentioned in a regular fashion in any course in
the Calculus of Variations. However, most
problems of antiquity came from geometry and
since there were no general methods to solve
such problems, each one of them was solved by
very different approaches. An interesting
collection of such problems can be found in the
book titled Stories about Maxima and Minima by
V. M. Tikhomirov.
Beginning of the modern methods of
optimization can be traced back to the growth of
the Calculus of Variations. In 1696 Johann
Bernoulli proposed the famous Brachistochrone
problem (Figure 1) which says that find the curve
along which a particle moving from one point to
another in a vertical plane does so in minimum
time.
This problem precisely gave rise to the Calculus
19
Figure 1: Brachistochrone curve.
of Variations and still continues to test all our
modern methods. Though this problem even
attracted the genius of Isaac Newton it was Euler
who first gave a framework to attack such
problems. A typical calculus of variations
problem consists of minimizing an integral under
some given boundary conditions. In fact it was
Joseph. L. Lagrange who in 1757 first gave a
general framework for tackling problems of the
calculus of variations. His method popularly
known as the Lagrangian multiplier rule still
continues to be used and remains to be one of
the central topics in optimization theory.
However the method for solving simple
unconstrained problems on the real line was
known to Fermat much before derivatives were
introduced in the mathematical literature. In
n
today's language if x Î R be a local minimum of
f : R n ® R then
It was Leibnitz who first gave more detailed
characterization of real-valued functions in terms
of their derivatives. This involved both first order
and second order characterization. However
calculus of variations continued to steal the
limelight and continued to do so till the
nineteenth century. However from the modern
point of view the problems of Calculus of
variations can be thought of as an infinite-
dimensional optimization problem and can be
solved using modern techniques. However its
methods are important since during those days
without any knowledge of infinite dimensional
problems Euler and Lagrange could develop a
general framework for solving such problems.
Even David Hilbert mentioned the importance of
the subject in his famous 1900 lecture in the
International Congress of Mathematicians. In
fact the calculus of variations later gave rise to
the modern subject of optimal control which has
large number of applications in engineering and
space science.
However during the early years of 20th century
there was hardly any development of
optimization theory. A new impetus to the
development of optimization began with
dramatic development of linear programming in
the late 1940's. A linear programming problem
consists of linear functions as objective and
constraints. It became apparent that linear
programming has tremendous applications in
economics, military operations, business,
engineering, etc. It was the development of the
simplex method for solving linear programming
problems that became the turning point of the
subject and George. B. Dantzig who formulated
this method became an icon and linear
programming became an important part of
applied mathematics. However in the early
1950's it was observed that there are important
application problems which involved nonlinear
functions as well as constraints represented by
inequalities. Thus in order to tackle the inequality
constraints some new mathematics was
developed and the first optimality condition in
this direction was given by Fritz John(1948) and
later refined by Kuhn and Tucker (1951). The
Kuhn-Tucker conditions turned out to be very
practical and were thus used to develop
algorithms for inequality constrained
minimization problem. However it was later
found that the use of inequality constraints was
also studied by W. Karush way back in 1939 and
thus the conditions are called the Karush-Kuhn-
Tucker conditions or simply KKT conditions. It is
20
important to note that Lagrange had only studied
equality constraints. Thus the KKT conditions
can be thought of as the Lagrangian multiplier
rule for inequality constraints.
2. Convexity: A Crucial Matter in Optimization
Theory
A KKT point need not be a minimum of a given
program. However the sufficiency of the KKT
conditions is guaranteed when the objective and
constraint functions are assumed to be convex.
This is precisely where convexity enters the
larger picture of optimization and goes on to play
a central role. Modern optimization by large is
based on convexity and notions relying on it. A
function f : C Í R n ® R is said to be convex over
the convex set if for any
we have
f ( x + l ( y - x)) £ l f ( y ) + (1 - l ) f ( x).
When f differentiable can have an equivalent
representation as follows
f ( y ) - f ( x) ³ áÑf ( x), y - xñ, "x, y Î C
It is clear from the above expression that every
critical point (point with of a
convex function is a global minima. In fact even
without differentiability one can easily show that
every local minima of a convex function is global.
This simple feature makes convexity such an
attractive thing in optimization theory. Convexity
also brought in a complete new feature to
optimization. It is that of non smoothness or non-
differentiability. In fact there are very simple
convex functions which are not differentiable at
the point of minimum. The function
is the prototype example, shown in
Figure 2.
Figure 2: A plot of .
Observe that this function is not differentiable at
x = 0 yet it does not have a derivative there.
Thus the notion of sub gradient was developed in
order to imitate the notion of the derivative at the
points of non-differentiability. For a real-valued
convex function defined over an open convex set
at any point there exists called the
subgradient of f at and satisfies the
following inequality
Let us note the subgradient need not be unique (
loss of differentiability at a point
corresponds to the fact that more that one
tangent plane can pass through the same point).
Thus the collection of all subgradients at a given
point is called the sub differential of f at
and is denoted as Further a point x is a
global minimum if and only if . This can
be indeed thought of as the generalization of
Fermat's rule in the differentiable case. In fact if
we consider the following simple convex
problem
Where f and are real-valued convex function
on R n . Let x0 be a minimum of the above
problem and also assume that there exists x
such that Then one can show that
there exists l ³ 0 such that
0 Î ¶f ( x0 ) + l¶g ( x0 )
and
The above conditions can be thought of as the
KKT type conditions for a convex problem in a
nonsmooth setting or a general setting. Thus
with the powerful tool like the subdifferential
convexity became a very amenable object even
in the nonsmooth setting. Infact Tucker coined
the term Convex Analysis in order to bring out
the importance of convexity and with the
publication of the classic monograph titled
Convex Analysis (Princeton, 1970) by R. T.
Rockafeller, which established convex analysis
as an important branch of modern mathematics.
It is important to note that linear programming
problems are a subclass of the convex
programming problems. As Rockafellar has
recently pointed out that the great watershed in
optimization is not really between linearity and
nonlinearity but between convexity and
nonconvexity. Since then convexity continues to
21
play a central role in optimization theory. In fact
so deep rooted is the notion of convexity that
usually nonconvex problems are tackled by
devising suitable convex approximations.
3. Smoothness of Functions
Though convexity has such power it was
observed that many application problems are
devoid of convexity yet are non-differentiable.
This lead Francis Clarke to study locally Lipschitz
nonsmooth function in details in the early
1970's. He developed the notion of generalized
gradient or the Clarke subdifferential. Clarke
used his subdifferential to develop KKT type
conditions for mathematical programming
problem as well as in developing necessary
conditions for optimal control problems. This
new approach in tackling nonsmoothness in
optimization gave rise to the subjects of
Nonsmooth Analysis and Nonsmooth
Optimization. However Clarke's approach
depended heavily on the tools of convex
analysis. A complete move away from convexity
was championed by Mordukhovich by
developing the now famous sequential
nonsmooth analysis. Mordukhovich succeeded
in developing a robust subdifferential for lower-
semicontinuous functions which is nonconvex in
general and is contained in the Clarke
subdifferential. Motivated by optimal control
Mordukhovich introduced the notion of Extremal
Principle which turned out have far reaching
consequences and which has application to
areas that go beyond traditional optimization.
4. Numerical Methods for Optimization
Meanwhile on the other side during the mid
1960's and 1970's there has been a tremendous
growth in the development of algorithms for
solving nonlinear programming problems. It first
began by trying to develop efficient and
implementable algorithms to solve the
unconstrained problem
The most efficient methods developed for
solving this problem used the principle of line
search, i.e. one moves from the current iterate
to the next one denoted as x+ in the following
manner
x+ = x + td ,
where d is called the descent direction satisfying
and t > 0 is called the steplength.
This leads to the fact that and thus
by moving along d one is able to improve the
objective function. Various methods like this
have been developed using the line search
philosophy. Further finding a minimum amounts
to finding the root of the equation Thus
various approaches like the Newton method or
Quasi-Newton method used in equation solving
can be applied to optimization. The Newton
method has the following scheme of iteration
Major contributions in this area have been made
by Davidon, Powell, Fletcher, Goldfard, Broyden,
Goldfarb, and Shanno. In solving constrained
problems the penalty method, the barrier method
and the sequential quadratic programming
scheme are important classical approaches.
It is important to note that the simplex method
for linear programming which has been effective
in solving many practical problems is not a
polynomial time algorithms. Thus there have
been efforts to develop a polynomial time
algorithm for linear programming which will be if
not better but at least as efficient as the simplex
method in practice. The major breakthrough
came in 1984 when Karmarkar announced his
now famous algorithm. In Karmarkar's method in
contrast to the simplex method one moves
through the interior of the feasible region and
move towards a vertex. Karmarkar's work gave
rise to the now important interior point methods
of convex optimization. The interior point
methods remain to be one of the most fertile
fields of modern optimization research. The
growth of interior point methods has also lead to
the growth of the modern areas of semi definite
programming and conic programming. Apart
from these the important areas associated with
modern optimization are semi finite
programming, vector optimization, global
optimization, bi-level programming,
mathematical programming under equilibrium
constraints, Abstract Convexity, just to name a
few.
5. My Interests in IITK
My research initially centered around nonsmooth
optimization and trying to understand and design
sub differentials for nonconvex functions and
22
applying them to develop necessary optimality
conditions. Relations of subdifferentials to
generalized convex functions were also one of
the areas that I have been interested in. Later on
during my post-doctoral days I worked on the
notion of a convexity factor which is a very
general object and posses many generic
properties of most of the well known sub
differentials. I also got interested in an emerging
area of optimization called Abstract Convexity
during my postdoctoral days and it remains to by
major interest. Abstract Convexity has grown
out of generalizing the well known global
property of a convex function namely, every
convex function is an upper envelope of the
family of affine functions majorized by it (or point
wise smaller that itself). The sub-area of
Abstract Convexity where I have worked most is
an area called Monotonic Analysis where in one
studies functions which are increasing and has
some additional properties like positive
homogeneity or convex along rays. This area has
important applications to Global optimization. In
this area I collaborate with Professor A. Rubinov
(Ballarat, Australia) and Professor J. E. Martinez-
Legaz (Barcelona, Spain). Apart from this I am
interested in studying various theoretical aspects
of vector optimization. Recently my interest also
shifted to bi-level programming and
mathematical programming with equilibrium
constraints. I plan to study optimality conditions
in this particular area since the most well known
constraint qualification fails. Further I plan to
have a major project in the area of Interior Point
Method theory and I intend to study how to
increase the speed of interior point algorithms for
conic programs by using suitable self-concordant
barriers. In fact there have been few encouraging
recent results in this direction from some leading
researchers. Also I plan to develop software with
a user friendly front end to use the interior point
methods in convex programming and have a
library of such programs. Apart from the
computational aspects there are also interesting
theoretical aspects that can be studied in interior
point theory, for example the detailed study of
self scaled cones and developing faster path-
following algorithms.
References
1. C. R. Bector, S. Chandra and J. Dutta,Principles of Optimization
Theory, Narosa Publishers, New Delhi, 2004.
2. D. P. Bertsekas, Convex Analysis and Optimization, Athena Scientific
Publication, 2003.
3. D. P. Bertsekas, Nonlinear Programming, Athena Scientific
Publication, 1999.
4. J. Borwein and A. S. Lewis, Convex Analysis and Nonlinear
Optimization, Springer, 2000.
5. F. H. Clarke, Optimization and Nonsmooth Analysis, Wiley
Interscience, 1983.
6. B. D. Craven, Control and Optimization, Chapman and Hall, 1995.
7. S. Dempe, Foundations of Bi-level Programming, Kluwer Academic
Publishers, 2002.
8. J. Jahn, Vector Optimization, Springer, 2004.
9. J. Renegar, A Mathematical View of Interior-Point Methods in Convex
Optimization, SIAM, 2001.
10. R. T. Rockafellar, Convex Analysis, Princeton University Press, 1970.
11. R. T. Rockafellar and R. J. B. Wets, Variational Analysis, Springer,
1998.
12. A. Rubinov, Abstract Convexity and Global Optimization, Kluwer
Academic Publishers, 2000.

About the author: Joydeep Dutta recieved his Ph.d in mathematics in 1998
from I.I.T. Kharagpur with specialization in Nonsmooth Optimization.
Consequently he did his post-doctoral studies in the Indian Statistical Institute,
New Delhi and the Autonomous University of Barcelona, Spain. His main
research interests are in Abstract Convexity, Nonsmooth Optimization and
Vector Optimization. He is currently an Assistant Professor in the Department
of Mathematics at the I.I.T. Kanpur

23
 Books Written by IITK Faculty

Engineering Optimization: Theory and Practice by Singiresu S. Rao

Publisher: New Age Intl. Pvt. Ltd., Delhi (Reprint, 2003)
This book provides the most practical, up-to-date and comprehensive
coverage of new and classical optimization techniques currently in
use throughout a wide range of industries. This book is designed
to serve as both daily working resource and an excellent graduate
level text, this book gives an in-depth coverage of linear
and non-linear programming, dynamic programming, integer
programming and stochastic programming techniques.

Network Flows by Ravindra K. Ahuja, Thomas L. Magnanti, James B. Orlin

Publisher: Prentice-Hall, 1993
The book presents a comprehensive introduction to network flows that brings
together the classic and the contemporary aspects of the field, and provides an
integrative view of theory, algorithms, and applications. "This book contains a lot of
great algorithms for network flow theory and it also contains many of the great
applications, which are very useful in practice." Tiravat Assavapokee, Georgia

Linear Programming and Network Models by S. K. Gupta

Publisher: Affiliated East-West Publishers. Delhi, 1984
The book presents in an elementary but rigorous manner the most important
aspects of linear programming essential to a beginner in operations research. This
book is currently out of print.

Linear Programming for Schools by S. K. Gupta

Publisher: Arya Book Depot, New Delhi, 1968
The book presents linear programming methods for school students. This book is currently out of
print..

more books on Page 6

24