LP: Linear Programming

Overview
LP is a tool for solving optimization problems with
linear cost functions and constraints. Motivated by
addressing military problems during World War II, LP
has now found a wide range of industrial applications
in banking, economics/finance, transportation,
petroleum, and IT, to name a few.

@2020 New York University Tandon

2/4/2020 44
School of Engineering
 Key topics to be covered:

• Standard Form and Basic Solutions

• The Simplex Method (G. Dantzig, 1947)

• Duality and Sensitivity,

and the Interior‐Point Method

@2020 New York University Tandon

2/4/2020 45
School of Engineering
 A Geometric Look at LP
Consider a second‐order linear programming
problem:
min P   x1  2 x2
subject to  2 x1  x2  2
 x1  x2  3
x1  3
x1 , x2  0.
@2020 New York University Tandon
2/4/2020 46
School of Engineering
 X2

6
The Feasible Region P = -15

P = -4
P = -2

-3 -1 3 X1

P=0
@2020 New York University Tandon
2/4/2020 47
School of Engineering
 Observations
• The minimizer occurs at the corner or
extreme point (3, 6).
• The feasible region (defined by linear
constraints) is convex.
• The geometric approach is useful for two‐
dimensional LP problems.

There is no uniqueness of LP solutions. Indeed, the optimal

solutions may be the line sengment between two corner points.
@2020 New York University Tandon
2/4/2020 48
School of Engineering
 Exercise for the Geometric Approach

Can you apply the geometric approach to solve

the following LP problem:

max P  6 x1  2 x2 ,
subject to
4 x1  5 x2  20,

3 x1  x2  6,
 x , x  0.
 1 2
@2020 New York University Tandon
2/4/2020 49
School of Engineering
 The Simplex Method
A more systematic tool

• Require a standard form

• Require an initial basic and feasible solution

• Applicable to any-size LP problem

@2020 New York University Tandon

2/4/2020 50
School of Engineering
 Standard Form
Using matrix‐vector notation, an LP problem in
standard form is written as:

min P  c x T

subject to Ax  b,
x  0,

mn
A  , b   , b  0. m

@2020 New York University Tandon

2/4/2020 51
School of Engineering
 Transformation to Standard Form
• A link between max and min:

max{P  cT x}   min{ P  cT x}

xS xS

• From “inequality” to “=“ using slack variable:

ai1 x1    ain xn  bi

ai1 x1    ain xn  si  bi
@2020 New York University Tandon
2/4/2020 52
School of Engineering
 Transformation to SF (cont’d)
• From “inequality” to “=“ using excess variable:

a j1 x1    a jn xn  b j

a j1 x1    a jn xn  e j  b j

• For a “free” or “unrestricted” variable x,

x  x1  x2 , with x1 , x2  0
@2020 New York University Tandon
2/4/2020 53
School of Engineering
 An Example
Convert the following LP problem into standard form:

max P  5 x1  3 x2  7 x3
subject to 2 x1  4 x2  6 x3  7
3 x1  5 x2  3 x3  5
 4 x1  9 x2  4 x3  4
x1  2, 0  x2  4, x3 free

@2020 New York University Tandon

2/4/2020 54
School of Engineering
 Example (cont’d)
New Unknowns

z1  x1  2, z2  x2 , z2  z3  4, z4  z5  x3
3 x1  5 x2  3 x3  z6  5
4 x1  9 x2  4 x3  z7  4

All new unknowns  zi i 1 are  0.

7

@2020 New York University Tandon

2/4/2020 55
School of Engineering
 Example (cont’d)
New Objective Function

P  5 x1  3 x2  7 x3
 10  5 z1  3 z2  7 z4  7 z5
 10  cT z
where
cT   5,  3, 0, 7,  7, 0, 0  ,
z T   z1 , z2 ,  , z7  .

@2020 New York University Tandon

2/4/2020 56
School of Engineering
 2 x1  4 x2  6 x3  7,
New Constraints z2  z3  4,
3 x1  5 x2  3 x3  z6  5
4 x1  9 x2  4 x3  z7  4

2 z1  4 z2  6 z4  6 z5  11,
 z  z  4,
 2 3

3 z1  5 z2  3 z4  3 z5  z6  11,
4 z1  9 z2  4 z4  4 z5  z7  12.
@2020 New York University Tandon
2/4/2020 57
School of Engineering
New Constraints

Az  b

11  2 4 0 6 6 0 0 
4   
0 1 1 0 0 0 0 
b   , A
11   3 5 0 3 3 1 0 
   
12  4 9 0  4 4 0  1
@2020 New York University Tandon
2/4/2020 58
School of Engineering
 Exercise
Convert the following optimization problem
into an LP in the standard form:

min P   x1  2 x2
subject to
x1  x2  4.
@2020 New York University Tandon
2/4/2020 59
School of Engineering
 Standard Form (cont’d)
min P  c x T

subject to Ax  b,
x  0,

mn
A  , b   , b  0. m

Assumptions:
1. m  n : Less constraints than variables.
2. The matrix A has full (row) rank.
@2020 New York University Tandon
2/4/2020 60
School of Engineering
 Extreme Points
A point x  S is an extreme point if x is not
inside a line segment connecting any two points
(y, z) of S:

x  y  (1  ) z , 0    1, y, z  x.

@2020 New York University Tandon

2/4/2020 61
School of Engineering
 A Geometric Result
Assuming a finite minimum of the LP problem exists,
it is attained at (at least) one extreme point of the
constraint set, or called vertices.

Proof. – See the book, where the constraint set,

S, defined by linear constraints, is convex:

y, z  S   y  1    z  S , 0    1.
@2020 New York University Tandon
2/4/2020 62
School of Engineering
 Basic Solutions
A point x is a basic solution if:
i) x satisfies the equality constraints of the LP
problem.
ii) the columns of the constraint matrix A
corresponding to the nonzero components of x are
linearly independent.

A basic feasible solution is a basic solution that

satisfies x > = 0.

@2020 New York University Tandon

2/4/2020 63
School of Engineering
Comment
• A basic feasible solution is an extreme point. Look at
the motivating example:

min P   x1  2 x2
subject to  2 x1  x2  2
 x1  x2  3
x1  3
x1 , x2  0.
@2020 New York University Tandon
2/4/2020 64
School of Engineering
 Standard Form

min P   x1  2 x2
subject to  2 x1  x2  s1  2
 x1  x2  s2  3
x1  s3  3
x1 , x2 , s1 , s2 , s3  0.

@2020 New York University Tandon

2/4/2020 65
School of Engineering
 Basic Solutions
• For the basis  x2 , s1 , s3  , the basic solution follows:

 x1 x2 s1 s2 s3    0 3  1 0 3
• For the basis s 1 , s 2 , s 3  , the basic solution follows:

 x1 x2 s1 s2 s3    0 0 2 2 3 
• For the basis  x 1 , x 2 , s 1  , the basic solution follows:

 x1 x2 s1 s2 s3    3 6 2 0 0 
Optimal !
@2020 New York University Tandon
2/4/2020 66
School of Engineering
 An Equivalence Theorem
A point x is an extreme point of the set
{x: Ax=b, x>=0} if and only if
it is a basic feasible solution.

@2020 New York University Tandon

2/4/2020 67
School of Engineering
 The Sufficiency
If x is a basic feasible solution, then after re‐ordering,

 xB   xB 
x       and A   B N 
 xN   0 
where B is an invertible matrix.
By contradiction, assume x is not an extreme point.
Then,
x  y  (1  ) z , y  0, z  0,   (0,1)
y   yB y N  , z   z B z N  , y  z.
T T

@2020 New York University Tandon

2/4/2020 68
School of Engineering
 The Sufficiency (cont’d)
Therefore, it holds

y N  z N  0, BxB  ByB  Bz B  b

Since B is an invertible matrix, we reach a contradiction.

@2020 New York University Tandon

2/4/2020 69
School of Engineering
 The Necessity
If x is an extreme point, we prove by contradiction that x is
a basic feasible solution:

 xB   xB 
x       , xB >0 and A   B N  , B full column rank
 xN   0 

If B is not full column rank, then there is a nonzero vector

p such that

Bp  B1 p1    Bk pk  0, with Bi the i -th column of B.

@2020 New York University Tandon

2/4/2020 70
School of Engineering
 The Necessity (cont’d)
It is direct to check that the following points are feasible
and distinct from x :

 xB  p   xB  p 
y  and z    ,   0 small
 xN   xN 
1 1
x  y  z , Contradiction !
2 2

@2020 New York University Tandon

2/4/2020 71
School of Engineering
Comments
A basic feasible solution is called degenerate vertex, if
one or more of the basic variables are zero.

The LP Problem, in this case, is called degenerate.

@2020 New York University Tandon

2/4/2020 72
School of Engineering
 The Simplex Method
Lecture objective
Comparing with the geometric approach, the simplex method
is a systematic, powerful tool applicable to linear programming
problems of any size.

Key points:
• Principles of the simplex method.

• Step‐by‐step implementation: simplex tableau

• Initialization via “Artificial Variables”

@2020 New York University Tandon
2/4/2020 73
School of Engineering
 Principles of the Simplex Method
• Begin with an (initial) basic feasible
solution, or an extreme point.

• Move to a (new) basic feasible solution,

to improve the performance function.

@2020 New York University Tandon

2/4/2020 74
School of Engineering
 An Example

min P   x1  2 x2
subject to  2 x1  x2  2
 x1  2 x2  7
x1  3
x1 , x2  0.

@2020 New York University Tandon

2/4/2020 75
School of Engineering
 X2

6
D
The Feasible Region

C
3

-3 -1 A 3 X1

@2020 New York University Tandon

2/4/2020 76
School of Engineering
 Conversion to Standard Form
min P   x1  2 x2
subject to  2 x1  x2  x3 2
 x1  2 x2  x4 7
x1  x5  3
x1 , x2 , x3 , x4 , x5  0.

In compact notation:
min P  cT x
subject to: Ax  b, x  0, b  0.
@2020 New York University Tandon
2/4/2020 77
School of Engineering
 Basic Feasible Solutions
Point A: (0, 0, 2, 7, 3)  xB  ( x3 , x4 , x5 ), xN  ( x1 , x2 )
subject to x3  2  2 x1  x2 , x4  7  x1  2 x2 , x5  3  x1.

next , increase x2 to 2 (while x1 =0!) to give:
Point B: (0, 2, 0, 3, 3)  xB  ( x2 , x4 , x5 ), xN  ( x1 , x3 )
based on the above equality constraints!
This step yields: P  4  5 x1  2 x3
subject to x2  2  2 x1  x3 , x4  3  3 x1  2 x3 , x5  3  x1.
 
@2020 New York University Tandon
2/4/2020 78
School of Engineering
Observations
• The slack variables are good candidates for
(initial choice of) basic variables.

• At each step, P is rewritten as a (linear) function

of non-basic variables only.

@2020 New York University Tandon

2/4/2020 79
School of Engineering
 General Formulas
min P  c xT

subject to Ax  b , A  B  N 
x , b  0.
 xB 
 
x     , P  cB xB  c N xN
T T

x 
 N
Bx B  Nx N  b , B invertible
1 1
x B  B b  B Nx N
@2020 New York University Tandon
2/4/2020 80
School of Engineering
 General Formulas (Cont’d)
P  y b  T

c  y N
T
N

T
xN
cˆN reduced costs
1
with y  c B
T T
B

At each step, setting xN  0 leads to

1 ˆ ˆ T 1
x  B b  b, P  c B b.
B B
@2020 New York University Tandon
2/4/2020 81
School of Engineering
 General Formulas (Cont’d)
P  Pˆ  cˆN xN
Let cˆ j be the entry of cˆN corresponding to x j .
If cˆ j  0 for some j , then P can be improved
(or minimized) by increasing x j from zero.
Let xt be such a variable chosen to enter the basis.
Then,
xB  b  B NxN  bˆ  Aˆt xt
ˆ 1

ˆ 1
with A  B A , A t -th column of A.
t t t

@2020 New York University Tandon

2/4/2020 82
School of Engineering
 General Formulas (Cont’d)

xB  bˆ  B 1 NxN  bˆ  Aˆt xt
can be rewritten as: i,
( x )  bˆ  aˆ x
B i i i ,t t

Only when aˆi,t  0, ( xB )i decreases as xt increases!

Therefore, increase xt using the rule:
 bˆi   bˆi 
xt  min  aˆi ,t  0  , noting ( xB )i    xt  aˆi ,t
aˆi ,t  aˆi ,t 
1i  m    

@2020 New York University Tandon

2/4/2020 83
School of Engineering
 General Formulas (Cont’d)
This yields the new basic variables and an improved
performance function:
x  bˆ  Aˆ x
B t t

Pˆ  Pˆ  cˆt xt , cˆt  reduced cost of xt

Remark:
When aˆi ,t  0 for all i, then, as xt increases
from zero, P   !
@2020 New York University Tandon
2/4/2020 84
School of Engineering
 Summary
 Initialization: Select a basic feasible solution
xB  bˆ  B 1b  0, with B the basis matrix.

 Optimality Test: The current basis is optimal, if

cˆTN  cTN  (cBT B 1 )T N  0.

Reduced costs

Otherwise, select a variable

xt that satisfies cˆt  0 as the entering variable.

@2020 New York University Tandon
2/4/2020 85
School of Engineering
 Recursive Step: Compute Aˆt  B 1 At .
Find an index s (determining the leaving variable,
s‐th. basic variable in B) such that:

bˆs  bˆ 
 m in  i aˆ i , t  0
aˆ s , t 1 i  m  aˆ i , t 

 The Pivot: From the “pivot entry” aˆs ,t , update the

basis matrix B and the vector of basis variables xB .
GO BACK TO STEP 2.

@2020 New York University Tandon

2/4/2020 86
School of Engineering
 An Exercise
min P   x1  2 x2
subject to  2 x1  x2  x3 2
 x1  2 x2  x4 7
x1  x5  3
x1 , x2 , x3 , x4 , x5  0.

In compact notation:
min P  c x, subject to: Ax  b, x  0, b  0.
T

@2020 New York University Tandon

2/4/2020 87
School of Engineering
 Check the sign of reduced cost at each step
Answer ˆcTN  cTN  (cBT B1 )T N . If all  0, then STOP.

Step 1:
xB   x3 , x4 , x5  , xN   x1 , x2 
T T

c B   0, 0, 0  , c N   1,  2 
T T

Step 2:
xB   x2 , x4 , x5  , xN   x1 , x3 
T T

c B   2, 0, 0  , c N   1, 0 
T T

@2020 New York University Tandon

2/4/2020 88
School of Engineering
 Step 3: xB   x2 , x1 , x5  , xN   x3 , x4 
T T

c B   2,  1, 0  , c N   0, 0 
T T

Step 4: xB   x2 , x1 , x3  , xN   x4 , x5 
T T

c B   2,  1, 0  , c N   0, 0 
T T

Optimal, because

 
T
1
cˆ  c N  c B B
T
N
T T
N  (1, 2)  0.
@2020 New York University Tandon
2/4/2020 89
School of Engineering