Numerical Methods for Optimization

7. Introduction to Linear Programming

* Adapted slides from Edoardo Amaldi

Content

¨ Examples
¨ Historical sketch
¨ Linear Programming (LP) definition
¨ Example
¨ Assumption of LP
¨ Geometry of Linear Programming
¨ Vertices of the feasible region
¨ Four types of Linear Programmes

2/34
7.1 Example
Rappresentazione grafica
max z = 3x1 + 5x2 8

s.t. 7
x2

x1 + x2 ≤ 12 6
x1 + 2x2 ≤ 16
5
x1 ≤ 10
x2 ≤ 6 4

3
x1, x2 ≥ 0
2

0
x1
-1
-1 0 1 2 3 4 5 6 7 8 9 10 11

Figura: Regione ammissibile

3/34
Example: graphical solution
Rappresentazione grafica
max z = 3x1 + 5x2 8

s.t. 7
x2

x1 + x2 ≤ 12 6
x1 + 2x2 ≤ 16
5
x1 ≤ 10
x2 ≤ 6 4 (3,5) (8,4)
3 z=44
x1, x2 ≥ 0 3x1 + 5x2 =36
2
3x1 + 5x2 =15
1

0
x1
-1
-1 0 1 2 3 4 5 6 7 8 9 10 11

Figura: Regione ammissibile

4/34
 Example: vertices
Rappresentazione grafica
max z = 3x1 + 5x2 8
s.t. x2
x1 + x2 ≤ 12 7
x1 + 2x2 ≤ 16 (0,6) z=30 (4,6) z=42
x1 ≤ 10 6
x2 ≤ 6
5
x1, x2 ≥ 0
4 (8,4) z=44

2
(10,2) z=40

0 (10,0) z=30
(0,0) z=0 x1
-1
-1 0 1 2 3 4 5 6 7 8 9 10 11
5/34
Figura: Regione ammissibile
 Example: vertices
Rappresentazione grafica
max z = 3x1 + 5x2 8
s.t. x2
x1 + x2 ≤ 12 7
x1 + 2x2 ≤ 16 (0,6) z=30 (4,6) z=42
x1 ≤ 10 6
x2 ≤ 6
5
x1, x2 ≥ 0
4 (8,4) z=44

2
(10,2) z=40

0 (10,0) z=30
(0,0) z=0 x1
-1
-1 0 1 2 3 4 5 6 7 8 9 10 11
6/34
Figura: Regione ammissibile
 Example: vertices grafica
Rappresentazione
max z = 3x1 + 5x2 8
s.t. x2
x1 + x2 ≤ 12 7
x1 + 2x2 ≤ 16 (0,6) z=30 (4,6) z=42
x1 ≤ 10 6
x2 ≤ 6 (6,5) z=43
5
x1, x2 ≥ 0
4 (8,4) z=44
Examples
(6,5)= 0.5(4,6) + 0.5(8,4) 3
(6,0)= 0.6(10,0)+0.4(0,0)
What about the objective 2
(10,2) z=40
function value? v
1
(6,5)= 0.5(4,6) + 0.5(8,4)
(6,0) z=30
z for (6,5)=> 0 (10,0) z=30
=0.5(42)+0.5(44)=43
(0,0) z=0 x1
(6,0)= 0.6(10,0)+0.4(0,0) -1
-1 0 1 2 3 4 5 6 7 8 9 10 11
z for (6,0))=>
s=0.6(30)+0.4(0)=18 7/34
Figura: Regione ammissibile
 Example: vertices grafica
Rappresentazione
max z = 3x1 + 5x2 8
s.t.
x1 + x2 ≤ 12 x2
x1 + 2x2 ≤ 16 7
x1 ≤ 10 (0,6) z=30 (4,6) z=42
x2 ≤ 6
6
x1, x2 ≥ 0
5

4 (8,4) z=44

(2,3)=7/9(0,3)+2/9(9,3) (2,3)
3
(0,3) (9,3)
(0,3)= 1/2(0,0)+ 1/2(0,6) 2 (10,2) z=40
(9,3)= 1/2(10,2)+1/2(8,4)
v
1
=>
(2,3)=7/18(0,0)+7/18(0,6)+ 0 (10,0) z=30
2/18(10,2)+2/18(8,4)
(0,0) z=0 x1
-1
Z for (2,3)=> -1 0 1 2 3 4 5 6 7 8 9 10 11
=7/18(0)+7/18(30)+2/18(40)+2/18(44)=21 8/34
Figura: Regione ammissibile
 Example
Rappresentazione grafica

max z = 3x1 + 3x2 8

x2
s.t. 7

(0,6) z=30 (4,6) z=42

x1 + x2 ≤ 12 6

x1 + 2x2 ≤ 16 5

x1 ≤ 10 4 (8,4) z=44
x2 ≤ 6
3 (2,3)
(0,3) (9,3)
x1, x2 ≥ 0 2 (10,2)
1 vz=40
0
(10,0)
xz=30
-1 (0,0) z=0 1
-1 0 1 2 3 4 5 6 7 8 9 10 11

Figura: Regione ammissibile

What can we say about the objective function values of (2,3)?

Z=7/18(0)+7/18(30)+2/18(40)+2/18(44) ≤ max {0,30,40,44}

9/34
Example

10/34
 7.2 Historical sketch

1825/26: Fourier presents a method to solve systems of linear inequalities and

discusses LPs with 2-3 variables.

1939: Kantorovitch lays the foundations of LP (Nobel prize, 1975)

1947: Dantzig independently proposes LP and invents the Simplex algorithm

11/34
 7.3 Linear programming

Definition
A Linear Programming (LP) problem is an optimization problem:

min f (x)
s.t.
x ∈ X ⊆ n

where
• the objective function f : X ® n is linear
• the feasible region X={ x ∈ n : gi(x) (=, ³ , £ )0, i ∈ {1,…, m} } and
• gi : n ®  linear functions " i = 1,…, m

12/34
 Linear programming

Definition
x* ∈ n is an optimal solution of the LP:

min f (x)
s.t.
x ∈ X ⊆ n
if f (x*) ≤ f (x) "x∈X

• A wide variety of decision-making problems can be formulated as

linear programs (LPs)
• Used in solving more complex problems

1/34
Linear programming

General form: min z = c1 x1 + … + cn xn

a11 x1 + … + a1n xn ≥ b1
(=)
(≤)
⋮ ⋮
am1 x1 + … + amn xn ≥ bm
(=)
(≤)
x1, …, xn ≥ 0c

13/34
Assumptions of LP

¨ Linear objective function and constraints

¨ Proportionality: contribution of each variable =
constant ´ variable
¨ Additivity: contribution of all variables = sum of
single contributions
¨ Divisibility the variables can take fractional (rational)
¨ Parameters are considered as constants which can be
estimated with a sufficient degree of accuracy

14/34
7.3.1 The transportation problem (single product)

Given
m production plants i = 1,…, m
n clients j = 1,…, n
cij unit transportation cost from plant i to client j
pi maximum supply (production capacity) of plant i
dj demand of client j
qij maximum amount transportable from plant i to client

Objective: determine a transportation plan that minimizes the total costs

while respecting plant capacities and client demands

15/34
 The transportation problem (single product)

Decision variables xij = amount of product transported from i to j,

with i = 1,…, m and j = 1,…, n

m n
min åå cij xij m n

i =1 j =1
Assumption on input
parameters å p ³ åd
i =1
i
j =1
j

n
å xij £ pi " i = 1,…, m (plant capacity)
j =1
m
å xij ³ d j " j = 1,…, n (client demand)
i =1

0 ≤ xij ≤ qij " i, j (transportation capacity)

16/34
7.4 Geometry of Linear Programming

Example: min - x1 -3x2

s.t. x1 + x2 ≤ 6 (I) x2
2x1 + x2 ≤ 8 (II) c
x1, x2 ≥ 0

(0,6)
(2,4)
-x1 -3x2 =-9

x1
(4,0)
(I)
(II) 17/34
 7.4.1 Vertices of the feasible region

Consider a LP with inequality constraints (easier to visualize).

Definitions: H = { x ∈ n : aTx = b } is a hyperplane and

H- = { x ∈ n : aTx ≤ b } is an affine half-space (half-plane in 2 ).

Each inequality constraint (aTx ≤ b) defines an affine half-space in the

variable space

19/34
 Polyhedron

Definitions:
Polyhedron P is the intersection of finitely many halfspaces

The feasible region X is a polyhedron P

P can bebounded(polytope)
P can be unbounded

P can be empty
20/34
 Convex set

Definitions: A subset X ⊆ n is convex if for each pair y1, y2 ∈ X, X contains the

whole segment connecting y1 and y2

x
. y2 ≠ convex
convex y1 y1 y2

[y1, y2] = { x ∈Rn : x = a y1 + (1 - a) y2 with a ∈ [0, 1] }

segment º {all the convex combinations of y1 and y2 }

21/34
 Polyhedron

Property: A polyhedron P is a convex set of n

any half-space is convex and the intersection of a finite

number of convex sets is also a convex set

22/34
 Vertices of the feasible region

x3

facet
H1
(0,0,3)
(1,0,3)
(0,1,3)
Algebraically:
not a vertex!
. (2,0,2)
~x is a vertex of P
.Polyedron P vertex x~
. H3 if ∄ y1, y2 ∈ P, y1≠y2
(0,0,0) .
(0,3,1)
(2,0,0) x1
and a ∈ (0, 1) s.t.

(0,3,0)
(2,2,0) ~x = a y1 + (1 - a) y2
(1,3,0)
edge
x2
H2

Definition: A vertex of P is a point of P which cannot be expressed as a convex combination

of two other distinct points of P
23/34
 Polytope

Definition: A polytope is a bounded polyhedron

Consequence: Every point x of a polytope P can be expressed as a convex

combination of its vertices

x4 .
P
. . x = a1 x1 +...+ a4 x4 (d = 0)

x1 . . x3

with ai ³ 0 and a1 +...+ a4 = 1

x2 24/34
 Intuitively, any convex combination of two vectors 𝐱, 𝐲 lies on the line
segment joining 𝐱 and 𝐲
Example
Any point 𝐲 of a non-empty bounded polyhedron can be represented
as a convex combination of its extreme points, that is:

Every point x of a polytope 𝐲

P can be 𝜆 𝐱 𝑠. 𝑡.
expressed 𝜆 of its
𝜆 combination
as a convex 1 0, ∀𝑖
vertices
In other words, a non-empty bounded polyhedron is the convex hull of
its extreme points
𝐱 Example: 𝐲 is a convex(d = 0)
𝐳 combination of 𝐱 , … , 𝐱
1 1
𝐱 𝐳 𝐱 𝐱
P 2 2
𝐲
1 1
𝐳 𝐱 𝐱
2 2
𝐱
𝐳 𝐲 𝐳 𝐳 ,.., 𝐱
𝐱
25/34
 Vertices of the feasible region

Property: A non-empty polyhedron P ={ x Î n : Ax ³ b, x ³ 0 } has a finite

number (³ 1) of vertices

Definition: Given a polyhedron P, a vector d Î n with d ¹ 0 is an unbounded

feasible direction of P if, for every point x0 Î P, the “ray” { x Î n : x = x0 + l d ,
l ³ 0 } is contained in P

P
d
x0 .
26/34
 Vertices of the feasible region

Theorem (representation of polyhedra -Weyl-Minkowski)

Every point x of a polyhedron P can be expressed as a convex combination of its
vertices x1,..., xk plus (if needed) an unbounded feasible direction d of P:
x = a1 x1 +...+ ak xk + d
where the multipliers ai ³ 0 satisfy a1 +...+ ak = 1

" xÎ[x1, x2], d = 0

P

d . x

x1 . .. x2 27/34
 The Fundamental theorem of Linear Programming

Consider a LP min{ cTx : x Î P } where P Í n is a non-empty polyhedron of the

feasible solutions (P ={ x Î n : Ax ³ b, x ³ 0 }). Then either there exists (at least)
one optimal vertex or the value of the objective function is unbounded below on P

Proof

Case 1: P has an unbounded feasible direction d such that cTd < 0

P is unbounded and the values z = cTx ® - ¥

along the direction d

28/34
 The Fundamental theorem of Linear Programming

Case 2: P has no unbounded feasible direction d such that cTd < 0

Any point of P can be expressed as: k

x = åai x + d
i

i =1

where x1,..., xk are the vertices of P, ai ³ 0 with a1+...+ak=1, and

d = 0, or d is a unbounded feasible direction

For any x ∈ P, we have d = 0 or cTd ³ 0 and hence

æ k
T i ö k
c x = c ç å a i x + d ÷ = å a i c x + c d ³ min c x
T T i T
{ }
T i

è i =1 ø i =1 1£i £ k

³
since ai ³ 0 "i and a1+...+ak=1 0
29/34
 Geometry of the fundamental theorem of Linear Programming

An interior point x ∈ P cannot be an optimal solution:

c = direction of fastest increase in z
(constant gradient)

x $ always an improving direction

In an optimal vertex all feasible directions (respecting feasibility for a

sufficiently small step) are “worsening” directions:
-c
improving directions
vertices
feasible directions 30/34
 Vertices of the feasible region

The theorem implies that, although the variables can take fractional
values, Linear Programs can be viewed as combinatorial problems:

we “only” need to examine the vertices of the polyhedron of the

feasible solutions!

Finite but often exponential number

Graphical method only applicabile for n ≤ 3.

31/34
 Examining all vertices

A full enumeration procedure implies checking all vertices and

choosing the best one.

# Variables #Constraints Computation Time

50 10 2.8 hours
50 20 545.5 days
100 20 16995921.2 years

Assuming a vertex can be examined in 1 microsecond

32/34
 7.4.2 Four types of Linear Programmes

Observation: Since min cTx, better solutions found by moving along -c

x2
A unique optimal solution c
.
x1

c
Multiple (infinitely many) optimal solutions

33/34
 Four types of Linear Programmes

c Unbounded polyhedron and

Unbounded LP unlimited objective function
value

Infeasible LP

Empty polyhedron (no

feasible solution)
34/34