Introduction

Geometry

Simplex

Duality

VU Algorithmics
Part V: Linear Programming
u Markus Leitner, Gnther Raidl,Mario Ruthmair
Algorithms and Data Structures Group Institute of Computer Graphics and Algorithms Vienna University of Technology

WS 2011/12

VU Algorithmics

M. Leitner

Introduction

Geometry

Simplex

Duality

Topics of this part

Introduction to Linear Programming Geometry of Linear Programming The Simplex Method Duality Theory

VU Algorithmics

M. Leitner

Introduction

Geometry

Simplex

Duality

Introduction

VU Algorithmics

M. Leitner

Introduction

Geometry

Simplex

Duality

Literature for this part

D. Bertsimas and J. N. Tsitsiklis, Introduction to Linear Optimization, Athena Scientic, 1997. 6.251J/15.081J Introduction to Mathematical Programming - Fall 2009, MIT OpenCourseWare (http://ocw.mit.edu). G. Nemhauser and L. A. Wolsey, Integer and Combinatorial Optimization, Wiley, 1999. A. Schrijver, Theory of Linear and Integer Programming, Wiley, 1986.

VU Algorithmics

M. Leitner

Introduction

Geometry

Simplex

Duality

What is Linear Optimization / Linear Programming?

Denition 1 (Linear Programming)
Linear programming is the problem of minimizing a linear cost function subject to linear equality and inequality constraints.

Example 2 (Exemplary Linear Program)

minimize 2x1 x2 + 4x3 subject to x1 + x2 + x4 3x2 x3 x3 + x4 x1 x3
VU Algorithmics M. Leitner

2 =5 3 0 0
5

Introduction

Geometry

Simplex

Duality

General Linear Programming Problem

In the general case we are given a cost vector c = (c1 , . . . , cn ) and seek to minimize a linear cost function c x =
n i=1 ci xi

over all n-dimensional vectors x = (x1 , . . . , xn ), subject to a set of linear equality and inequality constraints. Let M1 , M2 , and M3 be nite index sets for each of which we are given an n-dimensional vector ai and a scalar bi , used to form the i-th constraint. N1 and N2 be subsets of {1, . . . , n} indicating which variables xj are constrained to be nonnegative or nonpositive, respectively.
VU Algorithmics M. Leitner 6

Introduction

Geometry

Simplex

Duality

General Linear Programming Problem

min c x s.t. ai x bi ai x bi ai x = bi xj 0 xj 0 decision variables x1 , . . . , xn objective (cost) function c x free (unrestricted) variables xj , j N1 N2 /
VU Algorithmics M. Leitner 7

i M1 i M2 i M3 j N1 j N2

Introduction

Geometry

Simplex

Duality

Further Notations and Denitions

vector x satisfying all constraints is called a feasible solution or feasible vector set of feasible solutions is called feasible set or feasible region a feasible solution x that minimizes the objective function (x x for all feasible solutions x) is called an optimal feasible solution or optimal solution; value c x is then called optimal cost

VU Algorithmics

M. Leitner

Introduction

Geometry

Simplex

Duality

Transformation Rules
1 2 3 4

max c x min c x ai x = bi ai x bi ai x bi ai x bi ai x bi xj 0 and xj 0 are special cases of ai x bi (j-th unit vector ai , bi = 0)

feasible set of a linear programming problem can be expressed exclusively in terms of inequality constraints of the form ai x bi . Suppose there are m constraints (indexed by i = 1, . . . , m) A is a m n matrix with row vectors ai , i = 1, . . . , m b = (b1 , . . . , bm )
VU Algorithmics M. Leitner 9

Introduction

Geometry

Simplex

Duality

Denition 3 (Linear Program in General Form (Compact))

min s.t. cx Ax b x Rn

Denition 4 (Feasible Set)

P = {x Rn : Ax b}

VU Algorithmics

M. Leitner

10

Introduction

Geometry

Simplex

Duality

Where do Linear Optimization Programs arise?

Real world problems
may have a (straightforward) description as LP or may be approximated as LPs modeling is an important task! Integer Linear Programs last part of Algorithmics, mathematical programming (summer term) Transportations and Logistics Telecommunications Scheduling Manufacturing Engineering Portfolio optimization ...
VU Algorithmics M. Leitner 11

Introduction

Geometry

Simplex

Duality

Exemplary Transportation Problem (Data)

Given m plants n warehouses si supply of ith plant, i = 1, . . . , m dj demand of jth warehouse, j = 1, . . . , n cij transportation costs per unit from plant i to warehouse j
m i=1 si

n j=1 dj

(for simplicity)

Objective nd a minimum cost transportation plan such that

all demands are satised and and all supplies are respected

VU Algorithmics

M. Leitner

12

Introduction

Geometry

Simplex

Duality

Exemplary Transportation Problem (Formulation)

Decision variables xij . . . number of units to send from supply i to plant j Formulation (Linear Program)
m n

min
i=1 j=1 m

cij xij

s.t.
i=1 n

xij = dj xij = si
j=1

j = 1, . . . , n i = 1, . . . , m

xij 0

VU Algorithmics

M. Leitner

13

Introduction

Geometry

Simplex

Duality

Exemplary (Nurse) Scheduling Problem (Data)

Given planning horizon of seven days nurses work ve days in a row dj : number of nurses needed on day j, j = 1, . . . , 7

Objective identify the minimum number of nurses to be hired

VU Algorithmics

M. Leitner

14

Introduction

Geometry

Simplex

Duality

Exemplary Scheduling Problem (Formulation)

Decision variables xj . . . number of nurses starting to work on day j Formulation (Linear Program) min x1 x1 x1 x1 x1 x1 +x2 +x2 +x2 +x2 +x2 +x2 +x3 +x4 +x4 +x5 +x5 +x5 +x6 + x6 + x6 + x6 +x7 +x7 +x7 +x7 +x7

+x3 +x3 +x3 +x3 +x3

+x4 +x4 +x4 +x4

+x5 +x5 +x5

+x6 +x6

+x7 xj

d1 d2 d3 d4 d5 d6 d7 0

j = 1, . . . , 7
15

VU Algorithmics

M. Leitner

Introduction

Geometry

Simplex

Duality

Guidelines
How to formulate
1 2

dene your decision variables write constraints and objective function

Good LP formulations often have a small number of variables and constraints sparse constraint matrix Reminder: Models must be linear, e.g. you are not allowed to multiply two variables!

VU Algorithmics

M. Leitner

16

Introduction

Geometry

Simplex

Duality

Standard form problems

Denition 5 (Standard Form)
A linear programming problem of the form min s.t. cx Ax = b x0 is said to be in standard form. Interpretation x Rn , Ai is ith column of A then Ax = b can be written as n Ai xi = b i=1 Ai are resource vectors that should synthesize b by using a nonnegative amount xi of each resource ci is the unit cost of each resource
VU Algorithmics M. Leitner 17

Introduction

Geometry

Simplex

Duality

Reduction to standard form

obviously, standard form is special case of general form we will see that general form can be transformed into standard form

Note usually, the general form Ax b is used to develop the theory of linear programming when talking about algorithms, e.g. simplex method, standard form Ax = b, x 0, will be used (computationally more convenient)

VU Algorithmics

M. Leitner

18

Introduction

Geometry

Simplex

Duality

Reduction rules
Elimination of free variables
Assume xj is an an unrestricted variable in a general form problem xj xj+ xj with xj+ , xj 0

Elimination of inequality constraints

Given an inequality
n j=1 aij xj n j=1 aij xj

bi

introduce a slack variable si 0 + si = bi

n j=1 aij xj

Given an inequality
n j=1 aij xj
VU Algorithmics

bi

introduce a surplus variable si 0 si = bi

M. Leitner 19

Introduction

Geometry

Simplex

Duality

Reduction to standard form (Example)

The problem min s.t. 2x1 + 4x2 x1 + x2 3 3x1 + 2x2 = 14 x1 0 is equivalent to the standard form problem min
+ 2x1 + 4x2 4x2 + 3x1 + 2x2 2x2 = 14 + x1 , x2 , x2 , x3 0

+ s.t. x1 + x2 x2 x3 = 3

VU Algorithmics

M. Leitner

20

Introduction

Geometry

Simplex

Duality

Graphical Representation and Solution of LPs

VU Algorithmics

M. Leitner

21

Introduction

Geometry

Simplex

Duality

Graphical Representation of LPs

Motivation
provide useful geometric insights into the nature of linear programming problems LPs with two variables can (easily) be solved geometrically

HowTo (Minimization Problem)

1 2

determine feasible region move line corresponding to objective function in direction of decreasing objective value stop if no further moving is possible without leaving the feasible region optimal solutions are then given by all feasible points of objective function line
M. Leitner 22

VU Algorithmics

Introduction

Geometry

Simplex

Duality

Graphical Representation of LPs (Example 1)

min x1 x2 s.t. x1 + 2x2 3 2x1 + x2 3 x1 , x2 0 line z = c x = x1 x2 is perpendicular to c z = 2 is optimal value

Figure from Bertsimas, Tsitsiklis, 1997

VU Algorithmics

M. Leitner

23

Introduction

Geometry

Simplex

Duality

Graphical Representation of LPs (Example 2)

x1 + x2 1 and x1 , x2 0

Figure from Bertsimas, Tsitsiklis, 1997 VU Algorithmics M. Leitner 24

Introduction

Geometry

Simplex

Duality

Optimal Solutions of a Linear Program

For a Linear Program min c x s.t. Ax b with feasible set P = {x | Ax b} the following possibilities exist P = no feasible solution exists optimal cost is , i.e. inf{c x | x P} does not exist, and hence no feasible solution is optimal (LP is unbounded) a unique optimal solution exists multiple optimal solutions exist; set of optimal solutions may be bounded or unbounded
VU Algorithmics M. Leitner 25

Introduction

Geometry

Simplex

Duality

Oil Renery (Modeling)

create dierent end products from raw oil
heavy oil (H) medium oil (M) light oil (L)

dierent crack processes produce dierent quantities

crack process 1: 2H, 2M, 1L costs: 3 EUR crack process 2: 1H, 2M, 4L costs: 5 EUR

required quantities
3H, 5M, 4L

Objective: minimize costs

VU Algorithmics

M. Leitner

26

Introduction

Geometry

Simplex

Duality

Oil Renery (Variables)

Variables
x1 : level of production of crack process 1 x2 : level of production of crack process 2

Interpretation of x1 = 2.5
process 1 is applied to 2.5 units raw oil costs 2.5 3 EUR result 2.5 2 H, 2.5 2 M and 2.5 1 L Each non-negative vector (x1 , x2 ) R2 describes a certain level of production of both crack processes.

VU Algorithmics

M. Leitner

27

Introduction

Geometry

Simplex

Duality

Oil Renery (Constraints)

at least 3 units H 2x1 + x2 3 at least 5 units M 2x1 + 2x2 5 at least 4 units L x1 + 4x2 4 total costs z = 3x1 + 5x2

VU Algorithmics

M. Leitner

28

Introduction

Geometry

Simplex

Duality

Oil Renery (Linear Program)

min 3x1 + 5x2 s.t. 2x1 + x2 3 2x1 + 2x2 5 x1 + 4x2 4 x1 0 x2 0

VU Algorithmics

M. Leitner

29

Introduction

Geometry

Simplex

Duality

Oil Renery (Graphical Solution)

1 x2 0 1 0 1 1 0 C5 0 1 0 1 0 1 0 1 0 1 0 1 0 1 0 1 40 1 0 1 0 1 0 00000000 11111111 1 0 00000000 11111111 1 0 00000000 11111111 1 0 00000000 11111111 1 00000000 11111111 11 00 1 0 11111111111111 00000000000000 1 0 00000000 11111111 30 00000000000000 11111111111111 1 0 00000000 11111111 00000000000000 11111111111111 1 0 00000000 11111111 00000000000000 11111111111111 1 0 00000000 11111111 00000000000000 11111111111111 1 0 00000000 11111111 11111111111111 00000000000000 1 0 00000000 11111111 00000000000000 11111111111111 1 0 00000000 11111111 00000000000000 11111111111111 1 00000000 11111111 00000000000000 11111111111111 1 0 1 0 00000000 11111111 20 00000000000000region 11111111111111 1 0 00000000 feasible 11111111 11111111111111 00000000000000 1 0 00000000 11111111 00000000000000 11111111111111 1 0 00000000 11111111 00000000000000 11111111111111 1 0 00000000 11111111 00000000000000 11111111111111 1 0 00000000 11111111 00000000000000 11111111111111 1 00000000 11111111 11111111111111111111111 00000000000000000000000 00000000000000 11111111111111 1 0 00000000 11111111 10 11111111111111111111111 00000000000000000000000 00000000000000 11111111111111 1 0 00000000 11111111 11111111111111111111111 00000000000000000000000 00000000000000 11111111111111 1 0 00000000 11111111 11111111111111111111111 00000000000000000000000 00000000000000 11111111111111 1 0 00000000 11111111 00000000000000000000000 11111111111111111111111 00000000000000 11111111111111 1 0 00000000 11111111 11111111111111111111111 00000000000000000000000 00000000000000 11111111111111 1 0 00000000 11111111 11111111111111111111111 00000000000000000000000 00000000000000 11111111111111 1 0 00000000 11111111 1 0 1 0 1 0 1 0 1 0 1 0 1 0 11111111111111111111111 00000000000000000000000 00000000000000 11111111111111 1111111111111111111111111111111111 0000000000000000000000000000000000 x1 1 0 00000000 11111111 1 0 1 0 1 0 1 0 1 0 1 0 1 0 00000000000000000000000 11111111111111111111111 00000000000000 11111111111111 00000000 11111111 11111111111111111111111 00000000000000000000000 00000000000000 11111111111111 00000000 11111111 11111111111111111111111 00000000000000000000000 00000000000000 11111111111111 00000000 2 11111111 0 1 3 4 5 6 11111111111111111111111 00000000000000000000000 00000000000000 11111111111111 C4 00000000 11111111 1 1 0 50 1 0 1 0

C1

C2

C3

VU Algorithmics

M. Leitner

30

Introduction

Geometry

Simplex

Duality

Oil Renery (Graphical Solution)

1 x2 0 1 0 1 1 0 C5 0 1 0 1 0 1 0 1 0 1 0 1 0 1 0 111111111111111111111111111111 000000000000000000000000000000 1 40 111111111111111111111111111111 000000000000000000000000000000 1 0 111111111111111111111111111111 000000000000000000000000000000 1 0 111111111111111111111111111111 000000000000000000000000000000 1 0 00000000 11111111 000000000000000000000000000000 111111111111111111111111111111 1 0 00000000 11111111 000000000000000000000000000000 111111111111111111111111111111 1 0 00000000 11111111 000000000000000000000000000000 111111111111111111111111111111 1 0 00000000 11111111 111111111111111111111111111111 000000000000000000000000000000 1 00000000 11111111 11 00 1 0 11111111111111 00000000000000 111111111111111111111111111111 000000000000000000000000000000 1 0 00000000 11111111 30 00000000000000 11111111111111 000000000000000000000000000000 111111111111111111111111111111 1 0 00000000 11111111 00000000000000 11111111111111 111111111111111111111111111111 000000000000000000000000000000 1 0 00000000 11111111 00000000000000 11111111111111 000000000000000000000000000000 111111111111111111111111111111 1 0 00000000 11111111 00000000000000 11111111111111 000000000000000000000000000000 111111111111111111111111111111 1 0 00000000 11111111 11111111111111 00000000000000 111111111111111111111111111111 000000000000000000000000000000 z=19 e.g.: x=(3,2) 1 0 00000000 11111111 00000000000000 11111111111111 000000000000000000000000000000 111111111111111111111111111111 1 0 00000000 11111111 00000000000000 11111111111111 111111111111111111111111111111 000000000000000000000000000000 1 00000000 11111111 00000000000000 11111111111111 1 0 000000000000000000000000000000 111111111111111111111111111111 1 0 00000000 11111111 20 00000000000000 11111111111111 000000000000000000000000000000 111111111111111111111111111111 1 0 00000000 11111111 11111111111111 00000000000000 111111111111111111111111111111 000000000000000000000000000000 1 0 00000000 11111111 00000000000000 11111111111111 000000000000000000000000000000 111111111111111111111111111111 1 0 00000000 11111111 00000000000000 11111111111111 000000000000000000000000000000 111111111111111111111111111111 1 0 00000000 11111111 00000000000000 11111111111111 111111111111111111111111111111 000000000000000000000000000000 1 0 00000000 11111111 00000000000000 11111111111111 000000000000000000000000000000 111111111111111111111111111111 1 00000000 11111111 11111111111111111111111 00000000000000000000000 00000000000000 11111111111111 111111111111111111111111111111 000000000000000000000000000000 1 0 00000000 11111111 10 11111111111111111111111 00000000000000000000000 00000000000000 11111111111111 000000000000000000000000000000 111111111111111111111111111111 1 0 00000000 11111111 11111111111111111111111 00000000000000000000000 00000000000000 11111111111111 000000000000000000000000000000 111111111111111111111111111111 1 0 00000000 11111111 11111111111111111111111 00000000000000000000000 00000000000000 11111111111111 000000000000000000000000000000 111111111111111111111111111111 1 0 00000000 11111111 00000000000000000000000 11111111111111111111111 00000000000000 11111111111111 000000000000000000000000000000 111111111111111111111111111111 1 0 00000000 11111111 11111111111111111111111 00000000000000000000000 00000000000000 11111111111111 000000000000000000000000000000 111111111111111111111111111111 1 0 00000000 11111111 11111111111111111111111 00000000000000000000000 00000000000000 11111111111111 000000000000000000000000000000 111111111111111111111111111111 1 0 00000000 11111111 1 0 1 0 1 0 1 0 1 0 1 0 1 0 11111111111111111111111 00000000000000000000000 00000000000000 11111111111111 1111111111111111111111111111111111 0000000000000000000000000000000000 x1 000000000000000000000000000000 111111111111111111111111111111 1 0 00000000 11111111 1 0 1 0 1 0 1 0 1 0 1 0 1 0 00000000000000000000000 11111111111111111111111 00000000000000 11111111111111 000000000000000000000000000000 111111111111111111111111111111 00000000 11111111 11111111111111111111111 00000000000000000000000 00000000000000 11111111111111 000000000000000000000000000000 111111111111111111111111111111 00000000 11111111 11111111111111111111111 00000000000000000000000 00000000000000 11111111111111 000000000000000000000000000000 111111111111111111111111111111 00000000 2 11111111 0 1 3 4 5 6 11111111111111111111111 00000000000000000000000 00000000000000 11111111111111 C4 00000000 11111111 1 1 0 50 1 0 1 0

C1

C3

C3

VU Algorithmics

M. Leitner

31

Introduction

Geometry

Simplex

Duality

Oil Renery (Graphical Solution)

1 x2 0 1 0 1 1 0 C5 0 1 0 1 0 1 0 1 0 1 0 1 0 1 0 1 40 1 0 1 0 1 0 00000000 11111111 1 0 00000000 11111111 1 0 00000000 11111111 1 0 00000000 11111111 1 00000000 11111111 11 00 1 0 11111111111111 00000000000000 1 0 00000000 11111111 30 00000000000000 11111111111111 1 0 00000000 11111111 00000000000000 11111111111111 1 0 00000000 11111111 00000000000000 11111111111111 1 0 00000000 11111111 111111111111111111111 000000000000000000000 00000000000000 11111111111111 1 0 00000000 11111111 000000000000000000000 111111111111111111111 11111111111111 00000000000000 1 0 00000000 11111111 000000000000000000000 111111111111111111111 00000000000000 11111111111111 1 0 00000000 11111111 000000000000000000000 111111111111111111111 00000000000000 11111111111111 1 00000000 11111111 000000000000000000000 111111111111111111111 00000000000000 11111111111111 1 0 1 0 00000000 11111111 000000000000000000000 111111111111111111111 20 00000000000000 11111111111111 1 0 00000000 11111111 000000000000000000000 111111111111111111111 11111111111111 00000000000000 1 0 00000000 11111111 000000000000000000000 111111111111111111111 00000000000000 11111111111111 1 0 00000000 11111111 111111111111111111111 000000000000000000000 00000000000000 11111111111111 1 0 00000000 11111111 000000000000000000000 111111111111111111111 00000000000000 11111111111111 1 0 00000000 11111111 000000000000000000000 111111111111111111111 00000000000000 11111111111111 1 00000000 11111111 000000000000000000000 111111111111111111111 11111111111111111111111 00000000000000000000000 00000000000000 11111111111111 1 0 00000000 11111111 000000000000000000000 111111111111111111111 10 11111111111111111111111 00000000000000000000000 00000000000000 11111111111111 1 0 00000000 11111111 000000000000000000000 111111111111111111111 11111111111111111111111 00000000000000000000000 00000000000000 11111111111111 1 0 00000000 11111111 000000000000000000000 111111111111111111111 11111111111111111111111 00000000000000000000000 00000000000000 11111111111111 1 0 00000000 11111111 000000000000000000000 111111111111111111111 00000000000000000000000 11111111111111111111111 00000000000000 11111111111111 1 0 00000000 11111111 000000000000000000000 111111111111111111111 11111111111111111111111 00000000000000000000000 00000000000000 11111111111111 1 0 00000000 11111111 000000000000000000000 111111111111111111111 11111111111111111111111 00000000000000000000000 00000000000000 11111111111111 1 0 00000000 11111111 000000000000000000000 111111111111111111111 1 0 1 0 1 0 1 0 1 0 1 0 1 0 11111111111111111111111 00000000000000000000000 00000000000000 11111111111111 1111111111111111111111111111111111 0000000000000000000000000000000000 x1 1 0 00000000 11111111 000000000000000000000 111111111111111111111 1 0 1 0 1 0 1 0 1 0 1 0 1 0 00000000000000000000000 11111111111111111111111 00000000000000 11111111111111 00000000 11111111 000000000000000000000 111111111111111111111 11111111111111111111111 00000000000000000000000 00000000000000 11111111111111 00000000 11111111 000000000000000000000 111111111111111111111 11111111111111111111111 00000000000000000000000 00000000000000 11111111111111 00000000 2 11111111 000000000000000000000 5 111111111111111111111 0 1 3 4 6 11111111111111111111111 00000000000000000000000 00000000000000 11111111111111 C4 00000000 11111111 000000000000000000000 111111111111111111111 1 1 0 50 1 0 1 0

C1

C2

C3

VU Algorithmics

M. Leitner

32

Introduction

Geometry

Simplex

Duality

Oil Renery (Graphical Solution)

1 x2 0 1 0 1 1 0 C5 0 1 0 1 0 1 0 1 0 1 0 1 0 1 0 1 40 1 0 1 0 1 0 00000000 11111111 1 0 00000000 11111111 1 0 00000000 11111111 1 0 00000000 11111111 1 00000000 11111111 11 00 1 0 11111111111111 00000000000000 1 0 00000000 11111111 30 00000000000000 11111111111111 1 0 00000000 11111111 00000000000000 11111111111111 1 0 00000000 11111111 00000000000000 11111111111111 1 0 00000000 11111111 00000000000000 11111111111111 1 0 00000000 11111111 11111111111111 00000000000000 1 0 00000000 11111111 00000000000000 11111111111111 1 0 00000000 11111111 00000000000000 11111111111111 1 00000000 11111111 00000000000000 11111111111111 1 0 1 0 00000000000000000 11111111111111111 00000000 11111111 20 00000000000000 11111111111111 1 0 11111111111111111 00000000000000000 00000000 11111111 11111111111111 00000000000000 1 0 00000000000000000 11111111111111111 00000000 11111111 00000000000000 11111111111111 1 0 00000000000000000 11111111111111111 00000000 11111111 00000000000000 11111111111111 1 0 00000000000000000 11111111111111111 00000000 11111111 00000000000000 11111111111111 1 0 00000000000000000 11111111111111111 00000000 z=8.5, x*=(2, 11111111 00000000000000 11111111111111 1 11111111111111111 00000000000000000 0.5) 00000000 11111111 11111111111111111111111 00000000000000000000000 00000000000000 11111111111111 1 0 00000000000000000 11111111111111111 00000000 11111111 10 11111111111111111111111 00000000000000000000000 00000000000000 11111111111111 1 0 00000000000000000 11111111111111111 00000000 11111111 11111111111111111111111 00000000000000000000000 00000000000000 11111111111111 1 0 00000000000000000 11111111111111111 00000000 11111111 11111111111111111111111 00000000000000000000000 00000000000000 11111111111111 1 0 00000000000000000 11111111111111111 00000000 11111111 00000000000000000000000 11111111111111111111111 00000000000000 11111111111111 1 0 11111111111111111 00000000000000000 00000000 11111111 11111111111111111111111 00000000000000000000000 00000000000000 11111111111111 1 0 00000000000000000 11111111111111111 00000000 11111111 11111111111111111111111 00000000000000000000000 00000000000000 11111111111111 1 0 00000000000000000 11111111111111111 00000000 11111111 1 0 1 0 1 0 1 0 1 0 1 0 11111111111111111111111 00000000000000000000000 00000000000000 11111111111111 1111111111111111111111111111111111 0000000000000000000000000000000000 x1 1 0 111111111111111111 000000000000000000 00000000 11111111 1 0 1 0 1 0 1 0 1 0 1 0 00000000000000000000000 11111111111111111111111 00000000000000 11111111111111 000000000000000000 111111111111111111 00000000 11111111 11111111111111111111111 00000000000000000000000 00000000000000 11111111111111 00000000000000000 11111111111111111 00000000 11111111 11111111111111111111111 00000000000000000000000 00000000000000 11111111111111 00000000000000000 11111111111111111 00000000 2 11111111 0 1 3 4 5 6 11111111111111111111111 00000000000000000000000 00000000000000 11111111111111 C4 00000000000000000 11111111111111111 00000000 11111111 00000000000000000 11111111111111111 1 1 0 50 1 0 1 0

C1

C2

C3

VU Algorithmics

M. Leitner

33

Introduction

Geometry

Simplex

Duality

The Geometry of Linear Programming

VU Algorithmics

M. Leitner

34

Introduction

Geometry

Simplex

Duality

Linear Combination
Denition 6
Let x1 , x2 , . . . , xk R and 1 . = . Rk . . k Then y=
i=1 k

i xi

is called linear combination of x1 , . . . , xk .

VU Algorithmics

M. Leitner

35

Introduction

Geometry

Simplex

Duality

Specic Linear Combinations

Denition 7
If y=
i=1 k

i xi

and 0 conic k ane then y is called i=1 i = 1 k convex i=1 i = 1, 0 combination of x1 , x2 , . . . , xk .

VU Algorithmics

M. Leitner

36

Introduction

Geometry

Simplex

Duality

Sets of Linear Combinations

Denition 8
For = S Rn lin(S) cone(S) a(S) conv(S) linear conic is the ane convex

hull of S.

VU Algorithmics

M. Leitner

37

Introduction

Geometry

Simplex

Duality

Sets of Linear Combinations

VU Algorithmics

M. Leitner

38

Introduction

Geometry

Simplex

Duality

Sets of Linear Combinations

Denition 9
If L = lin(S) L = cone(S) L = a(S) L = conv(S) linear space cone , then L is a ane space convex set

VU Algorithmics

M. Leitner

39

Introduction

Geometry

Simplex

Duality

Subspaces

Denition 10
L Rn is a linear subspace of Rn there exists A Rmn with L = {x Rn | Ax = 0}.

Denition 11
L Rn is a ane subspace of Rn there exist A Rmn and b Rm with L = {x Rn | Ax = b}.

VU Algorithmics

M. Leitner

40

Introduction

Geometry

Simplex

Duality

Hyperplanes, Halfspaces, and Polyhedra

Denition 12 (Polyhedron)
A polyhedron P is a set that can be described in the form P = {x Rn | Ax b} where A Rmn and b Rm . (cmp. feasible set of LPs)

Denition 13
A set S Rn is bounded if there exists a constant K such that the absolute value of every component of every element of S is less than equal to K .

Denition 14 (Polytope)
A bounded polyhedron is called polytope, i.e. P {x Rn | ||x|| B} for some B 0.
VU Algorithmics M. Leitner 41

Introduction

Geometry

Simplex

Duality

Hyperplanes, Halfspaces, and Polyhedra

Denition 15
Let a be a nonzero vector in Rn and let b be a scalar (a) the set {x Rn | a x b} is called a halfspace (b) the set {x Rn | a x = b} is called a hyperplane a hyperplane is the boundary of a corresponding halfspace a hyperplane is an ane subspace (m = 1) halfspaces are polyhedrons each polyhedron is equal to the intersection of a nite number of halfspaces a polyhedron P is a convex set, i.e. if x, y P, then x + (1 )y P the feasible set of each LP is a convex set
VU Algorithmics M. Leitner 42

Introduction

Geometry

Simplex

Duality

Hyperplanes, Halfspaces, and Polyhedra

Figure from Bertsimas, Tsitsiklis, 1997

VU Algorithmics

M. Leitner

43

Introduction

Geometry

Simplex

Duality

Representation
Theorem (Minkowski and Weyl)
A polyhedron P Rn can be represented in the form P = conv(V ) + cone(E ) with V and E being nite subsets of Rn .

Remark polytopes can be represented as P = conv(V )

VU Algorithmics

M. Leitner

44

Introduction

Geometry

Simplex

Duality

Example

Polytope P represented as: P= x y R2 | x 1, y 1, x + y 1

VU Algorithmics

M. Leitner

45

Introduction

Geometry

Simplex

Duality

Faces
Denition 16 (Valid Inequality)
The inequality a x a0 is called a valid inequality for a polyhedron P if it is satised by all points in P.

Denition 17
Given a polyhedron P = {x Rn : Ax b}, F P is a face of P, if there exists a valid inequality a x a0 for P, such that F = {x P : a x = a0 }. a face F is a proper face if F = and F = P. A facet of P is a maximal (relative to inclusion) proper face of P. A corner point of P is a face of P consisting of one element.

VU Algorithmics

M. Leitner

46

Introduction

Geometry

Simplex

Duality

Faces

1 P

corner point: x_1 + x_2 <=2

facet: x_1 <=1

VU Algorithmics

M. Leitner

47

Introduction

Geometry

Simplex

Duality

Extreme Points
Denition 18 (Extreme Point)
Let P be a polyhedron. A vector x P is an extreme point of P if we cannot nd two vectors y, z P, both dierent from x, and a scalar [0, 1], such that x = y + (1 )z.

x is an extreme point of P u, v, w, y, z are no extreme points of P

Figure from Bertsimas, Tsitsiklis, 1997 VU Algorithmics M. Leitner 48

Introduction

Geometry

Simplex

Duality

Vertices
Denition 19 (Vertex)
Let P be a polyhedron. A vector x P is a vertex of P if there exists some c such that c x < c y for all y satisfying y P and y = x.

x is a vertex of P w is not a vertex of P

Figure from Bertsimas, Tsitsiklis, 1997 VU Algorithmics M. Leitner 49

Introduction

Geometry

Simplex

Duality

Active Constraints
Consider a polyhedron P Rn dened by linear equality and inequality constraints
ai x bi , ai x bi , ai x = bi , i M1 i M2 i M3

where M1 , M2 , and M3 are nite index sets, ai Rn , and bi R.

Denition 20
If a vector x satises ai x = bi for some i M1 , M2 , or M3 , we say that the corresponding constraint is active or binding at x . If n constraints are active at x , x satises a system of n linear equations in n unknowns. This system has a unique solution if and only if these n equations are linearly independent.
VU Algorithmics M. Leitner 50

Introduction

Geometry

Simplex

Duality

Theorem 21
Let x Rn and I = {i | ai x = bi } be the set of indices of constraints that are active at x . Then the following are equivalent (a) there exist n vectors in the set {ai | i I }, which are linearly independent (b) the span of the vectors ai , i I , is Rn , i.e. every element of Rn can be expressed as linear combinations of the vectors ai , i I (c) the system of equations ai x = bi , i I , has a unique solution

VU Algorithmics

M. Leitner

51

Introduction

Geometry

Simplex

Duality

Basic Solutions

Denition 22
Consider a polyhedron P dened by linear equality and inequality constraints, and let x be an element of Rn (a) vector x is a basic solution if
(i) all equality constraints are active (ii) out of the constraints that are active at x , there are n of them that are linearly independent

(b) if x is a basic solution that satises all of the constraints, we say that it is a basic feasible solution (BFS)

VU Algorithmics

M. Leitner

52

Introduction

Geometry

Simplex

Duality

Active Constraints and Basic Feasible Solutions

P = {(x1 , x2 , x3 ) | x1 + x2 + x3 = 1, x1 , x2 , x3 0}

3 constraints active at A, B, C , and D 2 constraints active at E , i.e. x1 + x2 + x3 = 1 and x2 = 0 A, B, and C are basic feasible solutions
Figure from Bertsimas, Tsitsiklis, 1997

VU Algorithmics

M. Leitner

53

Introduction

Geometry

Simplex

Duality

Extreme Point, Vertex, Basic Feasible Solution

Theorem 23
Let P be a nonempty polyhedron and let x P. Then, the following are equivalent: (a) x is a vertex (b) x is an extreme point (c) x is a basic feasible solution

Corollary 24
Given a nite number of linear inequality constraints, there can only be a nite number of basic feasible solutions.

Denition 25
Two distinct basic solutions are adjacent if there are n 1 linearly independent constraints that are active at both of them.
VU Algorithmics M. Leitner 54

Introduction

Geometry

Simplex

Duality

Polyhedra in Standard Form

P = {x Rn | Ax = b, x 0} Essential for the development of the simplex algorithm!

Theorem 26
Consider the constraints Ax = b and x 0 and assume that the m n matrix A has linearly independent rows. A vector x Rn is a basic feasible solution if and only if we have Ax = b, and there exists indices B(1), . . . , B(m) such that: (a) The columns AB(1) , . . . , AB(m) are linearly independent and (b) if i = B(1), . . . , B(m), then xi = 0.

VU Algorithmics

M. Leitner

55

Introduction

Geometry

Simplex

Duality

Basic Solutions for Polyhedra in Standard Form

Constructing basic solutions
1 2 3

Choose m linearly independent columns AB(1) , . . . , AB(m) Let xi = 0 for all i = B(1), . . . , B(m) Solve the system of m equations Ax = b for the unknowns xB(1) , . . . , xB(m) if a basic solution constr. is nonneg. it is a basic feasible solution if x is a basic solution, then xB(1) , . . . , xB(m) are called basic variables; the remaining variables are called nonbasic the columns AB(1) , . . . , AB(m) are called the basic columns and since they are linearly independent, they form a basis of Rm B(1), . . . , B(m) are the basic indices arranging the m basic columns next to each other, we obtain an m m matrix B called basis matrix BxB = b yields xB = B1 b

VU Algorithmics

M. Leitner

56

Introduction

Geometry

Simplex

Duality

standard form problem with n = 4 and m = 2 A1 , A2 form a basis; corresponding basic solution is infeasible (negative value of x2 is needed) A1 , A3 form another basis; basic solution is feasible
Figure from Bertsimas, Tsitsiklis, 1997

A1 , A4 do not form a basis (linearly dependent)

VU Algorithmics

M. Leitner

57

Introduction

Geometry

Simplex

Duality

The Full Row Assumption on A

Theorem 27
Let P = {x | Ax = b, x 0} be a nonempty polyhedron, where A is a matrix of dimensions m n with rows a1 , . . . , am . Suppose that rank(A) = k < m and that the rows ai1 , . . . aik are linearly independent. Consider the polyhedron Q = {x | ai1 x = bi1 , . . . , aik x = bik , x 0}. Then Q = P Which shows that the full row rank assumption on the matrix A in Theorem 26 results in no loss of generality.

VU Algorithmics

M. Leitner

58

Introduction

Geometry

Simplex

Duality

Degeneracy
at each basic solution we have at least n active constraints may be more than n (of course not linearly independent) degenerate basic solution huge impact on algorithms for solving LPs

Denition 28
A basic solution x Rn is said to be degenerate if more than n of the constraints are active at x.

VU Algorithmics

M. Leitner

59

Introduction

Geometry

Simplex

Duality

Degeneracy

Figure from Bertsimas, Tsitsiklis, 1997

The points A and C are degenerate basic feasible solutions. The points B and E are nondegenerate basic feasible solutions. The point D is a degenerate basic solution.
VU Algorithmics M. Leitner 60

Introduction

Geometry

Simplex

Duality

Degeneracy in Standard Form Problems

in standard form, the m equality constraints are active at a basic solution having more than n active constraints is the same as having more than n m variables at zero level

Denition 29
Consider the standard form polyhedron P = {x Rn | Ax = b, x 0} and let x be a basic solution. Let m be the number of rows in A. The vector x is a degenerate basic solution if more than n m of the components of x are zero. For a degenerate basic solution, there are typically multiple possibilities to choose the n m nonbasic variables, i.e. several bases correspond to the same basic solution.
VU Algorithmics M. Leitner 61

Introduction

Geometry

Simplex

Duality

Degeneracy and Representation

Degeneracy is, in general, not a geometric (representation independent) property, but rather depends on the particular representation of a polyhedron.

Example
P = {(x1 , x2 , x3 ) | x1 x2 = 0, x1 + x2 + 2x3 = 2, x1 , x2 , x3 0} n = 2, m = 3, and n m = 1 (1, 1, 0) is not degenerated, but (0, 0, 1) is P = {(x1 , x2 , x3 ) | x1 x2 = 0, x1 + x2 + 2x3 = 2, x1 0, x3 0} (0, 0, 1) not degenerated in this representation

VU Algorithmics

M. Leitner

62

Introduction

Geometry

Simplex

Duality

Existence of Extreme Points

Denition 30
A polyhedron P Rn contains a line if there exists a vector x P and a nonzero vector d Rn such that x + d P for all scalars .

Theorem 31
Suppose that the polyhedron P = {x Rn | ai x bi , i = 1, . . . , m} is nonempty. Then, the following are equivalent: (a) P has at least one extreme point (b) P does not contain a line (c) there exist n vectors out of a1 , . . . , am which are linearly independent

VU Algorithmics

M. Leitner

63

Introduction

Geometry

Simplex

Duality

Existence of Extreme Points

a bounded polyhedron does not contain a line the positive orthant {x | x 0} does not contain a line since a polyhedron in standard form is contained in the positive orthant, it does not contain a line either

Corollary 32
Every nonempty bounded polyhedron and every nonempty polyhedron in standard form has at least one basic feasible solution.

VU Algorithmics

M. Leitner

64

Introduction

Geometry

Simplex

Duality

Optimality of Extreme Points

Theorem 33
Consider the linear programming problem of minimizing c x over a polyhedron P. Suppose that P has at least one extreme point and that there exists an optimal solution. Then there exists an optimal solution which is an extreme point of P.

Figure from Bertsimas, Tsitsiklis, 1997

VU Algorithmics

M. Leitner

65

Introduction

Geometry

Simplex

Duality

Optimality of Extreme Points

Theorem 34 (stronger variant)
Consider the linear programming problem of minimizing c x over a polyhedron P. Suppose that P has at least one extreme point. Then either the optimal cost is equal to , or there exists an extreme point which is optimal.

Corollary 35
Consider the linear programming problem of minimizing c x over a nonempty polyhedron. Then, either the optimal cost is equal to or there exists an optimal solution.

VU Algorithmics

M. Leitner

66

Introduction

Geometry

Simplex

Duality

Summary
Conclusions regarding the solutions of LPs (a) If the feasible set is nonempty and bounded, there exists an optimal solution. Furthermore, there exists an optimal solution which is an extreme point. (b) If the feasible set is unbounded, there are three possibilities
(i) There exists an optimal solution which is an extreme point. (ii) There exists an optimal solution, but no optimal solution is an extreme point. (Only possible if the feasible set has no extreme points; This never happens in standard form problems) (iii) The optimal cost is .

(c) If the optimal cost is nite and the feasible set contains at least one extreme point, then there are only nitely many extreme points. (d) The number of extreme points may increase exponentially with the number of variables and constraints.
VU Algorithmics M. Leitner 67

Introduction

Geometry

Simplex

Duality

The Simplex Method

VU Algorithmics

M. Leitner

68

Introduction

Geometry

Simplex

Duality

Historical Notes simplex method was pioneered by Dantzig in 1947 comprehensive text: G. B. Dantzig, Linear Programming and Extensions, Princeton University Press, 1963 Basic Strategy start from a vertex (BFS) of the polyhedra
may be dicult to nd!

move from one BFS to another along the edges of the feasible set in a cost reducing direction
which directions / edges are possible? which one to choose?

repeat while further improvement is possible

global optimum?

VU Algorithmics

M. Leitner

69

Introduction

Geometry

Simplex

Duality

Optimality Conditions
basic strategy of many optimization algorithms: given a feasible solution, search its neighborhood to nd a nearby feasible solution with lower costs (as long as it exists) such algorithms terminate in a locally optimal solution which need not to be globally optimal LP: optimization of a convex function over a convex set!

Theorem 36 (Local Minima of Convex Functions)

Let F : Rn R be a convex function and S R be a convex set. Let x be an element of S. Suppose x is a local optimum for the problem of minimizing f (x) over S; i.e. there exists some > 0 such that f (x ) f (x) for all x S for which ||x x || . Then x is globally optimal, i.e. f (x ) f (x) for all x S.
VU Algorithmics M. Leitner 70

Introduction

Geometry

Simplex

Duality

Assumptions
In this chapter, we consider the standard form problem min c x s.t. Ax = b x0

where P is the corresponding feasible set A is a m n matrix with linearly independent rows Ai is the ith column of A ai is the ith row of A

VU Algorithmics

M. Leitner

71

Introduction

Geometry

Simplex

Duality

Feasible Directions
given a point x P we want to move in the direction of d Rn we do only want to consider choices for d that do not immediately take us outside the feasible set

Denition 37 (Feasible Direction)

Let x be an element of a polyhedron P. A vector d Rn is said to be a feasible direction at x, if there exists a positive scalar for which x + d P

VU Algorithmics

M. Leitner

72

Introduction

Geometry

Simplex

Duality

Basic Direction
given BFS x, i.e. Ax = b, basis matrix B, basic variables xB = (xB(1) , . . . , xB(m) ) (xB = B1 b) move away from x to x + d by selecting a nonbasic variable xj (initially zero) increasing it to x + d, with dj = 1 (zero for all other nonbasic variables) A(x + d) = b should hold (feasibility) and Ax = b holds and > 0, we need Ad = 0 0 = Ad = dB = B
1 n i=1 Ai di

m i=1 AB(i) dB(i)

+ Aj = BdB + Aj

Aj

which is called jth basic direction

VU Algorithmics

M. Leitner

73

Introduction

Geometry

Simplex

Duality

Basic Direction
equality constraints are respected when moving along the basic direction d nonnegativity constraints?
variable xj is increased other nonbasic variables stay zero

Other basic variables (a) suppose x is a nondegenerate BFS, i.e. xB > 0, and thus xB + dB 0 is is suciently small ( feasibility is maintained and d is a feasible direction) (b) if x is degenerate, d is not always a feasible direction! (a basic variable xB(i) may be zero while dB(i) is negative)
VU Algorithmics M. Leitner 74

Introduction

Geometry

Simplex

Duality

Eects on the Cost Function

jth basic direction d: rate c d of cost change along d is given by cB dB + cj , where cB = (cB(1) , . . . , cB(m) ), which is equal to cj cB B1 Aj
cj . . . cost per unit increase of xj cB B1 Aj . . . cost of the compensating change in the basic variables due to Ax = b

Denition 38 (Reduced Cost)

Let x be a basic solution, let B be an associated basis matrix, and let cB be the vector of costs of the basic variables. For each j, we dene the reduced cost c j of the variable xj according to the formula c j = cj cB B1 Aj (Note: reduced costs of basic variables are zero)
VU Algorithmics M. Leitner 75

Introduction

Geometry

Simplex

Duality

Optimality Conditions
Theorem 39
Consider a basic feasible solution x associated with a basis matrix B, and let c be the corresponding vector of reduced costs. (a) if c 0, then x is optimal (b) if x is optimal and nondegenerate, then c 0

Denition 40
A basis matrix B is said to be optimal if: (a) B1 b 0, and (b) c = c cB B1 A 0

VU Algorithmics

M. Leitner

76

Introduction

Geometry

Simplex

Duality

Development of the Simplex Method

Main Missing Part

How to move to a better BFS, when a protable basic direction is discovered assume that every BFS is nondegenerate reduced costs have been computed and c j < 0 (jth basic direction d) when moving along d, xj becomes positive; we say that xj (or Aj ) enters the basis

VU Algorithmics

M. Leitner

77

Introduction

Geometry

Simplex

Duality

How far can we move?

j : c j < 0: jth basic direction allows for a cost reduction since costs are decreasing along d we want to move as far as possible to x + d, where = max{ 0 | x + d P} if d 0, then x + d 0, for all 0, hence = (unbounded) if di < 0, then xi + di 0 xi /di
must be satised for every i with di < 0 = min{i|di <0} (xi /di ) since we only need to consider basis variables = min{i=1,...,m|dB(i) <0} (xB(i) /dB(i) ) > 0, because xB(i) > 0 (nondegeneracy)

VU Algorithmics

M. Leitner

78

Introduction

Geometry

Simplex

Duality

New Basic Feasible Solution y

y = x + d since xj = 0 and dj = 1, we have yj = > 0 Let l be a minimizing index, i.e. xB(l) xB(i) = min dB(l) {i=1,...,m|dB(i) <0} dB(i) =

in particular dB(l) < 0 and xB(l) + dB(l) = 0 basic variable xB(l) becomes zero, whereas nonbasic variable xj becomes positive xj should replace xB(l) in the basis, i.e. B is replaced by new basis matrix B by replacing AB(l) with Aj we have y = x and c y < c x
VU Algorithmics M. Leitner 79

Introduction

Geometry

Simplex

Duality

New Basic Feasible Solution y

Theorem 41
Let y, B, be constructed as detailed before. Then (a) The columns AB(i) , i = l, and Aj are linearly independent and, therefore B is a basis matrix (b) The vector y = x + d is a basic feasible solutions associated with the basis matrix B.

VU Algorithmics

M. Leitner

80

Introduction

Geometry

Simplex

Duality

An Iteration of the Simplex Method

1 2

start with a basis (columns AB(1) , . . . , AB(m) ), and associated BFS x compute reduced costs c j = cj cB B1 Aj for all nonbasic indices j; if all of them are nonnegative, current solution is optimal (algorithm terminates); else choose some variable j for which c j < 0 compute u = dB = B1 Aj ; if no component of u is positive, we have = and the optimal cost is (algorithm terminates) if some component of u is positive, let = xB(i) {i=1,...,m|ui >0} ui min

let l be such that = xB(l) /ul ; form new basis by replacing AB(l) with Aj (xj enters basis, xB(l) leaves basis)
M. Leitner 81

VU Algorithmics

Introduction

Geometry

Simplex

Duality

Correctness of Simplex (Nondegenerate case)

Theorem 42
Assume that the feasible set is nonempty and that every basic feasible solution is nondegenerate. Then, the simplex method terminates after a nite number of iterations. At termination, there are the following possibilities: (a) We have an optimal basis B and an associated basic feasible solution which is optimal. (b) We have found a vector d satisfying Ad = 0, d 0, and c d < 0, and the optimal cost is .

VU Algorithmics

M. Leitner

82

Introduction

Geometry

Simplex

Duality

The Simplex Method for Degenerate Problems

Suppose, we apply the same algorithm in the presence of degeneracy. Then, two new possibilities may be encountered: (a) If current BFS x is degenerate, can be equal to zero and thus y=x
happens if some xB(l) is equal to zero and dB(l) < 0 still, we can dene a new basis B, by replacing AB(l) by Aj

(b) if > 0, more than one of the original basis variables may become zero at x + d; since only one variable exists the basis, we get a new BFS which is degenerate basic changes while staying at the same BFS do not reduce cost degeneracy may lead to cycling (i.e. algorithm may loop indenitely)

VU Algorithmics

M. Leitner

83

Introduction

Geometry

Simplex

Duality

Simplex and Degeneracy (Example)

Figure from Bertsimas, Tsitsiklis, 1997

VU Algorithmics

M. Leitner

84

Introduction

Geometry

Simplex

Duality

Pivot Selection

simplex algorithm has certain degrees of freedom:

selection of nonbasic variable xj with negative reduced cost
Dantzigs rule: most negative reduced cost largest cost decrease (more computational eort!) Blands rule: smallest subscript (smallest index j such that c j < 0) (Devex, steepest edge, candidate lists, partial pricing,. . . )

selection of variable leaving basis (if multiple indices attain the minimum in denition of )
Blands rule: smallest subscript

rules for making such choices are called pivoting rules

VU Algorithmics

M. Leitner

85

Introduction

Geometry

Simplex

Duality

Full Tableau Implementation

Structure of the representation cT B1 b B B1 b or, in more detail cT xB B xB(1) . . . xB(m) c1 | B1 A1 | ... ... cn | B1 An | cT cT B1 A B B1 A

VU Algorithmics

M. Leitner

86

Introduction

Geometry

Simplex

Duality

Full Tableau Implementation Iteration

start with tableau associated with basis matrix B and BFS x examine reduced costs (0th row of tableau); if they are nonnegative, current solution is optimal; else choose some j with c j < 0 consider vector u = B1 Aj (jth column of tableau):
if no component of u is positive, optimal cost is ; termination for each i with ui > 0 compute xB(i) /ui ; let l be index of a row with smallest ratio; column AB(l) exists basis, Aj enters basis
4

1 2

add to each row a constant multiple of the pivot row (lth row) so that ul (pivot element) becomes one and all other entries of the pivot column become zero

VU Algorithmics

M. Leitner

87

Introduction

Geometry

Simplex

Duality

Full Tableau Implementation (Example from [1])

After introducing slack variables, we obtain the standard form problem

VU Algorithmics

M. Leitner

88

Introduction

Geometry

Simplex

Duality

Full Tableau Implementation (Example from [1])

Basic feasible solution: x = (0, 0, 0, 20, 20, 20); accordingly B(1) = 4, B(2) = 5, B(3) = 6; B = I

x1 has negative reduced costs (enters basis) pivot column u = (1, 2, 2); form xB(i) /ui minimum at i = 2 and i = 3; choose l = 2, i.e. x5 leaves the basis
VU Algorithmics M. Leitner 89

Introduction

Geometry

Simplex

Duality

Full Tableau Implementation (Example from [1])

New (degenerate) BFS: x = (10, 0, 0, 10, 0, 0); B(1) = 4, B(2) = 1, B(3) = 6

choose x3 to enter basis; pivot column u = (1, 1, 1) u3 < 0, hence calculate xB(i) /ui for i = 1, 2 only tie; we chose x4 to exit the basis
VU Algorithmics M. Leitner 90

Introduction

Geometry

Simplex

Duality

Full Tableau Implementation (Example from [1])

New (degenerate) BFS: x = (0, 0, 10, 0, 0, 10)

x2 enters basis, x6 exists basis

VU Algorithmics

M. Leitner

91

Introduction

Geometry

Simplex

Duality

Full Tableau Implementation (Example from [1])

New BFS: x = (4, 4, 4, 0, 0, 0)

optimal solution, since all reduced costs are nonnegative!

VU Algorithmics

M. Leitner

92

Introduction

Geometry

Simplex

Duality

Full Tableau Implementation (Example from [1])

Feasible Set; Path A-D-B-E

Figure from Bertsimas, Tsitsiklis, 1997 VU Algorithmics M. Leitner 93

Introduction

Geometry

Simplex

Duality

Full Tableau Implementation: Cycling (Example from [1])

Initial tableau (pivot elements are marked using *)

pivoting rules nonbasic variable with most negative reduced cost c j enters the basis out of eligible basic variables, the one with smallest index (subscript) exists the basis
VU Algorithmics M. Leitner 94

Introduction

Geometry

Simplex

Duality

After iteration 1:

After iteration 2:

VU Algorithmics

M. Leitner

95

Introduction

Geometry

Simplex

Duality

After iteration 3:

After iteration 4:

VU Algorithmics

M. Leitner

96

Introduction

Geometry

Simplex

Duality

After iteration 5:

After iteration 6: same basis and tableau that we started with (cycling!)

VU Algorithmics

M. Leitner

97

Introduction

Geometry

Simplex

Duality

Anticycling
appropriate pivoting rules prevent cycling Lexicographic pivoting rule
choose entering variable arbitrarily (with neg. reduced cost) for each row i with ui > 0, divide ith row of tableau by ui and chose lexicographic smallest row to identify variable leaving the basis

Blands Rule
nd smallest j for which c j < 0; xj enters basis in case of ties for exiting variables choose the one with smallest i

using one of these rules, the simplex algorithm terminates after nitely many steps

Denition 43 (Lexicographically Smaller)

A vector u Rn is said to be lexicographically larger (smaller) than another vector v Rn if u = v and the rst nonzero component of u v is positive (negative).
VU Algorithmics M. Leitner 98

Introduction

Geometry

Simplex

Duality

Finding an initial BFS

to start the simplex algorithm, an initial BFS is needed sometimes straightforward: Ax b, b 0 Ax + s = b (x, s) = (0, b) in general, however, nding a BFS is not easy ( auxiliary LP)

VU Algorithmics

M. Leitner

99

Introduction

Geometry

Simplex

Duality

Finding an initial BFS

Consider the problem
min c x s.t. Ax = b x0

w.l.o.g. we assume b 0 (possibly multiply some constraints by 1) introduce articial variables y Rm Auxiliary problem
m

min
i=1

yi

s.t. Ax + y = b x, y 0

which has the BFS x = 0 and y = b.

VU Algorithmics M. Leitner 100

Introduction

Geometry

Simplex

Duality

The Two-Phase Simplex Method (Phase I)

1 2

possibly multiply some constraints by 1 so that b 0 add articial variables y1 , . . . , ym , if necessary and apply simplex to auxiliary problem with cost m yi i=1 if optimal cost in auxiliary problem is positive, original problem is infeasible (termination) if optimal cost in auxiliary problem is zero, feasible solution to original problem has been found; if no articial variable is in the nal basis, eliminate articial variables and columns to obtain feasible basis for original problem Repeat: if lth basic variable is articial, examine lth entry of the columns B1 Aj , j = 1, . . . , n; if all entries are zero, lth constraint is redundant and is eliminated; otherwise apply change of basis: lth basic variable exists and xj enters (lth entry of jth column is nonzero)
M. Leitner 101

VU Algorithmics

Introduction

Geometry

Simplex

Duality

Two-Phase Simplex (Example from [1])

Initial LP

Auxiliary LP

VU Algorithmics

M. Leitner

102

Introduction

Geometry

Simplex

Duality

Two-Phase Simplex (Phase I, Example from [1])

BFS (x5 , x6 , x7 , x8 ) = b = (3, 2, 5, 1), B = I cB = (1, 1, 1, 1), c i = cB Ai

x4 enters the basis, x8 exits the basis

VU Algorithmics M. Leitner 103

Introduction

Geometry

Simplex

Duality

Two-Phase Simplex (Phase I, Example from [1])

B = I, only 0th row of the tableau changes

x3 enters the basis, x4 exits the basis

VU Algorithmics

M. Leitner

104

Introduction

Geometry

Simplex

Duality

Two-Phase Simplex (Phase I, Example from [1])

x2 enters the basis, x6 exits the basis (degenerate pivot with = 0)

VU Algorithmics

M. Leitner

105

Introduction

Geometry

Simplex

Duality

Two-Phase Simplex (Phase I, Example from [1])

x1 enters the basis, x5 exits the basis

VU Algorithmics

M. Leitner

106

Introduction

Geometry

Simplex

Duality

Two-Phase Simplex (Phase I, Example from [1])

cost is zero, i.e. feasible solution to the original problem x7 is still in the basis and we need to drive x7 out of the basis x7 is 3rd basis variable and 3rd entry of columns B1 Aj , j = 1, . . . , 4, (original variables) is zero indicates that A has linearly dependent rows and we can remove 3rd row (redundant constraint)
VU Algorithmics M. Leitner 107

Introduction

Geometry

Simplex

Duality

Two-Phase Simplex (Phase I, Example from [1])

Initial tableau for original problem:

we may now compute reduced costs of the original variables and apply simplex to original problem articial variable x8 was unnecessary (since x4 appears in a single constraint only) in this case (typical example: slack variable) we can let that variable in the initial basis
VU Algorithmics M. Leitner 108

Introduction

Geometry

Simplex

Duality

Big-M Method
Auxiliary problem with y Rm
n m

min
j=1

cj xj + M
i=1

yi

s.t. Ax + y = b x0 y0

combines both faces into a single one for suciently large M, articial variables are driven to zero

VU Algorithmics

M. Leitner

109

Introduction

Geometry

Simplex

Duality

Computational Complexity of the Simplex Algorithm

computational eort at each iteration: O(mn) number of iterations?
for all known variants, there are worst case examples such that exponentially many extreme points are visited (Klee-Minty cube) in practice usually only O(m) pivots are needed

Important open question

Is it true that for every pivoting rule there are examples where the simplex method takes an exponential number of iterations?

VU Algorithmics

M. Leitner

110

Introduction

Geometry

Simplex

Duality

Computational Complexity - Spanning Path

Figure from Bertsimas, Tsitsiklis, 1997

max
i=1

xi x1 1

xi1 xi 2n

1 xi1

i = 2, . . . , n 2n 1 pivots (costs are

111

unit cube has vertices, simplex needs strictly decreasing along path due to )
VU Algorithmics M. Leitner

Introduction

Geometry

Simplex

Duality

Alternative Approaches
There are algorithms that solve LPs in polynomial time: ellipsoid method: L. Khachian (1979): mainly of theoretical value (numerics, long running times) interior point methods: N. Karmarkar (1984), partly competitive (no warm start), very important for theoretical research! Simplex warmstart is possible (e.g. after adding rows or columns) has good practical (empirical) performance is by far the most commonly used method in practice and incorporated in most commercial codes for LP

VU Algorithmics

M. Leitner

112

Introduction

Geometry

Simplex

Duality

Summary
simplex moves from on extreme point of the feasible set to another, each time reducing the cost, until an optimal solution is reached cornerstones
optimality conditions (nonnegative of reduced costs) allow to test whether current basis is optimal systematic method for performing basis changes while not optimal

degeneracy causes substantial diculties including the possibility of cycling (since a change of basis may keep us at the same BFS, with no resulting cost improvement) cycling can be avoided by choosing suitable pivoting rules two-phase simplex can be used to obtain initial BFS simplex method is a rather ecient algorithm in practice

VU Algorithmics

M. Leitner

113

Introduction

Geometry

Simplex

Duality

Duality Theory

VU Algorithmics

M. Leitner

114

Introduction

Geometry

Simplex

Duality

Motivation
Consider the Primal Problem min c x s.t. Ax = b x0 with an optimal solution x (assumed to exist). Relaxed Problem: g (p) = min c x + p (b Ax) x0 where p is a price vector (with dimension of b) and p (b Ax) a penalty.
VU Algorithmics M. Leitner 115

Introduction

Geometry

Simplex

Duality

Motivation
Since the relaxed problem allows for more options, we should have g (p) c x . Indeed, g (p) = min c x + p (b Ax) c x + p (b Ax ) = c x
x0

since x is feasible and satises Ax = b. Hence, each p yields a lower bound g (p) for the optimal cost c x .

Dual problem
search for the tightest lower bound of the type max g (p) Main result in duality theory: Optimal cost in the dual problem is equal to the optimal cost c x in the primal (Violating Ax = b has no value if prices are chosen according to an optimal solution for the dual problem)
VU Algorithmics M. Leitner 116

Introduction

Geometry

Simplex

Duality

Motivation

By denition of g (p) we have g (p) = min c x + p (b Ax) = p b + min(c p A)x

x0 x0

Note that min(c p A)x =

x0

0 if c p A 0 otherwise

Thus, when maximizing g (p), we only need to consider values of p for which g (p) is not .

VU Algorithmics

M. Leitner

117

Introduction

Geometry

Simplex

Duality

Motivation
Dual (Linear Programming) Problem
The dual problem is the same as the linear programming problem max p b s.t. p A c

Note: we started with equality constraints Ax = b and ended with no constraints on the sign of the price vector p

VU Algorithmics

M. Leitner

118

Introduction

Geometry

Simplex

Duality

Motivation
We can replace inequality constraints (primal), i.e. Ax b, by Ax s = b, x 0, which can be written as [A | I] Leading to the dual constraints p [A | I] c | 0 , or equivalently p A c , p 0. x = b. s

VU Algorithmics

M. Leitner

119

Introduction

Geometry

Simplex

Duality

Motivation

Also, if x is free rather than sign-constrained, we use min(c p A)x =

x

if c p A = 0 otherwise

to end up with the constraint pA=c in the dual.

VU Algorithmics

M. Leitner

120

Introduction

Geometry

Simplex

Duality

Motivation
Construction of the dual of a minimization problem we have a vector of parameters (dual variables) p for every p we have a method for obtaining a lower bound on the optimal primal cost dual problem is a maximization problem that looks for the tightest lower bound for some vectors p we get (no useful information) we only need to maximize over those p leading to nontrivial lower bounds

VU Algorithmics

M. Leitner

121

Introduction

Geometry

Simplex

Duality

The dual problem

Let A be a matrix with rows ai and columns Aj . Then the following denes a pair of primal and dual problems:

min c x s.t. ai x bi ai x bi ai x = bi xj 0 xj 0 xj free i M1 i M2 i M3 j N1 j N2 j N3

max p b s.t. pi 0 pi 0 pi free p Aj c j p Aj c j p Aj = c j i M1 i M2 i M3 j N1 j N2 j N3

VU Algorithmics

M. Leitner

122

Introduction

Geometry

Simplex

Duality

Example

Primal min x1 + 2x2 + 3x3 s.t. x1 + 3x2 = 5 2x1 x2 + 3x3 6 x3 4 x1 0 x2 0 x3 free

Dual max s.t. 5p1 + 6p2 + 4p3 p1 free p2 0 p3 0 p1 + 2p2 1 3p1 p2 2 3p2 + p3 = 3

VU Algorithmics

M. Leitner

123

Introduction

Geometry

Simplex

Duality

Relations between primal and dual variables and constraints

PRIMAL constraints minimize bi bi = bi 0 0 free DUAL variables maximize 0 0 free cj cj = cj

variables

constraints

Theorem 44
If we transform the dual into an equivalent minimization problem and then form its dual, we obtain a problem equivalent to the original problem. Or more compact: The dual of the dual is the primal.
VU Algorithmics M. Leitner 124

Introduction

Geometry

Simplex

Duality

Duality Theorems
Theorem 45 (Weak duality)
If x is a feasible solution to the primal problem an p is a feasible solution to the dual problem, then p b c x. Hence the cost of any dual solution is a lower bound for the optimal cost.

Corollary 46
(a) If the optimal cost in the primal is (unbounded), then the dual problem must be infeasible. (b) If the optimal cost in the dual is + (unbounded), them the primal problem must be infeasible.

VU Algorithmics

M. Leitner

125

Introduction

Geometry

Simplex

Duality

Duality Theorems
Corollary 47
Let x and p be feasible solutions to the primal and the dual, respectively, and suppose that p b = c x. Then x and p are optimal solutions to the primal and the dual, respectively.

Theorem 48 (Strong duality)

If a linear programming problem has an optimal solution, so does its dual, and the respective optimal costs are equal.

Theorem 49 (Complementary Slackness)

Let x and p be feasible solutions to the primal and the dual problem, respectively. They are optimal if and only if:
pi (ai x bi ) = 0 i (cj p Aj )xj = 0 j
VU Algorithmics M. Leitner 126

Introduction

Geometry

Simplex

Duality

Summary

for each LP, we can dene a dual LP the dual of the dual is the primal strong duality theorem! allows to develop the dual simplex algorithm (important for warm start when solving LPs) (more advanced) duality theory provides powerful tools for further insights and understanding of LPs and polyhedra

VU Algorithmics

M. Leitner

127