Quadratic Programming

Insoon Yang

Department of Electrical and Computer Engineering

Seoul National University

CORE
Control + Optimization Research Lab

1 / 15
Quadratic programming (QP)

QP: minimizing a convex quadratic function over a polyhedron

1 ⊤
min z Hz + q ⊤ z + r
2 (1)
s.t. Gz ≤ w,

where H = H ⊤ ⪰ 0.

2 / 15
Geometric interpretation of QP

Figure: Geometric illustration of QP. The feasible set P , which is a polyhedron, is

shown shaded. The contour lines of the objective function, which is convex
quadratic, are shown as dashed curves. The point x⋆ is optimal.

3 / 15
Solution properties

Two possible cases (when the feasible set is nonempty):

1 The minimizer lies strictly inside the feasible polyhedron.
Q) How to find the minimizer?
2 The minimizer lies on the boundary of the feasible polyhedron.

4 / 15
Dual of QP
Consider a QP:
1 ⊤
min z Hz + q ⊤ z
2
s.t. Gz ≤ w.

What’s its dual?

When does strong duality hold?

5 / 15
KKT conditions for QP
1 ⊤
min z Hz + q ⊤ z
2
s.t. Gz ≤ w.

KKT conditions:
Primal feasibility

Dual feasibility

Complementary slackness

Stationarity

In the case of QP, KKT conditions provide necessary and sufficient

conditions for optimality.
6 / 15
Active constraints and degeneracies

Recall...
set of active constraints at z: A(z) = {i | Gi z = wi }
set of inactive constraints at z: N A(z) = {i | Gi z < wi }

Definition (Linear independence constraint qualification (LICQ))

We say that LICQ holds at z ⋆ if the matrix GA(z ⋆ ) has full row rank.

Definition
The QP is said to be
primal degenerate if there exists an optimal point z ⋆ such that the
LICQ does not hold at z ⋆
dual degenerate if its dual problem is primal degenerate.

7 / 15
Suppose the QP is not primal degenerate.

Then, λ⋆N A = 0 and GA H −1 G⊤

A is invertible.

Thus, the dual QP has a unique solution

LICQ always holds for dual QPs and thus dual degeneracy can never
occur for QPs with H ≻ 0.

8 / 15
Example: Constrained least squares problem

Constrained least spares problem or constrained linear regression problem

min ∥Az − b∥22

s.t. li ≤ zi ≤ bi , i = 1, . . . , n

is a QP.

9 / 15
Example: LP with Random Cost
min c⊤ z
s.t. Gz ≤ h
Az = b,
where the cost vector c ∈ Rn is random, with mean value c̄ and covariance
E[(c − c̄)(c − c̄)⊤ ] = Σ. Note that
E[c⊤ z] = c̄⊤ z, var[c⊤ z] = E[(c⊤ z − Ec⊤ z)2 ] = z ⊤ Σz.
A way to take variance (or risk) into account is to minimize
E[c⊤ z] + γvar[c⊤ z], γ ≥ 0.
The resulting risk-sensitive optimization problem is a QP:
min c̄⊤ z + z ⊤ Σz
s.t. Gz ≤ h
Az = b
10 / 15
Optimization Solvers

11 / 15
Idea of interior point methods

min c⊤ x
s.t. a⊤
i x ≤ bi , i = 1, . . . , 6

Figure: Central path for an LP with n = 2 and m = 6. The dashed curves show
three contour lines of the logarithmic barrier function ϕ. The central path
converges to the optimal point x⋆ as t → ∞.

12 / 15
Solvers for LP and QP

CPLEX
https://community.ibm.com/community/user/datascience/
blogs/xavier-nodet1/2020/07/09/cplex-free-for-students

Gurobi
https://www.gurobi.com/academia/
academic-program-and-licenses/

MOSEK
https://www.mosek.com/products/academic-licenses/

13 / 15
Solvers for convex optimization

CVX
http://cvxr.com/cvx/

14 / 15
Solvers for more general nonlinear optimization

Artelys Knitro
https://www.artelys.com/solvers/knitro/

SNOPT, IPOPT, etc...

15 / 15