ELEC0842-001

Convex Optimization
#01

2022 Fall
Seongah Jeong

Unicon
Ubiquitous and Intelligent Communication Lab
 Logistics

Unicon
Ubiquitous and Intelligent Communication Lab
 Instructor and TAs

• Instructor: Seongah Jeong

– IT1-717
– seongah@knu.ac.kr
– https://sites.google.com/view/uniconlab/home

• Office hours: Appointments by arrangement via email

• Offline class
• All lecture slides and notices will be posted @ LMS

• TA information will be posted @ LMS

Unicon 2
Ubiquitous and Intelligent Communication Lab
 Prerequisite

• Linear algebra
– Ref: Boyd & Vandenberghe

• Probability and stochastic processes

– Ref: Bertsekas & Tsitsiklis

Unicon 3
Ubiquitous and Intelligent Communication Lab
 References

• Boyd & Vandenberghe, “Convex Optimization”,

Cambridge University Press, 2004.
– http://stanford.edu/~boyd/cvxbook/bv_cvxbook.pdf

• Calafiore & El Ghaoui, “Optimization Models”,

Cambridge University Press, 2014.

Unicon 4
Ubiquitous and Intelligent Communication Lab
 Assignments

• Assignments will be posted @ LMS

• Submission with all the details via LMS

• Solutions usually available right after the due date

– Delay penalty!

Unicon 5
Ubiquitous and Intelligent Communication Lab
 Exams

• Midterm:
– Offline 1:30~2:00 hrs

• Final:
– Offline 1:30~2:00 hrs

• Allowed to use one cheating sheet

(A4-sized & double-sided)

Unicon 6
Ubiquitous and Intelligent Communication Lab
 Grading policy

• Participation 5%
– Attendance, in-class participation, questions & discussion,
any type of interaction with me

• Midterm 40%

• Final 45%

• Assignment 10%

Unicon 7
Ubiquitous and Intelligent Communication Lab
 Overview

Unicon
Ubiquitous and Intelligent Communication Lab
 Optimization

Definition: Find an optimization variable that

minimizes (or maximizes) an objective function.

The birth of the optimization field traced back to an

astronomy problem in the 1800s.

Let us talk about the problem to see how the theory

was developed.

Unicon 9
Ubiquitous and Intelligent Communication Lab
 An astronomy problem in the early 1800s

A new planetoid (미행성) was discovered: Ceres.

To figure out the trajectory, made 19 observations

(over 42 days).
However: Lost behind the glare of the sun.
Unicon 10
Ubiquitous and Intelligent Communication Lab
 Carl Friedrich Gauss (1777 ~ 1855)

A young mathematician, named

Gauss, took an approach to
figure out the trajectory.

In the process: Developed a

math problem that later laid the
foundation of optimization field.

Unicon 11
Ubiquitous and Intelligent Communication Lab
 Gauss’s approach

observation

Sun
Fix two points.
Draw an orbit that crosses the two points.

Wish to find an orbit (determined by ) that minimizes

the sum of the squared errors.
Unicon 12
Ubiquitous and Intelligent Communication Lab
 Gauss’s problem

Unicon 13
Ubiquitous and Intelligent Communication Lab
 Gauss’s problem

Unicon 14
Ubiquitous and Intelligent Communication Lab
 Gauss’s problem

least square

Called the “least-squares” problem.

Turns out: Opened up optimization field

Reasons:

1. Closed-form solution:
2. Efficient method to compute the solution.
Unicon 15
Ubiquitous and Intelligent Communication Lab
 What people tried from that time on

Tried to translate an interested problem to a

least-squares problem.

Various translation techniques have been developed.

However: Encountered many situations in which

translation is not doable.

Another breakthrough was made in 1939, which

revolutionized the optimization field.

Unicon 16
Ubiquitous and Intelligent Communication Lab
 Leonid Kantorovich in 1939
레오니트 칸트로비치

Tried to solve a military-related

problem during WWII:
Find an optimal planning of
expenditures-returns of soldiers.
지출-비용

In the process: Formulated an

optimization problem:
Linear Program (LP)

(or linear optimization problem)

Unicon 17
Ubiquitous and Intelligent Communication Lab
 Leonid Kantorovich in 1939

No closed form solution for LP.

However: Developed an efficient

algorithm that achieves the
optimal solution.

Won him the 1975 Nobel Prize in Economics!

Unicon 18
Ubiquitous and Intelligent Communication Lab
 What people tried from that time on

Tried to translate an interested problem to a

least-squares problem or linear program (LP).

Various translation techniques have been developed.

However: Encountered still many situations in which

translation is not doable.

Hence: Tried to come up with a class of tractable

optimization problems which can be solved efficiently.

Unicon 19
Ubiquitous and Intelligent Communication Lab
 A class of tractable optimization problems

Convex Optimization

Least-squares Linear Program

1800s (LP) 1939
Quadratic Program (QP) 1956

Second-Order Cone Program

(SOCP) 1994
Semi-Definite Program
(SDP) 1994
……
Unicon 20
Ubiquitous and Intelligent Communication Lab
 Course outline
Part I: Convex optimization (4~5 weeks)
Basic concepts and several definitions
Definition of convex optimization
LP, Least-squares, QP, SOCP, SDP
Algorithms for special cases
Part II: Duality (3~4 weeks)
Strong duality  Algorithms for general convex optimization
Weak duality  How to approximate non-convex problems
Part III: Applications (3~4 weeks)
Supervised learning
Generative Adversarial Networks (GANs)
Fair classifiers
Unicon 21
Ubiquitous and Intelligent Communication Lab