Optimization Partition Bound Extensions & Experiments

Majorization for CRFs and Latent Likelihoods

Tony Jebara & Anna Choromanska

Columbia University
Optimization Partition Bound Extensions & Experiments

Majorization
If cost function θ ∗ = arg minθ C (θ) has no closed form solution
majorization (IIS, GIS, etc.) uses a surrogate Q with closed form
to monotonically improve from initial θ0 .
Find bound Q(θ, θi ) ≥ C (θ)
where Q(θi , θi ) = C (θi )
Update θi+1 = arg minθ Q(θ, θi )
Repeat

Majorization preferred until [Wallach ’03, Andrew & Gao ’07].

For example, both IIS and GIS are slower than 1st order methods.
The culprit: loose and complicated bounds.

Let’s fix this!!!

Optimization Partition Bound Extensions & Experiments

Partition Function Bound

For log-linear model partition functions
Z (θ) = y h(y ) exp(θ > f(y ))
P

our tighter quadratic bound is

ln Z (θ) ≤ ln z + 12 (θ − θ̃)> Σ(θ − θ̃) + (θ − θ̃)>µ

Init z → 0+ , µ = 0, Σ = zI
For each y ∈ Ω {
α = h(y ) exp(θ̃ > f(y ))
l = f(y ) − µ
tanh( 1 ln(α/z)) >
Σ + = 2 ln(α/z)
2
ll
α
µ + = z+α l
z += α }
Optimization Partition Bound Extensions & Experiments

Extensions & Experiments

Graphical models: use efficient message-passing of bounds.
High dimensional models: use fast low-rank bounds.

Latent models: bounds find better local maxima, faster.