CHAPTER 7: SUPPORT VECTOR MACHINES (NOTES)

1) Support vector machines can be seen as a dual to maximum-margin

classifier. The dual is obtained by applying Langrangian optimization
theory to maximum-margin classifier optimization problem (In which we
solve the convex optimization problem by means of quadratic
programming) Why the concept of convexity has been introduced at
a very first place? The consequences of dual representation of
maximum-margin classifier (we term as SVM) is that it enables the SVM
to extend their applicability to non-linear datasets where using kernel
trick we can obtain a non-linear SVM classifier.
2) In order to increase the generalization additional concepts (i.e., slack
variables , cost variable) has been introduced to classify SVM into soft-
margin and hard margin classifiers. The mantra is “ SIMPLE
DECISION SURFACES TEND TO GENERALIZE BETTER THAN
COMPLICATED ONE’s”
3) Langrangian dual is a technique to derive a dual for the optimization
problem.
a. For that we need to construct a new optimization problem called
Langrangian optimization problem, based on primal problem (7.3)

b. „x‟ is the prime variable and „alpha‟ is the dual variable , the vector
alpha are called Langrangian multipliers. (7.5)

Now recalling that we formulate an objective function in Chapter 6

(6.17)
 And we tried to minimize it, and try to find the minimum points of
the curve using quadratic formulations. In this case Langrangian
optimization problem the Langrangian dual has to be minimized
with respect to x and maximized with respect to alpha which
implies that these are the saddle points on the graph. And this
saddle point represent a solution for L therefore the partial
derivative with respect to x will be zero (7.8)

Definition: Saddle point (a point at which a function of two

variables has partial derivatives equal to zero but at which
the function has neither a maximum nor a minimum value)
c. Now an interesting property which is referred to be as KKT
conditions (7.11 – 7.14)
 Means a solution to Langrangian is also a solution to primal
optimization problem. We know that primal objective function is
convex and has a unique minimum point, and that minimum is a
saddle point in Langrangian where x can be computed in terms of
alpha to present a dual variable Langrangian optimization problem
And both solution concides meaning x= alpha = minimum point only
if KKT conditions are met

4) DUAL MAXIMUM MARGIN OPTIMIZATION

Taking into account general form of Langrangian optimization problem as

(7.3) now we assume that we have a training set D, and it‟s objective

function as derived by maximum margin classifier (7.29) with subject to

constraints (7.30) , the values get plucked in above and formulate it in
Langrangian optimization form.. From the above discussion we know that
saddle points in Langrangian offers a solution to primal optimization
problem so we use KKT conditions to find the saddle points (7.35-7.39)
 That gives an interesting observation about point b

We therefore construct a dual representation of maximum margin

classifier that gives us a unique way to compute b (7.56)

Unlike the way we find b ( in chapter 6, revise it and add it here the
comparison of the two)

5)