Signal Processing for Big Data

Sergio Barbarossa

19/10/20 Signal Processing for Big Data 1

Clustering

Graph
partitioning

Given a graph, split in two complementary subsets S and S
c, let us associate different labels
to nodes belonging to different subsets:


si= 1, if i belongs to S,
si= -1, if i belongs to Sc

Note

0.5 (1-si sj ) = 0, if i and j belong to the same set,
0.5 (1-si sj ) = 1, if i and j belong to different sets
1 XX
Definition: Cut size = R = Aij (1 si sj )
4 i j

Problem: Split a graph in two subsets in such a way that the cut size is minimum

19/10/20 Signal Processing for Big Data 2

Clustering

Graph
partitioning

Cut size can be rewritten as

1
R = sT L s
4

Constraints: - number of nodes / cluster
- bounded norm

Problem formulation:
s = argmin sT L s

subject to si 2 { 1, 1}
N
X
s i = n1 n2
i=1
This is a combinatorial problem

19/10/20 Signal Processing for Big Data 3

Clustering

Graph
partitioning

Relaxed problem:


s = argmin sT L s
N
X
subject to s2i = N
i=1
XN
s i = n1 n2
i=1
Lagrangian: 0 1 0 1
N X
X N N
X N
X
L(s; , µ) = Ljk sj sk + @N s2j A + 2µ @n1 n2 sj A
k=1 j=1 j=1 j=1

Setting the gradient to zero, we get

Ls = s + µ1

19/10/20 Signal Processing for Big Data 4

Clustering

Graph
partitioning

Relaxed problem:

n2 n1
T
Multiplying from the left side by , we get
1 µ =
N

Introducing the vector
µ n2 n1
x := s + 1 = s + 1
N

we get
Lx = x

x
is then an eigenvector of L
n1 n2
The cut size can be rewritten as R =
N

19/10/20 Signal Processing for Big Data 5

Clustering

Graph
partitioning

Relaxed problem:


x
is then the eigenvector associated to the second smallest eigenvalue of :
L u2

The (real) solution is then
n1 n2

sR = x + 1
N

The closest binary solution is obtained by maximizing the scalar product sT sR

s
The optimal is achieved by assigning to the vertices with the largest
si = +1 n1
and to the other vertices
xi + (n1 n2 )/N si = 1

19/10/20 Signal Processing for Big Data 6

Clustering

Graph
partitioning

Example
u2
1.6

0.06

1.4
0.04
1.2

0.02
1

0.8 0

0.6
−0.02

0.4
−0.04

0.2

−0.06

0
−1 −0.5 0 0.5 1 1.5

19/10/20 Signal Processing for Big Data 7

Clustering

Graph
partitioning: Extension to K clusters
K
1 X
cut(A1 , . . . , AK ) := W (Ai , Ai )
2 i=1

with
XX
W (A, B) := aij
i2A j2B
Minimizing the cut does not prevent solutions where the partition simply separates
one individual vertex from the rest of the graph

To avoid this undesired solution, it is useful to introduce the normalized cut:

K K
1 X W (Ai , Ai ) X cut(Ai , Ai )
RatioCut(A1 , . . . , AK ) := =
2 i=1
|Ai | |Ai |
i=1

Unfortunately, minimizing RatioCut is NP-hard !

19/10/20 Signal Processing for Big Data 8

Clustering

Relaxed
minimum RatioCut (K=2)

Problem:

min RatioCut(A, A)
A⇢V

Introducing 8q
< |A|/|A|, if i 2 A
xi = q (1)
: |A|/|A|, if i 2 A
such that
n
T 1 X
x Lx = aij (xi xj )2 = . . . = |V | · RatioCut(A, A)
2 i,j=1
n
X n
X
T
xi = 1 x = 0 x2i = kxk2 = n
i=1 i=1

the partition problem can be formulated as

min xT Lx, subject to x ? 1, kxk2 = n and (1)
A⇢V

This problem is still NP-hard

19/10/20 Signal Processing for Big Data 9

Clustering

The
problem can be relaxed by removing the discreteness condition on x

The
solution of the relaxed problem is given by the eigenvector associated to
the second smallest eigenvalue of L


Generalization to K-cluster case

Introducing the matrix H 2 Rn⇥k whose coefficients are
(
p1 , if vi 2 Aj , i = 1, . . . , n; j = 1, . . . , k
hi,j = |Aj |
0, otherwise
we can write
n
X
RatioCut(A1 , . . . , Ak ) = hTi L hi = Tr HT L H
i=1
The relaxed problem becomes
min Tr HT L H , subject to HT H = I
A1 ,...,Ak

The solution is given by the k eigenvectors associated to the k smallest eigenvalues of L

19/10/20 Signal Processing for Big Data 10

Clustering

Clustering
algorithm [1]

[1] Ulrike von Luxburg, “A tutorial on spectral clustering”

19/10/20 Signal Processing for Big Data 11

Clustering

Selection
of number of clusters

The
number of clusters can be estimated by selecting the index i such that
the eigenvalue gap gi = i+1 i is sufficiently larger than the previous gaps


Numerical example

(the graph has been obtained from the data by building the 10-nearest neighbor graph) [1]

[1] Ulrike von Luxburg, “A tutorial on spectral clustering”

19/10/20 Signal Processing for Big Data 12

References

1.  Ulrike von Luxburg, “A tutorial on spectral clustering”

13
19/10/20 Signal Processing for Big Data