Máster Universitario en Ingeniería Biomédica

Advanced Methods In Medical Signals and Images

Segmentation of Medical Images

Juan Ortuño Fisac
je.ortuno@upm.es

Biomedical Imaging Technology (BIT) www.die.upm.es/im

Departamento de Ingeniería Electrónica, ETSIT-UPM

TSIM (2022-2023) - Segmentation

of medical images
 Outline

• Outline
– Segmentation of medical images
– Classical methods
– Morphological methods
– 1st exercise, image segmentation
– Active contours
– Graph-based methods
– Active shape models (ASM)
– Machine learning: Clustering
– 2nd exercise, clustering segmentation

TSIM (2022-2023) - Segmentation of medical images 2

 Introduction to graphs

Graph G=(V,E)
Abstract representation of a set of objects where some pairs are
connected by links
– Vertices or nodes (V)
– Edges (E)

V E

TSIM (2022-2023) - Segmentation of medical images 3

 Introduction to graphs

• Weighted graph w(i,j)

– Value that quantifies the similarity between vertices i,j

i j
w(i,j)
• Undirected graph
– Edges have no orientation
i j
w(i,j)=w(j,i)
• Directed graph
– Arcs, directed edges or arrow
i j i j

w(i,j)≠w(j,i)
TSIM (2022-2023) - Segmentation of medical images 4
 Introduction to graphs

Adjacency matrix of a graph

– Square Matrix A with rows and columns labeled by graph nodes, with a 1
or 0, in A(i,j), according to whether the vertices i and j and are adjacent
or not.
– The Adjacency matrix is symmetric for undirected graphs

N nodes NxN adjacency matrix A

TSIM (2022-2023) - Segmentation of medical images 5

 Images as graphs

• Image as an undirected graph

Vertices → pixels  0, if dist ( i , j )  r
w (i, j ) = 
Edges → pixel adjacency
0, otherwise

Image (pixels) Undirected Graph

TSIM (2022-2023) - Segmentation of medical images 6

 Images as graphs

• Pixel connectivity: neighboring pixels

2D images (square grid): types of connectivity
– 4-connected
– 8-connected

4-connected 8-connected

TSIM (2022-2023) - Segmentation of medical images 7

 Images as graphs

• Pixel connectivity: neighboring pixels

3D images (Cubic grid)
– 6-connected dist ( i , j )  1
– 18-connected dist ( i , j )  2
– 26-connected dist ( i , j )  3

6-connected 18-connected N-connected

TSIM (2022-2023) - Segmentation of medical images 8

 Images as graphs

• Generalized pixel connectivity

– Two pixels are connected by edges if the gray level is
similar

e.g., Gray level similarity

Neighbor connectivity
Gerenalized connectivity

TSIM (2022-2023) - Segmentation of medical images 9

 Affinity matrix

• Weighted graphs: Affinity

– Fuzzy relation defined between nodes
– Most common case: Gray level-based affinity

(
w ( i , j ) = exp − ( I ( i ) − I ( j ) ) 2 g2
2
)
– Affinity matrix: Generalized Adjacency matrix with weights

TSIM (2022-2023) - Segmentation of medical images 10

 Affinity matrix

• Affinity matrix W
– Is a weighted adjacency matrix
– Non connected pixels -> wi,j= 0 Neighboring pixels (Small kernel)

i = 0,..., N − 1
j

w
Neighboring pixels (Large kernel)

Wi , j = w ( i , j )
w
TSIM (2022-2023) - Segmentation of medical images 11
 Affinity matrix

Neighboring pixels (Small kernel)

Gray level affinity

w
Neighboring pixels (Large kernel)

TSIM (2022-2023) - Segmentation of medical images 12

 Oversegmentations as graphs

• Graphs from regions

– Initial oversegmentation based on:
• Watershed transform
• Mathematical morphology
• Split and merge

Oversegmentation Graph of adjacent regions

TSIM (2022-2023) - Segmentation of medical images 13

 Oversegmentations as graphs

• Superpixels:
– Atomic regions, which can be used to replace the rigid
structure of the pixel grid

Buyssens P, et al. “Eikonal based region growing for superpixels generation: Application to semi-
supervised real time organ segmentation in CT images”. IRBM, vol. 35(1),pp. 20-26, 2014.

TSIM (2022-2023) - Segmentation of medical images 14

 Oversegmentations as graph

Graphs from regions

– Affinity between regions:

(
w ( i , j ) = exp − d ( i , j ) 2 d2
2
)
– Histogram-based affinity: d(i , j) =  Lki − Lkj
k

– Gradient between regions: d (i , j) =  I ( fi , j ) ds

s

• Advantages:
– Dimensionality
• Set of pixels instead of individual pixels
– Complex statistics to compute affinity

TSIM (2022-2023) - Segmentation of medical images 15

 Segmentation of graphs

• Segmentation in graph theory:

– A segmentation S is a partition of the graph G(V,E) for
which every node V belonging to S is connected

TSIM (2022-2023) - Segmentation of medical images 16

 Graph cuts

• Graph cut
– Remove a subset of edges to partition the weighted graph G(V,E) into
two disjoint sets of nodes A,B (two sub-graphs):
A B = V ; A B = 
– Cut: sum of weights for the edges that have been removed
(also known as capacity)
cut ( A, B ) =  w ( i , j )
iA , jB

This is not a graph cut!

Graph cut
Unconnected node
TSIM (2022-2023) - Segmentation of medical images 17
 Graph cuts

• Minimum cut
– Cut that minimizes the sum of weights w
• Maximizes the similarity within sets.
 
min-cut(G) = argmin   w ( i , j ) 
cut ( A ,B )
iA , jB 

High energy cut Low energy cut

TSIM (2022-2023) - Segmentation of medical images 18
 Graph cuts

• Markers
– Additional nodes with “infinity” weights
– Associated to segments
w=

TSIM (2022-2023) - Segmentation of medical images 19

 Graph cuts with markers

• Markers

Graph cuts segmentation with markers (Cardiac MRI)

TSIM (2022-2023) - Segmentation of medical images 20
 Graph cuts with markers

Segmentation of myocardium in MRI

TSIM (2022-2023) - Segmentation of medical images 21

 S-T graph cuts

• S-T graph cut

– Two markers (source and sink) connected to the other nodes
– Solve the min-cut problem, so that the source is separated from the
sink
– Maximum flow problem: Finding the maximum flow from the source
to the sink

 
min-cut(G) = argmin  w(i ,S) +  w(j,T) +  w ( i , j ) 
cut ( A ,B )
 iA jA iA , jB 

Data costs Smooth costs

TSIM (2022-2023) - Segmentation of medical images 22

 S-T graph cuts

• Min-cut/max-flow theorem:
– The maximum flow passing from the source to the sink is
equal to the total weight of the edges in the minimum cut

TSIM (2022-2023) - Segmentation of medical images 23

 Graph cuts

Boykov algorithm
– Max-flow algorithm for undirected graphs
– Two markers (source and sink)
– Solved in polynomial time

• Searching trees from the

two terminals
• Passive nodes (internals)
• Active nodes (frontier)

Min-cut == max-flow:
the maximum source-to-sink flow is equal to the capacity of
the minimum cut
Boykov et al. An Experimental Comparison of Min-Cut/Max-Flow Algorithms for Energy
Minimization in Vision, IEEE Transactions on PAMI, (26) 9, 2004

TSIM (2022-2023) - Segmentation of medical images 24

 Graph cuts

• Drawbacks of min-cuts:
• Not normalized, so can segment small objects (e.g., noise)
• Compute inter-cluster difference, but not intra-cluster
similarity

Improvement: Normalized cuts

TSIM (2022-2023) - Segmentation of medical images 25

 Normalized cuts

• Normalized cuts (N-cuts)

– Graph cut criterion evaluates inter-cluster dissimilarity
– We’d like to maximize the within-cluster similarity compared to the
inter-cluster difference
– Given a graph V, Construct sub-graphs A, B such that their within-
cluster similarity is high compared to their association with the rest of
the graph

assoc ( A,V ) =  w (i, j )

iA , jV

V assoc(A,V)

assoc ( B,V ) =  w (i, j )

iB , jV
A B assoc(B,V)
TSIM (2022-2023) - Segmentation of medical images 26
 Normalized cuts

• Normalized cuts (N-cuts)

– Two partitions A,B of a graph V B = A =V − A

cut ( A, B ) =  w (i, j ) cut ( A, B ) cut ( A, B )

iA , jB
Ncut ( A, B ) = +
assoc ( A,V ) assoc ( B,V )
assoc ( A,V ) =  w (i, j )
iA , jV

cut(A,B) assoc(B,V)

V A B assoc(A,V)

J. Shi and J. Malik. Normalized Cuts and Image Segmentation. IEEE Trans.
Pattern Analysis and Machine Intelligence,vol. 22(8)8, 2000

TSIM (2022-2023) - Segmentation of medical images 27

 Normalized cuts

• Normalized cuts (N-cuts)

– Two partitions A,B of a graph V B = A =V − A

cut ( A, B ) cut ( A, B ) assoc ( A,V ) =  w (i, j )

Ncut ( A, B ) = + iA , jV
assoc ( A,V ) assoc ( B,V )
Ncut ( A, B ) = 2 − Nassoc ( A, B )
assoc ( A, A ) assoc ( B, B )
Nassoc ( A, B ) = +
assoc ( A,V ) assoc ( B,V )

assoc(B,B)

assoc(A,A) cut(A,B)

TSIM (2022-2023) - Segmentation of medical images 28

 Spectral clustering

• Methods using the spectrum (the set its eigenvalues) of the

affinity matrix are known as spectral clustering

TSIM (2022-2023) - Segmentation of medical images 29

 Spectral clustering

Minimum cuts using eigenvectors

– Affinity matrix W
– Find a vector y such as:
• Maximize: y W y = 
T

• Subject to: yT y = 1
– Eigenvalue problem: Wy =  y yT W y = yT  y

– If y is relaxed to take on real values, the min-cut can be approximated

efficiently by solving eigenvalue problem
• Use sparse matrices libraries (LAPACK)

– Finds eigenvector y with the largest eigenvalue λ

TSIM (2022-2023) - Segmentation of medical images 30

 Spectral clustering

Normalized cuts using eigenvectors yT (D - W ) y

Ncut = miny
– Affinity matrix W yT D y
NP hard!
– Diagonal matrix D D ( i ,i ) = w ( i , j )
jV
(y discrete values)
– Find a vector y such as:
• Minimize: y T ( D - W ) y
• Subject to: y T D y = 1
– Generalized eigenvalue problem: ( D - W ) y =  D y

– If y is relaxed to take on real values, the N-cut can be approximated

efficiently by solving the generalized eigenvalue problem
• Use sparse matrices solvers

– The eigenvector y with the second smallest eigenvalue λ!

(The smallest eigenvalue is 0)

TSIM (2022-2023) - Segmentation of medical images 31

 Spectral clustering

• Basic steps of the spectral clustering

– Segmentation from eigenvectors

Image Affinity matrix Eigenvector Segmentation by

(N píxeles) (NxN) (N values) eigenvector
thresholding

TSIM (2022-2023) - Segmentation of medical images 32

 Examples

A. Delrong, Y. Boykov, “Scalable Graph-Cut Algorithm for N-D

Grids.” IEEE Conf. on Comp. Vision and Patt. Rec. (CVPR), 2008

TSIM (2022-2023) - Segmentation of medical images 33

 Examples

• Graph cuts with markers

Thoracic CT

TSIM (2022-2023) - Segmentation of medical images 34

 Examples

• Bone segmentation in CT

Y. Boykov, V. Kolmogorov. “Computing Geodesics and Minimal

Surfaces via Graph Cuts.” ICCV-2003, vol. I, pp. 26-33

TSIM (2022-2023) - Segmentation of medical images 35

 Examples

• Graph Cuts en OsiriX

TSIM (2022-2023) - Segmentation of medical images 36

 Bibliography

• Y. Boykov and O. Veksler, “Graph Cuts in Vision and Graphics: Theories

and Applications”. In Handbook of Mathematical Models in Computer
Vision,. (Springer, 2006)

• https://cs.uwaterloo.ca/~yboykov/pubs.html

• J. Shi and J. Malik “Normalized Cuts and Image Segmentation” IEEE

Transactions on Pattern Analysis and Machine Intelligence, 22(0), 2000

TSIM (2022-2023) - Segmentation of medical images 37