voxelmorph.mit.

edu
+ in future FreeSurfer
release

Unsupervised Learning of Image

Correspondences in Medical Imaging Analysis
Adrian V. Dalca
HMS, MIT
 Small vessel
disease

Strok
e
Dalca et al, MICCAI,
2014 Sridharan*, Dalca* et al,
Progression with
age

31 42.5 54 65.5 77
years years years years years
average average average average average
Sridharan*, Dalca* et al, MBIA
Scan at age
Scan at age
50
60

Dalca et al, MICCAI,

2015 Batmanghelich, Dalca et al, IPMI 2013,
 0.5 sec

baseline ours ours-

(CPU) (CPU) GPU
 0.5 sec

baseline ours ours-

(CPU) (CPU) GPU
Registrati
on

field 𝜙
𝜙

moving scan fixed scan𝒇𝒇

𝒎𝒎
Registration is fundamental in
MIA
• Register scans to a template for analysis
• Register subject scans to each other for direct
comparison
• Clinical data alignment
e.g. before and after surgery
• Segmentation
propagate anatomical labels
• Related to alignment in other fields
computer vision, 1D signals, computational biology
Pairwise
optimization
scan scan𝒇𝒇
𝒎𝒎

field 𝜙
𝜙

warped scan
𝑚𝑚

�
𝜙𝜙𝑓 𝑚𝑓,
= argmin𝜙𝜙 ‖𝑚𝑚 ∘ 𝜙
𝜙− 𝑓‖ + 𝜆𝜆Reg(𝜙𝜙)
optimal deformation field images match smooth
field

• significant development
• slow for two images
Learning-based
methods

CNN

𝜃𝑔𝜃𝑚𝑚,𝑓 𝑓 field 𝜙
𝜙

• Supervised (have example triplets {𝑚𝑚,𝑓𝑓𝜙𝜙})

• Unsupervised (only have images {𝑚𝑚,})
𝑓 (voxelmorph)
Learning-based
methods

CNN

𝜃𝑔𝜃𝑚𝑚,𝑓 𝑓 field 𝜙
𝜙

• Supervised (have example triplets {𝑚𝑚,𝑓𝑓𝜙𝜙})

• Unsupervised (only have images {𝑚𝑚,})
𝑓 (voxelmorph)

• limited use of classical modelling

• fast for new image pair
Outlin
e
• Model
• Variational Inference with neural networks
• Optimization interpretation
…

• Results (runtime and accuracy)

Loss (ℒ)

• Extensions & Applications

• Training with limited data
• Training with auxiliary
data
• Training with clinical data
• Learn deformable
templates
 Framewor
Moving
k deformation

image
(𝒎𝒎)
…
Fixed
image network 𝜃𝑔𝜃
(𝒇𝒇) parameters 𝜃𝜃
Supervised:
2
ℒ= 𝜙
𝜙−𝑔 𝑔 𝜙
𝑔𝜙𝑔
 VoxelMorph
deformation
Moving
image
(𝒎𝒎)
…
Fixed
image network 𝜃𝑔𝜃
(𝒇𝒇) parameters 𝜃𝜃
Unsupervised:
ℒ = ‖𝑚𝑚 ∘ 𝜙𝜙− 𝑓‖ + 𝜆𝜆Reg(𝜙𝜙)
images match
smooth field
 VoxelMorph
Moving
Loss deformation

image
(𝒎𝒎)
…
Fixed
image network 𝜃𝑔𝜃
(𝒇𝒇) parameters 𝜃𝜃

ℒ =‖𝑗 �
+𝑗 𝜆𝜆Reg(𝜙
𝑖 ‖𝑚
𝑗−𝑚
𝑖𝑓𝑗𝑖∘
𝑗)𝑖𝑓𝑗𝜙
𝑖𝑖𝜙
𝑖𝑖
𝑗 𝑗,𝑖 𝑖
 VoxelMorph
Moving
Loss deformation

image
(𝒎𝒎)
…
Fixed
image network 𝜃𝑔𝜃
(𝒇𝒇) parameters 𝜃𝜃

ℒ(𝜃𝜃; data) = �
‖𝑗 +𝑗‖𝑚
𝜆)𝜆𝑚
𝑖Reg(𝑔
𝑖∘−
𝑖 𝜃(𝑚
𝑔𝑖𝑓𝑖𝜃𝑓))
𝑚
.𝑓𝑖𝑔
𝑓𝑖𝜃(𝑚𝑖𝑓𝑚
. 𝑖𝑓𝑖
𝑗 𝑗,𝑖 𝑖

𝜙𝜙
 VoxelMorph
Moving
Loss deformation
Moved (𝒎𝒎 ∘ 𝑧 𝑧
image
𝝓𝝓 )
(𝒎𝒎)
…
spatial
Fixed transfor
image network 𝜃𝑔𝜃 m
(𝒇𝒇) parameters 𝜃𝜃 moved image 𝑚𝑚∘ 𝜙 𝜙
should be close to𝑓 𝑓

ℒ(𝜃𝜃; data) = �
‖𝑗 +𝑗‖𝑚
𝜆)𝜆𝑚
𝑖Reg(𝑔
𝑖∘−
𝑖 𝜃(𝑚
𝑔𝑖𝑓𝑖𝜃𝑓))
𝑚
.𝑓𝑖𝑔
𝑓𝑖𝜃(𝑚𝑖𝑓𝑚
. 𝑖𝑓𝑖
𝑗 𝑗,𝑖 𝑖

𝜙𝜙
 Training
deformation
Moving
image Moved (𝒎𝒎 ∘ 𝝓𝝓
)𝑧 𝑧
(𝒎𝒎)
…
spatial
Fixed transfor
image network 𝜃𝑔𝜃 m
(𝒇𝒇) parameters 𝜃𝜃 moved image 𝑚𝑚∘ 𝜙 𝜙
should be close to𝑓 𝑓
• SGD based techniques
• Each image pair contributes slightly to 𝜃𝜃
Classical optimization: slightly update 𝜙
𝜙for an image
pair
 Registrati
Moving
on deformation

image
(𝒎𝒎)
…
Fixed
image network 𝜃𝑔𝜃
(𝒇𝒇) parameters 𝜃𝜃
Probabilistic
model

𝑚
𝑚 ∘ 𝑧𝜙𝑧 + 𝜖=
𝜖 𝑓𝑓
𝑧~𝑧 𝒩𝒩 ;𝑧0, Λ−1
stationary velocity field

smoothness via Laplacian

Atlas-based
registration
Data: 7000 training volumes, 250 validate, 250
test Baseline: ANTs optimization method
Runtime for a new 3D image
pair

51 0.5
sec sec
ANTs NiftyReg VoxelMorph VoxelMorph [GPU]
[CPU] [CPU] [CPU]
Anatomical volume
overlap
scan scan warped m (VM) warped m
m f (baseline)
pair 1

*algorithms only see images, no segmentation

maps
Accuracy via volume overlap
(Dice) baseline
VoxelMorph
Outlin
e
• Model
• Variational Inference with neural
networks
…

• Optimization interpretation
Loss (ℒ)
• Results (runtime and accuracy)

• Extensions & Applications

• Training with limited data
• Training with auxiliary
data
• Training with clinical data
• Learn deformable
templates
Amortized analysis: training with limited
data
0.
9 ANTs (Syn)
0.8 Test Set
5 Test Set
0. (Instance-
Specific
8 Opt.)
0.7
score
Dice

0.
7

0.6
5 10 25 50 100
Training set
0. size
6
Segmentation Maps available at
training
Moving 3D Image
(𝒎𝒎)

deformatio
n field
(𝝓𝝓𝑧)𝑧
… Moved (𝒎𝒎 ∘ 𝝓𝝓
)𝑧 𝑧
Fixed 3D Image (𝒇𝒇)
spatial
transfor
param: 𝜓𝜓 m

Moving Image
Segmentatio
ns (𝒔𝒔𝒎) Move Image
transform Segmentatio
spatia ns (𝒔𝒔𝒎𝒎∘ 𝝓𝝓
)𝑧 𝑧
Fixed Image l
Segmentatio
ns (𝒔
)𝒇𝒔𝒇
Dice Loss
(ℒ)
Test time
performance
0.9

0.8
5

0.8
score
Dice

0.7
5

0.7

0.6
5 VoxelMorph VoxelMorph w/
seg
0.6
SynthMorp
(do
h we need real
data?)
Hoffmann et al in submission
IIIIIIIIII IIIIII II II I III IIIIIII II IIIIIII
I IIIIII IIIII III III

Billot MIDL 2020

https://github.com/BBillot/lab2im Billot MICCAI
Do we need anatomical images to
train?
Do we need anatomical images to
train?
Movin Deformation 𝜙 𝑣𝜙𝑣
g image via 𝑣𝑣integration
𝑚𝑚
Modalit
y
agnostic
CNN
Fixe
ℎ𝜃𝜃(𝑚𝑚, )𝑓
d
image𝑓 𝑓

Hoffman et al, in
 Do we need anatomical images to
train?
Moving Movin
labels 𝑚𝑠𝑚 g image
Label 𝑚𝑚
Image Modalit
samplin
sampling y
g model
model agnostic
(𝑧𝑔𝑠)𝑧𝑠
,𝑓 )𝑧(𝐼𝑔𝑠𝑚,𝑚𝑠𝑓𝐼𝑠 CNN
Fixed Fixe
ℎ𝜃𝜃(𝑚𝑚, )𝑓
labels
𝑓 𝑠𝑓𝑠 d
image𝑓 𝑓

Hoffman et al, in
 Do we need anatomical images to
train?
Warped
Moving Movin Deformation 𝜙 𝑣𝜙𝑣 segmentatio
labels 𝑚𝑠𝑚 g image via 𝑣𝑣integration n
𝑚𝑚
Label 𝑚𝑠𝑠
∘𝜙
𝑚 𝑣𝜙𝑣
Image Modalit
samplin
sampling y
g model
model agnostic
(𝑧𝑔𝑠)𝑧𝑠
,𝑓 )𝑧(𝐼𝑔𝑠𝑚,𝑚𝑠𝑓𝐼𝑠 CNN
Fixed Fixe
ℎ𝜃𝜃(𝑚𝑚, )𝑓 Regularization term
labels d 𝑔 𝑔 𝑟 𝑟ℒ
𝑓 𝑠𝑓𝑠
𝑟𝑟

image𝑓 𝑓 Dice term

𝑠 𝑠 𝑖 𝑖ℒ
𝑑𝑑

Hoffman et al, in
b Skull-stripped x Cross-dataset (e.g. HCP-OASIS)
VoxelMorph - NMI SynthMorph (brains)
HyperMorph:
Amortized parameter
learning
Hoopes, Hoffmann, Fischl, Guttag, Dalca, IPMI 2021
Regularization Analysis
(hyperparameters)

ℒ = ‖𝑚𝑚 ∘ 𝜙
𝜙− 𝑓‖ + 𝜆𝜆Reg(𝜙𝜙)
images match
smooth field
 HyperMorp
h VoxelMorph Strategy

Train Multiple Models Validate Each Choose Optimal

Model Hyperparameter λ

Model Using λ1

Model Using λ2

Model Using λ3 λ

Repeat with More Precise λ

Values

Hoopes et al, IPMI 2021

 HyperMorp
h VoxelMorph Strategy HyperMorph Strategy

Train Multiple Models Validate Each Choose Optimal Train Single Efficiently Optimize
Model Hyperparameter λ Model Hyperparameter λ Using
Validation Data
Model Using λ1

Model Using λ2 HyperMorph

Model
Model Using λ3 λ λ

Repeat with More Precise λ {λ}

Values

Hoopes et al, IPMI 2021

 HyperMorp
h Learned template (𝒕𝒕)

deformatio
velocit n field (𝝓𝝓
𝑖) 𝑖 Moved (𝒕
𝒕∘ )𝑖𝝓
𝝓𝑖
y
field Spatial
𝑔
𝑣 𝑣𝜽 𝜽 𝑖 𝒗𝑖𝒗 Transfor
Image )(𝒙
𝑖 𝒙𝑖 m
integratio
n layer

Loss (ℒ)

Hyperparamet
er 𝜆𝜆
Input
 HyperMorp
h Learned template (𝒕𝒕)

deformatio
velocit n field (𝝓𝝓
𝑖) 𝑖 Moved (𝒕
𝒕∘ )𝑖𝝓
𝝓𝑖
y
field Spatial
𝑔
𝑣 𝑣𝜽 𝜽 𝑖 𝒗𝑖𝒗 Transfor
Image )(𝒙
𝑖 𝒙𝑖 m
integratio
θv n layer

Loss
(ℒ)
Hypernetwork (h)
Hyperparamet
er 𝜆𝜆
Input
Baseline
Comparison

Dice

Runtime (GPU-hours)
VoxelMorph (~10 models): 765
HyperMorph: 147
Optimal Hyperparameters vary by
dataset
Optimal Hyperparameters vary by
task
… even by anatomical
region!
Template construction
Dalca, Rakic, Guttag, Sabuncu, NeurIPS 2019
Joshi et al, 2004
Template
Construction
Learned template (𝒕𝒕)

param:
deformatio
Ψ atl velocit n field (𝝓𝝓
𝑖) 𝑖 Moved (𝒕
𝒕∘ )𝑖𝝓
𝝓𝑖
y
field Spatial
𝑔𝑔
𝑣𝑣
𝜽𝑣,𝑣
𝒕,𝑖 𝒕𝒙𝒙𝑖 𝑖 𝒗𝑖𝒗 Transfor
Image )(𝒙
𝑖 𝒙𝑖 m
integratio
n layer

Loss (ℒ)
 Conditional template
construction
Learned template (𝒕𝒕)
Attribute (𝒂
)𝑖 𝒂𝑖

deformatio
velocit n field (𝝓𝝓
𝑖) 𝑖 Moved (𝒕
𝒕∘ )𝑖𝝓
𝝓𝑖
y
field Spatial
𝑔𝑔
𝑣𝑣
𝜽𝑣,𝑣
𝒕,𝑖 𝒕𝒙𝒙𝑖 𝑖 𝒗𝑖𝒗 Transfor
Image )(𝒙
𝑖 𝒙𝑖 m
integratio
n layer

Loss (ℒ)
15 90
 29 weeks 32 35 38 41 44

dHCP Dey et al, in

Acknowledgement
sDDIG) Benjamin Billot (UCL CMIC)
Guha Balakrishnan (MIT CSAIL

Bruce Fischl (HMS/MGH LCN)

John Guttag (MIT CSAIL
DDIG)
Malte Hoffmann (MGH
LCN) Andrew Hoopes
(MGH LCN)
Eugenio Iglesias (MIT CSAIL, MGH HMS, UCL
CMIC) Kathleen Lewis (MIT CSAIL DDIG)
Marianne Rakic (MIT CSAIL DDIG, ETH)
Mert Sabuncu (Cornell ECE, HMS/MGH
LCN) Amy Zhao (MIT CSAIL DDIG)
vo elmorph
• Probabilistic generative • Very fast for new image pair
model for diffeomorphisms • State-of-the-art accuracy
• Variational Inference • Diffeomorphic deformations
• Unsupervised Neural • Uncertainty estimation
Network
voxelmorph.mit.edu
vo elmorph
• Probabilistic generative • Very fast for new image pair
model for diffeomorphisms • State-of-the-art accuracy
• Variational Inference • Diffeomorphic deformations
• Unsupervised Neural • Uncertainty estimation
Network
voxelmorph.mit.edu

• Limited training data  use VM as initialization

• Segmentation at training  better test Dice
performance
• No atlas  construct atlas automatically
• Synthesis  invariant representations
vo elmorph
• Probabilistic generative • Very fast for new image pair
model for diffeomorphisms • State-of-the-art accuracy
• Variational Inference • Diffeomorphic deformations
• Unsupervised Neural • Uncertainty estimation
Network
voxelmorph.mit.edu

• Limited training data  use VM as initialization

• Segmentation at training  better test Dice
performance
• No atlas  construct atlas automatically
• Synthesis  invariant representations