Professional Documents
Culture Documents
Day 2 - Friday - 1100 AM - Tony Reeves
Day 2 - Friday - 1100 AM - Tony Reeves
Anthony P. Reeves
Vision and Image Analysis Group
School of Electrical and Computer Engineering
Cornell University
VIA
Nodule Measurement Studies
VIA
Outline
• VOLCANO Goals
• A Benchmark set if image pairs
• Results from automated and manual evaluation
• Evaluation methodology
VIA
Relative Change in Size
VIA
Volume or size measurement
VIA
Ground Truth Challenge
A Challenge for this Challenge
There is a large amount of disagreement in experienced chest
radiologist estimates of pulmonary nodule volumes
– Lung Image Database Consortium (LIDC)1: 10 mm nodule limits of
agreement: (-23.4%, 27.7%)
1. Volunteer activity
1. Volunteer participation
Participation incentives:
1. Authorship on main publication
2. Limited effort required
3. Individual feedback on performance (all published results are
anonymized), bragging rights may be claimed
4. Herding cats
VIA
Colima’s volcano in Colima, Mexico
Overview
1. Data set size: The data set must not be too large as to require to much
time to evaluate:
50 image pairs for CAD, 24 image pairs for Manual
4. CT Image quality:
All scans are low-dose, high-resolution, non-contrast, whole-lung view
with thin slices (1.25 mm for most scans)
-- A major consideration was the data available to the study
VIA
Data Selection: Four Data Subsets
VIA
Nodule Size Distribution
12
mm3
5.8 5.8 – 7.3 7.3 – 9.1 9.1 – 11.5 11.5 – 14.5 14.5 – 18.3 18.3 mm
VIA
The VOLCANO Challenge Data Set
Methods: Dataset Preparation
VIA
Examples of Nodules
VIA
Examples: No Change
VIA
Example: Moderate Change
(~27%)
Scan 1 Scan 2
VIA
Example: Different Slice Thickness
3D Visualization
(axial view)
VIA
Example: Large Change Nodule
Scan 1 Scan 2
3D Visualization
(axial view)
VIA
The Challengers
Participants: 13 Teams
Team Members Team Members
Biomedsys Chris Schaefer, Jeffrey Kitware Rick Avila
J. Bowden New York Jane P. Ko, Kira Melamud,
Definiens Markus Kietzmann, University Henry Rusinek
Rene Korn, Markus Philips Rafael Wiemker
Dillmann
Siemens AG Grzegorz Soza, Christian
Duke Qiang Li, Jiahui Wang Tietjen, Matthias Thorn
Fraunhofer Jan H. Moltz, Jan- UCLA Michael F. McNitt Gray,
MEVIS Martin Kuhnigk Yanisley Valenciaga,
Gifu Tatsuro Hayashi, Maryam Khatonabadi
Xiangrong Zhou, University of Yoshiki Kawata, Noboru
Hiroshi Fujita Tokushima Niki
Image Thomas Duindam,
Sciences Bram van Ginneken VIA Cornell Anthony. P. Reeves, Artit.
Institute C. Jirapatnakul, Alberto. M.
Biancardi
Submitted Methods:
Automation Level
Methods were ranked according to the following scale:
VIA
The Challengers (17 methods)
VIA
Results
Results: Group A
zero-change, same slice thickness
Nodule
Example: Group A with large
median size change
3D
Visualization
(axial view)
Example: Group A with large
median size change
3D
Visualization
(axial view)
Results: Group C
Nodules with actual change – small
Best nothing
The method that estimates the smallest change for zero change
Slice master
The method with the least bias with respect to slice thickness
SIMBA VIA
VIA
‘10
Participants incentives
SIMBA VIA
VIA
VOLCAMAN Study Specifications - 1
– Participants may access the study through any web connected computer that has a
display with sufficient resolution and quality.
– Participants may take as much time to mark a nodule as they want; however, the
marking time is recorded as a study item.
– Participants should either complete the marking of the current nodule or cancel the
marking of the current nodule before ending a session.
SIMBA VIA
VIA
VOLCAMAN Study Specifications - 2
SIMBA VIA
VIA
VOLCAMAN Study Specifications - 3
SIMBA VIA
VIA
VOLCAMAN Study Web-based marking
Baseline image
Baseline image to with marks Follow up image
be marked to be marked
• Randomization
– The same nodule pair ordering will be used as for VOLCANO
– The order of the nodule cases for marking will be individually
randomized for each participant
SIMBA VIA
VIA
Volcano Progress Report
VIA