30th Annual International IEEE EMBS Conference
Vancouver, British Columbia, Canada, August 20-24, 2008
REVIEW - A Reference Data Set for Retinal Vessel Profiles
Bashir Al-Diri, Andrew Hunter, David Steel, Maged Habib, Taghread Hudaib and Simon Berry
Abstract— This paper describes REVIEW, a new retinal
vessel reference dataset. This dataset includes 16 images with
193 vessel segments, demonstrating a variety of pathologies and
vessel types. The vessel edges are marked by three observers
using a special drawing tool. The paper also describes the algo-
rithm used to process these segments to produce vessel profiles,
against which vessel width measurement algorithms can be
assessed. Recommendations are given for use of the dataset
in performance assessment. REVIEW can be downloaded from
http://ReviewDB.lincoln.ac.uk.
I. INTRODUCTION
This paper describes a new publicly-available reference
data set, REVIEW (Retinal Vessel Image set for Estima-
tion of Widths). REVIEW is available for download at:
http://ReviewDB.lincoln.ac.uk. A number of authors have de-
veloped algorithms to estimate the width of retinal vessels, as
these measurements may be of diagnostic value in a number
of medical conditions, including arteriosclerosis, stroke and
diabetic retinopathy [1]. A large number of authors have also
developed algorithms to segment the retinal vasculature (i.e.
to detect those pixels belonging to vessels), but without fol-
lowing this segmentation through to the actual extraction of
vessel diameter measurements. There are excellent databases
of retinal vessel pixel segmentations already in the public
domain (DRIVE [2] and STARE [3]), but none which include
width measurements. The REVIEW dataset is intended to fill
this gap. The dataset includes four subsets of images: high-
resolution, vascular disease, central light reflex and kick-
points, respectively. These are useful in presenting different
kinds of challenge, and benchmarking opportunities. Images
were assessed by three independent experts, who marked the
vessel edges. We present and justify the manual mark-up
method used below, and discuss the implications for bench-
marking and comparison of vessel measurement algorithms.
The mark-up method used is accurate to a sub-pixel level,
which allows for very accurate analysis of algorithms.
II. PREVIOUS WORK
A variety of manual or semi-automatic retinal vessel
measurement methods have been presented in the literature.
Bashir Al-Diri is with the Department of Computing and Informatics,
University of Lincoln, UK baldiri@lincoln.ac.uk
Andrew Hunter is the Dean of Research, University of Lincoln, UK
ahunter@lincoln.ac.uk
David Steel is with Sunderland Eye Infirmary, a Consultant Ophthalmol-
ogist, UK
Maged Habib is with Sunderland Eye Infirmary, a Specialist Registrar in
Ophthalmology, UK
Taghread Hudaib is with the Department of the Biological Sciences,
University of Lincoln, UK
Simon Berry is with Sunderland Eye Infirmary, an Optometrist, UK
978-1-4244-1815-2/08/$25.00 ©2008 IEEE.
Authorized licensed use limited to: University of Malaya. Downloaded on March 07,2024 at 03:25:09 UTC from IEEE Xplore. Restrictions apply.
Manual methods are slower than automated methods, and
are usually limited to the pixel resolution of the displayed
image. Most of these methods are defined with respect to a
vascular profile, and side-step the problem of defining the
“true” profile direction by requiring the user to nominate
the profile; e.g. by indicating two points with a mouse-
driven tool [4]. A 1D intensity profile may be constructed
between the indicated points using bilinear interpolation of
the pixels intersected by the line segment joining the two
points, and the width is defined with respect to this profile. In
the manual methods, the user nominates the perceived edges
of the vessel on this profile. This raises the question of the
proper definition of the vessel edge, with respect to a profile.
Rassam et. al. [5] defines the edge as lying at the kick points
– the first skew points on the slopes of the intensity profiles,
which probably correspond to the widest lateral extent of the
blood column. However, kick points are not always visible –
certainly not on smaller vessels or lower-resolution images –
and so user intuition is more often relied on. One method to
improve reliability is to measure the width along a length of
the vessel. Chapman et. al. [4] uses five profiles parallel to
the nominated one, spaced to either side along the vessel, and
averages the results from these. This may compensate both
for user error and image noise. A number of semi-automated
methods rely on the user to nominate the profile direction,
but then use an algorithm to determine the width along this
profile. Examples include Gregson’s rectangular profile [6],
the Full Width Half Maximum (FWHM) [7], the Gaussian
function [8], The sliding linear regression filter (SLRF) [4],
a double Gaussian [9] and a double Gaussian extruded in
two-dimension [10].
III. METHODS
A. Edge Marking
In this paper we introduce a new manual method to mark
the vessel edges and derive the vessel width. The human
observer first marks the vessel edge positions by clicking
a series of points with a mouse-driven tool; see figure 1.
The image can be zoomed to any desired level and the click
points are not limited to image pixel granularity, allowing
sub-pixel mark-up. The edge splines are then processed to
produce profiles, as follows:
1) A cubic spline is fit through the user-nominated edge
points, and regularly sampled (at spacing 0.2 pixels).
2) Choosing a point on one side as P1 , the nearest edge
point on the other side is found, and labelled P2 . Then,
the nearest point to P2 on the same side as P1 is located
and labelled; see figure 2(a). The point P3 is often, but
not always, the same as point P1 .
2262
 Fig. 1. An example of marking vessel edges by clicking on the edge.
3) The local segment direction (LSD) is calculated as the
mean of the perpendicular vectors on vectors P2~P1 and
P2~P3 ; see figure 2(b).
4) The vectors between P2 and a set of edge points from
the other side, including all the points between P1 and
P3 are calculated. The point P4 from that set with
a minimum deviation from π/2 is defined; see figure
diagram 2(c).
5) The point P4 is shifted along a B-spline fitted to the
edge points, to lie on a perpendicular profile with local
segment direction (LSD); see figure 2(d). The distance
between point P2 and shifted P4 is the local diameter.
The edge end points are not necessarily located on a per-
pendicular profile to the segment direction, and one side is
trimmed as necessary. See figure 3 for final results.
B. Observers
Human observers are subjective and prone to inter-
observer and intra-observer variability [11]. We thus provide
results from three independent observers, each of whom
marks a set of randomly selected vascular segments. The first
observer is an ophthalmic specialist with research interests in
automated assessment of diabetic retinopathy and evaluation
of retinal vascular geometry; he has five year experience
in retinal fundus images analysis. The second observer is
a specialist optometrist and a qualified secondary grader for
diabetic retinopathy; he has been previously involved with
different research projects involving handling and analysis of
digital images with tasks for defining vascular and non vas-
cular diabetic lesions. The third observer has a satisfactory
basic medical background being an experienced laboratory
Fig. 2. The vessel width extraction algorithm; (a) shortest distances
from both edges are calculated to provide two or three edge points; (b)
direction vectors at edge points and the local segment direction (LSD); (c)
the perpendicular direction to the LSD where point P4 should be located;
(d) P2 and P4 form a perpendicular profile with the local segment direction,
located on the tangent of the largest circle inside the segment.
2263
Authorized licensed use limited to: University of Malaya. Downloaded on March 07,2024 at 03:25:09 UTC from IEEE Xplore. Restrictions apply.
Fig. 3. Examples of extracting vessel diameters from their marked edges.
specialist. She received a training session in defining and
marking retinal vascular segments before starting indepen-
dent marking of the selected segments for the experiment.
Her initial training set was reviewed and independently
assessed by an ophthalmic consultant. The mean of the
measured diameters from the three observers is used as the
reference standard.
C. Performance evaluation
As mentioned earlier, vessel diameter measurement algo-
rithms fall into two classes – semi-automated algorithms that
work on defined profiles, and fully automated algorithms
that measure a full vessel. The reference dataset may be
used to assess either class of algorithm. In the latter case, it
is necessary to determine the diameter measurement of the
algorithm through each centre-point in the reference dataset.
If a width measurement algorithm has a consistent bias (non-
zero mean error), this can be compensated for by subtraction
or by multiplication of the bias, whereas fluctuations in the
error cannot be so-compensated. For this reason the mean
measurement error is ignored and the standard deviation of
the measurement error is used to assess the performance of
the algorithms. The width difference equals:
χi = ωi − ψi ,
where ωi is the estimated width and ψi is the corresponding
benchmark width. The standard deviation of the width dif-
ference, σχ , is a useful measure of algorithm performance.
It is unaffected by correction for additive bias. After correct-
ing width for multiplicative bias, the width differences, ηi
becomes
ηi = ωi (ψ̄/ω̄) − ψi .
The corresponding standard deviation, ση , characterizes
performance after correction for multiplication bias.
IV. MATERIALS
Images for the experiment were selected by a consultant
ophthalmologist from the fundus image database of the dia-
betic retinopathy clinic at Sunderland Eye Infirmary. Retinal
photography was performed to all patients according to a
standardised protocol as part of their routine clinical care.
Mydriasis was induced using tropicamide (1%) eye drops.
All digital fundus images were graded by a qualified diabetic
retinopathy grader using the developed system of assessment
 Image# Formal Grading Dataset Obs Mean Std Std 5% 95% Min Max
1 Level 3 = Severe Non-Proliferative Retinopathy Dev E
2 Level 2 = Moderate Non-Proliferative Retinopathy HRIS O 4.35 1.26 0.03 4.30 4.40 1.51 7.35
3 Level 3 = Severe Non-Proliferative Retinopathy O1 4.12 1.25 0.03 4.07 4.17 1.39 7.14
4 Level 1 = Minimal Non-Proliferative Retinopathy O2 4.35 1.35 0.03 4.30 4.41 1.21 7.61
O3 4.58 1.26 0.03 4.53 4.63 1.88 7.44
TABLE I
VDIS O 8.85 2.57 0.05 8.74 8.96 3.28 18.60
T HE DISEASE OF IMAGES IN THE HIGH RESOLUTION IMAGE SET. O1 8.50 2.54 0.05 8.39 8.60 2.48 18.33
O2 8.91 2.69 0.06 8.79 9.02 3.12 19.30
O3 9.15 2.67 0.06 9.04 9.26 3.68 18.47
CLRIS O 13.80 4.12 0.24 13.32 14.28 6.85 21.73
O1 13.19 4.01 0.24 12.72 13.66 6.71 21.27
of the EURODIAB study based on the principles of the O2 13.69 4.22 0.25 13.19 14.18 6.50 22.10
Modified Airline House classification scheme as described O3 14.52 4.26 0.25 14.02 15.01 7.11 22.65
by Aldington and his co-workers [12]. Four image sets were KPIS O 7.40 0.36 0.03 7.34 7.45 6.41 8.38
chosen to assess the accuracy and precision of the vessel O1 7.74 0.63 0.05 7.64 7.84 5.49 9.10
O2 7.29 0.49 0.04 7.21 7.36 6.07 7.94
width measurement algorithms in the presence of pathology O3 7.17 0.37 0.03 7.11 7.23 6.21 8.22
and a central light reflex, and to compare the performance TABLE II
of the proposed algorithms with manual measurements with D ESCRIPTIVE STATISTICS OF THE PROFILE DIAMETERS .
accuracy up to 0.25 of a pixel. These image sets contain
5066 manually marked profiles. The descriptive statistics of
the manual segment width measurements for the image sets HRIS VDIS CLRIS KPIS
are summarized in Table II and the standard deviations of Err SD Err SD Err SD Err SD
σξ ση σξ ση σξ ση σξ ση
the width measurements are summarized in Table III. O1 0.288 0.303 0.543 0.560 0.567 0.573 0.417 0.394
O2 0.256 0.256 0.621 0.616 0.698 0.707 0.317 0.322
A. The high resolution image set (HRIS) O3 0.285 0.284 0.669 0.649 0.566 0.540 0.326 0.332
The HRIS consists of four images. They are 60 degrees TABLE III
field macula centred photographs captured by a Cannon S TANDARD DEVIATIONS OF THE MEASUREMENT METHODS .
60 U V film camera. The 35 mm slides produced were
then digitalised by scanning with an Epson Stylus photo
RX 700 scanner to develop digital images with a resolution
of (3584 × 2438 × 3) pixels. They represent different sever 50 degrees dilated fundus images were captured per eye
grades of diabetic retinopathy. Table I illustrates the formal using the same standardised protocol as above with the Zeiss
grading of the images. The use of HRIS supports an evalu- fundus camera and JVC 3CCD. The resolution of the image
ation to sub-pixel accuracy. This set consists of four images is (1360 × 1024 × 3) pixels. Six of the images illustrate
where 90 segments containing 2368 profiles are manually different type of diabetic retinopathy, as summarized in
marked. The images were subsequently down-sampled by table IV.
a lateral factor of four for submission to the measurement This set consists of eight images where 79 segments
algorithms, so that the vessels widths are known to ±0.25 containing 2249 profiles are manually marked; see figure 5.
of a pixel, discounting human error see figure 4. This set of images is very noisy and suffers from pathologies,
and this manifests in the greater variance of the observers.
B. The vascular disease image set (VDIS)
C. The central light reflex image set (CLRIS)
The VDIS consists of eight digital captured images. The The CLRIS consists of two digital images, containing
images were selected randomly from the database of images 21 marked segments with 285 profiles. They are captured
of patients attending the diabetic retinopathy outpatient clinic using a Zeiss FF 450 fundus camera connected to a digital
at Sunderland Eye Infirmary. Two-field (Nasal and Temporal) 3-CCD JVC camera. The capture angle is 50 degrees; the
photographs are macula centred. The resolution is (2160 ×
1440 × 3) pixels. They were selected to represent early

Image# Formal Grading

1 Arteriosclerotic Arterio-Vascular Changes
2 Level 5 = Proliferative Retinopathy
3 Level 1 = Minimal Non-Proliferative Retinopathy
4 Level 1 = Minimal Non-Proliferative Retinopathy
5 Level 2 = Moderate Non-Proliferative Retinopathy
6 Normal
7 Normal
8 Level 3 = Severe Non-Proliferative Retinopathy
TABLE IV
T HE DISEASE OF IMAGES IN THE VASCULAR DISEASE IMAGE SET.
Fig. 4. Marked segments in the high resolution image set.

2264
Authorized licensed use limited to: University of Malaya. Downloaded on March 07,2024 at 03:25:09 UTC from IEEE Xplore. Restrictions apply.

Fig. 5. arked segments in the vascular disease image set.
Fig. 6. Marked segments in the central light reflex image set.
atherosclerotic changes in retinal vessels characterised by the
presence of arterial silver-wiring (exaggerated vascular light
reflex) secondary to stiffness and changes in the vascular
wall.
D. The kick point image set (KPIS)
The KPIS consists of two images, three segments with 164
gold standard widths, with a 60 degree field of view, and
photographed using a Canon 60uv fundus camera. All 164
vessel widths were selected from large diameter non-tortuous
vessel segments between bifurcations. The dimensions of the
high-resolution images were 3300 × 2600 pixels, approxi-
mately four times larger than the standard 760 × 570 lower
resolution fundus counterparts used for evaluation. Profiles
were identified by choosing profiles directions orthogonal to
the centre line direction recovered using the previous and
next pixels from the centre line determined by the “tram-
line”’ algorithm [13]; The three observers then identified
the kick-points. By down-sampling the images to a lower
resolution, these gold standard widths have higher accuracy
than expressed in the measured images so the measurements
can be assessed with sub-pixel accuracy. See figure 7. The
KPIS subset provides an alternative benchmarking method
to the edge-marked subsets.
2265
Authorized licensed use limited to: University of Malaya. Downloaded on March 07,2024 at 03:25:09 UTC from IEEE Xplore. Restrictions apply.
Fig. 7. Marked segments in the kick point image set.
V. CONCLUSIONS
We have briefly described a new standard dataset for
the evaluation of retinal vessel diameter measurement al-
gorithms. This dataset is intended to facilitate the develop-
ment and comparison of vessel measurement algorithms. It
includes a variety of normal and pathological images. The
dataset is marked up by three observers, using a specially-
developed tool to nominate points on the edges. These are
extrapolated, and an algorithm used to extract regularly-
spaced profiles, against which algorithms may be tested.
R EFERENCES
[1] X. Gao, A. Bharath, A.Stanton, A.Hughes, N. Chapman, and S.Thom,
“Quantification and characterisation of arteries in retinal images,”
Computer Methods Programs in Biomedicine, vol. 63, pp. 133–146,
2000.
[2] DRIVE, “Digital retinal images for vessel extraction,” 2007.
[3] STARE, “Structured analysis of the retina,” 2007.
[4] N. Chapman, N. Witt, X. Gao, A. A. Bharath, A. V. Stanton, S. A.
Thom, and A. D. Hughes, “Computer algorithms for the automated
measurement of retinal arteriolar diameters,” British Journal of Oph-
thalmology, vol. 85, pp. 74–79, 2001.
[5] S. M. B. Rassam, V. Patel, O. Brinchmann-Hansen, O. Engvold,
and E. M. Kohner, “Accurate vessel width measurement from fundus
photographs: A new concept,” British Journal of Ophthalmology,
vol. 78, pp. 24–29, 1994.
[6] P. Gregson, Z. Shen, R. Scott, and V. Kozousek, “Automated grading
of venous beading,” Computers and Biomedical Research, vol. 28,
no. 4, pp. 291–304, 1995.
[7] O. Brinchmann-Hansen and H. Heier, “Theoretical relations between
light streak characteristics and optical properties of retinal vessels,”
Acta Ophthalmologica, supplement, vol. 179, pp. 33–37, 1986.
[8] O. Chutatape, L. Zheng, and S. M. Krishman, “Retinal blood vessel
detection and tracking by matched gaussian and kalman filters,”
Proceedings of the 20th Annual International Conference of the IEEE
of Engineering in Medicine and Biology Society, vol. 6, pp. 3144–
3149, 1998.
[9] L. Zhou, M. S. Rzeszotarski, L. J. Singerman, and J. M. Chokreff,
“The detection and quantification of retinopathy using digital an-
giograms,” IEEE Transactions on Medical Imaging, vol. 13, no. 4,
pp. 619–626, December 1994.
[10] J. A. Lowell, A. Hunter, D. Steel, A. Basu, R. Ryder, and R. L.
Kennedy, “Measurement of retinal vessel widths from fundus images
based on 2-d modeling,” IEEE transactions on medical imaging,
vol. 23, no. 10, pp. 1196–1204, October 2004.
[11] J. S. Suri and S. Laxminarayan, Angiography And Plaque Imaging:
Advanced Segmentation Techniques, 2nd ed. CRC Press, 2003.
[12] S.Aldington, E.Kohner, S.Meuer, R.Klein, A.Sjolie, and T. E. I. C. S.
Group, “Methodology for retinal photography and assessment of
diabetic retinopathy: the eurodiab iddm complications study,” Dia-
betologia, vol. 38, no. 4, pp. 437–444, April 1995.
[13] A. Hunter, J. A. Lowell, D. Steel, A. Basu, and R. Ryder, “Non-linear
filtering for vascular segmentation and detection of venous beading,”
Technical Report, 2003.