Pixel-Based Iris and Pupil Segmentation in Cataract Surgery Videos
Using Mask R-CNN
Natalia Sokolova1 , Mario Taschwer1 , Stephanie Sarny2 3 , Doris Putzgruber-Adamitsch2 and Klaus Schoeffmann1
1
Klagenfurt University, Austria
2
Klinikum Klagenfurt, Austria
3
Medical University Graz, Austria
ABSTRACT
Automatically detecting clinically relevant events in
surgery video recordings is becoming increasingly important
for documentary, educational, and scientific purposes in the
medical domain. From a medical image analysis perspective,
such events need to be treated individually and associated
with specific visible objects or regions. In the field of
cataract surgery (lens replacement in the human eye), pupil
reaction (dilation or restriction) during surgery may lead
to complications and hence represents a clinically relevant
event. Its detection requires automatic segmentation and
measurement of pupil and iris in recorded video frames. In
this work, we contribute to research on pupil and iris seg-
mentation methods by (1) providing a dataset of 82 annotated
images for training and evaluating suitable machine learning
algorithms, and (2) applying the Mask R-CNN algorithm to
this problem, which—in contrast to existing techniques for
pupil segmentation—predicts free-form pixel-accurate seg-
mentation masks for iris and pupil. The proposed approach
achieves consistent high segmentation accuracies on several
metrics while delivering an acceptable prediction efficiency,
establishing a promising basis for further segmentation and
event detection approaches on eye surgery videos.
object segmentation, cataract surgery videos, mask R-
CNN, deep learning
1. INTRODUCTION
Cataract surgery is an eye natural lens replacement with
an artificial lens, and it is one of the most common surgi-
cal operations performed worldwide. Because of the rather
small size of operated organ, surgeons have to work with
tiny instruments under a microscope. Such microscopes are
equipped with a camera, and the video output signal can be
recorded and stored in an archive.
Cataract surgery videos are useful for educational and
documentary purposes in hospitals, but they may also be
analyzed automatically to detect the occurrence of adverse
events or complications during postoperative analysis. One
such event that may lead to complications is pupil reactions,
i.e. dilation or constriction of the pupil during the surgical
*This work was funded by the FWF Austrian Science Fund under grant
P 31486-N31.
978-1-7281-7401-3/20/$31.00 ©2020
Authorized licensed use limited to: IEEE
Universitas Airlangga. Downloaded on January 08,2024 at 10:38:01 UTC from IEEE Xplore. Restrictions apply.
procedure. Automatic detection and measurement of pupil
and iris in cataract surgery videos is, therefore, a necessary
preprocessing step of such automated analysis.
Most of the existing techniques that have already ad-
dressed this problem in medical domain search for the outer
circle of iris and pupil in the frame. However, eyes are
moving during surgery, and instruments cover parts of the
eye during essential moments. Also, it is not always possible
to find such circles in the frames, for example, if the visible
part of pupil or iris does not look round because of the used
instruments. In addition, the edge detection algorithms which
are mostly used to find those circles do not always work well
because of eye movements and focus conditions. Pixel-wised
segmentation detects the iris and pupil more precisely and
differentiates their areas from, for example, used surgical
instruments.
In this work, we investigate pixel-accurate pupil and iris
segmentation by a region-based convolutional neural network
(Mask R-CNN), which – to the best of our knowledge –
has not been applied to this problem before. We evaluate
the performance of Mask R-CNN with different backbone
networks for a manually annotated image dataset, collected
from different cataract surgeries. The backbone networks
were initialized with weights from COCO [1].
The contribution of this paper is two-fold: (1) we provide
a dataset1 of 82 annotated images for pixel-accurate pupil
and iris segmentation that can be used to train and evaluate
machine learning algorithms; and (2) we train and evaluate
the Mask R-CNN algorithm on this dataset and achieve
high accuracy on several metrics, establishing a promising
approach for further segmentation tasks on eye surgery
videos.
2. RELATED WORK
The problems of recognition, localization and tracking of
eyes in the images and videos have already been studied in
the research community. This topic was addressed in terms of
different use case scenarios. For example, iris segmentation
was used in the biometric identification approaches [2], [3],
[4] to localize the iris for further biometric analysis.
1 http://ftp.itec.aau.at/datasets/ovid/iris_pupil_seg/
 Iris segmentation was also applied to some medical imag-
ing tasks. For example, McConnon et al. [5] evaluated the
effect of different diseases on iris segmentation. Also, this
approach was applied to diagnose such diseases as pterygium
[6], strabismus [7] and cataract [8] in the images and even
to compare the eyes before and after cataract surgeries [9].
Iris and pupil localization was also applied to improve the
video analysis in the medical domain for different purposes
over the last years. In particular, it was used to assess sur-
geons by introducing the eye gase tracking control interface
for the surgical system in [10] using iris segmentation.
Some authors have already used segmentation methods of
pupil and iris in cataract surgery videos. For Lalys et al. [11]
the goal was to localize exactly where the iris and pupil are to
use it as a preprocessing step later. The images from cataract
surgery videos were first filtered and then analyzed using
color histogram. The authors defined at first the personal
color characteristics for every patient and then the thresholds
in obtained histograms to distinguish the boundaries between
pupil, iris and sclera. As a result, authors obtained for every
image a binary mask to segment an iris from sclera. This
method, as authors also mentioned, was not able to overcome
completely the problems of light conditions and used medical
instruments, and it requires an individual setup for every
analyzed cataract surgery video.
Even more progress was done in [12] and [13] research
works. There the main idea was to find the pupil area to use it
as a region of interest for further instruments localization and
classification. The images were first transformed to the YUV
color space, and then, with the assumption that the color of
pupil is different from everything else in the frame, the binary
mask was obtained. From this mask the most probable pupil
circle was retrieved using Hough transform [14]. To decrease
fault rate the pixel and circle centers counting were used.
One more pupil localization approach was presented in
[15]. This localization was used later as a preprocessing step
of a operation workflow retrieval in [16] to normalize the
cataract surgery videos. For this purpose, using Canny edge
detection and Hough transform algorithm the center of the
pupil was found in the frames to implement eye tracking to
further determine a region of interest.
Also, some analyses have been done in this area already.
For example, authors of [17] aimed to solve a problem of iris
registration under both rigid and nonrigid deformations for
use in refractive cataract surgery. They split a task of iris and
pupil segmentation into two parts. At the beginning they used
a random sample consensus algorithm to find most probable
circles of pupil to combine them afterwards into elliptical fit.
To find a boundary between iris and sclera, circular spines
algorithm was developed in this work. It searched for the
circle outside of the pupil which has the highest gradient
value between inner and outer areas.
However, none of these approaches used pixel-based seg-
mentation, which would allow us to localize pupil and
iris and accurately segment them from surgical instruments
and other tissues in a sufficiently precise way for further
automated analysis.
Authorized licensed use limited to: Universitas Airlangga. Downloaded on January 08,2024 at 10:38:01 UTC from IEEE Xplore. Restrictions apply.
3. MATERIAL AND METHODS
3.1. Collected dataset
For this work we used 35 cataract surgery videos, collected
at Klinikum Klagenfurt with the following parameters:
• a resolution of 540x720 pixels;
• a frame rate of 25 fps;
• recorded during years 2017–2018.
All of these videos are varying in terms of the zoom level,
eye color, brightness and blur levels. We extracted 82 frames
from these videos. 51 of them contain also surgical tools.
The frames were obtained from different phases of cataract
surgery. Therefore, the images contain different surgical tools
and show the eyes in different states; for example, with
natural or artificial lenses, or even without any lens.
All of the frames were manually annotated with free-form
regions of iris and pupil. Iris annotations do not include
the pupil area. Figures 1, 2 and 3 show examples of the
original frames (a), as well as corresponding annotations (b)
and predictions (c). All annotations were created and stored
in the MS COCO format using the COCO Annotator tool
[18]. The dataset was randomly split into training, validation
and test subsets with a ratio of 70:15:15.
3.2. Experimental setup
As segmentation and classification algorithm, we use Mask
R-CNN [19], which is known to work well for general
object segmentation. This region-based convolutional neural
network is an extension of Faster R-CNN [20] with an
additional branch to predict object masks. For every detected
object, Mask R-CNN outputs a binary segmentation mask, a
boundary box, a class category and confidence value.
In our evaluations we use two different backbone networks
for Mask R-CNN: ResNet-50 and ResNet-101 [21]. Both
models were initialized with COCO weights what allows us
to train our network on the rather small dataset. As shown in
Table I, training with random weights results in much lower
performance. We experiment with different hyperparameters
and achieve best validation results with the following values:
• learning rate of 0.002;
• momentum of 0.9;
• batch size of 1;
• 100 epochs;
• 5 validation steps.
For model selection, we use the training epoch with the
lowest validation loss. For ResNet-50 and ResNet-101 this
was epoch 52 and 73, respectively. We used the Mask R-
CNN implementation provided by [22] with Tensorflow as a
backend and an NVIDIA TITAN RTX N GPU with 24 GB
of RAM. This implementation does not work with occluded
object annotations and with annotated regions enclosing
holes (iris without pupil), so we had to slightly adapt it.
4. RESULTS AND DISCUSSION
Segmentation performance is evaluated using Intersection
over Union (IoU), which is the ratio of the intersection of the
ground truth segment with the predicted segment divided by
 (a) (b) (c)
Fig. 1: Initial (a), annotated (b) and predicted (c) frames without any surgical tool.

(a) (b) (c)

Fig. 2: Initial (a), annotated (b) and predicted (c) frames with secondary incision knife.

(a) (b) (c)

Fig. 3: Initial (a), annotated (b) and predicted (c) frames with the lowest iris IoU value.

the union of both of them. Using different IoU thresholds to
determine true positives, we compute mean average precision
(mAP) as in the original Mask R-CNN paper [19]. The value
mAP50:95 represents the average of mAP values with IoU
thresholds ranging between 50% and 95% in steps of 5%.
During training we discovered that the results for the net-
work with all trained layers are slightly better than when only
head layers were trained. We also found that the bounding
box computed from the predicted segmentation mask is more
precise than the boundary box output of the Mask R-CNN
algorithm. We therefore use these computed bounding boxes
for evaluation of boundary box segmentation.
Both Mask R-CNN models (with ResNet-50 and ResNet-
101 backbone networks, respectively) lead to similar evalua-
tion results, both for free-form mask segmentation (Table I)
and boundary box segmentation (Table II). For mask segmen-
tation, all of the iris test examples are recognized (mAP = 1)
with an IoU threshold of 75% and 80% by ResNet-101
and ResNet-50 models, respectively. Both models achieve
identical results for pupil mask segmentation, where all test
examples are recognized with at least 90% IoU. For boundary
box segmentation, both models correctly recognized all iris
and pupil test examples with at least 85% IoU (mAP85 = 1
in Table II).
To interpret these results further, we look at the test
example recognized with the lowest IoU value. According
to minimal IoU thresholds X with mAPX < 1 in Tables I
and II, this occurred for mask segmentation of an iris test
example recognized by the ResNet-101 model (mAP75 = 1
and mAP80 < 1). The actual IoU value for the predicted
iris mask is 78.4%, and the corresponding test example is
shown in Figure 3 (iris corresponds to red area). Apparently,
the predicted iris mask does not look too bad and may still
be usable for further processing or analysis. The reason for
this is that the area of this iris is rather small and has quite
long external and internal borderlines compared to the other
iris examples. In this case even little shortcomings in the
predicted region contribute significantly to the IoU value. At
the same time the pupil IoU value for the mask segmentation
in the same frame is 91.47% (cyan area in Figure 3c).

Authorized licensed use limited to: Universitas Airlangga. Downloaded on January 08,2024 at 10:38:01 UTC from IEEE Xplore. Restrictions apply.
 TABLE I: Free-form mask evaluation results. [3] R. Raghavendra, K. B. Raja, V. K. Vemuri, S. Kumari, P. Gacon,
Class (CNN) mAP75 mAP80 mAP85 mAP90 mAP95 mAP50:95 E. Krichen, and C. Busch, “Influence of cataract surgery on iris
Iris (ResNet-50) 1.0000 1.0000 0.7273 0.0909 0.0000 0.7818 recognition: A preliminary study,” in 2016 International Conference
Iris (ResNet-101) 1.0000 0.9091 0.9091 0.1818 0.0000 0.8000 on Biometrics (ICB), June 2016, pp. 1–8.
Pupil (ResNet-50) 1.0000 1.0000 1.0000 1.0000 0.0909 0.9091 [4] R. Keshari, S. Ghosh, A. Agarwal, R. Singh, and M. Vatsa, “Mobile
Pupil (ResNet-101) 1.0000 1.0000 1.0000 1.0000 0.0909 0.9091
Random weights periocular matching with pre-post cataract surgery,” in 2016 IEEE
Iris (ResNet-101) 0.5455 0.0909 0.0000 0.0000 0.0000 0.4091 International Conference on Image Processing (ICIP), Sep. 2016, pp.
Pupil (ResNet-101) 0.7273 0.7273 0.6364 0.2727 0.0000 0.6455 3116–3120.
[5] G. McConnon, F. Deravi, S. Hoque, K. Sirlantzis, and G. Howells,
“Impact of common ophthalmic disorders on iris segmentation,” in
TABLE II: Boundary box evaluation results. 2012 5th IAPR International Conference on Biometrics (ICB), March
Class (CNN) mAP75 mAP80 mAP85 mAP90 mAP95 mAP50:95 2012, pp. 277–282.
Iris (ResNet-50) 1.0000 1.0000 1.0000 0.9091 0.3636 0.9273 [6] R. G. Mesquita and E. M. N. Figueiredo, “An algorithm for measur-
Iris (ResNet-101) 1.0000 1.0000 1.0000 0.9091 0.1818 0.9191 ing pterygium’s progress in already diagnosed eyes,” in 2012 IEEE
Pupil (ResNet-50) 1.0000 1.0000 1.0000 0.7273 0.0909 0.8818
Pupil (ResNet-101) 1.0000 1.0000 1.0000 0.8182 0.0909 0.8909
International Conference on Acoustics, Speech and Signal Processing
(ICASSP), March 2012, pp. 733–736.
[7] N. Khumdat, P. Phukpattaranont, and S. Tengtrisorn, “Development
of a computer system for strabismus screening,” in The 6th 2013
Prediction efficiency is also quite similar for both evalu- Biomedical Engineering International Conference, Oct 2013, pp. 1–5.
[8] S. Patange and A. Jagadale, “Framework for detection of cataract and
ated Mask R-CNN models. The ResNet-101 model performs gradation according to its severity,” in 2015 International Conference
predictions at 6.07 frames per second (fps) in our test on Pervasive Computing (ICPC), Jan 2015, pp. 1–3.
environment, while the ResNet-50 model runs at 6.37 fps. [9] A. Lakra, P. Tripathi, R. Keshari, M. Vatsa, and R. Singh, “Seg-
densenet: Iris segmentation for pre-and-post cataract surgery,” in 2018
In summary, both evaluated Mask R-CNN models achieve 24th International Conference on Pattern Recognition (ICPR), Aug
promising and effective results for iris and pupil segmenta- 2018, pp. 3150–3155.
tion. According to mAP50:95 values, the ResNet-101 model [10] Z. Li, I. Tong, L. Metcalf, C. Hennessey, and S. E. Salcudean, “Free
head movement eye gaze contingent ultrasound interfaces for the da
tends to be slightly more effective than the ResNet-50 model, vinci surgical system,” IEEE Robotics and Automation Letters, vol. 3,
except for boundary box segmentation of the iris, where the no. 3, pp. 2137–2143, July 2018.
ResNet-101 model seems to overfit slightly. [11] A. Ektesabi and A. Kapoor, “Exact pupil and iris boundary detection,”
in The 2nd International Conference on Control, Instrumentation and
Automation, Dec 2011, pp. 1217–1221.
5. CONCLUSION AND FURTHER WORK [12] F. Lalys, L. Riffaud, D. Bouget, and P. Jannin, “A framework for the
recognition of high-level surgical tasks from video images for cataract
This work provides a diverse dataset of 82 images selected surgeries,” IEEE Transactions on Biomedical Engineering, vol. 59,
from 35 cataract surgery videos, where the regions of iris no. 4, pp. 966–976, April 2012.
and pupil have been manually segmented and annotated. [13] D. Bouget, F. Lalys, and P. Jannin, “Surgical Tools Recognition
and Pupil Segmentation for Cataract Surgical Process Modeling,”
We trained and evaluated the Mask R-CNN segmentation in Medicine Meets Virtual Reality - NextMed, vol. 173. Newport
algorithm on this dataset, which predicts free-form pixel- beach, CA, United States: IOS press books, Feb. 2012, pp. 78–84.
accurate segmentation masks for iris and pupil, in contrast [Online]. Available: https://www.hal.inserm.fr/inserm-00669660
[14] M. J. Swain and D. H. Ballard, “Color indexing,” International
to existing techniques that constrain predicted segments to Journal of Computer Vision, vol. 7, no. 1, pp. 11–32, 1991. [Online].
circular or elliptic regions. Our evaluations employ ResNet- Available: https://doi.org/10.1007/BF00130487
50 and ResNet-101 as backbone networks of Mask R-CNN [15] G. Quellec, K. Charrière, M. Lamard, B. Cochener, and G. Cazuguel,
“Normalizing videos of anterior eye segment surgeries,” in 2014 36th
and demonstrate consistent high segmentation accuracies on Annual International Conference of the IEEE Engineering in Medicine
several metrics while delivering an acceptable prediction and Biology Society, Aug 2014, pp. 122–125.
efficiency of 6 frames per second on current GPU hardware. [16] K. Charrière, G. Quellec, M. Lamard, G. Coatrieux, B. Cochener,
and G. Cazuguel, “Automated surgical step recognition in normalized
Future work may include extending the iris and pupil cataract surgery videos,” in 2014 36th Annual International Confer-
dataset to allow for more robust segmentation models and ence of the IEEE Engineering in Medicine and Biology Society, Aug
evaluation results. More importantly, the promising results 2014, pp. 4647–4650.
[17] D. Morley and H. Foroosh, “Computing cyclotorsion in refractive
reported in this study may lead to the development of cataract surgery,” IEEE Transactions on Biomedical Engineering,
accurate segmentation and measurement approaches for other vol. 63, no. 10, pp. 2155–2168, Oct 2016.
regions of interest in eye surgery videos like operation [18] J. Brooks, “COCO Annotator,” https://github.com/jsbroks/coco-
annotator/, 2019.
instruments or natural and artificial lenses. Tracking such [19] K. He, G. Gkioxari, P. Dollár, and R. B. Girshick, “Mask
regions over time may eventually allow us to detect events R-CNN,” CoRR, vol. abs/1703.06870, 2017. [Online]. Available:
that may be relevant for clinical studies, such as intra- http://arxiv.org/abs/1703.06870
[20] S. Ren, K. He, R. B. Girshick, and J. Sun, “Faster R-
operative pupil reactions in cataract surgeries. CNN: towards real-time object detection with region proposal
networks,” CoRR, vol. abs/1506.01497, 2015. [Online]. Available:
6. REFERENCES http://arxiv.org/abs/1506.01497
[21] K. He, X. Zhang, S. Ren, and J. Sun, “Deep residual learning for
[1] T. Lin, M. Maire, S. J. Belongie, L. D. Bourdev, R. B. image recognition,” in Proc. of the IEEE Conference on Computer
Girshick, J. Hays, P. Perona, D. Ramanan, P. Dollár, Vision and Pattern Recognition, 2016, pp. 770–778.
and C. L. Zitnick, “Microsoft COCO: common objects in [22] W. Abdulla, “Mask R-CNN for object detection
context,” CoRR, vol. abs/1405.0312, 2014. [Online]. Available: and instance segmentation on keras and tensorflow,”
http://arxiv.org/abs/1405.0312 https://github.com/matterport/Mask_RCNN, 2017.
[2] C. N. Devi, “Automatic segmentation and recognition of iris images:
With special reference to twins,” in 2017 Fourth International Confer-
ence on Signal Processing, Communication and Networking (ICSCN),
March 2017, pp. 1–5.

Authorized licensed use limited to: Universitas Airlangga. Downloaded on January 08,2024 at 10:38:01 UTC from IEEE Xplore. Restrictions apply.