A Benchmark for Visual-Inertial Odometry Systems

Employing Onboard Illumination

Mike Kasper1 Steve McGuire1 Christoffer Heckman1

Abstract— We present a dataset for evaluating the perfor-

mance of visual-inertial odometry (VIO) systems employing an
onboard light source. The dataset consists of 39 sequences,
recorded in mines, tunnels, and other dark environments,
totaling more than 160 minutes of stereo camera video and
IMU data. In each sequence, the scene is illuminated by an
onboard light of approximately 1300, 4500, or 9000 lumens.
We accommodate both direct and indirect visual odometry
methods by providing the geometric and photometric camera
calibrations (i.e. response, attenuation, and exposure times).
In contrast with existing datasets, we also calibrate the light
source itself and publish data for inferring more complex
light models. Ground-truth position data are available for a
subset of sequences, as captured by a Leica total station.
All remaining sequences start and end at the same position,
permitting the use of total accumulated drift as a metric for
evaluation. Using our proposed benchmark, we analyze the
performance of several start-of-the-art VO and VIO frame-
works. The full dataset, including sensor data, calibration Fig. 1: Example frames from each environment (i.e. tunnels,
sequences, and evaluation scripts, is publicly available online
at http://arpg.colorado.edu/research/oivio.
mines, woods, and office). Each row shows a sequence of
four frames, separated by a few seconds. These images high-
I. I NTRODUCTION light the primary challenges posed by our dataset: dynamic
illumination, motion blur, and poor camera exposure.
Given their versatility and relatively low cost, passive
cameras are arguably the most common sensor employed in
robotics applications. However, to be used reliably, sufficient
scene illumination is required. While these conditions are are robust to such illumination changes, but are far more
met in many scenarios, there is growing interest for robots susceptible to the motion blur and sensor noise we can expect
to work in darker environments, such as underground or when working in dark environments, due to inadequate
underwater. This is most evident in the recently proposed camera exposure [23], [13], [4].
DARPA Subterranean Challenge [3], but also highlighted by To assess the performance of existing methods and to aid
the emergence of workshops focused on the subject [20], the development of novel solutions to the aforementioned
[25] and companies fielding robots in this domain. challenges, we present a benchmarking dataset for visual-
In the absence of visual information, the robotics com- inertial odometry (VIO) systems working in environments
munity has primarily relied on depth sensors (e.g. LIDAR illuminated by a single, onboard light source. In total, the
and active depth cameras). While these sensors are robust to dataset contains 39 sequences, with over two hours of stereo
low-texture surfaces and poor illumination, they are often of camera video and IMU data. The sequences were recorded
limited range, lower resolution, and higher cost. More im- in a number of challenging environments, including tun-
portantly, their utility is reduced in geometrically ambiguous nels, mines, low-light indoor scenes, and nighttime outdoor
scenes, such as long hallways or tunnels. Traditionally, visual scenes. Several example frames can be seen in Figure 1.
cues have compensated for these limitations [9]. For each recorded sequence, we illuminate the scene with
In order to employ cameras in dark environments, robots a white LED light of approximately 1300, 4500, or 9000
can be equipped with an onboard light source. However, lumens. This allows us to assess how much performance
this would violate the brightness constancy assumption held depends on lighting strength. In contrast with other datasets,
by most visual perception systems, as scene illumination our benchmark not only provides the geometric and photo-
will change as a result of the robot’s movement. This is metric camera calibrations, but also a calibration of the light
particularly problematic for direct methods, which work on itself. By publishing a light model, we intend to promote
image intensities [8], [15], [4]. In contrast, indirect methods the development of novel VIO algorithms that relax the
1 Autonomous Robotics and Perception Group (APRG). Department of
brightness constancy assumption and model the dynamic
Computer Science. University of Colorado. Boulder, Colorado USA. Cor- illumination of the scene. As a point of comparison, however,
responding author: christoffer.heckman@colorado.edu we analyze several existing frameworks in Section VII.
Fig. 2: Frames captured while executing approximately the same turn with each lighting configuration. From left to right,
the depicted scene is illuminated with 1300, 4500, and 9000 lumens. Note how the camera’s auto-exposure compensates for
the different levels of illumination, but consequently produces varying amounts of motion blur. To aid visual inspection, we
have provided enlarged images for the regions outlined in red and green.
II. R ELATED W ORK
We draw guidance from the popular EuRoC dataset [2] in
terms of sensors and ground-truthing employed. It contains
11 sequences captured via a hardware-synchronized stereo
camera and IMU mounted on top of a micro aerial vehi-
cle. Ground-truth 6-DoF poses and 3-DoF positions were
captured via a Vicon Motion Capture system and Leica
MS50 laser tracker, respectively. The dataset does exhibit
some challenging lighting scenarios, where large regions of
the environment are poorly illuminated. However, lighting
remained constant while capturing each sequence. A limita-
tion of the EuRoC dataset is that it is only well-suited for
indirect methods; it not only lacks the camera response and
attenuation models required by direct methods, but also the
exposure times between stereo cameras are not synchronized.
In contrast, the TUM monoVO dataset [5] targets direct
odometry methods, providing full photometric calibration
and exposure times as reported by the camera sensor. How-
ever, this is a purely monocular dataset, lacking the second
camera and IMU sensor found in [2]. The curators of the
TUM monoVO dataset also opt for a different ground-
truthing strategy. All sequences start and end in the same
position, permitting the evaluation of VO frameworks in
terms of total accumulated drift over the entire sequences.
The published sequences contain a number of challenging
scenes, but again exhibit relatively static lighting.
More inline with our focus on onboard illumination is the
Oxford RobotCar dataset [12]. It contains over one year of
driving data recorded by a car outfitted with six cameras,
a LIDAR, and an IMU. It facilitates the development and
evaluation of a number of perception problems related to
autonomous vehicles. The sequences exhibit a variety of
weather conditions, captured at day and night. While night-
time sequences are illuminated by the car’s headlights, a sig-
nificant portion of illumination is contributed by streetlights.
Additionally, no model of the car’s headlights are provided.
Taking these concepts further, the ETH-ICL dataset [18]
focuses directly on the problem of visual SLAM in dy-
namically lit environments. The dataset contains both real
and synthetic sequences largely based on the TUM RGB-
D benchmark [24] and the ICL-NUIM dataset [7]. Each
sequence exhibits some form of dynamic lighting, either
by modulating global and local light sources, or by the
movement of a flashlight co-located with the camera. While
this dataset does contain sequences illuminated by an on-
board light source, only two sequences employ such a
lighting solution. Additionally, no model of the light source
is provided, which we believe could be leveraged to develop
novel methods for visual odometry.
In a different vein, the DiLiGenT dataset [22] is not in-
tended for VO research, but rather that of photometric stereo.
Photometric stereo, in contrast with binocular stereo, is a
technique that typically employs a single camera and one or
more lights to infer 3D geometry [26]. The DiLiGenT dataset
contains a series of images, taken of 10 objects, captured
by a stationary camera and different lighting configurations.
In addition to the images themselves the authors provide a
calibration of the employed light array, which consists of a
2D grid of 96 uniformly-spaced, white LED lights. We wish
to take a similar approach in our visual odometry dataset.
As can be observed by this brief review, just as there is
large diversity of VO solutions, the same can be said for VO
datasets. The dataset we present in the following sections
is particularly focused on underground environments with
sensing and calibration considerations to match, including
onboard lighting and the usage of stereo cameras and IMUs.
III. DATASET
All sequences in our dataset can be characterized as a
visual-inertial rig navigating dark environments, illuminated
by an onboard light source. We captured sequences in four
types of environments: (1) mines, (2) tunnels, (3) outdoors
at night, and (4) indoors where all other lights are turned
off. While some sequences may exhibit small amounts ex-
ternal lighting, the predominate illuminant in all scenes is
the onboard light source. To permit exploration of lighting
solutions, we roughly replicate each trajectory with three
different lighting intensities. A few examples frames are
shown in Figure 2. During data capture, the sensor rig was
either handheld or mounted on a ground-vehicle (Clearpath
Husky UGV). In the remainder of this section we provide
details about the sensor rig and ground-truthing strategies.
 Leica Prism Computer
Light
Cameras Batteries
IMU
Fig. 3: Our employed sensors include an Intel RealSense
D435i and a LORD Microstrain G3M-GX5-15 (not visible).
The onboard light source is a 9000 lumen, 100W, white LED
light. To modulate the light intensity we use a DC-DC boost
regulator. Long-term use of this light requires a large passive
heat-sink and fan. We equip a tracking prism when ground-
truthing position data with the Leica.
A. Sensor Setup
We capture inertial data with LORD Microstrain 3DM-
GX5-15 at 100Hz, and a stereo pair of 1280 × 720, grayscale
images with a Intel RealSense D435i at 30Hz. We opted for
the RealSense as it is a widely available consumer device
featuring a hardware-synchronized, fixed-lens, global-shutter
stereo camera. However, as it is primarily intended to be used
as an active IR depth sensor, these cameras do not filter out
infrared light. This does not negatively impact the acquired
visual information, but does require that we disable the IR
emitter during operation. Consequently, we do not publish
any depth maps with our dataset.
Each sequence in our benchmark is illuminated by an
onboard, maximum 9000 lumens, 100W, white LED light.
Long-term use of this light requires a large passive heat
sink and fan. Clearly, such a lighting system is not practical
for all robotics applications (e.g. micro air vehicles). We
therefore attempt to capture the same trajectory three times,
modulating the light’s intensity to approximately 100, 50,
and 15 percent of its full capacity. This allows us to evaluate
the performance of visual odometry systems working with
different lighting solutions.
The light and sensors are mounted inside custom housing,
equipped with a power supply and onboard computer for
logging data. As our sensor rig is self-contained, all captured
sequences exhibit consistent extrinsic calibrations, regardless
of the mobile platform employed (i.e. handheld or ground ve-
hicle). Depending on the ground-truthing system employed,
the rig may also be outfitted with a laser tracking prism. An
image of our sensor rig can be seen in Figure 3.
B. Ground-truth
In this work, we employ two methods for ground-truthing,
providing a balance between sampling resolution and trajec-
tory complexity. For a subset of our dataset, we employ a
Leica TCRP1203 R300 total station to acquire the ground-
truth position of our sensor rig at 10Hz. Unfortunately, main-
taining the line-of-sight required by any tracking solution
puts undesirable constraints on the nature of environments
and trajectories we can employ. This is especially true in
the narrow passage ways common in tunnels and mines.
We therefore provide an addition set of sequences, that start
and end in the same position, permitting the use of total
accumulated drift as another evaluation metric. As in [5], we
start and end each sequence with a 10-20 second sequence
of loopy camera motion, observing an easy-to-track scene.
We then process just these frames using the ORB-SLAM2
framework [14] to obtain accurate poses for the start and
end segments. As our dataset contains stereo images, we
need not resolve the scale ambiguity problem that arises from
monocular VO, as required in [5].
IV. DATASET F ORMAT
A. Data Format
We organize our dataset using the ASL dataset format,
similar to the the EuRoC dataset [2]. However, we must
also accommodate the additional photometric calibrations
and light source model. As in [2], each captured sequence
comprises a collection of sensor data. Each sensor comes
with a sensor.yaml file, specifying calibration parame-
ters, and a data.csv file, that either contains the sensor
data itself or references files in an optional data folder:
<sensor system id>
<sensor id>
sensor.yaml
data.csv
data
Again, ground-truthing systems are treated as separate
sensors. In a slight abuse of terminology, we treat the light
source as a sensor without a data.csv file. For example:
husky0
imu0
sensor.yaml
data.csv
cam0
sensor.yaml
data.csv
data
154723105215000000.png
154723105220000000.png
...
leica0
sensor.yaml
data.csv
light0
sensor.yaml
1) YAML Files: As with the EuRoC MAV dataset,
each sensor provides a sensor.yaml file, which
describes all relevant properties unique to the sensor.
Additionally, all YAML files share two common
fields: sensor type and T BS. The sensor type
field specifies one of the following sensor types:
{imu|camera|position|pose|light} and T BS is
a 4 × 4 homogeneous transformation matrix, describing the
sensor’s extrinsic relationship with the sensor rig’s frame.
All properties listed in the YAML file are assumed to be
static throughout the entire sequence.
2) CSV Data Files: The data.csv file either contains
all the data captured by the sensor throughout the sequence,
or references files in the optional data folder. In either case,
each line first begins with a timestamp (integer nanosec-
onds POSIX) denoting the time at which the corresponding
data was recorded. The subsequent fields for each sensor’s
data.csv file are presented in the following sections.
B. Sensors
1) Cameras: As with the EuRoC dataset [2], each line
of a camera’s data.csv file contains the timestamp and
file name of the captured image. We augment this by
including the exposure times and gains, reported by the
camera during capture. The sensor.yaml file specifies
the cameras intrinsics with the fields: camera model,
intrinsic coefficients, distortion model,
distortion coefficients, and resolution.
However, as our dataset accommodates direct VO meth-
ods, we also provide a photometric calibration, via the
response.csv and vignette.png files in the sensor
folder. The response.csv file contains 255 values, pro-
viding all camera inverse-response values over the domain
of 8-bit color depth. These values can be used as a simple
lookup to convert from captured image intensities to irra-
diance values. The vignette.png file is a monochrome
16-bit image with the same resolution as the camera. Each
pixel in the vignette.png specifies the attenuation factor
for its respective coordinates in camera images. Section V-B
describes our photometric calibrations process.
2) IMU: As with the EuRoC dataset, each line of an
IMU’s data.csv file contains the following fields: times-
tamp, ω S [rad s−1 ], aS [m s−2 ]. Where ω S ∈ R3 is the
angular rate and aS ∈ R3 is the linear acceleration, both
expressed in the sensor’s body frame. For all sequences, the
Microstrain’s data is published under the imu0 sensor. The
YAML file contains the noise densities and bias “diffusions”,
which stochastically describe the sensor’s random walk.
These are obtain in the same way as in the EuRoC MAV
dataset. We refer the reader to [2] for more details.
We also publish measurements from the RealSense
D435i’s onboard IMU (Bosch Sensortec BMI055) as imu1.
However, the RealSense does not capture accelerometer and
gyroscope data at the same rate. Rather than complicating
IMU data access, we opt to keep the same single-line file
format. We configure the RealSense to capture accelerometer
data at 250Hz and gyroscope data at 400Hz, and only report
the gyroscope measurement closest to each accelerometer
measurement. However, we still provide files for the indi-
vidual sensor streams in imu1/data folder.
3) Light: As previously mentioned, the onboard light is
treated like a sensor without a data.csv file. However,
it does have a sensor.yaml file. Like all sensors, this
YAML file provides the transformation matrix T BS, which
specifies the pose of the light with respect to the sensor
system. It also contains two additional fields: size and
lumens. Here, size is 2D vector indicating the horizontal,
and vertical size, in meters, of the LED patch. The lumens
field specifies the approximate light intensity, in lumens,
which is one of three values: 9000, 4500 or 1300.
4) Position, Pose: As described in Section III-B, we
employ two ground-truthing methods. These are represented
by the sensors leica0 and loop0. As with the EuRoC
dataset, the data.csv file for these sensors contains the fol-
lowing fields: timestamp, qRS , R pRS . Here, R pRS denotes
the position of the sensor with respect to the ground-truthing
reference frame R. The field, qRS is the four-element, unit
quaternion representing the orientation of the sensor.
Given that the Leica laser tracker only captures 3D posi-
tion, the qRS field is omitted from its data.csv file. In
contrast, the “loop-closure” method, described in [5], can
provide full 6-DoF poses. However, it only does so for the
very beginning and ending of the trajectory. Consequently,
its corresponding data.csv file contains far fewer entries
with a large gap in the reported timestamps.
V. C ALIBRATION
In this section we described the provided calibration
parameters and how they are obtained. These calibrations
include the camera intrinsics, extrinsics between the cameras
and IMU, and the photometric camera calibration. However,
in contrast with existing VO datasets, we additionally provide
multiple models for the onboard light source. To permit the
use of custom calibration models, we also publish all raw
sensor sequences for each mode of calibration.
A. Geometric Camera Calibration
We calibrate our visual-inertial sensor using the Vicalib
framework1 . This process yields a geometric calibration com-
prising the camera intrinsics and the camera-IMU extrinsics,
although recent work shows this may be estimated online
[16]. To permit the use of other camera models, we publish
all calibrations sequences.
B. Photometric Camera Calibration
As with the TUM monoVO dataset [5], we provide the
photometric calibration for each camera. This includes the
camera’s response and dense attenuation model. We obtained
these properties using the same process described in [5]. This
yields independent linear response functions Ul and Ur for
the left and right cameras, respectively. However, care must
be taken to ensure that observations of the same point in each
image elicit the same response. We therefore apply an affine
transformation to the response Ur to satisfy the equality
Ul (Il (p)) = α Ur (Ir (q)) + β. (1)
Here, p and q are coordinates in the left and right images
corresponding to the same world point. We obtain these from
1 https://github.com/arpg/vicalib

Fig. 4: Example frames from our light source calibration
sequences. The top row shows images for geometric cali-
bration, where the green ‘plus’ indicates the hand-annotated
specular highlight, and the red ‘x’ indicates the highlight
computed from the inferred geometry. The bottom row shows
images for intensity distribution calibration. The motion of
the rig is such that the entirety of the side-lobing from the
onboard illuminant was captured.
the irradiance images generated when performing the cali-
bration described in [5] on both cameras simultaneously. As
the two irradiance images will be centered on the calibration
target, data association is trivial. We then solve the linear
system constructed from these observations to determine the
affine transformation that best satisfies Eq. (1).
C. Light Source Calibration
In order to facilitate the development of novel visual
odometry algorithms, which anticipate changes in scene illu-
mination, we additionally calibrate the onboard light source.
Our dataset includes computationally simple light models,
which are feasible for real-time applications. However, we
also publish multiple types of calibration sequences so that
other models may be inferred in the future.
To obtain an initial geometric calibration, we first model
the light source as a planar patch in 3D space. As in [28],
[21], we employ reflective spheres as light probes, and
exploit the their contours and specular highlights, observed in
multiple stereo images and scene configurations, to construct
an inverse-rendering problem. Given prior knowledge of our
light source, we fix the dimensions of the planar patch,
reducing the parameter space proposed in [28] to that of a
single 6-DoF pose. Figure 4 depicts this calibration process.
The position of an isotropic point light, the simplest model
we provide, is computed as the center of the inferred area
light. Using this point light model (and assuming Lambertian
reflectance), the irradiance I found at a given point p is then
αL cos θ
I= , (2)
d2
where α is the surface albedo, L is radiance emitted by
the light, θ is the angle of incidence, and d is the distance
between p and the light source. As α is typically unknown
in a visual odometry setting, we do not provide an absolute
measure of L and our isotropic light models can therefore
be used for all three lighting configurations. In addition, we
argue that light occlusions (i.e. cast shadows) need not be
modeled, as the light source is located near the cameras.
However, when the sensor system is operating within
close proximity to objects of the scene, shadows may be
of concern. Additionally, the errors from using a point light
model will become non-negligible. Consequently, the area
light model should be employed in these circumstances.
Theoretically, this would require integrating all incident
radiance emitted from the entire surface of the LED patch.
In practice, incident radiance arriving from our area light
can be approximated, up to some scale, as a summation of
a finite number of samples
X cos θa
I∝ , (3)
d2a
a∈A
Given that our light does not employ any reflectors and
that the field of view of our cameras are relatively narrow,
we believe an isotropic light model offers the best trade-off
between fidelity and computational cost. However, should
more complex light models be desired, we provide addi-
tional calibration sequences for inferring the light’s intensity
distribution. Guided by similar efforts [1], [17], we capture
hundreds of images of a known calibration target, exhibiting
largely diffuse reflectance, at different depths and angles.
Figure 4 shows two example images from these sequences.
Using the provided camera calibration and exposure times,
these images should first be photometrically corrected. In-
tensity variations observed in the resulting images can then
be attributed to extrinsics relationship between the target
and light source. Similar approaches have been used to
infer radially symmetric light distributions [1] and full 4D
light models [17]. We provide separate light distribution
calibration data for each lighting configuration.
D. Timestamp Alignment
In order to spatially and temporally align ground-truth
data with sensor readings, we employ an implementation of
SplineFusion [11]. Here we use a maximum likelihood (ML)
state estimator to obtain the set of control points c ∈ SE3,
describing the trajectory as a cubic B-spline. As the spline is
C 2 -continuous, we can therefore infer the rotational velocity
and linear acceleration at any point for direct comparison
with IMU readings. This process yields a temporal offset for
each sensor measurement and the biases of each IMU.
VI. E VALUATION M ETRICS
We propose several metrics for evaluating VIO systems
using our dataset. As described in Section III-B, we employ
two distinct methods of ground-truthing: (1) trajectory po-
sitions, captured with a Leica laser tracker and (2) relative
start and end poses, obtained from the final loop-closure.
We therefore propose separate evaluation metrics for each
method. To assess how sensitive a VIO algorithm is to
the lighting solution, this analysis can also be performed
separately for each of the employed light configurations.
 15% Light 50% Light 100% Light 15% Light
OKVIS (Stereo + IMU) ORB-SLAM (Stereo)
Fig. 5: Color-encoded alignment error ealign for each run of three different odometry systems. Results are broken into three
regions based on onboard light intensity and each row depicts results from running the same trajectory multiple times.
Fig. 6: Frames where stereo ORB-SLAM consistently loses
tracking in the TN 015 HH 03 and OD 015 HH 02 data
sequences. Both frames exhibit extreme motion blur as well
as over- and under-saturated image intensities.
When ground-truth position data is available, we follow
[6], [24], [19] and evaluate performance using both relative
pose error (RPE) and absolute trajectory error (ATE). When
the sensor system is used as a handheld rig, we obtain
ground-truth poses for the beginning and end of the sequence
only. This approach, introduced in TUM monoVO dataset
[5], requires these segments exhibit loopy camera motion
and fixate on the same scene. In a sequence containing n
frames, let S ⊂ [1; n] and E ⊂ [1; n] denote the subset of
frames making up the start and end segments, respectively.
We concatenate two segments together and run stereo ORB-
SLAM [14] on the resulting sequence to obtain relative
positions pˆi ∈ R3 for each frame in S ∪ E. We then
independently align these segments with the corresponding
position estimated pi yielded by a visual odometry system
X framework [4]. As our main evaluation metric is accumulated
Tsgt := argmin (T pi − p̂i )2 (4)
T ∈SE3
i∈S
X the performance of stereo OKVIS, stereo ORB-SLAM and
Tegt := argmin (T pi − p̂i )2 .
T ∈SE3
i∈E
This provides us with two relative
q Ptransforms for evaluating
n
the alignment error ealign = n1 i=1 kTsgt pi − Tegt pi k. As
noted in [5], for fair assessment using this metric, it is
imperative that loop-closure is not performed.
We publish code for computing all evaluation metrics de-
scribed here. Users need only provide a CSV file representing
an estimated trajectory for a given sequence, where each line
contains a timestamp followed by a 3-DoF position or 6-DoF
pose. We first determine the appropriate SE3 (or optionally
SIM3) transform for aligning the two trajectories. A full 6-
DoF pose may also be provided, indicated by four additional
values representing the unit quaternion. Orientation error will
50% Light 100% Light 15% Light 50% Light 100% Light
10
0
ORB-SLAM (Mono)
300
OKVIS-Stereo
Number of Runs
OKVIS-Mono
200 ORB Stereo Fwd
ORB Stereo Bwd
ORB Mono Fwd
ORB Mono Bwd
100
DSO Fwd
DSO Bwd
0
0 2 4 6 8 10
ealign
Fig. 7: A comparison of several state-of-the-art systems
running on the proposed dataset. Here we show the number
of successful runs on the vertical axis, as determined by the
given alignment error threshold on the horizontal axis.
then be assessed using the poses yielded by our batch ML
estimator, as described in Section V-D. When evaluating
accumulated drift, however, we use the ground-truth 6-DoF
pose that is already available.
VII. B ENCHMARK
We evaluate the performance of several start-of-the-art
visual and visual-inertial odometry systems running on our
dataset. Specifically, we analyze the stereo and monocular
versions of OKVIS [10], the stereo and monocular versions
of ORB-SLAM [14], and the Direct Sparse Odometry (DSO)
drift, we disabled loop-closure when running ORB-SLAM.
The color-encoded tables presented in Figure 5 illustrate
(5)
monocular ORB-SLAM. Each respective table is further
sorted by the onboard light intensity employed. As runtime
performance is typically non-deterministic, we run the same
sequence 10 times and visualize their results in separate
columns. From these results, we can see that the stereo
OKVIS performs well on all sequences. Stereo ORB-SLAM
also exhibits good performance, however, tracking failed on
two sequences where the dimmest lighting was employed.
ORB-SLAM failed consistently on the same frames, shown
in Figure 6. Not surprisingly, both exhibit severe motion blur
and poorly saturated image intensities.
We compare all evaluated frameworks in Figure 7. The
plot shows the number of successful runs on the vertical-
axis, as determined by the alignment-error threshold on the
 OKVIS-Stereo OKVIS-Mono
t1 t3
t3
50
t2

Y-translation (m)

Y-translation (m)
40 40

30
t1
20 20
t2
10
t0 t0 0 0

−10
DSO Forward DSO Backward −60 −40 −20 0 −70 −60 −50 −40 −30 −20 −10 0
X-translation (m) X-translation (m)
Fig. 8: Top-down view of point-clouds generated by DSO ORB-Mono DSO
moving forwards (left) and backwards (right) through a mine. 30
100
Note the mine shaft itself is of uniform width. Corresponding
points have been labeled to aid visual comparison.

Y-translation (m)

Y-translation (m)
20 50

0
100 100 10

−50
Number of Runs

Number of Runs

80 80
0
−100
60 60

−30 −20 −10 0 −50 0 50 100 150

40 40
X-translation (m) X-translation (m)
OKVIS-Mono 100% ORB-Stereo 100%
20 OKVIS-Mono 50% 20 ORB-Stereo 50% Fig. 10: Top-down views of the trajectories produced by
OKVIS-Mono 15% ORB-Stereo 15%
0 0 each system on the TN 100 HH 02 data sequence. Here, the
0 2 4 6 8 10 0 2 4 6 8 10
ealign ealign total distance traveled is approximately 285m. The starting
100 100 position of each trajectory is at the origin (0, 0).
ORB-Mono 100% DSO 100%
ORB-Mono 50% DSO 50%
Number of Runs

Number of Runs

80
ORB-Mono 15%
60
40
20
0 0
0 2 4 6 8 10
ealign
Fig. 9: Lighting sensitivity analysis of each odometry system.
Performance is individually plotted for the three lighting
configurations: 15%, 50%, and 100% max intensity.
horizontal axis. An ideal framework would have a horizontal
line at 300. Notably, stereo ORB-SLAM initially performs
better than monocular OKVIS, but flattens out around 250
runs, as tracking was completely lost for many sequences.
As described in [27], we see that ORB-SLAM exhibits a
fair amount of motion bias, performing significantly better
when the sequences are run backwards. The same is true for
DSO when running on our benchmark. We conjecture this
to be a direct result of violating the brightness constancy
assumption. In general, corresponding points will become
brighter in subsequent frames when tracked through forward
motion. Consequently, direct methods will likely match with
darker, more distant points. The inferred depth will therefore
increase to satisfy the multi-view geometry problem. This
phenomenon is illustrated in Figure 8. Note how the scale-
drift is reversed under backwards motion.
In Figure 9, we evaluate each framework’s sensitivity to
the lighting solution. Both OKVIS and DSO appear to be
80
DSO 15%
60 quite robust to light intensity. This can be attributed to their
use of inertial information and direct alignment, making them
40
inherently robust to motion blur. In contrast, we can see that
20 both the stereo and monocular versions of ORB-SLAM are
far more sensitive to the lighting.
0 2 4 6 8 10
Finally, we provide individual trajectories generated by
ealign each framework, which are indicative of their general perfor-
mance. The shown sequence was captured in a steam tunnel,
has a total distance traveled around 285m, and starts and ends
in the same location. Both the stereo and monocular versions
of OKVIS exhibit very little drift, while monocular ORB-
SLAM and DSO’s poor performance is largely the result
of extreme scale drift. We note, however, both systems still
manage to reconstruct the overall trajectory well.
VIII. DATA ACCESS
Our entire dataset, including the raw data streams
and calibration sequences, is publicly available online at:
http://arpg.colorado.edu/research/oivio
To facilitate the evaluation of different frameworks, we also
provide scripts for changing image resolution and converting
from the ASL format to ROS bags. Additionally, code for
performing quantitative and qualitative analysis as well as
generating all figures presented here are also provided.
ACKNOWLEDGMENTS
We would like to thank Cesar Galan for his work in
designing and constructing our sensor rig. Financial support
from the DARPA Subterranean Challenge Award HR0011-
18-2-0043 is also gratefully acknowledged.
 R EFERENCES
[1] Takahito Aoto, Tomokazu Sato, Yasuhiro Mukaigawa, and Naokazu
Yokoya. Linear estimation of 4-D illumination light field from diffuse
reflections. In Asian Conference on Pattern Recognition (IAPR), 2013.
[2] M Burri, J Nikolic, P Gohl, T Schneider, J Rehder, S Omari, M Achte-
lik, and R Siegwart. The EuRoC MAV Datasets. International Journal
of Robotics Research (IJRR), 2015.
[3] Defense Advanced Research Projects Agency (DARPA). Subterranean
challenge, 2018. http://www.subtchallenge.com [accessed:
2019-02-01].
[4] Jakob Engel, Vladlen Koltun, and Daniel Cremers. Direct Sparse
Odometry. Transactions on Pattern Analysis and Machine Intelligence
(TPAMI), 2018.
[5] Jakob Engel, Vladyslav Usenko, and Daniel Cremers. A Photometri-
cally Calibrated Benchmark For Monocular Visual Odometry. arXiv
preprint, 2016.
[6] Andreas Geiger, Philip Lenz, and Raquel Urtasun. Are we ready for
autonomous driving? the KITTI vision benchmark suite. In Computer
Vision and Pattern Recognition (CVPR), 2012.
[7] Ankur Handa, Thomas Whelan, John McDonald, and Andrew J.
Davison. A benchmark for RGB-D visual odometry, 3D reconstruction
and SLAM. In International Conference on Robotics and Automation
(ICRA), 2014.
[8] M Irani and P Anandan. About Direct Methods. Vision Algorithms,
2000.
[9] Christian Kerl, Jurgen Sturm, and Daniel Cremers. Dense visual
SLAM for RGB-D cameras. In Intelligent Robots and Systems (IROS),
2013.
[10] Stefan Leutenegger, Simon Lynen, Michael Bosse, Roland Siegwart,
and Paul Furgale. Keyframe-based visualinertial odometry using
nonlinear optimization. International Journal of Robotics Research
(IJRR), 2014.
[11] Steven Lovegrove, Alonso Patron-Perez, and Gabe Sibley. Spline
Fusion: A continuous-time representation for visual-inertial fusion
with application to rolling shutter cameras. In British Machine Vision
Conference (BMVC), 2013.
[12] Will Maddern, Geoffrey Pascoe, Chris Linegar, and Paul Newman. 1
Year, 1000km : The Oxford RobotCar Dataset. International Journal
of Robotics Research (IJRR), 2017.
[13] Michael Milford, Eleonora Vig, Walter Scheirer, and David Cox.
Vision-based Simultaneous Localization and Mapping in Changing
Outdoor Environments. Journal of Field Robotics (JFR), 2014.
[14] Raul Mur-Artal and Juan Tardos. ORB-SLAM2: An Open-Source
SLAM System for Monocular, Stereo, and RGB-D Cameras. Trans-
actions on Robotics, 2017.
[15] Richard Newcombe, Steven Lovegrove, and Andrew Davison. DTAM:
Dense tracking and mapping in real-time. In International Conference
on Computer Vision (ICCV), 2011.
[16] Fernando Nobre, Michael Kasper, and Christoffer Heckman. Drift-
correcting self-calibration for visual-inertial slam. In International
Conference on Robotics and Automation (ICRA), 2017.
[17] Jaesik Park, Sudipta Sinha, Yasuyuki Matsushita, Yu Wing Tai, and
In So Kweon. Calibrating a non-isotropic near point light source using
a plane. In Computer Vision and Pattern Recognition (CVPR), 2014.
[18] Seonwook Park, Thomas Schöps, and Marc Pollefeys. Illumination
Change Robustness in Direct Visual SLAM. International Conference
on Robotics and Automation (ICRA), 2017.
[19] Bernd Pfrommer, Nitin Sanket, Kostas Daniilidis, and Jonas Cleveland.
PennCOSYVIO: A challenging Visual Inertial Odometry benchmark.
In International Conference on Robotics and Automation (ICRA),
2017.
[20] Nicholas Roy, Henrik Christensen, Dieter Fox, M. Ani Hsieh, Stuart
Young, Simon Ng, Dirk Schulz, Ethan Stump, Carlos Nieto-Grand,
and Christopher Reardon. International Conference on Robotics and
Automation (ICRA) Workshop on Robot Teammates Operating in
Dynamic, Unstructured Environments (RT-DUNE), 2018. https:
//manihsieh.com/icra-2018-workshop/ [accessed: 2019-
02-01].
[21] Dirk Schnieders, Kwan-yee Wong, and Zhenwen Dai. Polygonal Light
Source Estimation. In Asian Conference on Computer Vision (ACCV),
2009.
[22] Boxin Shi, Zhe Wu, Zhipeng Mo, Dinglong Duan, and Sai-kit Yeung
Ping. A Benchmark Dataset and Evaluation for Non-Lambertian and
Uncalibrated Photometric Stereo. In Computer Vision and Pattern
Recognition (CVPR), 2016.
[23] Zijiang Song and Reinhard Klette. Robustness of Point Feature
Detection. Computer Analysis of Images and Patterns (CAIP), 2013.
[24] Jürgen Sturm, Nikolas Engelhard, Felix Endres, Wolfram Burgard, and
Daniel Cremers. A benchmark for the evaluation of RGB-D SLAM
systems. In Intelligent Robots and Systems (IROS), 2012.
[25] Wennie Tabib and Nathan Michael. Robotics: Science and Systems
(RSS) Workshop on Challenges and Opportunities for Resilient Col-
lective Intelligence in Subterranean Environments, 2018. http://
rssworkshop18.autonomousaerialrobot.com/ [accessed:
2019-02-01].
[26] Robert J Woodham. Photometric Stereo: A Reflectance Map Technique
For Determining Surface Orientation From Image Intensity. Image
Understanding Systems and Industrial Applications, 1979.
[27] Nan Yang, Rui Wang, Xiang Gao, and Daniel Cremers. Challenges
in Monocular Visual Odometry: Photometric Calibration, Motion Bias
and Rolling Shutter Effect. arXiv preprint, 2018.
[28] Wei Zhou and Chandra Kambhamettu. Estimation of the Size and
Location of Multiple Area Light Sources. In International Conference
on Pattern Recognition (ICPR), 2004.