See discussions, stats, and author profiles for this publication at: https://www.researchgate.

net/publication/378234606

A High Frame Rate Stereovision Tracking System

Poster · May 2008

DOI: 10.13140/RG.2.2.12763.95526

CITATIONS READS
0 7

1 author:

Siraj Sabihuddin
W3NG Inc.
18 PUBLICATIONS 345 CITATIONS

SEE PROFILE

All content following this page was uploaded by Siraj Sabihuddin on 15 February 2024.

The user has requested enhancement of the downloaded file.

 F a s tT ra c k
A High Frame Rate Stereovision Tracking System
Jamin Islam1, Siraj Sabihuddin1, W. James MacLean1, Valeri Kirischian2 ,Vadim Geurkov2, Lev Kirischian2, Michael Belshaw3, Michael Greenspan3
1
Dept. of Electrical & Computer Engineering, University of Toronto; 2Dept. of Electrical & Computer Engineering, Ryerson University; 3Dept. of Electrical & Computer Engineering, Queen’s University

3) Rectification
Project Goals: To construct a
stereo vision system capable of Transforms a pair of stereo images
Hardware Design
tracking objects at 200fps. into a parallel camera geometry so ● Storage of N scanlines ● Inverse Warp
that corresponding point features in into dual-ported BRAMs computes warped
3D will lie on the same scanline in camera
data
camera control ● Provides 2x2 region of coordinate location
Motivation: Remote satellite both images. Address Generator
start
pixels for interpolation via homographies
● Computes the ideal

original
Contains 2 BRAMs to pixel intensity for a
capture, high speed motion
●
coordinates
Pixel Sync
Reduces the stereo correspondence Buffer Inverse Warp Control
alternate read for given warped
stereo extraction coordinate location
tracking. problem to a one-dimensional warped
coordinates
● Provides read signal
using Bilinear
search. read
address
Out of Range
OOR &
coefficients
& address to output Interpolation
2x2 pixel OOR &
computed disparities
region coefficients
HL & HR are 3x3 homographies rectification
data
stereo
data
Rectification-to-
which transform the image planes
System Components
Bilinear Interpolator
Stereo Interface
rectification disparity read

and make them collinear & control control

horizontal

1) Stereo Imaging 2) Calibration 4) Stereo 5) Tracking Homographies are calculated

Un-rectified
using calibration parameters Performance
Hardware 3) Rectification Extraction and implemented in hardware Stereo
as a 1st-order Taylor Series Resolution:
640x480 pixels Images
polynomial P

Left Images Right Images

Freq. (MHz) Frames/Sec
Images
Acquired
Images
warped
Depth
Acquired
Object
Pose 31.25 50.862 Result:
Acquired 62.50 101.72 Rectified
125.0 203.45 Stereo
HL HR
Images
CL CR

1) Stereo Imaging & Hardware Platform 4) Stereo Extraction

200fps Stereo Frame Grabber Hardware Design Stereo extraction is based on a Maximum Likelihood
USB Dynamic Programming (DPML) algorithm developed
by Cox et al.

μC Algorithm
200 FPS 200 FPS 1) Initialize cost and match matrices.
Sensor Xilinx Xilinx Sensor 2) Compute the cost and match matrix values b/w all
FPGA FPGA
XC3S200 XC3S200
right and left image pixels in a scanline (OC/NOC).
3) Compute optimal stereo disparities via a backward
Control of two Parameter Real-time High speed VGA DAC VGA DAC pass through the match matrix.
200fps image control via Visualization image comm. High Speed Interface 4) Repeat for all scanlines in the image.
sensors. PC over USB. via VGA DACs. with AP1100.
Parallelization
● Exploit anti-diagonal structure of DPML cost
AP1100 FPGA Platform matrix for SIMD parallelization
● Pixel level pipelining to improve throughput and
High Speed Interface
and Amirix PCI Interface clock frequency.
SRAM SRAM ● Interleaving to pipeline forward and backward

Bank Bank
Xilinx phases of algorithm – scanline level pipelining.
FPGA
XC2VP100
Performance 320x240 ● 1.24 fps
SW Result:
640x480 ● 0.18 fps
Processed data Standalone system
The figures to the right
320x240 ● 248 fps
Dense
shows depth estimation HW
stored in SRAM that operates
results at 128 pixel max.
640x480 ● 123 fps Disparity
memory banks. without PC. 320x240 ● 200 fps Map
disparity and the indicated 640x480 ● 200 fps Goal Right Input Image Ground Truth HW Result
resolutions.

2) Calibration 5) Tracking
Calibration is the process of Tracking with Iterative Range Data
Model
determining the position and Closest Point (ICP) From Stereo Object's
orientation of the cameras Calibration Extraction
Pre-filter Data Data Nearest Point
Pose
with respect to a world reference 1) Find correspondences between model Transform SVD
Parameters & data using last pose estimate -
ROI
Points Points
Neighbor
Pairs
system.
Intrinsic Parameters: Nearest Neighbor (NN).
Calibration determines values Focal Length 2) Find transform using correspondences - Transform
for two types of parameters: Principal Point Singular Value Decomposition (SVD)
3) Apply transform to update data 3250 Pre-filter / ROI NN Approach
Intrinsic & Extrinsic Skew Coefficient 200fps goal 3000

Radial Distortion 4) Repeat 10 times. T

Fixed point 16bit 2750 ● Initially, data is ● Parallel brute force
2500
filtered using region approach.
Number of NN cyc/sec

FPGA @ 100Mhz
Tangential Distortion 2250
2000
of interest based on
3040
1750 inferred XY SVD
Extrinsic Parameters: NN Brute Force 1500
1250
Using integers
coordinates
● Data is then
●
SVD performed by using Microblaze μC.
A checkerboard pattern (as shown Rotation Speed Comparison
1000
750
x86 @ 3.4ghz
converted to world ●
To decrease overhead on μC the
on the right) is used to discover Translation 500
250
188 95 XYZ form and a formation of the covariance matrix is
Compare using 2048 bounding box is used performed in hardware.
these calibration parameters for 0
CPU FPGA-1 cmp FPGA-32 to further sub-sample
each camera in the stereo pair. model points & 512 data. cmp
and invalid points are HW CPU
points. rejected.

View publication stats