See discussions, stats, and author profiles for this publication at: https://www.researchgate.

net/publication/4045874

Visual control of a robotic hand

Conference Paper · November 2003

DOI: 10.1109/IROS.2003.1248819 · Source: IEEE Xplore

CITATIONS READS
24 279

4 authors, including:

Ignazio Infantino Antonio Chella

Italian National Research Council Università degli Studi di Palermo
98 PUBLICATIONS 639 CITATIONS 291 PUBLICATIONS 2,315 CITATIONS

SEE PROFILE SEE PROFILE

Irene Macaluso
Trinity College Dublin
100 PUBLICATIONS 936 CITATIONS

SEE PROFILE

Some of the authors of this publication are also working on these related projects:

UnipaBCI Augmented Framework View project

Self-Consciousness and Theory of Mind for a Robot Developing Trust Relationships View project

All content following this page was uploaded by Antonio Chella on 21 May 2014.

The user has requested enhancement of the downloaded file.

 Visual Control of a Robotic Hand
Ignazio Infantino1 Antonio Chella1,2
1 2
ICAR-CNR sez. di Palermo
Viale delle Scienze, edif. 11,
90128, Palermo, Italy
(infantino, dzindo, macaluso)@pa.icar.cnr.it
Abstract -- The paper deals with the design and
implementation of a visual control of a robotic system composed
of a dexterous hand and stereo cameras. The aim of the proposed
system is to reproduce the movements of a human hand in order
to learn complex manipulation tasks. A novelty algorithm for a
robust and fast fingertips localization and tracking is presented.
Moreover, a simulator is integrated in the system to give useful
feedbacks to the users during operations, and provide robust
testing framework for real experiments (see video).
I. INTRODUCTION
The control of robotic systems has reached a high level
of precision and accuracy, but often the high complexity and
task specificity are limiting factors for large scale uses.
Today, robots are requested to be both “intelligent” and
“easy to use”, allowing a natural and useful interaction
with human operators and users. In this paper, we deal with
the (remote) control of an anthropomorphic robotic hand:
our approach is to provide a Human-Computer-Interface
[10][17][15] based on natural hand gesture imitation of the
operator showing his hand [8]. Different methods have been
proposed to capture human hand motion. Rehg and Kanade
[11][12] introduced the use of a highly articulated 3D hand
model for the tracking of a human hand [13][14]. For
tracking, the axes of the truncated cylinders that are used to
model phalanges, are projected onto the image, and local
edges are found. Fingertip positions are measured through a
similar procedure. A nonlinear least squares method is used
to minimize the error between the measured joint and tip
locations and the locations predicted by the model. The
system runs in real-time, however, dealing with occlusions
and handling background clutter remains a problem. Heap
and Hogg [6] used a deformable 3D hand shape model. The
hand is modeled as a surface mesh which is constructed via
PCA from training examples. Real-time tracking is achieved
by finding the closest possibly deformed model matching
the image. In [16], Cipolla and Mendoca presented a stereo
hand tracking system using a 2D model deformable by
affine transformations. Wu and Huang [19] proposed a two-
step algorithm to estimate the hand pose, first estimating the
global pose and subsequently finding the configuration of
the joints. Their algorithm relies on the assumption that all
fingertips are visible.
Our method uses fingertips as features, considering each
finger except the thumb as planar manipulator, and starting
from this hypothesis we compute inverse kinematics to
Haris Džindo1 Irene Macaluso1
DINFO Università di Palermo
Viale delle Scienze,
90128, Palermo, Italy
chella@unipa.it
control the robotic hand. This paper is structured as follows:
section II describes the whole system and the
experimentation setup; section III deals with the fingertip
localization and the tracking on image sequence; section IV
concerns the control of robotic hand by inverse kinematics
computation; section V reports the details about various
experimentations done; finally, some conclusions and future
work are drawn in section VI.
II. THE IMPLEMENTED SYSTEM DESCRIPTION
Our proposal deals with the design and implementation
of a visual control of a robotic system composed by a
dexterous anthropomorphic hand (the DIST-Hand [1]) and
calibrated stereo cameras. The aim of the present work is to
reproduce real-time movements of a human hand in order to
learn complex manipulation tasks.
The human hand is placed in front of the stereo couple. A
black panel is used as background and illumination is
controlled. Fingertips and wrist are extracted at each frame
and their 3D coordinates are computed by a standard
triangulation algorithm. Given a hand model, inverse
kinematics is used to compute the joint angles which are
translated to suitable commands sent to the DIST-Hand
controller.
The system (see figure 1) is composed by some common
modules that work in parallel on the two flows of images
sourcing from the stereo rig:
1) image pre-processing operation (noise filtering,
tresholding, edge detecting);
2) the fingertip tracking following the first estimation;
3) fingertip localization on image based on previous
position.
The coordinates of the fingertips are the input of the other
software modules responsible of:
1) 3D coordinate computation;
2) inverse kinematics;
3) actuator commands computation.
III. FINGERTIP LOCALIZATION AND TRACKING
This section deals with the localization of the fingertips
in long image sequences. In order to achieve real-time
performances state-space estimation techniques are used to where (r(k), c(k)) represent the fingertip pixel position at
reduce the search space from the full camera view to a kth frame, and (vr(k) , vc(k)) be its velocity in r and c
smaller search window centered on the prediction of the directions.
future position of the fingertips. In the following, a novel
algorithm is proposed to detect the fingertip position inside
the search window.

I*
Left Image Image Fingertip xreal 3D Coordinate
Preprocessing xpred Extraction Computation
Right Image Module Module Module

a b
Kalman Filter Inverse
Module Kinematics
Module

Qhuman
c
Initial estimates
Human-To-
Robot Mapping Fig.2 (a) Initial hand configuration used to calculate the first estimation of
Module the fingertip positions; (b) three points (yellow squares) with high
correlation response individuate a group and provide a fingertip candidate
Qrobot computed as centroid of such a group (red arrow); (c) circular templates
used to search fingertips.

To the communication
units Under the assumption that a fingertip moves with
constant translation over the long sequence and a
sufficiently small sampling interval (T), the plant equation
for the recursive tracking algorithm can be written as
Fig.1 The system architecture: two parallel set of procedures perform
separately the estimation of the fingertips on right and left images; the
coordinates extracted are the input of the procedures responsible for the X(k+1)=Ax(k)+w(k)
control of the robotic hand.
Where
The visual servoing procedure starts with a preliminary
setup phase, based on various image processing operations. 1 0 T 0
The procedure individuates the hand/arm on image, 0 1 0 T
compares it with robotic one and calculates exact size ratios. A =
0 0 1 0
Relevant features are tracked on image sequences, and 0 0 0 1
reconstructed in a three-dimensional space, using standard
computer vision algorithms [3]. These features are used as a and the plant noise w(k) is assumed to be zero mean with
guide to perform 3D reconstruction of the simulated covariance matrix Q(k).
arm/hand, and real system positioning. The reconstructed For each fingertip, the measurement used for the
movements are reproduced both by the simulator and the corresponding recursive tracking filter at time tk is the
robotic system in real time. image plane coordinates of the corresponding point in the
kth image. Thus the measurement is related to the state
A. Fingertip Tracking vector as:

In this section we are involved with prediction of Z(k) = Hx(k) + v(k)

feature points in long image sequences. If the motion of the where
observed scene is continuous, as it is, we should be able to
make predictions on the motion of the image points, at any 0 1 0 0
time, on the basis on their previous trajectories. Kalman H =
0 0 1 0
filter [1] [16] is the ideal tool in such tracking problems.
The motion of a fingertip at each time instance (frame) and the measurement noise v(k) is assumed to be zero mean
can be characterized by its position and velocity. For a with covariance matrix R(k).
feature point, define the state vector at time tk as After the plant and measurement equations have been
formulated, the Kalman filter can be applied to recursively
x(k)= [ r(k) c(k) vr(k) vc(k)]T estimate the motion between two consecutive frames and
track the feature point.
Assuming that the trajectory of a feature point has been
established up to the kth frame of a sequence, we describe
procedures for estimating the in-frame motion between the
kth and (k+1)th frames. First, for a fingertip, the predicted
location of its corresponding point z(k+1|k) is computed by
the KF. Subsequently, a window centering at z(k+1|k) is
extracted from the (k+1)th image and the fingertip extraction
algorithm is applied to the window to identify salient feature
points as described in the next section. The same process is
repeated for each fingertip.
The initial fingertips and wrist coordinates are computed by
assuming the hand is placed as in figure 2.a. Under such
hypothesis it is straightforward to compute the fingertip
coordinates.
B. Fingertip Extraction
This subsection describes a novelty algorithm to
perform robust and fast localization of the fingertips.
Once a search window is determined for each fingertip, the
feature point is searched for within that window. The overall
shape of a human fingertip can be approximated by a
semicircle. Based on this observation, fingertips are
searched for by template matching with a set of circular
templates as shown in figure 2.c. Ideally, the size of the
templates should differ for different fingers and different
users. However, our experiments showed that fixed window
size works well for various users. This result has been
proven by other authors [9]. We choose a square of 20x20
pixels with a circle whose radius is 10 pixels as a template
for normalized correlation in our implementation.
In order to find meaningful data we first perform an
adaptive threshold on the response images to obtain distinct
regions. For each correlation we select the points with the
highest matching scores (usually 2-3) which lies in
unconnected regions of the thresholded image.
Subsequently, near responses of different correlations
are grouped. We select the groups characterized by the
highest number of points associated and the centroid of such
groups are used to compute the fingertip candidates. In
order to calculate those candidates, we must combine the
information provided by each point in the group. However,
only the points corresponding to semi-circle correlation
responses are used in this stage.
Each point is associated with a vector whose direction
depends on the orientation of the corresponding semi-circle
(i.e. right - 0°, up - 90°, and so on). Its module is given by
the correlation matching score. The direction in which the
fingertip candidate should be searched is computed as the
angle formed by the vector sum of such vectors (see figure
2.b). Therefore, the fingertip candidate coordinate for each
group lies on the line through the previously computed
centroid and whose direction is given by the above
procedure. We use techniques of data association based on
the previous measurements and predicted state to choose the
correct fingertip among the selected candidates.
The robustness of this algorithm has been tested using
various fingers configuration: also in the more complicated
case of two very close fingers on the background palm the
system follow the correct feature. The addition of artificial
noise or the use of a lower image scale doesn’t degrade the
performance of the algorithm (see figure 3). No constraints
of the possible hand postures are necessary and gray-level
video images are sufficient to obtain good results.
Fig.3 Results of fingertips extraction in different hand postures; the
algorithm robustness has been tested also with artificial noise added.
C. Three-dimensional coordinate computation
Given the camera matrices it is possible to compute the
fundamental (or essential) matrix of the stereo rig [3].
Epipolar geometry is used to find correspondences between
two views.
In order to reduce computational efforts, matching
features are searched on the epipolar line in the search
window of the second image provided by KF by means of
normalized correlation. 3D fingertip coordinates are
computed by a standard triangulation algorithm.
IV. THE ROBOTIC HAND CONTROL
The coordinates of the four fingertips and the wrist are
used to solve the inverse kinematics problem for joint
angles, provided a kinematics model of a human hand. The
model adopted [7] is designed to remain simple enough for
inverse kinematics to be done in real-time, while still
respecting human hand capabilities.
The whole hand is modelled by a 27 DOF skeleton
whose location is given by the wrist’s middle point and
whose orientation is that of the palm. Each finger consists of
three joints. Except for the thumb, there are 2 degrees of
freedom (DOF) for metacarpophalangeal (MCP) joints, and
1 DOF for proximal interphalangeal (PIP) joints and distal
interphalangeal (DIP) joints. The thumb is modelled by a 5
DOF kinematic chain, with 2 DOF for the trapezio-
metacarpal (TM) and MCP joint and 1 for interphalangeal
(IP) joint.
In order to obtain meaningful configurations and to
reduce the dimensionality of the joint space we include hand
constraints [21] in the model. The constraints are of two
types: static (limit the range of finger motion) and dynamic
(describe relationships between some finger joints, i.e. qII-
IV II-IV
DIP=(2/3)q PIP). Besides, the thumb can be modelled as a
4 DOF manipulator. Since the robotic hand has four fingers
and it is not placed on an arm, we are not interested neither
in the posture of the pinky, nor in the position/orientation of
the hand. Then, given the hand constraints, it is possible to
reduce the number of DOF to 13.
A. Inverse Kinematics
In order to solve the inverse kinematics problem,
calibration of the human hand has to be performed to adapt
to various users. This process is must be done off-line using
an interactive procedure. We use 6 frames to represent
finger kinematics: Oc (the camera frame), Oh (the hand
frame whose origin is the wrist’s middle point and whose
axes are that of the camera frame) and Ofr(I-IV) (the finger-
root frame whose origin is at first flexion/extension joint of
each finger and whose axes are defined by the palm
orientation with z axes perpendicular to the palm plane and
x axis pointing towards the respective fingertip in the initial
configuration).
vector qi(k) may be computed by minimizing the following
function:
F (qi(k)) = D ( xi(k), k(qi(k)) ), i=I...IV
subject to the human hand constraints. The initial guess is
set to the previously computed joint vector qi(k-1) since
under the slow motions assumption we can state that the
joint angles vary little from time k-1 to time k.
The computed positions of the hand are sent to the
human-to-robot mapping unit which computes the
corresponding robot joint angles. These angles are
communicated by local network to the real hand, or by the
DDE mechanism to the simulator (see fig. 4).

Fig.4 A simulator is part of the system architecture, allowing to visualize

the reconstructed movements on the virtual robotic hand. The 3D model
used is very similar to real robotic hand and receive the same commands.
The simulator also includes the model of a PUMA-arm that in future will
complete our system architecture.

Under the assumption that the hand orientation doesn’t

change in time, we can compute the fixed transformation
matrixes Thfr(I-IV) during the initialization phase. In such a
way it is possible to express the coordinates of each
fingertip in the finger-root frame and solve for the joint state
vector. For each finger except for the thumb we compute the
joint variable qMCP-AAII-IV as the angle between the projection
of the fingertip on the palm plane and the x axes. By
rotating the finger-root frame around z axis by angle qMCP-
II-IV
AA , each finger II-IV can be represented as a 3-link
planar manipulator. The joint angles qMCP-FE, qPIP and qDIP
for the fingers II-IV are then found by solving the inverse
kinematics problem for such a manipulator. The thumb joint
angles are found directly by solving the inverse kinematics
problem for the kinematics model associated with it. Fig.5 . Hand tracking and control of robotic hand movements using web-
cameras.
Given the fingertip coordinates xi(k) and the forward
kinematics equation xi(k)=k[qi(k)], i=I...IV, and denoting
with D(x,y) the distance between two points, each state
Sample images Hand movement Success rate (%) The software (the described system and the simulator) runs
on a personal computer (Pentium III, 450 MHz), equipped
Opened hand 100% with two video grabber cards and an ethernet card. The
(initial position) movements command are send to robotic DIST-hand using
a TCP-IP link to it.
The various procedures implemented to perform the
visual control allow a frame rate of 10 images per second,
permitting a qualitative correct reproduction of the
Thumb 89%
movements of the real hand presented (see video).
(T)
If the system find inconsistencies of fingertip positions
or fails to follow them (mainly due a too rapid movements),
the user can establish a new visual control showing the hand
in the position of initialization. A more reactive behavior of
Index 92%
the system is obtained if it is used only the images of a
(I)
single camera: avoiding the 3-d reconstruction of the
fingertips, it is possible to perform the control of the robotic
hand using the variations of the length ratios from the initial
position.
Middle 91% Naturally, the precision is lower but it is enough to
(M) execute manipulation or grasping tasks (see example shown
in figure 5). We have tested the system performing 17
standard movements starting from the initial position:
opened hand (no movement), movements involving only
Ring 90% one finger (4 cases), movements involving two fingers (6
(R) cases), three fingers (4 cases), four fingers, closed hands
(see figure 6). For each final posture, it has been calculated
the percentage of success in reaching correct final position
by qualitative observation of the robotic hand posture. The
Two fingers 85% results shown in figure 6 are related to 20 acquired
(TI, TM,TR, sequences for each class of movements.
IM, IR, MR)
VI. CONCLUSIONS

Three fingers 80% The research demonstrates how to control in “natural

(TIM, TIR, way” a robotic hand using only visual information. The
IMR, TMR) proposed system is able to reproduce the movements of a
human hand in order to perform telepresence tasks or to
learn complex manipulation tasks.
Four fingers 74% Moreover, a novelty algorithm for a robust and fast
(TIMR) fingertips localization and tracking is been presented and
results shown the goodness of the proposed solution. Future
work will extend the visual control to a complete robotic
arm-hand system.
Closed hand 73%
(TIMR+P) VII. ACKNOWLEDGMENTS

This research is partially supported by MIUR (Italian

Fig.6 .We have tested the system performing 17 standard movements
starting from the initial position. For each final posture, it ha been
calculated the percentage of success in reaching correct final position by
qualitative observation of the robotic hand. The results shown are related
to 20 acquired sequences for each class of movements. Note: letters
T,I,M,R,P indicate the fingers.
V. EXPERIMENTAL RESULTS
We have tested our proposed method by using two
different stereo rigs: the first one was composed by two
Sony cameras, and the second one by two usb web-cams.
Ministry of Education, University and Research) under
project RoboCare (A Multi-Agent System with Intelligent
Fixed and Mobile Robotic Components).
VIII. REFERENCES
[1] S. Bernieri, A. Caffaz, G. Casalino, “The DIST-Hand”
Robot IROS '97, Conference Video Proceedings,
Grenoble (France), September 1997.
[2] C. K. Chui, G. Chen, Kalman Filtering: With Real-
Time Applications, Springer Series in Information
Sciences, 1999.
 [3] O. Faugeras, Three-Dimensional Computer Vision,
MIT Press, Cambridge, MA, 1993.
[4] B.P.K. Horn, Robot Vision, MIT Press, Cambridge,
MA, 1986.
[5] F. Lathuilière, J.Y. Hervé, “Visual Hand Posture
Tracking in a Gripper Guiding Application”, in Proc.
Of IEEE Int’l Conference on Robotics and
Automation, San Francisco, April 2002.
[6] A. J. Heap, D. C. Hogg, “Towards 3-D hand tracking
using a deformable model”, in 2nd International Face
and Gesture Recognition Conference, pp. 140-145,
Killington, Vermont, USA, October 1996.
[7] J. Lee, T.L. Kunii, “Model-based analysis of Hand
Posture”, in IEEE Computer Graphics and
Applications, pp. 77-86, 1995.
[8] C. Nolker, H. Ritter, “Visual Recognition of
Continuous Hand Postures”, IEEE Transaction on
Neural Network, Vol. 13, no. 4, July 2002.
[9] K. Oka, Y. Sato, H. Koike, ”Real-time tracking of
multiple fingertips and gesture recognition for
augmented desk interface systems”, in Proc. 2002
IEEE Int'l Conf. Automatic Face and Gesture
Recognition 2002 , pp. 429-434, May 2002.
[10] V. Pavlovic, R. Sharma, T.S. Huang., “Visual
interpretation of hand gestures for human-computer
interaction: a review”, IEEE PAMI, 19(7), 1997.
[11] J.M. Rehg, T. Kanade, “DigitEyes: Vision-Based
Hand Tracking for Human-Computer Interaction”, in
Proc. of the IEEE Workshop on Motion of Non-Rigid
and Articulated Objects, Austin, Texas, Nov. 1994,
pp.16-22.
[12] J.M. Regh, T. Kanade, “Visual tracking of high dof
articulated structures: an application to human hand
tracking”, in Proc. of Third European Conf. on
Computer Vision, Stockholm, Sweden, 1994, pp. 35-
46.
View publication stats
[13] R. Rosales, V. Athitsos, L. Sigal, S. Sclaroff, “3D
Hand Pose Reconstruction Using Specialized
Mappings”, in Proc. IEEE Intl. Conf. on Computer
Vision (ICCV’01), Canada, Jul. 2001.
[14] Y. Sato, M. Saito, H. Koike, “Real-Time Input of 3D
Pose and Gestures of a User’s Hand and Its
Applications for HCI”, in Proc. of Virtual Reality
2001 Conference, pp. 13-17, March 2001, Yokohama,
Japan.
[15] T. Starner, J. Weaver, A. Pentland, “Real-time
American sign language recognition using desk- and
wearable computer-based video”, IEEE PAMI, 20(12):
1371–1375, 1998.
[16] B.D.R. Stenger, P.R.S. Mendonca, R. Cipolla,
“Model-Based Hand Tracking Using an Unscented
Kalman Filter”, in Proc. British Machine Vision
Conference, Manchester, UK, September 2001.
[17] B.D.R. Stenger, P.R.S. Mendonca, R. Cipolla, “Model
based 3D tracking of an articulated hand”, in Proc.
CVPR‘01, pages 310–315, 2001.
[18] S. Waldherr S., Thrun S., Romero R, “A gesture-based
interface for human-robot interaction”, Autonomous
Robots, 9, (2), 2000, 151-173.
[19] Y.Wu, T. S. Huang, “View-independent Recognition
of Hand Postures”, in Proc. of IEEE CVPR'2000, Vol.
II, pp.88-94, Hilton Head Island, SC, 2000
[20] Y. Wu, J.Y. Lin, T. S. Huang, “Capturing Natural
Hand Articulation”, in Proc. of IEEE Int'l Conf. on
Computer Vision, Vancouver, Canada, 2001.
[21] Y. Wu, J. Y. Lin, T. S. Huang, "Modeling Human
Hand Constraints", in Proc. Of Workshop on Human
Motion (Humo2000), Austin, TX, Dec., 2000.