58
1, FEBRUARY 2005
Somatosensory Computation for Man–Machine
Interface From Motion-Capture Data and
Musculoskeletal Human Model
Yoshihiko Nakamura, Member, IEEE, Katsu Yamane, Member, IEEE, Yusuke Fujita, and Ichiro Suzuki
Abstract—In this paper, we discuss the computation of so-
matosensory information from motion-capture data. The efficient
computational algorithms previously developed by the authors
for multibody systems, such as humanoid robots, are applied to
a musculoskeletal model of the human body. The somatosensory
information includes tension, length, and velocity of the muscles,
tension of the tendons and ligaments, pressure of the cartilages,
and stress of the bones. The inverse dynamics of the muscu-
loskeletal human model is formulated as an optimization problem
subject to equality and inequality conditions. We analyzed the
solutions obtained by linear and quadratic programming methods,
and showed that linear programming has better performance. The
technological development aims to define a higher dimensional
man–machine interface and to open the door to the cognitive-level
communication of humans and machines.
Index Terms—Forward dynamics, inverse dynamics, man–ma-
chine interface, musculoskeletal human model, somatosensory in-
formation.
I. INTRODUCTION
T HE relationship between humans and machines will show
qualitative differences through technical developments of
communications and interactions. Such communications and in-
teractions would imply not only explicit and conscious signal
exchange, but also an implicit and subconscious one.
In this paper, we discuss the computation of somatosensory
information from motion-capture data. Somatosensory informa-
tion includes all the stimuli sensed by the sensory organs lo-
cated both outside and inside the body. We are particularly in-
terested in the internal somatosensory information such as ten-
sion, length, and velocity of the muscles, tension of the tendons
and ligaments, pressure of the cartilages, and stress of the bones.
The efficient computational algorithms previously developed by
the authors for multibody systems, such as humanoid robots,
Manuscript received October 21, 2003; revised March 26, 2004. This paper
was recommended for publication by Associate Editor J. P. Merlet and Editor
H. Arai upon evaluation of the reviewers’ comments. This work was supported
in part by the CREST program, in part by the Japan Science and Technology
Corporation, and in part by the Information-Technology Promotion Agency
(IPA) of Japan. This paper was presented in part at the International Symposium
on Adaptive Motion of Animals and Machines, Kyoto, Japan, March 2003.
Y. Nakamura and K. Yamane are with the Department of Mechano-In-
formatics, University of Tokyo, Tokyo 113-8656, Japan (e-mail: naka-
mura@ynl.t.u-tokyo.ac.jp; yamane@ynl.t.u-tokyo.ac.jp).
Y. Fujita is with Nomura Research Institute, Yokohama 223-0051, Japan
(e-mail: fujita@ynl.t.u-tokyo.ac.jp).
I. Suzuki is with Nintendo Corporation, Kyoto 601-8501, Japan (e-mail:
ichiro@ynl.t.u-tokyo.ac.jp).
Digital Object Identifier 10.1109/TRO.2004.833798
1552-3098/$20.00 © 2005 IEEE
Authorized licensed use limited to: Universidade Tecnologica Federal do Parana. Downloaded on June 23,2021 at 20:26:49 UTC from IEEE Xplore. Restrictions apply.
IEEE TRANSACTIONS ON ROBOTICS, VOL. 21, NO.
are applied to a high-dimensional musculoskeletal model of the
human body.
The somatosensory information sensed deep inside the
human body has not has much emphasis put on it for applica-
tions other than medicine and rehabilitation. Motion-capture
systems have been used to capture only the superficial changes
of the human body. By combining the motion-capture system,
the musculoskeletal model of the human body, and the effi-
cient computation of kinematics and dynamics, the authors
are exploring the possibility of realtime computation of the
somatosensory information.
We know from our daily experiences that we subconsciously
compute and estimate the somatosensory information of others,
even when we do not explicitly try to communicate with others
in verbal or nonverbal ways. For example, we can imagine
whether a person is tired or not by observing his/her walking
motion. We can also estimate which part of the body is the most
stressed in a lifting motion. It is also known in brain science
that a human uses the motor cortex not only for controlling
the muscles, but also for recognizing motions of others. Such
information is used in our cognitive process to understand the
other’s intention, mental state, and relationship with us. When
the man–machine interface obtains access to the somatosensory
information, machines would make the first step to understand
humans. Such implicit communication between humans and
machines could be termed as a cognitive-level communication.
The technology will enable us to build robots that naturally
reach out to humans in need. The same information would also
realize more realistic virtual humans [1], [2] by providing a
more detailed internal state.
A body of previous research on motion analysis and simu-
lation of a musculoskeletal human model has been conducted
in the field of sports science and medicine [3], [4]. Their main
purposes are to use the human somatosensory data to improve
motion patterns of athletes, and to monitor the recovery of those
who are in rehabilitation of mobility. Most of them adopt either
a simplified model of the whole body or a detailed model lim-
ited to a small part of the body, due to the high computational
cost of detailed whole-body human models. The computation
of the whole-body somatosensory information was investigated
in [5]–[7], although the efficiency of computational algorithms
was not paid much attention.
The authors developed an efficient algorithm for forward-dy-
namics computation of structure-varying kinematic chains to
handle articulated multibody systems with the highest gen-
erality [8]. The algorithm was recently improved to deliver
NAKAMURA et al.: SOMATOSENSORY COMPUTATION FOR MAN–MACHINE INTERFACE
Fig. 1. Musculoskeletal human model.
order- efficiency with a single processor [9], and order-
efficiency with as many processors as order- [10], where
is the number of bodies. A near-realtime optical motion-cap-
ture system was also developed by Kurihara et al. [11] in the
authors’ group, and will be used to capture the target behaviors.
We constructed the musculoskeletal human model so that the
dynamics-computation algorithms for multibody systems can
be applied in a straightforward way, as we explain in the fol-
lowing section in detail. Each of the muscles, tendons, and lig-
aments is represented by a single wire which has two ends and
any number of via-points and generates a tension. Muscles can
also actively change their length. We assume that the tension is
uniform throughout the wire, and that the via-points change only
the direction of the wire without friction. Cartilages are modeled
as linear springs.
The forward/inverse-dynamics computation of muscu-
loskeletal human models using motion-capture data allows
us to compute the somatosensory information. Applications
of this technology include sports science, rehabilitation, and
medicine. Being integrated with realtime motion-capture tech-
niques and efficient algorithms, the technology would also find
applications in the higher dimensional man–machine interface
and serve as the foundation of cognitive-level communication
between humans and machines.
II. MUSCULOSKELETAL HUMAN MODEL
A. Elements and Attributes
We designed the detailed human model shown in Fig. 1. It
is comprised of the skeleton and the musculotendon network.
The skeleton is a set of bones grouped into a suitable number
of body parts. The musculotendon network is a set of muscles,
tendons, and ligaments spreading and straining among bones.
Each attribute of these elements is modeled as follows.
1) Bone: Rigid-link element with mass and inertia.
2) Muscle: Active constrictive wire actuator.
Authorized licensed use limited to: Universidade Tecnologica Federal do Parana. Downloaded on June 23,2021 at 20:26:49 UTC from IEEE Xplore. Restrictions apply.
59
Fig. 2. Model of a simple muscle.
3) Tendon: Passive constrictive wire coupled with a
muscle to transmit its power.
4) Ligament: Passive constrictive wire that connects mul-
tiple bones to constrain their relative movement.
5) Cartilage: Passive linear spring with zero nominal
length that connects multiple bones and constrains their
relative movement.
B. Skeleton Model
We used polygon geometric data of the bones commer-
cially available for computer graphics [14]. We also produced
another set of polygon data by reconstructing the geometry
from computer tomography (CT) cross-sections of a human
skeleton model. Both models are based on the standard body of
European male.
In the beginning, for an experimentation of the whole-body
dynamics, we grouped the bones into a smaller number of rigid
links for simplification. For example, each of the head and hands
is counted as one rigid link. A foot is grouped into two bodies,
the toe and the rest. Thus, most of the facial muscles and short
musculotendons on the hands and feet are omitted. We applied
this simplification and grouped about 200 bones into 53 rigid
links.
C. Musculotendon Model
There are various types of muscles, tendons, and ligaments.
For example, there are large muscles like triangular muscles,
long muscles winding bones, and muscles with furcation, like
biceps. However, we can classify them into the following pat-
terns:
1) one to be replaced by a simple wire that connects an
origin and an end;
2) one to be replaced by a wire that has an origin, via-
points, and an end;
3) one to be replaced by several wires forming a fork at a
virtual bone;
4) one to be replaced by a set of simple wires;
5) one to be replaced by a combination of (1)–(4).
We describe the detail of the above patterns with typical ex-
amples in the following paragraphs.
1) Simple wire model: Most of the muscles, tendons, and
ligaments are modeled as a wire which has only an
origin and an end. Fig. 2 is the model of M. Teres Minor,
which ends at Humerus.
60 IEEE TRANSACTIONS ON ROBOTICS, VOL. 21, NO. 1, FEBRUARY 2005

Fig. 3. Muscle with via-points.

Fig. 5. Model of M. Biceps Brachii.

Fig. 4. Two models of a muscle with furcation.

2) Wire model with via-points: Some muscles, tendons,

and ligaments wreathe around bones and cannot be
modeled by simple wires connecting two points. We Fig. 6. Model of M. Pectoralis Major.
represent a curved path of a muscle, tendon, or liga-
ment by segmented lines. Namely, we place via-points
if a wire wreathes around a bone or if the tendons are
sheaved by a synovial sheath. A path of segmented
lines has an origin, an end, and any number of cor-
ners in the middle, called via-points. We assume that
each via-point is fixed to a bone, at which the muscle,
tendon, or ligament can slide without friction. Also
assumed is that a muscle, tendon, or ligament has a
uniform magnitude of tension throughout it, and only
changes the direction of tension at the via-points. An
example is shown in Fig. 3.
3) Wires connected to virtual links: Muscles, tendons, Fig. 7. Mechanism of elbow joint.
and ligaments are not always connected in series, but
may have furcations. Fig. 4 shows two simple cases
with furcations. Although the two models are equiva- wires. A typical example is M. Pectoralis Major, shown
lent, the right-hand case adopts the concept of virtual in Fig. 6.
link, which allows more flexible representation of the 5) Combined model: Some complex parts require a com-
muscle, tendon, and ligament networks. A virtual link bination of the above-mentioned models. Fig. 7 is the
is modeled as a small link without mass. example of the ligaments on the elbow. The anular lig-
Fig. 5 is the model of M. Biceps Brachii, which has a ament of the radius forms a ring, allowing indepen-
bifurcation and a via-point. Two muscles are connected dent pronation and supination of the radius while con-
to a tendon at the end of M. Biceps Brachii. The tendon straining the radius and ulna to interlock at flexion and
separates into two branches attached to two different extension of the elbow. We use a virtual link and a
bones. Therefore, the furcation model of the tendon is number of via-points to model this function, as shown
essential to completely model the function of M. Biceps in Fig. 8.
Brachii. Virtual links are needed to model the furcations
under the fundamental rule that a wire can have only one
pair of origin and end points. D. Musculoskeletal Model
4) Muscle model comprised of multiple wires: We mod- Consulting an anatomy text [15], we built the musculotendon
eled some wide-spanned muscles using several simple network model. The model covers most of the parts of the mus-

Authorized licensed use limited to: Universidade Tecnologica Federal do Parana. Downloaded on June 23,2021 at 20:26:49 UTC from IEEE Xplore. Restrictions apply.
NAKAMURA et al.: SOMATOSENSORY COMPUTATION FOR MAN–MACHINE INTERFACE 61

forward-dynamics algorithm simulates the whole-body motion

from the given wire tensions of the muscles.
The computational algorithm to be presented in this section is
based on the method proposed in [8], which was originally de-
veloped for general multibody systems, including closed kine-
matic chains, and even for those changing the structure in time.
The forward dynamics computation of musculoskeletal models
includes the following three steps.
Step 1) Compute , the Jacobian matrix of the
wire and spring lengths with respect to the general-
ized coordinates, defined as

Fig. 8. Model of ligaments on elbow joint. (1)

TABLE I where is the length of the wires and springs,

COMPLEXITY OF MUSCULOSKELETAL MODEL. THE 53 BONE GROUPS ARE is the generalized coordinate, and and
CONNECTED BY TWO 6-DOF BONES (HIP AND CHEST), TWO 1-DOF BONES are the number of wires and springs and DOF of
(TOES), TWO 0-DOF BONES (KNEECAPS), AND 47 3-DOF BONES (THE
OTHERS). EACH VIRTUAL LINK HAS 6 DOFs the model, respectively. and
in our model.
Step 2) Map the given wire tensions into the generalized
force using the Jacobian matrix obtained above.
Step 3) Compute the accelerations of the generalized coor-
dinates from the generalized force using the method
in [10] and integrate them to obtain the whole-body
motion.
For the definition of the generalized coordinates and the
generalized force, and their determination for complex con-
strained multibody systems, please refer to [8]. The following
culotendon networks except for those in the head, hands, and paragraphs give more detailed descriptions of the procedures.
feet. Step 1) Computation of the Jacobian matrix: In order to
The complexity of the model is summarized in Table I. The compute the Jacobian matrix of (1), we first consider
200 bones are divided into 53 bone groups, most of which are , the Jacobian matrix of the length of the th wire
connected to the neighboring bones by 3-degree-of-freedom with respect to the generalized coordinates. The ma-
(DOF) spherical joints. The exceptions are to cover complex trix satisfies the following relationship:
mobility due to constraints such as surface contacts and closed
(2)
loops. The skeletal system has 155 DOFs. In addition, we
introduce 28 virtual links, which have DOFs in where is the length of the th wire.
total. The DOF of the kinematic chain is, therefore, 323. We We suppose that the th wire is comprised of
used 366, 56, 91, and 34 wires to model the muscles, tendons, points, including the via-points and the
ligaments, and cartilages, respectively. Accordingly, we have origin and end points. Let the length between the
547 wires, in total. th and th via-points be . Also let the Ja-
Our model should be able to approximate the function of most cobian matrix of with respect to the generalized
types of muscles and tendons. However, we do not consider the coordinates be . Then can be represented
following phenomena that might occur in the real human body: by the sum of as follows:
• realtime addition or removal of via-points: moving the
joints may introduce new via-points due to new contacts (3)
between a muscle and bone, or eliminate a via-point by
taking off a muscle from a bone; Using , the position of the th via-point of the th
• interwire contacts: we do not model the contact forces be- wire, can be described as follows:
tween multiple wires, although we do consider those be-
tween wires and bones by using the via-points. (4)
Differentiating (4) with respect to the generalized
III. DYNAMICS COMPUTATION OF MUSCULOSKELETAL MODEL coordinates yields the following equation:
A. Forward Dynamics
Forward dynamics is the computation to estimate the mo-
tion, namely accelerations, of the generalized coordinates for
(5)
the given generalized forces. In our musculoskeletal model, the

Authorized licensed use limited to: Universidade Tecnologica Federal do Parana. Downloaded on June 23,2021 at 20:26:49 UTC from IEEE Xplore. Restrictions apply.
62
(6)
where denotes the Jacobian
matrix of with respect to the generalized co-
ordinates, and can be calculated using well-known
methods such as [16]. We first compute by (3)
and (6), and then obtain by copying to the th
row of .
Step 2) Mapping wire tensions into joint torques: We com-
pute the joint torques from the given
wire and spring tensions of muscles, tendons, liga-
ments, and cartilages. The mapping is linear and rep-
resented by the following equation:
(7)
Step 3) Solving for accelerations: Without losing the gener-
ality of [8], we improved the efficiency of computa-
tion in [9] and [10]. We use these algorithms to solve
for the accelerations of the generalized coordinates.
For an articulated multibody system consisting of
bodies, the computational complexity of the algo-
rithm is . If the algorithm is implemented on
a parallel processing system with processors,
the computational time is reduced to . The
same algorithm is used for parallel computation on
any number of processors for higher efficiency, and
even for serial computation on a single processor, as
well, without modification.
B. Inverse Dynamics With Force Distribution
Inverse dynamics computes the tensile strength of the mus-
cles, tendons, and ligaments and spring forces of the cartilages
that realizes the motion obtained through, for example, motion
capturing. The computation requires the following two steps.
Step 1) Apply the standard Newton–Euler inverse dynamics
computation to the given motion data, and compute
the joint torques. We can treat the mechanism as
a tree-structure kinematic chain with a fixed base,
assuming that the chain is cut open if it has closed
loops, and also assuming that free joints are actively
driven. This concept originates from the basic idea
of [8].
Step 2) Map the joint torques to the wire tensions . For this
computation, we first determine the generalized co-
ordinates that may vary in time due to the change
of constraints. We use , the Jacobian matrix of the
wires with respect to the generalized coordinates,
and , the Jacobian matrix of all the joints of the
serial tree-structure kinematic chain with respect to
the generalized coordinates for the mapping [8]. The
mapping problem is not straightforward because the
number of the wires are much greater than the DOFs
of the whole system. We use mathematical program-
ming methods to solve the problem with redundancy.
In the following sections, we first formulate the optimization
problem for computing the muscle tensions, and then provide
two methods for solving the problem.
Authorized licensed use limited to: Universidade Tecnologica Federal do Parana. Downloaded on June 23,2021 at 20:26:49 UTC from IEEE Xplore. Restrictions apply.
IEEE TRANSACTIONS ON ROBOTICS, VOL. 21, NO. 1, FEBRUARY 2005
C. Optimization of the Tensions
The relationship between the generalized forces and the
wire tensions is represented by (7). On the other hand, the
relationship between the joint torques and the generalized
forces is provided by the following equation:
(8)
The generalized forces are computed by simply substituting
the joint torques computed in Step 1 into the above equation.
The remaining problem is, therefore, to solve (7) for using
the generalized force . Since the number of wires (547) is
greater than the total DOFs (323), which is even greater than
the DOFs of the skeleton system (155), computing from
is a highly redundant problem.
The wire tensions of the muscles, tendons, and ligaments only
work in a contractive direction, while spring tensions of the car-
tilages work in both contractive and expansive directions. The
characteristics are represented by the following constraint con-
dition:
(9)
where is a matrix which extracts the tensions of muscles,
tendons, and ligaments from . It consists of as many unit row
vectors as the total number of the muscles, tendons, and liga-
ments. Each row vector has one at the column corresponding to
a muscle, tendon, or ligament.
Technically, solving (7) is a problem of actuation redundancy
[12] subject to an equality condition of (7) and an inequality
condition of (9).
The goal of the inverse-dynamics computation of muscu-
loskeletal human models is to estimate the activity of muscles.
To meet this goal, we have to carefully set the performance
index for the actuation redundancy problem. We will also have
to investigate how humans use the redundancy and formulate
the performance index to incorporate the knowledge.
In order to proceed with the discussion of inverse dynamics
computation, we now assume that such knowledge is repre-
sented in the following form of nominal use of muscles:
(10)
where is the muscle tensions derived from biology-based
methods, is the joint variable, and represents the time.
may be obtained from electromyographic (EMG) data measured
simultaneously with the motion data, or from a database de-
scribing how humans use the actuation redundancy during a
particular task. Vector includes all the sensory information re-
quired to compute , such as EMG data and contact forces. The
goal is to find that satisfies the dynamics (7) and minimizes
the difference from . Our problem can now be represented by
the following expression.
Find that minimizes
(11)
subject to the following constraints:
(12)
(13)
NAKAMURA et al.: SOMATOSENSORY COMPUTATION FOR MAN–MACHINE INTERFACE 63

1) Solving With Quadratic Programming: Solving

(11)–(13) forms a quadratic programming (QP) problem,
which is commonly represented as the following optimization.
Find an extreme value of the objective function

(14)
under linear equality and inequality constraints for . If the ma-
trix is symmetric positive semidefinite, the problem is called
a convex QP problem, for which efficient algorithms are known.
In our case, is an identity matrix from (11) and, therefore, pos-
itive definite. We may also introduce the weighting ratios among
the components of as long as the matrix remains positive defi-
nite. Note that in (11). Also note that the third
term in (11), , is independent to and, therefore, can
be disregarded for optimization.
2) Solving With Linear Programming: Although many effi-
cient algorithms are known for QP, they are still computation-
ally expensive for our large problem. For applications where the Fig. 9. Example of forward dynamics.
computational efficiency is more important than the accuracy,
we can also apply linear programming (LP) as an approximating of , because LP does not allow square root in the cost func-
solver of the original problem. tion.
By defining We can apply LP to the problem because the constraint con-
(15) ditions of (21)–(24) and the performance index of (20) are all
linear. The simplex method is known as an efficient algorithm
(16)
for LP.
we can equivalently reformulate the problem of (11)–(13) as
follows. IV. EXAMPLES OF COMPUTATION
Find that minimizes
A. Forward Dynamics
(17) The forward dynamics computes the motion of the whole
subject to the following constraints: body for a given set of muscle tensions. The tendons, ligaments,
and cartilages can be modeled by hard springs and dampers.
(18) However, it was not easy to design a practical controller to com-
pute the muscle tensions that produce a relevant example motion
by the forward-dynamics computation.
(19) The example we adopted here is for a preliminary test which
simulates a free motion with zero muscle forces under the
The above formulation is still a QP problem. We simplify it to gravity while the head position is constrained. Fig. 9 shows
form a LP problem by introducing the variable vector several snapshots from the simulation, where the whole body
as follows. gradually stretches down due to the gravity. The simulation was
For constant vector with positive components, find and executed on a Pentium IV 2-GHz machine, and the computation
that minimize time was 0.5 s for each step.
It is also possible to apply the muscle tensions obtained by the
(20)
inverse-dynamics computation, but it would not yield the same
subject to the following constraints: motion as the reference because of the following two reasons.
• As shown in the results of inverse-dynamics computation,
(21)
we cannot compute the muscle tensions exactly equivalent
to the joint torques required to perform the reference mo-
tion, due to the limitation of our musculoskeletal model.
(22) • Even with a complete model, discrete-time integration and
collisions/contacts will also cause the discrepancy.
It is, therefore, essential to include a robust controller to control
(23) the model in the virtual world.
(24)
If all components of are one, (20) approximates (17), while B. Inverse Dynamics
with uneven positive components approximates the quadratic In this section, we show the results of inverse-dynamics com-
performance index with a weighted norm. We apply (20) instead putation. In inverse dynamics, as we saw in the previous section,

Authorized licensed use limited to: Universidade Tecnologica Federal do Parana. Downloaded on June 23,2021 at 20:26:49 UTC from IEEE Xplore. Restrictions apply.
64 IEEE TRANSACTIONS ON ROBOTICS, VOL. 21, NO. 1, FEBRUARY 2005

determination of is the fundamental and essential problem.

It requires extensive experimental research to build a database
of how the human uses the muscles. Such a database will be ex-
tremely important for further research of somatosensory com-
putation. We plan to continue the research and carry out experi-
ments for accumulating the data in our future work. In this par-
ticular paper, we assume and apply LP and QP to focus
on the computational problems.
We captured a kicking motion using the motion-capture
system developed in our group [11]. The data contain 243
frames captured every 30 ms for 7.29 s.
The skeletal system has 155 DOFs and 323 DOFs, including
the virtual links, driven by the 366 muscle wires. Through com-
putation of the example, however, we found that the driving in-
puts are not redundant, and not even sufficient for producing Fig. 10. Comparison of computational time between LP and QP for inverse
arbitrary whole-body motions. In fact, there was no solution dynamics.
of for the generalized force computed from the captured
kicking motion. This was particularly prominent for the joint
torques of the vertebras in the neck, which has seven 3-DOF
spherical joints, but not so many muscle wires. The results sug-
gest that the forces acting on the whole body cannot be approx-
imated only by the simple wire model of major muscles. In the
real human body, the tissue and organs, which are not included
in our model, passively generate the distributed forces and sup-
port the skeletal system. It would be necessary to build a more
precise model taking account of the effects of the tissue and or-
gans.
To avoid the essential problem in the current computation,
we introduce another variable vector and replace
the equality constraint of (21) with an inequality constraint as
follows.
For constant vectors and with positive components, find
, , and that minimize Fig. 11. Comparison of  j 0 J fj between LP and QP for inverse
dynamics.
(25)
subject to the following constraints: subject to the following constraints:
(26) (32)
(27)
where indicates the weighting parameter similar to .
(28) Note that the above QP problem has 547 variables to optimize,
(29) which is the dimension of , while the LP problem of (25)–(30)
has 1417 variables of , , and .
(30)
For the computation of convex QP problems, we used the
This replacement is conducted to relax the problem and to primal-dual interior-point algorithm routines of the free ob-
ensure the existence of a solution by allowing an error which ject-oriented software package for solving convex QP problem
represents the permissible range of error in the joint torques. (OOQP [17]). For the computation of LP problems, we used the
The permissible error is intuitively tuned by setting the ratio of simplex method routines of the GNU free LP library (GLPK
the magnitudes of and . In our implementation, each of [18]). All the computation was executed on a PC with a single
and was set to have uniform components, and we simply Pentium IV 2-GHz processor.
select the ratio of their magnitudes . Note that The results of computation are summarized by Figs. 10–12.
this formulation is effective even if the model is complete and We selected the weighting parameters as and
an analytically exact solution exists, because the equation may . Since the performance indexes have the dimension
lose the solution due to singular configurations. of in QP and in LP, we cannot simply compare
For comparison, we also reformulated the QP problem of the two weighting parameters.
(11)–(13) as follows. Fig. 10 shows that LP takes 2.98 s for each frame, on average,
Find that minimizes while QP takes 14.2 s. LP is 4.77 times faster in computation
than QP, in spite of the fact that LP has 2.59 times more variables
(31) than QP.

Authorized licensed use limited to: Universidade Tecnologica Federal do Parana. Downloaded on June 23,2021 at 20:26:49 UTC from IEEE Xplore. Restrictions apply.
NAKAMURA et al.: SOMATOSENSORY COMPUTATION FOR MAN–MACHINE INTERFACE
Fig. 12. Comparison of jf j between LP and QP for inverse dynamics.
Fig. 13. Example of inverse dynamics.
Fig. 11 represents , the squared magnitude of the
error of constraints. The error becomes very large for LP around
frames 50 and 110, the latter of which corresponds to the end of
the kick. In general, we can reduce the magnitude of the error by
using larger or . In this experiment, however, increase
of could not reduce the error of LP, while increase of
could reduce that of QP. This difference would be intuitively
explained by the fact that LP assumes that the optimal solution
is obtained at one of the vertices of the space formed by the
inequality conditions.
Fig. 12 shows for both LP and QP. Since we assumed
, the two curves are equivalent to the squared magnitudes
of . Although increase of tended to reduce the error of
equality constraints as mentioned above, it admitted the increase
of . The large fluctuation of the QP curve is due to the relatively
smaller weight for . In other words, the fluctuation can
be reduced at the cost of the error of equality constraints.
In this particular example, the comparison of the computa-
tional time and the two graphs suggest that the LP problem de-
scribed in (25)–(30) is an efficient formulation of the inverse-dy-
namics problem in comparison with the QP problem.
The result of LP computation is visualized in Fig. 13, where
the colors of muscle wires change from yellow to red (from
white to gray) as their tensions increase. The colors of other
passive wires for tendons and ligaments also change from light
blue to dark blue (from white to gray) as their tensions increase.
The animation of this motion is available at http://nobunaga.t.u-
tokyo.ac.jp/publications/archives/video/ along with other mo-
tions, including walk and squat.
Authorized licensed use limited to: Universidade Tecnologica Federal do Parana. Downloaded on June 23,2021 at 20:26:49 UTC from IEEE Xplore. Restrictions apply.
65
In the inverse-dynamics computation, we have determined
the muscular force by using mathematical optimization
methods. We have not yet started the physiological study to
make a database of how we take advantage of the redundancy
of our actuation system. The computational methods need to be
extended to integrate the physiological knowledge, because the
computed forces without such knowledge lack the consistency
with the actual human muscular forces. The contribution of this
paper is the formulation and implementation of a mathematical
method that would realize the somatosensory computation with
physiological knowledge.
V. CONCLUSION
We proposed a method for modeling the human mus-
culoskeletal dynamics, where the dynamics-computation
algorithms developed for kinematic chains are extended to
compute the muscle and tendon tensions. We constructed a
detailed human model based on this modeling method.
We developed computational algorithms for inverse and for-
ward dynamics of a wire-based human model on the basis of
efficient multibody dynamics-computation algorithms [9], [10].
The computational algorithms were successfully implemented
and used for numerical experiments to show their performance.
The computational problem of inverse dynamics is reduced
to an optimization problem. The problem was formulated in two
forms, namely, as a QP problem and as a LP problem. The two
formulations were compared from the viewpoint of computa-
tional time, accuracy, and stability of the solution. The results
presented in this paper suggested that the LP is more efficient,
accurate, and stable.
The optimization criteria needs to be specifically and appro-
priately chosen to simulate how the human uses the muscle re-
dundancy. We are currently searching for the criteria and mod-
eling it. Concurrent use of EMG at motion capturing would be
a promising approach.
The forward and inverse dynamics computations of the mus-
culoskeletal human model using motion-capture data would en-
able the computation of the somatosensory information. Appli-
cations of this technology include sports science, rehabilitation,
and medicine, as it was originally targeted. Integrated with re-
altime motion-capture techniques and efficient algorithms, the
technology would also find its applications in the higher dimen-
sional man–machine interface and serve as the foundation of
cognitive-level communication between humans and machines.
REFERENCES
[1] J. Gratch, J. Rickel, E. Andre, J. Cassell, E. Petajan, and N. I. Badler,
“Creating interactive virtual humans: Some assembly required,” IEEE
Intell. Syst. Mag., vol. 17, pp. 54–63, Apr. 2002.
[2] E. de Sevin, M. Kallmann, and D. Thalmann, “Toward real time virtual
human life simulations,” Comput. Graph. Int., pp. 31–37, 2001.
[3] M. Kaya, H. Minamitani, K. Hase, and N. Yamazaki, “Motion analysis
of optimal rowing form by using biomechanical model,” Proc. IEEE
Engineering in Medicine and Biology Conf., pp. 715–716, 1995.
[4] T. Komura, Y. Shinagawa, and T. L. Kunii, “Creating and retargetting
motion by the musculoskeletal human body model,” Visual Computer,
vol. 16, no. 5, pp. 254–270, 2000.
[5] S. L. Delp and J. P. Loan, “A software system to develop and analyze
models of musculoskeletal structures,” Comput. Biol. Med., vol. 25, pp.
21–34, 1995.
66 IEEE TRANSACTIONS ON ROBOTICS, VOL. 21, NO. 1, FEBRUARY 2005

[6] , “A computational framework for simulating and analyzing human Katsu Yamane (M’00) received the B.S., M.S., and
and animal movement,” IEEE Comput. Sci. Eng. Mag., vol. 2, pp. 46–55, Ph.D. degrees in mechanical engineering from Uni-
Jan. 2000. versity of Tokyo, Tokyo, Japan, in 1997, 1999, and
[7] N. Suzuki and A. Takatsu, “3D and 4D visualization of morpholog- 2002, respectively.
ical and functional information from the human body using noninvasive He was a Research Fellow of the Japan Society
measurement data,” Atlas of Visualization, vol. 3, pp. 213–224, 1997. for the Promotion of Science from 2000 to 2002,
[8] Y. Nakamura and K. Yamane, “Dynamics computation of struc- and a Postdoctoral Fellow at the Robotics Institute,
ture-varying kinematic chains and its application to human figures,” Carnegie Mellon University, Pittsburgh, PA, from
IEEE Trans. Robot. Automat., vol. 16, pp. 124–134, Apr. 2000. 2002 to 2003. Since 2003, he has been an Assistant
[9] K. Yamane and Y. Nakamura, “O(N) forward dynamics computation of Professor with the Department of Mechano-Infor-
open kinematic chains based on the principle of virtual work,” in Proc. matics, University of Tokyo. His research interests
IEEE Int. Conf. Robotics, Automation, vol. 3, 2001, pp. 2824–2831. include dynamics simulation of human figures, control of humanoid robots,
[10] , “Efficient parallel dynamics computation of human figures,” in and physically-based animation in computer graphics.
Proc. IEEE Int. Conf. Robotics, Automation, vol. 1, 2002, pp. 530–537. Dr. Yamane received the King-Sun Fu Memorial Best Transactions Paper
[11] K. Kurihara, S. Hoshino, K. Yamane, and Y. Nakamura, “Optical mo- Award and an Early Academic Career Award from the IEEE Robotics and Au-
tion capture system with pan-tilt camera tracking and realtime data pro- tomation Society in 2001 and 2004, respectively. He is a member of the Robotics
cessing,” in Proc. IEEE Int. Conf. Robotics, Automation, vol. 2, 2002, Society of Japan and ACM SIGGRAPH.
pp. 1241–1248.
[12] Y. Nakamura, Advanced Robotics: Redundancy and Optimiza-
tion. Reading, MA: Addison-Wesley, 1991.
[13] F. D. Carlson and D. R. Wilkie, Muscle Physiology. Englewood Cliffs,
NJ: Prentice-Hall, 1974.
[14] (2002) The Viewpoint Medical Products: Viewpoint Corporation. [On-
line]. Available: http://www. viewpoint.com/
[15] W. Kahle, H. Leonhardt, and W. Platzer, Taschenatlas der Anatomie,
Band 1: Bewegungsapparat. Stuttgart, Germany/New York: Verlag,
1986.
[16] D. E. Orin and W. W. Schrader, “Efficient computation of the Jacobian
for robot manipulators,” Int. J. Robot. Res., vol. 3, no. 4, pp. 66–75, 1984.
[17] Object-oriented software for quadratic programming (OOQP). [Online].
Available: http://www.cs.wisc.edu/~swright/ooqp/ Yusuke Fujita received the B.S degree in mechanical
[18] Free Software Foundation. GNU linear programming kit (GLPK). [On- engineering and the M.S. degree in information sci-
line]. Available: http://www.gnu.org/soft-ware/glpk/glpk.M.html ence and technology from the University of Tokyo,
Tokyo, Japan, in 2002 and 2004, respectively.
His research interests include motion planning and
dynamics computation of human figures.
Yoshihiko Nakamura (M’87) received the B.S., Mr. Fujita is a member of the Robotics Society of
M.S., and Ph.D. degrees in precision engineering Japan.
from Kyoto University, Kyoto, Japan, in 1977, 1978,
and 1985, respectively.
He was an Assistant Professor at the Automation
Research Laboratory, Kyoto University, from 1982
to 1987. He joined the Department of Mechanical
and Environmental Engineering, University of
California, Santa Barbara, in 1987 as an Assistant
Professor, and became an Associate Professor in
1990. Since 1991, he has been with Department of
Mechano-Informatics, University of Tokyo, Tokyo, Japan, and is currently a
Professor. His fields of research include redundancy in robotic mechanisms,
nonholonomy of robotic mechanisms, kinematics and dynamics algorithms for
computer graphics, nonlinear dynamics and brain-like information processing,
and robotic systems for medical applications. He was the principal investigator
of “Brain-like Information Processing for Humanoid Robots” under the CREST
project of the Japan Science and Technology Corporation. He is the author of
the textbook, Advanced Robotics: Redundancy and Optimization (Reading, Ichiro Suzuki received the B.S. and M.S. degrees
MA: Addison-Wesley, 1991). in mechanical engineering from the University of
Dr. Nakamura received excellent paper awards from the Society of Instru- Tokyo, Tokyo, Japan, in 2001 and 2003, respectively.
ment and Control Engineers (SICE) in 1985 and from the Robotics Society of He is now with Nintendo Corporation, Kyoto,
Japan in 1996 and 2000. He was also a recipient of the King-Sun Fu Memo- Japan.
rial Best Transactions Paper Award, IEEE TRANSACTIONS ON ROBOTICS AND Mr. Suzuki received a Research Promotion Prize
AUTOMATION, in 2001 and 2002. He is a member of the ASME, the SICE, the from the Robotics Society of Japan in 2003. He is a
Robotics Society of Japan, the Japan Society of Mechanical Engineers, the In- member of the Robotics Society of Japan.
stitute of Systems, Control, and Information Engineers, and the Japan Society
of Computer-Aided Surgery.

Authorized licensed use limited to: Universidade Tecnologica Federal do Parana. Downloaded on June 23,2021 at 20:26:49 UTC from IEEE Xplore. Restrictions apply.