Emotional Engineering Volume 4

Shuichi Fukuda Editor
Emotional
Engineering
Volume 4
Emotional Engineering Volume 4
Shuichi Fukuda
Editor
Emotional Engineering
Volume 4
123
Editor
Shuichi Fukuda
Keio University
Yokohama
Japan
ISBN 978-3-319-29432-2 ISBN 978-3-319-29433-9 (eBook)

DOI 10.1007/978-3-319-29433-9
Library of Congress Control Number: 2012472706
© Springer International Publishing Switzerland 2016

This work is subject to copyright. All rights are reserved by the Publisher, whether the whole or part
of the material is concerned, specifically the rights of translation, reprinting, reuse of illustrations,
recitation, broadcasting, reproduction on microfilms or in any other physical way, and transmission
or information storage and retrieval, electronic adaptation, computer software, or by similar or dissimilar
methodology now known or hereafter developed.
The use of general descriptive names, registered names, trademarks, service marks, etc. in this
publication does not imply, even in the absence of a specific statement, that such names are exempt from
the relevant protective laws and regulations and therefore free for general use.
The publisher, the authors and the editors are safe to assume that the advice and information in this
book are believed to be true and accurate at the date of publication. Neither the publisher nor the
authors or the editors give a warranty, express or implied, with respect to the material contained herein or
for any errors or omissions that may have been made.
Printed on acid-free paper
This Springer imprint is published by Springer Nature

The registered company is Springer International Publishing AG Switzerland
Preface
The progress of network technology is amazing. In addition to humans, things are

now being connected, as Internet of Things (IoT), Cyber Physical Systems,
Industrie 4.0, etc., indicate. When IoT was proposed, the sensor technology was not
mature enough to realize its goal. But with its remarkable progress, now everything
is quickly getting connected.
It contributed greatly to flexible and adaptive production, but what is more
important is it will change engineering from extrinsic to intrinsic. Traditional
engineering has been product-focused. Engineers believed if they produced a good
quality product, their customers would be happy and satisfied. They regarded
customers just as passive consumers. But why are users called customers? That is
because they are very active and would like to customize their products. They have
intrinsic motivations to actualize themselves.
The concept of User Experience (UX) opened engineers’ eyes. Until this concept
is proposed, engineers paid attention only to products. They considered processes
just in terms of efficiency or cost reduction. UX demonstrated processes also yield
values. And more often, process values are more important than product values,
because processes are more deeply related to intrinsic motivation.
One important aspect of motivation is learning to grow. Traditional hardware
products do not grow, although they function very well. Motivation and emotion
originate from the same Latin word “movere = move.” Customers are full of rich
intrinsic motivations, and they would like to take actions to achieve their goals. And
when they overcome the difficulties, they feel the sense of achievement and ful-
fillment, which brings the greatest emotional satisfaction. Once their challenges are
successful, they would like to go one step further toward a little more difficult
challenge.
Traditional engineering looks at production from producer’s viewpoint, i.e.,
from technology, and they made tremendous efforts to provide the richest experi-
ence with their sophisticated products. But these experiences are extrinsic. What
customers really want is to realize their intrinsic motivations. They would like to
challenge for making their dreams come true.
v
vi Preface
Biologists point out that only human can think about the future. Animals can use
tools, but these tools are picked up from nature and they use them just for the
immediate purpose. Humans spend a great amount of time and efforts in making
tools to realize their dreams. That is why humans are called Homo Faber. But
traditional engineering is still shortsighted. Engineering which sees the far distant
future is strongly called for.
In other words, traditional engineering has been producing products, and their
focus was what and how: What products they should develop and how they can
produce them better? Engineers believed if what and how are selected appropri-
ately, they could offer happy experience to their users.
But what customers really would like producers to take into consideration is
why: Why do they want such a product? They expect producers to create a new
experience, which is intrinsically motived.
The quickly progressing network and such emerging technologies as additive
manufacturing, 3D printing, etc., have a great potential to respond to such desires of
customers. The current stage of networking is focused on the current framework of
society and industry, but at the next stage, it will become very much creative. It will
change its structure very flexibly and adaptably to respond to the personal intrinsic
need or desire of each customer.
Thus, engineering is quickly moving from what and how to why. And to develop
why engineering, we have to study more about motivation, because it is very much
multifaceted and it varies widely from case to case.
The chapters in this book cover a wide range of topics. The editor hopes the
reader will find clues in these chapters as to how they can develop such intrinsic
motivation-focused engineering. These chapters are very rich in themselves, but it is
hoped that by connecting and integrating them with focus on intrinsic motivation,
the reader will find a guiding principle.
Finally, I would like to thank all authors from the very bottom of my heart for
contributing such excellent chapters, and I would also like to thank Mr. Anthony
Doyle, Ms. Janet Sterritt, Mr. Balaji Sundarrajan and Ms. Swetha Sethuraman at
Springer.
Shuichi Fukuda
Contents
1 Age of Smart Orchestration . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1

Shuichi Fukuda
2 Composing a Product Network for Emotional Performance . . . . . . 23
Shuichi Fukuda
3 Humor Engineering in Smart Environments . . . . . . . . . . . . . . . . . 37
Anton Nijholt
4 Engineering for Human Pleasure . . . . . . . . . . . . . . . . . . . . . . . . . 59
Masashi Okubo
5 Aesthetic of Prosthetic Devices: From Medical Equipment
to a Work of Design. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 73
Stefania Sansoni, Leslie Speer, Andrew Wodehouse and Arjan Buis
6 Exploration of Users’ Cross-Cultural Differences
by Using the Kansei Engineering Approach . . . . . . . . . . . . . . . . . 93
Vanja Čok and Jože Duhovnik
7 The Kansei Design Approach at Toyota Motor Europe . . . . . . . . . 119
Carole Favart, Daniel Esquivel Elizondo, Dr. Alexandre Gentner
and Théo Mahut
8 Five Senses Theater: A Multisensory Display for the Bodily
Ultra-Reality . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 145
Yasushi Ikei, Koichi Hirota, Tomohiro Amemiya
and Michiteru Kitazaki
9 Making Emotion and Decision via Affecting Self-Perception. . . . . . 165
Sho Sakurai, Takuji Narumi, Tomohiro Tanikawa
and Michitaka Hirose
10 Neural Basis of Maternal Love as a Vital Human Emotion . . . . . . 189
Yoshiaki Kikuchi and Madoka Noriuchi
vii
viii Contents
11 Expectation Effect Theory and Its Modeling . . . . . . . . . . . . . . . . . 199

Hideyoshi Yanagisawa
12 Ma: Exploration of the Method for Grasping Intangible
Emptiness . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 213
Masato Hirahatake, Nobuyuki Kobayashi and Maeno Takashi
Chapter 1
Age of Smart Orchestration
Shuichi Fukuda
Abstract We, engineers, have been devoting tremendous efforts to achieve

high-quality functions. But our primary focus has been on an individual product.
Yesterday, changes were small and smooth so that they were predictable. We could
foresee the operating conditions so that we could develop functions that worked
well. But today, changes take place so frequently and extensively. The difference
between the changes of yesterday and today is their differentiability. Today’s
changes are sharp and angular so they are not differentiable. Thus, we cannot make
predictions. As the outer world changes unpredictably, we have to proceed by trial
and error. Thus, not only goals (products) are important, but how we can get to the
goal, i.e., processes become increasingly important. Further, as products are used no
more by themselves alone and their operating conditions are getting more and more
networked, performance does not mean just functions of a final product anymore.
We have to consider it through their product life cycle, i.e., life cycle of a group of
products connected in operation. This chapter points out what an important role
emotion plays in today connected world.
1.1 Why Humans Make Tools and Are Called Homo

Faber?
Biologists say that only human can see the future. That is why human makes a tool.
One of the definitions of human is Homo Faber. Human makes a tool. It is because
as humans can see the future, they can have dreams. They make tools to realize their
dreams. They would like to invent something that is not available in nature. That is
the core of engineering. Engineering is here to make our dreams come true. It is an
S. Fukuda (&)
System Design and Management, Keio University, 4-1-1, Hiyoshi, Koho-Ku, Yokohama
223-8526, Japan
e-mail: shufukuda@gmail.com
© Springer International Publishing Switzerland 2016 1

S. Fukuda (ed.), Emotional Engineering Volume 4,
DOI 10.1007/978-3-319-29433-9_1
2 S. Fukuda
activity of creation. Animals can use tools. But they can only select things out of
nature which can be utilized as tools. And they use them just to satisfy their
immediate needs. So what they are doing is nothing but selection.
1.2 Open World and Emotional Satisfaction
Our world is changing very quickly. It used to be a Closed World. There was a clear
boundary so that rational approaches were effective. Set theory holds, and as we
accumulate our experience, we can apply induction and structure them into
knowledge. Once a frame of knowledge is established, we can apply it deductively
to solve other problems. But our world is now expanding very rapidly and the
boundaries disappear. It becomes an Open World (Fig. 1.1).
As Herbert Simon pointed out [1], our rationalities are bounded. If the number of
variables is small, rational approaches are effective, but when it becomes very large
and diverse, such problems as combinatorial explosion, etc., come up and we cannot
apply rational approaches any more. Simon proposed satisficing (satisfy + suf-
fice = satisfy enough), which is none other than emotional satisfaction.
In fact, when it comes to global optimization and considers simulated annealing,
there is no guarantee that the result is the optimum. We repeat many trials and after
enough number of trials, we assume the best result is the optimum (Fig. 1.2).
In other words, we repeat the trials until we feel satisfied enough and we regard
the best result as the optimum. Thus, it also is related to emotional satisfaction.
Although Simon’s Bounded Rationality and Satisficing, and global optimization
Fig. 1.1 Closed world and

open world
Fig. 1.2 Global optimization

1 Age of Smart Orchestration 3
look at the problems from different perspectives, they are common in the sense that
both solve the problem with attention paid to emotional satisfaction.
Pragmatism is known as a philosophy to deal with the Open World. Its essence is
Shakespeare’s “All’s well that ends well.” We have to note that it says “it ends
well” and it does not say anything about “the result is optimum.” The result may or
may not be optimum, but the result is well or in other words is satisfactory enough.
Thus, pragmatism which originated in UK and became the philosophy representing
US shares its basic idea with Simon’s.
Interestingly enough, another economist John Maynard Keynes pointed out [2]
that economic agents make decisions rationally when it comes to short-term
expectations, but they rely on confidence when they are long-term. Keynes did not
use the word emotion. He used the word confidence, but confidence is one of the
emotions.
Charles Sanders Peirce who is also known as the father of pragmatism proposed
abduction. Its basic idea is to come up with a hypothesis or a model and apply it to
the current problem and if it works, then that is fine, if not, repeat the process until
satisfactory result is obtained. This also relates to emotional satisfaction. Although
he called it the third logic, it is utterly different from induction and deduction, which
are truly logical and rational. We should recall that Peirce lived in an age of
American frontier and the world was expanding from day to day and the boundaries
were quickly disappearing at that time. The world he lived in was typically an Open
World. And it also should be remembered that pragmatism originated in UK. She
was known as a seafaring country and dominated the Seven Seas. Thus, the worlds
of UK and US in this age were truly open.
Although much far later, Donald A. Schon published a book “The Reflective
Practitioners; How Professionals Think in Action” [3]. This book points out the
importance of trial and error and how we make decisions while we are in action. It
points out that we make decisions based on our past experience and emotional
confidence, as there is no logical or rational way in some of our professions such as
design, management, and medicine. We make decisions based on our knowledge
and experience but the problem space in these area are too large so that we have to
try them to see if they work or not and make a final decision. This is nothing other
than emotional decision making.
Shewhart proposed PDSA, plan–do–study–act cycle (Fig. 1.3). This approach is
fundamentally the same as abduction. Both come up with a hypothesis and apply it
to the problem and see if it works or not. If it works, then it is fine. Otherwise,
repeat the process until satisfactory result is obtained. Although Peirce called
abduction the third logic, this PDSA approach did not come from arguments about
logics, but it came from the design of experiments. Design of experiments (DOE) is
a systematic method to determine the relationship between factors affecting a
process and the output of that process. Although it is systematic, it is not rational or
logical at all. It is trial-and-error approach and is nothing other than what Schon
described as reflective practice or thinking in action.
4 S. Fukuda
Fig. 1.3 Shewhart cycle
All these approaches are basically pragmatic and solve problems, based on
emotional satisfaction. These facts demonstrate how emotion plays an important
role in decision making in an Open World.
1.3 Unpredictable Changes and Emotional Satisfaction
Another big change in our world is the fact that changes take place more frequently
and more extensively. But what is more important is that although there were also
changes yesterday, they were smooth and differentiable, so that we could predict the
future. But today, the changes are angular so that they are not differentiable. Thus,
we cannot predict the future anymore (Fig. 1.4).
If there is a clearly defined boundary, then we can control adaptively. Our
traditional adaptive control was developed with clearly defined boundary in mind.
Although the environment may be unstructured, if it does not change with time,
then by trial and error, we could reach our goal. This is the way a robot navigates
through unstructured environments. But today’s problems are different from these
Fig. 1.4 Changes of

yesterday and today
problems of yesterday. Today, environments or situations change with time and

what makes the problem very difficult is they change in non-predictable manner.
Thus, we have to cope with the non-predictable time-varying conditions.
This can be compared to a ship navigating through uncharted waters. Not only
navigation chart is unavailable, but at sea, the weather changes unexpectedly.
Today, the weather is very beautiful, but tomorrow, it may change to hurricane. So
situations change from moment to moment. Thus, we have no other choice than to
solve the problem by trial and error.
What is important in solving such a problem is how we learn from failures. If the
failure is a complete failure, then that is the end of our life. But if we survive, we
will learn what hypotheses or assumptions did not work. Or in other words, we have
to find out what piece of past experience we can apply to the current problem. This
is the problem of trust. We have to identify which pieces of our experience we can
apply with trust and confidence to solve the current problem. Again, this explains
why Peirce’s, Schon’s, and Shewhart’s ideas work today in an Open World.
1.4 Fast Adaptability and Trust
The word “fast adaptability” is now getting very popular these days. This is because
yesterday our constraints or situations changed, but their changes were smooth so
that we could predict their behaviors. But today changes are not smooth so that they
are not differentiable, therefore not predictable.
Yesterday, we could introduce adaptability easily because the outside world did
not change appreciably or changed in a predictable manner. But today the outside
world changes in an unpredictable manner so that not only rational approaches are
no more applicable in a straightforward manner, but we have to develop another
trial-and-error or pragmatic approach to cope with this situation. Adaptability is the
only one we should have focused yesterday, but today we have to consider how fast
we can adapt to the changes in addition. Yesterday, products were produced in mass
and the operating conditions and environment did not change appreciably, or if they
did, they changed in a predictable manner. But today their changes are so fast and
non-predictable, we have to develop another approach. This is why “Open xxx” is
getting wide attention these days. The word “open” in such usages mean in most
cases connected or networked.
As the world did not change appreciably or it changed in a predictable manner,
the goal could be set at the beginning and we did not have to change it so that how
fast and effectively we could reach our goal was our main interest. Thus, most of
our working framework at that time was a tree-structured. Everyone has his or her
position and role and he or she was expected to perform his or her role perfectly.
This is because a tree structure has only one output node and if an output does not
change, a tree structure is most effective. But the greatest disadvantage of a tree
structure is hard or not flexible to cope with the quick changes. Such fast adaptable
performance cannot be expected from a tree structure (Fig. 1.5).
6 S. Fukuda
Fig. 1.5 Tree
Fig. 1.6 Network
But if we change it to a network structure, any node in a network can be an

output node so that a really flexible and adaptable performance can be expected
(Fig. 1.6).
1.5 Convergent Engineering and Divergent Engineering
To describe this discussion in another way, our traditional engineering was con-
vergent. As the goal did not change appreciably, we tried to find out the best
applicable method, technology, etc. to solve the problem. In other words, we have
been focusing our attentions on incoming links to a node. This is convergent
engineering (Fig. 1.7).
Fig. 1.7 Convergent

engineering
But to cope with the changing situations, we have to find out what goals we can
reach with our current resources (knowledge, technology, etc.). This is nothing
other than exploration. When we have to explore the new world, we have nothing
other than current resources to rely on. Theodore Roosevelt said “Do what you can,
with what you have, where you are.” This is the spirit of exploration and this way of
thinking is needed to solve the problems which are facing us today. As the prob-
lems vary very frequently and very extensively, we have to explore the new ways to
solve them. We have to know where we can reach with our current resources. This
is divergent engineering. It focuses its attention on outgoing links (Fig. 1.8).
If the problem is simple, then one man may solve it this way. But the problems
facing us today is very much complex and complicated. This calls for team
working, i.e., more heads are better than one. We need to solve the problem by
cooperating with others who can work together with us. Everybody has his or her
own capabilities and the capabilities vary from person to person. Some have many
diverse capabilities. Others may not. The number and extent of their capabilities
vary from person to person. But if they can find a connecting link between them and
if they connect their links all together, we can constitute a network (Fig. 1.9).
Fig. 1.8 Divergent

engineering
8 S. Fukuda
Fig. 1.9 Divergent nodes
It should be stressed that a network is not hard like a tree, but very much soft.
This network varies adaptively with the change of the outer world. Thus, it is very
much situational adaptive, because such a network can be developed any time, any
place to appropriately cope with the changing environments and situations. It is an
open network. In fact, the word Open which we encounter in most cases today
implies such an open network or an adaptive network. This approach will enable
fast adaptability most effectively.
1.6 Needs for Creating Another New Market
In addition to these change of our world in terms of space and time, we have to note
that we are quickly approaching the saturation level in Gompertz curve (Fig. 1.10).
As Weber–Fechner law Eq. (1.1) teaches us, we can easily recognize the dif-
ference if the level is low. When we speak in a small voice, everybody will
recognize the voice is raised when we speak in a little louder voice. But when we
speak in a loud voice from the first, most people cannot recognize if the voice is
raised and becomes a little louder. As the loudness of voice increases, we have to
raise our voices in much greater amount.
D R=R ¼ Constant ð1:1Þ
where R is the level of stimulus
Fig. 1.10 Gompertz curve

The functions and quality of our products are now quickly approaching the
saturation level of Gompertz curve. When their qualities were very low, our cus-
tomers could easily recognize their improvements. But today it is very difficult to
recognize how much they are improved.
Thus, we need to explore and create a new market. We have to find a blue ocean
instead of fighting on the red ones [4]. If we stay on the same track and keep
developing or keep improving our products, our customers do not understand how
better they become and they do not appreciate engineers’ efforts. We have to get off
such beaten tracks and explore the new tracks in order to create a new market.
Again, this calls for divergent engineering. Then, our customers will understand the
difference and appreciate our efforts. We have to remember that this new product
does not have to be a good quality from the first. Rather, only basic functions will
satisfy them enough. The fact that they now have a new kind of product will excite
them and as software development teaches us, if the quality grows with time, our
customers will be pulled in and they will become a lifetime customer. It is much
easier than to attract customers with high quality from the first. In fact, high-quality
products only focus on one time value. But such continuous prototyping approach
will produce lifetime value and develop lifetime relations with customers.
1.7 Hardware Development and Software Development
To understand the difference between one time and lifetime values, let us take up
hardware and software developments and compare them.
Hardware products are developed with fixed functions. They are developed to
satisfy the design requirements, and the final products are delivered which comply
Fig. 1.11 Hardware

development
10 S. Fukuda
Fig. 1.12 Software

development
with and satisfy all design requirements. Thus, hardware products are delivered as
finished goods (Fig. 1.11).
Software development used to be carried out in the same manner as hardware. But it
was soon found out that hardware and software are basically different. Hardware is
physical, and software is non-physical. So software developers introduced continuous
prototyping approach. All functions are not offered from the first. First only the basic
functions are offered and after users get used to the system and get confident, a little
higher functions are offered. Functions are upgraded step by step and upgrades are
carried out after the developer knows that their customers get used to the system. Thus,
the system grows with time and with the customer (Fig. 1.12).
And when customers get used to the system and get confident, they put trust in
the system. The more confident they become, the more trust they put in the system.
Hardware development style is exactly as the word hard indicates. It is difficult
to adapt to the changes. But software development is literally soft. It can adapt to
the changes very flexibly. Hardware development style is primary for mass pro-
duction. Unless we do not produce products in mass, it takes lots of money to
change the production system. Software development, on the other hand is soft, and
very much adaptable, it can easily personalize the system. Such a difference comes
from the fact that software is non-physical and hardware is physical. The new
emerging technology of additive manufacturing, however, will change the whole
situation, which will be explained later in Chap. 2.
1.8 Dream and Expectation
Let us come back to the discussion about dreams and engineering. What are the
difference between dream and expectation? We could possibly divide them very
roughly in the following way. Dream is not rational, but expectation is more
rational. And most dreams are about the distant future, but expectations are more
about the near future and they are based on our past experiences. Expectations are
more likely to happen. Thus, when we say expectation, we presume its high pos-
sibility of realization.
Table 1.1 Dream versus Future Rational? Based on

expectation
Dream Distant Not rational ?
Expectation Near Rational Past experience
Therefore, it is more appropriate or realistic to say that engineering is here to

satisfy our expectations than to say that it is to make our dream come true. Indeed,
the final goal of engineering is to make our dreams come true, but in our daily
activities its goal is to satisfy our expectations, which are more realistic, but most
probably they originate from our dreams (Table 1.1).
1.9 Expectation Management
The word “expectation management” is often heard in business field, especially in

marketing. They flame customers’ expectations in order to increase the market share.
But we, engineers, have to remember that this activity is very much associated with
engineering. It is engineers’ job to find out what products will be accepted by
customers. It will lead us to what products we should develop. Engineers should
consider more how we can design and create markets beyond products. But
expectations discussed in marketing are about products.
Expectation management is also heard in management field. In the field related
to engineering, expectation management is often used in project management. The
expectations here are different from the ones discussed above. They are focused on
processes, not products.
What we would like to discuss here is expectation management in consulting
field. For consultants, what is more important than providing an excellent solution
is how they can follow the lines of reasoning of their customers. When their
customers ask them for help, in most cases, customers have solved the problem
already their way or have been getting closer to the solution their way. But cus-
tomers are not so confident about their way or they need advice to improve their
way of problem solving. So even if the consultant can offer a 100 % perfect
solution, it will not satisfy their customers or would not be accepted. The customers
would like to have their problems solved their way, not in a way they cannot fully
understand and hence they cannot put trust. So even if a 100 % perfect solution is
offered, the customers are not convinced or satisfied, even though the customer’s
approach may be 80 % and it is not so good as the one the consultant offers. What
the consultant should do is to help them solve the problem their way, no matter
whether it is 80 % or 70 % good, and not 100 % good. What the customers expect
from consultants is not a 100 % good solution, but they would like them to solve
the problem their way. If the problem is solved their way, and although it may not
be 100 % good, the customers feel very happy and satisfied because the problem is
solved their way. They can accept the result easily and comfortably, if the problem
12 S. Fukuda
Table 1.2 Expectation management in various sectors

Sector Action Goal Focus Comment
Business Marketing: Larger Product We should remember it is
flaming customers’ market engineers’ job to create a
expectations share market
Management Project management Scheduling Process Engineering
projects
Management Human resource Effective Process
management company
Consultant Follow the line of Convincing Process Customer’s acceptance is
reasoning of customers most important
customers
is solved their way. Thus, expectation management in consulting business focuses

their attention on processes (Table 1.2).
1.10 Sympathy and Empathy
Now, let us discuss sympathy and empathy in order to clarify the difference
between yesterday and today. To describe their difference concisely, sympathy is
the feeling that you care about someone else’s, while empathy is the ability to share
someone else’s feeling. In other words, when we say sympathy, your world and
someone else’s world are different. You live in your world, but you can care about
the feeling of someone else’s who is living in another world. Empathy, on the other
hand, your world and someone else’s world are unified (Fig. 1.13).
Fig. 1.13 Sympathy versus empathy

Therefore, we may compare sympathy and empathy to Web 1.0 and Web 2.0. In
Web 1.0, the producer and the user are different, but the producer assumes what the
user is expecting and they can produce such products and deliver them. The world
of producers and the world of users are distinctly separated, but when the world is
small and closed, they can understand the world of users and can produce what they
want. But as the world expands and becomes open, there are many ways of getting
to the goal and more often than not there are many diverse goals. Thus, as discussed
in the above as expectation management in consulting business, users would like to
have systems that work their ways. Thus, processes become more important in an
Open World than products.
Engineering is moving in the same direction as Web technology. It used to focus
on products, but we are moving into the age when processes become more
important than products. Product value is important, of course, but processes are
increasing its importance rapidly and process value needs more attention. Thus,
engineering is moving from Engineering 1.0 to Engineering 2.0 just like Web
technology.
Such transition was caused by the change of our outer world. There were small,
if ever, changes in our environments and situations yesterday. So we can make a
decision at the beginning. And in most cases, only one decision making is enough.
It is something like a railroad. As railroads have tracks, the only choice we can
make is to select which track or which train to get on to reach our destination. The
decision is made before we act. And once on a train, we are just carried to our
destination without worrying anything about decision making. But today environ-
ments and situations change so often and extensively that we have to make deci-
sions whenever there is a change. It is something like a voyage.
Our traditional approach was model-based. We make decisions based on a fixed
model. But today we have to carry out multistage decision making in order to
respond to the outer changes. Therefore, engineering yesterday was product-
focused. But as the lines of reasoning vary from situation to situation, more attention
must be paid to processes. So we could say we are moving from the age of products
to the age of processes.
1.11 Product Value and Process Value
Thus, engineering is now moving from product-centric to process-focused. If we

recall Maslow’s Hierarchy of Human Needs [5], this is very natural and convincing.
Humans look for physical satisfactions at the bottom level but as they go up, they look
more for mental satisfactions. And at the top level, they look for self-actualization. We
have to note that self-actualization is very much process-centric (Fig. 1.14).
Challenge is the core and mainspring of all human activities. And challenge is
one of the self-actualizing activities. Let us take mountain climbing for example. If
our goal is just to get to the top of the mountain, we could ask for a helicopter to
14 S. Fukuda
Fig. 1.14 Maslow’s

hierarchy of human needs
bring us there. If the goal is just getting to the top, then it will be achieved easily
that way. But this is not self-actualization. How we get to the top or the processes of
getting there is important. If the path is very severe and challenging, it will satisfy
our challenging spirits and when we get to the top, we feel deeply satisfied because
we can demonstrate our capabilities. Thus, the process is important, not the result.
Therefore, if we recall Maslow’s Hierarchy, it is quite natural that engineering is
moving from product-based to process-focused.
Value is defined as
Value ¼ Performance=Cost ð1:2Þ
But when hardware products were dominant in market, this performance implied
the functions of a final product, because hardware is developed with fixed functions
to meet design specifications. So to be strict, the value in this equation implies
nothing other than profit to the producer, because users do not know how well the
product will perform until after they put it to use. Expectations vary from customer
to customer. And to be more rigorous, hardware products are physical so that they
deteriorate immediately after delivery. The functions described on catalogs are
those at the time of delivery and such deteriorations are not taken into consideration
and not explained. Prices are determined based on the functions at the time of
delivery.
But software is different. At the time of purchase, customers do not really know
how well it works. Hardware is physical so customers can guess how they would
work. But software is non-physical so it is very difficult for a customer to understand
how it works before he or she begins to use it. And they do not deteriorate and as users
get used to the system more, it works better. This is very much contrary to hardware
behaviors. Hardware performance deteriorates with time, but software breaks in
better and works better with time. This is because hardware is product-based, but
software is process-focused. Thus, in software, the value in Eq. (1.2) is truly user’s
value. Performance in Eq. (1.2) in software is really performance, or how well the
system works under user’s condition.
As described earlier, software is developed in a continuous prototyping manner.

The functions grow with time and with customers. In other words, software
development is managing customer’s expectations in a very clever way. Software
developers sell processes, not products. Thus, it can grow expectations step by step
as is observed in continuous prototyping development and so users recognize easily
that their expectations are met and feel very much satisfied. Thus, the denominator
performance in software development is truly performance and the value is truly
user’s value (and profit for the producer).
Software development is in a way utilizing imprinting. It is something like
McDonalds business model. They paid their attention to imprinting and focused on
children first because once imprinted they will become their lifetime customers.
This is the same as Mom’s dishes. Software development is utilizing imprinting,
experience, storytelling, etc., in a very clever way.
Thus, what is important and what we have to learn from software development is
how value is shared between the producer and the user.
1.12 Process-Focused Business
Japan is very rich with process business. For example, flower arrangements and tea
ceremonies are typical ones. Flowers may be arranged better by experts, but learning
how to arrange flowers is a challenge and is self-actualization. Some people arrange
flowers very differently from others. How they arrange flowers reflects their
personalities.
It is very interesting if we note that many ladies attend courses to learn brazing in
Japan to produce their own accessories, while the number of brazing workers is
decreasing. Workers are paid but they leave. Ladies have to pay, but more ladies
will come and join. This demonstrates how self-actualization is important for us and
how processes yield emotional satisfaction.
Another interesting example in the field of welding is that in Japan there is
National Qualification Test for Welding. It would surprise you if you know a large
portion of successful candidates in highest level qualification test is (open-air)
sculptors. This is because in open-air sculptures, each artwork is different and what
makes the job very difficult is that the object is not designed in such a way that they
can weld it easily and what makes the job more difficult is the parts to be welded are
not well preprocessed for welding. They did not undergo pre-stage machining as is
usually done at factories. Further, a sculpture is placed outside in an open-air so the
environments are very severe. Sculptors challenge all these difficulties to achieve
their artwork and in order to achieve this goal, they learn and acquire such highest
skills. This also demonstrates how we attach value to self-actualization and we have
to examine the meaning of process value in engineering.
We have to note that a software development curve or a continuous prototyping
curve is nothing other than a learning curve. The more we learn, the more confident
we become and this drives us to the higher expectations.
16 S. Fukuda
These facts suggest that we can leave such work as spot welding in automotive
industries to our customers. Most of the parts to be spot-welded need rigidity, but
not strength. So even if their skills are not good as workers, we could possibly leave
spot welding to our customers. Then, they would enjoy doing it their way and they
feel more satisfied and will be attached to the finished car because they made it
themselves, although it may be only a small portion of the whole job.
We introduce robots because the number of workers is decreasing. Further,
robots will reduce cost. The introduction of robots may increase value in the sense
of the producer, i.e., their profit, but it does not enhance value in any way on the
part of the customer. Indeed, costs may be reduced and the degree of variability
would be far less. But personalization needs more variability and if customers can
be a player in the game, they do not hesitate to pay more. Why do we have to care
so much to reduce the denominator cost? We should pay more attention to increase
the numerator performance. We should re-examine Eq. (1.2).
If customers can join such manufacturing activities, they can create their
experience and they can tell their stories to others. This also satisfies Maslow’s
needs so that it would increase emotional value. The products which our customers
join in design and manufacturing may not be so perfect or sophisticated as experts
do without such interruptions. But we have to remember again how customers will
be emotionally satisfied if they can get involved in production. Again, customers
pay for their satisfactions, not for product quality.
1.13 Performance as Emotional Satisfaction
The above discussion demonstrates how performance has not been duly taken into
consideration in design and manufacturing from the standpoint of emotional
satisfaction.
In order to understand its importance better, let us take a steering wheel for
example. Wheels which steer very sharply will appeal to the young ones because
they respond to their quick actions. But such sharp responsive wheels will give
uneasiness to seniors. So the evaluation of functions depends largely on users and
situations. It is not function, but emotional satisfaction that we should consider in
design and manufacturing. In short, our traditional notion of performance in product
development was situation-independent. But performance is very much situation-
and context-dependent. The increasing importance of process values indicates such
underlying important of taking situation and context into consideration when we
discuss performance.
1.14 Performance in Business: MRO (Maintenance,

Repair, and Operations)
The word performance is quickly spreading in business sectors. These perfor-

mances are a little bit different from performance which was used yesterday and
which is discussed in the previous section.
In a Closed World, the producer could foresee how their products will be used.
That is why Engineering 1.0 or the separation of the producer and the user could be
established and worked well. In these days, what the user expected from the pro-
ducer is to develop a product that would work in the expected operating conditions,
and it was the user who should take care of operational issues. Functions were
designed to operate in such user-responsible operating conditions. If the functions
were well developed, they should work well in operation.
And what is most important is that in these days products were used indepen-
dently. Even if they were used together, we did not have to pay attention to how
they would behave as a group. Thus, it was enough if we consider only the relation
of functions of a single product and its operation. But today, products are used in a
connected manner so that the word performance implies not only that of a single
product, but that of a group of products. Thus, performance today means group
performance.
Let us first take up the example of power by the hour or PBTH. The term PBTH
was originally coined by Bristol–Siddeley to respond to the diverse requirements
for their engines by business jet customers. Business jet operations vary very
widely from customer to customer. PBTH is a strategy to meet such diverse
requirements. But we should note that the original idea of PBTH is not necessarily
taking group performance into consideration. Their main focus was how to deal
with a wide diversity of operating conditions and their primary concern was a single
engine.
Rolls-Royce later expanded their idea to commercial airplane operations. Their
core idea is to share value between the producer and the user. We should note that
commercial airplane operations are network-structured. And this network is
becoming more and more complex. When Bristol–Siddeley brought up the idea of
PBTH, it was how we can make a node flexible, if we use graph theory terms. But
Rolls-Royce expanded the idea to a network.
Bristol–Siddeley and Rolls-Royce realized that value considered only from the
producer perspective does not satisfy the user needs or their expectations. But
Bristol–Siddeley only considered a single engine and its user, but Rolls-Royce
considered a group of engines and how they will be used in a networked operation.
In commercial airplane operation, engines have to be inspected and may have to
be replaced while planes fly from airport to airport. If it takes time to replace an
engine, it will interrupt scheduled flights and sometimes it will cause a big financial
damage to the airline companies, because their operation is networked and if
operation is interrupted at one airport, it will influence those at other airports in a
chain manner. If the engines are deployed adequately to airports, then airline
18 S. Fukuda
companies can keep their on-time performance rating high so that they can satisfy
their passengers. In other words, Rolls-Royce can keep airline network intact by
introducing PBTH.
Thus, airline companies receive the benefit, but it must be noted that it is also
beneficial to engine companies. They can organize their maintenance jobs better
and put them on schedule so that they can save time and labor. Thus, PBTH
established win–win–win relations among engine companies, airlines, and pas-
sengers. Satisfactions are far greater than those they feel when just a single engine
functions work well. Such benefits were brought about because engine developers
expanded their thoughts from just a single engine to a group of engines and how
they will be used in a networked operation. Such network-focused perspective
made everyone in the game emotionally and financially satisfied.
Another good example is Komatsu. They produce heavy construction equipment
in Japan. Although their business is primarily B to B and they sell their vehicles to
rental or lease companies, they developed Komtrax and put this online monitoring
system on their vehicles in order to identify how their end customers are operating
them on their construction sites. As conditions vary widely from site to site,
Komatsu have to know such data in order to design and manufacture well
fit-for-the-purpose ones. But their focus is primarily on a single equipment.
The greatest advantage this brought to them is not only expanding the functions
of their single equipment and increase their sales to rental and lease companies,
because end users specify Komatsu vehicle at the time of their rental or lease, but
what is more beneficial to them is they can identify timing for replacement or for
repair. Komatsu can organize their maintenance jobs and they can negotiate with
their customers about what time will work best for them for maintenance or for
repair. So construction companies are no more harassed by sudden failures or by
untimely inspections. They can pinpoint good timing for such jobs which would not
disturb their schedules. Thus, it is very beneficial for them, but it is also beneficial
for Komatsu. Although construction equipment is not networked, Komatsu can
prepare parts in time and they can run the job with the minimum number of
maintenance workers. Komtrax is a good example of how to schedule maintenance
better in spite of very widely varying operations. This is the case of adaptive
performance.
PBTH and Komtrax demonstrate how it is important to pay attention to pro-
cesses or how products are used by customers. When producer–customer relation is
a single node–single node relation, they did not have to care about processes or
work flows and how their products are used. But in an age of network, performance
should be studied from the standpoint of a network. In other words, attention should
be paid more to work flows or processes and how we can let them flow smarter
becomes critically important.
Recently the word performance contracting is becoming widely used. This also
indicates that as we are entering an age of networks, we have to re-examine contract
on the basis of such networks of products and operations instead of discussing the
functions of a single product. In other words, performance contracting may be
interpreted as a contract to establish win–win relation between the producer and the
user by sharing the same notion of performance, which indicates the increasing
importance of process values. In other words, performance comes to mean per-
formance of orchestration.
1.15 Repair and Renovation: Emotional Performance
Although maintenance, repair, and operations are put together and are called MRO,
maintenance and repair are very much different. Maintenance is to restore the
degrading functions back to its original design requirement levels. So there could be
preventive maintenance. Even before a product or a machine fails, we can prevent
degrading before it begins. The remarkable progress of sensors developed such
technology as prognostics and health management (PHM) to keep machines
working in best conditions. Repair is always post-. There is no pre- in repair.
But we have to note that there is another big difference between the two.
Let us take our health for example. The word health in PHM is machine’s health.
What we are going to discuss here is our health. We all know doctor’s health and
our health are different. When a doctor says “You are healthy. But you have to
drink less to stay healthy,” then, do you feel you are healthy, if you love to drink?
We feel healthy when we can spend our life happily in our own ways. So feeling
healthy and being healthy in a medical sense are very much different. How a person
feels healthy varies from person to person. Just in the same way, machines work
best in their own ways. Users feel their machines work best when they satisfy not
only their needs but also their preferences.
Maintenance is technical, but repair is very much emotional. Repair needs
diagnosis before treatment. But maintenance is basically treatment and diagnosis is
not so much called for. Repair needs an engineering doctor, not a person with
doctor’s degree, but a person who has the same capabilities as medical doctors and
can carry out engineering diagnosis just as medical doctors do. As the interpretation
of health varies from person to person, there is no perfect recovery from illness. If a
patient feels satisfied and feel healthy, then the medical diagnosis and treatments
were successful. Likewise, there is no perfect recovery by repair. But what is more
important is while maintenance is nothing other than just putting everything back to
its original, repair is to keep the machines or the products running in their best
working conditions. It may not be a perfect health from the eyes of a medical
doctor, but it is our health we would like to recover. Best repair recovers our health.
Thus, repair needs much higher engineering capabilities. It needs the capabilities of
diagnosis in addition to those of treatments. And it should be noted that correct
diagnosis is not a good one for the patients. Diagnosis that meets patient’s
expectation is good from patient’s standpoint. Thus, repair must consider cus-
tomer’s expectations. Thus, it is more emotional than technical. In this sense, repair
and consulting business have lots in common.
In other words, medical health and maintenance are rational, but our health and
repair are emotional. It is described in Sect. 1.12 that the many recipients of the
20 S. Fukuda
highest level qualification in welding are now sculptors. This is because their
artwork is in essence the same as repair. Or it may be better to say the other way
around. Repair is fundamentally an art. As there is no pre-stage preparations done
on their materials and the conditions change from sculpture to sculpture, sculptors
have to carry out diagnosis before starting to weld. This is nothing else than the
procedures of repair welding.
In fact, the word repair comes from the same Latin word as prepare and it means
to prepare again. Thus, repair is to prepare the products to let them work again in
their best conditions, best in the sense of customers. So what differentiate it from
new product developments is that in repair, materials have been used and design
and manufacturing start with considerations about their past histories. Therefore,
repair calls for higher engineering knowledge and capabilities of diagnosis and
adequate judgment considering customers’ expectations, and thus, repair is more
difficult than new product development.
Behavior economist insists the importance of user experience (UX). I totally
agree. Their assertions are very much to the point. But they do not tell us how to
keep the value of UX. Their pointing out of the importance of UX can be inter-
preted as their way of saying doctor’s health and our health are different. They
assert our health is very important. But regrettably they do not tell us how we can
maintain our health.
In engineering, it is repair. Regrettably enough, although UX is attracting wide
attention, no so much attention is paid to repair. But both are the same in essence.
Repair is a way to keep UX as long as possible.
Although UX is sometimes discussed in connection with stories, their relation is
not so much often discussed. But when it comes to repair, we should keep in mind
stories play a very important role in diagnosis and judgement. Why do customers
want repair is because they would like to keep their UX as long as possible and it is
very much associated with their stories of using the product.
Harry–Davidson Owners Group (H.O.G.) is a good example. The members visit
HD factory often and it is not to observe what is going on there, but their main
interest is how they can share their stories of repairing their motor bikes themselves
or those of keeping them in good shape with workers. This storytelling or sharing of
UX put them more closely together and they truly become HD’s customers, not just
their motor bike riders. Thus, HD is selling emotional satisfaction beyond the
technical functions of their motor bikes.
In Japan, home renovation is now getting wide attention these days, because
Japan is one of the typical quickly aging countries and people would like to spend
their senior life in their old familiar homes, which have many stories about their life,
so they are very much attached to them.
Apart from such local topics, renovation becomes globally important and it will
increase its importance very quickly in the near future. We have to note that repair
and renovation are not necessarily the same. Repair is to recover our health. This
health implies how we spend our life in the past and we would like to get back to
the conditions of the past which we enjoyed. So Repair is to get back to the best
days of the past. Renovation is looking back into the future. Although we use many
parts or components of the past, we will attach many new ones to prepare for the
future.
Remanufacturing is also getting wide attention these days, but we should
remember that remanufacturing is technical and although repairing jobs may be
associated, it is not for securing emotional satisfaction as repair does.
But it must be stressed that the importance of remanufacturing is increasing very
rapidly. It is creating a new big market other than the new production, and
sometimes it may be bigger than the old, traditional ones. We have to remember
that although we are focusing our main attention to the production of new products,
we may not be able to stay on this track any more in the near future, because our
resources are limited. We have to change our perspective and pay more attention to
remanufacturing or repair. Although repair calls for very knowledgeable engineers
for diagnosis, the following jobs can be carried out by other less knowledgeable
workforce. The capabilities which are called for in these works are no different from
the current production of new products or remanufacturing. Thus, the concern that
the number of workforce will be reduced will not be the case. On the contrary, there
will be more jobs at many different levels because repair is so diverse. The number
of products may decrease, but the number of jobs will increase. Repair or reno-
vation will become major industrial activities in the latter part of the twenty-first
century.
And it should also be pointed out that sometimes old materials which have
stories yield better emotional satisfaction than new ones. Fritz Hansen in Denmark
use animal leather with scratches for their highest grade chair, although they can
produce far better quality artificial leather. This is because they can attach stories to
their chair if they introduce natural leather. Scratches bring more emotional satis-
faction than high-quality artificial leather.
Thus, repair or renovation brings emotional satisfaction to customers and per-
formance in this case is again emotional performance.
1.16 Summary
Up to now, we have been doing our best to play our own musical instruments
better. But as Internet of Things (IoT) [6], Cyber-Physical Systems (CPS) [7], and
Industrie 4.0 [8] indicate products are getting more and more connected. We cannot
play our own instruments alone. We have to play as a member of orchestra. But this
orchestra is different from other conventional orchestra. There is no conductor or
director. We have to orchestrate by our own initiatives. Thus, we need an ability of
empathy in this connected world to achieve the greatest emotional performance.
Thus, it is not just an age of orchestration, but it is an age of smart orchestration.
22 S. Fukuda
References
1. http://www.nobelprize.org/nobel_prizes/economic-sciences/laureates/1978/
2. Keynes, J.M (1936) The general theory of employment, interest and money, Palgrave
Macmillan, London
3. Schon, D.A (1984) The reflective practitioner: how professionals think in action, Basic Books,
New York
4. Kim, C, Mauborgne, R (2005) Blue ocean strategy: how to create uncontested market space and
make competition irrelevant, Harvard Business Review Press, Boston
5. Maslow, A.H (1943) A theory of human motivation, Psychological Review, 50 (4), 370-396
6. http://www.rfidjournal.com/articles/pdf?4986, Retrieved 30 Sep 2015
7. http://www.nsf.gov/pubs/2008/nsf08611/nsf08611.pdf, Retrieved 30 Sep 2015
8. http://www.acatech.de/fileadmin/user_upload/Baumstruktur_nach_Website/Acatech/root/de/
Material_fuer_Sonderseiten/Industrie_4.0/Final_report__Industrie_4.0_accessible.pdf,
Retrieved 30 Sep 2015
Chapter 2
Composing a Product Network
for Emotional Performance
Shuichi Fukuda
Abstract Up to now, we have been discussing performance, focusing our attention

on an individual product or a product family at the utmost. But now as Internet of
Things, Cyber-Physical Systems, Industrie 4.0, etc., are spreading very quickly, and
products are getting more and more connected. So, we have to widen our view and
consider products as a society and how each product can contribute to the society
and what role it should play. This calls for a new perspective. We have to remember
that connecting best performing products does not bring best performing society.
We have to assign an appropriate role to each product and to make it serve its
function for the society. In short, we need to orchestrate our products to achieve
best performance as a team. To achieve this goal, emotion plays an important role,
since performance is not just functions, but it means how much emotional satis-
faction a product or a team of products provides to our customers. Thus, perfor-
mance is quickly moving from functional performance to emotional performance,
and again, it is no more that of an individual product, but that of a team of products
or a product society.
2.1 Introduction
In Chap. 1, it is discussed that as things are getting connected, we have to stop

discussing individual products and move toward discussing how products will work
together as a network.
But most of the current discussion is how we can form a network by putting
currently existing products together. But what is stressed in Chap. 1 is that we
should go further than that. This approach is still stuck with our traditional
product-focused perspective.
S. Fukuda (&)
System Design and Management, Keio University, 4-1-1,
Hiyoshi, Kohoku-ku, Yokohama 223-8526, Japan
e-mail: shufukuda@gmail.com

DOI 10.1007/978-3-319-29433-9_2
24 S. Fukuda
We have to recognize 11 best does not make up a best team, i.e., forming a team
with 11 best players does not work. Best 11 is not 11 best. Each player may not be
best as an individual, but if they team up in the best way, they can establish a best
team [1].
Therefore, we have to change our focus from forming a network of best func-
tioning products to composing a best functioning network of products. The word
composing is used in the same sense and meaning as in musical composition. We
compose music to express our feelings. So network composition means we compose
a network of products to satisfy our emotions. It is not just an assembly of product
functions. It is far beyond that. A team performs far better, if composed appro-
priately, than a single player does. Team of players produce music that sounds very
differently from that of each single player. Thus, we have to move from traditional
performance, which focuses on individual products to network performance. We
have to move from 11 best toward best 11.
2.2 Decomposition
Music may also have started from decomposition, but today, musicians’ primary
focus is on how to compose music and few, at least to my knowledge, study how
the real world is decomposed into music.
In engineering, on the other hand, the necessity of decomposition was realized
very early when products became large and complex. In the real world, most
physical things are continuous or analog, at least to our eyes. When we made tools
or products in the early days, they are quite small and simple. So we designed and
manufactured them as we learned from nature. Thus, they remained continuous or
analog all the way or all through their life cycle. But after we succeeded in pro-
ducing such small and simple products, our expectations grew much higher and we
moved toward producing much larger and complicated products. Therefore, we
could no more produce them as a continuous or analog object. We have to
decompose them and break them into smaller parts, which are small enough to deal
with.
2.3 Modularization
The word modularization is getting very popular and coming to be used widely
today. But if we look at it as discretization, its history is very long. When products
became larger and more complex than we could handle, we broke the object into
parts and assembled them into a product. It is our wisdom how we can deal with
such large and complex objects with our limited resources.
Decomposition discussed in Sect. 2.2 is one of such examples. Of course, it
called for another sophisticated technology for assembling these parts into a final
2 Composing a Product Network for Emotional Performance 25
product. But the progress of technology was so rapid that soon we could assemble
many different parts into a product as we like. Thus, today, when we say modu-
larization, it implies not only discretization, but also assembling of discretized
parts. So from now on, let us assume that modularization means both discretization
and assembly. In the earliest days, the focus of modularization is on physical sizes
or dimensions. We just broke large size products into smaller size parts.
But soon not only sizes but the number of functions increased with increasing
complexity. We found out that if we break the product not only into smaller sizes,
but into parts with particular functions, we can design more complex products far
easier. Thus, modularization soon came to mean functional modularization. And
although sizes were not less important, the greatest attention came to be paid on
how we can divide functions and allocate them appropriately into parts.
This is the history of modularization of hardware products. But when software
emerged as new technology, it is no more physical so modularization became solely
the problem of functional modularization.
With increasing complexities, hardware and software came to be used together
as we can easily observe in mechatronic systems. Today, software became an
indispensable partner for hardware. So now, we have to consider modularization
from both perspectives, physical and non-physical.
Looking at modularization from the standpoint of graph theory, it is nothing
other than representing products as a network. Each node has attributes (sizes,
functions, etc.), and a link is the interface between the nodes (parts).
But we have to remember that if we look at modularization from a different
perspective, i.e., from the standpoint of design, not only we have to consider
decomposition for making things easier, but we also have to pay attention to how
we can compose them better for design.
We decomposed or discretized our continuous objects into modules, or nodes in
graph theory terms, mostly based on our experience from nature. So we did not pay
too much attention how attributes can be allocated to modules. Software changed
the scene. Software modules are non-physical and it is totally artificial so we have
to consider how we allocate attributes to each module. Hardware followed suit,
because with increasing complexity of functions, we have to consider how attributes
can be allocated best to each module from the standpoint of design. We need to
design a module with only attributes needed for its functions.
Modularization is getting wide attention these days in automotive industry,
especially in the field of passenger cars. But the history of modularization is very
long and how we divide modules depends on how we allocate functions. So if the
functions are common to all models, you can use the identical model for all of them.
Automotive companies call it a platform and they put on different kinds of modules
on top of it to characterize their models. But the same idea was already carried out
in truck industry from long time ago (Fig. 2.1).
The identical chassis can be used for all purposes. But the cargo body has to be
developed case by case to meet different needs. So truck producers divide the
chassis and cargo body, and different companies develop and produce them.
Passenger car industries are just following suit.
26 S. Fukuda
Fig. 2.1 Modularization of a

truck
What should be stressed is that modularization in this sense is now going on in

every industrial sector. Some deal with very large systems and others deal with very
small systems, so modularization is going on very rapidly with products with
different sizes with different degrees of complexity.
2.4 Finite Element Method (FEM)
Thus, modularization is modeling a continuum with discrete elements. This

expression will remind us of finite element method. Its modularization is very
simple and straightforward. As the degree of freedom of a continuum body is
infinite, we cannot analyze it as it is. Therefore, we developed a new model and
reduced it to finite and calculable degree. FEM is a tool for analysis. So their goal is
very clear.
Let us take up FEM of a solid body here. The goal is to calculate stress, strain,
and deformation, so they developed a very clever model based on a truss structure.
A truss structure is composed of triangular units and each unit has three members
connected at the joints or nodes. Each member is two-force member and is subject
to either tension or compression.
Just as triangulation is used to survey lands, truss member modeling enabled us
to model any shape as an assembly of triangular units. And an analysis can be made
very easily if we assume that each triangle is covered with a membrane with no
resistance and force (shown in blue in Fig. 2.2) and is only transmitted to members
through joints or nodes (shown in yellow in Fig. 2.2) either as tension or com-
pression. In fact, it is a very simple model of a spring (Fig. 2.2).
Modularization as seen in FEM is an excellent idea which revolutionized analysis
of a continuum body. In short, geometry is modeled by triangles and stress and strain
analysis is made possible by introducing the idea of a truss structure. Today, FEM is
used very widely in many industries, and without it, we cannot design and manu-
facture such very complex and complicated machines, structures, etc. But FEM is
Fig. 2.2 FEM triangular

element
used primarily for analysis. Then, what technology will enable us to design a product
with the concept of modularization? It is additive manufacturing.
2.5 Additive Manufacturing (AM)
Recently, additive manufacturing is attracting wide attention and is spreading very

quickly. It is expected that AM will enable us to design and manufacture any shape
without too much difficulty. And in addition, CAD data can be directly realized as a
physical entity. This discussion is too much simple. In reality, it is not so
straightforward. But we are certainly going in the direction where we can modu-
larize products and attach any attributes to each module as we like, with the
emergence of AM.
Interestingly enough, AM not only enables us to attach any attribute to each
module, but it will also attach a new meaning to vacancies. They are called chasm, gap,
slit, etc., but in most cases, they have been recognized as unfilled or unoccupied and
have been regarded as undesired spaces or as spaces that do not serve for any purpose.
Truss structures utilize spaces in a very sophisticated manner, but these empty
spaces are result of engineering design to reduce weight and the amount of material.
So although they look vacant, they have meanings.
What underlines our traditional design requirements is our experience of phys-
ical entities which are found in nature and most of them are continuum and with a
complicated combination of attributes. It is complicated, not complex. If complex,
we may be able to separate them into each attribute more easily. Thus, our
recognition of physical objects is in most cases continuum bodies with many
not-easy-to-separate attributes. Or rather such combinations characterize the nature
of physical entities we experience. Thus, it has not been easy for us to decompose
attributes and to compose them in a different manner, as we like. AM provided us
with such a capability of composing attributes as we like and as we need. Now, we
can design a new physical entity by picking up desired attributes and combine them
as we wish. AM is a versatile tool to produce unexperienced physical entities.
But as to the separation of attributes, there are other examples we should be
aware of. Prof. Shigeru Ban, architect who received the 2014 Pritzker Architecture
Prize, developed homes and other architectures using papers. His idea is very much
associated with paper folding. Most of us think that thin paper is weak so we cannot
28 S. Fukuda
build an architecture with it. But weakness is a property related to strength and
rigidity is another. But most of us fail to discriminate them. If we design appro-
priately, we can secure rigidity so that we can build homes, churches, etc., using
papers (Photos 2.1, 2.2 and 2.3).
Photo 2.1 Paper homes in

India
Photo 2.2 Cardboard

cathedral (interior)
Photo 2.3 Cardboard

Cathedral (exterior)
Many architects are now paying attention to papers, and they are pursuing paper
architecture [2]. This demonstrates how it is important to decompose attributes in a
proper way. This idea came from origami, or paper folding. Although we fold
papers, most of them are very simple ones (Photo 2.4). We have to know we can
fold them into very complicated ones (Photos 2.5 and 2.6). In fact, it is not an
exaggeration to say that any shape can be folded.
In addition, origami or paper folding plays a very important role in developing
deployable structures, which are required in space.
Photo 2.4 Origami crane
Photo 2.5 Origami serpent
Photo 2.6 Origami tree

30 S. Fukuda
These examples are very much interesting as they demonstrate how our tradi-
tional culture serves for the progress of engineering. But they are still focusing on
tangible attributes. But if we remember the phrase “silence is golden,” vacancies or
emptiness should have some meanings in our life. Up to now, vacant spaces have
been created unintentionally after pursuing to realize the desired functions with
desired geometry. Or they were results of engineering design as described in the
above with a truss structure example. Anyway, these vacant spaces were leftovers
from engineering design, and they were not intentionally created. But if silence is
golden, then we have to change our focus from words to silence. That is, from
tangible space to intangible space or vacant spaces. At least to my knowledge, we
have never tried to design a product by focusing first on vacant spaces. It is
certainly important when to keep silent during conversations. Thus, silence has
meanings. We have to design when we should keep silent to communicate better.
Likewise, vacancies or empty spaces have meanings. But up to now, we only paid
attention to words and forgot how to design when to keep silent.
Quite interestingly enough, Prof. Shunji Yamanaka and his group at the
University of Tokyo are now starting a project how to design empty space (Photos
2.7, 2.8, 2.9, and 2.10).
They utilize AM, and AM made such an approach possible. AM opened doors to
the new world of design where words and silence can be orchestrated, and let them
work together harmoniously.
In fact, if we recall Fourier transform analyzes time series and it transforms a
function of time to a function of frequency, and further space and time are asso-
ciated with it, it is reasonable enough that silence is a signal and has a meaning. So
is empty space, too.
AM has a potential to let us communicate much better with the outer world than
the time when we paid our attention only to words. It will make our design much
more interactive and communicative. Our designs up to now have been more or less
one-way communication.
Photo 2.7 Empty space

design-1

design-2

design-3

design-3
32 S. Fukuda
AM provides us with a versatile tool to design a product by using not only

tangible physical attributes, but also intangible attributes. We have to remember
again the importance of “silence is golden.”
2.6 Visual Words
We discussed in the previous section that vacancy or emptiness has significant

meanings. In other words, vacancies or emptiness has meanings because there is
substance elsewhere or other portions are occupied or filled.
We can find the opposite example in natural language processing and computer
vision. Natural languages are processed based on grammar and word order. But
there is an approach called Bag of Words, which disregards them and just counts the
number of times a word appears.
Long time ago, I attended a meeting of elementary school teachers in the USA.
At that time, they were very eager to learn from Japan. But they could not process
Japanese on their computers, because theirs were one-byte machines. To process
Japanese, they had to have a two-byte machine. In Japan, natural language pro-
cessing was very much prevalent these days, but all of them processed the language
based on grammar. But none of these methods could be applied because their
machines were one-byte computers.
What surprised me was one teacher stood up and said that if we note Kanji or
Chinese character (Japanese in Chinese characters), their images are very dense or
thick, while Hiragana or Katakana, which are Japanese phonetic alphabets, is very
sparse or thin. So if we regard Kanji as images, we can pick them up easily. If we
classify them as patterns, we can count the number of the appearance of the same
Kanji. As Kanji is keywords in Japanese sentence, we can estimate what the sen-
tence means if we make up the histogram of the appearance of each Kanji. You can
ask Japanese or someone who understands Japanese what all these Kanji mean.
Then, you know the keywords, so now you can guess what the sentence means.
All Japanese know Kanji is an image character, but none of us Japanese noticed
there is such a way to understand Japanese sentence. Indeed, this may be the way
we all Japanese do in everyday life without being aware of, or this is certainly the
way when we read very fast.
This was quite an eye-opener to me and I realized how true it is that outsiders see
most of the game. Dense image patches have dense meanings, and sparse image
patches have meanings of less importance. This is the spatial relationship among
Kanji, Hiragana, and Katakana. This is certainly a very clever feature extraction.
Interestingly enough, image processing researchers expanded the idea. They
developed an approach to process images in the same way. Just as we notice Kanji
in text and keywords in communication, there are feature portions in images. And
they serve for us to recognize the image. They play a role of keywords. The
researchers call such feature image portions Visual Words. This is just contrary to
the idea of how we can make the most of empty spaces described above.
We modularize an image by the degree of importance or the degree of catching

our attention. If a very prominent feature can be identified, then we can recognize
the image at once. If we cannot, we will go down to the next lower level and look
for some other feature portions. Repeating these procedures step by step, we can
recognize the whole image.
Although this follows the way of traditional modularization approach, we should
realize we can easily extend it to emotional modularization and we can satisfy our
customers more. We should remember that traditional engineering modularization
has paid its primary attention to functions. Although past engineering design uti-
lized modularization very much, its purpose was only for achieving better func-
tionality, for reducing complexity, or for reducing cost and labor. And we did not
pay attention to the problem of how we perceive and recognize. In traditional
engineering, modularization has been carried out based on design intentions. It was
done with engineering advantages on mind. But in other business fields, modu-
larization focused on perception is being carried out to satisfy their customers
emotionally.
2.7 Emotional Modularization
In other business fields, modularization focused on emotional satisfaction has been

carried out for a long time. Most typical one is fashion industry.
Let us take a wedding dress for example (Photo 2.11).
Every lady would like to wear a wedding dress just for her alone. But not every
lady is rich. So they have to wear rental ones. But rental shops cannot prepare a
different wedding dress for every lady. So they hold a wedding dress fashion show
and observe which part of the dress ladies are paying attention to. Then, they
prepare distinctively individualistic parts which would appeal to ladies with dif-
ferent personalities or preferences. And they prepare the common platform. They
Photo 2.11 Wedding dress

34 S. Fukuda
combine them and create personality or uniqueness. So when ladies rent such a
wedding dress, they feel this is the dress just for her. Thus, ladies feel very much
satisfied.
This idea is fundamentally the same as the one automotive industries are prac-
ticing with passenger cars. But automotive companies are focusing their attention
on such matters as cost reduction, efficiency, etc. The main interest of fashion
industry is to provide emotional satisfaction. Indeed, ladies would not care to pay
more, if they are fully satisfied. All fashion industry is practicing such emotional
modularization and pursuing emotional performance.
Engineering companies are now chasing after them. Daihatsu, for example,
developed Copen which is composed of changeable parts (Photos 2.12 and 2.13).
At this moment, experts are producing these parts, but it is expected customers
themselves can produce such changeable parts and will enjoy changing them in the
Photo 2.12 Daihatsu

Copen-1
Photo 2.13 Daihatsu

Copen-2
near future with the help of such technologies as AM. Thus, we may have car codes
just as we have dress codes. In fact, dress code is a composition of a network of
dress parts to meet situational and cultural requirements.
2.8 Composing a Network for Product Society
Up to now, we have been discussing performance from the standpoint of a product

or a product family. But now, all products come to be connected. Therefore, we
have to widen our view and consider a product society. Each product has a different
role in the society, and how we organize them and harmonize them to realize a
better society is our next challenge. As described in the previous sections, emotion
will play a critical role in linking all artifacts together in order to adapt to the
fast-changing situations and to provide satisfaction to our customers.
References
1. The secret is to work less as individuals and more as a team. As a coach, I play not my eleven
best, but my best eleven. Knute Rockne http://www.brainyquote.com/quotes/quotes/k/
knuterockn390851.html, Retrieved 7, Oct, 2015
2. http://origami.c.u-tokyo.ac.jp Retrieved 7, Oct. 2015
Chapter 3
Humor Engineering in Smart
Environments
Anton Nijholt
Abstract Enjoyment is one of the positive emotions we expect to have when

visiting environments that have been designed to provide us with entertainment
experiences. However, enjoyment is also part of our daily life, whether we are at
home, in our office environment, in public environments, or on the move from one
environment to the other. Humor is important in our daily experiences. We our-
selves and our daily conversational partners are expected to have a sense of humor.
We cannot maintain friendly relationships with family members, friends, and col-
leagues without having this sense of humor. Humor is one of the most appreciated
traits in human–human interaction. In this chapter, we investigate the possible role
of humor in smart environments or, from an opposite perspective, how smart digital
technology, its sensors, and actuators can help to introduce humorous situations and
interactions in digitally enhanced physical worlds. With these words in mind, we
discuss accidental, spontaneous, scripted, and staged humor. We will also draw
upon humor as it appears nowadays in artificial physical environments, such as
video game environments. Conclusions will be drawn about being able to include
humorous interactions and about being able to participate in humorous activities in
smart technology-supported environments and events.

Keywords Humor modeling Smart environments Sensors Actuators Video

games Entertainment technology
3.1 Introduction
It is quite unusual to talk about the modeling of humor. Humor seems to escape
from every rational explanation and therefore also from every algorithmic calcu-
lation. How can we ever expect that a non-human such as a computer can under-
A. Nijholt (&)
Faculty EEMCS, Human Media Interaction, University of Twente, PO Box 217,
7500 AE Enschede, The Netherlands
e-mail: a.nijholt@utwente.nl

DOI 10.1007/978-3-319-29433-9_3
38 A. Nijholt
stand a joke like: ‘A dyslexic man walks into a bra.’? Or ‘Three men walk into a
bar… Ouch!’ Admittedly, these are two well-known meta-jokes, variants of a
particular kind of jokes that start with ‘Three men walk into a bar …,’ and therefore,
they do not ‘only’ require real-world knowledge about stereotypes and their con-
tradictions, but also about a culture of joke telling. Moreover, a joke-telling culture
in Europe or the USA will be different from a joke-telling and humor-sharing
culture in, for example, Arabic countries. A Hollywood culture that has embraced
icons as Marilyn Monroe will be more receptive to jokes about dumb blondes than a
culture where women are expected to cover their hair or both face and hair.
Obviously, there is more about humor than jokes. Humor appears in sitcoms, in
movies, and on stage, but humor also appears in the street, in public spaces, and on
public transport. We encounter humor in daily interactions, not only in spontaneous
conversational face-to-face interactions between humans, but also when encoun-
tering unexpected situations, not necessarily involving other human partners. Such
unexpected situations and events can be accidental, but they can also be created,
just as we can create jokes. Throwing a banana peel on the street can lead to a
humorous event. What about digital banana peels in the streets of smart cities?
What about digital pranking?
Our aim in this chapter was to investigate how we can use smart sensors and
actuators embedded in smart environments to create or help to create humorous
events. There are various ways to use smart technology to realize or help to realize
humor. A smart environment can have a sense of humor and make decisions how to
use its sense of humor to create humorous situations or otherwise act in a humorous
way for its inhabitants. A less autonomous way to deal in a humorous way with its
inhabitants is to have smart technology embedded in the environment that does not
always act as can be expected from its appearance, context of use, or function. This
smart technology will surprise us in a humorous way when we try or need to use it.
A third way to use smart technology in order to create humorous events is when
inhabitants of smart environments can arrange or introduce sensors and actuators in
such a way that humorous situations can be expected to emerge. At this moment,
we can only expect to have some success with the latter two ways. And we have to
lower our aims to present a humor survey from which we can learn how to
introduce humor and what kinds of humor in smart environments, rather than being
able to provide guidelines for designing sensor and actuator configurations that
facilitate humor creation.
In Sect. 3.2, we have some observations on humor theories. Formal approaches
to humor aim at the description and analysis of particular instances of humor, in
particular jokes and wordplay. They do not yet allow an algorithmic detection,
analysis, or generation of humor in general. We nevertheless can learn from
attempts to describe or categorize humor, even when these attempts focus on verbal
humor and more in particular on jokes. It may be possible to generalize aspects of
these theories to the description, understanding, and creation of visual jokes,
humorous products, humorous behavior, and humorous events.
In Sect. 3.3, we discuss humor as it appears or can be created in real life, in
products, cartoons, animations, movies, and games. In particular, we will look at
3 Humor Engineering in Smart Environments 39
virtual game and communication environments, such as multiplayer video games.

Hence, it concerns not necessarily task-oriented environments where people can
communicate with each other, play games, comment on others, or cooperate or
compete with others. Section 3.4 is then concerned with humor as it appears or can
be created in interactive, digitally enhanced environments and devices, that is, in
smart environment. We collect ideas about the generation of humor in smart
environments that are based on what we see happening in game environments. That
is, we look at forms of humor such as pranking, trolling, and griefing that are not
necessarily considered to be humorous from the point of view from the recipient.
Some conclusions are presented in Sect. 3.5.
3.2 Understanding and Modeling Humor
We know about the modeling of intelligence (artificial intelligence), knowledge

(knowledge representation), and reasoning about knowledge. Agent research
informs us about modeling beliefs, desires, intentions, and even emotions. Emotion
modeling is also part of affective computing research. Expressions of enjoyment,
for example, laughter and facial expressions, are also topics of research.
Unfortunately, there is hardly research on the computational modeling of humor.
Formalizing the creative process of humor generation and humor understanding
seems to be more difficult (or less important) than formalizing intelligence,
knowledge and knowledge use or understanding affect and affect expression.
However, it does not mean that in human–computer interaction technology,
knowledge about human–human interaction, affect, and humor cannot be used.
Such knowledge, even if it is imperfect, can be used to make human–computer
interaction more effective, more natural, and more entertaining, and it can make our
life in smart environments and digital cities more enjoyable.
3.2.1 Humor Theories: Superiority, Relief, and Incongruity
Many philosophers, psychology, and cognitive science researchers have contributed

to what can be called humor theory. They have been helped by (computational)
linguists and computer scientists who focused on models to analyze and generate
verbal humor. These attempts have been useful, but not very successful. We shortly
review these theories and some computational linguistics, natural language pro-
cessing, and human–computer interaction applications where we can recognize
aspects of these theories or where these theories are explicitly addressed. Usually,
three main theories of humor are distinguished [20, 26].
Superiority or disparagement theory is about our role in perceiving or creating
humor. We and others listening to a joke or experiencing a humorous situation feel
superior to the main subject in the joke or situation. Our respect to this person
40 A. Nijholt
diminishes because vulnerabilities become visible, and physical, moral, and social
behavioral shortcomings become clear. Clearly, this humor happily accepts the
stereotype characterizations we have of women (blondes), mothers, lawyers, Jews,
Muslims, et cetera. So we can enjoy jokes about blondes that do stupid things,
lawyers that make right what is wrong, or population groups, minorities, or
immigrants that show their stupidity in procedures how to change a light bulb. This
superiority humor also shows when we laugh because someone slips over a banana
peel, someone’s stupid behavior leads to a humorous situation, or when we play a
prank on someone. When modeling humor, one may ask how this superiority aspect
of humor can be included in formal models of humor appreciation and creation.
Relief or release theory is about the functions of humor. Why do we need humor
and why are we open for humor? One answer might be that we love to be superior
to others, but in relief theory, the focus is rather on how humor escapes certain
censors in our consciousness that tell us how to behave and think. Humor addresses
inappropriate behavior and inappropriate thoughts. A society that tolerates humor
allows us to pass these censors and express inappropriate thoughts through humor,
and is the society does not allow this humor it can be expressed among friends and
other trusted members of a community. Inappropriateness does not only refer to
mentioning and make explicit ‘taboo’ behavior (e.g., making references to sexual
acts), but it can also address inappropriate reasoning. There are many jokes that
follow a kind of reasoning that contradicts our cognitive logics, making us aware
how we can be fooled ourselves, being a victim of the joke, but nevertheless feel
relieved when we understand how the joke has played with our cognition and how
we have been fooled.
Finally, we mention the incongruity or the incongruity resolution theory. While
the earlier mentioned theories informed us about why we are happy with and ask for
a confrontation with humor (superiority theories) or why we appreciate and expe-
rience humor that introduces issues that are not assumed to appear in daily life
conversations or interactions (relief theories), the incongruity viewpoint emphasizes
the role of cognition in understanding humor. That is, in understanding a humorous
remark or a humorous situation, a cognitive shift is required, where the shift
requires replacing one interpretation, the initial and usually stereotypical interpre-
tation, with an interpretation that emerges because of additional information that
contrasts the earlier available information. In other than language media, ambiguous
information can be presented in parallel. For example, in a cartoon, we can have a
contrasting visual ambiguity or we can have cross-modal ambiguities, where
information becoming available from one modality contrasts information becoming
available from a different modality. The humor incongruity theory usually refers to
language ambiguities that are resolved while processing utterances. But
non-language or non-language-only incongruity humor can be distinguished as
well. In this chapter, our focus is on incongruity humor. But, incongruities are not
necessarily language incongruities. There can be incongruity between appearance
and behavior, incongruity between words and image, or between a context of use of
a product or tool and its intended use and context of use. Unfortunately, other than
language, incongruities have hardly been investigated. In contrast to the other
humor theories, we can deliberately create incongruities by making variations of an

existing situation, making changes in an existing product, or changing a particular
behavior. Digital technology, its sensors and actuators, allows making changes in
objects, behaviors, and environments in a way that has never been possible before.
3.2.2 Toward Computational Humor?
We introduced the notion of computational humor in 1996 [15]. During a one-week

workshop, humor researchers discussed computational approaches to verbal humor.
A similar workshop was held in 2006 [29]. Also in this case, the focus was on
verbal humor. This focus is understandable given a long tradition of research in
linguistics in general and in computational linguistics (CL) and natural language
processing (NLP) since the advent of the computer. In CL, the main interest is in
modeling the syntax, semantics, and pragmatics of natural language. In NLP, we
look at applications such as machine translation or natural language and speech
interaction with a computer. Clearly, these NLP applications can profit from and
need advancements in CL. Verbal humor can profit from advancements in CL as
well, but rather than following rules of regular language use and language con-
structions, it prefers non-literal language use, wordplay, ambiguities, and unusual
and unexpected changes of context that require reinterpretations of meaning. Hence,
we can understand that modeling verbal humor is not the first priority of researchers
in CL or NLP. But, as becomes clear from the above-mentioned publications and
more recent work that will be mentioned below, there nevertheless is ongoing
research aiming at the computational modeling of verbal humor. Although this
research is not very successful and progress is slow, it nevertheless provides us with
information about why certain activities, admittedly, language design activities, can
be considered to be humorous.
Examples from this research confirm our observations on the role of incon-
gruities in generating humor and how it can be done in non-language situations.
This will be further explored in further sections, in particular sections that are
devoted to using digital technology (sensors, actuators, robotics) to generate humor
or facilitate humorous situations in future smart worlds. But concerning the gen-
eration of verbal humor, we can, for example, mention JAPE [5], a program that
generates puns, HaHa acronym [30], a program that generates humorous acronyms,
programs that auto-correct search questions [36] in a humorous way, and programs
that aim at generating humorous remarks in natural language processing applica-
tions, for example, when having a conversation with a virtual human, a chatbot, or a
social robot [34]. In these latter applications, there need to be some analysis of the
input of the human conversational partner in order to generate a humorous feed-
back. The incongruity point of view with overlap and opposition can easily be
recognized in this research. Unfortunately, recognizing the possibility to generate a
humorous remark from such an analysis and then producing a humorous remark
seem to be far beyond current possibilities to model natural language use in general.
42 A. Nijholt
But generation of verbal humor where the domain and the context limits the way
humor can or needs to be expressed can help to design humorous and enjoyable
applications.
Of course, modeling humorous behavior, humorous interactions, and humorous
situations or events is a task that can be considered as even more difficult as
modeling verbal humor. In previous years, we have seen a growing interest in a
computational approach to the modeling of social and physical behavior of humans
and human daily life activities. This interest became alive with the emergence of
notions such as the ‘disappearing computer,’ ‘ambient intelligence,’ and ‘pervasive
computing.’ These viewpoints assume the existence of computational intelligence
that has become part of our daily life environments. Smart sensors and actuators
being part of an Internet of Things monitor and interpret our behavior and activities
and do not only support them, but also anticipate them. Sensors can collect
audiovisual information, but also tactile and physiological information. Moreover,
these sensors can register and understand explicit commands or requests, and
compute appropriate feedback. Recognizing and interpreting facial expressions,
body postures, gestures, nonverbal speech, and social signals in face-to-face
communication are now well-established research areas. There is also much interest
in tracking individuals, detecting and interpreting (social) group behavior and
multiparty interaction. However, we can repeat what we said about modeling
language behavior; the interest is on modeling regular and stereotypical patterns,
rather than on irregularities, incongruities, unexpected and unusual behavior,
interactions, and activities that we associate with humor.
3.2.3 More on Incongruity Humor
We will spend some more observations on incongruity humor. AI research has

allowed us to use frames and scripts [27] to describe stereotypical situations and
humorous situations that oppose such situations. Unfortunately, there is no way to
generalize a description of one particular humorous situation to an algorithmic
viewpoint that not only knows about understanding humor, but also knows about
gathering elements that are available in a particular situation and that can be
integrated in order to make up a funny situation. Nevertheless, computational
approaches to verbal humor have been the focus of humor researchers that have a
background in computer science and human–computer interaction. Their aim is to
find algorithms that compute humor. That is, algorithms are able to compute the
humorous meaning of a remark, a joke, or a paragraph in a text. Or, algorithms, in a
particular situation, are able to generate a humorous remark, a joke, or a short
humorous text. There is good progress in research on natural language under-
standing and natural language generation. This is not really the case for verbal
humor.
We illustrate the formal approach to verbal incongruity humor with the fol-
lowing example. We have two friends talking to each other:
“Why did you run away from the operation table?”

“The nurse said “Don’t worry, it is a simple operation, no need to get so nervous.””
“She was just trying to calm you down!”
“Yes, but she said it to the doctor!”
We have two stereotypical but ‘clashing’ or opposing situations. The first per-
spective on the situation is that of a professional medical team, ready to do a routine
operation, calming down the nervous patient. The second perspective is that of
someone performing a new task for the first time in public, being nervous, and some
friendly reassurance is provided by a more experienced person supervising the task.
As mentioned earlier, AI research has introduced ‘scripts’ to describe stereo-
typical situations. In this case, the two scripts have a huge overlap, but at the same
time are extremely contrasting or opposing. Our first perspective changes into the
second one because of new information that is becoming available. We make a
cognitive shift from a first interpretation of the situation to a second one. It is not
just a slight misunderstanding, but it requires, due to the opposition in the per-
spectives, a completely different interpretation of the situation. In humor research,
overlapping scripts and strong opposition in scripts are called the ‘necessary’
conditions for humor. Whether they are also necessary conditions or whether other
conditions need to be introduced very much depends on their formalization, and
unfortunately, these formalizations hardly exist. In humor research literature, we
can find some observations on additional conditions. Incomplete and ambiguous
descriptions of a situation allow expectations, overlap, and surprise interpretations.
In addition to the cognitive viewpoint, it is often possible to detect aspects of
superiority [12] or relief in verbal jokes. Often, there is some kind of unexpected
diminishment of one of the main characters when shifting from the first to the
second interpretation of the situation [1], in agreement with a less obvious
stereotypical second situation.
3.2.4 Incongruity and Computational Approaches
It seems that the incongruity theory allows us to conclude that for each particular
joke, we can find two overlapping and opposing scripts with which to explain the
joke, but there is no chance yet that we can design algorithms that with arbitrary
text or arbitrary joke as input recognizes which script (of the myriad of scripts that
are necessary to describe every possible situation) should be chosen and because of
which ‘overlap’ cues the algorithm needs to choose a different, usually less likely
script, and because of which ‘opposing’ cues the algorithm has some certainty in
concluding that humor is intended. Maybe we should call it an unwholesome way
to continue computational humor research with that aim.
There is no chance that we can ever describe all our knowledge with scripts and
can design algorithms to decide when a script should be replaced by a different one
because of new information that is introduced or new topics that enter a conver-
sation or are addressed in a text. Clearly, this is a general problem in artificial
44 A. Nijholt
intelligence and natural language understanding by computers. That is, we may ask
how we get knowledge representation and reasoning modeled and implemented in
order to have a computer understand a joke such as ‘A dyslexic man walks into a
bra,’ among all utterances, it has to understand from its models and reasoning
capabilities that have become part of algorithms that decide that this is funny.
Humor research with the aim to create algorithmic understanding of verbal humor
as it appears in jokes or in spontaneous humor cannot be expected to yield usable
results in the foreseeable future. It certainly does not mean that we cannot learn from
this research. It can help us to understand how to design humor or to introduce
conditions that are necessary for creating humor in a particular situation. We can
sometimes give an incongruous (purposely wrong and contrasting) interpretation to
an utterance. There is overlap, and there is opposition. We can change a drawing in a
cartoon by adding a text that leads to incongruity—again, overlap and opposition.
We can design a product with an appearance that contrasts its function—here again,
overlap and opposition. Sometimes, we create humor by making slight changes to an
environment or events that are meant to confuse others, and their confrontations with
this unexpectedness are humorous. This creation can happen spontaneously, just
seeing the possibility and being in a playful mood. Sometimes, it takes the form of
well-prepared practical jokes. Rather than making physical changes to an environ-
ment in order to generate a humorous situation, we can also think of observing a
particular event and expressing an interpretation that satisfies some of the less
important aspects of the event, but opposes the main characteristics of the event.
Instead of looking at incongruous verbal and physical changes and interpreta-
tions of utterances, drawings, products, and events, we can also think of introducing
new elements and conditions that make it probable that humor will emerge. For
example, we can change a physical environment by throwing a banana peel on the
street. There are two situations, one with banana peel and one without banana peel,
and there is overlap and the banana peel being there is the condition that can change
the non-humorous event of a person walking on the street to an overlapping event
(same person, same street) where the controlled walking changes into a loss of
control of limbs and balance, resulting in a fall—again, overlap and opposition.
Throwing a banana peel on the street is not an incongruous act, and it does not
guarantee that someone will slip on it. The emergence of humor is facilitated, not
guaranteed. But clearly, all elements of humor theory are there. There is overlap,
and there is behavioral incongruity. Moreover, there are superiority and dispar-
agement elements involved, and we feel relieved that it did not happen to us.
3.3 Incongruities: Humor in Cartoons, Products,

and Movies
Our main aim in this chapter is to discuss the ways humor can appear in future,
digitally enhanced environments. But first, we need to make a transition from
humor as it appears in language to humor as it appears and is constructed in other
media. How do these multimodal media environments compare with language

environments? Language is used to tell jokes. Language may be accompanied with
gestures, with facial expressions, and with speech prosody when a joke is delivered.
Obviously, these nonverbal aspects of joke delivery occur and are dependent on the
feedback we receive from our conversational partner(s). We should also mention
that language allows us to create situations that are impossibly or highly improbable
in real life, maybe with the exception of augmented, mixed-reality, or virtual reality
and video game environments.
In this section, we will explore humor as it appears in real life and as it appears
and as it is constructed in cartoons, in products, and in movies. Just as is the case
with jokes, humor in cartoons, products, and movies is designed, and it does assume
a different than consuming role of the human partner in this humor. Instead of
listening (or reading), this humor has to be consumed by seeing only, seeing and
listening, or other combinations of perceptual modalities and multisensory inte-
gration. Humor can also emerge when interacting with a product, a device, or
social, affective and intelligent agents that resemble human beings, such as social
robots or virtual humans. In this section, however, we look at ‘traditional humor’ as
it appears in ‘real life,’ in TV sitcoms, in advertisements, in product design, in
cartoons, on stage, and in movies.
Here, we have already an important distinction. In ‘real life’ humor, we can be
active as a creator of humor, we can audience humor (whether we are creator or
not), we can be involved as a participant that makes a humorous situation possible,
and we can be ‘victim’ of humor. When using products, looking at cartoons,
watching TV, and going to the cinema, we consume humor, and although our sense
of humor is addressed, there is no real-time interaction between us and the
humor-sensitive generating device, product, or environment.
But clearly, as it is in the case of jokes, when looking at a cartoon, when being
surprised by a product, and when laughing when seeing a particular movie scene,
we draw upon knowledge that comes from education and real-world experiences.
For that reason, we will first look at humor as it appears in traditional real-life
situations.
3.3.1 Humor in Real-Life Situations
Various authors have introduced inventories of humor. They contain lots of dif-
ferent kinds of linguistic humor, so, here, we extract the other forms of humor that
are mentioned and that receive usually much less attention and explanation. These
inventories show us what we consider humorous. What kinds of events or behavior
or interaction are considered to be humorous? There is no explicit explanation why
we consider them humorous. There is no development of theory. But we certainly
can analyze many or maybe all items in these inventories in terms of the three
humor theories we mentioned before (superiority, relief, and incongruity theory).
However, this has hardly been done. Modest attempts to analysis can be found
46 A. Nijholt
when we see similar situations appear in movies, on stage (comedy), or in other

controlled situations (for instance in games or in music). In [19], we find five
categories of humor as it can occur in objects, persons, or situations. We take the
descriptions of these categories as they appear in [23].
• Deficiency in an object or person. This is about physical deformity, ignorance or
stupidity, moral shortcomings, or actions that fail. Hence, it is about inferiority,
weakness, and absentmindedness. In the case of actions that fail or performed
awkwardly, it can also because of a defective tool or a chance event. In a
practical joke, someone is deliberately causing a failure of action.
• One thing/situation seeming to be another. This is about mimicry/imitation,
imposter, pretense, and mistaken identity. Comedies often depend on mistaken
identity or situation. Anthropomorphizing animals, inanimate things, or
sequences of events also belongs to this category.
• Coincidence in things/situations. For example, unexpected repetition in events
or, for example, in comedy, in lines. Everywhere we expect uniqueness,
unexpected repetition can have a humorous effect.
• Incongruous juxtaposition. The incongruous effect is obtained by having
physical, social, and psychological opposites appear together in a situation. In a
comedy team, we can have physical differences (short versus tall, thin versus
fat) and personality differences that help to lead to funny situations. But we can
also get incongruous juxtaposition when, for example, a vagabond takes a
dinner in an expensive restaurant.
• Presence of things in appropriate situations. This category can be generalized to
the presence of ‘things’ (objects, people, behavior, opinions) in situations where
they can be considered as inappropriate, or to situations where sequences of
events inappropriately intersect. Clearly, not necessarily leading to a humorous
situation, but certainly helping to let a humorous experience occur.
One important difference with incongruity humor as we saw it in language jokes
is that in order to perceive an incongruity, there is not always the necessity to have a
strict sequential processing of information. In jokes, we have a setup a stereotyped
situation, and then new information comes in, usually the punch line, which forces
us to reinterpret the previous information. In real life, we can have conflicting or
ambiguous perceptual information that is presented at the same time. There is more
immediate display of incongruities. Sound may contrast visual information. Devices
can function in unexpected ways. Events, situations, and objects often allow dif-
ferent viewpoints and even contrasting viewpoints. Hence, they allow different,
sometimes opposing, interpretations.
Also in [3], a category ‘action humor’ can be found. Berger and Morreal do not
address multimedia generation of humor. In [7], Berger’s typology is adapted and
used to analyze humor in TV commercials. Their main categories are (1) slapstick,
(2) clownesque humor, (3) surprise, (4) misunderstanding, (5) irony, (6) satire, and
(7) parody. There are no observations that address particular modalities or incon-
gruities that cross modalities. Such observations would allow us to discuss how in
multimedia display of events, allocating different modalities to different aspects of a

generated event, incongruities between modalities can be introduced. Such an
approach, not for the creation of humorous events, but for the design of humorous
products can be found in [18]. We will return to this research later.
Interestingly, when returning to the issue of giving some explanation or theo-
retical foundation to why the items mentioned in these inventories have been
included, we need to look at a very early twentieth-century essay by the French
philosopher Bergson [4]. Rather than starting with observations on his main
interest, explaining humorous events in comedy (French comedy, e.g., Molière), he
took examples from real life and metaphorical, mechanical, views on real life to
explain why events, activities, or behavior can be humorous. Freud, when intro-
ducing his views on the function of humor, also made references to technology such
as the buildup and release of tension as can be seen in the functioning of steam
engines. Bergson did not discuss the function of humor, but he tried to explain
humor by looking at its appearance and how this contrasts with more regular
appearances of events. Bergson’s view is about incongruities that contrast natural
and regular behavior and events with a mechanical viewpoint where events include
a human being not able to adapt to a mechanically progressing situation or adapting
to this situation in an unexpected and surprising way. So, Bergson introduces
‘mechanical inelasticity’: ‘We laugh every time a person gives us the impression of
being a thing.’ Physical inelasticity and mental inelasticity are distinguished, where
‘physical’ refers to lack of flexibility to adapt behavior and movements to the
unexpected and ‘mental’ refers to making wrong presuppositions, assuming that
things will happen as usual, and have wrong and ‘mechanical’ interpretations of
events. He introduced three ‘mechanical’ principles that provide us with views on
how to analyze and create humorous situations: (1) repetition, e.g., the snowball
effect (simple and innocent action starts an unstoppable series of unplanned events),
(2) inversion (e.g., role inversion, a child tells a grown-up how to behave), and
(3) reciprocal inference—different series of events (story lines) that meet, have
overlap, mix, and are opposing and hence allow different interpretations. In this
way, Bergson identifies some underlying principles of incongruities as they appear
in the inventories mentioned above (see also [21] for a more detailed explanation of
Bergson’s views on humor).
3.3.2 Humor in Cartoons and Movies
Cartoons can be studied as drawn jokes [14], but cartoon drawings usually contain
text in captions or text balloons, offering many extra cross-modal incongruities, that
is, incongruities between text and drawing. We can have absurdities and can have
meta-cartoon humor, where a character comments to the reader about its behavior
or where the cartoonist enters the drawings or plays with the frame in which the
drawings have to appear. In cartoon drawings and accompanied text, there is no
need to follow laws of logic, physics, or even the regular laws of cartoon drawing.
48 A. Nijholt
In a more extreme form, this is also the case for animated movies. In these cartoons,
we have the extra dimensions of change of environment, nonverbal behavior,
movement, speech, and sound. There is no problem, on the contrary, because it
allows extra humorous effects, to introduce incongruities that are mentioned in the
typologies that we introduced in the previous subsection. Moreover, since the
movie director has fully control over his tools, not differently from a stand-up
comedian that configures language constructs to create a joke or a cartoonist that
draws a cartoon or a series of cartoon drawings that end with a punch line drawing,
this movie director can also introduce incongruities that follow from not obeying
laws of logic, laws of physics, and laws of behavioral conduct. In his autobiog-
raphy, Chuck Jones, author of the Roadrunner cartoons, explains his laws of car-
toon creation [16]. A playful view on the laws of physics in cartoon movies can be
found in [25]: ‘O’Donnell’s Laws of Cartoon Motion.’
Obviously, we can find lots of humor in comedy movies with Buster Keaton,
Harold Lloyd, Laurel and Hardy, and Charles Chaplin. In [8], an attempt is made to
characterize and explain this humor. In fact, Carroll provides excellent explanations
of humor exploitation in silent movies. He considers ‘sight gags,’ and here, we
summarize his observations as we did in [22]. Carroll mentions that a sight gag
provokes amusement because of the juxtaposition of incompatible interpretations.
He identifies six distinct categories of sight gags in films, and in each of them, the
directors appear to play with different interpretations. We can recognize such
playing with interpretations in his opinion, from the perspectives of both the
spectator (the viewer of the film) and the characters in the film. The six categories
that are mentioned are as follows:
(1) The mimed metaphor, where we can see an object either literally or figura-
tively (Charlie Chaplin treating a boot as a meal);
(2) The switch image, where we are presented with a view on a particular situation
or event, but when zooming in or out, or with a change of camera position we
learn that we misinterpreted the initial, visually ambiguous, scene;
(3) The switch movement, where an actor attempts to have his behavior reinter-
preted (e.g., from inappropriate to appropriate) by other characters in the film;
(4) The object analog, similar to the mimed metaphor in the sense that an object is
used or treated in an unusual way, but it has similarities to an object that is
meant to be used that way; again, this requires two interpretations: One is the
literal one and the other is the metaphorical one;
(5) The solution gag, maybe not completely distinct from the previous categories,
where the audience enjoys the wit of the protagonist to escape from a
threatening situation by behaving or using tools in incongruous ways;
(6) The mutual interference or interpenetration of two (or more) series of events
(or scenarios).
As mentioned by Carroll, this latter category is the most frequent form of the
sight gag. Series of events can be staged with the director’s aim to produce different
plausible interpretations. Creating different points of view that are plausible can be
aimed at the audience; that is, they can be aimed at fooling the audience. But it can
also be the case that the audience is aware and gets its enjoyment from the char-
acters that are not aware of an interpretation of events that is available for the
audience. In both cases, the audience can enjoy what is happening. There are
incongruities to be resolved by the audience, and there are incongruities that can be
observed and enjoyed by the audience while watching the characters trying to deal
with them.
As we noted in the previous section on real-life humor, in contrast to the telling
of jokes, there can be both a sequential way of presenting or detecting incongruities
and there can be a more immediate display of incongruities. The scenes do not
change when we become conscious that there is more than one interpretation, which
means there is complete perceptual overlap. The humorous effect is obtained by
opposing interpretations (e.g., made possible by a changing camera view, or by
emphasizing a possible metaphorical view), going from inappropriate to appropriate
behavior or the other way around, or using a tool or handling an object in an
unusual way. We will return to this latter aspect in the next subsection.
Notice also that in contrast to the typology of Morreal, there is explicit attention
for the role of the creator of humor and possible ways, using camera techniques and
design of narratives, to mislead the audience in order to create a humorous event.
There is also the possibility to make the audience an ally in the creation of a
humorous event by letting them have more information about situations than is
available for those who have to suffer a prank. Clearly, an audience can also enjoy
humorous confusion and humorous misunderstandings that can occur because of a
director’s playing with narratives where there is no ‘suffering’ at all. As our aim in
this chapter is to collect views on humor creation, rather than on understanding,
these are useful observations from the point of view to use them when we want to
discuss the creation of humor in smart environments where rather than ‘just’ a
camera we have digital technology in the form of sensors, actuators, multimedia,
and computing power at our disposition.
3.3.3 Humor in Products
It is certainly the case that in cartoons and comedy movies, directors, script writers,
composers, and artists (movie stars) are involved in a humor creation processes.
But, obviously, humor usually appears in every movie where actors interact and get
involved in events. In cartoons such as the Roadrunner or Pink Panther cartoons,
there is no need to follow whatever rules of every day’s common sense, appropriate
behavior, and regular physics require. In common with jokes, cartoons, and movies,
products are designed. And designers are using their freedom to design humorous
products. In smart environments, we will have smart objects and tools. We can
interact with them using different modalities (speech, touch, smell, taste, vision),
and they can enter the interaction using their intelligence and its translation to a
multimedia interaction act display. However, before being able to discuss such
50 A. Nijholt
issues, we need to collect views on the design of humorous products. Unfortunately,

but not different from views we collected in the previous sections on jokes, real-life
events, cartoons, and movies, digital technology has not yet a role in observations
on the design of humorous products. But obviously, various kinds of incongruities
can be distinguished when we look at products.
A product designer delivers products that have a particular representation, a
particular function, and a particular context of use. Hence, we can have an
incongruity in each of these viewpoints. In an excellent paper by Yu and Nam [38],
such incongruities are investigated. In this paper, product design is looked upon
from a humor relief theory perspective, a humor superiority theory perspective, and
a humor incongruity theory perspective. As mentioned earlier, relief and superiority
theories are mainly about the function and the effect of humor. This does not
necessarily mean that we cannot take these aspects into account when intentionally
creating humor. However, we will not do this here and focus on the creation of
humor by introducing incongruities.
We illustrate the incongruities mentioned in [38] with some of their examples.
We can have an incongruity in representation, for example, shape incongruity,
where bathroom mat seems to be composed of egg scales and we are afraid to step
on it, or a lamp that has the form of a matchstick. We can look at the operational
aspect of a product: What is its function and can we achieve a humorous effect by
giving a familiar product an unexpected function? One example is a balloon that has
to be blown in order to display the address details of a chest physician, that is, have
the balloon functioning as a visiting card. We can look at incongruities from the
context of use point of view. One example is to attach a computer keyboard to a
wall and use it as a key storage rack. Another example is a key ring that is plugged
in into a fake socket near your door and when unplugged reminds you of
unplugging unnecessary appliances, saving energy and money.
There are some remarks that need to be made about this research and that require
further investigations, as is the case with all good research. Firstly, when having
observations about incongruities in each of the three aspects, it is desirable and it
should be possible to distinguish between perceptual modalities that create or are
helpful in creating incongruities. For example, we can have incongruity in repre-
sentation or appearance, but we may ask what has caused this incongruity and
where we have opposing information obtained from our observations. From an
appearance point of view, we can get opposing information from different sources,
even when the overlap (whatever way we experience it, it is and remains a bath
map) is a physical hundred percent. Hence, we can have cross-modal incongruity
while experiencing a particular product. Its taste can be different from what we
expect, its smell can be different from what we expect, and its touch or its sound can
be different from what we expect. For example, squeezing a bath duckie may cause
it to roar like a lion instead of giving a quacking sound. Such cross-modal
incongruities have been investigated in [18].
We can learn from these approaches to humor in product design, as we can learn
from humor in cartoons and movies, and humor in real-life and in jokes.
Congruities can appear in context, function, and appearance. Cross-modal
incongruities can appear inside each of these aspects and between these aspects.
More detailed views on incongruities have not yet been explored.
3.3.4 Humor in Video games
Spontaneous humor appears in real life. Spontaneous humor can also appear in
game environments. Video games are synthesized environments that allow us to
interact with the environment or with other gamers in competing or collaborating
ways. We can communicate and negotiate with others about our contributions to a
game, we can take part in activities, and we can explore game environments on our
own. There are games that have been designed with the aim to have humorous
interactions between gamer and game environment.
Unlike the passive listening and seeing experience when someone tells us a joke,
when we watch a sitcom or a humorous TV commercial, when we watch a stage
performance, or when we watch a movie, video games are about interaction. In
principle, a user’s interactions are anticipated, including the many ways a user can
fail to perform a certain interaction. Game designers, who maybe not the designers
of first-person shooter games, have introduced humor in games, for example, in
adventure games, games where users have to exploit the environment and have to
make decisions about how to continue, and while doing so get humorous com-
mentary and sometimes are confronted with humorous actions of the non-playing
characters (NPCs) or humorous behavior of the environment. In some games,
players are challenged, sometimes implicitly, to reach a goal as fast as possible.
Other games can be played more leisurely and allow relaxed social interaction. In
massively multiplayer online role-playing games (MMORPGs), we have teams of
players that discuss strategies and try to humiliate and destroy opponents, preferring
creative and humorous ways to do so.
In games that employ humor, we can find back the incongruity techniques that
were mentioned in the previous typologies. At this moment, there is not sufficient
artificial intelligence that allows situation-aware autonomous behavior of NPC.
Hence, it is almost impossible to give them an active role in attempts to create
spontaneous humor. Although it is not spontaneous, in good games ‘canned’ humor
is integrated in the game similar to the integration of sound and music in a game.
Situational humor can be introduced in MMORPGs by actions of human players
without that NPCs and game environment are aware of this and without a game
designer intentionally having facilitated this kind of humor. On the contrary, humor
often emerges because of this unawareness. We enter a gray area between canned
humor and situational humor when the game designer introduces ‘laws’ in a
physical engine that do not match with what we are used to in our daily life
environments. Maybe we can walk through walls, see through walls, become
invisible, or have eyes in the back of our head. Having unusual game physics, weird
shortcuts, non-Euclidean geometries, intriguing perspective play, and other sur-
prising game elements (e.g., the use of panels and the portal mechanics in the Portal
52 A. Nijholt
game) will certainly help to make a gamer’s act intentionally or unintentionally

humorous. But, and Portal is a nice example, verbal commentary on the actions of
the gamer provides a nice way to make a game humorous.
However, there is a different kind of humor that has found popularity with
gamers and in game communities. This is what we call mischievous game humor.
Although the majority of gamers are happy to play a game or have a virtual life in a
game environment, there are also lots of gamers that find satisfaction and enjoyment
in playing a game in a way that was not intended by the designers. They try to hack
the game engine and the reward system, giving them an advantage to other players.
But they can also look for ‘accidental humor,’ by finding and exploring bugs and
weak points in a game design to create incongruous situations and strange behavior
of their own or others’ game characters. The general idea is that a game designer
cannot anticipate every action a gamer can take and therefore cannot always choose
an appropriate response for each of these actions. A gamer can choose to make
unlikely decisions or to follow unlikely continuations of a conversation with a NPC.
These activities are sometimes recorded in humorously commented ‘walk-throughs’
of a game or in Machinema [32], edited recordings of strange avatar behavior
because of shortcomings in game design and game engine, for example, in the
collision detection algorithms.
In some games, for example, in Sims 4, this humor has been given game rules.
You can offend other players, slap them, or use a rule that turns your avatar in a
streaker. More interestingly is this mischievous humor when a gamer uses his
knowledge of the game environment or is able to make changes to the game
environment in such a way that he can play a hoax on someone or disturb the game
play of others. Trolling [6, 33] is a rather innocent form of this behavior. There are
communities of gamers that find pleasure in doing this, they share knowledge and
experiences, and they sometimes team up to cause distress to other players. When
they purposely aim at annoying other players, this is called griefing [13], and in
game communities, this is considered to be antisocial behavior. Game hackers have
fun in finding ways to modify game event decisions. Game hackers sometimes
collaborate in virtual teams to attack a particular game and have fun finding
weaknesses in the design.
3.4 Humor in Smart Environments
The notions of smart environments, pervasive computing, and ambient intelligence

all refer to situations where our environment, whether it is in our home, in our
office, or in a public space, is enhanced by digital technology that is able to monitor,
anticipate, and support our activities, knowing about our preferences, our person-
ality, our emotions, and our moods, and, maybe, also knowing and learning about
our sense of humor and adapting to it. However, we will also have the possibility,
maybe not everywhere and not always without constraints, to configure our own
sensor and actuator (robotics) configurations and to explore smart environments,
and to modify them to our preferences, including a preference to introduce

humorous situations.
In ‘real life,’ we have humorous interactions, situations, and events. This real life
is full of technology. In the humor typologies, we mentioned and even in the design
of humorous products that we discussed that technology does not play a role.
However, we know that whenever new technology is introduced, it is often
accompanied with humor. In particular, in movies, clumsy or not intended use of
new technology is exploited: from assembly line (Charlie Chaplin) to a modern
kitchen (Jacques Tati). First-time use of new technology can lead to humorous
situations. New technology, especially extraordinary flexible and from day-to-day
modifiable digital technology, can give us the opportunity to creatively play a trick
on our colleagues and roommates.
Also in [28], shortcomings of technology are mentioned as humor-creating
opportunities. In his eyes, this is in particular true when we attempt to provide
interaction technology with human-like properties. This coincides, but now from a
digital viewpoint, with Bergson’s ‘mechanical’ viewpoint. There is human-like
behavior, but it has a humorous effect because of its imperfectness. Silber also
refers to Linda Stone’s [31] notion of ‘continuous partial attention’ and her
observations more frequent scanning behavior and multitasking that we see happen
nowadays due to parallel streams of information that is delivered to us (public
displays, smartphones, tablets, laptops). This can lead unintended juxtapositions,
mental mismatches, and incongruities in our perception. This partial attention can
be compared with Bergson’s observations that humorous situations often happen
because of absentmindedness.
Video games are not possible without smart technology. In fact, video games can
be considered as interactive smart environments. However, although these smart
environments address activities in physical worlds, whether they are real or fantasy
worlds, the worlds are virtual worlds. There are ‘sensors’ and ‘actuators’ acting
inside these worlds, controlled by a game engine and taking responsibility of the
behavior of the game characters, the events, the narrative, and the interaction with
the gamer. The smart environments that are now becoming part of our world or are
becoming the real world have digitally controlled sensors and actuators and monitor
and guide (or control) the behavior of its inhabitants. Rather than a game engine, we
have smart environment engines. So we may ask whether the various kinds of
humor we mentioned in the previous subsection on humor in video games can be
expected to happen in smart environments as well.
In the next short subsections, we have some additional observations on these
aspects of introducing humor in smart environments.
3.4.1 Smart and Playable Environments
We have become familiar with the notions of pervasive computing, the ‘disap-
pearing’ computer, and the Internet of Things. Smart technology becomes
54 A. Nijholt
integrated in our domestic and work environments, but also in public and urban
environments. Sensors and actuators (or robotics) will be everywhere, whether it is
in our home, our kitchen, the kindergarten, the school, concert hall, museum or
stadium, or on the streets. And, obviously, we carry them with us, whether it is in
our smartphone, smart watch, or smart glasses or in the clothes we are wearing.
This smart technology can introduce more smartness in our life, helping us to lead a
more active and efficient life.
However, there is more than efficiency in our daily life, our daily activities, and
our daily interactions with partners, family members, colleagues, or whoever we
meet during a day, which is not efficiency-oriented. Often it will be not. Therefore,
it is rather unsatisfactory that social, playful, and humorous aspects usually are not
addressed. They are of course the most difficult issues to address, and they are
addressed in research on affective computing, on social behavior of virtual agents
and robots, and sometimes in research on tangible interfaces. Rather than seeing
such research reflected in the design of smart environments, there are initiatives to
introduce such aspects in the context of digital cities. The notion of ‘playable cities’
has been introduced to emphasize that smart technology can be used to introduce
playful elements in a city. It also provides a role for the city dwellers [10, 24]. They
are assumed to play and contribute, rather than just experience and consume. As an
example, we can mention the ‘Hello Lamp Post’ project in Bristol (UK) where
citizens were given the opportunity to communicate with lamp posts, mailboxes,
and other street furniture that kept a memory of such conversations and used it in
subsequent interactions with other passers-by.
3.4.2 User Initiatives: Exploration and Design

in Smart Spaces
There are some tendencies we should mention to support our views. Nowadays, we
see successful attempts to engage children in designing and programming games
and playful applications and tangibles that use sensors and actuators (robotics) with
the help of tools such as Scratch, Makey Makey, and simple microprocessors such
as the Arduino microcontroller [9]. Children become grown-ups who, despite that
they are not necessarily experts and computer scientists, know that tinkering with
digital technology can be fun, and they can pimp, personalize, and modify con-
figurations on their own and in addition design their own playful extensions of their
home environment. As an example, in [2], an overview of objects is presented that
have been designed by students in a Delirious Home project, a smart home with a
sense of humor.
Home activities can be extended to activities in the neighborhood. In human–
computer interaction research, there is already a tendency to design for commu-
nities and engaging communities in design that addresses their local issues. For
example, in the Neighborhood Networks project [11], a research group cooperated
with a local community and supported them in learning about sensors and actuators
and implementing them in their neighborhood to monitor traffic and air pollution
and triggering playful alerts when speeds were too high or air quality too low. Other
projects have looked at providing public artists or political activists with tools that
can support their activities [17].
In [37], commonalities in these creative practices are discussed. They mention
inventive leisure practices, in which activities such as hacking, tinkering, DIY, and
crafts are included. Whether it is about game environments, playful and smart cities,
domestic settings, or neighborhood activities, the following similarities can be
distinguished: sharing of ideas, resistance, repurposing and challenging authority,
skill development, learning by teaching, managing reputation, being member of a
social group, sharing norms, and senses of identity. Hence, it is not a big step to go
from mischief humor in games to mischief humor in smart and playable environ-
ments, including cities. In game environments, our activities are monitored and
restricted by game rules. They take effect by virtual sensors and actuators. In
digitally enhanced physical environments, we have physical sensors and actuators,
and rather than just aiming at efficiency, they can be employed for generating
humorous situations as well. These digitally enhanced physical environments are
already used for gaming. Various video games have been translated to games that
can be played in (digitally enhanced) physical environments. Urban games are
designed that are making use of the wearables of the players and location-based
sensors.
Our future life will take place in environments that resemble game environments
and that are cocontrolled by engines (kitchen engines, home engines, public space
engines, etc), similar to game engines. This smart technology can be equipped with
scripted humor, but we can as well expect ‘accidental,’ spontaneous, and mischief
humor, as we mentioned before for game environments. Moreover, we can expect,
again similar as in game environments, the emergence of communities or short-term
collaborations, for example in hackathons, in order to find weak spots, make
modifications [35], and ways to ‘play’ the smart environment or the digital city in
ways never intended by the designers. Hackers can collaborate to change config-
urations of sensors and actuators in such a way that humorous situations can occur.
Obviously, also comparable with game environments, there will be antisocial
behavior, and not every sense of humor will be appreciated by the ‘victims’ of such
humor.
3.5 Conclusions
In this chapter, we surveyed the various ways humor appears in our life. Rather
than, as is usually the case, having a focus on verbal humor, we investigated humor
as it appears in situations and events, both in real life and in movies, in products,
and in game environments. We argued that humor will become part of smart
environments. Using sensors and actuators, designers can design humor using our
observations or they can facilitate humor creation by others by providing tools. By
56 A. Nijholt
observing what is happening in game environments and in game communities and

arguing that our future smart environments will resemble game environments, we
can expect similar kinds of humor as it appears in game environments, such as
spontaneous humor, scripted humor, trolling, griefing, and hacking. Not all kinds of
humor will be appreciated by the recipients.
References
1. Apter MJ (1982) The experience of motivation: The theory of psychological reversals.

Academic Press, London
2. Bellet A, Kabel C (2014) ECAL Delirious Home. ECAL University of Art and Design
Lausanne, Switzerland. Document is downloadable from http://www.ecal.ch/download/
wysiwyg/ea8a5a6c9051e6d819c0431fb0b29478.pdf/TextsPresentationCaptions.pdf
3. Berger AA (1993) An Anatomy of Humor. Transaction Publishers, New Brunswick, NJ (first
edition in 1976)
4. Bergson H (1900) Laughter. An essay on the meaning of the comic. Translated from Le Rire.
Essai sur la signification du comique. Gutenberg project (2003)
5. Binsted K, Ritchie G (1997) Computational rules for generating punning riddles. HUMOR 10
(1):25–76
6. Buckels EE, Trapnell, PD, Paulhus DL (2014) Trolls just want to have fun. Personality and
Individual Differences 67:97-102
7. Buijzen M, Valkenburg P (2004) Developing a Typology of Humor in Audiovisual Media,
Media Psychology 6(2):147-167
8. Carroll N (1996) Theorizing the Moving Image (Cambridge Studies in Film). Cambridge
University Press, Cambridge
9. Davis D, Kafai Y, Vasudevan V, Lee E (2013) The education arcade: crafting, remixing, and
playing with controllers for Scratch games. In Proceedings of the 12th International
Conference on Interaction Design and Children (IDC ‘13). ACM, New York, NY, USA,
439-442
10. de Lange M (2015) The Playful City: Using Play and Games to Foster Citizen Participation. In
Social Technologies and Collective Intelligence, A. Skaržauskienė (ed.), Mykolas Romeris
University, Vilnius, 426-434
11. DiSalvo C, Louw M, Coupland J, Steiner M (2009) Local issues, local uses: tools for robotics
and sensing in community contexts. In Proceedings of the seventh ACM conference on
Creativity and Cognition (C&C ‘09). ACM, New York, NY, USA, 245-254.
12. Grönroos AM (2013) Humour in Video Games: Play, Comedy, and Mischief. Master’s Thesis,
Aalto University, School of Art, Design and Architecture, Department of Media, Media Lab,
Helsinki
13. Gruner, CR (1997) The Game of Humor. A Comprehensive Theory of Why We Laugh.
Transaction Publishers, New Brunswick, USA
14. Hempelman CF, Samson AC (2008) Cartoons: Drawn jokes. In (Raskin, 2008), 609-640
15. Hulstijn J, Nijholt A (eds.) (1996) Computational Humor: Automatic Interpretation and
Generation of Verbal Humor. Proc. Twente Workshop on Language Technology 12,
University of Twente, Netherlands
16. Jones C (1989) Chuck Amuck: The Life and Times of an Animated Cartoonist. Farrar, Straus
and Giroux, New York
17. Kuznetsov S, Paulos E, Gross MD (2010) WallBots: interactive wall-crawling robots in the
hands of public artists and political activists. In Proceedings of the 8th ACM Conference on
Designing Interactive Systems (DIS ‘10). ACM, New York, NY, USA, 2010, 208-217
18. Ludden GDS, Kudrowitz, BM, Schifferstein HNJ, Hekkert P (2012) Surprise and humor in
product design. Designing sensory metaphors in multiple modalities. Humor 25(3):285-309
19. Morreal J (1983) Taking Laughter Seriously, State University of New York Press, New York
20. Mulder MP, Nijholt A (2002) Humour Research: State of the Art. Center for Telematics and
Information Technology. University of Twente, the Netherlands
21. Nijholt, A (2015) Incongruity Humor in Language and Beyond: From Bergson to Digitally
Enhanced Worlds. (Invited) In: Fourteenth International Symposium on Comunicación Social:
retos y perspectivas, Volumen II, Ediciones Centro de Lingüística Aplicada, Santiago de
Cuba, Cuba, 594-599
22. Nijholt A (2014) Towards Humor Modelling and Facilitation in Smart Environments.
Proceedings of the 5th International Conference on Applied Human Factors and Ergonomics
(AHFE 2014), T. Ahram, W. Karwowski, T. Marek (Eds.), Krakow, Poland, 2992-3006
23. Nijholt A (2015) The Humor Continuum: From Text to Smart Environments (Keynote Paper).
Proceedings International Conference on Informatics, Electronics & Vision (ICIEV),
Kitakyushu, Fukuoka, Japan, IEEE Xplore, 1–10
24. Nijholt A (2015) Designing Humor for Playable Cities. Proceedings of the 6th International
Conference on Applied Human Factors and Ergonomics (AHFE 2015), T. Ahram, W.
Karwowski (Eds.), Section Advances in Affective and Pleasurable Design. Procedia
Manufacturing (3C), Edited By Yong Gu Ji and Sooshin Choi, Elsevier (ScienceDirect),
Las Vegas, USA, 2178–2185
25. O’Donnell M (1985) Elementary Education. An Easy Alternative to Actual Learning.
Alfred A. Knopf, New York
26. Raskin V (ed.) (2008) The primer of Humor Research. Mouton de Gruyter, Berlin, Germany
27. Schank R, Abelson R (1977) Scripts, plans, goals, and understanding: An inquiry into human
knowledge structure. Lawrence Erlbaum Associates Hillsdale, NJ
28. Silber MJ (2013) Digital Humor Theory. M.Sc. Thesis, School of Art and Design, Pratt
Institute, New York, USA
29. Stock O, Strapparava C, Nijholt A (eds.) (2002) The April Fools’ Day Workshop on
Computational Humour. Proc. Twente Workshops on Language Technology 20 (TWLT 20),
ITC-IRST, Trento, Italy
30. Stock O, Strapparava C (2003) Getting Serious about the Development of Computational
Humor. In: Proceedings of the 18th international joint conference on Artificial intelligence
(IJCAI’03). Morgan Kaufmann Publishers Inc., San Francisco, CA, USA, p 59-64
31. Stone L (2006) Attention: The *Real* Aphrodisiac. Talk at the Emerging Technology
Conference, San Diego, CA, 2006, http://itc.conversationsnetwork.org/shows/detail739.html
32. Švelch J (2014) Comedy of contingency: Making physical humor in video game spaces.
International Journal of Communication 8, 2530-2552
33. Thacker S, Griffiths MD (2012) An Exploratory Study of Trolling in Online Video Gaming.
International Journal of Cyber Behavior, Psychology and Learning (IJCBPL) 2(4):1-17
34. Tinholt HW, Nijholt A (2007) Computational Humour: Utilizing Cross-Reference Ambiguity
for Conversational Jokes. In: Masulli F, Mitra S, Pasi G (eds.) WILF 2007, LNAI vol. 4578,
Springer, Heidelberg, p 477-483
35. Unger A (2012) Modding as Part of Game Culture. In Computer Games and New Media
Cultures. A Handbook of Digital Games Studies, J. Fromme, and A. Unger (eds.). Springer,
Dordrecht, Netherlands, 509-523
36. Valitutti A, Toivonen H, Gross O, Toivanen JM (2012) Decomposition and Distribution of
Humorous Effect in Interactive Systems. In: AAAI Fall Symposium: Artificial Intelligence of
Humor 2012, AAAI Technical Report FS-12-02 Artificial Intelligence of Humor, 96-100
37. Wang T, Kaye J (2011) Inventive leisure practices: Understanding hacking communities as
sites of sharing and innovation. In Proceedings CHI ‘11 Extended Abstracts on Human Factors
in Computing Systems, ACM New York, NY, USA, 263-272
38. Yu Y, Nam TJ (2014) Let’s Giggle! Design Principles for Humorous Products. DIS 2014,
June 21–25, 2014, Vancouver, BC, Canada, 275-284
Chapter 4
Engineering for Human Pleasure
Masashi Okubo
Abstract We engineers are trying to let human’s life wealthier. Because a person
will be able to carry out a more intellectual activity, it will be necessary to relieve
the human of complicated work. The engineering have been developing for relaxing
people as one subject until now. However, to be able to do something easily may
take pleasure and a motivation from them. The system has to understand the
relationship between the user and the system and the situation to motivate the
person, and it is necessary to have the function that can change its service for
adaptation to circumstances. In this chapter, an idea is demonstrated to design the
system to achieve the motivation of the person.
4.1 Introduction
Chinese character “楽” has meanings of “RAKU” and also “TANOSHI.” In

English, “RAKU” means relax and easy, and “TANOSHI” means enjoy and fun.
Not being noticed usually, these words have completely different meanings. Sitting
on a sofa and taking a rest makes us feel relaxed. But does it make us enjoyable?
You must sweat a lot to enjoy playing soccer, and making you sweat a lot is not a
relax situation and easy staff. Many accomplishments brought by postwar engi-
neering developed in Japan have facilitated various things. There are some products
making even pleasure easy. From the standpoint of human’s motivation, it is
considered that people can be more motivated by making themselves enjoyable than
by being themselves easy. That is, the direction of engineering needed to be
reconsidered to motivate people. The paradigm shift from the engineering that
makes a person easy to the one that can inspire his/her motivation is necessary.
Professor Kawakami and his colleagues have proposed a concept of “Fuben-eki”
M. Okubo (&)
Doshisha University, 1-3 Tatara-Miyakodani, Kyotanabe, Kyoto 610-0321, Japan
e-mail: mokubo@mail.doshisha.ac.jp

DOI 10.1007/978-3-319-29433-9_4
60 M. Okubo
[1, 2]. There have been things to lose by convenient products and systems con-
sidered to be useful in our daily life. Fuben-eki is an effort to design products and
systems in taking advantage of having inconvenience consciously, which bring
benefits. On the other hand, sometimes, facilitation leads to further motivation and
inconveniences lead to discouragement. In short, in order to take full advantage of
these ideas, individual’s personality and psychological state are necessary to be
considered adequately.
4.2 Flow Theory
A psychological concept called “flow theory” proposed by Mihaly Csikszentmihalyi,

psychologist, has been paid much attention especially in sociological field [3, 4]. Flow
is the mental state of operation in which a person performing an activity is fully
immersed in a feeling of energized focus, full involvement, and enjoyment in the
process of the activity, and this concept has been widely adopted in various fields.
Flow is considered to be experienced when a person perceives that his/her skill level
matches the challenge level of the operation (see Fig. 4.1). However, sometimes, the
engineering products lead our state of mind from this flow area. On the other hand, it is
considered that video games can induce flow state easily [5, 6]. Therefore, we
investigated the elements of flow theory from the viewpoint of the player’s psycho-
logical mind.
Fig. 4.1 Flow is considered

to be experienced when a Task
person perceives his/her skill
level matches the challenge Game system
level of the operation,
However, sometimes, the
engineering products lead us
from this state of mind. On
the other hand, video game is
considering the relationship
between player’s skill and Flow
game situation
Engineering
products
Skill
4 Engineering for Human Pleasure 61
4.3 Proposal for Approach for Cooperative-Type Fighting

Game Based on Player’s Personality
The author of this article has indicated the effectiveness in designing the game
situation focusing on the user’s motivation and performance in the previous
researches on the game system [7]. This chapter is to explore the direction of
engineering in next generations based on the research outcomes.
4.3.1 Cooperative-Type Fighting Game
Not only a conventional video game such as one-to-one fighting type, but also a
cooperative-type game in which teaming up with the other player to beat their
enemy or reach the goal is getting popular recently. Such cooperative-type fighting
game is likely to get a player servilely and bored depends on the relationship
between the player and the partner. Besides, the enemy’s skill is also seemed to be
affected to the player’s psychological state. Therefore, inspiring player’s motivation
and great performance is necessary to be considered on making a team as shown in
Fig. 4.2. This chapter examines an impact of the relationship of skill within a team
or between a team on the player’s performance and psychological state in
cooperative-type fighting game, and based on the result, the possibility to propose
the game system which is appropriate depends on a player’s type.
4.3.2 Outline of Proposed Cooperative-Type Fighting Game
A proposed system was developed to investigate the influence of relationship

among player’s skill, partner’s skill, and enemy’s skill. The game player in the
proposed system is presented two-digit number randomly and asked add these
numbers. As shown in Fig. 4.3, the player and the partner make one team and
compete with their counterpart. However, the game system also plays the role as the
Fig. 4.2 Relationship between not only player’s skill and enemy’s skill, but also player’s skill and
partner’s skill may influence the player’s performance and state of mind
62 M. Okubo
Fig. 4.3 Graphical user

interface of game system.
Each point is shown by not
only digit number, but also
bar graph
Table 4.1 Nine experimental Skill of game <Enemy =Enemy >Enemy

game conditions based on the
skill of player, partner, and Player < partner HH HE HL
enemy Player = partner EH EE EL
Player > partner LH LE LL
partner and the enemy of the game player, and the game player did not know about
that. Both teams answer the prepared numerical calculations, respectively, until the
total numbers of correct answers given by both teams reached a hundred. The team
that makes more correct answers wins. The skill, i.e., average of answer time, of
partner and enemy sets on three levels: “Low = L,” “Equal = E,” and “High = H,”
based on the player’s average answer time sensed in solving several calculations.
Each level presents a ratio of the participant’s average answer time to the partner’s
or enemy’s average answer time: Low = 1.5 times, Equal = 1.0 times, and
High = 0.5 times. Totally, nine kinds of experimental conditions are prepared as
shown in Table 4.1.
4.3.3 Validation of Impact of Partner’s and Enemy’s Skill

on Player’s Performance and Mind of State
Using the proposed system, the participant solves numerical questions under the
condition of controlling the partner’s and enemy’s skills. Then, the participant’s
performance and psychological mind are analyzed from the answer time and
questionnaire.
4.3.3.1 Experimental Procedure
A participant was instructed to enter a room and solve several two-digit additions in
practice. After being informed that the partner and the enemy were in the different
rooms, the participant was asked to choose the one out of nine conditions shown in
Table 4.1. The participant and partner made one team and compete with the enemy
team by solving one hundred additions with playing the game. Thirty-six university
male/female students aged between 18 and 22 were participated as the participants.
Each participant performed the experiment under all nine conditions in random
order. The answer time and the number of correct answers of each participant were
scored every nine conditions. The participant was asked to fill out a questionnaire
each time. The questionnaire used one-to-five scale to describe applicable levels
based on the eight elements of the flow theory: “clear goals,” “intense concentration
of attention,” “loss of self-consciousness,” “distortion of time,” “immediate feed-
back,” “balance between challenge and skill,” “sense of control situation and
activity,” and “intrinsic value in the activity.” The sum of seven elements’ scores
except “distortion of time” was analyzed as flow score. Additionally, based upon a
Circumplex Model of affect of human emotions proposed by J.A. Russell with
horizontal axis representing pleasant/unpleasant and with vertical axis representing
awakening/sleeping on horizontal axis [8], the participant answered 12 survey items
in one-to-four scale: tense, angry, unpleasant, depressed, bored, tired, relaxed, at
ease, satisfied, glad, astonished, and excited, and 4 survey items were answered by
many participants in pre-experiment: eager, impatience, resignation, and compo-
sure. Moreover, the participant gave a subjective evaluation for the partner’s skill
when scoring own skill as 100 points, for the enemy team’s skill when scoring own
team’s skill as 100 points, respectively, and evaluated fun of the game on a
100-point scale.
4.3.3.2 Experimental Results
For the relationship with the skill of participant and the partner, and the skill of
participant and the enemy, the number of participants who took the highest and
lowest average answer time is shown in Fig. 4.4. It is considered that the lower the
average answer time showed higher performance and the higher the average answer
time showed lower performance. Also, Fig. 4.4 shows that in the relationship with
the participant’s and partner’s one, when the partner’s skill was higher than the
participant, there were the least number of the participants who had the longest
average answer time and the largest number of the participants who had the shortest
average answer time. On contrary, when the participant’s skill is higher than the
partner’s one, there were the largest number of participants who had the longest
average answer time and the least number of the participants who had the shortest
average answer time. In short, it showed that the higher the partner’s skill became,
the higher the participant’s performance could be. In the relationship with the
participant’s and the enemy’s skill, there were few differences between the number
of the participants who had the longest and shortest average time. These results
indicated that the relationship with the participant’s and partner’s skill was more
likely to impact on the participant’s performance than the relationship with the
participant’s and the enemy’s skill.
64 M. Okubo
Number of subjects who showed the highest parformance

Number of subjects who showed the lowest parformance
36
32
28
24
20
16
12
8
4
0
Partner Partner Partner Enemy Enemy Enemy
Fig. 4.4 Number of participants who show the highest (red bars) and the lowest (green bars)
performance in relationship between their skill and partner’s or enemy’s skill
In each relationship with the participant and the partner, the participant’s emo-
tion evoked in questionnaire was shown in Fig. 4.5, in which the survey items
describing the emotion were answered in one-to-four scale as follows: (1) applica-
ble, (2) little applicable, (3) not very applicable, and (4) not applicable. The answers
1 and 2 were scored 1 point, and the answers 3 and 4 were scored 0 point to average
out the all participants’ answers. Moreover, Fig. 4.6 shows the participant’s emo-
tion evoked by varying the participant’s and the enemy’s skill. Thus, it shows the
results from Figs. 4.5 and 4.6 that the difference with the partner’s skill has a low
1
Partner Partner Partner
0.9
0.8
0.7
0.6
0.5
0.4
0.3
0.2
0.1
0
Fig. 4.5 Average of questionnaire about the state of mind for each relationship between
participant’s skill and partner’s skill
1
Enemy Enemy Enemy
0.9
0.8
0.7
0.6
0.5
0.4
0.3
0.2
0.1
0
Fig. 4.6 Average of questionnaire about the state of mind for each relationship between
participant’s skill and enemy’s skill
impact on the participant’s emotion and the difference with the enemy’s skill has a
greater impact on the participant’s emotion. That is, it was found that the difference
with the enemy’s skill is likely to have a greater impact on the participant’s emotion
than the one with the partner’s skill.
4.3.4 Classification of Player’s Performance
Using the result, as shown in the previous section, the relationship with the part-
ner’s skills has a greater impact on the participant’s performance than the one with
the enemy’s skill, and 36 participants were classified focusing on the difference
between the participants and the partners’ skills. Figures 4.7 and 4.8 show the
classification result using Ward method based on the 3 conditions of partner’s skill
—“Low,” “Equal,” and “High”—against the average answer time compared to the
participant’s skill. The classification result of 3 types of participants is shown in
Fig. 4.8. It is shown that the 1st group participants showed the higher performance
when the partner’s skill was equal to that of the participant and the lower perfor-
mance when the partner’s skill was lower than that of the participant. Also it is
shown that the 2nd group participants showed the higher performance when the
partner’s skill was higher than that of the participant and the lower performance
when the partner’s skill was equal to that of the participant. In the 3rd group, the
participants’ performances tended to become higher when the partners skills were
lower than those of the participants.
66 M. Okubo
Ward Method
Fig. 4.7 Classification based on the performance in relationship with partner’s skill
Fig. 4.8 Classification by Ward method based on the performance in relationship with partner’s
skill. Green background shows the relation in which the participant showed the highest
performance
Using each tendency seen in these 3 types, the players were classified more
easily. Figure 4.9 shows the simple classification result focusing on the relationship
with the partner, under which the participant showed best performance. The 1st type
is called “equal type”: the participants who had the highest performance when their
skills were equal to those of the partners. Thirteen participants were classified into
this type. The 2nd type is called “high type”: the participants who had the highest
Equal type High type Low type
Fig. 4.9 Simple classification based on the performance in relationship with partner’s skill. Green
background shows the relation in which the participant showed the highest performance
performance when the partners’ skills were higher than those of the participants.
Fourteen participants were classified into this. The 3rd type is called “low type”: the
participants who had the highest performance when the partners’ skills were lower
than those of the participants. Nine participants were classified into this type.
4.3.5 Characteristics Depending on Type
Figure 4.10 shows the performance of equal-type participants by focusing on the

relationship with the participant’s and partner’s skills and the one with the partic-
ipant’s and enemy’s skills. The equal-type participants naturally show the highest
0.6 ** P < 0.01

**
* P < 0.05
0.7 † P < 0.10
†
** *
Average answer time
0.8
0.9
1.1
1.2
1.3
1.4
Fig. 4.10 Average and standard deviation of performance of equal-type participants by focusing
on the relationship with the participant’s and partner’s skills and the one with the participant’s and
enemy’s skills
68 M. Okubo
**
35
**
28
Flow score
21
14
7
Fig. 4.11 Average and standard deviation of flow score of equal type by focusing on the
relationship with the participant’s and partner’s skills and the one with the participant’s and
enemy’s skills
performance when the participants and partners’ skills are as equals and show the
lowest performance when the partners’ skills are low. Moreover, in the relationship
with the participant’s and enemy’s skills, the performance of the participants tends
to lower when the enemies’ skills are low. Thus, the equal type is likely to show the
higher performance with following 3 conditions: equal partner, equal enemy, and
high-level enemy.
Figure 4.11 indicates the average flow score of equal type by focusing on the
relationship with the participant’s and partner’s skills and the one with the partic-
ipant’s and enemy’s skills. The flow scores are almost same in varying the par-
ticipants’ and partners’ skill levels. In the relationship with the participant’s and
enemy’s skills, flow scores tend to be high with the enemies having equal or low
skills.
Figure 4.12 shows the average of the fun of the game in equal type by focusing
on the relationship with the participant’s and partner’s skills and the one with the
participant’s and enemy’s skills. It is shown that the fun of the game and the flow
score indicate the same tendency.
From these results, by providing equal-type participant with an equal-level
partner and enemy, it is more likely to enable a high performance and satisfaction.
Moreover, the same kinds of analysis were conducted in high type and low type,
respectively. Summarizing these results, Table 4.2 shows the appropriate game
situations sorted by each player’s type using the average answer time.
*
100 **
90
80
70
60
Fun
50
40
30
20
10
0
Fig. 4.12 Average and standard deviation of fun of game of equal type by focusing on the
relationship with the participant’s and partner’s skills and the one with the participant’s and
enemy’s skills
Table 4.2 Appropriate game Type of player Partner’s skill Enemy’s skill
situations sorted by each
player’s type using the Equal type Equal Equal
average answer time High type High Low
Low type Low Equal or low
4.3.6 Discussions
This study investigated the impact of a player’s skill level in a competitive game on
the player’s performance and state of mind. Moreover, we propose an approach for
recommending the appropriate game situation for the game player based on the
examination result. The result indicates there is a high possibility that the rela-
tionship with the partner’s skill has more impact on the player’s performance than
one with the enemy’s skill. On the other hand, the relationship with the enemy’s
skill is more likely to have the impact on the player’s psychological status com-
pared to the one with the partner’s skill. Therefore, we classified the players into the
following 3 types by focusing on their performance levels based on the relationship
between the players and partners: “equal type” with highest performance when the
skill of the player and partner is equal, “high type” with the highest performance
when the partner’s skill is higher than that of the player, and “low type” with the
highest performance when the partner’s skill is lower than that of the player.
Moreover, we evaluated the performance, motivation, and state of mind of each
player’s type. As the result, equal type is more likely to enable the high perfor-
mance and satisfaction in providing a competitive game under the condition of an
equally skilled partner and an equally skilled enemy. High type is more likely to
70 M. Okubo
enable the high performance and satisfaction in providing a competitive game under
the condition of a higher skilled partner and a lower skilled enemy. Low type has
high chance to enable the high performance and satisfaction in providing a com-
petitive game under the condition of a lower skilled partner and a lower or equally
skilled enemy.
4.4 From System Offering to Status Offering for Human

Pleasure
When designing a game, gaining users’ attentions and making them play contin-
uously are considered. The flow theory matches very well to a game design in this
regard. If a user has low skill, the game situation is adjusted to simple manner in
order to keep user’s motivation. As a user’s skill level advances, the situation
becomes complicated and difficult. Some game software’s succeeded in adjusting
user’s task to user’s skill aiming at keeping their motivation. As shown in Fig. 4.13,
however, there are limited patterns of the user skills and problems they are given.
Development of a system based on user’s personality needs to have the function
that can change according to the broken line shown in Fig. 4.13.
That is, if the situation can be adapted to the user, we have a potential to inspire
the motivation continuously. On the other hand, a system related to a person has
been focused on only facilitating a user’s burden so far. It has not been considered
how the facilitation impacts on the user’s mental status and motivation. In other
words, engineers often pay more attention to develop a system, but user’s
Fig. 4.13 Game situation is

adjusted to simple manner in
Task
order to keep user’s

motivation as a user’s skill Flow
level advances, and the
situation becomes
complicated and difficult
User’s skill
motivation is up to the user’s issue. If the system can be designed from the
standpoint of user’s motivation by considering the relationship between the user
and the system and the situation, a user’s pleasure-oriented system inspiring user’s
motivation will be realized in future.
4.5 Conclusions
In order for a person to engage more in intellectual work, he or she must be released
from complicated and cumbersome works. So far, engineering has focused pri-
marily on removing such miscellaneous works from a person and making the things
easier. On the one hand, sometimes that easiness can get rid of pleasure and
motivation from a person. To inspire the motivation, the system needs to understand
the relationship between the user and the system and situation. Also, it needs to
have a function that can change its’ service for adaptation to circumstances flexibly.
In this chapter, an idea is suggested to design the system inspiring person’s
motivation.
References
1. H. Kawakami T. Hiraoka (2012): Incorporation of evolutionary computation for implementing

benefit of inconvenience, Int. J. of Advancements in Computing Technology, Vol.4, No.22,
pp.248-256.
2. Y. Hasebe, H. Kawakami, T. Hiraoka, K. Naito (2015): Card-type tool to support divergent
thinking for embodying benefits of inconvenience, Web Intelligence, Vol.13, No.2, pp.93-102.
3. Mihaly Csikszentmihalyi (1998): Finding Flow: The Psychology of Engagement with Everyday
Life (Masterminds Series) Paperback.
4. Mihaly Csikszentmihalyi (2013): Handbook of Research Method for Studding Daily Life,
Guildford press.
5. W.Admiraal,J.Huizenga,S.Akkerman,G.Dam (2011), :The Concept of Flow in Collaborative
Game-Based Learning, Computers in Human Behavior 27.
6. Mamiko Sakata, Tsubasa Yamashita, Masashi Okubo (2013): Factor Models for Promoting
Flow by Game Players’ Skill Level, HIM/HCII 2013, Part 1, LNCS 8016, pp.534-544
7. Masashi Okubo, Shingo Yamada and Mamiko Sakata(2015): Design for Game Situation Based
on Player’s Personality, Proc. of the 6th International Conference on Applied Human Factors
and Ergonomics (AHFE 2015), 10pages
8. James A. Russell, ‘A Circumplex Model of Affect’, Journal of Personality and Psychology,
Vol.39, No.6, pp.1161-1178(1980).
Chapter 5
Aesthetic of Prosthetic Devices: From
Medical Equipment to a Work of Design
Stefania Sansoni, Leslie Speer, Andrew Wodehouse and Arjan Buis
Abstract Aesthetic of prosthesis design is a field of research investigating the visual

aspect of the devices as a factor connected to the emotional impact in prosthetic users.
In this chapter, we present a revised concept of perception and use of prosthetic devices
by offering a view of ‘creative product’ rather than ‘medical device’ only.
Robotic-looking devices are proposed as a way of promoting a new and fresh per-
ception of amputation and prosthetics, where ‘traditional’ uncovered or realistic devices
are claimed not to respond with efficacy to the aesthetic requirements of a creative
product. We aim to promote a vision for a change in the understanding of amputation
—and disability in general—by transforming the concept of disability to super-ability
and to propose the use of attractive-looking prosthetic forms for promoting this process.
5.1 Introduction
‘Prosthetic’ is a term that refers to devices designed to replace a missing part of the
body, for example an artificial arm, leg, or finger. Our research focuses on the
aesthetic of transtibial prosthetic devices, or rather devices replacing the limb
segment below the knee.
S. Sansoni (&) A. Wodehouse

DMEM Department, University of Strathclyde, James Weir Building, 75 Montrose Street,
Glasgow G1 1XJ, UK
e-mail: stefania.sansoni@strath.ac.uk
A. Wodehouse
e-mail: andrew.wodehouse@strath.ac.uk
L. Speer
Design Department, San José State University, One Washington Square, San José, CA
95192-0225, USA
e-mail: Leslie.Speer@sjsu.edu
A. Buis
Biomedical Engineering, University of Strathclyde, Wolfson Building, 16 Richmond Street,
Glasgow G1 1XQ, UK
e-mail: arjan.buis@strath.ac.uk

DOI 10.1007/978-3-319-29433-9_5
74 S. Sansoni et al.
Prosthetic users state that it is important for a device to feel comfortable to wear
and functional to use (i.e. lightweight, movement in the ankle), but they also require
visual appeal in the devices to fulfil their emotional needs and connect the look of
the related product to their body image. Unlike the extended work to date on
prosthetics which has largely focused on the technical improvement of the devices
[1–4], the field of research into aesthetic of prostheses is new, as little interest in this
sector of prosthetic design has been recorded.
By ‘aesthetic of prosthetic devices’, we mean the visual aspect (i.e. the appearance)
of the products; in other words, this term refers to the prosthesis form and how it
looks. The form of the device involves non-pragmatic aspects (i.e. unconnected with
functionality or comfort) and is directly related to the emotional impact on the users,
on their body image, and the impressions on the external observers.
In our work, we discuss the role of the form of the device as a factor connected to
the emotional design aspects of this medical product. Let us consider what we mean
by emotional design; ‘Everything that we see evokes some kind of emotional
response. […] Love, fear, acceptance, sadness, friendship, happiness, satisfaction—
these are all valuable emotions, each may be evoked by a designer, either intentionally
or not, in the design of a product’.1 Defining prostheses as an emotional product is
particularly appropriate considering that this kind of device is strictly related to the
body image of a person with a physical impairment. Our research aims to address an
innovative point of view by proposing one of the first studies to revise the concept of a
medical device and to promote a new vision of it. The device should not merely stand
as a supportive medical product for the patient, but also as a product able to enhance
positive emotions in the user.
In considering the visual aspect of prostheses for below-knee devices, the models
resembling the realistic appearance of a human leg are identified with the term ‘cos-
metic’ (Fig. 5.1a, b), while ‘artificial’ prostheses identify devices with an appearance
dissimilar to a human leg (Fig. 5.1c–e). Within the category of artificial-looking
models, we identify ‘robotic’ devices (Fig. 5.1d) as a distinctive design type from the
uncovered design (Fig. 5.1c). With this term, we do not refer to devices with built-in
complex functionality—as the stereotype of the word might lead one to think—but
simply to the visual aspect of the device as clearly non-realistic and aesthetically
elaborated. Our need to define and utilise this term resides in the fact that no specific
term for these kinds of designs has yet been identified (they are usually referred as
‘prosthetic cover’, ‘non-realistic’ or simply ‘artificial’). Under our definition of
‘robotic’, we include devices making use of ‘fairings’ for the cover, or rather ‘intricately
designed panels that fit over prosthetic legs—the fairings create a shell around the
traditional prosthesis, giving the mechanical limb a more natural shape’2 (Fig. 5.1d).
Within the category of robotic designs, we find monolithic models, or rather prostheses
with a homogeneous and continuous design from the tibia to the feet (Fig. 5.1e).
1
http://www.studiofynn.com/journal/emotional-design-what-it.
2
http://www.huffingtonpost.com/2012/05/18/bespoke-innovations-prosthetics-that-rock_n_
1525455.html.
5 Aesthetic of Prosthetic Devices … 75
Fig. 5.1 Cosmetic foam-covered (author photograph) (a), PVC highly realistic (©2012Rosemary
Williams) (b), basilar uncovered (author photograph) (c), robotic cover design (UNIQ, 2015)
(d) and monolithic model (Jordan Diatlo design) (e) prosthetic devices
Our research direction addresses the appearance of robotic devices as way of

making a positive change to amputee’s perception of their amputation and the
prosthesis, and explores the role of robotic designs as a key factor in eliminating the
social stigma connected to amputation.
The visual choices currently offered in most of the public UK prosthetic centres are
limited to what is considered essential for the patient’s motion needs, and little account
is taken of the appearance of the devices. These choices usually include the uncovered
device (Fig. 5.1c), and a basic foam-covered ‘cosmetic’ prosthesis (Fig. 5.1a). Robotic
designs (i.e. Figure 5.1d, e) are usually available from a limited number of private
companies and often difficult to access by most prosthetic users. By considering that
the large majority of prosthetic users have access only to basic designs like the
cosmetic or uncovered device, we have identified the restricted range of aesthetic
options on offer to users as an issue. The appearance of these devices generally does
not correspond to the visual requirements of the users and does not stimulate the
elimination of the social stigma in users and external observers.
Our investigation aims to discuss the state of the art of the aesthetic of prostheses
and to propose a revised understanding of this aspect of prosthetic design. We
believe that the concept of social stigma correlated with the amputation can be
reworked by switching the understanding of the prosthetic form from a medical
support product only to a creative design. Specifically, our assumption is that
robotic-looking devices can provide the answer by better addressing the aesthetic
needs of amputees of today, to counter the realistic and uncovered devices which
represent an old and traditional vision of prostheses. Modern times require a change
in the vision of prosthetic devices, prosthetic users and disability in general.
The ‘conformist’ realistic (i.e. Fig. 5.1a) looking design is still connected to the
old-fashioned idea that the visual requirement of a good prosthesis is to closely mimic
the lost limb. This is one of the two options offered by the NHS, which states on its
website3 that ‘a prosthetic limb should feel and look like a natural limb’. Based on this
3
www.nhscareers.nhs.uk/explore-by-career/allied-health-professions/careers-in-the-allied-health-
professions/prosthetist-and-orthotist/.
same aesthetic direction, some private companies provide elaborate and expensive
prosthetic solutions taking the form of a prosthetic device almost identical to a real
limb (i.e. Fig. 5.1b).
Consider the uncovered device model (i.e. Fig. 5.1c), the second option offered
by the NHS. Here, we find a medical support product of poor appearance, without
any elaborate or appealing visual interface. The components of the socket and the
area of the tibia are artificial- and minimalistic-looking, together with a foot shape
resembling a sketchy reproduction of a real foot. The socket and the foot are often
skin colour, in contrast to a shiny metal-coloured or black-coloured pole connecting
the two components. This kind of design is a poor mix of mismatched sections,
attempting to merge realistic with non-realistic components. The result is an aes-
thetically non-harmonious prosthesis resembling neither a robotic model nor a
realistic device.
When considering the fact that those designs are the most accessible options for
prosthetic users, our cardinal observation was that the form of these models did not
respond adequately to the needs of the wearers. This observation was supported
during our data collection by 16 amputees out of the 19 we interviewed. All were
wearing either an uncovered or cosmetic device, and all 16 said they were dissat-
isfied with the form of their device. Following the focus on the problem of tradi-
tional looking devices having been designed with a form not responding to the
users’ needs, our research presents devices with a robotic form (i.e. Fig. 5.1d, e) as
innovative designs perceived internally (i.e. the user wearing the device) and
externally (i.e. observers) as better adapted for visual acceptance of the product. We
believe that the majority of prosthetic users are ready for a change in their prosthetic
aesthetic and that in 2015 times are ripe for a change in the image of prosthetic
devices. Robotic devices can respond in an innovative way to the needs of pros-
thetic users. The reason for this statement resides in the fact that a robotic device
does not try to fake the resemblance of the lost limb, and neither is it a minimalistic
design of support. This kind of prosthesis represents a visually developed design
work, aimed to revise the image of prostheses from medical products to visually
appealing products.
The attractiveness of robotic devices has to be viewed in a general context, as it
cannot apply as a universal rule for all prosthetic users (or external viewers) in their
perception of their device. When considering our recent qualitative data collection,
it was shown that, of a total number of 19 prosthetic users, making use of traditional
devices, who were interviewed, 12 stated that they were dissatisfied with the aes-
thetic of their devices, 4 of them were undecided, and only 3 of them stated they
were satisfied. Prosthetic users were shown a small set of prosthetic devices and
asked to describe them as attractive or non-attractive and to indicate their choice in
relation to their preferred option. The participants were all prosthetic users wearing
an uncovered or cosmetic model.
These data show that besides our driving idea that a consistent number of
amputees would benefit from the use of a robotic model, there is also a percentage
of amputees that do not fit in with this idea. In the following paragraphs, we will
present examples of the first and second category.
When describing the cosmetic leg (i.e. Fig. 5.1a, b) that was offered after the
amputation, most of the users stated comments like ‘it just looks ugly…it just looks
false (K)’, ‘it looked like [if I was wearing] an old lady pair of tights’ (C) or
‘between 1 and 10 I am satisfied 4 with the appearance of my device’ (R).
Similarly, in the context of describing the appearance of their uncovered device,
we recorded from most of the user comments such as that the prosthesis was ‘too
skinny’, making the trousers fold in a very unpleasant way, or, in the extreme case,
‘there is nothing that makes me like them’ (JS).
Our belief that cosmetic devices would not be suitable for all users is connected
to the idea that the uncanny valley (UV) feeling can apply to these models of
prostheses. The UV [5] affirms that artificial entities trying to reproduce human
features (e.g. robots, puppets, prosthetics) that show a very high level of
human-likeness generate a negative feeling instead of attraction [6]. In our previous
study, we detected that the UV might not universally apply for prosthetic devices in
observers (Sansoni, Wodehouse, [7]. However, by considering the general principle
of the UV and by taking into account the declarations of some prosthetic users, our
idea is that external observers might feel an unpleasant sensation when looking at a
prosthetic user and discovering that the limb, which they initially mistook for a real
leg, is an artificial prosthesis.
E, a 54-year-old wearer of a cosmetic prosthesis from the age of 2 stated that
‘We do not have a choice of prosthesis, you just wear what you are given—it will
be nice to choose the design sometimes, but I have never had an option…’ This
point shows that for prosthetists, the visual appearance of the device may, in a case
like that, be considered non-relevant that they will not even consult with the patient
before assigning a model.
The visual aspect of prosthetic devices is an underestimated element of the
design, and the designs provided often do not meet the expectations of the users.
Our research direction is that the idea of ‘suitable’ prosthetic aesthetic for users
should switch from the representation of a realistic limb (cosmetic) and/or
uncovered device to an individual appealing robotic product.
Unlike the previous examples, some categories of users have a different view of
their device, and the taste for both the appearance and design of a prosthetic device
is strongly individual. As a matter of fact, not all prosthetic users find the
appearance of the prosthetic device as their main priority or attach any importance
to it at all for their device. It appears that all users have as their first priority the
factor of comfort, some of them express functionality as a second priority and
pleasant appearance as their third requirement (user group 1—Fig. 5.2); other users
rate appearance in second place, even despite a lower level of functionality (user
group 2—Fig. 5.2). Some categories of users are more functional-oriented and
attribute little or no importance to the look of their device (user group 3—Fig. 5.2).
User group 3 includes people for which dissatisfaction with the traditional
prosthetic design does not apply.
Some prosthetic users prefer the use of an uncovered device and do not seek a
more visually enhanced model. It has been shown that some (former) soldiers view
Fig. 5.2 Representation of the priorities for the issues of comfort, functionality and aesthetic in
prosthetic devices by below-knee prosthetic users
their amputation as a symbol of pride, and by showing the minimal easy-to-notice

uncovered device, they are displaying a status symbol of their profession.
Other amputees can find themselves attracted to realistic devices for reasons
connected to fashion. For instance, some women wearing formal clothes showing
the legs consider it more appropriate to display a device which better combines with
their clothes in a natural way, rather than in an artificial device. The same issue can
be applied to males, for example Scottish amputees wearing formal traditional
clothing which shows their legs (the kilt).
The reasons for people being attracted to devices can be different, and taste is
strictly personal, so we do not intend to explore all the reasons behind the pref-
erence for a cosmetic, an uncovered or a robotic device.
Our position for this chapter acknowledges the fact that the choice of a robotic
device over more ‘traditional’ devices would not satisfy the whole population of
prosthetic users. However, we stress the importance of the direction of our work as
we believe that a negative perception of traditional devices can apply to a consistent
number of amputees, and we believe that a revision of prosthetic devices using a
different approach is needed—where the use of robotic devices can overcome
stigma. In other words, we propose the vision of a robotic device as a matter of
innovation in the context of prosthetic design.
5.2 Social Stigma in Disability
Perceived social stigma is defined as an individual’s perception that others hold

negative stereotypic attitudes about him or her as a result of a disability [8]. This
factor has been linked to problems of adjustment towards amputation and in our
opinion also affects amputees’ confidence in showing their prostheses and in the
choice of devices.
Specifically, we believe that ‘hiding’ a medical device by using some ‘mimetic

model’ (i.e. a cosmetic device) might foster the perception of stigma in both the
user and any external observer, whereas the use of a device with a robotic look
might reduce it. The more a disability, in our case an amputation, is ‘hidden’, the
less able the user will be to cope with the acceptance of the disability, and the more
likely external observers will feel a sense of ‘surprise’ or discomfort. For instance, a
prosthetic user hiding the amputation in a situation where they could wear short
clothes (i.e. summer, seaside) might demonstrate low self-confidence in exposing
his disability. This might be partly due to a natural process of acceptance or of a low
confidence generated by feedback from external observers. When focusing on
prostheses, the more the amputation is ‘covered’, the less the benefit to the user, and
the greater the likelihood of generating a sense of ‘surprise’ in external observers.
5.2.1 Stigma in Amputees
During our data collection, we recorded the presence of amputees who had
developed acceptance of their amputation—for instance, users described
self-acceptance and were supported by external observers not showing negative
feedbacks towards the ‘disability’. Besides the amputees who had accepted the
missing limb, there were some who had not developed acceptance of their body
image, in some cases because external observers tended to make them feel
uncomfortable on wearing an artificial-looking (i.e. uncovered device). These
people are therefore within the category that suffers from the stigma of amputation.
In this section, we will refer to the statements of some users via a random letter to
ensure their anonymity.
G and R were two amputees and prosthetic users, who were part of our data
collection group of 19 users. Their statements are particularly relevant in order to
explain the issue of stigma. G stated that he believes that there is a ‘stigma’ around
the idea of amputation, and this is also why some people want to ‘hide’ the
amputation behind a realistic leg. In his case, he wears an uncovered device, and he
described an episode in which he was walking in a public place in his village,
wearing shorts. A person told him that he should hide his leg and not show it to
people, as the appearance of the device was ‘not appropriate’. Similarly, R
described an episode where she was going out with a friend in a public place and
how that friend was strongly suggesting that she should either wear long trousers, or
wear a cosmetic device, in order not to attract the attention of other people to her
missing limb. The opinion expressed by the friend of R did not stop her from
expressing a preference for a robotic device. However, her friend’s views had made
her understand that her personal choice of a robotic prosthetic would not be wel-
comed by everyone; displaying a disability can be considered ‘embarrassing’.
These episodes suggest that the perception of amputation is still a taboo for some
people in Western society and that although it was fortunately not the case in our
examples, comments and behaviour of external observers could potentially have a

negative effect on the confidence of users in showing their prosthetic devices.
Similar to the previous examples, we recorded the more extreme position of two
other users, M and D, supporting the idea that displaying an amputation is (or was
for many years) a strong issue for them.
M stated that he needed his prosthesis to look as ‘normal’ as possible and that he
would have been ashamed to show other people his missing limb. This user
highlighted that showing an artificial device (i.e. by wearing shorts or no shoes),
and therefore exposing his disability, was not an appropriate choice. M stated this
point of view not because of his personal attraction to realistic devices, but by
focusing on the concern of not making the artificial limb noticeable to external
observers.
D affirmed that, since his amputation occurred when he was just in his twenties,
for many years, he had an issue about allowing people to view his disability. As a
person with a ‘genuine passion for going to the seaside during summer’, he
described that it took him many years before returning to the seaside, but when he
decided to go there showing his prosthesis, he chose a colourful model—more
visible than other options. By that time, he had stopped caring so much about the
opinion of others, as the point was to go to the seaside to have fun.
Additionally, D stated that immediately after amputation, he had opted for a
cosmetic device, although this model being more uncomfortable for him to wear.
He averred that at this stage, people attach more importance to the (cosmetic)
aesthetic of the device, and at a later stage, they are more willing to ‘sacrifice’ the
realism of a leg for one uncovered (considered to have more functionality). In the
second phase after the amputation (i.e. after many years), he chose an uncovered
device, as he felt his knee more ‘free’—‘even if it is uglier to see’.
Despite D developing more conscious acceptance and less ‘shame’ on showing
the amputated leg, he stated strongly that apart from the seaside, he chose to wear
long trousers to cover the prosthesis all year round.
The last two statements highlight how strong the influence of external observers
can be in affecting the confidence of users in showing the prosthetic device and
choosing which device to show. Furthermore, the example of D is evidence that
amputees have to face the fact that the very few robotic options available are not
aesthetically appealing at all. This issue might further discourage amputees to
abandon the idea of using a cosmetic-looking leg, as the uncovered device stands
like an ‘ugly’ option to show.
User J showed the opposite case in terms of aesthetic attraction of his prostheses.
This user stated that he lost his legs in a car accident…and stated that he ‘loved his
legs!’ and ‘I’d liked to have them back’. He added that realistic prostheses would
have been his favourite option as they could have reminded him of the lost limbs.
He liked to have ‘something that copied what I used to have’.
Despite his desire to have realistic legs, his current devices are robotic, an
appearance that does not meet his requirements, but he states that ‘I don’t care as
other people like it and say “it’s cool, you look like a robot”’. Therefore, despite his
statement of not caring about the opinion of others in the choice of his legs, he
demonstrates the acceptability of the perceived non-attractive look of his prosthe-

ses, as other people say they like his robotic devices.
This last statement supports the idea that, regardless of the appearance of the
prosthesis (i.e. cosmetic or robotic), some users are affected by the opinion of other
people in choosing the aesthetic of the prosthesis. Most importantly, it shows that
robotic devices, despite their different appearance, might gain more acceptance
from observers than traditional devices.
5.2.2 A Change in the Concept of Prosthetic Devices
The visual appearance of the medical products has always been associated with the
image of items for ‘solving a problem’, or rather a technical vision of devices as a
means of support for a human impairment. The background of the designers of
these ‘technical’ products has been exclusively clinical and engineering and
accounts for the appearance of a medical device not going beyond its medical
function. In other words, these designs completely omit emotional design, often
resulting in an unpleasing visual appearance. The appearance of these products
often negatively impacts on the interaction of the patient with the medical device,
which can be seen as non-user-friendly.
Fortunately, in recent years, the design of medical products has improved, with
the introduction of more emotionally appealing designs. For example, the bath
board launched in 1998 by A&E Design is one of the first positive visually
appealing medical designs. Similarly, the KaVO dental unit (Fig. 5.3a) shows a
pleasing-looking design displaying comforting features and colours, and aims to
suggest that going to the dentist can be seen as a positive experience: this design is
described by Dan Harden as a product that ‘looks like it won’t hurt’ [9]. By
considering the role of orthotic products, the designer F. Lanzavecchia [10] inter-
prets her neck collar design (Fig. 5.3b) as an extension of the body and aims to
achieve aesthetic comfort for the wearer in different situations by proposing an
alternative to the traditional ‘bulky’ neck collar model. Moreover, Pullin [11]
describes how simple everyday orthotic products, such as eyewear glasses
(Fig. 5.3a, c) are no longer considered a disability, but rather as fashion items.
Where people in the past avoided using glasses as it was ‘shameful’ to display a
device for visual impairment, nowadays this orthotic product is considered a beauty
accessory.
In addition to the improvement of the design of medical products, the design of
prostheses has improved in the years, and innovative theories regarding the
understanding of prosthetic devices have been introduced. A point of view from the
fashion sector is provided by the amputee and athlete top model Aimee Mullins,
who states that a prosthetic limb no longer represents the need to replace loss, but
can be conceived as a fashion accessory. The prosthesis can stand as a symbol
whereby the wearer creates him/herself like an architect and continuously changes
identity by choosing different models [12] (Fig. 5.4a shows the set of legs of the top
Fig. 5.3 KaVO dental units ‘E80 Vision’ (a), ‘Proaesthetic’ Absent Neck Brace design
(Francesca Lanzavecchia—Photocredit Davide Farabegoli) (b) and eyewear by Cutler and
Gross (c)
model). The designer Freddie Robins shows an example of fashion clothing—a

knitted wool item—designed expressly for an above limb amputee (Fig. 5.4b). The
cloth, named ‘at one’, focuses the attention of the observer on the tattoo-style
writing that has a special meaning for the model wearing it, placed on the missing
left-arm area. The special design with one sleeve missing and the decoration aims to
exalt the unicity of the model rather than focusing on ‘something missing’.
A second example of design work for revising the vision of above limb amputees is
found in the design of a golden prosthetic hand by the sculptor Jacques Monestier.
The artist proposed an alternative artistic design that represents a provocative
alternative to a realistic hand or a hook design. As Monestier explains, ‘amputees
often suffer a loss of self-image. I wanted to transmute what might be considered a
disfigurement into something marvellous and exotic. I wanted to create a hand that
would no longer cause shame and repulsion. I wanted amputees to be proud to have
a prosthetic hand and pleased to look at it. And for the people around them, I
wanted the prosthetic hand to be an object of healthy curiosity, a work of art’ (page
16, Design Meets Disability).
The topic of perception of prosthetic devices also includes student academic
research. For instance, Nguyen [13] discusses how the prosthetic design should be
‘more connected to latent needs related to the feminine identity of female amputees’
and how they should be connected to the idea of beauty rather than to mere
functionality.
In the past few years, media and fashion campaigns have aimed to positively
raise awareness of the concept of showing and performing with amputation. The
popular Paralympic games are the first major example. By referring to the London
Paralympic 2012 and acknowledging the Commonwealth Games in Glasgow 2014,
the BBC4 stated that ‘More than two-thirds of people believe attitudes towards
4
http://www.bbc.co.uk/news/uk-28175349.
Fig. 5.4 The set of prosthetic legs of the amputee top model Aimee Mullins—screen shot located
at https://www.ted.com/talks/aimee_mullins_the_opportunity_of_adversity accessed 01.12.2015,
(Aimee Mullins 2009) (a), Catherine long wears ‘at one’ knitted wool (Freddie Robins 2001) (b)
disabled people have improved since the Paralympic Games in 2012, a survey has
suggested’.
Prosthetic users K and G agree with this finding, and they reported that ‘for so
long amputation and disabilities have been hidden behind closed doors’, but thanks
to the media influence of the Paralympic games, people are becoming generally
much more accepting of the phenomenon of amputation, and used to the idea of a
prosthetic device. Accordingly, in 2012, the McCann Worldgroup released a poster
campaign to promote ticket sales for the London 2012 Paralympic Games
(Fig. 5.5a, b). The campaign included Paralympic stars and emphasised the power
and physical performance of the athletes, making them appear less disabled, and
almost like superheroes.
A campaign by Debenhams chose the long jump silver medal winner Stefanie
Reid as model for a dress which leaves the legs uncovered (Fig. 5.5c). As reported
by the Daily Mail5 regarding this choice, ‘the aim is to further challenge perceived
norms of the fashion industry showing that a broader range of body and beauty
ideals is a good thing’. Similarly, Kenneth Cole chose in the ‘We All Walk in
Different Shoes’ Advertising Campaign a series of 11 emotionally arresting pho-
tographs that celebrate diversity. By including the amputee top model Aimee
Mullins, it was stated that ‘the hope is to dispel all forms of social prejudices while
also exemplifying diversity’.6
A more provocative example demonstrating the emerging visual role of ampu-
tees is found in the model and singer Viktoria Modesta (Fig. 5.5d). Through music
5
http://www.dailymail.co.uk/femail/article-2304574/The-Paralympian-amputee-glamorous-gran-
size-18-swimwear-model-stars-new-Debenhams-campaign.html.
6
http://www.prnewswire.com/news-releases/we-all-walk-in-different-shoes-56807927.html.
Fig. 5.5 McCann Worldgroup a poster campaign for the London 2012 Paralympic Games (a and
b) (located at https://adsoftheworld.com/blog/london_2012_paralympic_games_campaign, acces-
sed 01.12.2015, McCann Worldgroup, 2012), the amputee athlete Stefania Reid for the fashion
campaign of Debenhams (located at www.dailymail.co.uk/femail/article-2304574/The-
Paralympian-amputee-glamourous-gran-size-18-swimwear-model-stars-new-Debenhams-
campaign.html, accessed 01.12.2015, Debenhams, 2013 (c), and the model and singer Viktoria
Modesta (Jon Enoch photography) (d)
videos, modelling and shows, she demonstrates the use of robotic prosthetic devices
as a strength point of her artistic image rather than as a weakness.
The singer states ‘the time for boring ethical discussions around disability is
over. It’s only through feelings of admiration, aspiration, curiosity and envy that we
can move forward’.7
7
“Modesta kāju zaudējusi Latvijas ārstu nolaidības dēļ” (in Latvian). Delfi.lv. December 16, 2014.
Retrieved January 16, 2015.
These examples represent only a small selection of the media communications

encouraging the elimination of stigma for amputees, and in most cases, they
encourage the display of robotic devices and not hiding the ‘disability’. These
works have probably helped the acceptance of external people towards the idea of
amputation and alternative (i.e. non-realistic looking) designs.
Despite the existence of these realities promoting a change in the perception of
the standard model of amputation, more advances have to be achieved in the
understanding of prosthetic product designs. Despite the fact that perception of
disability and amputation has positively improved in the past few years, there is still
room for improvement to be made. The issue of an old, traditional and incorrect
vision of amputation, viewed as something to be ‘hidden’ and being ‘ashamed’ to
show to others, is still present. The consequence of this view is little interest in the
design of prostheses as aesthetically appealing models.
Some people could argue that the importance of the aesthetic of prosthetic
devices should not be considered, as prosthetic users could simply cover the
prostheses under long trousers. However, we recorded many male amputees who
stated that they wanted to display their prostheses to others and not to have to hide
their amputation. As a matter of fact, it was not rare for us to meet amputees
wearing shorts in winter, or, in one case, we met a user wearing a special pair of
trousers which displayed only the artificial leg and covered the healthy leg. In the
case of women, the possibility of wearing skirts or shorts was also a matter of
importance; they did not want to renounce to their femininity, and they wanted the
option to wear a sporty comfortable look at the gym. Tastes in which prostheses for
females to display could have been either a very realistic device or, more often, a
nice and unique robotic design.
The issue of the appearance of the prosthesis seemed to be relevant also for users
who were wearing long trousers. Most of the people we spoke to stated that they did
not feel comfortable wearing an uncovered device under their long clothing, due to
incompatible anatomy. The result of wearing this model is the uncanny effect of a
skeletal leg when the trousers fold on the prosthesis. Similarly, most of the users
disliked the idea of wearing a ‘bulky’ cosmetic model. They stated the preference
for a robotic device, even when the device was not visible to other people. Our idea
is that a prosthesis responding to the needs of the user can provide confidence to the
wearer, even when not directly visible to external observers.
5.3 APD: From Medical Product to Appealing Work

of Design
The works mentioned in the previous section propose examples of a change

ongoing in the understanding of medical and prosthetic devices. Following this
current of thought, we aim to go beyond the traditional meaning of prostheses and
rediscuss the conventional vision of amputation. Our point of view is that the
aesthetic of prostheses is an aspect playing an important role in prosthetic design

and should be not undervalued. Specifically, we believe that the aesthetic of the
device affects both the self-body vision of the amputee and the impression that the
external observers and society have of the product and the understanding of dis-
ability of the wearer.
5.3.1 Overcoming the Stigma: From Disability

to Super-Ability
The thought of some prosthetic user interviewees is that, if a device has to be

noticed, they would much prefer to attract the attention of external observers for the
beauty of the prosthetic design, rather than for an uncovered device or a bad
mimicry of their lost leg (i.e. cosmetic device).
Our aim is to promote the beginnings of a different perception of amputation and
of prosthetic devices. The objective is to move from a negative old-style phase
where the prosthetic device is viewed as a matter of disability, or worse, something
to be ashamed of, to an appealing robotic-looking device that stands for
‘super-ability’ (Fig. 5.6—Manifesto promoting this concept) rather than ‘disabil-
ity’. For instance, the device can stand as something that the amputee is confident to
wear and to show to others; that is, it can have appealing futuristic features, it can
look modern and robotic, or it can look feminine and graceful. The idea is that the
device should represent a skilled work of design, or a fashion accessory, something
to be seen to creatively play to the user’s own identity. In return, the attention of
external observers would switch from noticing a missing limb to noticing an
appealing product and therefore to change the approach to the understanding of the
‘disability’. The person noticing the limb would for instance be willing to provide a
comment like ‘this design looks cool!’ rather than ‘how come the limb loss
occurred?’
This concept applies, for instance, to other less severe forms of disability. For
instance, some people with no visual impairment, in order to play with their image
and look, wear fake glasses with an appealing frame. This example is put forward as
an explanation of how that concept of visual ‘disability’ has changed over time,
from something to ‘hide’ to something to be socially accepted and considered as a
standing for fashion. On the subject of prostheses, when Oscar Pistorius was hosted
by the University of Strathclyde in November 2012 as a guest speaker, he was told
by someone in the audience that, after seeing him running at the Olympic and
Paralympic games, his 5-year-old son asked for prosthetic running blades as a
Christmas present. The child, a non-amputee, was seeing the devices as cool and as
desirable to wear. He was not imagining the prostheses as supports for allowing an
amputee to run, but viewing these devices as a symbol of super-ability, as some-
thing to be proud to wear in front of his friends.
Fig. 5.6 Manifesto ‘super-ability’—University of Strathclyde’s images of research, May 2015
Will prosthetic devices be perceived as a product to be proud to show off and as

a design work? This question is linked to the fact that prosthetic users need to feel
proud and confident in using their artistic prosthetic devices. Our aim is to
encourage this image of the use of prosthetic devices and therefore improve the
image of amputation. What we want to suggest is to change the image of ampu-
tation from the perception of a stigma and a disability to a super-ability—in other
words, to view a device as a new and appealing part of the body, to be connected to
the image that the user wants to promote.
5.3.2 Emotional Design for Prosthetics
Until a few years ago, the product design process was mainly focused on the
functionality of the product and did not place much importance on the visual
appeal. However, nowadays, the concept of product design has completely chan-
ged. Consumers do not require only functionality in products, but also seek an
emotional impact; they wish the product to communicate something to them.
However, we question why a large number of everyday products with a shorter term
of usability are endowed with a high emotional aesthetic appeal, whereas a special
and intimate product, such as a prosthetic device, is designed and conceived as
either a poor copy of the previous limb, or an unpleasant looking ‘skeletal’ device.
We assert that the robotic prototype should be considered as the best design for
promoting a different image of amputees and prosthetic devices.
The vision that the visual appearance of prosthetic devices is a matter of
importance for the user is a novel field of investigation, as both the academic and
industry interests in the field have been limited until now. Examples of authors
interested in the subjects can be found in the research of Murray [14, 15] and
Nguyen [13].
Academic research specifically focused on robotic models as a way of revising
the concept of prosthetic devices has been narrow. Influential researchers investi-
gating this field can however be found, i.e. in Vainshtein [12] and Pullin [11]. The
first author explored the role of prosthetic devices as a fashion accessory and a way
of creating a new identity. Similarly, Pullin investigated the new role of prosthesis
as more than a medical device. The author states ‘within design for disability, where
terms still tend to come exclusively from clinical and engineering backgrounds, the
dominant culture is one of solving problems. A richer balance between
problem-solving and more playful exploration could open up valuable new direc-
tions’. A few companies have been interested in the design process of robotic
devices by proposing aesthetically elaborated carbon fibre prosthetic covers—like
the US companies Unyq8 or the The Alternative Limb Project.9
However, the examples mentioned are only partially relevant and do not fully
cover the theme of the revision of the concept of prostheses for the well-being of the
users.
The most relevant research around the topic of image of prosthetic devices can
be found in the Simple Limb Initiative of the San José State University. By uniting
prosthetists and the works of university design students, the project aimed to design
robotic prostheses for the elimination of the social stigma for the children of
developing countries.
The Simple Limb Initiative is a project initiated in 2013, under a collaborative
design project between Prof. Leslie Speer from San José State University (SJSU)
and Prof. Gerhard Reichert from the Fachochschule Schwäbisch Gmünd (HfG SG).
8
www.unyq.com.
9
www.thealternativelimbproject.com/.
Together, they and their students worked on conceptualising affordable prosthetic

designs that focused on aesthetics without compromising function.
It is the belief of the project that people, amputees or not, wish to be surrounded
by and look at beautiful objects in their environment, including the objects they
wear and see others wearing. Celebrating one’s ‘disability’, through positive
association, assists the amputee with everyday reactions from the public and as a
result provides confidence and pride in themselves and their abilities.
Simple Limb Initiative’s ongoing research with amputees has shown that there is
a desire for attractive and beautiful prostheses that can augment their physical
appearance in a positive way (Fig. 5.7). Through the project, it was found that both
prosthetists and engineers who work in the clinics need convincing as to the value
of beauty in these types of products. Engineers, quite righty, believe that func-
tionality is primary and that the amputee wishes to have something to replace the
missing limb that visually replicates a human appendage. As discussed earlier in
this chapter, the uncanny valley is experienced by onlookers and this can result in a
kind of isolation for the amputee. This suffering and isolation is what has prompted
the Simple Limb Initiative to do the work they are doing.
The project was supported by Prosthetic Solutions in Santa Clara, California,
and Mahavir Kmina in Medellin, Colombia, where multiple products have been
designed and are undergoing user testing. Each round improves upon the previous,
and the most recent results are visually appealing, have a variety of aesthetic
choices, function well, and all cost under $30USD in parts to manufacture.
Materials used were sourced in multiple locations around the world, are common
and easy to obtain, and in many of the solutions rely on local craft industries to
fabricate.
During the projects, student teams designed products that would be used in
Colombia. This geographic focus gave student teams the opportunity to develop
solutions that addressed visual and aesthetic preferences in a specific region. This
also gave them the ability to expand their material choices by actively finding a
wider variety of material and fabrication resources. Trying to find the
aesthetic/functional/cost balance for such a demanding product is an ongoing
challenge and was faced during the final part of the student project prosthetic. The
acceptance of the aesthetic of the legs will be tested to assess whether the patients
actually are interested in prostheses that are beautiful and functional.
The students’ work has resulted in a number of insights. Finding aesthetic
solutions that purposely avoided the uncanny valley yet incorporated aspects of
cultural identification through form and detail proved challenging, but over the
three rounds have evolved and resulted in positive reactions from amputees. The
inspiration for the project has been centred on a culture (region) and the art,
architecture, fashion, and design that is native to that region. Finding opportunities
to incorporate the talents of local artisans and materials were helpful in focusing
aesthetic efforts.
Future explorations for the Simple Limb Initiative will include research activities
working with amputee partners to gain insights into their aesthetic preferences, and
Fig. 5.7 Simple Limb Initiative Transfemoral Prototypes Round 2 (left to right: a Natalie
Mukhtar; b Richard Lotti; c Adam Fujihara; d Eskady Haile)
both positives and negatives will be engaged. The results of this research will feed
into the next round of prototyping and will drive discreet component design, along
with overall aesthetic.
5.4 Conclusion
Our research aim is to offer a different perception of the medical product—and,

specifically, of the prosthetic device; as a way to promote in users and external
observers a different image of disability and to encourage the perception of the
device as a good-looking design item rather than a visually unattractive medical
device. We propose the use of robotic devices for allowing amputees to wear a
personalised and attractive ‘new part of their body’ and present this design for
allowing both users to gain a positive self-body vision and external observers to
view the concept of disability under a new, positive light. By using a unique and
artistic robotic replacement of the limb, amputees should perceive themselves and
be perceived as ‘super-abled’ rather than ‘bearer of stigma’.
By acknowledging the personal differences in users for their tastes and their ideal
device, and deeper psychological dynamics for their body vision, we believe that
times are right for encouraging a more matured vision of amputation and a different
approach in prosthetic design is needed as a first step in this matter. The majority of
prostheses currently produced are designed to mainly address the pragmatic needs
of the user and, by following an old traditional prosthetic form, do not take into
account their emotional requirements. Our assumption is that a change in the image
of disability is needed and that significant help can be provided by revising the
image of prostheses to meeting the expectations of users based on how they would
prefer their appearance to be.
We recognise as a limitation of our work the fact that our principles might not
apply to all people and in all the cases. For example, there are people that might
have little or no concern on the visual aspect of their prostheses and are happy to
use an uncovered device, or users who are more comfortable to wear a realistic
looking device rather than artificial-looking one, or others that might be happy to
wear different designs for different occasions (i.e. realistic prostheses for formal
occasions). Additionally, we recognise that the topic of disability and amputation is
a wide and complex psychological process, and the issue of acceptance and
well-being is a delicate course and cannot be reduced to the design of the prostheses
only.
Our work hopes to promote a new design system in which a user has the option
to choose an attractive robotic prosthesis and to feel comfortable to wear it in a
public place. The idea is for the amputee not to attract attention of people for his
disability, but for the visual appeal and originality of the prosthetic product.
Our hope is that the understanding of disability will be more positively perceived
in the near future and that our research could inspire people involved in the
prosthetic design process—i.e. prosthetic users, prosthetists, prosthetic designers,
and external observers—to change their views of prostheses in terms of improving
prosthetic designs for meeting the prosthetic user expectations. The core motivation
of this process is the users’ general well-being and to promote more critical
knowledge in the field of aesthetic of prosthetic devices.
References
1. Cheetham, Marcus, Suter, Pascal, & Jäncke, Lutz. (2011). The human likeness dimension of
the ‘uncanny valley hypothesis’: behavioral and functional MRI findings. Frontiers in human
neuroscience, 5.
2. Hahl, Jill, Taya, Minoru, & Saito, Makoto. (2000). Optimization of mass-produced trans-tibial
prosthesis made of pultruded fiber reinforced plastic. Materials Science and Engineering: A,
285(1–2), 91–98. doi:10.1016/s0921-5093(00)00720-6
3. Klute, Glenn K, Kallfelz, Carol F, & Czerniecki, Joseph M. (2001). Mechanical properties of
prosthetic limbs: adapting to the patient. Journal of rehabilitation research and development,
38(3), 299.
4. Mak, A. F., Zhang, M., & Boone, D. A. (2001). State-of-the-art research in lower-limb
prosthetic biomechanics-socket interface: a review. 38(2).
5. Mori, M. (1970). The Uncanny Valley. Energy, 7(4), 33–35.
6. MacDorman, Karl F., Green, Robert D., Ho, Chin-Chang, & Koch, Clinton T. (2007). Too real
for comfort? Uncanny responses to computer generated faces. Computers In Human Behavior,
25(3), 695–710.
7. Sansoni, Stefania, Wodehouse, Andrew, McFadyen, A, & Buis, Arjan. (2015). The aesthetic
appeal of prosthetic limbs and the uncanny valley: The role of personal characteristics in
attraction. International Journal of Design, 9(1), 67–81.
8. Rybarczyk, Bruce, Nyenhuis, David L, Nicholas, John J, Cash, Susan M, & Kaiser, James.
(1995). Body image, perceived social stigma, and the prediction of psychosocial adjustment to
leg amputation. Rehabilitation Psychology, 40(2), 95.
9. Sweet, F. (1999). Frog Design: Form Follows Feeling: Crown Publishing Group.
10. Vainshtein, O. (2012). ‘I Have a Suitcase Just Full of Legs Because I Need Options for
Different Clothing’: Accessorizing Bodyscapes. Fashion Theory-the Journal of Dress Body &
Culture, 16(2), 139–169. doi:10.2752/175174112x13274987924014
11. Pullin, G. (2009). Design meets disability: Mit Press.
12. Vainshtein, O. (2011). Being Fashion-able: Controversy around Disabled Models.
13. Nguyen, D. D. (2013). The beauty of prostheses: designing for female amputees.
Massachusetts Institute of Technology.
14. Murray, C. D. (2005). The social meanings of prosthesis use. Journal of Health Psychology,
10(3), 425–441.
15. Murray, CD, & Fox, Jezz. (2002). Body image and prosthesis satisfaction in the lower limb
amputee. Disability & Rehabilitation, 24(17), 925–931.
Chapter 6
Exploration of Users’ Cross-Cultural
Differences by Using the Kansei
Engineering Approach
Vanja Čok and Jože Duhovnik
Abstract Nowadays, product design for the global market has to meet the needs and
demands of more than a single group of users. Besides the required functionality, the
emotional value of the product also plays an important role in user satisfaction. One
of the aspects that an interdisciplinary design team should consider while developing
a new product is the users’ cross-cultural differences in emotional responses toward
product elements. The objective of this study was to use the adjusted Kansei engi-
neering methodology, named the Kansei cross-culture (KCUL) model, in order to
discover the differences or similarities in emotional responses toward shapes and
colors of diverse groups of participants. This study demonstrates this methodology
on two groups of students, one from Central Europe and the other from South Asia
(India). We presumed that the observed participants will have different associations
toward certain shape and color samples, as well as that they will differ in the intensity
of responses. The data were collected through Semantic differential questionnaires
and the five-level Likert scale. The survey results were then analyzed using the Factor
analysis, the Independent t-test, and the Linear mixed models.

Keywords Kansei engineering Global product conquering Central Europe

South Asia (India) Shapes Colors Product elements
6.1 Introduction
It is necessary to consider the interaction of the user with the product as a multi-
sensory experience [1]. If designers possessed a greater understanding of the
user/consumer experience, it would be easier for them to design better products [2].
V. Čok (&) J. Duhovnik (&)

Faculty of Mechanical Engineering, Lecad Laboratory,
Askerceva 6, 1000 Ljubljana, Slovenia
e-mail: vanja.cok@lecad.fs.uni-lj.si
J. Duhovnik
e-mail: Joze.duhovnik@lecad.fs.uni-lj.si

DOI 10.1007/978-3-319-29433-9_6
94 V. Čok and J. Duhovnik
Schmitt [3] emphasized that emotions are crucial when developing new designs and
for communicating with consumers as well. As described in Govers and Mugge [4],
the product personality can be defined by a set of characteristics that people use to
describe and discriminate products from others. Designers use product attributes to
design esthetically appealing products. The relationships of physical properties
(e.g., shape and color) of product designs with product attributes and esthetic
appraisal are often considered to be generalizable over product categories and
markets [5]. The cultural background of the user is an important factor that could
have a major effect on product experience. Culture consists of multilayers and is a
dynamic body of value systems that is altered by social change [6]. Early links
between culture and design were found in the area of social anthropology where
intercultural characteristics and the development of civilization were evaluated on
the basis of found objects [6]. Numerous studies have demonstrated both cultural
differences and cultural universality [7, 8]. The findings in the cross-cultural psy-
chology and intercultural anthropology suggest that cognition reflects the cultural
context, because it affects the emotions [9, 10]. People do not respond to the shape,
structure, and function of products, but rather to their individual and cultural
meanings [11]. Cross-cultural context in which product designers and buyers
operate could differ [12]. Alexander [13] argues that cultural norms in design are
especially important because they affect the human inner feelings and personal
preferences. Therefore, the interdisciplinary project team should, in accordance
with the cultural environment of the user, determine the appropriate material, color,
texture, shape, dimensions, graphics, and geometry of the prospective product [14].
6.2 Cross-Cultural Aspect in Kansei Engineering
In literature previews, we can find a few case studies where researchers use the
Kansei engineering methodology for a cross-cultural comparison of users’ emo-
tional feedbacks toward product design. Lokman et al. [15] have investigated
cultural differences in emotional responses to a Web site using the PrEmo computer
system. Chen and Chiu [16] have investigated cross-cultural differences in asso-
ciation with Kansei words toward mobile phones using the Factor analysis.
A holistic approach to the study of cultural differences and the external outlook
of industrial products has been developed by Maolosi [6] and Lin [17]. The Maolosi
model [6] has identified four key factors to determine the intercultural-oriented
appearance of a product. The model included the following factors: (1) material
factors, (2) social practices, (3) emotional factors, and (4) technology/design fac-
tors. Lin [17] claims that a cultural product refers to the three levels of culture by
Leong and Clark [18], Norman [19], and Maolosi [6]. According to Leong and
Clarke [18], the three cultural levels of product design are: (1) the inner level,
including special content, such as stories, emotions, and cultural characteristics;
(2) middle or mid-level, dealing with function, operational concerns, usability, and
safety; and (3) the outer level, dealing with color, texture, shape, decoration, surface
6 Exploration of Users’ Cross-Cultural Differences … 95
pattern, line quality, and detail [17, 18]. Product cultural design features could be
integrated in Norman’s [19] levels of the product as well: visceral, behavioral, and
reflective. Lin [17] believes that reflective design features are the most vulnerable to
variability, as a result of differences in culture, experience, and education, as well as
in individual differences. The focus of our research lies in a new methodology for
detecting cross-cultural differences in emotional feedbacks of the user toward
product design features. Since in our case study our special interest was toward
colors and shapes, we have made a literature preview concerning this area.
6.2.1 Shapes
Many researchers investigate the emotional responses of users with regard to shape
and color in different cultures [20–22]. Bloch [23] discovered that one particular
way a culture has an impact on the design preference is through the mechanisms of
prevailing styles and fashion. Designers expect consumers to prefer products that
communicate meanings that are desirable within a particular culture or subculture
[23]. Individualistic aspect, claims Hofstede [24], is based on free will and inde-
pendence of the other, while the collectivist perspective is based on harmony and
dependence [25]. While Western culture supports an open way of expressing
emotions, East Asian culture promotes emotional balance and control [10, 25]. This
theory is, according to Nisbett [10], linked with the finding that people who belong
to the Eastern culture easily identify relationships between multiple objects, while
Westerners tend to focus primarily on individual objects and their details.
In this regard, every individual has their own internal control mechanism of
self-regulation, which controls its own behavior and the consequences of behavior.
Zhang et al. [25] found that individuals with a strong self-regulation (individualistic) by
Hofstede [26] perceive angular shapes as more attractive, while individuals with
interdependent self-regulation (collectivists) perceive rounded shapes as more attractive.
Batre [27] defined different cultural markers in the context of Web interfaces and
discovered that the major differences between the four Web sites (US1, US2, Greek,
and Italian) are in the color coordination, the graphics, background images, and
navigation icons. Simon [28] proved that Asians do not like triangles and squares on
Web sites, while North Americans and Europeans prefer a combination of these
shapes. Specific design elements also depend on the scope of the product [27, 29–31].
Reinecke and Bernstein [32] note that user interfaces, the aspect of usability, and
esthetics all depend on the cultural background. Desmet [33] found significant
differences in the evaluation of the external appearance of the car, when comparing
Dutch and Japanese car models. Studies have also shown that the cultural back-
ground of the individual is a determining factor in the recognition of the importance
of road signs [34]. Attention to detail is common to all Asian cultures. Chinese,
Malaysians, Indonesians, and Thai value complexity and decoration of mixed and
vivid colors [35]. The three principles that describe Asian esthetic sensibility are:
(1) complexity and decoration, (2), harmony, and (3) naturalism [36].
6.2.2 Colors
In addition to shapes in a particular culture or region, the meaning of colors may

also vary [20]. Communication value of color, association, and color symbolism is
influenced by demographic factors such as gender, age, and ethnicity [37].
Osgood et al. [38] conducted tests on various products and discovered that the
interaction between the color and the product is statistically significant. According
to the EPA model, “activity” and “potency” are equivalent to hue and saturation of
color. For example, the more saturated the color, the greater its potency.
Berg-Weitzel Laar [39] proved the difference in package design of deodorants for
women by using the Hofstede dimensions [26]. The “feminine” cultures use more
contrast and brighter colors, while the “masculine” cultures use slight and har-
monious colors with low contrast. The study of cross-cultural color preferences of
Web site features revealed that Asians overwhelmingly suggested the use of less
bright colors (suggesting more green and less blue) while Europeans and North
Americans requested lighter/brighter colors with more images to make the sites
appear more modern [28]. Meaning of colors differs among different countries. In
China, red means love, joy, and happiness and is considered the most attractive,
whereas in India red is associated with lust, ambition, power, and energy [40]. In
India, yellow applies for a pleasant and symbolizes authority, while white is
associated with death [41]. Generally, white symbolizes mourning and death in East
Asia [42], yet in Australia, New Zealand, and the USA, it symbolizes happiness and
purity [43]. Color creates a trademark and is also the simplest way to change the
meaning of the product [20].
6.3 KCUL (Kansei Cross-Culture) Model
In order to discover cross-cultural differences regarding user’s emotional responses

to the visual element of a product, we have developed a new methodological model
named the KCUL.
The KCUL model consists of eight stages. The analogy of the procedure orig-
inates from the Nagamachi [44] Kansei engineering process. Specifically from
Kansei engineering type 1 (KE1), KCUL model and KE1 have a similar sequence
of phases or methodological steps. During phase one (I), we define the task,
objectives, and the purpose of the product. At this stage, the decision for the product
development strategy is made by the company. Next or during phase two (II), we
focus on the user profile, where we gather information about the characteristics of
target users. During phase three (III), the semantic space or dimension is reduced.
During phase four (IV), we integrate and introduce a procedure of geometrization
(geometrizing). After the procedure of geometrization, elements concerning shape
and color are classified into categories. During phase five (V), we call catego-
rization and we classify the profile of the customer, their characteristics, the culture
and the properties of product elements. This is where the KCUL model and Kansei
engineering type 1 differ from each other. During phase six (VI), we perform
evaluation and analysis (VII) of data. Here, we discover the differences between
diverse groups of the user’s emotional responses by using several statistical tech-
niques. During the synthesis phase (VIII) of the KCUL model, we associate the
user’s cultural background with product characteristics. Based on the results
gathered from the KCUL model procedure, the interdisciplinary project team then
decides on further steps in the development process. The main decision must take
into consideration whether product properties will be adapted to the local envi-
ronment or whether the product should be made universal. The phases of the KCUL
model are described below in detail (Fig. 6.1).
(a) The Task, Goal, and Purpose of the Product
In the beginning, the purpose and product development goal should be defined.
The purpose of the Kansei product is that the requirements, needs, and wishes of the
end user are met. Therefore, the product design should evoke positive feelings in
the user. Other aspects of the product should also be fulfilled. Only when the
product’s physical function is assured, we can develop affective components.
Usually, during product development those steps are performed concurrently.
(b) Users Profile
During this phase, we gather information about the target user and their char-
acteristics. To get a broader insight, it is necessary to look into the lifestyle, habits,
and culture index of the user. Other demographic data of the target user, such as
age, gender, and level of education, are also valuable.
(c) Limitation of Semantic Dimensions
We identified four main semantic dimensions of the product. These are the
esthetic, functional, social, and future attributes. In particular when considering
different groups of users, it is important to use basic, understandable, and common
words or adjectives with which product characteristics are described. Kansei words
or adjectives, which will represent the main semantic dimensions of the product, are
selected and reduced using both qualitative and quantitative methods.
Brainstorming should be conducted with experts from the industry to gather a
diverse selection of Kansei words, which will be associated with the future product.
At the same time, the product developer should follow the product strategy and
collect Kansei words from the literature, Internet, and magazines, which fit the
product domain. The quantitative method includes preliminary studies during which
we reduce the amount of adjectives with, e.g., Factor analysis.
(d) Preparation of Product Elements
Samples for further evaluation shall be selected in two ways. First, there is a
selection of pictures of existing products on the market from Web sites, journals,
and magazines. The second way is to collect conceptual sketches and concepts,
Fig. 6.1 KCUL model
which are based on semantic dimensions and designed by design engineers using
their own imagination. Then, the product or sketch samples are converted from 3D
models into a 2D abstract shape contour and color elements by using the procedure
of simplification (geometrization and abstraction) based on Lewalski’s [45] theory
of elements reduction (Fig. 6.2).
(e) Categorization
Contour shapes, colors, and other product features are categorized according to
the criteria of gestalt principles. Visual elements are sorted in predefined categories
according to their properties. With categorization of product elements, it is easier to
recognize the links between product characteristics and psychological feelings of
the user.
(f) Evaluation
Product elements or samples are used for the preparation of questionnaires or
other types of measurements. The questionnaire consists of a 5-, 7-, 9-, or 11-point
scale. Samples of shapes or colors are estimated together with Kansei adjectives in a
so-called Semantic differential evaluation scale [38].
(g) Analysis
Data are analyzed by a variety of statistical techniques. First, it should be
determined how the differences in emotional responses of cross-cultural users will
be observed. We proposed that researchers should observe the following: the dif-
ference in the intensity of the response, differences in perception or semantic
structure (comparing perception) and with investigating the influencing factors,
which affect emotional response, such as, gender, education, and location.
(h) Synthesis
During synthesis, we combine the findings from all types of analysis. Then, the
interdisciplinary team, based on results and guidelines, decides on further steps in
the development process. The target decision is whether the properties of the
product will be adapted to local markets or if the product will be global and
universal.
6.4 Case Study
6.4.1 Research Aim
We intend to test the KCUL model and discover whether South Asians (Indians)
have the same intensity of emotional responses and associate shape samples with
the same meaning of bipolar adjectives as Central Europeans. We would also like to
identify which factors influence the emotional responses of compared groups of
participants.
6.4.2 Participants
The subjects of this research were 137 engineering students from India (Chennai
n = 13, Bangalore n = 18, and Mumbai n = 24) and Central Europe (Hungary
n = 28, Croatia n = 28, and Slovenia n = 28). They were undergraduate, master,
and PhD students. There were 33 females and 104 males aged between 25 and 40.
Their educational background was mechanical engineering, industrial design
engineering, and product design. The research was performed using Semantic
differential questionnaires.
6.4.3 Preparation of Product Elements
The aim of geometrization (geometrizing) was to transform a concrete product or

sketch into abstraction. To get an initial idea of how users perceive separate product
elements, it is necessary to get a variety of different shapes and colors (Figs. 6.3 and
6.2, and Table 6.1).
Fig. 6.2 Preparation of samples
Fig. 6.3 Sets of angular and rounded shape samples
Table 6.1 Color samples

Colors RGB scale Hue Saturation Value
R G B H (°) S (%) V (%)
Red 237 28 36 357 87 92
Green 202 219 42 64 80 85
Blue 0 173 221 192 100 86
Yellow 255 242 0 56 100 100
Magenta 236 0 140 323 100 92
White 255 255 255 273 0 100
Black 0 0 0 344 0 0
Gray 128 130 133 208 2 51
Orange 247 147 30 32 88 96
Ten samples of contour shapes were used in this survey and represented with
black and white in order to avoid the influence of color, (Fig. 6.3). Shape contour
characteristics (2D) were taken from pellet burners designs (3D). For clear data
interpretation, we divided these samples into two categories based on the angularity
of their shapes, (Fig. 6.3). Criteria for separation were straight or curved corners. In
this experiment, we intentionally did not associate the shape with any shape
symbolism.
Separated and divided from shape samples, nine color samples were chosen for
evaluation. We gathered samples from several Web pages and magazines, which
describe color trends. The color sample characteristics are presented in Table 6.1.
6.4.4 Affective attributes
We have defined four product characteristics, which were observed during this
experiment (Fig. 6.4). Adjectives were collected from different sources and litera-
ture. Later on, they were arranged in regard to product characteristics. All selected
adjectives were general, used daily, and understandable to all participants.
Fig. 6.4 Main product characteristics from reduced semantic space

The subject used a 5-point Likert scale to rate shape esthetic attribute (1 = very
ugly and 5 = very beautiful), shape functional attribute (1 = very uncomfortable
and 5 = very comfortable), shape social attribute (1 = very feminine and 2 = very
masculine), shape functional attributes (1 = very unreliable and 5 = very reliable),
(1 = very complex and 5 = very simple), and shape future attributes (very unfu-
turistic–very futuristic).
6.4.5 Procedure
The questionnaire was divided into three parts. The first part dealt with demo-
graphic questions related to gender, age, location of residence, ethnicity, and
education. The second part of the questionnaire examined the emotional response of
respondents to shape and color using the Semantic differential technique according
to Osgood [44]. Finally, the 5-level Likert scale was used for rating samples of
shapes and colors associated with bipolar Kansei adjectives. For statistical analysis,
we transposed values and changed the order of bipolar Kansei pairs together with
values, e.g., (1) beautiful–(5) ugly to (1) ugly–(5) beautiful. All questionnaires were
in English.
6.4.6 Statistical Methods
Data were analyzed with both univariate and multivariate statistical models
(Table 6.2). First of all, we have the conduct (1), descriptive, and inference
statistics. Then, on the basis of the mean, standard error, and review of the distri-
bution, we defined certain criteria for the definition of positive, negative, and
neutral emotional responses. With (2) the Student’s t-test, we discovered the dif-
ference in the intensity of response and the difference in the mean of two
Table 6.2 Sequence of statistical methods for users’ emotional feedbacks comparison
Seq. Method Output Indicator
1 Descriptive and Mean, standard error, data Data review
inferential distribution
statistic
2 Student’s t-test Discovering differences in the mean Difference in the
between two target populations intensity of the
response
3 Factor analysis Correlation between the Kansei Difference in
(FA) adjectives, shapes, and colors perception (semantic
structure)
4 Linear mixed Identification of factors affecting Effects on the
models (LMM) emotional response emotional response
independent group samples. With (3) Factor analysis, we observed the differences
between the two groups of participants in their perception or semantic structure of
Kansei adjectives, shapes, and colors. With (4) Linear mixed models, we observed
that which factors influence or have an effect on emotional responses. Statistical
analysis was performed in SPSS version 22 and version 2013 XLSTAT, while
statistical significance was set at p < 0.05.
6.4.7 Criteria for Emotional Responses Definition
Data obtained from the questionnaires were first analyzed by using descriptive and
inferential statistics. Mean values of individual shapes and colors in conjunction with
Kansei pairs were divided into three categories. Kansei pairs, which have been together
with shapes and colors, estimated with the mean ðxÞ above, below, and on 3 (interval of
x = 2.86 to 3.14). Estimated shapes or colors that have reached x > 3 were considered
as beautiful, simple, masculine, reliable, comfortable, and futuristic. While shapes or
colors which have reached mean of x < 3 were considered as ugly, complex, femi-
nine, uncomfortable, unreliable, and unfuturistic. When determining the boundaries
between the mean, which is neutral, we considered the lower and upper limits of
standard error or confidence interval. If the average is x = 3 or is located between the
x = 2.86 to 3.14, the response of the respondents was perceived as neutral.
6.5 Results
6.5.1 The Differences in the Intensity of Emotional

Response
With the Student’s t-test, we found statistically significant differences in the mean
of the two observed groups, which consisted of South Asians and Central
Europeans. By Levene’s test for equality of variances, we have identified 3 sta-
tistically significant results for which inequality of variances was found, which we
then considered during data interpretation.
6.5.1.1 Colors
Kansei bipolar adjectives (ugly–beautiful) Statistically significant differences

were found in the following colors:
Green: (t = 2.557, p = 0.012) South Asians have a higher mean and perceive it
as beautiful, while Central Europeans are neutral in their scores.
Magenta: (t = 3.922, p = 0.000) South Asians perceive it as a beautiful, whereas
Central Europeans as ugly (Figs. 6.5 and 6.6).
Fig. 6.5 Mean values of scores for colors in relation to Kansei bipolar adjectives of Central
Europeans. The value of error bars is standard error
Fig. 6.6 Mean values of scores for colors in relation to Kansei bipolar adjectives of South Asians.
The value of error bars is standard error
Kansei bipolar adjective (complex–simple) Here, we found statistically signifi-

cant differences between individual colors.
Green: (t = 4.511, p = 0.000) with a higher mean South Asians rated it as
simple, while Central Europeans perceive it as complex.
Magenta: (t = 2.515, p = 0.013) both groups of respondents perceived it as a
complex, but South Asians have a higher mean in comparison with Central Europeans.
Black: (t = −4.175, p = 0.000) Central Europeans have a higher mean, but like
South Asians both have perceived it as simple.
Orange: (t = 2.573, p = 0.011) South Asians have neutral scores, while Central
Europeans perceive it as a complex (Figs. 6.5 and 6.6).
Kansei bipolar adjective (feminine–masculine) Here, we have observed statis-
tically significant differences in the following colors.
Green: (t = −3.511, p = 0.001) Central Europeans were neutral in their scores,
while South Asians perceive it as feminine.
Blue: (t = −2.371, p = 0.020) both groups perceived it as masculine. However,
Central Europeans have a higher mean in comparison with South Asians.
Magenta: (t = 3.430, p = 0.001) both groups grade it as feminine, but South
Asians had a statistically significant higher mean compared to Central Europeans.
Gray: (t = −2.549, p = 0.013) both groups grade it as a masculine color.
However, Central Europeans statistically have a significantly higher mean than
South Asians (Figs. 6.5 and 6.6).
Kansei bipolar adjective (unreliable–reliable) Statistically significant differences
appear in the following colors.
Green: (t = 2.475, p = 0.015) South Asians perceive it as reliable, while Central
Europeans perceive it as unreliable.
Yellow: (t = 2.130, p = 0.035) South Asians perceive it as reliable, while
Central Europeans perceive it as unreliable.
Magenta: (t = 2.618, p = 0.010) both groups of respondents perceive it as un-
reliable, with South Asians having a higher mean than Central Europeans.
Kansei bipolar adjective (uncomfortable–comfortable) Statistically significant
differences were found in the following colors.
White: (t = 3.333, p = 0.001) both groups of respondents perceive it as com-
fortable, but South Asians have a higher mean compared to Central Europeans.
Red: (t = −3.518, p = 0.001) Central Europeans perceive it as comfortable,
while South Asians perceive it as uncomfortable (Figs. 6.5 and 6.6).
Kansei bipolar adjective (unfuturistic–futuristic Statistically significant differ-
ences were found in the following colors.
Blue: (t = 3.282, p = 0.001) both groups of respondents perceive it as futuristic.
However, South Asians have a higher mean than Central Europeans.
White: (t = 4.230, p = 0.000) South Asians perceive it as futuristic, while

Central Europeans are neutral in their scores.
Black: (t = 2.267, p = 0.025) South Asians are neutral in their scores, while
Central Europeans perceive it as unfuturistic.
Gray: (t = 4.416, p = 0.000) South Asians perceive it as futuristic, while Central
Europeans as unfuturistic (Figs. 6.5 and 6.6).
Generally, we found that groups mostly differ in their perception of the color
magenta. South Asians perceived it as beautiful, while Central Europeans found it
ugly but futuristic. South Asians, when compared to Central Europeans, were dis-
tinctly more positive in their assessment of the color green. In general, South Asians
gave more positive meanings to colors, which are more saturated, such as blue,
green, and magenta. Both groups positively evaluated the colors blue and white.
6.5.1.2 Shapes
Kansei bipolar adjective (ugly–beautiful) Statistically significant differences

were found only in the following shape.
Shape I: (t = 2.935, p = 0.002) both groups of respondents score it beautiful,
while South Asians have a higher average score compared to Central Europeans.
The unequal variance from Levene’s test was considered (Figs. 6.9 and 6.10).
Kansei bipolar adjective (complex–simple) Statistically significant differences
were found in the following shapes.
Shape A: (t = −2.174, p = 0.031) despite a higher mean value by Central
Europeans, both groups assessed it as simple.
Shape D: (t = 2.458, p = 0.016) both groups of respondents assessed it as
complex but South Asians have a higher mean of scores in comparison with Central
Europeans.
Shape F: (t = 2.844, p = 0.006) both groups of respondents perceive it as
complex but South Asians have a higher mean of scores compared to Central
Europeans.
Shape E: (t = −2.351, p = 0.020) with a higher mean of scores Central
Europeans have rated it as simple but South Asians as complex.
Shape G: (t = −3.415, p = 0.001) with a higher mean of scores Central
Europeans perceive it as simple but South Asians as complex.
Shape I: (t = 2.126, p = 0.035) both groups of respondents assessed it as simple
but South Asians have a higher mean in comparison with Central Europeans.
NOTE: When considering D- and F-shapes, we took into consideration
inequality of variances from Levene’s test (Figs. 6.9 and 6.10).
Kansei bipolar adjective (feminine–masculine) No statistically significant dif-

ferences between groups of respondents were found when taking into consideration
this adjective (Figs. 6.7, 6.8, 6.9, and 6.10).
Kansei bipolar adjective (unreliable–reliable) Statistically significant differences
were found in the following shapes.
Shape A: (t = −2.301, p = 0.023) Central Europeans perceive it as comfortable,
while South Asians were neutral in their scores.
Shape D: (t = 2.085, p = 0.039) South Asians were neutral in their scores, while
Central Europeans perceive it as unreliable (Figs. 6.9 and 6.10).
Kansei bipolar adjective (uncomfortable–comfortable) Statistically significant
difference was found only with shape F.
Shape F: (t = −4.070, p = 0.000) Central Europeans perceive it as comfortable
while South Asians perceive it as uncomfortable.
Kansei bipolar adjective (unfuturistic–futuristic) Statistically significant differ-
ence was found in several shapes.
Shape A: (t = 3.509, p = 0.001) was perceived as unfuturistic by both groups of
respondents while South Asians have a higher mean of scores compared to Central
Europeans.
Fig. 6.7 Mean values of scores for angular shapes in relation to Kansei bipolar adjectives for
Central Europe. The value of error bars is standard error
Fig. 6.8 Mean values of scores for angular shapes in relation to Kansei bipolar adjectives for
South Asia (right). The value of error bars is standard error
Shape B: (t = 2.369, p = 0.019) South Asians perceive it as neutral, while

Central Europeans perceive it as unfuturistic.
Shape F: (t = −4.414, p = 0.000) both groups of respondents perceive it as
futuristic but Central Europeans have a higher mean compared to South Asians.
Shape J: (t = −2.394, p = 0.027) both groups of respondents assess it as fu-
turistic but Central Europeans have a higher mean compared to South Asians.
NOTE: With Shape A, we consider the inequality of variance in Levene’s Test
(Figs. 6.7 and 6.8). In this particular case, we should consider that the equality of
variances was not assumed.
Both groups of participants perceive angular shapes as masculine, uncomfortable,
and unfuturistic, while the rounded shapes evoked more positive responses.
Rounded shapes were perceived as beautiful and feminine in most cases. Shape B is
an exception because it was perceived as masculine despite having rounded edges.
At the same time, feminine shapes are perceived as futuristic. Both groups attributed
the most positive adjectives to Shape B.
Fig. 6.9 Mean values of scores for rounded shapes in relation to Kansei bipolar adjectives for
Central Europe. The value of error bars is standard error
6.5.2 The Difference in Perception and Semantic Structure
When examining eigenvalues, we have found that it is necessary to only keep the
first and the second factor because the values of other factors are low and negligible
(Figs. 6.11 and 6.12). This means that these two factors are sufficient to explain the
variability of other variables.
Correlation matrix with the values of the Pearson correlation coefficient indicates
the Kansei bipolar adjective. When considering subjects from Central Europe, a
high correlation between bipolar Kansei bipolar adjectives is calculated: ugly–
beautiful and uncomfortable–comfortable (r = 0.831, p = 0.000); simple–complex
and unreliable–reliable (r = 0.886, p = 0.000); and feminine–masculine and un-
reliable–reliable (r = 0.714, p = 0.001). South Asians have the highest correlations
between the following Kansei bipolar adjectives: complex–simple and unreliable–
reliable (r = 0.787, p = 0.000); complex–simple and uncomfortable–comfortable
(r = 0.735, p = 0.000); and unreliable–reliable and uncomfortable–comfortable
(r = 0.866, p = 0.000).
The dimensions of emotional responses are composed from two factors that have
factor weights on different Kansei bipolar adjectives (Table 6.3). After eigenvalues
Fig. 6.10 Mean values of scores for rounded shapes in relation to Kansei bipolar adjectives for
South Asia. The value of error bars is standard error
Fig. 6.11 Eigenvalues for Central Europe
and factor contribution have been checked, the first two factors for both India and
Central Europe had the highest contribution (India; 81.7 %) and (Central Europe;
87.9 %). The values in bold correspond for each variable to the factor for which the
squared cosine is the largest (Table 6.3).
Fig. 6.12 Eigenvalues for South Asia
For Central Europe, the Kaiser–Meyer–Olkin (KMO) measure of sampling

adequacy was 0.604, while the result of a Bartlett’s test was statistically significant
p < 0.001. This means that the data in relation to the KMO rate correspond for
further evaluation with the Factor analysis. For extraction, we used the principal
axis factoring. The first factor explains 52.2 % of the variance, while F2 explains
35.67 % of variance. F1 contains four most influential Kansei adjectives: reliable,
simple, masculine, and unfuturistic. F2 contains two Kansei adjectives: beautiful
and comfortable (Table 6.3). Factors were named as “classical” and “attractive-
ness” because of the variables that compose them. We named the two main factor
dimensions as F1 “classical” and F2 “attractiveness.”
For South Asia, the Kaiser–Meyer–Olkin measure of sampling adequacy was
0.620, while the result of a Bartlett’s test was statistically significant p < 0.001. The
first factor F1 explains 54.2 % of variability and the second, F2, explains 27.5 %
(Table 6.3). F1 contains five Kansei adjectives: simple, reliable, futuristic, com-
fortable, and beautiful. F2 contains one Kansei adjective, which is masculine
(Table 4.1). The factors were named the same as with Central Europeans but the
structure of factors and meaning was opposite to South Asians. For example, F1
Table 6.3 Component matrix of factors for Central Europe and India
Bipolar adjectives Central Europe South Asia (India)
F1 (52.19 %) F2 (35.70 %) F1 (54.21 %) F2 (27.46 %)
Ugly–beautiful 0.95 0.77 −0.42
Complex–simple 0.90 0.80 0.38
Feminine–masculine 0.80 0.93
Unreliable–reliable 0.94 0.89 0.38
Uncomfortable–comfortable 0.96 0.94
Unfuturistic–futuristic −0.89 0.32 0.57 −0.55
Values in bold are different from 0 with a significance level alpha = 0.05
Fig. 6.13 Observations of axes F1 and F2 and vectors of variables for Central Europeans
was named “attractiveness” while F2 as “classical.” Red and green are colored
vectors in Figs. 4.5 and 4.6. Red-colored vectors represent variables: beautiful,
comfortable, reliable, simple, and futuristic, which are positioned on the axis of F1
and F2. The green-colored vectors represent the opposite pairs of variables: ugly,
uncomfortable, unreliable, complex, and unfuturistic.
South Asians in comparison with the Central Europeans also differ in the per-
ception of shapes and colors. Central Europeans generally associate the functional
attribute (simple and reliable), the social attribute (masculine), and the futuristic
attribute (unfuturistic) with shapes that have angular edges or lines, such as Shape A
and achromatic colors (Fig. 6.14). South Asians associate the functional attribute
(simple, reliable, and comfortable), the esthetic attribute (beautiful), and the future
attribute (futuristic) with shapes B and I (with rounded edges) and colors blue,
green, and white (Fig. 6.13). South Asians associate the angular shapes A and E
and the color black with the social aspect (masculine).
For a detailed idea about the position of factor scores on the graph, values are
presented in Tables 6.4 and 6.5.
6.5.3 Effects on the Emotional Response
First, we tested fixed effects for Kansei pairs (6), colors (9), geographical locations
(2), and gender (2) separately (Table 6.6). A statistically significant fixed effect was
discovered for the Kansei adjectives and colors, while gender and geographic
location were statistically insignificant. Statistically significant two-way
Fig. 6.14 Observations of axes F1 and F2 and vectors of variables for South Asians
Table 6.4 Factor scores of Central Europeans

Observation F1 F2 Observation F1 F2
Green −1.289 −0.329 a 2.520 −1.901
Blue 0.987 2.458 b 1.934 1.325
Yellow −0.343 0.603 c 0.278 −2.975
Magenta −2.710 −1.805 d −2.104 0.792
White 1.932 1.726 e 1.394 −0.772
Black 2.427 0.439 f −3.959 0.015
Gray 1.650 −1.232 g 1.003 −1.980
Orange −0.897 −0.040 h −1.072 −0.626
Red −0.094 0.990 i −0.329 1.729
j −1.330 1.583
Values in bold are different from 0 with a significance level alpha=0.05
Table 6.5 Factor scores of South Asians

Observation F1 F2 Observation F1 F2
Green 1.527 −0.787 a −0.975 2.270
Blue 2.925 −0.312 b 2.297 1.375
Yellow 0.049 0.069 c −2.914 1.315
Magenta −1.013 −1.557 d 0.041 −1.705
White 3.503 0.621 e −1.333 1.479
Black 0.044 1.387 f −3.193 –2.606
Gray −0.075 0.517 g −2.173 1.467
Orange 0.359 −0.553 h −0.466 −0.531
Red −1.212 −0.466 i 2.003 −0.925
j 0.605 −1.058
Table 6.6 Type III test for fixed effects

Source Denominator df F-value Sig. Numerator df
Adjective 1929 20.604 0.000 5
Color 1330 43.425 0.000 8
Location 135 0.550 0.460 1
Adjective * location 1929 3.962 0.000 5
Color * location 1331 7.194 0.000 8
Color * adjective 511 17.741 0.000 40
Color * adjective * location 511 4.618 0.000 40
Gender 132 3.280 0.072 1
Color * gender 1297 0.643 0.742 8
Gender * location 132 2.288 0.133 1
Color * gender * location 1297 1.887 0.058 8
interactions were between: Kansei adjective x geographic location, color x geo-

graphic location, and the color x Kansei adjective, while the interactions between
color x gender and geographic location were statistically insignificant. The
three-way statistically significant interactions are color x Kansei pair x geo-
graphical location, while the color x gender x geographic location are statistically
insignificant (Table 6.6). This means that South Asians and Central Europeans
perceive colors differently or give different meanings to colors. The results confirm
the assumption that the interaction between geographic location, colors, and Kansei
adjectives affects the responses of respondents.
Before observing the interactions, we observed the results from the fixed effect
experiment for the shape (10), adjective (6), location (2), and gender (2) separately
(Table 6.7). The criterion for measuring significance was set to p < 0.001.
A statistically significant fixed effect was found for the shapes, which indicates that
Table 6.7 Type III test for fixed effects

Source Denominator df F-value Sig. Numrator df
Intercept 135.182 9923.373 0.000 1
Shape 1323.053 14.545 0.000 9
Adjective 2680.917 1.987 0.077 5
Location 135.779 0.085 0.771 1
Gender 131.614 0.651 0.421 1
Shape × adjective 530.934 42.494 0.000 45
Shape × location 1288.067 2.384 0.011 9
Adjective × location 2680.917 1.055 0.383 5
Shape × gender 1313.408 1.542 0.128 9
Gender × location 131.614 2.538 0.114 1
Shape × gender × location 1313.408 1.302 0.231 9
Shape × adjective × location 530.934 4.169 0.000 45
different shapes trigger different participant responses. For adjective and location, a
statistically significant effect was not found, which indicates that adjectives or
geographical location alone did not affect the responses in our research. Gender
alone was also statistically insignificant. This means that there was no difference
found in the perception of shape contour between male and female participants. In
this case, we can confirm that variety of shape contours do affect a response or
emotional feedback of individual.
During the next step, the two-way interactions between: shape × adjective;
shape × location; adjective × location; shape × gender; and gender × location
were analyzed. The shape × adjective interaction is statistically significant, which
indicates a connection between different shapes and different meaning of adjectives
that influence participant responses. The other two-way interactions: shape × lo-
cation, adjective × location, shape × gender, and gender × location were statis-
tically insignificant to the responses of the participants. Therefore, we can confirm
the assumption of hypothesis 2 that the interaction between shape × adjective has
an influence with individual responses. The three-way interaction test showed that
shape × gender × location is also statistically insignificant (Table 6.7).
However, the shape × adjective × location interaction was statistically signifi-
cant for the responses. The results from the interaction test indicate that South
Asians and Central Europeans have a different understanding of the meaning of
shape. The results from all of the stimuli demonstrate the similarities and dissim-
ilarities in shape perceptions between both geographical locations.
6.6 Conclusion
The case study demonstrated the use of the KCUL model. An important contri-
bution of the method is the systematic approach. The method should be imple-
mented in the early stage of the product development process, when the designer is
looking for new solutions or conceptual concepts at an abstract level. The upgraded
model KI 1, the KCUL model with its process, serves as a complementary
methodology to the product development process and contributes to new types of
methods to detect cross-cultural differences among the user’s perception of visual
product elements.
The goal of this paper was to compare and discover emotional feedbacks toward
shape and color samples between two culturally diverse groups of participants. Data
were gathered by using the Semantic differential technique and the 5-level Likert
scale. Later on, three different statistical methods were used in order to find
potential differences in: semantic structure, intensity of emotional responses, and
effects during emotional responses.
The results show that the participants from both groups from South Asia and
Central Europe have similar psychological feelings toward shapes, while more
obvious differences were found in the perception of color. The two groups differ in
semantic structure and in the intensity of responses or scores.
South Asians in comparison with Central Europeans gave higher scores to

rounded shapes.
Observing the structure of shapes regardless of their angularity, the shapes that
are asymmetric over the horizontal centerline are typically accepted as complex.
The results are consistent with the findings of Gordon [46], who claims that
angular shapes give an impression of strength and power. In contrast, curves are
generally considered more beautiful than straight lines [46]. The results from the
interaction tests confirm the results from the Independent t-test that South Asians
and Central Europeans have a different understanding of the meaning of shape and
color. Besides, the research showed that using the Linear mixed model procedure
requires the performing of a three-way interaction in order to recognize which
combination of factors influences the emotional responses of participants.
References
1. Bordegoni M, Cugini U, Ferrise F (2013) Multisensory User Experience Design of Consumer

Products, Emotional Engineering, Vol. 2, editor Suichi Fukuda., Springer London Heidelberg,
New York Dordrecht: 219 – 240
2. Desmet PMA, Hekkert P (2009) Special issue editorial: Design & emotion.Int J Des, 3,2:1-6
3. Schmitt BH, Simonson A (1997) Marketing Aesthetics: The Strategic Management of Brands,
Identity and Image, New York: Free Press
4. Govers P, Mugge R (2004)” I love my jeep, because it’s tough like me: the effect of product
personality congruence and attachment”, Proceedings of the fourth international conference on
design and emotion. Ankara, Turkey
5. Blijlevens J, Creusen MEH, Schoormans JPL (2009) How Consumers Perceive Product
Appearance: The Identification of Three Product Appearance Attributes. International Journal
of Design, 3,3: 27-35.
6. Moalosi R., Popovic V, Hickling-Hudson A (2007) Product Analysis Based On Botswana’s
Postcolonial Socio-Cultural Perspective. Int J Des, 1, 2: 35 – 43.
7. Matsumoto D, Ekman P (1989) American–Japanese cultural differences in intensity ratings of
facial expressions of emotion. Motivation and Emotion,(13):143–157.
8. Rosenberg EL, Ekman P, 1994. Coherence between expressive and experiential systems in
emotion. Cognition and Emotion, 8(3): 201–229.
9. Hofstede G, 1980. Culture’s Consequences: International differences in work related values.
Beverly Hill, CA, Sage.
10. Nisbett RE, 2003. The geography of thought: How Asians and Westerners think differently…
and why. New York: Free Press.
11. Krippendorf K (2006) The Semantic Turn: A New Foundation for Design, Boca Raton
12. Hsu SH, Chuang MC, Chang CC (2000) A semantic differential study of designers and users
product form perception. International Journal of Industrial Ergonomics,(25): 375 – 391.
13. Alexander RD (1979) Darwinism and Human Affairs. University of Washington Press, Seattle
14. Crilly N, Moultrie J, Clarkson P J (2004) Seeing things: consumer response to the visual
domain in product design, Design Studies, 25(6), November
15. Lokman AM, Ishak KK, Hadiana A (2013) PrEmo and Kansei: A Comparative Analysis.
International Journal of Basic and Applied Sciences. Insan Akademika Publications
16. Chen K, Chiu SC, Lin FC (2007) Kansei Design with Cross Cultural Perspectives, Aykin N
(Ed.): Usability and Internationalization, Part I, HCII 2007, LNCS 4559, Springer-Verlag
Berlin Heidelberg: 47–56.
17. Lin RT (2007) Transforming Taiwan aboriginal cultural features into modern product design:
A case study of a cross-cultural product design model. Int J Des,1(2): 45 – 53.
18. Leong D, Clark H (2003) Culture-based knowledge towards new design thinking and practice-
A dialogue. Design Issues, 19, 3: 48 – 58.
19. Norman DA (2002) ‘Emotion & design: attractive things work better’ Interactions, IX 4: 36–42.
20. Aslam MM (2006) Are You Selling the Right Colour? A Cross-cultural Review of Colour as a
Marketing Cue, Journal of Marketing Communications, 12(1), pp.15 – 30.
21. Zhang Y, Feick L, Prince LJ (2006) The Impact of Self-Construal on Aesthetic Preference for
Angular Versus Rounded Shapes, Society for Personality and Social Psychology, PSPB, 32,
6:794 – 805.
22. Adelabu OS, Yamanaka T (2012) Towards Understanding Kansei through Aesthetic
Experience and Value Perception in Cultural Design, Kansei Engineering International
Journal, 1,2: 51 – 60.
23. Bloch PH (1995) Seeking the Ideal form: Product Design and Consumer Response. Journal of
Marketing, 59,3:16 – 29.
24. Hofstede G (1997) The Archimedes effect. Working at the interface of cultures: 18 lives in
social science. M. H. Bond. London, Routledge: 47 – 61.
25. Zhang Y, Feick L, Prince LJ (2006) The Impact of Self-Construal on Aesthetic Preference for
Angular Versus Rounded Shapes, Society for Personality and Social Psychology, PSPB, 32,6:
794 – 805.
26. Hofstede G (2001) Culture’s Consequences: Comparing Values, Behaviors and Organizations
across Nations 2nd Ed. Sage Publications.
27. Badre AN (2000) The effects of cross cultural interface design orientation on World Wide
Web user performance. Source: http://www.cc.gatech.edu/gvu/reports/2001/abstracts/01-03.
html/.
28. Simon SJ (1999) A Cross Cultural Analysis of Web Site Design: An Empirical Study of
Global Web Users, Eighth Cross-Cultural Consumer and Business Studies.
29. Chau PYK, Cole M, Massey AP, Montoya-Weiss M, O’Keefe RM (2002) Cultural differences
in the online behavior of consumers. Communications of the ACM, 45(10):138–143.
30. Faiola A., Ho CC, Tarrant MD, MacDorman KF (2011) The Aesthetic Dimensions of U.S. and
South Korean Responses to Web Home Pages: A Cross-Cultural Comparison, Int J Hum
Comp Inter, 27(2):131–150.
31. Tractinsky N (2006) Aesthetics in Information Technology, Human Computer Interaction and
Management Information Systems: Foundations, Armonk, New York: ME Sharpe.
32. Reinecke K, Bernstein A (2011) Improving performance, perceived usability, and aesthetics
with culturally adaptive user interfaces. ACM Trans. Comput.-Hum. Interact. (18)2, Article 8,
29 pages. DOI 10.1145/1970378.1970382 http://doi.acm.org/10.1145/1970378.1970382,
viewed on 19. 7. 2014.
33. Desmet PMA, Hekkert P (2002) The basis of product emotions. v: Green W., in
Jordan P. (2000)(Eds.), Pleasure with Products, beyond usability, London: Taylor & Francis.
34. Ou YK, Liu YC (2012) Effects of sign design features and training on comprehension of traffic
signs in Taiwanese and Vietnamese user groups. International Journal of Industrial
Ergonomics, 42:1 – 7.
35. Khalid HM, Helander MG (2004) A framework for affective customer needs in product design
Theor. Issues in Ergon. Sci., 5(1): 27–42.
36. Schütte H, Ciarlante D (1998)Consumer Behavior in Asia (London: Macmillan).
37. Yang F (2001) Transformations in New Immigrant Religions and the Global Implications.
American Sociological Review, 88: 269 – 88.
38. Osgood CE, Suci GJ, Tannenbaum PH (1957) “The Measurement of Meaning”. University of
Illinois Press: USA.
39. Berg-Weitzel L, Van Den Laar G (2001) Relation between culture and communication in
packaging design, Journal of Brand Management, 8, 3: 171–184.
40. Grossman RP, Wisenblit JZ (1999) What we know about consumers’ color choices, Journal of
Marketing Practice: Applied Marketing Science, 5(3): 78–88.
41. McManus IC (1981) The aesthetics of color, Perception,10: 651–666.

42. Ricks DA (1983) Big Business Blunders: Mistakes in Multinational Marketing (Homewood,
IL: Dow Jones-Irwin).
43. Neal CM, Quester PG, Hawkins DI (2002) Customer Behaviour: Implications for Marketing
Strategy, 3rd edn (Roseville, NSW:McGraw-Hill).
44. Nagamachi M, Lokman AM (2011)Innovations of Kansei engineering, Taylor & Francis
45. Lewalski ZM (1988) Product Esthetics: An interpretation for Designers, Carson City, NV:
Design & Development Engineering Press
46. Gordon K (1909) Esthetics, New York, Henry Holt and Co.:166–168.
Chapter 7
The Kansei Design Approach at Toyota
Motor Europe
Carole Favart, Daniel Esquivel Elizondo, Dr. Alexandre Gentner

and Théo Mahut
Abstract The aim of this chapter is to present a first understanding of the Kansei
design approach and how it can be bridged within an industrial context. It proposes
a theoretical point of view of our division build on researches still conducted to link
Kansei philosophy and user experience in a first time and then to understand how an
interaction can impact the full user experience. Furthermore, the automotive context
of these researches leads us to develop and use several methodologies and tools that
are presented in this chapter. We develop them in order to increase what users live
with their products from an early design point of view, beyond the product itself.
7.1 Introduction
Kansei is a wide word, encompassing complex notions such as emotion, sensation,

semantic, affectivity, and need and approaches through different fields such as
Kansei engineering, Kansei science, or Kansei design. This chapter will not define
these fields, or even tackle the understanding of the full Kansei’s scope. What we
will present you in the following sections is our own vision of what we used to call
the Kansei design at Toyota Motor Europe. It highlights our researches focused on
C. Favart (&) D. Esquivel Elizondo (&)

Kansei Design, Toyota Motor Europe R&D, Hoge Wei. 33B,
1930 Zaventem, Belgium
e-mail: carole.favart@toyota-europe.com
D. Esquivel Elizondo
e-mail: daniel.esquivel@toyota-europe.com
Dr. A. Gentner (&)
Kokekoko SPRL, 1 rue Sainte Marie bte31, 1080 Brussels, Belgium
e-mail: alexandre@kokekoko.eu
T. Mahut (&)
Laboratoire Conception de Produits et Innovation, Arts et Métier ParisTech,
151 bd de l’Hôpital, 75013 Paris, France
e-mail: theo.mahut@yahoo.fr

DOI 10.1007/978-3-319-29433-9_7
120 C. Favart et al.
user-centered subjective conception according to three key understandings: The first

one is dedicated to what our division is constituted of a history (Sect. 7.1.1); a
structure (Sect. 7.1.2); and an approach and positioning (Sect. 7.1.3). Then, we
propose a first approach on our plenty of researches bridging Kansei process and
user experience scope (Sect. 7.2), highlighting a Kansei experience framework that
we consider as our predevelopment activities’ territory. Furthermore, researches are
conducted to link this vision of Kansei experience to interaction design. Indeed, we
believe that the way we design memorable interactions can impact and increase the
full user experience. This vision is presented in the section named experiential
interaction (Sect. 7.3). Finally, the very industrial position of our researches and the
constitution of the Kansei design team with both profiles of researchers and
designers (engineers, designers, styling designers, etc.) lead us to create, test, and
use several tools and methodologies that the last part of this chapter introduced as
the industrial materialization of our research.
7.1.1 KD History
Although Toyota started selling cars in Europe during the 1960s, Toyota Motor
Europe (TME) was only set up as a holding and regional headquarter in 2002. The
mission of each R&D Center is linked to specific leading-edge fields that might
nurse and generate value to the global company. Europe is considered as the most
complex zone in the automotive world, due to contrasted cultural aspects as well as
contextual, and it stands to be a reference for driving dynamics. As example, the
future of mobility, such as “autonomous driving”, is widely tested on this continent.
From our Kansei perspective, this is conducted through prospective studies related
to the impact of new type of motion control on human behaviour and perception.
Since 2008, the economic crisis dramatically changed some conservative way of
thinking toward a more user-centered approach where the emotional side of the
experience became a mandatory requirement.
To create a new ethics of auto mobility, in sum, will require a deep shift in automotive
emotions, including our embodied experiences of mobility, our non-cognitive responses to
cars, and the affective relations through which we embed cars into personal lives, familial,
networks, and national culture [40].
Kansei design division (KD) has been created in 2005 at first to assess liking and
study new KPIs for target setting, based on sensory stimuli, etc., which quickly
raised the crucial need for new methodologies and tool studies within breakthrough
strategy “TME uses Kansei engineering techniques for assessing, but has come to
recognize the need for a design approach in the first two parts: understanding and
creating, i.e., the need for a Kansei design approach to create a Kansei space”[26].
And if “Kansei” is Japanese word that stands for people’s affective process
generated following sensory perception, it encompasses notions such as emotions,
feelings, and impressions. It is one reason for which we are strengthening our
7 The Kansei Design Approach at Toyota Motor Europe 121
Fig. 7.1 Kansei span of activities from Kansei Design perspective
approach on Kansei design researches on the upstream phases of the development

process and developing it by collaborative studies with Kansei engineering as well
as Kansei science experts (Fig. 7.1).
7.1.2 KD Structure
KD is a division of TME-R&D, at the intersection of design and product planning,

playing the role of translator or bridge between departments and warrant of the
overall consistency. The multidisciplinary team is composed by designers (auto-
motive, product, and interaction), engineers, and researchers, some of them with
new double-education profiles, such as business. The role is cross-functional, which
means many collaborative activities with major divisions in Europe and Japan.
KD team has the particularity of being the only division with no “head division”
at Japan headquarter (TMC—Toyota Motor Corporation). It is also quite young
(approach initiated in 2003 and division created in 2005) and relatively small (5
core members and around 9 non-permanent members).
They are playing an active role in both advanced models and vehicle develop-
ment studies, supported by in-depth research with academics in early phase.
It focuses on the Kansei and subjective aspects (mainly not rational and logical
aspects) of consumers’ perception. From that perspective, it participates in research
and development activities related to future vehicles and to mobility in general.
KD’s approach and the fields tackled have evolved together with the maturation of
the division. Thanks to long-term collaboration that started 10 years ago with
academics (Fig. 7.2).
Fig. 7.2 Evolution of TME-Kansei Design over time
Fig. 7.3 Simplified

representation of the Kansei
Competency Center structure
Since 2011, KD management initiated the Kansei Competency Center

(KCC) (Fig. 7.3), to strengthen this cross-divisional management, which interacts at
different levels for innovative concept studies and projects. “Tailored short-term
teams with radically diverse yet relevant skills help identify opportunity areas
quickly” [1]. It consists in a collaboration platform permitting the stakeholders of
R&D projects to collaborate more easily on Kansei-related topics and to communicate
about them to Toyota headquarters with a unique European voice. Even though it does
not much change the nature of the collaborative activities that were already in place,
this structure enhances the visibility and acknowledges the utility of Kansei-related
collaborations between product planning, style design, and engineering departments.
As it will be explained in the next section, within a flexible and agile structure,
methodologies and tools are studied which are tested, broken down, and then
deployed through an iterative PDCA cycle (Plan Do Check Act) .
In summary, creation of new methodologies toward innovative Kansei design
strategy starts to influence and nurse the mid- to long-term vision of the future
model lineup for Toyota and Lexus. In the next section, we will explain the current
status of our way to proceed.
7.1.3 KD Approach and Positioning
The approach initially followed by designers (i.e., based on the experience and
intuition and on abductive reasoning) focused on the multisensory qualities of a car
and its overall consistency (e.g., materials and color). It rapidly integrated a com-
plementary quantitative point of view based on a more scientific reasoning (i.e.,
analytical approach). The start of the collaboration with the CPI Laboratory played
a major role in this evolution. It also permitted better study of the response to
perception (i.e., associated meaning, emotions) of potential users and thereby made
it possible to better guide design directions. Both are now being combined in a set
of integrative tools and methodologies. Supported by Ph.D. researches, it sub-
stantially contributed to establishing this approach inside (use in development
projects) and outside (publications) the company.
With time, interaction also became an additional field of study. Recently, the
notion of “experience” gained importance. It is now almost exclusively used when
describing KD’s concerns and field of study. Indeed, this notion encompasses
several fields (i.e., perception, response to perception, interaction) while staying
focused on the user’s affective mental process. As a whole, it is now referred to as
the “Kansei design” approach (integrative thinking and focusing on experience).
Alexandre Gentner was the first Ph.D. researcher to work for KD. His researches
as well as the different projects he conducted impacted greatly the recent evolution
of the division regarding the type of thinking used and fields tackled. This research
also contributed to establish the structure of KD research and predevelopment
activities. In that sense, this dissertation will highlight the main theoretical (i.e.,
framework, model) and some of the practical characteristics (i.e., tools, method-
ologies) of the “Kansei design” approach. Without communicating confidential
information related to ongoing vehicle development projects, it will explain how the
approach can be applied to the early phase of industrial development projects (i.e.,
tools, methodologies leading to early representations) and will give hints about how
the approach contributes to downstream design activities.
KD’s current activities impact the vehicle development process at different
stages. In upstream phases, experience-focused concepts are investigated. In later
stages, KD is involved in the design development of specific vehicle parts and
materials impacting the resulting user experience, as well as in the evaluation of
perceived Kansei qualities of the full vehicle under development. Most of these
activities imply collaborative activities with other TME and TMC departments,
suppliers, and/or external partners. The cross-functional importance of such con-
siderations was acknowledged by top management and led to the recent creation of
the “Kansei Competency Center” described previousely.
As mentioned earlier, the focus of Kansei design researches is the upstream
phases of the development process. This means the researches are related to
experience-focused concepts. These concepts represent experience directions that
might influence different types of future development projects. Three contexts for
these early representations of user experience can be distinguished: “exploratory
concept” (propose new experience concepts for future breakthrough products),

“product lining strategy” (identify user experience logics and directions for future
platform products), and “predevelopment direction” (prepare grade and character
strategies for future incremental products).
7.2 User Experience and Kansei Process
The notion of experience (used in the terms user experience, product experience,
and experience design) is now used more and more in the literature when describing
a (instrumental or not) human–artifact interaction. Ortíz Nicólas and Aurisicchio
[34] analyzed 11 user experience frameworks from the literature in an attempt to
bring together in a consistent overview the rapidly growing and disjointed literature
on the subject. The conclusion of this research suggested that even if the per-
spectives and focus points of the 11 researchers were different, common constituent
elements (user, interaction, artifact, context) and aggregates (subjective, conscious,
emotional, interconnected, dynamic) of user experience were acknowledged by the
majority of the perspectives reviewed.
Kansei studies are usually cross-disciplinary and involve researchers from fields
such as brain sciences, psychology, and engineering design, and design or mar-
keting research. Although the word Kansei is widely used in Japanese design
research literature, it is usually only briefly defined as an introduction to the context
of the study presented and is interpreted in a variety of ways [25]. Some of the
reasons pointed out are that the notion is impossible to transpose directly into
English, that it is closely connected to the Japanese culture [42], and that the
literature intending to provide a definition struggled over time to come up with a
single and clear definition [27]. Lévy et al. [27] described it as the function of the
brain related to “emotions, sensitivity, feelings, experience and intuition, including
interactions between them” (p. 9). It is further described as originating in one’s
sensory perception and personal characteristics (Kansei means) and providing as
output a qualitative meaning and value of the environment (Kansei result). Notably,
Lévy et al. indicated that the flow between Kansei means, process, and results is not
strictly linear and that these different aspects influence each other.
Part of the originality of KD is that it intends to combine notions from the
“Western” emotional and experience design research field as well as from Eastern
Kansei research. As could be seen in the previous sections, both have in common
the fact that they describe the human subjective process involving an affective
dimension and following the perception of artificial construction (product, inter-
action, service, etc.). Indeed, in the same way experience is distinguished from
usability (arising from the logical behavioral level [33]), Kansei is distinguished
from chisei (leading to intellectual understanding) and opposed to risei (logic
process). In order to define a clear context for the experiments, we will use this
section to put both points of view in perspective and build a summary “Kansei
experience framework” that will be used as a basis for the following discussions.
First, it is interesting to note that there is a difference in terms of focus point

between the two notions: “Kansei” is centered on one’s subjective mental pro-
cesses, whereas “experience” is described from a point of view that encompasses an
environment (including at least a product) and a user. This second perspective is
clearly expressed by the statement, “experience is not a property of the product but
the outcome of human-product interaction” [10: p. 63]. The emotional and expe-
rience design perspectives generally only describe visible outputs of mental pro-
cesses (e.g., pleasure, appeal, emotions, semantic association). These visible
outputs correspond to the Kansei direct consequences described by Lévy et al. [27].
As mentioned previously, one reason for it is that emotional and experience
design models “are not intended to be ‘true’ to reality (realism), but to be useful
(instrumentalism)” [5: p. 9]. It is only very recently that researchers [7] have
attempted to link emotional and experience design frameworks and models from the
field of neuroscience.
This initiative is very interesting because it puts them at the same level as Kansei
research frameworks. The Kansei process [27] and the neuroscience model [7] both
describe mental processes taking their origins in signals captured by one’s senses
and influenced by one’s personal characteristics such as personal values, past
experiences, and sociocultural references. In that sense, Kansei process corresponds
to both roads (i.e., high road and low road) presented in the neuroscience model as
it is defined as “gathering the functions related to emotions, sensitivity, feelings,
experience and intuition, including interactions between them” [27].
Both perspectives can actually be combined in a framework covering the context
of a user–product interaction in a given environment (context of an experience), the
Kansei process, and the result of this process: perceived Kansei qualities. They
correspond to Kansei direct consequences [27] including user responses such as
pleasure, meaning elaboration, and primary and secondary emotions [7]. This
framework is presented in Fig. 7.4.
As explained previously, rational understanding (related to the understandability
of functions, usability) is dissociated from the perceived Kansei qualities but is still
represented on the framework as a factor influencing the Kansei process. The other
Fig. 7.4 Generic framework combining Kansei and user experience design perspectives [16]
Fig. 7.5 Kansei experience framework [16]
influencing factors represented include the user’s personal characteristics and at-
tributes from the environment.
A simplified framework has been introduced by Gentner [16]: the “Kansei
experience framework” (Fig. 7.5). It is centered on the specific focus of this dis-
sertation. In this framework, the centers of interests are the notions of experience
and of Kansei process (i.e., not rational processes), as well as the three core entities
of user experience: the user’s personal characteristics, the user’s perceived Kansei
qualities, and the attributes of the environment.
The Kansei process is represented as creating a link between the three user
experience entities. Perceived Kansei qualities encompass notions such as pleasure,
meaning, and emotions. Personal characteristics cover notions such as the user's
culture, values, personality traits, mind-set, as well as memory. Finally attributes
from the environment include descriptions of products, interactions, and elements of
the context involved in an experience. Notably, the framework also retains the four
constituent elements of an experience identified by Ortíz Nicólas and Aurisicchio
[34]: user, interaction, artifact, and context.
7.3 Experiential Interaction
As we saw previously through the Kansei experience framework, concrete links

between user experience and interaction have already been done. Nevertheless, the
relation between user experience and interaction is still difficult to identify, because
of it youthfulness. So how to define interaction in order to state on the right balance
between user experience and interaction?
Researches conducted at KD by Ph.D. student Théo Mahut [28] highlight that
interaction is the action accomplished by the user or by the artifact on each other
that influences or modifies the user’s motor, perceptive, cognitive, and affective
systems [19]. Interaction can be physical (driving a car) or non-physical (con-
templating a car) [21]. Interaction binds user, artifact, and even the context [16, 43].
Furthermore, according to Desmet and Hekkert, interaction refers to instrumental
interaction (function), but also to non-instrumental (no function) and even to

non-physical interaction (no touch), because each of these consequences can gen-
erate physiological, motivational, or motor responses [3, 33]. So, we consider
interaction as a dialogue between user and product (service or a system), in a
particular context. And this dialogue is not specifically based on the use of an
advanced technology. Interaction surrounds the design process, which creates a way
to make it easily usable, useful, desirable, and profitable in relation with the
product. During the early stages of design, designers tried to respond to a given
design brief and find the good conception. It is because the product plays a critical
role as a precursor to consumers’ cognitive and affective responses [8].
To build a bridge between user experience and interaction, this section suggests
a new notion: “the experiential interaction.” Indeed, if the user experience field
encompasses many situations dedicated to this dynamic phenomenon such as the
driving experience, a simple interaction is much more focused on a microlevel, for
example, the way we touch, we hear, we smell, and we watch. Each of these
elements is considered as interaction. So, how to consider the set of meaningful
interactions that an element can provide? We consider in our work that the user
experience (like driving a car) depends on the way we live a simple interaction.
Nevertheless, if an interaction deeply impacts the user experience, we can
acknowledge that a sequence of interactions is even more decisive in our user
experience. This is the reason why we characterized the sequence of interactions as
an experiential interaction. For example, a seat, a steering wheel, and a gearbox are
products. But if we consider them through their impact on the user experience, we
focus on the experiential interaction impacts.
The notion of experiential interaction is for us a way not only to focus on an
interaction, but also to understand and design sequences and transitions between
each interaction. Working on what we call experiential interaction is for us a
suitable solution for the understanding of the entire scope of features that the user
experience includes, pointing out a direct link between interaction and user
experience.
Furthermore, our understanding of experiential interaction leads us to take a
position on the understating of interaction: The two following sections highlight our
vision of unilateral interaction and reciprocal interactions, represented in Fig. 7.6.
Fig. 7.6 Link between user experience; experiential interactions; and unilateral and reciprocal
interactions
7.3.1 Unilateral Interaction
Can we consider simple emotion as an interaction? The emotional design field

acknowledges the fact that an emotion is a response to an interaction. A stimulus
impacts internal and external cognitive processes according to the community
[12, 29]. It affects the dominance [32], the activation, and the positive and negative
affects of the user [2, 9, 36, 44]. Those kinds of reactions to the stimulus imply what
[19] characterizes as changes of the user. These changes are user feedbacks even if
there is no product’s state modification. This section proposes to call these
responses the “unilateral interaction.” It refers to responses that do not affect the
state of the product. For example, [33] highlights the visceral relation to a product,
where interactions are mainly focused on the ability to see, hear, or even touch to
feel. Furthermore, these kinds of unilateral interactions that affect feedbacks are
close to what [22] consider as automatic responses in interaction. If an interaction is
a link between the user and the artifact, characterized through feedbacks, we can
acknowledge that looking at something is an interaction because the stimulus (e.g.,
a blue carpet) involves the user system through his sensory process (mainly based in
this case on the ability to see). It produces feedbacks like changes of the user state.
Furthermore, we can also observe the reverse phenomenon, interactions from
Artifact to User: the way an Artifact captures information through its sensors
impacts his responses.
Thus, “unilateral response” impacts the user and produces changes or feedbacks.
It affects the way one perceives the other (interchangeably, depending on who is
capturing the stimuli: the user or the artifact). This entire process is named “the
unilateral interaction,” by opposition to the reciprocal interaction advocated in the
next section.
7.3.2 Reciprocal Interaction
If unilateral interactions do not imply a functional reaction through a state’s evo-

lution from the user (unilateral interaction), others can. The physical response of the
user (or the artifact) is a typical choice reaction task in which each stimulus is
assigned to a unique physical response. The main particularity of this reciprocal
response is that it shifts the other conditions (user’s or artifact’s conditions).
According to Proctor and Van Zandt [35], interaction is defined through “response
selection” and “response execution,” and the execution could be dependent on motor
responses. Thus, the way a user responds to a stimulus could also be a response
based on user motor capacity. It means that interaction could be a reciprocal system
where the user interacts physically to stimuli. To conclude, the reciprocal interaction
is proposed as a process where the user characteristics identify a stimulus and
respond to this stimulus through different steps: response selection and then
“response execution.” In this case, this is a “reciprocal response.” It impacts both the
user and the artifact’s status and produces changes or feedbacks to the user and the
artifact. This reciprocal response finally impacts the perception of the other (user or
artifact). This entire process is named the reciprocal interaction.
Finally, this section highlighted that we can build links between user experience
and interaction. Thus, a sequence of unilateral and reciprocal interactions can be
regrouped as the experiential interaction. And then, a set of experiential interactions
can be understood as what is called the full user experience.
7.4 Tools and Methodologies
Experience-centered tools and methodologies have been created to support the

different design activities such as the ones described by Bouchard and Aoussat [4]
(see Fig. 7.7).
This model has the particularity of being fractal as it can describe information
processes at different levels. At a microlevel, it can be used to describe the reflexive
conversations between the designer’s mental representations and externalized rep-
resentations. In this case, the “seeing–drawing–seeing” cycle described by Schön
and Wiggins [41] can be superposed with the “information–generation–evaluation”
cycle. The time span of such a cycle is typically counted in seconds or minutes. At a
more macrolevel, used also to discuss the tools and methodologies, it can be used to
represent a early-phase design activities typically leading to 2D or 3D visualization
of a product concept or to guidelines related to design strategy (i.e., idea/concept
representations, character/differentiation strategy). In this latter case, the design
informational cycle’s time span is counted in weeks or months [4].
All the tools and methodologies that will be presented have as common char-
acteristics the fact that they contribute to improving the user experience (through
information, generation, or evaluation and decision activities), but they differ in the
way they treat the “user.” Depending on the tool or methodology, he/she can be
treated either as a subject (observed and questioned) or as a partner (participatory
design). When treated as a subject, “users” are either directly (e.g., interviews) or
indirectly (e.g., field observations) involved in the design activities [39]. In fact,
Fig. 7.7 Design activities—design informational cycle (adapted from [4])

at this NCD stage, it is preferable to speak about a “desired targeted user.” For this
reason, the term “user” will be put between quotation marks (i.e., “user”).
7.4.1 Tools and Methodologies Related to the Design

Activity “Information”
Design team members gather various types of information in order to build

knowledge related to the project’s context. The activity consists in questioning the
initial intentions from various perspectives (intended user, political, economical,
environmental, social, technological, brand) by collecting and organizing data. The
knowledge built is thereby mutually shared and can also be used as a source of
inspiration in order to prepare generation and other subsequent activities [37].
7.4.1.1 Tool to Represent Experience Directions: EUPI
Kansei qualities such as meanings and emotions are used by the user to describe its
experience with a product or service. The same process might be applied to more
abstract concepts like brand. Multiple comparisons are possible on the relative
weight on each keyword of KQ. In addition, multidimensional mappings can be
calculated to visualize their relative distances. However, these frameworks are
dependent on the products under study which limits its relevance to be used to
position future products or more abstract concepts like brand. The aim of this
research is to develop a tool based on experience independent from the products
allowing multiple comparisons of products and brands becoming a system of ref-
erence to discuss an “intended experience.”
One of the components of experience is the user’s personality consisting of an
aggregate of behaviors greatly influencing the user’s tendency to appreciate or
prefer specific objects or services. Among several branches of research, “person-
ality trait approach” states that each individual personality is a unique combination
of various traits that interact among each other. We have used the five-factor model
of personality, which is the most widely used tool in research, in order to study the
Kansei qualities that are triggered by each personality trait. The Kansei card tool
was used to facilitate the generation of KQ. It consists of a collection of iconic
images from different sectors of inspiration: animals, sports, leisure activities,
gestures, instruments of music, etc.
The personality trait under study was displayed and explained. Participants,
mostly designers, were given 10 min to walk around the room and go through the
set of Kansei cards on the table, pick the visual stimuli that they feel to be correlated
with the personality trait, and write on a Post-it the reason why they chose this
image. After laying all images on a board, several participants clustered elements
expressing similar ideas. Each workshop was conducted twice with a different
population of designers. Clusters generated were then grouped into families by one
member obtaining 3 families of experience per personality trait. In total, 30 “ex-
periences” were generated containing Kansei cards and keywords.
The semantic content was further structured inside each experience depending
on its nature: values, personality, emotions, and meanings. Additional keywords
had to be added in order to complete this classification. New visuals were generated
moving away from iconic content (Kansei cards) with a more specific imagery. This
imagery was structured as well according to 3 levels of abstraction: human
behaviors and fashion, product and spaces, and shape and pattern.
The 30 experiences were quantitatively correlated with a set of 10 target cus-
tomer segments by an expert panel. We could analyze these data in order to gen-
erate a 2D mapping space of the 30 experiences and target customers. Axes were
named according to the positioning of the experiences. We use this space as a
visualization tool to position intended experiences of products and brands.
Concerning brands, visionary panel evaluated the correlation of several brands with
each experience board allowing us to map and compare the different experience
territories.
7.4.1.2 User Research
As an activity aiming at improving future user’s pleasure, emotions, and well-being

when in contact with Toyota products, user research tools are key starting points for
our process. They can be either indirectly involved such as for the EUPI tool
introduced previously or directly. In the latter case, they can be involved as subject
or as partner.
When involving them as subjects, typical face-to-face and online tools are used.
Online tools are being used when bigger participant pools and mostly quantitative
data are needed. In 2012/2013, a research collecting descriptions of experiences
provided by products that participants owned was for instance conducted. It per-
mitted to identify 15 distinct types of experience based on real-life situations. Those
combine descriptions of the product, interaction and context of use attributes as well
as the Kansei qualities elicited [18]. It is a good example of online research con-
ducted by KD in the field of user experience.
Face-to-face individual and focus group interviews are also used complementary
to online research when involving participants as subjects. These are particularly
relevant to collect qualitative data regarding for instance a target audience’s life-
styles and habits in terms of interactions with products or services.
When it comes to understand their opinions on prospective topics, such as
discussing potential future experiences, we tend to involve them as partner of the
research activity. This means that they are asked to discuss and build on top of
material previously created by the KD team (such as images, multisensory samples,
and storyboards). In this case, the research covers both “information” and
“generation” activities. The notion of cocreation will be further discussed in the
following section related to “generation” design activity.
7.4.2 Tools Related to the Design Activity “Generation”
This activity consists in the generation of new ideas and new concepts. This is
achieved using the collected data, mental images, and other information contained
in the members’ memory. The design team members thereby generate intermediate
(physical and/or digital) representations.
In this section, two types of tools will be presented. The first one corresponds to
cocreation tools enabling the discussion of Kansei-related topics with users and
other divisions within Toyota. It will be illustrated by the Kansei cards tool. The
second one are tools used by Kansei design to design great user experiences
focusing on interactions. In this case, tools from the fields of service and interaction
design will be discussed.
7.4.2.1 Cocreation: Kansei Cards
The intention of Kansei cards is to enable participants of collaborative design

sessions to identify and communicate their experience-related expectations or
impressions regarding a context. Researches such as the ones conducted by Eckert
and Stacey [11] and McDonagh and Denton [31] showed that pictures have the
ability to convey a wide range of design information. This notion motivated the use
of picture as a main vector of information for this tool.
The Kansei cards tool is composed of different sets of pictures (i.e., families).
Each family should focus on particular categories of design information, and the
pictures from each family should cover the widest possible spectrum of variations
within these categories. Through a pilot study, it was validated that participants
easily extracted key design information from the pictures and used these pictures to
illustrate one of their ideas [15]. Another learning point from the pilot study was
that it was easier to analyze and compare the association made with families of
items commonly known across age-groups and cultures (music instrument or ani-
mals vs. cities or cartoon heroes).
Over fourteen families of Kansei cards have been created. Most of them are
image-based and follow specific themes (i.e., patterns, landscapes, animals, music
instruments, sports). Additional three families of keywords corresponding to
semantic keywords, emotions, and values complement the tool. Table 7.1 presents
11 families of Kansei cards together with the main design information they com-
municate. The real cards measure approximately 10 10 cm each and are made
out of rigid cardboard.
Two use cases of Kansei cards will now be presented. More details and
explanation about them can be found in a previous publication [17].
A first possible use of the Kansei cards are by participatory design sessions
involving potential future users. Guided with predefined samples and a protocol
including association sessions and semidirected interviews, the participants are able
to construct their own images of intended user experiences related to a specific topic
Table 7.1 Example of 11 families of Kansei cards

Family topic Main category of design Example of pictures
information
Simple shapes – Semantic descriptor
– Visual attribute (shape)
Animals – Value
– Semantic descriptor
– Emotion
– Product characteristic
– Gesture
Natural – Value
landscapes – Semantic descriptor
– Emotion
Chairs – Style
Sports – Value
– Emotion
– Interface characteristic
– Temporal context
Arm gestures – Semantic descriptor
– Emotion
– Gesture
Semantic – Semantic descriptor
keywords
Emotions – Emotion
(e.g., future hybrid car atmosphere) including pictures and words. Statistical analysis
later permits to construct main experience directions based on the participants’ input.
The directions cover design information with low to high levels of abstraction and
include explicit keywords and inspirational pictures and color harmonies (Fig. 7.8).
In the second use case presented here, the Kansei cards are used to support user
experience-related discussions and creativity sessions in the context of cross-
divisional and multicultural teams. Participants are in this case asked to discuss the
position and finally place selected categories of Kansei cards on a two-dimensional
mapping in line with the topic of the generation activity (Fig. 7.9).
Serene/peace of mind Energy

Harmony/pure Duality
Relaxing Power/dynamism
Joy
Hue: light colors, often Hue: bright, saturated,

cold (blue, green, white) vivid colors
Contrast of hue: Contrast of hue:

L-D contrast: L-D contrast:
Fig. 7.8 Example of two experience directions identified following participatory design sessions
Fig. 7.9 Two-dimensional

mapping with Kansei cards
positioned on it
When comparing it with a standard presentation of the topic of the generation

activity, this preliminary activity involving Kansei cards appears to have positive
impact on the concepts later generated in terms of both quality and quantity [17].
This led us to think that the use of the cards tends to break down walls between
participants of different genders, functions, and nationalities. In that sense, the cards
probably positively influence factors such as reciprocal understanding, cross-cul-
tural communication, and trust [20] within the various teams of big corporations
such as Toyota.
7.4.2.2 Tools for the Generation of New Interactions
This category of tools enables Kansei design to design great user experiences
focusing on interactions. In this case, the tools used come mostly from the fields of
service and interaction design.
In order to explore new interactions, bodystorming and role-playing [24] are
commonly used among the Kansei design team members. These tools allow
exploring and exchanging about new solution spaces during ideation sessions in
groups.
Fig. 7.10 Example of a

blueprint
Scenarios are also used for all interaction design projects. These narrative
approaches allow the design team to better explore and convey its views about new
experiences [38]. They permit the validation of hypotheses (regarding the way users
act, think, and experience) made by the designer during the information activities
[14]. The user journey narration identified is then often transcribed in storyboards
[6]. The fact that they rely on series of drawings in the style of a cartoon allows a
quicker and more immersive understanding on the different design team members.
In the case of scenarios involving an ecosystem of stakeholders (typically the case
of service design projects), blueprints are used [23]. It displays the different stages
of the scenario on a horizontal flowchart above and below the line of visibility to
the user. The user experience touch points and the functions are placed above the
line of visibility, and the back-stage processes and related stakeholders corre-
sponding to the different user experience stages are organized accordingly below
the line of visibility (Fig. 7.10).
7.4.3 Tools and Methodologies Related to the Design

Activity “Evaluation and Decision”
It corresponds to the activity of assessing the proposed concepts and deciding

which ones (if any) should be maintained in the development process. Depending
on the context, the decision can be taken by persons involved in the design process
or by decision-makers external to it.
7.4.3.1 Quantifying User’s Kansei: SQP
Kansei qualities like appeal are proven to be key drivers of preference in many
studies involving real customers. One of KD's role is to input in the process of new
concept development the future expectations of customers. Inside the field of
Kansei Engineering, its first challenge was to create a tool to define the relative
importance of each component (steering wheel, seats, instrument panel, etc.)
towards the overall interior appeal perceived by customer. Customer observation

and ad hoc interviews can help design team grasp the spoken needs of the customer.
However, aesthetic appeal is proven to be an outcome that lies mostly in the
subconscious processes of human cognition; therefore, language-based tools, by
definition cognitive conscious processes, are not the most suitable. Our aim is to
create a tool providing us with a quantified measure of the factors influencing the
appeal outcome using subconscious approach. More specifically, this tool would
define to what extent each component in car interior influences the overall interior
appeal. It was named internally by Daniel Esquivel, sensory quality prioritization
(SQP), and became well adopted by all division partners.
Participants are no longer asked to provide which components they feel are more
important in the car interior in terms of appeal, given that this is a conscious
measure, but rather on to what extent each of them is appealing, through Likert
scales. Finally, the participant provides a score of the overall appeal of the car
interior. After we aggregate all input into a two-way matrix of appeal scores of
components and overall appeal, we run a linear correlation calculation. This will
calculate to what extent the appeal scores of each component are correlated with the
overall appeal score of the car interior. In this way, we obtain a “weight” (corre-
lation score) for each component, ensuring to tap on the subconscious process of
participants, who were never asked directly to provide us with this “weight.” The
principle behind this tool is basically Kansei engineering, which aims to use
engineering techniques to predict or assess subjective criteria of design. One of the
requirements to apply this methodology is to have a number of entities (5–10) with
similar function purposes. It will hardly work if we compare cars with bicycles.
Several applications of this methodology are possible, depending on the
concept/product development stage. We have successfully applied this tool to
predict overall car interior appeal. However, it can be used as well to define more
specific product attributes in product development: i.e., maximize the appeal level
of steering wheels to define which elements have a stronger positive correlation:
decorations, surface finish, shapes, patterns, controls, touch properties, and gloss
levels. Or it can be used to predict the touch appeal of plastic samples with 15
different paint effects.
7.4.3.2 Kansei in Motion—Eye-Tracking Tool
The Kansei qualities to express the user experience evolve in time depending on the
degree of interaction with the product or service. During static interactions, users
can explore the vehicle in many ways: walking around the car to admire the
different proportions and shapes and getting inside the car to feel the seat material
and the overall interior atmosphere. However, in active interaction, the user inter-
acts with the product with the intention to accomplish a task or function: turn the
knobs to regulate the air con, turn on the engine, and drive and park the car in town.
The quality of active interactions can be broken down in 3 topics of study: intu-
itivity, usability, and meaningfulness. How to measure them? Questionnaires and
Likert scales can give us an idea; however, it does not tell us the reasons behind this
score. Vision is the most important element when interacting with products. Which
tools could be used to study to get further understanding of our gaze behavior? Our
aim is, by analyzing gaze patterns in interaction, to provide a quantifiable indication
of the system performance.
In order to develop a method to study the gaze in active interaction, a fixed
sequence of interactions was defined, in both driving conditions and showroom
conditions. Ten participants conducted the test followed by an interview. In addi-
tion to the eye-tracking, they were video-recorded in order to keep track of the hand
movements. One of the tasks to conduct while wearing the eye-tracker is “to find a
specific radio station.” While driving, participants had to identify the button to turn
ON the audio and then find the FM mode and scroll stations until the station they
had to find. After obtaining the data from all drivers, calculations were conducted
and heat maps obtained. It showed evidence of the poor level of intuitivity of the
interface of one of the cars under study. Pairing the results of eye-tracking with the
hand-tracking from the video-recordings helped designers identify confusing ele-
ments in the layout and define a more natural mental model of audio interaction.
7.4.3.3 Evaluation Platform
The way users interact with products influences the way users live their experi-
ences. This Kansei design department way of thinking leads us the creation of a
platform dedicated to interactions through the user experience vision. The aim of
this evaluation platform is to understand how a meaningful interaction can influ-
ence, impact, and regulate the experience users are living.
Concretely, we are creating an evaluation platforms organized in two steps. The
first step is dedicated to the understanding and assessment of any kind of existing
interaction and user experience to highlight guidelines, principles, and future
opportunities.
On the other side, the second step assesses a creation, a concept, or a prototype
through the same criteria than an existing product in the step one. The goal is to
prove by comparison that we improved the product. Thus, the evaluation platform
aims at evaluating existing products to highlight guidelines for conception, to
finally evaluate the new creation through the same criteria than the original product.
We are developing this evaluation platform through methodologies and tools. The
following section proposes a first and summarized understanding of this platform:
Approaching the evaluation phase means to evaluate both existing human–
product interactions and prototype of interactions’ concept. To reach this goal of
evaluating interactions from early design or existing design, we decided to assess
interactions and user experience through the Kansei criteria:
– The first one is the “sensory level.” It focuses on the sensory involvement of a
user while interacting. For example, while driving, a user needs his capacity to
see, to hear, to grasp the steering wheels, and so on. Evaluating users’ sensory
involvement is also helpful to take off information such as time to find an

element, the attractiveness of elements, zones, and sounds, if it is a peripheral
interaction or not. All of these data are collected by both objective and sub-
jective measurement tools, such as eye-tracking, camera, and hand-tracking, and
by user’s transcription of sensory perceived significance, through discussions
and questionnaires.
– The second criterion is the emotional one, focused on the understanding of how
an interaction is affecting the user. Three kinds of information could be col-
lected: the psychological impact on the user; the physiological impact (deter-
mine a subject’s emotional state by interpreting the physiological response to a
stimulus), and the behavioral impact (it contributes to the formalization of the
emotional states). This emotional level is also based on objective and subjective
measurement tools. To collect objective data, different tools could be used, such
as electrocardiogram and face reader tools to translate facial expression to
emotional state (“affectiva tool” by McDuff et al. [31]). The subjective data
could be collected by emotional cards that the user can point out after the
experience. Both objective and subjective emotional data allow understanding
user’s emotional reaction to an interaction.
– And finally, the last level is the semantic one and focuses on the way the user
perceives the artifact. This level could be understood through three kinds of
attributes: the practical attribute (related to the use in terms of pleasure and
effectiveness), the symbolic attribute (about aspect of esteem and social per-
ception), and the aesthetic attribute (focused on aesthetic impression that is felt
about an object through the senses). Objective measurement tools could be
relevant for highlighting specific data such as task success, learnability, effi-
ciency, number of errors, and time per task, thus mainly “practical attribute”
information. But on the subjective side, a self-measurement method aims at
understanding user’s perception of interaction and experience, through a larger
scope.
So finally, these evaluation criteria, with the entire set of tools and measurement
methods that they convey, are only relevant for user experience and interaction if
they are combined, compared, and gathered. Indeed, the only way to accurately
evaluate interaction is to use as much criteria as possible and to combine all the
collected data to highlight specific features.
Furthermore, to be able to compare, analyze, and understand how an interaction
can impact the user experience, we use these previous tools and methodologies at
different levels: First of all, we use these sensory, emotional, and semantic tools to
assess the full user experience. Then, we use the same criteria and tools to assess the
experiential interaction. And finally, we use the same criteria and tools to assess the
three main phases that the user has encountered: the learning phase (interactions to
understand and recognize the product); the setup phase (interactions to adjust the
product and its parameters); and the operate phase (interactions related to the use of
the product).
7.4.4 Tools Related to the Design Activity “Communication”
This activity consists in presenting the result of the design informational cycle to
stakeholders of the design team and/or to prepare material to be used for upcoming
cycles. The design team can adapt the type of representation and of design infor-
mation conveyed depending on the audience. The four tools presented below
exemplify different types of tools used by the KD team in order to illustrate and
communicate user experience-related intention internally. The examples given are
all related to projects already published and therefore present in the public domain.
7.4.4.1 Mood-Boxes
In order to communicate with interior atmosphere and experience intentions, the

mood-box too was created. It has been used both for user research and for internal
communication [15].
Mood-boxes are transparent boxes (37 26 6 cm), open on the top and
displaying a composition of inspirational elements such as fabrics or products as
well as metallic and paint samples. This way, they convey design information
through vision and touch sensory channels. For the creation of the mood-boxes,
material was gathered from material libraries, as well as from shops (furniture,
fashion, fabric, art and design). This type of representation permitted me to convey
a very specific atmosphere (combining concrete and abstract design information) in
a small tangible space.
An example of mood-box is displayed together with additional inspirational
objects and materials in Fig. 7.11. This corresponds to one of the three directions
(i.e., the light and organic direction) resulting from a research in Europe about
future hybrid vehicle interiors [15].
7.4.4.2 Narrative Videos
For interaction and service design projects, other tools are necessary in order to
communicate user experience intentions. Narrative video is one of them. They
Fig. 7.11 Example of

mood-box and inspirational
objects and materials
Fig. 7.12 Snapshots from the communication video of the “Window to the World” project (2011)
present situations or complete user journeys in an immersive way and highlight the
added value for the user without necessary to have to build a fully functional
interactive prototype or deploy a service. This enables time and cost reduction in
order to achieve a proof of concept from a user experience point of view.
During the past years, narrative videos have been used multiple times by the KD
team. They either were produced internally or were outputs from collaborations.
This is the case of the “Window to the World” project that resulted from a col-
laboration with CIID and tackled the interaction between car occupants and their
environment in a poetic and seamless way. Snapshots of the video are presented in
Fig. 7.12, and the entire shot is available online (http://bit.ly/15sb6A3).
7.4.4.3 Storyboard Scenarios
Storyboard scenario was already introduced for generation activities as a tool

commonly used by the KD team. For communication activities, the intention is the
same: communication user experience intentions using series of drawings. The
execution is nevertheless different. Whereas in a generative context the quality of
the drawings is not primary, its importance grows significantly when the purpose is
to communicate to a wider audience than the design team only.
Even if the necessary time to create a communication scenario is higher than that
in a generation context, this tool remains very time- and cost-efficient in order to
communicate a user experience intention in a narrative way (e.g., in comparison
with narrative videos). Figure 7.13 displays an example of storyboard scenario used
as communication tool for the “Window to the World” project.
Fig. 7.13 Example of storyboard scenario from the “Window to the World” project (2011)
7.4.4.4 Prototypes
Prototyping is a common communication practice in many situations. Building on

low-fidelity prototypes from the generation phase, final prototypes can be created in
order to have a wider audience (management, internal stakeholders, potential users),
testing the experience in “real” conditions. The nature of such user experience
prototypes is very diverse. They can take the form of vehicle functional parts or
elements, color and trim proposals, HMI and interface simulation, as well as mobile
application.
7.5 Conclusion
As it was pointed out,our approach to design changed from designer-centric

towards user-centric, in order to fit with quick changing situations, now expected by
every user [13]. What we presented is the vision and new way of thinking of what
we call the Kansei design approach at TME. It enables us to overcome these new
challenges related to fast evolving customer expectations in term of user
experience. As it was explained this freshness and adaptability is made possi-
ble by the use of tools often resulting from a close collaboration between academic
and industrial worlds. We also showed how it appears crucial to build on the wide
field of research related to Kansei in order to create optimized methodologies and
tools to link emotional feelings with objective and subjective measurements. When
writing this chapter, our aim was not to define these fields, or even describe the
entire scope of researches related to Kansei, but to share some of our studies,
introducing thereby the industrial context of our researches. We explained, through
some examples of what we achieved so far, how we are highly convinced that
emotional research, always linked to Kansei philosophy, is from our point of view
the most efficient way to succeed.
If we consider that the research conducted during the past decade at TME was
experimental, it is now time to consolidate some of these building blocks and
generate all necessary conditions to create a sustainable activity that might be
deployed even more broadly.
References
1. Altringer (2013) A New Model for Innovation in Big Companies-Harvard business review
2. Bales, R. F. (2001). “Social Interaction Systems: Theory and Measurement”, Transaction
Publishers.
3. Bongard (2013). (Kerstin Bongard-Blanchy) Bringing the user experience to early product
conception: From idea generation to idea evaluation.
4. Bouchard, C., & Aoussat, A. (2003). Design process perceived as an information process to
enhance the introduction of new tools, International Journal of Vehicle Design, 31(2),
162-175.
5. Christoforidou, D., Motte, D. (2009). Responses to Product Features: an Affective
Neuroscience Perspective. In Proceedings of the International Conference on Research into
Design, Bangalore, India.
6. Chung, H., & Gerber, E. (2010). Emotional-Storyboarding : A Participatory Method for
Emotional Designing for Children. In K. Sato, J. Gregory, and T. MacTavish (Eds.),
Proceedings of Design and Emotion conference. Chicago, MI: IIT.
7. Colombo, S. (2012) A model of affective, cognitive and hedonic response to products.
In J. Brassett, P. Hekkert, G. Ludden, M. Malpass, J. McDonnell (Eds.), Proceedings of
Design and Emotion Conference. London: UAL.
8. Crilly, N., Moultrie, J., & Clarkson, P. J. (2004). Seeing things: consumer response to the
visual domain in product design. Design Studies , 25, 547–577.
9. Desmet, P.M.A. (2002). Designing emotions (Doctoral dissertation), Delft University of
Technology, Delft, The Netherlands.
10. Desmet, P.M.A., & Hekkert, P. (2007). Framework of product experience. International
Journal of Design, 1(1), 57–66.
11. Eckert, C., & Stacey, M. (2000). Sources of inspiration: a language of design. Design Studies,
21, 523–538.
12. Forlizzi, J. and Battarbee, K., 2004, Understanding experience in interactive systems. In
Proceedings of the 2004 conference on Designing Interactive Systems (DIS 04): processes,
practices, methods, and techniques (New York: ACM), p. 261
13. Fukuda, S., (2013), “Emotional Engineering” vol 2, Spinger, 2013
14. Fulton Suri, J. (2003). The experience evolution: developments in design practice, The Design
Journal, 6(2), 39–48.
15. Gentner, A. (2010). How can semiology and emotional studies support a design process? A
multi sensory study based on emotional design and semiotics and the specific case of shapes
and colors. Arts & Métiers ParisTech, Paris, France.
16. Gentner, A. (2014). Definition and representation of user experience intentions in the early
phase of the industrial design process: a focus on the kansei process (Doctoral dissertation),
Arts&Métiers ParisTech, Paris, France.
17. Gentner, A., Bouchard C., Badoil, A., & Favart, C. (2014). Kansei cards: A visual tool
supporting the investigation, discussion, and representation of the kansei-related intentions of
a product to be designed, In Proceedings of KEER conference.
18. Gentner, A., Bouchard, C., Favart, C. (2013). Investigating user experience as a composition
of components and influencing factors. In Proceedings of Int. Association of Societies of
Design Research conference.
19. Gil, S. (2009). Comment mesurer les émotions en laboratoire ? Revue électronique de
Psychologie Sociale, 4. 15–24.
20. Graff, D., Koria, M., & Karjalainen, T.M. (2011). Theoretical model of cross-functional team
effectiveness. In T.M. Karjalainen, M. Koria, M. Salimäki (Eds.), IDBM papers vol. 1,
Helsinki, Finland: IDBM Program, Aalto University.
21. Hassenzahl & Tractinsky 2006: User experience - a research agenda, Behaviour & Information
Technology, 25:2, 91–97
22. Hommel, B., & Prinz, W. (Eds.) (1997). Theoretical issues in stimulus response compatibility.
Amsterdam: North-Holland.
23. Kalakota, R., Robinson M. (2004), Services Blueprint: Roadmap for Execution,
Addison-Wesley, Boston.
24. Larssen, A. T., Robertson, T., & Edwards, J. (2007). Experiential Bodily Knowing as a Design
(Sens)-ability in Interaction Design. In L. Feijs, S. Kyffin, B. Young (Eds.), Proceedings of
Design and semantics of form and movement conference.
25. Lee, S., Harada, A., & Stappers, P.J. (2002). Pleasure with products: Design based on Kansei.
In W.S. Green, and P.W. Jordan (Eds.), Pleasure with products: Beyond the usability (Chapter
16). London, UK: Taylor & Francis.
26. Levy Pierre, (2013) Beyond Kansei engineering: The emancipation of Kansei Design.
International Conference on Kansei Engineering and Emotion Research. Paris, France : LCPI
A&M Paristech
27. Lévy, P., Lee, S., & Yamanaka, T. (2007). On kansei and kansei design: A description of
Japanese design approach. Proceedings of the 2nd Congress of International Association of
Societies of Design Research (IASDR07), The Hong Kong Polytechnic Institute-School of
Design, Hong Kong.
28. Mahut, T., Bouchard, C., Omhover, J.F., Favart, C., & Esquivel D. (2015). Interaction, the
core of the user experience, Proceedings of Desform conference.
29. Mandler, G. (1982). The structure of Value; Accounting for Taste, in T. Tighe & al (Eds.).
Affect and Cognition: The 17th Annual Carnegie Symposium (pp. 3-36), Hillsdale, NJ
(Lawrence Erlbaum Associate).
30. McDonagh D., & Denton H. (2005). Exploring the degree to which individual students share a
common perception of specific trend boards: Observations relating to teaching, learning and
team-based design. Design Studies, 26, 35–53.
31. McDuff, D., El Kaliouby, R., Senechal, T., Amr, M., Cohn, J., & Picard, R. (2013).
Affectiva MIT Facial Expression Dataset (AM-FED): Naturalistic and Spontaneous Facial
Expressions Collected “In the Wild”, Proceedings of IEEE Conference on Computer Vision
and Pattern Recognition.
32. Mehrabian, A. (1980). “Basic Dimensions for a General Psychological Theory of Emotion”,
Wadsworth Publishing Company.
33. Norman, D. 2004. Emotional Design: Why we love (or hate) everyday things. New York, NY:
Basic Books.
34. Ortíz Nicólas, J.C., & Aurisicchio, M. (2011). A scenario of user experience. In S. J. Culley,
B. J. Hicks, T. C. McAloone, T. J. Howard, and P. Badke-Schaub, Proceedings of The 18th
International Conference on Engineering Design (ICED 11), Impacting through Engineering
Design: Vol. 7. Human Behaviour in Design (pp. 182–193).
35. Proctor, R. W., & Van Zandt, T. (1994). Human factors in simple and complex systems.
Boston: Allyn & Bacon.
36. Russell, J.A. (1980). A circumplex model of affect. Journal of Personality and Social
Psychology, 39, 1168–1178.
37. Sanders, E. (2005). Information, Inspiration and Co-creation. In Proceedings of the 6th
International Conference of the European Academy of Design. University of the Arts, Bremen,
Germany.
38. Sanders, E. (2006). Design research in 2006. Design Research Quarterly, 1(1), 1–8.
39. Sanders, E., & Stappers, P.J. (2008). Co-creation and the new landscapes of design, CoDesign:
International Journal of CoCreation in Design and the Arts, 4(1), 5–18.
40. Sheller, M. (2005). Automotive Emotions. Automobilities, 221.

41. Schön, D.A., & Wiggins, G. (1992). Kinds of seeing and their functions in designing, Design
Studies, 13(2), 135–156.
42. Schütte, S. (2005). Engineering Emotional Values in Product Design - Kansei Engineering in
Development (Doctoral dissertation), Linköping University, Linköping, Sweden.
43. Thompson D (1992) On growth and form. In: Bonner JT (ed) Cambridge University Press,
Cambridge, UK
44. Van Gorp, T. (2008). Design for Emotion and Flow. Retrieved 8th August, 2008, from http://
www.boxesandarrows.com/view/design-for-emotion.
Chapter 8
Five Senses Theater: A Multisensory
Display for the Bodily Ultra-Reality
Yasushi Ikei, Koichi Hirota, Tomohiro Amemiya

and Michiteru Kitazaki
Abstract The present paper describes a multisensory virtual reality (VR) system
built for the exploration of the bodily ultra-reality. First, we introduce a new term
‘ultra-reality’ which was advocated by a government committee to explore future
media technology in Japan. The ultra-reality has been recognized from both the
aspects of the super-reality and the meta-reality. The author proposed a hypothesis
that the ultra-reality has its base on the VR (the second reality), and it is embodied
as rational and emotional entity built on pieces of rendered reality. The ultra-reality
is the third reality grasped at the cognitive level rather than perception level. The
author placed the forth reality hypothesis where the self-body is virtualized by
projecting information backward from the body input to the brain. The dynamic
virtual body illusion is the objective of the research to create in which the self-body
makes a part of VR media to duplicate experience of other person. We consider that
multisensory presentation plays a principal role for this bodily ultra-reality. To
provide a framework for realization of the ultra-reality, we introduced exhibit
prototypes of the multisensory presentation system. The five senses theater system,
the FiveStar, provides multisensory stimulations to the user for the creation of
ultra-realistic experiences. The contents of the exhibits were the mixed reality
bodily experience, the interaction with CG creature, and the virtual first-person trip
Y. Ikei (&)
Tokyo Metropolitan University, 6-6, Asahigaoka, Hino, Tokyo 1910065, Japan
e-mail: ikei@computer.org
K. Hirota
The University of Electro-Communications, 1-5-1 Chofugaoka, Chōfu 182-8585, Tokyo,
Japan
e-mail: hirota@vogue.is.uec.ac.jp
T. Amemiya
NTT Communication Science Laboratories, NTT Corporation, Tokyo, Japan
e-mail: amemiya.tomohiro@lab.ntt.co.jp
M. Kitazaki
Toyohashi University of Technology, 1-1 Hibarigaoka, Tempaku-cho, Toyohashi 441-8580,
Japan
e-mail: mich@tut.jp

DOI 10.1007/978-3-319-29433-9_8
146 Y. Ikei et al.
to the tourist sites. The multisensory display devices that were used to create the
bodily ultra-reality were described to show the method and characteristics of this
hypothesis.
8.1 Introduction
Recently, 3D technologies have become more popular in the market providing a

spatial and natural interaction scheme to the digital media than a decade ago.
Stereoscopic TV/movie content is constantly provided as advanced presentation
media. Spatial input device such as Microsoft KINECT or Leep Motion device
introduced a new interface with the digital world. Motion input has already been
used ordinarily in the convenient function of smartphone/tablet devices. However,
these media have not been integrated to create a multisensory whole-body expe-
rience of a digital world with multiple modalities allowing us to work and live
within the mediated another reality.
In the real world, we feel the sense of presence of a space though we are not
always conscious that the world is actually present in front of us, since the existence
of the space and objects is felt absolutely apparent. The conviction of existence is
established by information that comes from multiple sources [5] of senses in
real-time about the environment. Thus, the presence we recognize in the real
environment is not created only with visual information or body motion but with all
the modalities through which we interact [19] with the real world. The experience
of a virtual space should involve information from multiple modalities, synthesized
by a stimulus integration system. However, such multisensory integrated presen-
tation has not been developed sufficiently where we could truly feel ourselves
immersed in another space. There are technical challenges for this realistic pre-
sentation not only in the device quality of each sensory display but also in inte-
gration scheme where the mechanism of human cognition of reality is projected
inherently. The latter aspect in creating the reality (presence) has also been dis-
cussed for about two decades from the advent of virtual reality technology in 1989.
In 2005, a new term of ‘ultra-reality’ was introduced by a government committee
in Japan, and then a forum1 was organized on the theme. The aims of the forum
involved:
• Super-high-definition and stereo images,
• High-realistic sound space generation, and
• Multisensory (five senses) communication.
These are the areas that would cover the ultra-reality research, though they may
not necessarily give sufficient description of the notion. Since then, a new definition
or hypothesis for this synthesized reality has been explored by several projects
1
The Ultra-Realistic Communications Forum.
8 Five Senses Theater: A Multisensory Display … 147
administrated by National Institute of Information and Communications

Technology (NICT) in Japan under the name of ‘Research and development of
ultra-realistic communication technology with an innovative 3D image technique.’
The authors started a research as one of those projects related to creation of sense of
presence in the direction of the ultra-reality. In the research, we have formed a
working hypothesis of the ultra-reality as compared to the virtual reality.
8.2 Presence and Ultra-Reality
8.2.1 Sense of Presence
The ultra-reality is basically recognized as a kind of reality or what introduces the

sense of presence that is created artificially by devices that produce information to
human senses. Thus it includes basic natures of virtual reality such that it makes one
feel like being there by way of high-resolution pixels, a wide color range, complete
coverage of sense organs, a 3D interactive space, self-projection, an autonomous
response of the world, and so on, while being here in the actual space. All of or part
of these features would be required for the creation of sense of presence.
Since psychological mechanism of the sense of presence has not been elucidated
fully to provide an established basis on which the technology could build a system
of an effective presence generator. Andoh [1] advocated three factors that constitute
the sense of presence.
1. Spatial factor: sensations of stereognostic, texture, and immersiveness.
2. Temporal factor: sensations of motion and synchrony.
3. Body factor: sensations of self-existence, agency, interactivity, and emotion.
In addition, the sources of sensation of presence are both external (perceptual) and
internal (memory). The external sources are sensory information from the physical world,
while the internal sources are the human memory built from experience or learning.
Since the sensation of presence is essentially subjective, it may not necessarily
be easy to measure precisely what people perceived. Questionnaire [26] is a prin-
cipal method to represent processes occurred in the mind. In a sense, it is the only
method to look into what was really going on in the psychological space of the
person. On the other hand, other measures were proposed differently by researchers.
Reflexive responses [11] obtained by measurement of the behavior of a participant
to critical/emotional events shown by a virtual reality system would reveal the
extent to which the participant was involved in the virtual environment. The dis-
crimination accuracy [19] between the real and virtual environments may show the
state that one perceives the environment. Perceptual constancy in a virtual space
[20] would provide the information on low-level perceptual processing activity to
capture the environment. Externalization of perceived stimulus [14] is also fun-
damental processing that projects the mechanism for acquiring the world image.
148 Y. Ikei et al.
Attention change count [15] from virtual to real is, in a sense, a direct measure of
the extent to which the participant is immersed in the artificial environment. The
values for these criteria indicate the amount to which participants believe the
stimulus of the system is of natural environment, not composed but another world is
really there.
8.2.2 A Working Hypothesis of Ultra-reality
It would be possible to place the ultra-reality at the ultimate of the perceptual reality
which gives the sense of presence in terms of factors or measures written above.
However, the ultra-reality would also accept another interpretation. Two terms
relating to ultra-reality, super-reality and meta-reality, were presented in [8]. The
super-reality is the extreme of the direction of current technology development. It
pursues more resolution, more colors, and more field of view to enhance the reality.
It is super-real in the sense of perception level of capturing the world. On the other
hand, the meta-reality is thought as a cognitive level reality, not necessarily real in
all the direct senses of perceptual reality, but it may cause more subjectively real
(natural) understanding and/or emotional response of the user. It depends more on
the content and multisensory structure which accepts content-oriented rendering
than fidelity (similarity)-oriented rendering.
The hypothesis of the ultra-reality in contrast to the virtual reality might be
proposed as follows. Figure 8.1 illustrates one interpretation of the ultra-reality by
the first author of the present article. This indicates that the feature of the
Fig. 8.1 Virtual reality and ultra-reality

ultra-reality is described by two axes of subjective property established on the basis

of perceptual reality rendered by the virtual reality technology. The axes are of the
rational understanding and the emotion of the user about the reality regarding the
presented space. The ultra-reality forms a cone originating from the last decade of
twentieth century with an increasing fan shape that develops with the x-axis which
indicates the amount of virtualization of the reality over the time.
The x-axis indicates the extent of virtuality of the artificial environment. The left
half-space of x-axis (x < 0) is the natural environment without electronic aug-
mentation or electronic substitution of real environment. It is under conventional
artificial development of real environment chiefly by a mechanical approach. The
right half-space (0 < x) is the virtualized (electronically augmented) environment.
The virtuality of the space is increased to the right direction. In other words, this x-
axis shows the status of artificial rendering in a spatiotemporal (4D) conventional
space.
The other two axes are introduced for the core part of the ultra-reality. These
axes provide the rendering policy of the space for cognitive meaningful interaction
with, and subjective feeling of the artificial space. The space is rendered to provide
cognitive and emotional relation to the meaningful world for the users. The z-axis of
Fig. 8.1 that comes over to us indicates the rationality of the space, in other words,
the explanatory message of the environment. The more the environment is virtu-
alized (to x-direction), the more the complicated idea can be expressed with an
artificial device and its explanation provided in the space. The rationality indicates
the cognition of subjective nature of the environment other than perceptual nature
of it. On the other hand, the y-axis shows the emotional aspect of the environment.
The extent to which subjective emotion is induced or to which the value judgment
relating to emotional preference is caused is regulated by artificial rendering of the
world content. The upward direction indicates the extent of pleasant impression
evoked by the created environment.
The ultra-reality is a state in a cone on the right half-space and depicted at a
specific level of virtuality by a bundle of arrows or a 2D fan shape (or a disk) that
spans on a rationality and emotion plane. That is a cognitive/emotional rendering
based on a virtual reality 4D space rendering. Virtual reality rendering is basically
directed to perceptually correct reproduction of a spatiotemporal space. While on
the other hand, the ultra-reality rendering tries to deliver comprehensible messages
and also emotional experience based on virtual reality rendering. In this rendering,
specifically for the later, we believe that multisensory presentation is essentially
important in the presentation directed to bodily sensation (haptic interaction) which
provides one of the bases of emotion. In addition, the multisensory rendering is
rather intended to integrated effectiveness than to accuracy in each modality.
The reality we handle might be named for conveniently classified understanding
of interrelation. The reality appears in the x-direction, established by the virtual
reality technology is considered to be the second reality as compared to the original
natural (first) reality. The ultra-reality rendered based on the virtual (second) reality
might be the third reality. We would like to introduce the forth reality in the next
section to discuss bodily consciousness as part of reality.
150 Y. Ikei et al.
8.3 The Forth Reality Hypothesis
8.3.1 Back-Projection
Involvement of a self-body to the virtual space is a prerequisite factor as described

in [24]. This advocated that essential characteristics of the virtual reality are the
real-time interaction, the 3D space, and the self-projection. The last factor,
self-projection, is not very obvious issue, and it has not been sufficiently imple-
mented and elucidated since it has complicated cognitive and sensory aspect that is
currently investigated in light of neuroscience.
Usually the self-projection means that projected self-avatar is observable by both
the current user of a VR space and other persons in the same VR space. It is an
objective projection as it is apparently examined by anyone in the space—the first
level of self-projection. On the other hand in light of a subjective view point—the
second level of self-projection, the self-projection is considered that the user feels as
if his/her body is reside in the avatar, or in a weaker sense, his/her body coexist both
at the avatar and the actual (real) self-body (virtual coexistence).
In this subjective projection, the cognitive (conceptual) self-body is virtualized
and placed at the avatar [2] although the real body still exists in the real world and
holds the control of the avatar. In this situation, it would be very interesting to think
whether the avatar could control the cognitive self-body. That is, the avatar that has
a scenario of an experience drives the cognitive self-body to provide a copy of the
experience to the user. In reality, the scenario of the avatar is of special interest if it
is of someone else who had a variety of valuable experience, e.g., world travels,
physical presentations of top athletes, artistic skills of a living national treasure
artist, or even a ride in a theme park and learning experience at the university.
The projection is from the avatar to the cognitive self-body, which is a
back-projection (Fig. 8.2) to usual self-projection. The external experience (of one
person) is projected to the user through the self-body. In this case, the self-body
itself is a medium to present the other person’s body. It may be written that the user
dives into the other person’s body. We call this the virtual body technology by
back-projection in which the user relives the experience of the other person. The
virtual body is a rendered cognitive self-body by the back-projection interface.
8.3.2 Dynamic Virtual Body
Cognition of a self-body and its rendering have been investigated in the context of a
virtual body in a VR space [22, 23]. A multisensory presentation is a crucial part of
implementation by which a virtual body might be really considered as a self-body
as subjective projection. A high-quality sense of presence is expected to be realized
if we can properly integrate multisensory stimuli that produce the sensation of a
virtual self-body as if it is resided in the VR space, not in the actual state of the
Fig. 8.2 Back-projection in

which a virtual body is
rendered
physical body. There are many evidences that a passive multisensory perception
can easily create virtual body illusion [3] even if it was a whole body [2].
We discuss here the method to create a dynamic virtual body illusion where the
body is recognized in a different motion state from the real state of an actual body.
For example, a virtual body of yourself is walking while the actual your body is
sitting. This could also be said that your actual body is the medium that renders a
virtual body in a different state in conjunction with the VR rendering apparatus. As
a part of VR rendering system, the actual body displays a virtual body into the
user’s brain. In the course of this design implementation, the actual body is
physically moved appropriately by some haptic devices2 to evoke a virtual body
sensation that creates a virtual experience. This is the goal of our multisensory
display system introduced in the present paper. We consider this kind of
ultra-reality that virtualizes the body may be called the forth reality.
To utilize the actual body to represent virtual body motion in a virtual world,
multisensory input to the actual body should be generated and integrated inge-
niously. Among them, vestibular and haptic sensations are specifically crucial for
the experience of spatial body motion and presence of the body in a virtual space.
A vestibular device needs to create a first-person sensation of own virtual body
movement. The vestibular device is not the one popularly used for a vehicle sim-
ulator but the special one for representing the self-body.
The present study proposes a multisensory information display (the Five Senses
Theater, [12]) that evokes the sensation of the virtual body motion. Based on the
display integrating multisensory presentation, we investigate a new methodology to
create first-person reliving sensation of a bodily experience that was performed and
2
Devices similar to an exoskeleton mechanism that assists user’s movement.
152 Y. Ikei et al.
obtained as other person’s activity in the past. For the first stage of this dynamic
virtual body representation, we focused on representing a walking motion of a
person as a kind of playback of prerecorded body motion. If we could achieve this,
it is expected to lead to relive other person’s voluntary experience. This opens a
possibility to obtain experiences of other people, for example, visits to various sites
by other people. It would also provide the user the opportunity of learning through
the experience of other people, or in a sense, it might be considered as a kind of
memory replication.
8.4 A Display System for Dynamic Virtual Body
8.4.1 Multisensory VR System Prototype
A display system that produces clues to senses of multiple modalities [12] was
developed where the ultra-reality described above was partially implemented. The
display system, FiveStar (Five Senses Theater), presents stimuli to the five senses
except for the gustatory sensation. The FiveStar is an interactive personal display
system that consists of a large 3D visual display, a 5.1-channel surround audio
system, haptic/tactile display, a wind/scent delivery system, and a vestibular body
motion display subunits.
Fig. 8.3 Multisensory display (FiveStar)

Figure 8.3 shows a schematic of the FiveStar. These subsystems are integrated to
provide a base for an interaction to the user following a particular scenario. More
specifically, the objective of the system is to provide the platform to build a core
technique for a virtual body used in a multisensory communication and a package
content experience. For the creation of the ultra-reality on the system, we focused on
rather cognitive characteristics than the fidelity of each device that has often technical
limitation relative to the quality of human senses. The system pursues the effective
rendering of realistic experience in which the rationality and emotion of activity are the
points to evaluate; it is not necessarily of high fidelity-oriented presence rendering in
the sense of the second reality. The integration over modalities compensates with each
other, substitutes and augments the total presentation.
8.4.2 Rendering Levels
We assumed a multilevel structure for the FiveStar rendering design. Figure 8.4 shows
the conceptual three levels: (1) scene level, (2) sensation level, and (3) physical level.
Actual interactions are exchanged at the physical level between the sensors and motors
of both sides of the user and the FiveStar. The information of this level is described in
physical quantity. In this bottom level, information exchanged has no direct meaning
of the higher level in itself. The capability of the physical device for this level is
limited specifically in haptic and vestibular devices. The second, sensation level
determines the range and resolution of the physical level to effectively produce the
interaction performed by the user. The information of the second level is described by
sensory response and motion input. This level interaction is designed to implement the
top level functions. The top level, the scene level, handles the context to be performed
by the user and the FiveStar. The scene is interpreted and designed in terms of both
cognitive and emotional aspects. We assume that the scene is rendered to express or
transfer intention of the producer and the user. The modality presentation in the second
level needs to be integrated in this direction.
Fig. 8.4 Rendering levels of the FiveStar

154 Y. Ikei et al.
8.5 Prototype Contents of the Multisensory Emotional

Rendering
The development of an ultra-realistic space prototype has started in 2009 to

investigate the method and inherent mechanism to create the space. As stated
above, the ultra-reality is considered to emerge essentially in the aspects of
high-level cognition of rationality and accompanying emotion. We investigated the
methods in trial implementations of specific scenes that involved the aspects. Four
test-bed spaces were produced. Since basic displays for modalities have been
developed concurrently with building the specific scene, functions of displays were
slightly different with the builds.
8.5.1 Mixed Reality Bodily Experience (Magic School)
The first prototype of the FiveStar was demonstrated in 2009 for an early stage
evaluation of its element. The exhibit was a mixed reality experience featuring a
magic casting. The participant sat on the seat and casted a magic with a real magic
wand device to him/herself image in the mirror rendered stereoscopically on the
screen (Figure 8.5). The magic in the form of colorful particles flies to the
self-image on the mirror. The magic particle that flows from the tip of the real wand
Fig. 8.5 Mixed reality experience that involved 3D visual/audio magic particles, haptic feedback
(at hands, feet), and wind/smell sensation was presented. a 3D projection screen with the
participant inside the virtual mirror. b A participant swings a haptic feedback wand. Seat vibration,
feet motion, and wind/smell response of magic exertion were presented
is presented stereoscopically in the real space. After the magic reached to the virtual
image of the participant, the system produced a multiple-modality event to the
participant as magical effects created and controlled by the participant. This content
for the Asiagraph 2009 exhibit was introduced as a wizardry training at a virtual
magic school.
In this implementation, a haptic (force and tactile) feedback at the hand was
provided by the real magic wand in response to the input motion of the participant
who controlled the particles of magic shed at the tip of the wand. The response
events of the world were given also to the body surface by tactile stimulators on the
seat, to the feet by a motion board under the shoes that provides stimuli to
somesthesia originated from the event in the VR world behind the mirror image that
was concurrently projected to the real space. A wind and smell presentation was
synchronized with the magic particles of four kinds and delivered to around the face
of the participant. The smell presentation was the most interested modality except
for the visual presentation according to the questionnaire response obtained from
the participants.
The reality treated in this prototype was unique in several points. The space
presented was an interactive 3D space that was both virtual and real at the same
time. The VR space in the virtual mirror was built based on captured image of the
participant and it was augmented with visual rendering of magic effects. The events
in the VR space (behind the mirror) such as collisions of particles to the virtual
mirror and to the image of the participant were projected to the real space with
haptic, somesthetic, and wind/scent stimulations. Although the perceptual infor-
mation sources were divided within virtual and real space, they are recognized as an
integrated single event to form a new kind of reality sensation. In this setting, the
participant was conscious of two bodies of an image in the VR space and the real
self-body. Since the directions of the bodies are opposite, the two body images are
felt rather separated.
8.5.2 Interaction with CG Creature (Japanese Specter)
The prototype was extended and presented3 (Fig. 8.6) at Asiagraph 2010 in Tokyo.
The content was an experience of a fantasy space where a guest player sitting on the
system was brown from the current place on the Earth to another space where a
Japanese specter (Fig. 8.7a) lived and casted evil magic particles to the player. The
system produced a multiple-modality interaction to the player. The player protects
the particles of magic by holding a shield with an amulet called ‘ofuda.’ After the
In the title of ‘Ultra-ordinary experience through five senses media’.

3
156 Y. Ikei et al.
Fig. 8.6 FiveStar system at Asiagraph 2010. It consists of a 3D visual display with shutter
glasses, a 7.1-channel audio system, an 8-channel smell/wind source, a 3D force-feedback device,
a tactile device, a seat vibrator, 2-dof feet motion device, and a 3-dof motion seat
Fig. 8.7 a A Japanese specter (Yokai) of the fantasy VR world. The player is ‘blown’ into this
space from the exhibit hall with an intense air flow and vestibular stimulus. b A 3D screen and the
player interacting with the VR scene
player received the magic particles sufficiently into the amulet, the player could
throw the amulet to the specter. When the amulet defeated the specter, it changed
the specter to a friendly character. Then, the specter brought the player back to the
original real world where the player was sitting. Thus, the player experiences and
interacts with an extraordinary world with multiple sensations.
Figures 8.6 and 8.7b show the configuration of the system. The seat was driven
by 3D motion base system to give flying sensation (from the Earth to the yokai
world) and the impact event in both physical and emotional senses. A stereoscopic
visual system rendered the world and a 7.1 surround audio added the sound space.
An eight-channel wind scent source system, a compressed air jet system, tactile
stimulators to the body on the seat, and a 2D feet motion generator system with a
3D motion seat were integrated to create the ultra-realistic experience.
In this implementation, the motion seat provided a large body motion to present
a transition between the real space and a fantasy space. The motion indicated a
distance of the scene changes as a first-person experience. This rendering was
essentially a vehicle metaphor for the transition. The content was designed by CG
modeling which has to move in real-time, and thus, its presence quality was
restricted by the performance of the visual rendering hardware. Although visual
reality was limited, the multisensory stimuli provided emotional effects relating to
the context.
8.5.3 Virtual First-Person Trip
This is a direct implementation of the virtual body. A real scene-based virtual trip
prototype was demonstrated at Asiagraph 2011 in Tokyo. The virtual trip sites were
Hakodate in Hokkaido (northern island in Japan) and Asakusa in Tokyo, the both of
them are popular destinations for tourists. The system provided a virtual walking
experience around the sites with a 3D vision/audio, wind and smells, vestibular
sensation, feet motion, and sole stimulation device generating a bodily sensation of
walking while sitting on the motion seat. (see Fig. 8.8) The user viewed, on a
3D-LCD panel, the scene of Asakusa Senso-ji Temple (Fig. 8.9a) and Mt.
Hakodate, Hachiman-zaka slope (Fig. 8.9b) recorded by a 3D video cam.
Fig. 8.8 Virtual trip system consisting of a motion seat, a foot motion device, sole stimulators, a
wind/smell system, 5.1-channel audio system, and a 3D visual display
158 Y. Ikei et al.
Fig. 8.9 Scenes of virtual trip. First-person virtual bodily walking in a Asakusa, b Hachiman-zaka
slope
Walking sensation was created by the devices, while the 3D video scene went
through a flat road or a slope with a soft/hard ground, and up/down stairs. The
sensation of a cyclic walking motion of a body was created by the periodic small
vibration of a seat and a cyclic 2D feet motion with a two-point (heel and toe) sole
stimulation. These motions were synchronized to the sway of video images intro-
duced by walking motion of a person who recorded the scene. The motion of the
display devices was designed and adjusted based on the subjective equality of
sensation observed in the real walk. A trip to Asakusa included experiences of
riding a subway, climbing stairs, walking down the shop street, smelling baking of
Japanese cracker, smelling an incense burner for good luck, feeling splashing water.
Hakodate tour provided sensations of riding a tramway/a ropeway gondola, feeling
sea winds, and smelling flowers at Mr. Hakodate, stepping on soft ground in a park,
etc. Walking sensation on this device is described in Sect. 8.7.
8.6 Display Devices of the System
The system presents a particular scene with modalities integrated to generate a

realistic impression. The integrated devices for this purpose have not fully been
developed so far. A haptic system and a body motion system are those would give
new reality sensation not experienced only with an audio/visual channel currently
used in a conventional display system.
8.6.1 Haptic System for Hand
The haptic system for the user’s hand consists of a string-pulled 3D force-feedback
device (Fig. 8.10a) and a stick with a six-channel tactile device (Fig. 8.10b). The
haptic device presents reaction force and tactile stimulation to the hand and palm,
Fig. 8.10 Haptic system. a String force display for pulling a wand and an amulet, b Magic wand
equipped with six vibrators
related to an event during the interaction. The force-feedback device (Fig. 8.10a)
consists of four motors and strings suspending the magic wand/amulet device.
Three-dimensional force is applied to the tip of the device that has about an
800-mm square workspace. Figure 8.10b shows the magic wand (stick) device with
a tactile grip that has six vibrators for the palm. The position of the device is
measured by string lengths of the force-feedback device, and the orientation is
obtained by a magnetic spatial sensor (Patriot, Polhemus).
In the scenario of the second prototype (FiveStar 2010), the player defended
against the magic particles shed from the specter by a half-transparent shield (in a
virtual space) with an amulet (center of Fig. 8.10a, in both virtual and real spaces)
that is attached to the wand device. The collision of particles at the shield in a
virtual space caused force perturbation with impact forces on the wand as well as
tactile stimuli on the grip.
8.6.2 3D Motion Seat and 2D + 2D Feet Motion Device
The system for somesthetic (bodily) sensation consists of a three degree of freedom
(dof) motion seat with a 2-dof feet motion system (Fig. 8.11) and the foot (sole)
stimulation device (Fig. 8.12a). This system produces vestibular stimulus that
changes the user’s head position in addition to the body posture by the feet motion
160 Y. Ikei et al.
(a) (b)
(c)
Linear actuators
Fig. 8.11 a 3-dof motion seat in use with the force display and feet motion plate, b seat is driven
by three linear electrical actuators, one at the front and two (left, right) at the rear of the seat, c base
linear actuators
Fig. 8.12 Lower limb stimulator. a sole stimulator that installs two (heel and toe) vibrators,
b 2-dof horizontal motion plate (base), two vertical motion pedals, and heel/toe tactile stimulators
on the pedals
relative to the seat. Along with the feet motion, the force impact at the sole of a foot
while real walking was presented by the foot stimulation device.
The 3D motion seat is driven by three electric linear actuators with a 100-mm
displacement and a 200 mm/s max velocity. The possible motions are up-and-down,
pitch, and roll directions. These motions are combined to generate the desired tra-
jectory of the seat. The 2D feet motion board installs two motors that drive the top
plate 85 mm to the x–y-directions in a horizontal plane.
Eight-channel tactile feedback devices were installed under the cushion material
of the surface of the 3D motion seat. The stimulator unit is a full-range speaker that
creates stimulations on the back and the thigh of the participant to enhance the event
and motion in the space. Two vibro-stimulators were attached at the upper arm to
present tactile sensation of a wind or grazing collision while walking among people.
The lower limb stimulator (Fig. 8.12b) consists of a 2D horizontal motion plate,
2D vertical motion pedals, and sole stimulators on the both ends of the pedal. These
devices contribute to create sensation of a spatial bodily motion by making user’s
feet move both in a 2D plane, and by lifting lower legs at the heel simultaneously.
Synchronous motion of them induces the sensation of a voluntary walking motion
of the self-body.
8.7 Rendering of Walking Motion
Walking is a full-body motion controlled by the spinal cord and the brain based on
vestibular sensation, kinesthesis, tactile sensation, and vision with an intention to
walk originated at the cortex [16, 25]. It is a voluntary motion; however, not all of
the motion is under conscious control. The cyclic rhythm of leg motion is mainly
controlled by the CPG (central pattern generator, [7]) in the spinal cord and the
brain stem without direct intervention of subjective volition from the cortex. This
might allow to place a hypothesis that a passive body motion generated by an
external device could be perceived as a part of voluntary walking motion.
The motion of real walking in terms of vestibular sensation is represented as a
main component by the up–down (lift) motion length. The length of the lift was
around 30 mm according to our measurement, although it depends on the body size
of the participant. The trajectory of the heel in a real walk takes the vertical lift
about 200 mm and the horizontal displacement in the forward/backward direction
about 600 mm. We produced this amplitude of a vertical motion by the display
system. The result was that this motion evoked an extremely larger sensation than
actual sensation the subject received during the real walking. This might be ascribed
to the difference in sensory processing in which sensory input could be attenuated
when a voluntary motion was executed [4, 13].
An appropriate amplitude of a vertical motion (with roll and pitch motion) was
experimentally searched by the participant based on the method of adjustment. The
adjustment involved three linear actuators so that the resultant motion had three
degrees of freedom. The vertical motion that the participant adjusted was about 1 to
2 mm at the headrest of the seat that was far smaller than the real walking motion.
The amplitude of seat motion was about one-fifteenth of the amplitude observed
in the real walking. A foot lift motion of a participant was created by two vertical
motion pedals in Fig. 8.12b to evoke a sensation of walking while the participant
sat on the seat with his shinbone set approximately vertical and the angle of the
knee joint held to 90 degrees. Based on the real trajectory of a heel, a similar
trajectory was designed and produced. The result showed that the sensation of heel
lift was most similar to the real walking when the lift length was about 10–30 mm.
This is around one-tenth of the real lift motion of heel during walking.
In both cases of either stimuli, the passive reception of motion could impart a
part of sensation of walking at the magnitude of extremely smaller than the real
ones. This suggests that the virtual body is created by relatively small devices.
162 Y. Ikei et al.
Although other kind of body motion to be realized will require more mechanisms
for passive drive of other parts of a body, this asymmetry of magnitude adds value
to the system design.
8.8 Conclusion
Three prototype systems of FiveStar were developed to investigate the creation

method and the property of an ultra-realistic space. In the first prototype, a mixed
reality space experience was created using the multisensory displays where the VR
space behind the rendered mirror and the real space were merged. We installed
haptic, somesthetic, wind/scent displays in addition to the audiovisual displays. An
interaction between a magic wand and particles presented multisensorily was
implemented to give a sensation of control over a 3D real/VR space. In this sce-
nario, the magic particle went across the real space to the VR space; then after the
image of the participant was hit behind the mirror by the particle, the response
occurred simultaneously in two spaces so that it was viewed as a single-event
experience. In this framework, the participants have understood the space construct
intuitively, although it was not a simple immersion to a single virtual world. The
participant might have monitored it analytically or simply enjoyed the world. The
issues of rationality and emotion of the experience were involved in this framework.
The second prototype dealt with the experience rather at a usual first-person
viewpoint to the artificial world than a mixed reality world of the first prototype.
The participant was looking into the presented world or environment to find the
position of him/herself and interacted with the world that evolved visually in front
with direct (real world) feedback in other modality. This might involve one aspect
in ultra-realistic experience compiled in a package media like a game world. The
factor relating to the ultra-reality was more emotional than rational in the case of
this content.
The third prototype was to relive the trip recorded by other people. The par-
ticipant was forced to follow the experience of other person’s sensation. A factor of
reality is, in a sense, suppressed where the participant can not interact with the
virtual environment by his/her own will. However, as a movie gives a kind of
reality, passivity does not necessarily mean significant degradation of sense of
presence. Instead, the heteronomously driven sensations including somesthesia
generated rather intense sensation that induced subjective experience of the
environment.
In these cases, the contribution of multiple-modality presentation seemed to have
a critical impact at a cognitive level, so that we have preliminarily investigated the
effect of each modality presentation of the exhibits regarding the ultra-reality
factors.
The authors consider that the rendered fantasy world was a good example to
demonstrate how the multiple-modality presentation was able to be integrated
effectively from the ultra-reality creation point of view. We will investigate in a
more quantitative sense the modality interaction in producing integrated sensation

of reality. The relation between two factors (rationality and emotion) and modality
integration would be discussed in terms of the FiveStar performance. The inter-
action among modalities to enhance the whole effect of expression for communi-
cation and authoring a content will be the issues to pursue in this project hereafter.
In the last implementation we introduced, a new display hypothesis that projects
sensation of walking by passive use of the body. After measuring a body trajectory
during a real walk, an equivalent trajectory of devices that projected a virtual
walking was compared with the real walk. The equivalent amplitude of vertical
motion on the display was as small as about one-tenth to one-twentieth of the real
motion. This small value is good for the implementation of the display since its
mechanical structure can be much reduced to allow more places for use.
This hypothesis of a dynamic virtual body rendered by the back-projection
seemed to work according to our observations, and we confirmed that it determined
the stimulus range that enabled the back-projection. Observed large difference
between the real body motion and its equivalence on the display is considered to be
related to the particular brain process of modulating sensory information based on
the efference copy and the presynaptic inhibition [18] in addition to the conscious
intention to walk [9]. The mechanism needs to be elucidated in conjunction with the
field of brain science.
Since the reality is only perceived through the bodily interaction with the world,
so that the body itself limits the reality. If we can assume the self-body as a medium
of VR space media, our experience will be augmented and expanded based on the
VR system, since the body is the base of all the experience. The direct consequence
of this is that your experience can be that of the other person. The user can relive
other’s experience as if it occurred originally on the user. This may be said that the
user’s body is replaced with the other’s [17]. In other words, the user dives into the
other’s body and obtains his experience as depicted in [6, 10, 21]. The dynamic
virtual body provides a conceptual base of this future image.
Acknowledgements The authors wish to thank Professor Michitaka Hirose at the University of
Tokyo for his valuable advice on the present research. We thank Dr. Koji Abe for his great
contribution to the virtual trip project. This research was supported by the MIC/SCOPE
#141203019 and a Grant-in-Aid for Scientific Research (A) from MEXT, and a past funding for
the ultra-realistic communications technologies of NICT in Japan.
References
1. H. Andoh. Factors of the sense of presence, chapter 6.2.2. VRSJ, 2010.

2. O. Blanke. Multisensory brain mechanisms of bodily self-consciousness. Nature reviews.
Neuroscience, 13(8):556–71, aug 2012.
3. M. Botvinick and J. Cohen. Rubber hands ’feel’ touch that eyes see. Nature, 391:756, 1998.
4. K. E. Cullen. Sensory signals during active versus passive movement. Current Opinion in
Neurobiology, 14:698–706, 2004.
5. P. K. Dick. Total Recall. Gollancz, 2012.
164 Y. Ikei et al.
6. M. R. Dimitrijevic, Y. Gerasimenko, and M. M. Pinter. Evidence for a spinal central pattern

generator in humans. Annals Of The New York Academy Of Sciences, 860(1):360–376, 1998.
7. K. Enami, M. Katsumoto, and R. Nishimura. Current techniques for. The Journal of the
Acoustical Society of Japan, 64:322–327, 2008.
8. P. Haggard. Conscious intention and motor cognition. Trends in cognitive sciences, 9(6):
290–5, Jun 2005.
9. N. K. Hayles. How We Became Posthuman: Virtual Bodies in Cybernetics, Literature, and
Informatics, volume 4. 1999.
10. G. H. D. Hecht, M. Reiner. Multimodal virtual environments: Response times, attention, and
presence. Presence: Teleoperators and Virtual Environments, pages 515–523, 2006.
11. R. M. Held and N. I. Durlack. Telepresence. Presence: Teleoperators and Virtual Environ-
ments, 1:109–112, 2015.
12. Y. Ikei, K. Abe, K. Hirota, and T. Amemiya. A multisensory vr system exploring the ultra-
reality. In Proc. VSMM2012, pages 71–78, 2012.
13. F. M. Lambert, D. Combes, J. Simmers, and H. Straka. Gaze Stabilization by Efference Copy
Signaling without Sensory Feedback during Vertebrate Locomotion. Current Biology, pages
1–10, 2012.
14. J. M. Loomis. Distal attribution and presence. Presence: Teleoperators and virtual environ-
ments, 1, pp., 1:113–119, 1992.
15. J. Perry. Gait Analysis: Normal and Pathological Function, volume 12. 1992.
16. V. I. Petkova and H. H. Ehrsson. If I Were You: Perceptual Illusion of Body Swapping. PLoS
ONE, 3:9, 2008.
17. S. Rossignol, R. Dubuc, and J.-P. Gossard. Dynamic sensorimotor interactions in locomotion.
Physiol Rev, 86(Physiol Rev 86: 89154, 2006;):89–154, 2006.
18. T. B. Sheridan. Musings on telepresence and virtual presence. Presence: Teleoperators and
virtual environments, (1):120–126, 1992.
19. S. Shimojo. pages 21–29. 2015.
20. M. Shirow. Ghost in the Shell. Dark Horse Comics, 1996.
21. A. S. M. Slater. A virtual presence counter. Presence: Teleoperators and virtual environments,
9:413–434, 2000.
22. M. Slater, E. D, S. H, and V. M. Inducing illusory ownership of a virtual body. Frontiers in
neuroscience, 3(2):214–20, 2009.
23. M. Slater, J. McCarthy, and F. Maringelli. The influence of body movement on subjective
presence in virtual environments. Human Factors: The Journal of the Human Factors and
Ergonomics Society, 1998.
24. S. Tachi, M. Sato, and M. Hirose. Virtual Reality. Corona Publishing Co.,LTD., 2010.
25. M. W. Whittle. An Introduction to Gait Analysis. 2007.
26. B. G. Witmer and M. J. Singer. Measuring presence in virtual environments: A presence
questionnaires. Presence: Teleoperators and Virtual Environments, 7(3):225–240, 1998.
Chapter 9
Making Emotion and Decision
via Affecting Self-Perception
Sho Sakurai, Takuji Narumi, Tomohiro Tanikawa

and Michitaka Hirose
Abstract We propose a model for generating high-level cognitions, especially

emotions and decisions. The development of virtual reality (VR) technologies has
made it possible to re-create multi-sensory experiences with high accuracy and
resolution. Many studies have tried to reproduce not only the environment but also
the psychological experience, such as emotion and decision with VR technology.
However, VR has not yet been realized to generate the absolute subjective expe-
rience because the influence of the environment on the human mind in the human
information process has barely been discussed. On the other hand, in psychology or
cognitive science, many studies have shown that the psychological state and
self-body image interact mutually. The “body image,” which is the representation
of one’s body recognized in the brain, is formed based on the perception of
self-body (self-perception) through multi-sensory integration received from the
environment and one’s body. Given this, appropriately providing artificial sensory
stimuli seems capable of generating emotions and decisions in light of the influence
of the environment on self-perception. Based on these findings, we propose a
methodology “self-perception-based VR” for making emotions and decisions by
affecting self-perception. The current paper discusses and offers a guide for
“self-perception based VR.”
S. Sakurai (&) T. Narumi (&) T. Tanikawa (&) M. Hirose (&)

The University of Tokyo, 7-3-1 Hongo, Bunkyo-Ku, Tokyo, Japan
e-mail: sho@cyber.t.u-tokyo.ac.jp
T. Narumi
e-mail: narumi@cyber.t.u-tokyo.ac.jp
T. Tanikawa
e-mail: tani@cyber.t.u-tokyo.ac.jp
M. Hirose
e-mail: hirose@cyber.t.u-tokyo.ac.jp
S. Sakurai
Tokyo Metropolitan University, 6-1-1 Asahigaoka, Hino, Tokyo, Japan

DOI 10.1007/978-3-319-29433-9_9
166 S. Sakurai et al.
9.1 Introduction
The VR has developed technologies to duplicate physical sensory stimuli. These

technologies have enabled the creation of various virtual experiences equivalent to
the real ones or those never experienced in the actual world. However, VR has not
been sufficiently developed to reproduce a perfect realistic world indistinguishable
from the actual one.
Conventional VR has tried to (re)produce the mind by the (re)production of an
environment with physical stimuli resembling actual physical characteristics,
through improving quantity of stimuli, display resolution, accuracy. However, the
same information is always interpreted differently depending on the individual’s
experience or memory.
We will illustrate the difference with a painting as an example. Someone who
felt that a painting is beautiful would be able to immerse in the painting and gain
happiness or satisfaction from it. In contrast, someone who felt uncomfortable with
the painting would withdraw his/her eyes from the painting and refuse to ever look
at it. Even if existing of the paintings is a truth, the reality about the painting is
“beautiful painting” for former individual, whereas “uncomfortable painting” for
latter individual.
Conventional VR has left the interpretation of meaning of artificial sensory
information to one’s recognition. However, generating a particular psychological
experience is difficult due to the limited debate on the influence of the environment
on human emotions or decisions.
Meanwhile, an abstract representation, such as comic or animation, enables us to
be immersed the story and to empathize with fictional characters. Such represen-
tations do not always reproduce actual physical characteristics.
Given these cases, a different approach from conventional VR is considered
essential to achieve the generation of absolute reality: VR’s final goal. To achieve
the goal, a new approach is required to reconsider the mechanism of the higher
cognitive ability in human information process.
Meanwhile, the long-term studies have discussed the mind/conscious generation.
Such researches have shown that the recognition of one’s mind state mutually
interacts with one’s recognition of his/her body through multi-sensory integration.
Self-body is recognized with a core of the perception of the one’s body
(self-perception). Human can separate themselves from their self-body and the
environment from within much information. The sensation received from the
self-body and environment is integrated and separated in the process of perception.
Given these findings, the recognition of body image seems variable if the sen-
sory stimuli provided from outside can be integrated with the sensations received
from one’s body in the perceptual process. Additionally, appropriate change of
body image is considered necessary for making emotions and the decisions.
Based on these suggestions, we propose a model for making high-cognitive
abilities centered on emotions and decisions by affecting self-perception using VR.
Here, the VR based on the model is referred as to “self-perception-based VR.”
9 Making Emotion and Decision via Affecting Self-Perception 167
In this chapter, the mechanism of self-perception and multi-sensory integration

process will first be summarized. Then, we will discuss concrete approaches to
“self-perception-based VR” referencing engineering studies for creation of psy-
chological experiences. Final, we will offer a guide of “self-perception-based VR.”
9.2 Self-Perception in the Human Information Process
Emotion and decision will first be defined. Next, findings regarding the
self-perception process and mechanism of generating emotions and the decisions
will be summarized.
9.2.1 The Definition of Emotion and Decision
“Emotion” is used as a term for phenomenon related to movement of the mind.

How to best define emotion has debated for years [1, 13, 44]. However, it is defined
differently depending on the research field and is still controversial.
Sometimes, “affect,” “feeling,” and “mood” are used as equivalent terms with
“emotion.” These terms are used differently according to the strength and the length
of term of the psychological state.
For the current study, we treat “emotion” as comprising “emotion,” “affect,”
“feeling,” and “mood.” We define emotions as “subjective experiences and actions,
which are caused by changes in bodily responses, including facial expressions and
physiological reactions.” Subjective experiences and behaviors caused by changes
in bodily responses comprise reactions closely tied to affect, such as sadness,
happiness, anger, positive/negative or pleasant/unpleasant feelings, and other sub-
jective elements (i.e., value judgments based on affect).
Meanwhile, “decision” means the attitude of the mind to keep for taking a given
action. Decision-making behavior is “what to do next.” In general, the mind-set is
described using the word “decision-making,” which means that a cognitive process
for the selection and determination of things among options can be undertaken. In
this chapter, we use this definition of the decision.
Decision making is influenced by the environment. For example, the more
complex environment takes a longer time to undertake decision-making [14]. As
mentioned above, the body perception mutually interacts with the environmental
perception. Given this, the change in self-perception also would influence decisions.
On the other hand, emotional condition strongly affects decisions [36].
Therefore, a change in emotion depending on the alternation of self-perception
could have a larger impact on decisions.
9.2.2 The Mind Generator: One’s Body

and the Environment
There are three general phase of the human information process: “sensation,”
“perception,” and “cognition.” First, much the sensory stimuli are received through
sensory receptors in “sensation.” Second, the sensory information received is
chosen and created the meaning of it in “perception.” Third, the perceived sensory
information is interpreted and understood based on past experiences, memories, and
cumulative concepts in “cognition.”
In various fields, many researchers have argued that the recognition of the body
and mind changes almost synchronously; for example, recent physiological studies
have demonstrated that the change in blood flow [21] due to the action of the
automatic nervous system in accordance with the emotional change induces specific
physiological changes to body temperature [20, 34], skin conductance [12, 30], and
tremor [35].
Meanwhile whether emotion or bodily responses change first has been dis-
cussed. In this regard, recent studies tend to be positive theories that body recog-
nition changes prior to changes in emotions/decisions [31, 53]. In fact, regulation
body affects emotional experience [56, 57]. Additionally, some studies clarified that
the essential qualification of the change in the psychological state is not the actual
bodily change but getting the “feeling” that the actual body changes [35].
Meanwhile, Schachter et al., state that any physiological changes can be related
to several emotions; thus, discreet emotions cannot be completely determined by
only the change in bodily response (two-factor theory of emotion) [53]. The
recognition of any physiological change and the estimation of the physiological
change’s reason (attribution of causality) determine what type of emotions are
evoked. In this regard, Dutton et al. demonstrate that different emotions would be
evoked according to how people interpret their own environments, including sit-
uations when similar circumstances and bodily responses arise [18].
These theories would indicate that one’s psychological state and body is
unconsciously understood through the observation of the environment, though
one’s mind is recognized as centering the recognition of his/her body.
Furthermore, cognitive dissonance theory [22] and self-perception theory [7] are
argued as follows: When the self-intrapsychological state is ambiguous, human
knows his/her psychological condition, including emotion or attitude through
observation of own bodily condition, others’ conditions, and the environment. This
means that one’s self-psychological state is recognized depending on the infor-
mation clearly recognized.
The theories stated by Festiger et al., and Bem would have a comprehensive
description of the claims about the relationship between the recognition of self-body
and self-mind. Based on these findings, it seems that the recognition of one’s body
causes his/her psychological mind, since the mutual correspondence between the
self-body and self-mind in a certain environment is empirically known.
One’s body is recognized based on the perception of one’s body with integration
of sensory information received from the self-body and the environment. The
amount of information provided from these is almost infinity. Nevertheless, human
can separate them. The sensory information is organized in the process of per-
ception, and the distinction between these cannot be possible without one’s body.
In this regard, Gibson states that psychological activity has everything to do with
the environment, since the environmental perception attaches to the body percep-
tion accordingly [25]. This means that the perception of oneself (self-perception) is
the separation of one’s body from the environment centering his/her body perceived
through the integration of information regarding these.
Therefore, making emotions and decisions seem to be accordingly possible if
self-perception change. In order to alter the self-perception with the artificial stimuli
provided from the outside, a need exists for the appropriate presentation and
integration of the stimuli into the one’s body based on an understanding of the
self-perception process and the perceptual property.
9.2.3 Self-Perception and Body Image
The sensations to receive the bodily information and environmental information are
different. Human have senses to perceive the external environment (external sense)
and senses to perceive the bodily condition (internal sense). Figure 9.1 illustrates
the classification of these senses.
The internal sense is directly linked to the perception of one’s body. The body
perceived reliance on the internal sense is called as “body schema” [41, 54]. The
body schema, is the representation of the spatial positional relationship of the body
perceived mainly based on through somatic sensation.
Fig. 9.1 The classification of sensation. The sensations described in the higher point of this figure
enable sensory information to be perceived farther away from the body
However, a body is not known based only on the body schema; for example,
while the existence of the viscera in self-body is known, it is hard to know the
exact position or movement of the viscera with only the internal sense directly.
The external sense complements the internal sense to know such unknowable
senses. The representation of the body recognized depending on the integration of
both the internal and external senses is called “body image” [54]. This has a close
relationship with the self-conscious.
The body image and body schema are not always same; for example, “phantom
limb” is a good example of the body image differ from the body schema. This is a
phenomenon in which a nonexistent arm or leg is felt as if it exists [41]. In another
instance, the condition of face or the back of self cannot be understood until s/he
uses an object like a mirror.
Here, the boundary of the environment and body is generally considered the
physical skin. However, some external objects, which are not a body, are recog-
nized as part of one’s body in some cases; for instance, the “rubber hand illusion” is
well-known as the example of it. This is a illusion that a rubber hand is felt as if
own hand when tactile stimulus is presented to the rubber hand and an actual hand
put out of sight at same time [4, 9]. Conversely, some parts of one’s body would be
felt as though they are not part of body [29]. This means that the recognized body
image is formed beyond the physical confines. The distinction of where one’s body
ends and the environment begins is determined based on the subjective senses of
self-ownership and self-agency in the phase of perception [32]. A sense of
self-ownership is the feeling of “this is I.” A sense of self-agency is a feeling of “I
operate this.” These senses are perceived through understanding spatial or temporal
relationships between internal and external senses [4]. When the sensations are
caused in the external object, the object can be rephrased as an “extension of one’s
body.”
The body image extends and is updated adaptively to encourage the appropriate
behavior of human intentionally at difference times. Therefore, the self-perception
and body image are variable by affecting the internal and external senses
appropriately.
9.2.4 Perceptual Property
The world in the brain that is recognized through the human information process is
not necessarily similar to the actual physical world. In some cases, the physical
characteristics are perceived differently from the actual ones as an example of the
“phantom limb” [41] as cited above.
The property can be divided into a case due to the physical cause and a case due
to the cognitive bias. In the case due to physical cause, the perception would change
under the influence of single sensation modality or interaction of a number of
sensation modalities [17]. In the meantime, in the cases due to the cognitive bias, it
is known that perception is influenced by various types of cognitive biases, such as
past experiences, knowledge, concepts accumulated in the brain, and emotional

condition [28].
This difference is the same holds for self-perception. Given this, self-perception
also seems to be changeable by the perceptual property.
Regarding the use of the former case, we are going to use the influence of
multi-sensory interaction on self-perception with sensory stimuli from outside the
body. The influence of cognitive bias is assumed to be difficult to use for versatile
engineering interfaces, since it is individually different due to being acquired after
birth. Therefore, it is not our purpose to investigate the application of the effect of
the cognitive bias on how it affects self-perception.
9.3 The “Self-Perception-based VR” Model
Based on these findings, we propose a model for making emotions and decisions
with “self-perception-based VR.”
Here, the pathways for self-perception could be varied through three pathways:
internal sense, external sense, and correspondence between the internal and external
senses. “Self-perception VR” includes approaches for affecting each of the three
pathways to modify a body image related to the desired mind state in different
ways. Each approach will be discussed in Sects. 9.4, 9.5, and 9.6 with referring
related engineering studies. Based on the discussion, we will also give an allover
picture of the methodology in Sect. 9.7.
Figure 9.2 illustrates the conceptual model of “self-perception-based VR.” The
leftmost of this model depicts a physical body and the environment. The rightmost
of this model represents the correspondence between the body and environment
in the cognition process.
Fig. 9.2 The “self-perception-based VR” conceptual model. The leftmost of this model depicts a
physical body and the environment. The rightmost of this model represents the correspondence
between the body and environment in the cognition process. This methodology generates
emotions/decisions due to the change in the perception of self-body by affecting the internal sense,
external sense, and integration process of the internal and external senses
9.4 “Cross-Modal Approach”: Affecting Self-Perception

Through the Internal Sense
This section will explain the first approach for varying the body image by affecting
the internal sense, such as somatic and visceral sensations (Fig. 9.3).
Humans cannot grasp the exact conditions inside the self-body directly. In
particular, the change in the self-body caused by autonomic change is less con-
scious and cannot be controlled. One of the ways to affect the internal sense is
intervention inside of the body and controlling organs directly. However, practical
use of such methods is difficult due to the danger of physically and mentally taxing.
In this regard, we focus on the cross-modal effect indirectly on affect the internal
sense. Our sensation is highly interconnected. Perception through one sensory
system is changed by stimuli simultaneously received through other senses. This
phenomenon is referred to as “cross-modality” [55]. We use of this cross-modal
effect to affect the perception of the internal sense. This first approach is referred to
as the “cross-modal approach”.
9.4.1 Related “Cross-Modal Approach” Research
The famed early investigations of this approach would be the “false heartbeat
experiment” [61]. This experiment result shows that the speed of false heartbeat is
influenced the affection of a woman in a picture. In this case, the sound of the
heartbeat (false heartbeat) modified the perceived heartbeat through cross-modal
effect (auditory—visceral). This perceptional change would distrupt the
physiological/psychological excitements as reported in Dutton’s experiment [18].
Fig. 9.3 The “cross-modal approach” model. This approach generates a psychological state based
on the recognized body image through the changing of the perception of the internal sense
There are other studies for affecting emotion through modifying the perception
of the self-heartbeat; for example, the “empathetic heartbeat” requires watching a
movie of people with a tense feeling and hearing the sound of fast heartbeat
gradually getting louder through a headphone in the dark by placing a stethoscope
over one’s chest. This process blurs whether the owner of the heartbeat hearing is
one self or the other person. This creates empathy with others’ tense feeling [3].
A system proposed by Nishimura et al., presents the vibration stimulus functions
as a false heartbeat to one’s chest to evoke affection toward others [39]. The result
of their experiment shown that the speed of the false heartbeat tends to
increase/decrease the attractiveness of a woman in a picture. This have similar
tendency as Valins’s study [61].
Thus, the perception of the self-heart beat and awareness of self-emotion are
affected by stimuli provided from the environment. Since the heart rate corre-
sponding with various emotional states is known experimentally, the modified
heartbeat perception seems able to change one’s emotion. Meanwhile, the per-
ception of the same heartbeat evokes a different emotion according to the envi-
ronment as a two-factor theory of the emotion state. Therefore, the key to
evoking the preferred emotion using this approach is the design of what strikes
oneself as the causal attribution of the heart rate change.
The body temperature and skin conductance also relate to many types of emotions
[20, 34]. In particular, a report exists outlining that many perople have the common
image of the body temperature distribution associated from some specific emotional
states [40]. Furthermore, the physiological change in accordance with evoking an
emotion is not always only one. Focusing on this, Sakurai et al. have proposed systems
for evoking a number of emotions differently using a limited number of stimuli.
For example, “Comix: beyond” is aimed at evoking seven types of positive/
negative emotions using three types of stimulus: the vibration stimulation (false
heartbeat), thermal stimulus (false body temperature), and pressure stimulation (false
tight chest feeling) [47]. The actual tight chest feeling is caused by myocardial con-
traction. This system aims to cause the perception of physiological/psychological
choking feelings, since the tight chest feeling would literally create a tightening
feeling. The intensity of these stimuli changes according to the scene in a comic that
one is reading. The comic is used as a causal attribution of the change in the
recognition of one’s body. The comic could also make it possible to share the emotion
of a fictional character. Through user studies, it is possible to evoke various emotions
depending on the presented stimuli [48]. Meanwhile, when any of stimuli brought
strangeness, a desired emotion could not evoke. This means that the emotional state
could not alter if the stimuli are not felt as self-body changes.
“Communious mouse” also enables empathizing with others’ emotions that
described in remarks on the Internet [45]. This system provides thermal stimulation
and vibration stimulus to a palm through a mouse device. When the mouse cursor is
put on any remarks on the web browser, the system analyzes the emotional states
that each of the remarks mean. The result of the analysis changes the intensity of
each stimulus. The change in the modified perception of the skin temperature and
pulse is attributed to the remarks the cursor is put on. The change of stimuli
evokes the following four types of emotions differently: happiness, sadness, anger,
and surprise. The effect of this system has been underway in experiments.
The system temporally named “emotional controller,” displays thermal stimulus
and nebulized water through a game controller in order to affect the perception of
skin temperature and conductance responses. The preliminary experience they
conducted shown the positive outcome that this system is capable of augmenting an
impatient feeling [51]. Additionally, the impatience would generate difficulty in
playing the game. This means that the system seems to enable subjective assess-
ment of game skill depending on the generated emotion.
These studies also account for the by cross-modal effect (many types of haptic—
somatic/visceral). In order to solve the above problem of strangeness, the “com-
munious mouse” [45] and “emotion controller” [51] are also applied to the
“body-augmentation approach,” which will be explained in Sect. 9.6.
Another example of using the haptic sense, “Chilly chair” augments fear and
surprise due to the generation of false horripilation [23]. Although muscles actually
raise goose bumps, this system creates goose bumps by combing one’s skin hair
using static electricity. The false goose bumps increase fear and surprise when a
movie kindles fear.
These studies indicate that emotion can be evoked via the internal sense with the
stimuli provided outside. Meanwhile, some studies revealed that a decision is
affected through the change of the perception of the internal sense influenced with
the cross-modal effect.
The one of such study is “augmented satiety” proposed by Narumi et al. [38]. A
sense of satiety, one of organs senses, is linked to making decision concerning
eating behaviors. The sense of satiety is influenced by not only the condition of
one’s body but also other environmental factors, such as the prominence of food,
size of cutlery, and fellow dinners [15, 43, 63]. The appearance of the size of the
food has also a great effect on the relatively estimated amount of food [42].
Based on this, the augmented satiety system changes only the appearance of the
size of finger food through a head-mounted display (HMD). Using this system, they
showed the appearance of the size of food can increase/decrease food intake by
about ± 10 % [38].
A report also exists on how the ratio of the size of the dish and food also affects
food consumption [64]. Based on this finding, Sakurai et al. constructed a system
named “CalibraTable,” which has the shape of tabletop for adapting to a variety of
situations, such as general meals [49]. This system projects an image of a white
dish (virtual dish) around food and controls the size of the virtual dish. The size of
the virtual dish affects the sense of satiety and food intake.
Furthermore, Suzuki et al. proposed the “illusion cup” can affect drink intake
[59]. This system modifies the appearance of the height of a drinking cup based on
the knowledge that the height of a glass biases liquid consumption [5].
These systems show that visual stimuli can modify visceral sensation with the
cross-modal effect (visual—visceral) and affect decision making on food intake.
A sense of satiety is considered empirically understood after birth based on the
estimation on the food amount [42], such as the relationship between emotion and
bodily condition.
9.4.2 Discussion About the “Cross-Modal Approach”
As these studies show, the external sense can affect the perception of the internal
sense based on the cross-modal effect. Additionally the perceptional change in the
internal sense is considered capable of varying emotion and decision. On the other
hand, this approach could be applied to the internal sense that can take the
cross-modal effect.
Particularly, to improve the effect of evoking emotion using the false physio-
logical stimuli, consideration must be given to the difference between the perceptual
property between the internal and external senses; for example, even the same
thermal perception, body temperature inside of a body, is insusceptible to envi-
ronment temperature, whereas skin temperature is amenable to environment. The
strangeness of the presented stimuli means differences between temporal, spatial,
and intensity between each sense must be considered.
Nevertheless, we believe that the “cross-modal approach” is applicable in var-
ious situations since this approach can be used in already accumulated findings of
the relationship between the body, environment, and mental state. Using this
approach seems to evoke the different emotions in the same environment with the
same stimulus. Evoking the same emotion in a different environment using stimuli
differently appears difficult.
A sense of satiety is a less conscious ambiguous sense until it is integrated with
the other sense. The example of other senses with properties similar to a sense of
satiety include the sense of thirst, uresiesthesia, and carnal appetite. Therefore, this
seems to affect decision making and behavior by changing these senses using a
“cross-modal approach.” For instance, control of drink intake could be possible via
changing the perception of a thirsty feeling and change the aggressiveness in sexual
activity by varying the carnal appetite.
We also consider that the proper design of the environment, including any
interfaces, related to a particular behavior can be attributed to the change of one’s
body to the environment and can evoke desired emotions and decisions related to
the behavior. This has the advantage of needing no time for users to adapt to the
environment using knowledge of such experimental correspondences.
9.5 “Self-Observation Approach”: Affecting

Self-Perception Through the External Sense
This section will explain the second approach in affecting self-perception. This
approach modifies a body image through the external sense, such as visual and
auditory sensations (Fig. 9.4).
The self-body is regarded as not only the controlled object but also the obser-
vation object. In the self-observation process, the perception of the external sense
strongly influences on the recognition the one has.
Fig. 9.4 The “self-observation approach” model. This approach changes a body image by
affecting the external sense to supplement the internal sense
Based on this, this second approach changes self-body image through modifying
self-body projected onto the environment. This second approach is referred to from
this point as the “self-observation approach.”
9.5.1 Related “Self-Observation Approach” Research
In Sect. 2.2, we illustrated the influence of the self-body is observed objectively

through the external sense with the example of a mirror. A mirror is a part of the
environment. However, many people ordinarily believe without a doubt that the
body reflected in the mirror is the self-body.
Some studies show that the emotional state is affected by the self-face super-
imposed computer-generated information through the system just like the mirror;
for example, the “Tear machine” can induce a powerful distressed emotional state
[2]. This apparatus looks like a mirror. However, participants see tears streaming
down their face in the mirror. The participant is directed to feel distress and sadness
in response to strong empathic cues.
“Incendiary reflection” can also evoke an emotion due to the change appearance
of one’s face [67]. Although this system is in the shape of a triple mirror, a display
arranged halfway between two mirrors shows the deformed self-face. This display
feedback the picture of self-facial expression deformed at real-time. This system
was shown that pleasant/unpleasant feeling evoked when self-facial expression
appeared to be smiley/sad even though actual self-facial expression is being neutral.
The influence of the appearance of the self-face on emotion is explainable in
terms of the facial feedback hypothesis. Facial expression is culture-independent,
has universal human features and has an innate base [19]. Therefore, a certain
degree of knowledge exists about the mapping of facial expression and the
emotional state. Regarding the relationship between the face and emotions, the
facial feedback hypothesis states that one’s facial expression influences the one’s
emotion [60]. Moreover, a change of appearance in the self-face has a powerful
effect on one’s emotions regardless of whether the change is conscious or uncon-
scious [16, 58]. Moreover, Kleinke et al. reported that watching the self’s facial
change in a mirror has a greater effect on changing his/her emotional states [33].
“Incendiary reflection” can also predict the wearing of a muffler in accordance
with the emotional state due to the modified appearance of the self-face [67]. This
means that the self-body image recognized acts that are not only evoking emotion
but also making decision as an earlier study described [36].
One report outlines that sonority of the self-voice also has an effect on attitude
regarding self-emotion [27]. In an experiment Hatfield et al. conducted, participants
were asked to recite a novel with feedback of the self-voice through a headphone.
Intonation and tone of the self-voice were modulated in real time. The result of this
experiment shows that the sonority of the self-voice impacted on the impression
of the emotional state is received from the novel.
These studies are considered sufficient to support the work on the conscious
of the self-psychological mind of altered perceptions by the external sense.
9.5.2 Discussion About the “Self-Observation Approach”
The “self-observation approach” could vary the recognition of the observable body
surface. The psychological condition can be objectively found in changing body
surface. The term “expressed emotion” signifies the emotion represented with such
apparent bodily change, such as facial expressions or the sonority of voice [10]. The
meaning of the expressed emotion seems to be known, for instance, in Ekman
et al.’s report [19]. The visual contact of the self-body strongly influences on the
recognition of self-emotion. Given these reports, the “self-observation approach”
appears to have a powerful effect on the generation of various clear emotions due to
the change in the apparent surface of the body as a display of emotion. This
changing of the emotional state also influences on how decisions are made.
Meanwhile, the “self-observation approach” is applicable only in situation
where something to reflect self-physical body exists, since self-observation is not
possible without an environment for watching of oneself objectively. Expressed
emotion includes the movement action of a body. For example, Sasaki et al.
reported that motor actions of a finger movement due to active flicking input
influence emotional response to picture looking [52]. Given this finding, a proper
design for induction body movement or affecting static sensation would also gen-
erate the desired emotion or decision.
9.6 “Body-Augmentation Approach”: Affecting

Self-Perception Through the Designing
of the Correspondence Between the Internal
and External Senses
This section will explain a third approach for changing body image through the
construction of a new relationship between the internal and external senses
(Fig. 9.5).
A body image sometimes augments to the environment through understand-
ing of the spatial and temporal correspondence between internal and the external
senses; for example, the perception of temporal coincidence of haptic, somatic
(internal), and visual (external) senses generates a feeling as if the observation
target through the visual sensation is part of one’s body. The “rubber hand illusion”
is the known as famous example of this phenomenon [4, 9].
Meanwhile, the body image extends to the environment in an action with
agency. This expansion is caused by the success of the prediction of the spatial and
temporal change of the external sense in accordance with the active change of the
internal sense [24, 32].
In this regard, we will discuss gaming with a game controller. A game character
and a player are entirely physically different. However, if the player succeeded in
predicting the correspondence between the movement of his/her body and the
action of the character, the character is likely to be felt as reflection of the player’s
intention. Hence, the players feel pain or shock when the character is damaged,
although the players themselves are not actually damaged. This means that what
Fig. 9.5 The model of the “body-augmentation approach,” which generates a psychological state
with augmentation of the recognized self-body image by designing new correspondence between
the internal and external senses
happened to the character is felt as having to the player himself/herself. In this case,
an experience of an integration of the player’s body (internal sense) and the
character (external sense) is created.
This integration makes the observation object become part of his/her body. The
body schema perceived around a hand is reshaped in tune with operation object.
Moreover, the boundary between the hand and the operational object becomes
vague due to the extension of the body image beyond the confines to the physical
body. This example indicates that the change in the augmented body influences the
perception of physical body and psychological mind.
We attempt to change the body image through such paired-associative learning
of the internal and external senses. This third approach is henceforth referred as the
“body-augmentation approach.”
9.6.1 Related “Body-Augmentation Approach” Research
As one of the interface applies the “body-augmentation approach,” “Interactonia

Balloon” [46] evokes a tense feeling by shifting causal attribution of one’s respi-
ratory condition from self-intention to the augmented body. A tense feeling, which
is one emotional state, accelerates and shallows the breathing. One’s respiratory
condition evokes psychological choking feeling. Given the mutually relationship
between respiration, physiological state, and psychological one, the choking feeling
could be evoked if the physiological smothering feeling is attributed to the
environment.
A balloon, which is incorporated in this apparatus, inflates due to breath-holding.
The balloon also deflates due to breathing in/out. These different movements of the
balloon from the usual one require to learning phase to a understand the corre-
spondence between self-respiration and the balloon movement. This learning would
arise from a sense of agency regarding the balloon. At this time, the smothering
feeling is attributed to intentional breath-holding.
However, the air becomes difficult to emerge from the balloon growing beyond a
certain size. This makes the balloon less likely to deflate regardless of the respi-
ratory condition. This means that the balloon moves differently from the prediction
of correspondence with breathing condition. Such unpredictable movement of the
balloon seems to cause a feeling of being unsure of what is happening to the
self-body, since the balloon moves without a relationship to one’s conscious. This
conscious evokes a tense feeling.
The recognition of the correspondence between self-respiration and the move-
ment of the balloon would be addled owing to an impatience comes from a potential
to break the balloon via overexpansion, in addition to the paradoxical movement of
the balloon. The attribution causality of choking feeling is likely to shift the causal
attribution of the choking feeling is likely to shift the causal attribution of the
feeling from active breathing to the change in the balloon. This shift of the attri-
bution causality of the feeling of oppression magnifies the tense feeling [50].
However, if the movement of a balloon is considered to have no relationship

with self-respiration, a choking feeling was not evoked. Likewise, a tense feeling
failed be evoked when other environmental factors, such as environmental noise,
were felt to be annoying. This study has not clarified the process of the shifting of
the causal attribution of a feeling of smothering, and the intensity of a sense of
agency that arises; therefore, more elaborate investigation is required.
The “Interactonia Balloon” expands one’s body image to the environment
through paired-associative learning of the internal and external senses, which are
not originally connected to each other. On the other hand, when the body image is
augmented to the environment, the border between the physical body and envi-
ronment, which are perceived as a part of the body image, is blurred as men-
tioned earlier. If the stimulus is presented through the environment included in the
body image, it seems confused as to whether the stimulus came from the envi-
ronment or the actual body. Therefore, the presented stimulus from the augmented
body is perceived as the change of the self-body without strangeness.
The effect of the ambiguity of the boundary in the mind is applied to the
“communious mouse” [45] and “emotion controller” [51] described in Sect. 9.3.;
for example, a sense of agency is caused in the mouse through learning and pre-
dicting the correspondence between self-hand motion and the mouse cursor. As
mentioned above, such expansion of the body image blurs the line between the
hands and interface, which is controlled with the hands. The visual pulsating of the
cursor would be likely to affect strongly the recognition of changing the pulse.
Besides, each of the system provides the stimuli through an interface, which has
ill-defined border with the hand in conscious. This could make the user feel that the
stimuli result from not the mouse presents but the change of body of self naturally.
9.6.2 Discussion About the “Body-Augmentation

Approach”
The “body-augmentation approach” would generate emotion by designing the

correspondence between the internal and external senses.
The largest difference between the “cross-modal/self-observation approaches”
and “body-augmentation approach” is whether the correspondence between the
internal and external senses, which is used to modify the body image, is already
known or unknown.
The “cross-modal/self-observation approaches” use the already-known mutual
relationship between the body, environment, and psychological state. The corre-
spondence of the three is understood inherently or empirically. Therefore, this
seems to enable the generation of various types of mental state. Meanwhile, it is
difficult to stain the relationship.
In response, the “body-augmentation approach” requires the phase to a paired-
associative learning of the correspondence between the body and environment,
since the correspondence relationship is unknown. What type of emotion or
decision is caused by the learnt correspondence causes is also unknown. Therefore,

the important key is how to design new link between internal and external senses.
Moreover, the “body-augmentation approach” could re-create the relationship
between the body and the environment in many ways. This has the advantage of
being able to induce the psychological state that cannot be achieved by only the
self-body.
This section refers to related works that pertain to only evoking emotion.
However, the emotional state influences the decision making and intellectual ability
as shown by the earlier works [36, 67]. Given this, it seems that the body-
augmentation approach acquires the ability to generate the decision and the intel-
lectual capacity as well as emotion. In other words, unconsciously making better
decisions or higher performances than the original would be possible.
9.7 General Representation of “Self-Perception-based

VR”
Thus far, we had proposed a “self-perception-based VR” model. This is designed to

affect each of the three types of pathways of self-perception in human information
processing using VR. We also discuss the advantages and problems of each
approach. Figure 9.6 shows an integrated model of the three approaches explained
in Sects. 9.4, 9.5, and 9.6.
Fig. 9.6 The integrated model of three types of “self-perception-based VR.” One’s emotions and
decisions comes from the physical sensations in any pathways for self-perception
Fig. 9.7 General representation of the “self-perception-based VR” model. The core of this model
is how the boundary between the body and environment is created
Each approach changes the perception of the self-body based on the

self-knowledge of the correspondence between one’s body and environment; in
other words, any of the approaches affecting the self-consciousness where one’s
body ends and the environment begins. This means that a “self-perception-based
VR” model, which the paper proposes, is nothing more or less than the construction
of a boundary between the self-body and the environment in the mind. Figure 9.7
shows a general representation of the “self-perception-based VR” model.
The approach of “self-perception-based VR” appears as a vague black box, since
the parameterization of the perception of the multiple senses. However, as noted
throughout this paper, many studies have accumulated findings about the rela-
tionship between the environment/body and the body/psychological state. Various
studies have also shown the influence of multi-sensory integration on perception
and cognition in the human information process. Moreover, some studies have
developed the measure of self-consciousness and sense of self-agency. These
studies would discover and ascertain new knowledge. We believe that
“self-perception-based VR” is feasible and developable with further research.
9.8 Conclusion and Future Works
The “self-perception-based VR” model can be described as follows: changes to

body image by affecting self-perception with use of the accumulated knowledge of
the correspondence between the body and environment in the opposite way. Our
model will have enabled VR to generate subjective experience without relying on
the reproducibility of the physical information. Unlike conventional VR,

“self-perception-based VR” has no need for meaning interpretation of the input
sensory information for making a subjective mind, since the new relationship
between the stimuli comes from outside and a body before the input information is
recognized. Therefore, the reproduction of the environment that is a strong
resemblance to the actual world is not indispensable. Thus, being able to affect
mind state with only physical stimulation is an enormous advantage to engineering.
Conventional engineering has used an information model applying the senso-
rimotor loop to a computer metaphor for understanding humans [65]. The senso-
rimotor loop is simple bottom-up information process, which involves movement
resulting from the recognition of perceived sensations and input senses that result
from the movement [11]. However, the influence of multi-sensory integration on
the recognition and top-down effect of accumulated recognition experience on the
perception have been rarely discussed. Therefore, engineering technologies had
treated human mind as only noisy date.
In contrast, our model can be considered to extend the idea of the human
information model based on the conventional computer metaphor. Therefore, our
model has a wide field of application; for example, our model could be used in the
field human interface involving presenting information to humans.
We believe that our model can be applied to engineering techniques for pro-
ducing “empathy” in various human–computer interaction. Regarding the definition
of “empathy,” Stodland defines it as a emotional response that comes from the
perception of another person’s experience of an emotion or another person’s
experience of an emotion [8]. In this chapter, we define “empathy” as an influence
of the recognition of the subjective experience or the movement of an observation
object on the self-mind; for example, our proposed model could be used on a
life-long basis with the model for recording behavior and experience as digital data.
Our model also will be helpful for solving health problems, such as obesity and
depression, or the various intellectual abilities and communication skills in the
healthcare and academic fields. For instance, there are many approaches for mod-
ifying self-behavior based on the consciousness of the issues in order to induce the
subjective condition in a desirable direction. It is hard to feel that these approaches
are not mentally taxing. A certain degrees of the effect of these approaches are
shown, while applying our model in such fields, these issues might be resolvable by
directly affecting one’s mental condition.
The model will be able to develop a technology for creating empathy between
human and computer; for instance, our model can be applying to the avatar and the
teleoperation system. The operability of such system has been supported through
the development of physical accuracy in accordance with the user’s movement.
However, the operability is influenced by a sense of self-agency and
self-ownership. Therefore, in such a situation, our model will be helpful for
improvement of the ability to handle without depending on only the original control
capability of the user or machine accuracy improving.
Furthermore, our model would be able to contribute to the development of
psychology and cognitive science. As we have shown throughout this chapter, our
model would have a feasibility of virtual generation and switching psychological

experience using the findings of psychology and cognitive science. This would
enable the creation of never before possible experimental conditions. Therefore,
investigating the effect of the creation of emotions or decisions with our model on
the action or whole recognition enables to find new knowledge.
Meanwhile, we can also know the self-mind state through the observation of the
environment as noted in some studies [7, 22, 25]. The perception of the self-body and
environment mutually interacts as mentioned earlier. Therefore, the “self-perception-
based VR” model designs the self-body perception into a model of designing envi-
ronment perception.
As mentioned earlier, the problem of conventional VR is the lack of sufficient
discussion of the environmental impact on the human mind and conscious.
Nevertheless, recent researches has elucidated the environmental perception of the
unconsciously effects on self-perception and decision performance [38, 49, 58].
Furthermore, the presence of other people and particular environments effect high-
cognitive ability, such as emotions, decisions, and intellectual ability; for example,
emotion or tempo movement is considered “infectious” by observing one another’s
bodily responses or actions [65]. Besides, some VR studies develop various abilities
with application, such findings and approaches for changing self-perception intro-
duced in this chapter [6, 37].
Thus, VR technologies for creating human minds and consciousness will be able
to develop through the further validation of mutual interaction between the envi-
ronment, self-body, and cognition.
References
1. Anderson T (1998) Circumplex Models of Personality and Emotion. R. Plutchik & H. R. Conte
(Eds). (1997). Washington, DC: American Psychological Association. Psychotherapy Research
8:359–360. doi:10.1080/10503309812331332457
2. Andersson U (2001) The Tear Machine. http://www.weststarland.com/wp/2011/04/tear-m
achine-tearful-reflection/. Accessed 12 Jun 2013
3. Ando H, Watanabe J, Sato M (2012) Empathetic heartbeat. Proceedings of the 20th ACM
international conference on Multimedia - MM ‘12. doi:10.1145/2393347.2396530
4. Armel K, Ramachandran V (2003) Projecting sensations to external objects: evidence from
skin conductance response. Proceedings of the Royal Society B: Biological Sciences
270:1499–1506. doi:10.1098/rspb.2003.2364
5. Attwood A, Scott-Samuel N, Stothart G, Munafò M (2012) Glass Shape Influences
Consumption Rate for Alcoholic Beverages. PLoS ONE 7:e43007. doi:10.1371/journal.pone.
0043007
6. Ban Y, Sakurai S, Narumi T, Tanikawa T, Hirose M (2015) Improving work productivity by
controlling the time rate displayed by the virtual clock. Proceedings of the 6th Augmented
Human International Conference on - AH ‘15. doi:10.1145/2735711.2735791
7. Bem D (1967) SELF-PERCEPTION: AN ALTERNATIVE INTERPRETATION
OF COGNITIVE DISSONANCE PHENOMENA. Psychological Review 74:183–200.
doi:10.1037/h0024835
8. Berkowitz L (1969) Advances in experimental social psychology. Academic Press, New York
9. Botvinick M, Cohen J (1998) Rubber hands ‘feel’ touch that eyes see. Nature 391:756–756.
doi:10.1038/35784.
10. Butzlaff R, Hooley J (1998) Expressed Emotion and Psychiatric Relapse. Arch Gen Psychiatry
55:547. doi:10.1001/archpsyc.55.6.547
11. Card S, Moran T, Newell A (1983) The psychology of human-computer interaction.
L. Erlbaum Associates, Hillsdale, N.J.
12. Carlson N (1977) Physiology of behavior. Allyn and Bacon, Boston
13. Dalgleish T, Power M (1999) Handbook of cognition and emotion. Wiley, Chichester,
England
14. Davidson A, Bar-Yam Y (2006) Environmental Complexity: Information For
Human-Environment Well-Being. Unifying Themes in Complex Systems 157–168. doi:10.
1007/978-3-540-35866-4_16
15. De Castro J (1994) Family and friends produce greater social facilitation of food intake than
other companions. Physiology & Behavior 56:445–455. doi:10.1016/0031-9384(94)90286-0
16. Dimberg U, Söderkvist S (2010) The Voluntary Facial Action Technique: A Method to Test
the Facial Feedback Hypothesis. J Nonverbal Behav 35:17–33. doi:10.1007/s10919-010-
0098-6
17. Driver J, Spence C (2000) Multisensory perception: Beyond modularity and convergence.
Current Biology 10:R731–R735. doi:10.1016/s0960-9822(00)00740-5
18. Dutton D, Aron A (1974) Some evidence for heightened sexual attraction under conditions of
high anxiety. Journal of Personality and Social Psychology 30:510–517. doi:10.1037/
h0037031
19. Ekman P, Friesen W (1969) The Repertoire of Nonverbal Behavior: Categories, Origins,
Usage, and Coding. Semiotica. doi:10.1515/semi.1969.1.1.49
20. Ekman P, Levenson R, Friesen W (1983) Autonomic nervous system activity distinguishes
among emotions. Science 221:1208–1210. doi:10.1126/science.6612338
21. Elam M, Wallin B (1987) Skin blood flow responses to mental stress in man depend on body
temperature. Acta Physiologica Scandinavica 129:429–431. doi:10.1111/j.1365-201x.1987.
tb10609.x
22. Festinger L (1962) Cognitive Dissonance. Sci Am 207:93–106. doi:10.1038/
scientificamerican1062-93
23. Fukushima S, Kajimoto H (2012) Chilly chair. ACM SIGGRAPH 2012 Emerging
Technologies on - SIGGRAPH ‘12. doi:10.1145/2343456.2343461
24. Gallagher S (2000) Philosophical conceptions of the self: implications for cognitive science.
Trends in Cognitive Sciences 4:14–21. doi:10.1016/s1364-6613(99)01417-5
25. Gibson J (1979) The ecological approach to visual perception. Houghton Mifflin, Boston
26. Hatfield E, Bensman L, Thornton P, Rapson R (2014) New Perspectives on Emotional
Contagion: A Review of Classic and Recent Research on Facial Mimicry and Contagion.
Interpersona 8:159–179. doi:10.5964/ijpr.v8i2.162
27. Hatfield E, Hsee C (1995) The Impact of Vocal Feedback on Emotional Experience and
Expression. Journal of Social Behavior and Personality 10:293–313.
28. Hilbert M (2012) Toward a synthesis of cognitive biases: How noisy information processing
can bias human decision making. Psychological Bulletin 138:211–237. doi:10.1037/a0025940
29. Holmes E, Brown R, Mansell W, Fearon R, Hunter E, Frasquilho F, Oakley D (2005) Are
there two qualitatively distinct forms of dissociation? A review and some clinical implications.
Clinical Psychology Review 25:1–23. doi:10.1016/j.cpr.2004.08.006
30. Hökfelt T, Johansson O, Ljungdahl Å, Lundberg J, Schultzberg M (1980) Peptidergic
neurones. Nature 284:515–521. doi:10.1038/284515a0
31. James W (2011) The Principles of Psychology (Volume 2 of 2). Digireads. com Publishing
32. Jeannerod M (2003) The mechanism of self-recognition in humans. Behavioural Brain
Research 142:1–15. doi:10.1016/s0166-4328(02)00384-4
33. Kleinke C, Peterson T, Rutledge T (1998) Effects of self-generated facial expressions on
mood. Journal of Personality and Social Psychology 74:272–279. doi:10.1037//0022-3514.74.
1.272
34. Levenson R, Ekman P, Friesen W (1990) Voluntary Facial Action Generates Emotion-Specific
Autonomic Nervous System Activity. Psychophysiology 27:363–384. doi:10.1111/j.1469-
8986.1990.tb02330.x
35. Marshall J, Schnieden H (1966) Effect of adrenaline, noradrenaline, atropine, and nicotine on
some types of human tremor. Journal of Neurology, Neurosurgery & Psychiatry 29:214–218.
doi:10.1136/jnnp.29.3.214
36. Naqvi N, Shiv B, Bechara A (2006) The Role of Emotion in Decision Making: A Cognitive
Neuroscience Perspective. Current Directions in Psychological Science 15:260–264. doi:10.
1111/j.1467-8721.2006.00448.x
37. Nakazato N, Yoshida S, Sakurai S, Narumi T, Tanikawa T, Hirose M (2014) Smart Face.
Proceedings of the 17th ACM conference on Computer supported cooperative work & social
computing - CSCW ‘14. doi:10.1145/2531602.2531637
38. Narumi T, Ban Y, Kajinami T, Tanikawa T, Hirose M (2012) Augmented perception of
satiety. Proceedings of the 2012 ACM annual conference on Human Factors in Computing
Systems - CHI ‘12. doi:10.1145/2207676.2207693
39. Nishimura N, Ishi A, Sato M, Fukushima S, Kajimoto H (2012) Facilitation of affection by
tactile feedback of false heratbeat. Proceedings of the 2012 ACM annual conference extended
abstracts on Human Factors in Computing Systems Extended Abstracts - CHI EA ‘12. doi:10.
1145/2212776.2223796
40. Nummenmaa L, Glerean E, Hari R, Hietanen J (2013) Bodily maps of emotions. Proceedings
of the National Academy of Sciences 111:646–651. doi:10.1073/pnas.1321664111
41. Ramachandran V, Blakeslee S (1998) Phantoms in the brain. William Morrow, New York
42. Rolls B, Engell D, Birch L (2000) Serving Portion Size Influences 5-Year-Old but Not
3-Year-Old Children’s Food Intakes. Journal of the American Dietetic Association 100:232–
234. doi:10.1016/s0002-8223(00)00070-5
43. Rolls B, Morris E, Roe L (2002) Portion size of food affects energy intake in normal- weight
and overweight men and women. American Journal of Clinical Nutrition 76:1207–1213.
44. Russell J (1980) A circumplex model of affect. Journal of Personality and Social Psychology
39:1161–1178. doi:10.1037/h0077714
45. Sakurai S, Ban Y, Katsumura T, Narumi T, Tanikawa T, Hirose M (2014) Communious
mouse. SIGGRAPH Asia 2014 Emerging Technologies on - SA ‘14. doi:10.1145/2669047.
2669057
46. Sakurai S, Katsumura T, Narumi T, Tanikawa T, Hirose M (2012) Interactonia Balloon.
SIGGRAPH Asia 2012 Emerging Technologies on - SA ‘12. doi:10.1145/2407707.2407722
47. Sakurai S, Katsumura T, Narumi T, Tanikawa T, Hirose M (2013) Comix: beyond.
SIGGRAPH Asia 2013 Emerging Technologies on - SA ‘13. doi:10.1145/2542284.2542287
48. Sakurai S, Katsumura T, Narumi T, Tanikawa T, Hirose M (2014) Evoking Emotions in a
Story Using Tactile Sensations as Pseudo-body Responses with Contextual Cues. Lecture
Notes in Computer Science 241–250. doi:10.1007/978-3-319-07731-4_25
49. Sakurai S, Narumi T, Ban Y, Tanikawa T, Hirose M (2013) Affecting Our Perception of
Satiety by Changing the Size of Virtual Dishes Displayed with a Tabletop Display. Virtual,
Augmented and Mixed Reality Systems and Applications 90–99. doi:10.1007/978-3-642-
39420-1_11
50. Sakurai S, Narumi T, Katsumura T, Tanikawa T, Hirose M (2015) Basic Study of Evoking
Emotion Through Extending One’s Body Image by Integration of Internal Sense and External
Sense. Human Interface and the Management of Information Information and Knowledge
Design 433–444. doi:10.1007/978-3-319-20612-7_42
51. Sakurai S, Narumi T, Tanikawa T, Hirose M (2015) A Proposal of a Game Controller for
Evoking Emotion by Creating an Illusion of Perception of One’ s Bodily Response around
Hand. In: ASIAGRAPH2015. pp 15–19
52. Sasaki K, Yamada Y, Miura K (2015) Post-determined emotion: motor action retrospectively
modulates emotional valence of visual images. Proceedings of the Royal Society B: Biological
Sciences 282:20140690-20140690. doi:10.1098/rspb.2014.0690
53. Schachter S, Singer J (1962) Cognitive, social, and physiological determinants of emotional
state. Psychological Review 69:379–399. doi:10.1037/h0046234
54. Schwoebel J, Coslett HB (2005) Evidence for Multiple, Distinct Representations of the Human
Body. Journal of Cognitive Neuroscience 17:543–553. doi:10.1162/0898929053467587
55. Shimojo S (2001) Sensory modalities are not separate modalities: plasticity and interactions.
Current Opinion in Neurobiology 11:505–509. doi:10.1016/s0959-4388(00)00241-5
56. Shimojo S, Simion C, Shimojo E, Scheier C (2003) Gaze bias both reflects and influences
preference. Nature Neuroscience 6:1317–1322. doi:10.1038/nn1150
57. Simion C, Shimojo S (2007) Interrupting the cascade: Orienting contributes to decision
making even in the absence of visual stimulation. Perception & Psychophysics 69:591–595.
doi:10.3758/bf03193916
58. Strack F, Martin L, Stepper S (1988) Inhibiting and facilitating conditions of the human smile:
A nonobtrusive test of the facial feedback hypothesis. Journal of Personality and Social
Psychology 54:768–777. doi:10.1037//0022-3514.54.5.768
59. Suzuki E, Narumi T, Sakurai S, Tanikawa T, Hirose M (2014) Illusion cup. Proceedings of the
5th Augmented Human International Conference on - AH ’14. doi: 10.1145/2582051.2582092
60. Tomkins S, Karon B (1962) Affect, imagery, consciousness. Springer, New York
61. Valins S (1966) Cognitive effects of false heart-rate feedback. Journal of Personality and
Social Psychology 4:400–408. doi:10.1037/h0023791
62. Van Ittersum K, Wansink B (2012) Plate Size and Color Suggestibility: The Delboeuf
Illusion’s Bias on Serving and Eating Behavior. J Consum Res 39:215–228. doi:10.1086/
662615
63. Wansink B, Painter J, Lee Y (2006) The office candy dish: proximity’s influence on estimated
and actual consumption. Int J Obes Relat Metab Disord 30:871–875. doi:10.1038/sj.ijo.
0803217
64. Wansink B, Van Ittersum K, Painter J (2006) Ice Cream Illusions. American Journal of
Preventive Medicine 31:240–243. doi:10.1016/j.amepre.2006.04.003
65. Wolpert D, Ghahramani Z (2000) Computational principles of movement neuroscience. Nat
Neurosci 3:1212–1217. doi:10.1038/81497
66. Watanabe K (2008) Behavioral speed contagion: Automatic modulation of movement timing
by observation of body movements. Cognition 106:1514–1524. doi:10.1016/j.cognition.2007.
06.001
67. Yoshida S, Sakurai S, Narumi T, Tanikawa T, Hirose M (2013) Incendiary reflection.
ACM SIGGRAPH 2013 Talks on - SIGGRAPH ’13. doi:10.1145/2504459.2504503
Chapter 10
Neural Basis of Maternal Love as a Vital
Human Emotion
Yoshiaki Kikuchi and Madoka Noriuchi
Abstract Maternal love, which may be the core of maternal behavior, is essential
for the mother–infant attachment relationship and is important for an infant’s
development and mental health. Therefore, maternal love can be considered a vital
human emotion. Using video clips, we examined patterns of maternal brain acti-
vation in response to infant cues. We performed functional magnetic resonance
imaging (fMRI) measurements while 13 mothers viewed video clips of their own
infants and other infants (all approximately 16 months of age) who demonstrated
two different attachment behaviors. We found that a limited number of the mother’s
brain areas were specifically involved in maternal love, namely orbitofrontal cortex
(OFC), striatum, anterior insula, and periaqueductal gray (PAG). Then, we pro-
posed a schematic model of maternal love, based on integration of the two systems
in the OFC: the dopamine reward system (OFC, striatum) and the interoceptive
information processing system (OFC, insula, PAG). Additionally, we found a
strong and specific brain response in a mother viewing her distressed infant. The
neural activation pattern was found in the dorsal OFC (dOFC), the dorsolateral
prefrontal cortex (DLPFC), ventrolateral prefrontal cortex (VLPFC), dorsomedial
prefrontal cortex (DMPFC), dorsal anterior cingulate cortex (dACC), posterior
cingulate cortex (PCC), caudate nucleus, supplementary motor area (SMA), and
posterior superior temporal sulcus/temporoparietal junction (pSTS/TPJ). These
results showed a highly elaborate neural mechanism, based on the above neural
basis of maternal love, mediating the diverse and complex maternal behaviors that
mothers engage in when raising and protecting their own infants.
Y. Kikuchi (&) M. Noriuchi

Department of Frontier Health Science, Division of Human Health Sciences,
Graduate School of Tokyo Metropolitan University, 7-2-10, Higashi-Ogu,
Arakawa-Ku, Tokyo 116-8551, Japan
e-mail: ykikuchi@tmu.ac.jp
M. Noriuchi
e-mail: noriuchi@tmu.ac.jp

DOI 10.1007/978-3-319-29433-9_10
190 Y. Kikuchi and M. Noriuchi
10.1 Maternal Love as a Vital Human Emotion

(Introduction)
Maternal love is one of the most powerful motivational factors underlying the
behavior of a mother as she cares for and protects her infant. That is, maternal love
is essential for the maternal behavior in which a mother makes sacrifices necessary
to care for her infant; this love is a dynamic force that empowers a mother to
maintain vigilance and sustain the exhausting schedule involved in protection and
nurturing of an infant. Bowlby [1] stated that a mother’s love during a child’s
infancy and childhood is as important for mental health of a child as are vitamins
and proteins for physical health. The amount of love involved in a mother’s
interactions with her infant has a profound influence on the stability of the mother–
infant relationship and the quality of the mother–infant attachment. Therefore, it can
be said that maternal love is an exceptional human emotion that is essential for our
species preservation. To clarify the neural basis of human maternal love and related
behaviors is crucial to understand normal as well as abusive and neglectful
mothering [2–8].
10.2 Methods
10.2.1 Neuroimaging Based on Infant Attachment

Behaviors
Mother–infant attachment is not a unilateral process that depends solely on the

mental state and attitudes of the mother. The infant plays an important part in
mother–infant attachment because infant behavior has a powerful effect on a
mother’s emotions. Therefore, strong maternal attachment, which is expressed in
the mother as affectionate behavior, vigilance, and protectiveness, should be
induced by an infant’s attachment behaviors. We assumed that the neural correlates
responsible for maternal behavior and perhaps its core, maternal love, could be
realistically assessed by observing a mother’s brain activity as she viewed video
clips of her own infant demonstrating a variety of attachment behaviors [9, 10].
We focused on mothers of infants who were approximately 16 months of age,
had developed a specific tie to their mothers, and displayed clear attachment
behaviors [9]. Attachment behaviors, exhibited by a mother’s own infant and other
infants in two different situations, were presented to the mothers as video stimuli. In
the first situation, the infant was smiling at his/her own mother while playing with
his/her mother (play situation: PS). In a second situation, the infant showed distress
when his/her mother left (separation situation: SS) (see Fig. 10.3). While mothers
experience happiness when watching video clips of their own infant in the first
situation, they should feel worry and protective when shown video clips of their
own infant in the second situation. As mothers are impelled to protect their infants,
10 Neural Basis of Maternal Love as a Vital Human Emotion 191
a biologically essential mechanism for species preservation, the neural system

mediating maternal behavior should be shown more clearly when a mother views a
situation depicting her own infant in distress.
10.2.2 Subjective Ratings of Feelings
After the fMRI scan, the mother was asked to rate her feelings (happy, motherly,
joyful, warm, love, calm, excited, anxious, irritated, worry, and pity) while viewing
sample video clips selected from the video stimuli. The subjective ratings described
as happy, motherly, joyful, warm, love, calm, and excited were significantly higher
when mothers viewed their own infants, compared with other infants in the PS, and
they were also higher for motherly love and excited in the SS. In addition, there
were no significant differences in the subjective feelings of motherly love, sug-
gesting that mother’s love exists in the mother herself, regardless of how she
responded to her own infant in any situation.
10.3 Maternal Love
10.3.1 Neural Basis of Maternal Love
Based on the assumption that maternal love is invariant, existing in the mother
herself, regardless of how she responds to her own infant in any situation and also
as shown in the results of mother’s subjective feelings [2], we found a limited
number of a mother’s brain areas that could be specifically involved in maternal
love, namely the right OFC (Fig. 10.1a), anterior insula (Fig. 10.1b), PAG
(Fig. 10.1c), and the striatum.
The OFC plays an important role in the reward system; it receives ascending
dopamine projections from the ventral tegmental area (VTA) and is critical in
representing a stimulus reward value [11, 12]. Correlation analyses showed that the
magnitude of activation in the right OFC was positively correlated with a mother’s
intensity of worry, and the magnitude of activation of the left OFC was positively
correlated with the intensities for feelings of joy and happiness (Fig. 10.1a). All
these feelings, both positive and negative, are important for facilitating maternal
behavior.
The PAG has direct connections with the OFC [13], which may explain the
equally specific activation of the OFC in relation to maternal love. The PAG also
receives direct connections from the limbic areas and contains a high density of
oxytocin receptors [14]. In fact, maternal behaviors may be inhibited when PAG is
pharmacologically or physically targeted [15, 16]. Our finding supports the notion
that the PAG is involved in the maternal love that a mother feels for her infant.
Fig. 10.1 The OFC activations when mothers viewed their own infants versus other infants (a).
Scatter plots depict the positive correlations between the activity of the left OFC and the intensity
of joy and happiness (right) and between the activity of the right OFC and the intensity of worry
(left). Activation in the anterior insula was revealed when mothers viewed their infants versus other
infants (b). Activation in the periaqueductal gray (PAG) was revealed when mothers viewed their
own infant versus other infants (c)
Additionally, the PAG is known to be involved in endogenous pain suppression

during one’s experience of intense emotional experiences, such as childbirth;
moreover, a previous study demonstrates that this brain activity is facilitated
through oxytocinergic action [17].
The anterior insula is involved in processing caress-like touching between
individuals [18]. This is considered to be important for affiliative behavior between
the mother and infant. The insular cortex is organized in a hierarchical caudal–
rostral direction, whereby primary sensory inputs projecting to the posterior insula,
including somatosensory, vestibular, and visceral inputs, are progressively pro-
cessed and integrated across modalities in the middle insula [19, 20]. The insula
differentiates sympathetic and parasympathetic activity [21, 22]; electrical stimu-
lation of the right insular cortex elevates diastolic blood pressure and heart rate
while stimulation of the left insula decreases heart rate [23, 24]. Sympathetic
activity appears to be represented in the right hemisphere [21, 24].
The striatum receives strong projections from the OFC [25], and it plays an
important role in stimulus reward learning. In turn, this learning is mediated by
afferent dopamine input so that responses associated with predictions of greater

reward in a given context are reinforced, and hence, they are more likely to be
subsequently selected. In addition, the striatum contains cells that respond to food
and drink rewards, and it has been shown to be activated by monetary reward
stimuli [26, 27], cocaine [28], and sexual arousal in humans [29].
10.3.2 Schematic Model of Maternal Love
Based on the above results, we can propose a schematic model of maternal love
(Fig. 10.2). The OFC and striatum are included in the dopamine reward system, and
they mediate reward evaluation and motivation. Moreover, the OFC, insula, and
PAG are included in the interoceptive information processing system; they are
related to what are known as “homeostatic emotions.” These are emotions specif-
ically relevant to our ability to self-regulate through homeostasis. That is, the
mother’s brain is activated in such a way that the OFC integrates the two systems,
the reward system and the interoceptive processing system, and such integration of
these two systems motivates maternal behavior. Based on this model, it is rea-
sonable to assume that a mother’s own infant is not only her own reward, acting as
a motivation for caretaking, but that the infant is also responsible for the mother’s
Fig. 10.2 A schematic model of the neural basis of maternal love based on our findings. The
orbitofrontal cortex (OFC) and striatum are included in the dopamine reward system, and the OFC,
insula, and PAG are included in the interoceptive information processing system. The former
system mediates reward evaluation while interacting with the motivation of a mother to care for
her infant, and the latter mediates homeostatic emotions and feelings related to the realization of
motherhood that is a mother living with her infant. The OFC plays an important role in integrating
the two systems. Based on the model, it may be said that mother’s infant is not only a reward but
also acts to protect the mother through producing homeostatic emotions
homeostasis. That is, in addition to the mother protecting her infant, the infant also
protects the mother.
10.4 Maternal Behavior Based on Maternal Love
10.4.1 Mother’s Responses to Her Own Infant’s Distress
Mothers showed a specific pattern of neural response to their own infant’s distress
(SS) compared with their own infant’s smiling (PS) (Fig. 10.3). The neural acti-
vation pattern appeared in the dOFC, DLPFC, VLPFC, DMPFC, dACC, PCC,
SMA, caudate nucleus, and pSTS/TPJ (Fig. 10.3 separation situation versus play
situation). That is, complicated neural processing might be required for a mother to
quickly recognize and respond to an infant’s cue that he/she is in distress.
The dOFC is related to behavioral choice [30], and its activity may therefore
reflect a selection of appropriate maternal strategies reacting to an infant’s distress.
The higher activity in the caudate nucleus is involved in motor programming,
suggesting that the initiation of emotion-induced behavior [31, 32] was evoked
when a mother viewed her own infant in distress. The right VLPFC is involved in
decoding facial expressions of emotions [33–35]. Accordingly, its activity suggests
that when a mother views her own infant’s attachment behaviors, i.e., as calling for
his/her mother, the mother recognizes the infant’s emotions based on the infant’s
facial expressions. The DMPFC is involved in making sense of an emotional
experience [36, 37] and in representing the emotions or mental states evoked by
interpersonal interactions [36]. In such conflict situations, the dACC activation is
Fig. 10.3 Lateral and sagittal views of the brain activations when mothers viewed their own
infants in the separation situation versus play situation (right) or the play situation versus
separation situation (left) superimposed on a template structural brain
involved in conflicts; it acts as an alarm that signals the DLPFC, which, in turn, is
related to an executive function. On the other hand, the PCC subserves visual
attention to salient stimuli and is related to memory recollection [38]. Furthermore,
lesions to both the ACC and PCC have been shown to impair maternal behavior in
rats [39, 40]. Therefore, activation of these brain regions may indicate that a mother
is paying attention to her own infant, who demonstrates strong attachment
behaviors; in short, she recognizes her infant’s emotional and mental states evoked
by separation from his/her mother. Additionally, we found activation of the
pSTS/TPJ. Mothers may immediately try to interpret their infants’ distressed states
not only by grasping an infant’s intention but also by attending to the infant’s
emotional states; that is, mothers perceive cues involving the infant’s biological
motion and gaze direction [41–43]. The DLPFC is involved in constructing reap-
praisal strategies that can modulate activity in multiple emotion-related limbic
areas. Furthermore, the DLPFC participates in the conscious experience of emotion,
inhibition of potentially excessive emotion, meaning that it is vital to the process of
monitoring one’s own emotional state in making personally relevant decisions [44].
In our study, the DLPFC activation was associated with a mother’s complex
emotional state when viewing her own infant in SS, as positive emotions such as
love and motherly feelings coexisted with negative ones such as anxiety and worry.
In this complex situation, a mother’s emotional responses to her own infant might
be appropriately regulated as she monitors her own emotional states and inhibits
excessive negative emotional effects so as not to display adverse expressions to the
distressed infant. Our findings that a mother responds more strongly to her own
infant’s crying (SS) than to her child smiling (PS) seem to be biologically mean-
ingful in terms of adaptation to specific demands associated with successful infant
care.
10.4.2 Schematic Model of Maternal Behavior
Based on these findings, a schematic model of maternal behavioral response to an

infant’s distress can be depicted as shown in Fig. 10.4. A mother recognizes and
understands her infant’s mental and emotional states from her own infant’s cues
(DMPFC, VLPFC, pSTS/TPJ). Through this saliency processing (PCC), an alarm
signal (dACC) is conveyed to DLPFC which instantiates an executive function.
Then, decision making and choice of appropriate behaviors are made (dOFC,
DLPFC), and the necessary motor programs are prepared/simulated in the
motor-related regions (caudate nucleus, SMA). All of these neural processes for
protecting mother’s own infant depend on the neural basis of maternal love (OFC,
striatum, insula, PAG) (Fig. 10.2).
Fig. 10.4 A schematic model of maternal behavior when her own infant is in distress is shown. In
this type of situation, a mother recognizes and understands her infant’s mental and emotional states
from her own infant’s cues (DMPFC, VLPFC, pSTS/TPJ). Through this saliency processing
(PCC), an alarm signal (dACC) is conveyed to DLPFC which serves an executive function. Next,
decision making and selection of appropriate behaviors are made (dOFC, DLPFC), and the
necessary motor programs are prepared/simulated in the motor-related regions (caudate nucleus,
SMA). All of these neural processes for protecting mother’s own infant depend on the neural basis
of maternal love (OFC, striatum, insula, PAG)
References
1. Bowlby J.: Child Care and the Growth of Love. London: Pelican (1953)
2. Noriuchi M., Kikuchi Y., Senoo A.: The functional neuroanatomy of maternal love: mother’s
response to infant’s attachment behaviors. Biological Psychiatry 63; 415-23 (2008)
3. Kikuchi Y: The female brain supporting the life of human beings. The Trend of Science: SCJ
(Science Council of Japan) Forum, 6; 94-99 (2012)
4. Kikuchi Y:Neural basis of emotion supporting the humanity - Its adaptive mechanism - .
Japanese Journal of Physiological Anthropology 18; 61-66 (2013)
5. Kikuchi Y.: Neural basis of the humanity. In: Lecture Note Series in Mathematical Sciences
Based on Modeling and Analysis (Hagiwara I, Atomi Y, Kikuchi Y, Otomi K, Naito K, eds.),
135-144, Meiji Institute for Advanced Study of Mathematical Sciences (MIMS), Center for
Mathematical Modeling and Applications (CMMA) (2015)
6. Kikuchi Y.: Neural basis underlying the humanity – towards ‘New Brain Science’ for
understanding human beings. Committee of simulation for heart and brain, Science of Council
of Japan (SCJ), Sep l 7 (2013)
7. Kikuchi Y., Noriuchi M.: The Surprising Reason We Find Baby Cute. The Huffington Post,
TED (Technology Entertainment Design) Weekends, Aug 8 (2013) (http://www.
huffingtonpost.com/madoka-noriuchi/wired-to-prefer-cute-sexy_b_3697845.html)
8. Kikuchi Y., Noriuchi M.: Baby’s little smiles: Building a relationship with Mon. How smiles–
and pouts –are helping researchers probe the essence of the complex mother-infant bond.
Scientific American Mind Matters. Sep 23 (2008) (http://www.scientificamerican.com/article/

smiles-and-mother-baby-bond/)
9. Bowlby J.: Attachment and Loss: Volume 1- Attachment. New York: Basic Books (1969)
10. Bell SM., Ainsworth MD.: Infant crying and maternal responsiveness. Child Dev 4:1171–
1190 (1972)
11. O’Doherty J., Kringelbach ML., Rolls ET., Hornak J., Andrews C.: Abstract reward and
punishment representations in the human orbitofrontal cortex. Nat Neurosci 4:95–102 (2001)
12. Schoenbaum G., Chiba AA., Gallagher M.: Orbitofrontal cortex and basolateral amygdala
encode expected outcomes during learning. Nat Neurosci 1:155–159 (1998)
13. Cavada C., Company T., Tejedor J., Cruz-Rizzolo RJ., Reinoso-Suarez F.: The anatomical
connections of the macaque monkey orbitofrontal cortex. A review. Cereb Cortex 10:220 –242
(2000)
14. Jenkins JS., Ang VT., Hawthorn J., Rossor MN., Iversen LL.: Vasopressin, oxytocin and
neurophysins in the human brain and spinal cord. Brain Res 291:111–117 (1984)
15. Lonstein JS., Stern JM.: Site and behavioral specificity of periaqueductal gray lesions on
postpartum sexual, maternal, and aggressive behaviors in rats. Brain Res 804:21–35 (1998)
16. Miranda-Paiva CM., Ribeiro-Barbosa ER., Canteras NS., Felicio LF.: A role for the
periaqueductal grey in opioidergic inhibition of maternal behaviour. Eur J Neurosci 18:667–
674 (2003)
17. Lund, I., Yu, LC., Uvnas-Moberg, K., Wang, J., Yu, C., Kurosawa, M., Agren, G., Rosen, A.,
Lekman, M., Lundeberg, T.: Repeated massage-like stimulation induces long-term effects on
nociception: contribution of oxytocinergic mechanisms. Eur. J. Neurosci. 16 (2), 330– 338
(2002)
18. Olausson H., Lamarre Y., Backlund H., Morin C., Wallin BG., Starck G., et al.: Unmyelinated
tactile afferents signal touch and project to insular cortex. Nat Neurosci 5:900 –904 (2002)
19. Berlucchi G., Aglioti SM.: The body in the brain revisited. Exp Brain Res 200: 25–35 (2010)
20. Craig AD.: How do you feel- now? The anterior insula and human awareness. Nat Rev
Neurosci 10: 59–70 (2009)
21. Craig AD.: How do you feel? Interoception: the sense of the physiological condition of the
body. Nat Neurosci 3 (2002)
22. Oppenheimer S.: Cerebrogenic cardiac arrhythmias: cortical lateralization and clinical
significance. Clin Auton Res 16: 6–11 (2006)
23. Zhang ZH., Oppenheimer S.: Insular cortex lesions alter baroreceptor sensitivity in the
urethaneanesthetized rat. Brain Res 813: 73–81 (1998)
24. Zhang ZH., Oppenheimer S.: Electrophysiological evidence for reciprocal insulo-insular
connectivity of baroreceptor-related neurons. Brain Res 863: 25–41 (2000)
25. Haber SN.: The primate basal ganglia: Parallel and integrative networks. J Chem Neuroanat
26:317–330 (2003)
26. Elliott R., Newman JL., Longe OA., Deakin JF.: Differential response patterns in the striatum
and orbitofrontal cortex to financial reward in humans: A parametric functional magnetic
resonance imaging study. J Neurosci 23:303–307 (2003)
27. Schultz W.: Multiple reward signals in the brain. Nat Rev Neurosci 1:199 –207 (2000)
28. Breiter HC., Rosen BR.: Functional magnetic resonance imaging of brain reward circuitry in
the human. Ann N Y Acad Sci 877:523–547 (1999)
29. Karama S., Lecours AR., Leroux JM., Bourgouin P., Beaudoin G., Joubert S., et al.: Areas of
brain activation in males and females during viewing of erotic film excerpts. Hum Brain Mapp
16:1–13 (2002)
30. O’Doherty J., Critchley H., Deichmann R., Dolan RJ.: Dissociating valence of outcome from
behavioral control in human orbital and ventral prefrontal cortices. J Neurosci 23:7931–7939
(2003)
31. Aldridge JW., Thompson JF., Gilman S.: Unilateral striatal lesions in the cat disrupt
well-learned motor plans in a GO/NO-GO reaching task. Exp Brain Res 113:379 –393 (1997)
32. Jaeger D., Gilman S., Aldrige JW.: Neuronal activity in the striatum and pallidum of primates
related to the execution of externally cued reaching movements. Brain Res 694:111–127
(1995)
33. Adolphs R.: Neural systems for recognizing emotion. Curr Opin Neurobiol 12:169 –177
(2002)
34. Nakamura K., Kawashima R., Ito K, Sugiura M., Kato T., Nakamura A., et al.: Activation of
the right inferior frontal cortex during assessment of facial emotion. J Neurophysiol 82:1610 –
1614 (1999)
35. Hornak J., Rolls ET., Wade D.: Face and voice expression identification in patients with
emotional and behavioral changes following ventral frontal lobe damage. Neuropsychologia
34:247–261 (1996)
36. Ochsner KN., Bunge SA., Gross JJ., Gabrieli JD.: Rethinking feelings: An fMRI study of the
cognitive regulation of emotion. J Cogn Neurosci 14:1215–1229 (2002)
37. Lane RD., Fink GR., Chau PM., Dolan RJ.: Neural activation during selective attention to
subjective emotional responses. Neuroreport 8:3969 –3972 (1997)
38. Maddock RJ.: The retrosplenial cortex and emotion: New insights from functional
neuroimaging of the human brain. Trends Neurosci 22: 310–316 (1999)
39. MacLean PD.: The Triune Brain in Evolution (Role in Paleocerebral Functions). New York:
Plenum Press (1990)
40. Slotnick BM.: Disturbances of maternal behavior in the rat following lesions of the cingulate
cortex. Behaviour 29: 204 –236 (1967)
41. Allison T., Puce A., McCarthy G.: Social perception from visual cues: Role of the STS region.
Trends Cogn Sci 4:267–278 (2000)
42. Gallagher HL., Frith CD.: Functional imaging of ‘theory of mind’. Trends Cogn Sci 7:77– 83
(2003)
43. Wicker B., Michel F., Henaff M-A., Decety J.: Brain regions involved in the perception of
gaze: A PET study. Neuroimage 8:221–227 (1998)
44. Reiman EM., Lane RD., Ahern GL., Schwartz GE., Davidson RJ., Friston KJ., et al.:
Neuroanatomical correlates of externally and internally generated human emotion. Am J
Psychiatry 154:918 –925 (1997)
Chapter 11
Expectation Effect Theory and Its
Modeling
Hideyoshi Yanagisawa
Abstract A discrepancy between prior expectation and posterior experience

evokes emotions, such as surprise, satisfaction, and disappointment, affecting the
perceived product and service value. Furthermore, expectation affects perceived
experience. This psychological phenomenon, called the expectation effect, is a key
to designing the affective experience of a product and a service. Experimental
findings of this effect exist in a variety of disciplines. In this chapter, the author
presents computational models of the expectation effect using information theory
and neural coding principles. These models estimate its occurrence conditions, its
intensity, and two patterns of the expectation effect, i.e., contrast and assimilation.
The author discusses an essential mechanism of human perceptions involving prior
expectations based on simulation results of the models.
11.1 Expectation Effect
Thomas Fuller said “Good is not good when better is expected.” Similarly,
expectation disconfirmation is an important factor when designing a satisfactory
product and service. Marketing studies suggest that a customer’s satisfaction with a
product is influenced by the disconfirmation between prior expectation and per-
ceived quality as well as by the quality itself [11, 12, 14] (double arrow in
Fig. 11.1). Expectation confirmation is an appraisal component that affects emo-
tions such as contentment, satisfaction, disappointment, and dissatisfaction [4, 10].
Expectation disconfirmation surprises people and induces emotions that affect the
overall liking of a product [9].
Expectation itself often affects the expected experience (dashed arrow in
Fig. 11.1). This effect, known as the expectation effect, has been observed in
multiple disciplines and different cognitive processes such as emotion and
H. Yanagisawa (&)
The University of Tokyo, 7-3-1, Hongo, Bunkyo, Tokyo, Japan
e-mail: hide@mech.t.u-tokyo.ac.jp

DOI 10.1007/978-3-319-29433-9_11
200 H. Yanagisawa
Expectation
Satisfaction
Disappointment
Expectation Disconfirmation Emotion

effect
Artifacts Experience
Quality
Performance
Fig. 11.1 Expectation disconfirmation and expectation effect
perception [17]. The expectation effect changes the disconfirmation between

expectation and experience. Thus, the expectation effect is an essential factor to
ensure the satisfactory products and services.
11.2 Expectations in Sensory Modality Transitions
In a time sequence of user experience (UX) of a product, users shift from one
sensory state to another in cyclic interactions involving action, sensation, and
meaning [8]. We expect that users would predict subsequent states between such
transitions of state (Fig. 11.2). For example, we expect a meal to taste a certain way
based on how it looks, the weight of a product before lifting it, the usability of a
mouse by looking at it, etc. This prior prediction affects posterior perception, that is,
the expectation effect.
The size–weight illusion is one example of the expectation effect, in which
people perceive a smaller object as heavier than a bigger one when the weights of
the two objects are identical [6]. In this situation, people expect a bigger object to be
Expect Expect
Appraisal Tactile Sound
Appearance
impression impression
Sense Vision Touch Audition
Action Look Grasp Hit
Fig. 11.2 Expectations in sensory modality transitions

11 Expectation Effect Theory and Its Modeling 201
heavier than a smaller one, but perceive the opposite, even though the weight is
actually the same. In other words, the disconfirmation between visual prediction
and weight perception works as an expectation effect. Such weight illusion occurs
with different materials and surface textures. The present author found that visual
expectation changes tactile perceptions of surface texture [17]. In food science,
researchers investigated the effects of visual expectations with regard to food and its
actual taste [3].
11.3 Information, Surprise, and Uncertainty
According to information theory [13], the amount of information gained from an

event can be quantified using the prior probability of the event. An unexpected
event provides us with a greater amount of information than an expected one. The
information gain between prior and posterior beliefs represents a measure of
“surprise,” and a certain degree of surprise attracts sensory attention [7]. Hence,
information gain between prior expectation and posterior experience affects the
intensity of the expectation effect.
Furthermore, the amount of expected information or entropy of prior belief is a
condition that determines the occurrence of the expectation effect. In a situation of
high entropy, in which future events are unpredictable, more weight should be
given to bottom-up sensory input than to top-down prediction [15]. Thus, we
predicted that the expectation effect occurs when the entropy of prior beliefs is less
than a certain value.
Now, we consider a transition from prior event y 2 V to posterior event x 2 U,
where U and V are sets of events on posterior and prior states, respectively. Here,
event represents the state of one’s experience with an artifact. We assume one
expects posterior event x through prior event y. The error of expectation is defined
as the deviation of expected experience sy from perceived experience sx (without
prior expectations) as Dsxy ¼ sx sy .
In a transition from expectation to experience, one experiences x with prior
expectation sx. Thus, the expectation effect exy is defined as exy ¼ sxy sx where sxy
is the perceived experience of x with prior expectation y.
In the prior state of a transition, we assume one has a belief distribution that can
be represented as subjective probability distribution PX for a random variable of
posterior event X. The subjective probability of a posterior event x, Px, represents
the difference between prior expectation and posterior experience. For example, if
Px is close to 1.0, one experiences an event x with low expectation disconfirmation.
On the other hand, if Px is small, one does not believe that event x occurs and
experiences a large expectation disconfirmation with surprise. Information theory
202 H. Yanagisawa
Px
Gained information
P2
I x = – log Px
(surprise!)
I1
P1
I2
X Px
x1 x2 P1 P2
Fig. 11.3 Gained information is proportional to surprise
suggests that one gains information amount Ix after experiencing an event x in

posterior state as follows:
Ix ¼ log Px ð11:1Þ
It is known that the amount of gained information Ix in such state transitions

represents a degree of “surprise” induced by experiencing a posterior event
(Fig. 11.3). Itti and Baldi [7] verified that information gain, which is a generalized
Ix, corresponds to human attention in an experiment on gaze attention. We
hypothesized that Ix positively affects the intensity of the expectation effect [18].
Furthermore, the expected uncertainty of an event is an important factor for
balancing sensory information and prior expectation during perceptual synthesis
[15]. The expected uncertainty of events X can be measured using entropy HX as
follows:
X
HX ¼ Px log Px ¼ E ðIx Þ ð11:2Þ
x2U
where E(a) is an expected value of a and HX represents an amount of expected

information or expected uncertainty, such that expectation requires a contextual
situation where HX is less than a certain value (Fig. 11.4).
From above discussion, we hypothesized that the expectation effect ex occurs
under certain level of entropy HX and the degree of the effect depends on the
amount of gained information Ix as follows:
jex j ¼ f ðIx Þ if HX [ hH ; jex j ; 0 if HX \hH ð11:3Þ
where f(x) represents a positive function and hH is a threshold value of entropy. In

[18], an experimental result of the visual expectation effect of tactile texture sup-
ported the hypothesis.
px px
Low entropy meaning certain expectation High entropy meaning uncertain expectation
Top-down(expectation) is dominant Bottom-up(sensory input) is dominant
Fig. 11.4 Entropy represents uncertainty of expectations
11.4 Contrast and Assimilation
Two patterns of expectation effect are commonly observed: contrast and assimi-
lation [17]. As shown in Fig. 11.5, contrast is an effect that magnifies the difference
between prior expectation and posterior experience. Assimilation is an effect that
assimilates posterior experience into prior expectation. It is important to understand
whether the expectation effect is contrasting or assimilating, because they exag-
gerate or diminish the perception of expectation disconfirmation as a factor of
satisfaction, respectively.
Expectation effect
assimilation contrast
Diminished
disconfirmation
Expectation error
Exaggerated disconfirmation
Expectation Experience without

expectation
Fig. 11.5 Contrast and assimilation

204 H. Yanagisawa
11.5 Modeling Perception Involving Expectation Effect
We define perception as an estimation of external physical property, such as the

weight of an object. Sensory stimulus from the external physical world, such as
pressure applied to a hand, is transformed to patterns of neural signals. We call the
neural representation of an external physical variable encoding. Based on the pat-
tern of neural signals, our brain estimates the physical variable. We call this esti-
mation process decoding. We assume that sensory stimuli are encoded as certain
firing rates of neural populations. This type of neural coding is called rate coding.
Based on the firing rate distributions from a sensory stimulus, R, our brain forms the
likelihood function, kðhjRÞ, of a physical variable, h. On the other hand, a physical
property has certain frequency distributions in the world. Human beings learn such
frequency distributions throughout their life. Based on such learned distributions,
human beings predict a physical variable, h, before experiencing sensory stimulus.
For example, in the SWI, people predict the weight of an object by looking at it
before actually lifting it up. Predicted physical variable should follow certain
probability distributions. We define such distribution as prior, PðhÞ. Recent studies
in neuroscience showed that estimation of a physical variable, that is, decoding,
follows the Bayesian estimator, e.g., [2, 5]. Based on Bayes’ theorem, our brain
estimates the distributions of perceptions or posterior, PðhjRÞ, using prior and
likelihood.
kðhjRÞPðhÞ
PðhjRÞ ¼ P ð11:4Þ
X kðhjRÞPðhÞ
Since the denominator of the right-hand side of Eq. (11.4) is a constant for
normalization, the posterior is proportional to the product of prior and likelihood.
PðhjRÞ / kðhjRÞPðhÞ ð11:5Þ
A peak of posterior, hpost , is an estimate of a physical variable. We can define the

expectation effect, e, as the difference between hpost and the maximum likelihood
value of the obtained firing rate, hlik .
e ¼ hpost hlik ð11:6Þ
We define expectation disconfirmation, d, as a difference between a peak of

prior, hpri , and hpost .
d ¼ hpost hpri ð11:7Þ
We call the difference between hpri and hlik prediction error, D. Therefore, the
expectation disconfirmation is a sum of the prediction error and expectation effect.
d ¼ Dþe ð11:8Þ
We can define contrast and assimilation as follows.
Contrast: e[0 if D [ 0; e\0 if D\0 ð11:9Þ
Assimilation: e\0 if D [ 0; e [ 0 if D\0 ð11:10Þ
Equation (11.4) indicates that the Bayesian estimate, hpost , always comes close
to a peak of prior, hpri , and forms a peak of the likelihood estimate of sensory
stimulus, hlik . We call the effect attractive influence of prior. The attractive influ-
ence alone involves assimilation as an expectation effect. The question then arises:
How does contrast occur?
Wei and Stocker [16] proposed a neural encoding framework based on the
efficient coding principal to create a direct link between prior and likelihood.
According to the encoding framework, the Bayesian estimate shifts away from the
peaks of the prior distribution. This phenomenon corresponds to the contrast pattern
of the expectation effect. Efficient coding hypnosis [1] proposes that the tuning
characteristics of a neural population are adapted to the prior distribution of a
sensory variable such that the neural population optimally represents the sensory
variable. In [16], efficient coding defines the shapes of the tuning curves in physical
space by transforming a set of homogeneous neurons using a mapping, F−1, that is,
the inverse of the cumulative of the prior, F.
Zh
F ð hÞ ¼ Pð xÞ dx ð11:11Þ
1
Therefore, the likelihood shape is constrained by the prior distribution, showing

heavier tails on the side of lower prior density. In other words, efficient encoding
typically leads to an asymmetric likelihood function whose mean value is away
from the peak of prior. The Bayesian estimate is determined by a combination of
prior and shifted likelihood means, and it shifts away from the prior peak. We apply
this efficient encoding to explain contrast in our model. Figure 11.6 shows how the
Bayesian estimate (perceived value), hpost , shifts from a peak of the asymmetric
likelihood function away from a peak of prior. We call the perceptual shift re-
pulsion influence. The repulsion influence increases as the distance between prior
distribution and peak of likelihood, that is, prediction error, D, increases, because
the extent of asymmetry of likelihood increases away from peak of prior.
Figure 11.7 summarizes our hypothetical model of perception. Based on the
efficient encoding principle, prior changes the shape of the likelihood function
asymmetry while encoding the sensory stimulus of the physical variable, h, as a
206 H. Yanagisawa
Fig. 11.6 Contrast effect Contrast

caused by asymmetric
likelihood function based on
efficient coding
Prior
Likelihood
Estimate Mean of likelihood
firing rate of the neuron population, R. The Bayesian decoder integrates the prior
distribution, PðhÞ, and asymmetric likelihood function, kðhjRÞ, and forms posterior
distributions. As a result, we perceive a peak of the posterior as an estimate of the
physical variable, that is, perception.
Expectation
Bayesian decoding
Prior Prior integrates prior and likelihood
information p( )
Perception
Efficient coding
shapes likelihood Posterior
asymmetry p( | R)
Estimates
Likelihood
Stimulus
( | R)
Sensation
Physical variable
Fig. 11.7 Model of human perception with prior expectation

11.6 Three Factors of Expectation Effect
Repulsion influence increases as the prediction error increases, due to asymmetry of

the likelihood function. Repulsion influence involves contrast. Thus, the prediction
error is a factor that decides a condition of the expectation effect.
We assume two more factors of the expectation effect: external noise and
uncertainty. The shape of the likelihood function is affected by the noise of the
external stimulus. An external noise modifies the shape of the likelihood function
by convolving it with noise distributions. Symmetric external noise distributions do
not change the mean of likelihood, but they increase its overall width. Thus, the
attractive influence of prior relatively increases, and the Bayesian estimate, hpost ,
shifts toward the peak of prior. If the attractive influence of prior exceeds the
repulsion influence of asymmetric likelihood, the expectation effect may change
into assimilation from contrast.
Variations of prior distributions are indicators of prediction uncertainty. The
variation in prior impacts the attractive influence. In the Bayesian estimation, a
small variation in prior means certain prediction and involves a strong attractive
influence. Conversely, a big variation in prior means uncertain prediction and
involves weak attractive influence. Thus, we defined the expectation effect, e, as a
function of three factors: prediction error, D; variation of prior (uncertainty), r2pri ;
and variation of external noise, r2noise .

e ¼ f D; r2pri ; r2noise ð11:12Þ
11.7 Computer Simulations of Expectation Effect
Using the equation for expectation effect, we conducted a computer simulation to

investigate the effects of the three above-mentioned factors on the expectation
effect. We focused on the conditions of contrast and assimilation and the extent of
the expectation effect. We used normal distributions for prior, homogeneous like-
lihood, and posterior. We choose the following as conditions of the simulation
parameters: prediction error of 100 steps; uncertainty, r2pri , of ten steps within [50,
200]; and external noise, r2noise , of ten steps within [5, 50]. The standard deviation of
homogeneous likelihood was set as 0.04. We calculated the expectation effect using
the function (11.12) for all combinations of the above-mentioned conditions for the
three factors.
Figure 11.8 shows an example of the simulation result of the expectation effect
as a function of the expectation error. Each line represents a condition of uncer-
tainty (small: 80, big: 90) and external noise (small: 15, big: 20). A positive value
represents contrast, and a negative value, assimilation. Figure 11.8 revealed three
findings.
208 H. Yanagisawa
2
Big uncertainty
Smallnoise
1.5
Big uncertainty
Expectation effect
1 Big noise
Small uncertainty
Small noise
0.5 contrast
-0.5 Small uncertainty

Big noise assimilation
-1
0 20 40 60 80 100 120
Expectation error
Fig. 11.8 Simulation result of expectation effect as a function of expectation error for different
conditions of expectation uncertainty and external noise
1. The expectation effect functions as an assimilating effect when the expectation

error is small. As the expectation error increases, the expectation effect increases
and changes to the contrasting condition. Around the peak of prior, where the
prediction error is small, the shape of the likelihood function was close to
symmetric, the repulsion influence was small, and the attraction influence of
prior is dominant. Thus, assimilation occurred. As the prediction error increases,
the extent of the likelihood asymmetry increases, and the repulsion influence
increases. Thus, the expectation effect shifts to the contrast condition.
2. The extent of the expectation effect, jej, is bigger when uncertainty is lower for
both assimilation and contrast. With respect to assimilation, the attractive
influence of prior increases in the Bayesian estimation as the variation of prior
(uncertainty) decreases. On the other hand, the repulsive influence increases
from a certain value of prediction error as the variation of prior decreases. In
other words, certain predictions involve a sharp expectation effect regardless of
the condition (contrast or assimilation).
3. The prediction error at which assimilation changes to contrast increases as the
external noise increases. External noise weakens the repulsive influence. In the
Bayes’ estimation, the attractive influence of prior becomes stronger than the
repulsive influence of likelihood. Thus, the area of assimilation in the prediction
error increases when the external noise exceeds prediction error and uncertainty.
We observed the above-mentioned trends for all possible combinations of
conditions for uncertainty and external noise. We also observed special cases
wherein patterns of only contrast and only assimilation occur. Figure 11.9 shows
contours of prediction errors when assimilation changes to contrast for all
200 1.0
0.9
0.8
0.7
150
0.6
Uncertainty
0.5
0.4
100
0.3
0.2
0.1
50 0
5 10 15 20 25 30 35 40 45 50
External noise
Fig. 11.9 Contours of expectation errors when assimilation shifts to contrast for different
conditions of uncertainty and external noise
combinations of uncertainty and external noise. The prediction error, the z-axis, is
normalized between zero and one. Zero of the contour represents a case where only
contrast occurs, whereas one of the contours represents a case where only assim-
ilation occurs. As shown in Fig. 11.9, the area where uncertainty is high and
external noise is small denotes cases where only contrast occurs. In this area, the
repulsive influence of asymmetry likelihood is dominant compared to the attractive
influence of uncertain prediction. On the other hand, the area with low uncertainty
and big external noise shows only assimilation. The attractive influence of prior is
dominant for certain predictions compared to the repulsive influence, which is
weakened by the external noise.
11.7.1 Neural Mechanism of Expectation Effect and Its

Ecological Meanings
The result of the computer simulation (Fig. 11.8) showed that prediction error
affected the extent of the expectation effect and worked as a factor of either the
assimilation or the contrast condition. The pattern of expectation effect shifted from
assimilation to contrast as the prediction error increased. The prediction error
increases the likelihood repulsive influence against prior attractive influence during
Bayesian estimation (decoding). We discuss the meaning of the psychological
210 H. Yanagisawa
phenomenon from an ecological viewpoint. Contrast exaggerates expectation

disconfirmation so that human beings pay attention to novel stimuli with surprise
[7] and try to gain information from unexpected phenomena. This ecological
function may provide an opportunity to learn novel information and renew prior
knowledge, that is, prior distributions. However, due to limitations of cognitive
resources, such as short-term memory content and energy, human beings cannot
pay attention to each unexpected phenomenon. Assimilation may work as a filter to
select which unexpected phenomena should be paid attention to. In other words,
human beings ignore marginal prediction error. This ecological function is rea-
sonable in that it saves the energy resources of the human brain.
The simulation results in Fig. 11.8 show that uncertainty decreased the extent of
the expectation effect and external noise increased the assimilation due to the
decreasing repulsive influence during the Bayesian estimation. We can explain
these phenomena with our hypothetical model as follows. Prior distributions of low
variation, namely certain predictions, attracted a Bayesian estimate against the
likelihood function of noisy stimuli when the prediction error and likelihood
asymmetry are small. The repulsive influence decreased as uncertainty and external
noise increased. The contrast weakened with big uncertainty and big noise. Human
beings rely on their prior distributions when the external stimulus is noisy. Certain
prior predictions may increase this dependency, and thus, the extent of assimilation
becomes prominent. On the other hand, human beings should pay attention to big
prediction errors of certain predictions and clear external stimuli. Therefore, con-
trast increased with small uncertainty (certain prediction) and small external noise
(clear stimulus).
Acknowledgements This work was supported by JSPS KAKENHI Grant Number 15K05755,
the Design Innovation (DI) Laboratory at the University of Tokyo (UTokyo) and its corporate
partners. We would like to thank to Professor Tamotsu Murakami, Professor Satoshi Nakagawa,
Dr. Kazutaka Ueda, Mr. Kenji Takatsuji, Mr. Natsu Mikami, and members of the Design
Engineering Laboratory at UTokyo for supporting this project.
References
1. Barlow, H. B. (1961). Possible principles underlying the transformation of sensory messages.

Sensory Communication, 217-234.
2. Brayanov, J. B., & Smith, M. A. (2010). Bayesian and “anti-Bayesian” biases in sensory
integration for action and perception in the size–weight illusion. Journal of Neurophysiology,
103(3), 1518-1531.
3. Deliza, R., & MacFie, H. J. H. (1996). The generation of sensory expectation by external cues
and its effect on sensory perception and hedonic ratings: a review. Journal of Sensory Studies,
11(2), 103-128.
4. Demir, E., Desmet, P., & Hekkert, P. (2009). Appraisal patterns of emotions in human-product
interaction. International Journal of Design, 3(2), 41-51.
5. Ernst, M. O., & Banks, M. S. (2002). Humans integrate visual and haptic information in a
statistically optimal fashion. Nature, 415(6870), 429-433.
6. Flanagan, J. R., & Beltzner, M. A. (2000). Independence of perceptual and sensorimotor

predictions in the size-weight illusion. Nature Neuroscience, 3(7), 737-741. Retrieved from
http://www.ncbi.nlm.nih.gov/pubmed/10862708
7. Itti, L., & Baldi, P. (2009). Bayesian surprise attracts human attention. Vision Research, 49
(10), 1295-1306.
8. Krippendorff, K. (2005). The semantic turn: A new foundation for design: CRC.
9. Ludden, G. D. S., Schifferstein, H. N. J., & Hekkert, P. (2012). Beyond surprise: A
longitudinal study on the experience of visual-tactual incongruities in products. International
Journal of Design, 6(1), 1-10.
10. Murakami, T., Nakagawa, S., & Yanagisawa, H. (2011). Proposal of “Expectlogy” as design
methodology. Paper presented at the International Conference on Engineering Design 2011.
11. Oliver, R. L. (1977). Effect of expectation and disconfirmation on postexposure product
evaluations: An alternative interpretation. Journal of Applied Psychology, 62(4), 480-486.
12. Oliver, R. L. (1980). A cognitive model of the antecedents and consequences of satisfaction
decisions. Journal of Marketing Research, 460-469.
13. Shannon, C. E., Weaver, W., Blahut, R. E., & Hajek, B. (1949). The mathematical theory of
communication (Vol. 117): University of Illinois press Urbana.
14. Spreng, R. A., MacKenzie, S. B., & Olshavsky, R. W. (1996). A reexamination of the
determinants of consumer satisfaction. The Journal of Marketing, 15-32.
15. Strange, B. A., Duggins, A., Penny, W., Dolan, R. J., & Friston, K. J. (2005). Information
theory, novelty and hippocampal responses: unpredicted or unpredictable? Neural Networks,
18(3), 225-230. doi:http://dx.doi.org/10.1016/j.neunet.2004.12.004
16. Wei, X.-X., & Stocker, A. (2012). Bayesian Inference with Efficient Neural Population Codes.
In A. P. Villa, W. Duch, P. Érdi, F. Masulli, & G. Palm (Eds.), Artificial Neural Networks and
Machine Learning – ICANN 2012 (Vol. 7552, pp. 523-530): Springer Berlin Heidelberg.
17. Yanagisawa, H., & Takatsuji, K. (2015a). Effects of visual expectation on perceived tactile
perception: An evaluation method of surface texture with expectation effect. International
Journal of Design, 9(1), 39-51.
18. Yanagisawa, H., & Takatsuji, K. (2015b). Expectation effect of perceptual experience in
sensory modality transitions: modeling with information theory. Journal of Intelligent
Manufacturing, 1-10. doi:10.1007/s10845-015-1096-7
Chapter 12
Ma: Exploration of the Method
for Grasping Intangible Emptiness
Masato Hirahatake, Nobuyuki Kobayashi and Maeno Takashi
Abstract To adapt to rapid social changes in modern society, new communication

forms and methods must be made with regard not only to logic but also to emotion.
In particular in the design field, in which emotions and sensitivity enhance the
impression of the intended recipients, designers are beginning to evoke the power
of “emptiness,” which attracts attention and stirs the imagination. In Japanese
society, “emptiness” is known as the concept of ma. Ma has had a lasting signif-
icance in Japanese society. However, this concept remains difficult to grasp, and
consequently, it is not used efficiently in communication. Therefore, to promote the
active use of ma, the methods for understanding the diverse uses of ma merit
analysis. This study develops an analytical method to approach ma, which is hoped
to engender a stronger grasp of the type, scope, and way of using ma in certain
circumstances. Using this method, this research analyzed ma used in non-language
vocal storytelling in US and Japan TED videos as a type of communication. The
results reveal differences in the uses of ma and related trends with respect to
storytelling.
12.1 Introduction
In recent years, emotional and logical capacities are demanded to cope with
ever-complicating social environmental changes. In a closed society with a pre-
dictable direction of societal progress, processes follow a logical structure; for
M. Hirahatake (&)
The System Design and Management Research Institute,
Graduate School of System Design and Management, Keio University,
Yokohama, Japan
e-mail: masato.hirahatake@gmail.com
N. Kobayashi M. Takashi
Graduate School of System Design and Management, Keio University,
Yokohama, Japan
e-mail: n-kobayashi@kato-works.co.jp

DOI 10.1007/978-3-319-29433-9_12
214 M. Hirahatake et al.
instance, products are transmitted from the producers to the consumers in a linear
and automatic manner. However, in today’s open society, predicting the future is
more challenging, and in the same vein, the relations between producers and
consumers have become more interactive, thus requiring higher emotional sensi-
tivity in communication [1]. In such a background, design has attracted attention.
Design does not only consider the form of objects, but also refer to creating
human environment; as such, design is a human action as an essential act of
communication [2]. In this regard, producers must offer experiences that relate to
users’ emotions, and these experiences must be viewed by users as special.
Therefore, producers are compelled to design both the form and method of com-
munication that would effectively deliver to consumers special impressions to
which they are not exposed yet.
Given this context, the concept of emptiness is receiving warranted attention. At
the basic level, emptiness is insubstantial, referring to silence and white space.
However, it can be understood as being, regardless of the nothing. Although
emptiness is nothing, it can bear special meaning in various situations. In fact,
emptiness at times can be more powerful than words, as in the adage “Eloquence is
silver, but silent is golden.” Hara, a renowned Japanese designer, said that although
emptiness is nothing, it is contrary to “not being.” Indeed, emptiness provides a
space within which imaginations can roam free, vastly enriching powers of per-
ception and mutual comprehension. Emptiness has such potentials [3]. Clearly,
emptiness has many meanings and possesses the power to inspire impressions. For
designers, emptiness is interpreted actively for its special meanings.
Japanese society has considered the concept of emptiness, encapsulated in the
term ma, whose origin is assumed to date back to around the thirteenth century [4].
It is used to refer to silence, whitespace, and space and has been utilized in
Japanese daily life and traditional culture. Nonetheless, the concept of ma remains
poorly understood because of the contrast in its characteristics of being insub-
stantial, extremely vague, and complexity. These challenges impede the easy dis-
cussion of ma [5].
Thus, this chapter will provide an outline of the concept of ma to shed light on it.
The current research has developed an analysis method to grasp how ma is used in
specific situations. Section 12.1 presents an introduction of the topic, and Sect. 12.2
offers definitions that clarify the concept of ma. A classification of ma is offered
based on the discussion in Sects. 12.2 and 12.3. The analysis method developed in
this study is introduced in Sect. 12.4, which will focus on ma as silence or pause in
communication. In Sect. 12.5, the use of ma in communication is analyzed, par-
ticularly its use in the storytelling of the US and Japan TED videos. The final section
sums up the chapter and highlights future prospects related to the study of ma.
12 Ma: Exploration of the Method for Grasping Intangible Emptiness 215
12.2 Defining ma
Simple nothing but not emptiness, ma has been challenging to define clearly. In the
present attempt, three viewpoints are considered. The first viewpoint is on deter-
mining where ma is found and what it is. In answering these questions, ma can be
understood, including its characteristic of being extremely vague.
The second viewpoint focuses on how ancient people discovered simple
emptiness and why they came to use emptiness. In this regard, the intellectual
tradition of philosophy will be overlooked in favor of culture studies [6], in which
ma has been considered [7].
The final viewpoint endeavors to define ma with consideration for its evolution,
which will be linked to the analysis method developed in this study.
12.2.1 Ma in Japanese Society
According to Minami, ma is found in three domains of Japanese life, that is,

ordinary life, martial arts, and arts in Japan [8]. Therefore, this section will discuss
the types of ma that can be observed in these three domains, including their roots in
Japanese history.
12.2.1.1 Ma in Japanese Daily Life
The reputable Japanese dictionary Koujien [9] lists eight explanations for ma. First,
it is an interval between two physical objects. Second, it is a unit of length. Third, it
is a space that is divided by a folding screen or a fusuma, which is a traditional
Japanese partition. Fourth, it is an interval in time, referring to pauses in the
production of rhythm in Japanese music or Japanese dancing, and includes the
meaning of the overall sense of rhythm. Fifth, ma is a moment of silence inserted
between lines to leave a lingering sound in a play. Sixth, it refers to a moderate
time, an opportunity, and a chance. Seventh, it indicates the momentary status.
Lastly, ma also refers to the place where ships are anchored or a port.
Apart from its dictionary meanings, ma is integrated into over 200 kanji terms.
Examples include sukima, which means spaces between physical objects, such as
between tables and chairs; yukima, which is a pause in snowfall; and kumoma,
which means between clouds and similar natural phenomena.
Further, ma is used in many phrasal idioms. For example, it is in maniau, which
describes a scenario where a person boards a train departing shortly before it
departs, in which ma means timing. In magaii, which means comfortable conver-
sation between people, ma expresses the rhythm and tempo in conversation. These
idioms refer to positive meanings. Meanwhile, in maganukeru, or a conversation
that continues ceaselessly toward one direction, ma is the silence, end, or rhythm
excluded in the never-ending conversation. In magawarui, which is used to

describe a person who talks to people who do not wish to talk, ma indicates the
timing and atmosphere of a situation.
As mentioned before, Japanese society uses various expressions of ma in
everyday scenarios. Ma is thus a temporal, spatial, and unsegmented concept.
12.2.1.2 Ma in Japanese Martial Arts
Ma is also observed in various forms of martial arts. In kendo, a traditional form of

Japanese martial arts began in the seventeenth century, ma appears in maai, which
means distance between opponents. Ma is an important concept as it is crucial to
winning. Kendo teaches the way to adapt to the movement of the opponent, which
includes managing the distance with the enemy in battle [10].
In “Immovable Intelligence,” letters by Takuan, an Edo-era Zen Buddhist monk,
to a famous samurai [11], ma expressions are described in the context of kendo. An
example is “not a hair allowed ma [between],” which means immediate response
toward the enemy in battle. In other words, to attack an enemy, act quickly such that
successive actions have no spaces in between, not even a hair’s breadth.
In another example, “Nor do you feel concerned with the opposition ma
[between] yourself and the enemy, for the latter then takes advantage of you:
Therefore, have no thought even of yourself. To try to be on the alert, to have
your attention keyed up to the highest pitch, this is all well for beginners, but it
will end in your mind being carried away by the sword,” ma can be interpreted
as time.
Thus, ma is used to express interval in space or a moment in time.
12.2.1.3 Ma in Japanese Traditional Arts
Ma is heavily used in the domain of Japanese arts, from the design of Japanese
traditional buildings to photography, music, and entertainment, such as rakugo,
kabuki, nougaku, and modern cinema [8].
In Japanese traditional entertainment rakugo, ma plays a crucial role in eliciting
laughter from the audience. Originating in the nineteenth century in Tokyo and
Osaka, rakugo is a type of storytelling of lighthearted, heartwarming stories set in
the Edo era. The rakugoka, or the comedian or storyteller, acts out one or two parts
in a story using small tools for generating a sense of reality. The rakugoka remains
mostly seated on stage throughout a performance, making the audience laugh with
only his storytelling and facial expressions.
Basho Kingentei, a famous rakugoka, described ma as follows: “Jokes can make
the audience laugh upon the merit of their content.” However, in eliciting laughter,
cues from the storyteller are essential. Thus, the fun is not only in the words
themselves but also in ma, which effectively makes a story more attractive and
interactive [12]. Here, ma refers to silence in the conversation or the atmosphere of

a situation.
Another popular traditional entertainment is noh, which is registered as a cultural
heritage by the UNESCO. Zeami, a noh genius in the fifteenth century, wrote the
book Fushikaden (The Flowering Spirit) [13], which reveals how ma is a secret to
the art of noh. In this book, the meaning of ma is expressed as “a moment not to do
action in performance.” In other words, ma is a stop in movement [7].
These uses comprise only one part of ma in Japan. As mentioned above, ma is
seen as an important ingredient of everyday life, martial arts, or the arts. Its
meanings include silence, time, space, and atmosphere or status. At the same time,
ma is essentially vague because formulating a concrete meaning is difficult, given
its contextual nature. As it is, ma remains tacit knowledge. Whether in communi-
cation or entertainment, ma is judged as either positive or negative depending on
the senses of the speaker.
At this point, a discussion on why the Japanese are interested in ma would be
helpful in outlining the reach and nature of ma.
12.2.2 Ma in Japanese Tradition
How did the ancient Japanese discover ma as emptiness or nothing and then came
to use it widely? This tradition of ma is influenced by Zen. The philosophy of
nothing of Zen affected not only the religious life of the Japanese, but also its arts
and sense of beauty [14]. The section below shall discuss the relationship between
Zen and ma, as well as the present-day characteristics of the intellectual traditions in
the West and the East.
12.2.2.1 Intellectual Tradition in the West and the East
Nonaka explained that philosophy is classified into two, namely the Western and
Eastern intellectual traditions, each bearing unique characteristics: the concept of
existence in the West, referring to the dichotomy of the subjective and objective,
and that of nothing in the East as transcending the dichotomy [6].
According to Nonaka, the Western philosophy of existence mainly consists of
language and reason. In other words, it is based on the mind–body dualism, pro-
posed by Descartes. Existence provides the dichotomy between, for example, mind
and body, object and subject, and mental and physical. Other known philosophers,
such as Hegel and Kant, based their own work on this philosophy.
Meanwhile, the Eastern nothing is a non-language philosophy, relying mostly on
the sensible. In other words, in nothing, the objective and the subjective are not
divided. The primitive state of subjectivity as self is not established in a moment but
unified in chaos, indicating transcendence of dichotomies [6, 15].
Fig. 12.1 Kumoma
A specific example is when a person finds a beautiful scene, such as the break
between clouds shown in Fig. 12.1, while walking. Many Japanese in the past have
stopped in their tracks to take in such a moment, and maybe compose a haiku-poem.
Any person would be moved by such a great scene. In this moment, the objectivity
(self) and subjectivity (scenery being viewed) are not separated in one’s con-
sciousness, or in other words, both are in unity. After the moment of pure appre-
ciation passes, judgment divides the subjectivity and objectivity, when a person
exclaims, “How beautiful is the scenery.” Thus, the unfractionated objective and
subjective mean the transcendence of the dichotomy, a state before judgment where
subjectivity (oneself) and objectivity (scenery) are not separated [16]. In the
Western concept, it is similar to “the concept of the flow” put forward by Professor
Csikszentmihalyi Mihály [15].
Examples of intellectual traditions in the East include the philosophy of Kitaro
Nishida, a famous philosopher. Nishida named the experience of unity between the
objective and the subjective “pure experience.” Nishida is called the father of
Japanese philosophy, having integrated his experiences into his philosophy by
logically translating them into words, with the aim of transcending dichotomy.
Notably, his philosophy focused on Zen [15].
12.2.2.2 Zen and ma
As for Zen, how does it influence Japanese society and Japanese thinking? Zen is a
form of Buddhism originating in Japan, but with roots from India via China,
introduced to Japan in the thirteenth century [14]. Zen is the Japanese equivalent for
Dhyâna, which “represents human effort to reach through meditation zones of
thought beyond the range of verbal expression by contemplation.” The purpose of
meditation is to be convinced of a principle that underlies all phenomena, and, if
Fig. 12.2 Karesansui
possible, of the Absolute itself—in other words, to put oneself in harmony with the
Absolute [17].1
This thought bears the same meaning of the unfractionated objective and sub-
jective. That is, Zen aims to acquire the ideal state of nothing, which is the tran-
scendence of dichotomies: no division between the self and others, subject and
object, and mind and body, only unification.
Zen influences not only Japanese religious life, but the Japanese traditional arts
of gardening, tea ceremony, building construction, and painting, among others.
Daisetsu Suzuki, a famous Buddhist Scholar well known for translating Zen
literature into English, explained the connection between Zen and Japanese culture
concretely. According to Suzuki, a Zen priest was not only a priest, but also an
artist, unlike in other religions, and was encouraged to come into contact with
foreign cultures. As such, Suzuki noted that where “all other schools of Buddhism
have limited their sphere of influence almost to the spiritual life, Zen has gone
beyond it. Zen has internally entered into every phase of the cultural life of the
people.” For instance, Zen influence in the arts has engendered the unique char-
acteristics of Eastern art, compared with Western art.
In Western art, the characteristic concept of beauty is proportion and balance. In
the East, the focus is on imperfection and asymmetry, especially in Japanese art
[18], expressed as extra space or emptiness, which may be viewed in the West as
deficiencies and shortcomings. Figures 12.2, 12.3, and 12.4 depict such examples.
Karesansui, or the traditional garden in Japan, suibokuga, or traditional painting,
and chashitsu, or traditional Japanese building, all present a sense of emptiness, or
in the words of Suzuki, extra space and emptiness as deficiencies.
1
This description of Zen is taken from Nitobe, who was quoting Patrick Lafcadio Hearn.
Fig. 12.3 Suibokuga
Fig. 12.4 Chasitsu
However, the imperfection reference above is not literal. “This imperfection

itself becomes a form of perfection” [14]. That is, through the imagination,
Japanese art transforms from imperfection to perfection. Emptiness as deficiency is
nothing but emptiness. Thus, in Japanese art, human imagination is required to
translate the deficiencies or imperfections. Cha, in Teaizm, shown in Fig. 12.5, is
an example of imperfection, piquing the imagination of those who drink green tea.
Okakura, a famous tea master in Japan, said that “Tea-ism… is essentially a
worship of the Imperfect, as it is a tender attempt to accomplish something possible
in this impossible thing we know as life” [19].
Fig. 12.5 Cha
Suzuki said the essence of Japanese artistry is suggestiveness [20]. Further,

Donald Keene, a researcher of Japanese culture, said that the sense of beauty in
Japanese culture empowers people to comprehend suggestiveness [21].
The concept of nothing in Zen philosophy then gradually changed to ma as a
symbol of emptiness that can be perceived through the senses [7]. Thus, the defi-
ciencies pointed out by Suzuki comprise ma. In this way, ma as the deficiencies in
Japanese art serves to promote suggestiveness, which stimulates the imagination to
perceive perfection in the imperfection.
The terminus of the Zen philosophy is nothing in the Eastern intellectual tra-
dition. It preaches the concept of the unfractionated objective and subjective as
transcending dichotomy. The essence of such a philosophy came to cultivate not
only Japanese spiritual life but also esthetic sensitivity.2 Such sensitivity germinates
the hidden power of seeing invisible things and hearing silent voices. Ma as the
symbol of imperfection in Japanese art relates to suggestiveness, which helps
translate imperfection into perfection.
12.2.2.3 Definition of ma
Gunter Nitschke, a German architect, said that ma is spatial experience, comparable

to “mysterious space born from a result of the external placement of the symbol”
[22]. Meanwhile, for Gamou, a Japanese musician, ma is a concept relating to the
2
Among the many factors that cultivated Japanese sensitivity, this study referred to Zen as a major
factor. Nakamura, a famous philosopher, called the Japanese sensitivity Emotional Naturalism and
argued that the Japanese cultivated sensitivity through enjoying nature [14]. The enjoyment of
nature, according to Suzuki, lacks religious connection and explicit doctrine [20].
dimensions of a performance, as rhythm between times from the perspective of time

distance [23]. Various definitions have been proposed by different experts. Indeed,
ma has been considered from both aspects of space and time for its broadness in
meaning, as discussed above.
As for the view of cultural research, Nishiyama provided the following definition
of ma: “It is a sense of distance that occurs in a section in time and space” [4]. This
definition encompasses both points of time and space and may be considered the
simplest to understand. However, “sense of distance” is subjective, indicating
ambiguity. The present work thus proposes to define ma as a hiatus that occurs in a
section in time and space. Subsequent uses of ma in this study will be based on this
definition.
At this point, various aspects of ma have been presented, from the mystic ma,
rooted in Japanese culture, to ma that is only silence, space, and a white blank. The
presence of ma in various domains has been established as well. Particularly in the
art realm, Zen philosophy has greatly contributed to the perception of the intan-
gible, because it preaches nothing, as is considerably reflected in Japanese art. The
philosophy of Zen nothing cultivates the sensitivity to find meaning in insubstantial
objects, which enriches the imagination when facing only emptiness, or ma.
Ma is a crucial concept in Japanese society, regardless of its ambiguity in
meaning. The special properties of ma thus need to be determined to facilitate the
effective application of this concept.
12.3 Structure Analysis of ma
In analyzing the structure of ma, this work intends to specify the target range of ma.
This concept will be classified into four quadrants, and then the target range can be
identified. Subsequently, the composition of ma for specific situations can be
clarified.
12.3.1 Classification and Range of ma
Two elements affect the pattern of ma. The first is the existence of senders and
receivers. The second is intention. Based on these, ma can be classified into two
axes.
The vertical axis is an axis of the sender’s intentions. This axis is divided into
two: either the sender has an intention or not and whether the sender intentionally or
unintentionally utilizes ma.
The cross-axis refers to the receiver’s intention, which is also divided into two
parts. One relates to the receiver accepting ma from the sender, who is sending ma
intentionally. In other words, the receiver recognizes ma. The other refers to the
Fig. 12.6 Four quadrants of

ma and set target
receiver accepting ma with no intention, which means the receiver only accidentally
recognizes ma. Figure 12.6 illustrates these points.
The upper left is the quadrant in which the sender and the receiver mutually
intend to send and receive ma, respectively; that is, both recognize ma. This
quadrant thus refers to the interaction of ma, mainly observed in communication. Of
course, in communication, the sender and the receiver might not always carry such
intention, but in the present work, the assumption is that they do.
Next, the bottom left is the quadrant relating to the scenario where the sender has
no intention, only the receiver. That is, the receiver accepts ma with intention. This
quadrant is called sensitivity of ma seen in such phenomena as kumoma and
yukima, which inspire the haiku composition.
The upper right quadrant shows the sender having intentions, but not the
receiver. Here, ma serves as a channel, space, or pause. For example, in the design
of a public space, the space between buildings is added to serve as the way through
which people could pass. In this case, the passer-by does not recognize the space as
ma but only as a space with the purpose of providing a path.
The fourth, at the bottom right, is the quadrant for the case where the sender and
receiver both do not have any intention to communicate ma. That is, ma occurs
accidentally in ordinary life, such as ma that occurs “between” passers-by in a
street. They might not mutually recognize ma, but ma exists.
Using the classification of ma into four quadrants, this study aims to help in the
positive use of ma in various scenes. As for the communication of ma, the upper left
quadrant is significant in describing the behavior and interaction between senders
and receivers.
12.3.2 Classification of ma in Communication
In the communication of ma, how can ma be used? To address this question, a

further classification of ma is needed to obtain more concrete descriptions. As such,
situations are presented below to specify how ma is used in communication.
According to Sueda, communication is broadly divided into language and
non-language [24].3 Language is further divided into two types: language and
non-language. Language refers to spoken language, and in this case, ma corre-
sponds to rhythm as well as the rise and fall of the voice tone. Sueda, however,
included these aspects under non-language vocal prosody. This ma is thus marked
N/A in this work. Meanwhile, non-language is divided into two types: literary and
sign language. Literary language is written language, and in this case, ma corre-
sponds to the spaces between lines.
As for non-language, it can be classified roughly into two types: non-language
vocal and non-language non-vocal. The former includes prosodic features and
paralanguage, or ma as rhythm, silence, pause, tone, and timing. Meanwhile, the
latter has six types: surface characteristic, physical contact, physical movement,
smell, space, and time.
Ma as surface characteristic can be seen in the space between clothes. As for
physical contact, ma is the pause when people clench their hands in a handshake.
However, this scenario is not common, and in this study, this point is set as N/A. In
the type physical movement, ma refers to the stopping of movement, as discussed in
Zeami’s Fushikaden (The Flowering Spirit) mentioned previously. As for smell, the
moment without scent in between scents is ma. However, as this is not a general
expression, it is also marked N/A in this work. Ma as non-language non-vocal-type
space is in the distance, literary space, and spatial state, related to ma in karesansui.
As for time, ma refers to the temporal state. An example is the scenario in cinema
where an actor or actress does not do any motion in a scene. Takeshi Kitano, a
famous Japanese movie director and comedian, referred to such a type of ma, a
definition adopted in the present study [25] (Fig. 12.7).
The next section will focus on ma in the non-language vocal realm, namely
silence or pause. According to the view that “speech is silver, but silence is golden”
in communication, then silence as a crucial aspect of communication needs to be
examined. Although ma (other than silence) is also included in the non-language
vocal category, the focus is only on silence to avoid confusion.
12.3.3 Classification of ma in Communication Model
The structure of ma in the non-language vocal category will be described based on

the communication model proposed by James Benjamin. According to Benjamin,
The term “language” used in the study corresponds to Sueda’s term “verbal.”
3
Fig. 12.7 Classification of ma in communication
communication includes nine elements, which are divided into six components and
three activities. The six components are source, message, channel, receiver, noise,
and context; the three activities are decoding, encoding, and feedback [26].
As regards the components, the source is the sender who translates an idea from
a thought into words. In the current work, source is expressed as sender for sim-
plicity. The message is the meaningful conceptualization of the original thought by
the sender. The channel is a medium through which the message is sent. In the case
of oral communication, it corresponds to air. On the other end in the receiver, the
listener receives a message created by the sender. Noise refers to specific types of
interference that hamper the decoding of the receiver in communication. For
instance, speaking voices in a class lecture are a type of noise. Context pertains to
the environment of the participants, encompassing the physical, psychological, and
communication aspects.
As for the three activities, encoding refers to the ability to translate a thought into
words, whereas decoding is the process of translating a message expressed from the
sender’s thought. Feedback arises out of the interaction between a sender and a
receiver.
Fig. 12.8 Ma in the modified communication model (based on J. Benjamin)
In the above communication model, ma is located in the message. Benjamin

stated that a message is divided into cord and language.4 The cord refers to the rules
and constraints of communication, such as grammar, whereas the language is
simply the words (which include language message and non-language message). As
mentioned previously, ma falls under a part of non-language messaging.
Figure 12.8 presents the structural placement of ma.
12.4 Analysis Method for ma
The discussion below will explain the design of the analysis method for ma. The
two axes and four quadrants of ma are referenced in the design.
12.4.1 Vertical Axis of ma: Interpretation
First, the axis of interpretation is divided into two parts: logical and emotional
interpretation. According to Jeremey Donovan, ma has four effects [27]. The pre-
sent work focuses on two characteristic points: the effect of ma to let the receiver
understand what the sender said and the effect of ma to let the receiver have a
dramatic impression. That is, information is understood in the former as logical and
4
According to Benjamin, language is a complicated system of symbols. Symbols are configured on
the basis of rules, called grammar. In the present study, this basis is called code and is set, along
with language, as an element.
reasonable, and in the latter as emotional or sensitive. Therefore, this interpretation

axis is divided into two: logic and emotion.
12.4.2 Cross-Axis of ma
Second, the cross-axis is the assumed axis of key message, also divided into two
parts: One relates to the moment before a person delivers a key message and the
other relates to the moment after the said key message is delivered. To illustrate this
point, Shijaku Katsura, a renowned rakugoka, proposed the “theory of relaxation of
tension.” According to this theory, a primary factor to elicit laughter is the relation
between tension and relaxation that creates ochi as key message, which is used in
the climax of a performance [28]. In this chapter, we do not closely explain to this
theoretical content, but we consider about relation with tension of relaxation.
This theory of tension and relaxations relates to ma in communication. When the
receiver gets ma before getting the key message, he/she is tense and would tend to
focus his/her attention to the sender’s speaking. Meanwhile, when the receiver gets
ma after the key message, he/she will be in a relaxed state while processing the
sender’s message within ma. Therefore, two time points emerge as significant:
before and after the delivery of the key message.
12.4.3 Synthesis of ma
The two axes discussed produce four quadrants. The vertical axis pertains to the
receiver’s logic or emotion-based interpretation while receiving ma from the sender
in communication. The cross-axis refers to the relationship between ma and the key
message: before or after the receiver gets the key message.
The four quadrants of consideration, comprehension, attention, and impression
are illustrated in Fig. 12.9.
First, in the upper left quadrant is consideration, or the time of ma before the
receiver gets the key message, which relates to the receiver’s logical interpretation.
Here, the receiver considers what the sender will say. Consideration consists of
words that serve as a subordinate clause connecting to a main clause. For example,
when a sender says, “Now, I’d like to talk about one story…,” the sender inserts
ma, which enables the receiver to think, What will he say.
Second, the upper right quadrant is comprehension, or the time of ma occurring
after the receiver gets the key message, also relating to the receiver’s logical
interpretation. This moment marks the receiver’s process of comprehending what
the sender said. Comprehension consists of words that serve as the main clause of
the descriptive explanation. For example, if a sender says, “I have come to
understand gradually what he said to me” and follows this with ma, then the
receiver can comprehend what the sender said within this moment of ma. That is, it
Fig. 12.9 Synthesis of ma
lets the receiver sort out the sender’s descriptive explanation in the receiver’s mind.
This comprehension of ma is considered a common situation.
Third, in the bottom left quadrant is attention, or ma occurring before the
receiver gets the key message, intended for the receiver’s emotional interpretation.
This ma focuses on the receiver’s attention to the sender. Attention consists of such
phrases introduced by “and,” “it,” and “then.” For example, if a sender uses such
phrases in the middle of a conversation and then immediately put ma of time, the
receiver’s attention will be piqued unconsciously to the sender’s speaking.
Finally, in the bottom right quadrant is impression, or the time of ma referring to
after the receiver gets the key message, again intended for the receiver’s emotional
interpretation. This ma gives the receiver an impression. Impression is the main
clause with accent, stress of tone, speed, and so on. For example, when the sender
exclaims, “At this moment, finally, I have overcome my past,” the sender gives the
receiver an impressive influence that stirs the receiver’s imagination, accomplished
by the ma of time inserted by the sender immediately after these emotional words
(Fig. 12.9).
12.5 Case Analysis
12.5.1 Developed Approach
This part will present the analysis method of ma. The target of this method is
“TED,” which has attracted increasing attention in recent years. TED is short for
Technology, Education, Design, and in a TEDTalk, a speaker shares an experiential
story within the limited time of 18 min. Using the analysis method, the study will
determine how ma is used in TED held in the USA and Japan. In the analysis, ma is
measured in terms of silence, number of seconds, and number of times, using the
software ELAN.
First, 50 US and Japan TED videos were chosen for a comparative analysis of
how ma is used in these countries. Second, the top 10 most viewed videos were
selected. In the case of US videos, they are referred to as TED Conferences [29] on
the Internet, whereas in the case of Japan, they are TED YouTube videos [30] (as of
July 31, 2015, Japan TED videos are only hosted in YouTube).
Third, the videos were divided into three parts: the beginning, middle, and end.
This division is based on the three-act structure [31]. Subsequently, the use of ma in
these parts was analyzed. In this analysis, the length of ma was counted in terms of
seconds.
The analysis also measured the number of seconds and times when ma was or
exceeded 0.7 s within the 18-min videos. Nakamura, in a study on sensitivity in
information science, reported that the comfortable lengths of pauses or ma are 0.35,
0.7, and 1.4 s, or in doubled values [32]. The present work adopted 0.7 s as the
basic time, as 0.35 s may be misunderstood as breathing, whereas a 1.4-s ma rarely
occurred in the videos.5
12.5.2 Results
12.5.2.1 US Results
The analysis measured the number of seconds and times of ma for the US and
Japan TED videos, and the results below indicate the quantity and percentages.
• Count: Number of seconds of ma in US TED Conference videos (Table 12.1)
“Total number of seconds of ma in each video division”

In total, ma accounted for 375.16, 443.89, and 539.36 s in the beginning, middle,
and end, respectively. The trend shows that ma is more often observed as the video
progressed from beginning to end.
“Total number of seconds for every type of ma”
The most common ma is comprehension, lasting a total of 462.26 s, whereas the
least common is attention at 184.23 s.
“Tendency of changes in the type of ma with respect to video progression”
Comprehension type of ma was the most common type in the beginning (159.24 s)
and middle (162.55 s), but was superseded by impression in the end (209.94 s). In
contrast, attention was the least common throughout (50.75 s in the beginning,
5
In case the audience laughed after an interval of ma, ma was assumed to be 1 s. If ma as silence
lasted 3 s, for example, then it was assumed to be 1 s.
Table 12.1 Tally of the count of the number of seconds of ma in US TED conference videos
Count: number of seconds of ma in US TED conference videos
Beginning Middle End Type/Total
Consideration 69.58 85.68 116.80 272.07
Attention 50.75 61.34 72.14 184.23
Comprehension 159.24 162.55 140.48 462.26
Impression 95.59 134.32 209.94 439.85
Time/Total 375.16 443.89 539.36 1358.41
Fig. 12.10 Count: number of

seconds of ma in US TED
conference videos

times of ma in US TED
conference videos
61.34 s in the middle, and 72.14 s in the end). On the whole, the number of
comprehension decreased with time change, whereas that of consideration, atten-
tion, and impression increased (Figs. 12.10 and 12.11).
Table 12.2 Count: number Count: number of times of ma in US TED conference videos
of times of ma in US TED
conference videos Beginning Middle End Type/Total
Consideration 71 79 97 247
Attention 52 56 55 163
Comprehension 149 139 106 394
Impression 83 103 161 347
Time/Total 355 377 419 1151
• Count: Number of times of ma in US TED Conference videos (Table 12.2)
“Total number of times of ma in each video division”

The count found 355, 377, and 419 uses of ma in the beginning, middle, and end,
respectively. The total number of times of ma use generally increased as the video
progressed.
“Total number of times for every type of ma”
Among all types of ma, comprehension was the most used at 394 times, whereas
attention had the lowest count of 163 times.
In the video divisions, comprehension was the most common ma in the beginning
(149 times) and middle (139 times), superseded by impression in the end (161
times). Meanwhile, attention had the lowest count throughout: 52, 56, and 55 times
in the beginning, middle, and end, respectively. Overall, the number of times
comprehension was used decreased with time change, whereas the count for con-
sideration, attention, and impression increased. This tendency is similar to the
results for number of seconds.
• Percentage: Number of seconds of ma in US TED Conference videos
(Table 12.3)
Table 12.3 Percentage: number of seconds of ma in US TED conference videos

Percentage: number of seconds of ma in US TED conference videos
Beginning (%) Middle (%) End (%) Type/Total (%)
Consideration 18.55 19.30 21.66 20.03
Attention 13.53 13.82 13.37 13.56
Comprehension 42.45 36.62 26.05 34.03
Impression 25.48 30.26 38.92 32.38
Time/Total 100.00 100.00 100.00 100.00
“Total percentage for every type of ma”

Among all types of ma, comprehension scored the highest at 34.03 %, whereas
attention was the lowest at 13.56 %.
“Tendency of changes in the type of ma in percentage according to video
progression”
Comprehension accounted for the largest part of ma in the beginning (42.45 %) and
middle (36.62 %), but not in the end, in which the most common ma was
impression (38.92 %). Attention accounted for the smallest part of ma throughout:
13.53 % in the beginning, 13.82 % in the middle, and 13.37 % in the end
(Fig. 12.12).
• Percentage: Number of times of ma in US TED Conference videos (Table 12.4)
“Total percentage in every type of ma”

Comprehension had the highest percentage of 34.23 %, whereas attention had the
lowest at 14.16 %.
Fig. 12.12 Percentage:

number of seconds of ma in
US TED conference videos
Table 12.4 Percentage: number of times of ma in US TED conference videos

Percentage: number of times of ma in US TED conference videos
Consideration 20.00 20.95 23.15 21.46
Attention 14.65 14.85 13.13 14.16
Comprehension 41.97 36.87 25.30 34.23
Impression 23.38 27.32 38.42 30.15
Time/Total 100.00 100.00 100.00 100.00

number of times of ma in
US TED conference videos
“Tendency for changes in the type of ma in percentage throughout the video”

In the beginning and middle parts, comprehension comprised the bulk of ma at
41.97 and 36.87 %, respectively, overtaken by impression in the end at 38.42 %.
Meanwhile, attention was consistently the least used: 14.65 % in the beginning,
14.85 % in the middle, and 13.13 % in the end (Fig. 12.13).
12.5.2.2 Japan Results
Count: Number of seconds of ma in Japan TED YouTube videos (Table 12.5)

“Total number of seconds of ma in each video division”
In total, ma accounted for 475.27, 490.07, and 486.86 s in the beginning, middle,
and end, respectively.
Table 12.5 Count: number Count: number of seconds of ma in Japan TED YouTube
of seconds of ma in videos
Japan TED YouTube videos
Consideration 94.89 102.16 115.46 312.51
Attention 92.77 96.44 99.43 288.64
Comprehension 187.20 188.69 136.81 512.70
Impression 100.41 102.78 135.16 338.35
Time/Total 475.27 490.07 486.86 1452.20

seconds of ma in Japan TED
YouTube videos
“Total number of seconds for every type of ma”

The most common type of ma is comprehension, clocking a total of 512.70 s,
whereas the least common was attention at 288.64 s.
Throughout the video, comprehension was the most common type of ma (187.20 s
in the beginning, 188.69 s in the middle, and 136.81 s in the end). By contrast,
attention was the least common type (92.77 s in the beginning, 96.44 s in the
middle, and 99.43 s in the end) (Fig. 12.14).
• Count: Number of times of ma in Japan TED YouTube videos (Table 12.6)
“Total number of times of ma in each video division”

The count found that ma was used 368, 391, and 374 times in the beginning,
middle, and end, respectively.
“Total number of times for every type of ma”
Among all types of ma, comprehension was the most used at 380 times, whereas
consideration had the lowest count of 258 times.
Table 12.6 Count: number Count: number of times of ma in Japan TED YouTube videos
of times of ma in Japan TED
YouTube videos
Consideration 76 86 96 258
Attention 86 88 94 268
Comprehension 137 143 100 380
Impression 69 74 84 227
Time/Total 368 391 374 1133

times of ma in Japan TED
YouTube videos

In the video divisions, comprehension was consistently the most common type of
ma, counted 137, 143, and 100 times in the beginning, middle, and end, respec-
tively. Meanwhile, impression had the lowest count throughout 69, 74, and 84 times
in the beginning, middle, and end, respectively (Fig. 12.15).
Of all the ma types, comprehension accounted for the largest part at 35.30 %,
compared with attention, the smallest at 19.88 %.
“Tendency of changes in the type of ma in percentage according to video
progression”
Comprehension comprised the largest part of ma in the beginning (39.39 %) and
middle (38.50 %), but not in the end, in which impression took the lead (28.10 %).
Attention had the lowest percentages throughout 19.52 % in the beginning,
19.68 % in the middle, and 20.42 % in the end. The US and Japan videos showed
significant differences in terms of trends in the number of seconds and times of ma
(Figs. 12.16 and 12.17; Table 12.7).
• Percent: Number of times of ma in Japan TED YouTube videos (Table 12.8)

Comprehension had the highest percentage at 33.54 %, whereas attention the
lowest at 20.04 %.
“Tendency of changes in the type of ma in percentage throughout the video”
In the beginning, middle, and end, ma was mostly comprehension at 37.23, 36.57,
and 26.74 %, respectively. Meanwhile, impression was consistently the least used:
18.75 % in the beginning, 18.93 % in the middle, and 22.46 % in the end.

number of seconds of ma in

number of times of ma in
Table 12.7 Percentage: number of seconds and times of ma in Japan TED YouTube videos
Count: number of seconds of ma in Japan TED conference videos
Consideration 19.97 20.85 23.72 21.52
Attention 19.52 19.68 20.42 19.88
Comprehension 39.39 38.50 28.10 35.30
Impression 21.13 20.97 27.76 23.30
Time/Total 100.00 100.00 100.00 100.00
Table 12.8 Percentage: number of times of ma in Japan TED YouTube videos

Percent: number of times of ma in Japan TED YouTube videos
Consideration 20.65 21.99 25.67 22.77
Attention 23.37 22.51 25.13 23.65
Comprehension 37.23 36.57 26.74 33.54
Impression 18.75 18.93 22.46 20.04
Time/Total 100.00 100.00 100.00 100.00
12.5.2.3 Conclusion of Analysis
The analysis yielded three insights on the use of ma in US and Japan TED videos.
The first is on differences in types of ma used in the video divisions or time
change. In the US videos, comprehension type of ma was the highest in both the
beginning and middle parts, whereas impression gradually rose toward the end. In
the case of Japan, comprehension was consistently high throughout the beginning,
middle, and end. Such a difference between the US and Japan videos may be
attributed to prosody. The criterion of impression is intensity in intonation and
accent. In American TED videos, impression use gradually rose from beginning to
end, which indicates a high number of storytellers skillfully using prosody in their
speech, particularly toward the end of their presentation.
Annette Simmons described the importance of ma (silence), “Silence includes
the power to let people amplify the sensible and emotional elements of the story,”
further noting that silence enables the audience to add meaning or color to a story
when used with proper speed and timing in speech [33]. The US TED speakers may
have intentionally used ma based on this theory.
Meanwhile, in Japan, comprehension maintained high usage throughout the
videos. The speakers inserted ma after descriptive explanations without adopting
any accent. That is, no impressive ma is used in the climax, unlike the case in
US TED.
Second, the results revealed differences between the count of ma in US and
Japan TED videos, summarized in Table 12.9. Although the count of ma times in
Japan TED videos is about half of that in US TED videos, the total number of
seconds is nearly equal. Indeed, overall, Japanese storytellers utilize ma more than
their American counterparts. Particularly, the number of seconds and times of ma in
the beginning and middle in Japan TED videos exceeds that in their US
counterparts.
This point is related to the conclusion of Bekku, a linguist in Japan, regarding
the structural character of the Japanese language: Japanese does not have accents
and intonations, and thus, to imply change in meaning or emphasis, ma is used in
speech [34].
Table 12.9 Comparative numbers for US and Japan TED videos

Type Country Beginning Middle End Type/Total
Consideration America 69.58 85.68 116.80 272.07
Japan 94.89 102.16 115.46 312.51
Attention America 50.75 61.34 72.14 184.23
Japan 92.77 96.44 99.43 288.64
Comprehension America 159.24 162.55 140.48 462.26
Japan 187.20 188.69 136.81 512.70
Impression America 95.59 134.32 209.94 439.85
Japan 100.41 102.78 135.16 338.35
Time/Total America 375.16 443.89 539.36 1358.41
Japan 475.27 490.07 486.86 1452.20
Table 12.10 Comparison of the average in number of seconds of ma between the US and
Japan TED videos
Type Country Beginning Middle End Type/Total
Consideration America 0.98 1.08 1.20 3.27
Japan 1.25 1.19 1.20 3.64
Attention America 0.98 1.10 1.31 3.38
Japan 1.08 1.10 1.06 3.23
Comprehension America 1.07 1.17 1.33 3.56
Japan 1.37 1.32 1.37 4.05
Impression America 1.15 1.30 1.30 3.76
Japan 1.46 1.39 1.61 4.45
Time/Total America 1.06 1.18 1.29 1.18
Japan 1.29 1.25 1.30 1.28
Third, as regards the comparison in the quantity of ma per instance between the
US and Japan TED, Table 12.10 summarizes the average ma per time/incidence.
Two main characteristic differences emerge.
The first difference is in the incidence of attention in the end part for number of
second and times. In the US case, the average of ma (attention) is 1.31 s per times
in the end, whereas in the Japanese case, it is 1.06 s. The figure of the US case is
larger than that in the Japanese case only for the incidence of attention in the end
part.
According to Donovan, one of the effects of ma is to emphasize the word that
follows it [27]. This difference that appears in climax suggests that ma is actively
used.
The second difference is in the incidence of impression in the end part for
number of seconds and times. In the US case, the average of ma (impression) is
1.3 s per time in the end, whereas in the Japanese case, it is 1.61 s. The largest
difference of 0.3 s may again be attributed to Bekku’s theory.
12.6 Conclusion
This study presented a method for analyzing ma use to contribute to the more active
use of this communication function. Using the developed method, differences were
found in the number of seconds and times of ma use, as well as tendency, between
American and Japanese storytelling.
As for future work, three directions may be taken. First, the findings here will
serve as a foundation for continued research on ma, which has been proven to
influence audience impression. In particular in US storytelling, the end part
demonstrates a significant feature. The criterion of impression in this work was
intensity in intonation and accent. As this study focused on the two countries, such
as Japan and the USA, for analysis, future research can examine trends in other
countries, such as France.
Second, future work can delve into the relationship between stories and ma. This
study built a hypothesis on stories for which ma is useful. For example, storytellers
tend to share stories based on their own experience, which, unlike ordinary
fact-based narrative description, draws listeners’ emotions in addition to offering
logical understanding. Future studies may include, with respect to analyzing ma in
stories, the relationship between ma and the three parts of a story: the beginning,
middle, and end.
Third, this study focused on non-language vocal ma, and thus, other aspects of
ma, such as space and white space, will require future attention. These endeavors
can be expected to expand the current understanding of the hidden influence of ma,
which can be translated into practical communication strategies.
References
1. Fukuda, Shuichi. Emotional Engineering Service Developpment. London: Springer, 2011.

2. Okude, Naoto. Desgin Language. Yokohama: Keio University Press, 2002.
3. Hara, Kenya. White. Tokyo: Chuou-kouron-sya, 2008.
4. Nishiyama, Matsunosuke. Reseach of “Ma.” Tokyo: Kodan-sya, 1983.
5. Takeo, Okuno. Structure of “Ma.” Tokyo: Shuei-sya, 1983.
6. Nonaka, Ikujiro. Knowledge Creating Company. Oxford: Oxford University Press, 1995.
7. Fujiwara, Seiichi. “Ma”: Basic Philosophy of Japanese Culture. s.l.: Japan University Press,
2005.
8. Minami, Hiroshi. Research of “Ma.” 1983.
9. Shinmura, Izuru. Koujien. Tokyo: Iwanami-syoten, 2008.
10. Koda, Naohide. Rsearch of martial arts. Tokyo: s.n., 1987.
11. Takuan. Immovable Intelligence. Tokyo: Tachibana-Kyoyo-bunko, 2011.
12. Nagai. Research of “Ma.” Tokyo: Kodan-sya, 1983.
13. Zeami. The Flowering Sprits. Tokyo: International Kodan-sya, 2006.
14. Suzuki, Daisetsu. ZEN and Japanese culture. Tokyo: International of Kodan-sya, 2005.
15. Nonaka, Ikujiro. Methodology of knwoledge Creation. Tokyo: Toyo-keizai-shinpo-sya,
2003.
16. Asami Hiroshi. Phylosophy of Sunshin. Tokyo: Noto Printer, 2004.
17. Nitobe, Inazo. Bushido. Tokyo: International Kodan-sya, 1998.

18. Takahashi, Hironobu. Fundamental research about the Japanese beauty consciousness.
Tokyo: Architectural Institute of Japan, 2004.
19. Okakura, Tenshin. The book of tea. Tokyo: International Kodan-Sya, 1998.
20. Suzuki, Daisetsu. Japanese Spirituality. Tokyo:Iwanami-Bunko, 1972.
21. Keene, Donald. Japanese Sensitivity. Tokyo: Chuou-Bunko, 2008.
22. Berque, Augustin. Vivre l’espace au Japon. s.l.: Chikuma-shoten, 1985.
23. Gamou. Research of “Ma.” Tokyo: Kodan-sya, 1983.
24. Sueda, Kiyoko. Communication study. Tokyo: Syohaku-sya, 2011.
25. Kitano, Takeshi. Structure of “manuke.” Tokyo: Shintyo-sya, Inc., 2014.
26. Benjamin, James B. COMMUNICATION: Concept and Contexts. s.l.: Nihei-sya, 1990.
27. Donovan, Jeremey. How to deliver TED Talk. s.l. : Shincho-sya, 2014.
28. Nomura, Masaaki. linguistics of Rakugo. Tokyo: Kodansya-Gakujyutsu-bunko, 2013.
29. TED. Brows TED talks. [Online] TED Conferences, LLC. [Accessed July 31, 2015.] https://
www.ted.com/talks?language=ja.
30. TED Tokyo. YouTube. [Online] [Accessed July 31, 2015.] https://www.youtube.com/
watch?v=O3CnWavAWzc.
31. Wellman, Wendell. A Writer’s load map. Tokyo: Firm-art, Inc., 2003.
32. Nakamura, Toshie. Information KANSEI of “Ma”. Fukuoka: Japan Society For Fuzzy
Theory and Intelligent Informatics, 2002.
33. Simmons, Annette. The Story Factor. Tokyo: Umi-to-tsuki-Sha, 2001.
34. Bekku, Sadanori. Research of “Ma”. s.l.: Kodan-sya, 1983.

Emotional Engineering Volume 4

Uploaded by

Copyright:

Available Formats

You might also like

Emotional Engineering Volume 4

Uploaded by

Document Information

Copyright

Available Formats

Share this document

Share or Embed Document

Sharing Options

Did you find this document useful?

Is this content inappropriate?

Copyright:

Available Formats

Emotional Engineering Volume 4

Uploaded by

Copyright:

Available Formats

Shuichi Fukuda Editor

ISBN 978-3-319-29432-2 ISBN 978-3-319-29433-9 (eBook)

Library of Congress Control Number: 2012472706

© Springer International Publishing Switzerland 2016

Printed on acid-free paper

This Springer imprint is published by Springer Nature

The progress of network technology is amazing. In addition to humans, things are

1 Age of Smart Orchestration . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1

11 Expectation Effect Theory and Its Modeling . . . . . . . . . . . . . . . . . 199

Abstract We, engineers, have been devoting tremendous efforts to achieve

1.1 Why Humans Make Tools and Are Called Homo

© Springer International Publishing Switzerland 2016 1

1.2 Open World and Emotional Satisfaction

Fig. 1.1 Closed world and

Fig. 1.2 Global optimization

Fig. 1.3 Shewhart cycle

1.3 Unpredictable Changes and Emotional Satisfaction

Fig. 1.4 Changes of

problems of yesterday. Today, environments or situations change with time and

1.4 Fast Adaptability and Trust

Fig. 1.5 Tree

Fig. 1.6 Network

But if we change it to a network structure, any node in a network can be an

1.5 Convergent Engineering and Divergent Engineering

Fig. 1.7 Convergent

Fig. 1.8 Divergent

Fig. 1.9 Divergent nodes

1.6 Needs for Creating Another New Market

D R=R ¼ Constant ð1:1Þ

where R is the level of stimulus

Fig. 1.10 Gompertz curve

1.7 Hardware Development and Software Development

Fig. 1.11 Hardware

Fig. 1.12 Software

1.8 Dream and Expectation

Table 1.1 Dream versus Future Rational? Based on

Therefore, it is more appropriate or realistic to say that engineering is here to

1.9 Expectation Management

The word “expectation management” is often heard in business ﬁeld, especially in

Table 1.2 Expectation management in various sectors

is solved their way. Thus, expectation management in consulting business focuses

1.10 Sympathy and Empathy

Fig. 1.13 Sympathy versus empathy

1.11 Product Value and Process Value

Thus, engineering is now moving from product-centric to process-focused. If we

Fig. 1.14 Maslow’s

Value ¼ Performance=Cost ð1:2Þ

As described earlier, software is developed in a continuous prototyping manner.

1.12 Process-Focused Business

1.13 Performance as Emotional Satisfaction

1.14 Performance in Business: MRO (Maintenance,

The word performance is quickly spreading in business sectors. These perfor-

1.15 Repair and Renovation: Emotional Performance

Abstract Up to now, we have been discussing performance, focusing our attention

In Chap. 1, it is discussed that as things are getting connected, we have to stop

© Springer International Publishing Switzerland 2016 23

Fig. 2.1 Modularization of a

What should be stressed is that modularization in this sense is now going on in

2.4 Finite Element Method (FEM)