Mixed-Initiative Interaction and Robotic Systems

Julie A. Adams and Pramila Rani

Nilanjan Sarkar

Electrical Engineering and Computer Science

Vanderbilt University
Nashville, USA.
[julie.a.adams, pramila.rani@vanderbilt.edu]

Mechanical Engineering
Vanderbilt University
Nashville, USA.
Nilanjan.sarkar@vanderbilt.edu

Abstract
A truly collaborative human-robot interaction framework
should allow the participating agents to assume and
relinquish initiative depending upon their own capabilities
and their understanding of the environment. The goal of this
work is to define and develop a mixed-initiative human
robot collaborative architecture in which affect-based
sensing plays a critical role in initiative switching. Affectbased sensing implies that the robot detects the humans
emotional state in order to determine which actions to
pursue. We have conducted an extensive literature review of
Mixed-Initiative Interaction that has provided a basis for our
architectural development. In particular, we are applying
Rileys (Riley 1989) general model of mixed-initiative
interaction to our architecture development. We have
developed a preliminary architecture and are now collecting
affect-based participant data that will be used to test the
system. The purpose of this paper is to provide an overview
of Rileys model, its application in our development, present
our preliminary architecture, and present the affect-based
interaction constraints that affect the architecture
development.

Introduction
In the foreseeable future, humans will remain a critical
component in robotic systems. The robotics field has not
developed fully autonomous capabilities for real-world
situations. The recent DARPA Grand Challenge is an
excellent example. The Grand Challenge mapped a 149
mile route between California and Nevada that robots were
to autonomously navigate with no human intervention.
Teams were provided with complete route information just
a few hours before the race. The result was that none of the
robots made it through the entire route. Carnegie Mellon
Universitys Red Team made the longest autonomous trek
(7.4 miles) before it ran off course, became stuck, and
caught on fire. The participating teams should be
applauded for their efforts and the advances that they
made, but this race illustrates the need for humans-in-theloop. In the Red Teams situation, a human could have
intervened to inform the robot that it was potentially going
to run off the course. If the robot had run off course, then
the human could have intervened in an attempt to help the
robot understand what went wrong and how to remedy the
situation. Such relationships are not strict teleoperation;
rather collaboration is needed between the human and the
robot. If the human is to be elevated to a true supervisory
position, an environment facilitating such collaborate must

be created. Mixed-initiative interaction (MII) is one

mechanism for creating this relationship.
Our focus is based upon providing MII for affect-based
human-robotic interaction. We have developed an MII
architecture that will be incorporated into an affect-based
robotic system. This architecture has been developed based
upon an extensive literature review of the MII research.
The following section provides a historical MII literature
overview. This is followed by a brief description of the
affect-based system and its applications. We then describe
Rileys (1989) general MII model, our MII architecture
and its development based upon Rileys model. This is
followed by a discussion on Initiative and Affect-based
Sensing. A general discussion is then presented, which is
followed by our conclusions and future work.

A Review of Mixed-Initiative Interaction

One of the first references to the term Mixed-Initiative
(MI) was by Carbonell (Carbonell 1971). Carbonell
associated the term with Computer Assisted Instruction
(CAI) in his SCHOLAR expert system. This system was
designed to maintain a dialogue with students during
instruction. Carbonells objective was to create a CAI
system that would permit students to ask the system
questions thus creating a two-way interaction. He also felt
that in order to attain truly mixed initiative, the system was
required to contain real knowledge so that it was capable
of understanding and answering the students questions.
An important question is: What is Initiative?
Donaldson (Donaldson & Cohen 1998) defined four
theories regarding initiative and the associated applications
scope within a planning domain. Their objective was: to
explore productive synthesis of the complementary
strengths of both humans and machines to build effective
plans more quickly and with greater reliability. The
primary message from this work is that initiative is
associated with control, goals, and conversational turns
while providing insight into interruptions, plan failures,
differences in beliefs, lack of interactivity, and degree of
commitment to problem solving.
Allen (Hearst 1999) felt that mixed initiative might not
necessarily involve a human. He defines mixed initiative
as a flexible interaction strategy where each agent can
contribute to that task that it can do best. The idea is to
allow the agent who knows best to proceed and coordinate
the activities of the other team members.

Horwitz (Horwitz 1999) defines MII in a humancomputer interaction context. He uses the term to refer too:
the methods that explicitly support an efficient, natural
interleaving of contribution by users and automated
services aimed at converging on solutions to problems.
The idea was to develop a shared understanding of the
goals and provide the capability for both entities to solve
the problem in the most appropriate manner.
Ramakrishnan et al. (Ramakrishnan et al. 2002) claim
that true MII cannot occur unless users are able to take
out-of-turn interactions. They have developed a speechbased system that provides such interactions.
We surveyed over thirty articles. As can been observed
from the discussion thus far, there are many interpretations
of MII. In addition to the varied interpretations, this
concept has been applied to many domains including:
Planning (Burnstein et al. 2003, Hearst 1999,
Mitchell 1997),
Agents (DAloisi et al. 1997, Lester et al. 1999,
Rich & Sidner 1998, Skinner Unknown),
Human-machine interaction (Bell et al. 2000,
Fleming & Cohen 1999, Goldman et al. 1997,
Penner et al. 1997), and
Understanding dialogue and discourse (Chu-Carroll
& Brown 1997, Donaldson & Cohen 1997, Guinn
1998, Ishizaki et al. 1999, Lemon et al. 2001).
Kortenkamp et al. (Kortenkamp et al. 1997) introduce
MII for human-robot teams. The MII occurs at the
planning level of the 3T architecture. They list four
advantages to adding MII: Increased flexibility of overall
solutions; more robust system behavior; more tractable
planning; and improved user involvement in planning.
Sherwood et al. (Sherwood et al. 2001) also incorporates
MII at the mission planning stage for a Mars Rover.
Anzai (Anzai 1994) raises the issue of incorporating MII
into human-robot interaction. Anzai states call the
mixture of robot-to-human communication with the one
(communication) from humans to robots as mixed initiative
interaction, Anzais platform is that prior HRI work
has focused on the human specifically communicating to
the robot but not robots communicating to humans. MII
was intended to provide the capability for the robot to
request information from the human.
Horiguchi and Sawaragi (Horiguchi & Sawaragi 2001)
provide MII for a mobile robot via force-feedback through
the joystick. The joystick is the primary interaction
capability to control the robot. They provide a graphical
display of additional information but this display does not
permit the human to command the robot. The strength of
the joystick inputs is employed to determine the humans
initiative level when commanding the robot. This group
(Zheng et al. 2004) recently started to extend their work to
the Urban Search and Rescue domain.
Bruemmer et al. (Bruemmer et al. 2002, 2003)
incorporate MII into a fully teleoperated system for
hazardous environments. In this case, the MII permits the

robot to use initiative to ensure it does not harm itself or

the environment. Their hypothesis is that the MII should
permit a reduction in instrumentation, number of human
operators, human exposure to hazardous materials, and the
overall mission time.
Goodrich et al. (Goodrich et al. 2001) feel that the
primary element in MII is the on-going dialogue between
human and robot in which both parties share responsibility
for mission safety and success. Their work focuses on
providing MII that includes multiple levels of autonomy.
Finally, Murphy et al. (Murphy et al.2000) developed a
MII system for urban search and rescue. Their system
employs an intelligent agent between the human operator
and the robots. The purpose of this intelligent agent is to
inform the human of the presence of potential victims
while providing the operator with perceptual assistance.

Affect-Based Human-Robot Interaction

Over the years, the traditional impression of emotions as
being maladaptive has changed to one of being functional.
The latest scientific findings indicate that emotions play an
essential role in rational decision-making, perception,
learning, and a variety of other cognitive tasks (Picard
1997). Hence it appears that endowing robots with a
degree of emotional intelligence would open the doors to
more meaningful and natural interaction between humans
and robots. The current trend in human-robot interaction
requires explicit communication from the human. A
human communicates with a robot by either typing-in or
speaking explicit instructions. Additional modes such as
eye gaze, head motion, gestures, and electromyogram
signals are also being used to command and control robots.
Such communication modes are generally effective in
many applications; however, a human-robot interaction
(HRI) that relies solely on explicit communication ignores
the potential gain of implicit communication via affectbased HRI. Affect-based HRI requires the detection,
interpretation, and response to the humans emotional
(affective) states by the robot. This can also be referred to
as affective communication since the humans affective or
emotional states are identified and recognized. Given the
crucial role theoretically ascribed to emotions in human
social communication (Scherer 1984, Smith & Lazarus
1990), the gains of endowing HRI with affective
communication could be substantial.
There are numerous potential applications of emotionsensitive robots. A robot that is capable of sensing
emotional states such as panic, fatigue, stress, and
inattention could immediately take meaningful actions to
assist a human during search and rescue operations, fire
fighting, space/remote planet/underwater exploration, and
warfare. Rehabilitation robotics is yet another application
of such versatile robots. Robotic devices for rehabilitation
could employ the emotion-sensing capability to provide
exercise sequences that are comfortable but still

challenging for the patient. A prototype Nursebot (Roy et

al. 2000) was developed to assist the elderly by reminding
them when to take medicine, monitoring for falls, and
providing a video stream to a remote observer.
Emotional sensing could augment the utility of such
healthcare robots. Other applications in this domain might
include industrial robots that can sense worker fatigue on
the shop floor and take necessary precautions to avoid
accidents. The robotic toy industry could also benefit from
such research where a robot that could understand and
respond to a childs emotions thus becoming an
extraordinarily engaging toy. These potential applications
could eventually lead to personal robots that act as
understanding human companions.

Physiological Responses for Emotion Recognition

There exist good evidence that the physiological activity
associated with affective state can be differentiated and
systematically organized (Bradley 2000). Therefore, our
work focuses on using physiological signals to detect
emotions. Emotions are closely associated with the
autonomic nervous system (ANS). ANS regulates the
body's internal environment. It is composed of the
sympathetic branch, which generally functions in
emergency situations, and the parasympathetic branch,
which dominates during periods of relaxation. The
transition from one emotional state to another, for
example, from relaxed to anxious is accompanied by
dynamic shifts in ANS activity indicators. The
physiological signals we examined include various
parameters of cardiovascular activity (Frijda 1986, Cinotti
et al. 2001), including: interbeat interval, relative pulse
volume, pulse transit time, electrodermal activity (tonic
and phasic response from skin conductance (hman et al.
2000)) and electromyogram (EMG) activity (facial activity
including activity of the corrugator supercilii [eyebrow]
and masseter [jaw], (Smith 1989)). These signals were
selected because they can be non-invasively measured and
are relatively resistant to movement artifact. Various signal
processing techniques such as Fourier transform, wavelet
transform, thresholding, and peak detection, were used to
derive the relevant parameters from the physiological
signals. All of these parameters are powerful indicators of
a humans underlying emotional state response. We have
exploited this dependence of a humans physiological
response on emotions to detect and identify affective states
in real-time using advanced signal processing techniques.
A prototypical robotic system based on Brook's
subsumption architecture that can adapt its autonomy level
based on affective sensing has been designed and
implemented by Rani et al (Rani et al. 2004).

General Mixed-Initiative Interaction Model

Riley (Riley 1989) defined a general mixed-initiative
interaction model for human-machine systems (to the best
of our knowledge, ours is the first application to HRI). His
model defines a taxonomy containing two factors that
define the level of intelligence and the level of autonomy.
The combination of the two factors determines the actual
model components that compose the final system. The
taxonomy defines twelve automation levels and seven
intelligence levels. As the levels of intelligence and
autonomy increase a larger number of system components
are included from the general model.
The model is essentially a loop through a structure that
represents the machine (robot), the human operator, and
the world. The machine (robot) and human operator
representations surround the world module and each of the
two larger representations are divided into input and output
modules. A complete system model (highest levels of
automation and intelligence) includes four inference
modules: operator state, operator intent, world state, and
operators knowledge. Such a model would also include
modules for determining information to provide to the
human operator, planning actions and maintaining a selfmodel. At this level of intelligence and automation, the
human essentially becomes a partner with the system with
the ability to provide information, request information, as
well as command and/or control the system.
The autonomy and intelligence levels are determined by
completing a Function Allocation Issues and Tradeoffs
(FAIT) analysis (Riley 1992). The FAIT analysis identifies
potential human factors issues and requirements. The full
analysis conducts a functional decomposition to identify
characteristics specific to the equipment, the human
operators, and the operational environment. The analysis is
composed of two components. The first contains two sets
of questions regarding autonomy and intelligence. The
results of these questions determine the appropriate
autonomy and intelligence levels for the particular system,
thus defining the particular model (modules) that the
system requires. The second component requires that each
system module be analyzed to identify the specific
characteristics that are relevant to system operation.
The autonomy levels include: no autonomy, information
fuser, simple aid, advisor, interactive advisor, adaptive
advisor, servant, assistant, associate, partner, supervisor,
and full autonomy. The intelligence levels include: simply
providing raw data (no intelligence), procedural, context
responsive, personalized, inferred intent responsive,
operator state responsive, and operator predictive.
We have developed a preliminary MII architecture for
affect-based HRI based upon completing the first
component of the FAIT analysis. The second component
of the FAIT analysis is quite involved and time
consuming. Due to on-going data collection necessary for

understanding the affect based sensing, we have not yet

completed the second FAIT analysis component.

Proposed Architecture
Based upon the results of completing the first
component of Rileys FAIT analysis, the resulting system
requires various levels of automation up to and including
full autonomy. As well, the system requires levels of
intelligence from the raw data level up to and including
operator predictive capabilities.

Figure 1. The MII human-robot collaborative architecture.

Figure 1 provides the high level mixed-initiative human

robot collaborative architecture. It can be seen that both
the robot and the human obtain and receive information
from the world. They interact with each other via the HRI.
Each entity (human and robot) performs an inference
based upon all received information to form
representations of the world, itself, and its partner. The
robot and the human dynamically modify the goals and
constraints by interacting via the HRI. Once the goals and
constraints are determined, they are sent to the planning
block where the steps to accomplish the given mission are
outlined and validated with the human and the robot. The
final plan is then passed to the execution block that
executes the steps one by one. The execution of each plan
step changes the world state, as well as the human operator
and the robot states. These changes are transmitted to the
robot and the human operator who accordingly update
their representations.
Figure 2 provides the details that feed into the Operator
State block (from the Robots Inference block in Figure 1).
The figure demonstrates how implicit and explicit
communication is employed to determine the human
operator's physical and mental state. The affect-sensing
block determines the humans implicit state (affective state
prediction using physiological signals) and the HRI along
and world state help determine the explicit operator state.

Figure 2. Operator state module detail.

Figure 3. Affect sensing block.

Figure 3 provides the Affect Sensing block details (from

Figure 2). The humans physiological signals indicate the
humans affective state and are measured using
biofeedback sensors (Cardiac activity, electrodermal
response etc.). The information regarding the humans
affect state from the previous time instant can be used to
determine the current operator state. Therefore the robot is
supplied with the previous operator state for purposes of
context-based reasoning. The block hold signifies the
operation of retaining or holding the current operator state
so that it can be used in the future as a representation of the
previous operator state.

Figure 4. Dynamic goal and constraint change block detail.

Figure 4 provides the block diagram that demonstrates

the process of dynamically changing the goals and
constraints in real time (Dynamically determined Goals
and Constraints block in Figure 1). The human operator
and the robot manipulate the goals and constraints through
the HRI. The HRI is connected to the goals block to
facilitate this interaction. Either the human or the robot
(whoever has the initiative to change the goals and
constraints) modifies the existing goals and constraints to
suit the long term and short-term objectives as required.

to assume and relinquish initiative depending upon their

capabilities, resources, inferences, and interactions with
each other. The architecture should also enable the system
to quickly react to the changing environment by
dynamically changing the goals and constraints. The
planning block with an embedded feasibility analysis
component will enable detailed plan analysis before
execution. The process of forming models of each other
and the world creates the agents knowledge of each
others capabilities and limitations. This is a critical
requirement for initiative switching. Hence, this
architecture should enable us to implement fast and
reliable affect-based as well as standard mixed-initiative
interactions between humans and robots.

Initiative and Affect-based Sensing

Figure 5. Planning and control block detail.

Figure 5 provides the Planning block detail (from Figure

1). Here again the agent who has the initiative/authority to
conduct planning does the planning. If the robot directs the
planning, these activities are communicated to the human
via the HRI. If the human is responsible for the planning,
this is directed via the HRI. The planner uses the current
goals and constraints as well as the last task that was
executed to determine the best plan. The plan that is
generated is cross-validated before execution. This
validation requires sending the plan to the agents via the
HRI. Once the plan is confirmed then it is sequentially
executed and each task is validated before execution.
It should be noted that the MII starts at the HRI level
where either the human or robot assumes initiative to
dynamically determine the mission goals and constraints.
For instance, if the task is to repair part of a workstation,
the goals (what to repair, what the repaired performance
level should be, etc.) and constraints (available resources,
time constraints, etc.) need to be dynamically determined.
The planning block generates the sequence of tasks
necessary to achieve the goals. Planning also involves the
task distribution between the agents and can be achieved
by either the human or the robot. This is determined by the
interaction between the two agents via the HRI. Once
completed the plan is verified via the HRI. Upon
verification, the plan is executed. Either of the two agents
can then assume initiative, which is done through the HRI.
Either the human or the robot can intervene in the
execution sequence by raising an interrupt that stops the
execution if a threat is perceived or an error occurs. This
architecture should be flexible enough to allow the agents

A crucial question in MII is: Why take initiative? This

issue was touched upon in the Introduction section and we
further evaluate this issue as it relates to our system. Goals
and constraints dynamically change with changing world,
robot, and operator states. If there is a sudden robot
breakdown, the system constraints may change. Any world
state change may also change the mission goals. For
instance, if the goal is to track a white object and the object
color changes, the goals may change accordingly.
The human maintains internal world and robot state
representations. The human also has an internal
representation of himself or herself. The human may detect
any changes in the robot or world states that prompt the
human to make changes to the mission goals and
constraints. Similarly the robot (which also has
representations of the world, the human and itself) can
detect unpredicted changes in the world and human
operator states that prompt modifications to the goals and
mission constraints. If the robot detects extreme stress or
panic in the human, the robot may make efforts to rescue
the human from a risky situation or abort the mission.
Hence, both the human operator and the robot have the
authority to take initiative dependent upon their
environmental perceptions.
An important question is: When should a particular
agent assume initiative? Some triggers that may cause the
robot or human to take initiative away from the other agent
include: the human or robot is in danger, expresses an
inability to perform, or requests assistance; the world state
changes; or the resource requirements change.
The next question is: How to assume initiative? This is a
domain dependent issue but manners in which the human
or robot can take initiative include: modification of the
mission goals or plan; altering the resource constraints,
altering the manner of execution; or aborting the mission.
While these are basic questions, there are additional
constraints and requirements when developing MII for an
affect-sensitive architecture. First, the robot should be able
to detect, recognize, and respond to human affective states

while executing routine tasks. Required capabilities to

accommodate this requirement include:
Reactive response to certain affective states (i.e.
fatigue, drowsiness, inattention, etc.).
Deliberative response to other affective states (i.e.
anger, frustration, etc.).
Knowledge and ability to prioritize the tasks at
hand.
Learning in order to diagnose affective states.
Knowledge of when and how to take the initiative.
An additional requirement is the provision for a
communication/interaction capability between the robot
and human. This mechanism is necessary to permit
negotiation with the human regarding impending tasks and
goals. This interaction may occur via speech, standard
graphical user interface capabilities or other multimodal
interaction capabilities.

opportunity to take initiative. The Rs-Ho situation occurs

when the Human offers initiative to the robot and the robot
seizes the initiative. Rs-Hs occurs when both the human
and the robot simultaneously seize initiative. In this case
the human has higher priority and he or she seizes the
initiative. Finally, the Ro-Ho situation occurs when both
the human and the robot simultaneously offer initiative. In
this case neither the human nor the robot believes that the
task can be done. When this case arises, the following
options exist: share the initiative; if possible, replan; shut
down the system and/or abort the task; start the task all
over again; consult the main control station.

Initiative Control Matrix

Initiative can be seized either by the human or the robot.
In cases of conflict the human operator has the ultimate
control and gains the initiative. There exist four possible
means of seizing the initiative.
The first method requires the robot to seize the initiative
from the human operator. This may occur when the human
is drowsy, sleepy, or dangerously inattentive. In this case,
the robot may cut the human off from the system, go into
an autopilot mode, or may simply shut the process down.
The robot may offer the initiative to the human operator.
If the human is bored or under challenged, the robot may
offer the human the initiative to assume some of the
autonomous tasks such as planning, updating environment
variables, conducting feasibility checks, etc. The purpose
is to maintain the humans vigilance level by making the
task more interesting or interactive.
The human may also seize the initiative from the
machine. If the world state unexpectedly or suddenly
changes, the human may seize initiative. This may be done
because the human perceives that the robot may not be
equipped to handle the current situation.
Finally, the human may offer the initiative to the robot.
This may occur when the current task is an easy, routine
job, or is so complicated that it is beyond the humans
capabilities. If the task is routine, then this will typically
occur when the human operator has more pressing tasks to
complete. When the task is complicated, the human may
offer initiative because he or she does not feel confident in
the ability to successfully complete the task.
Based upon the four combinations of offering and
seizing may occur. They are: Robot offers Human Seizes
(Ro-Hs); Robot seizes Human offers (Rs-Ho); Robot
seizes Human Seizes (Rs-Hs); and Robot offers Human
offers (Ro-Ho).
The Ro-Hs situation occurs when the robot offers
initiative to the human and the human operator seizes the

Figure 6. Human/Robot Initiative-Switching Matrix.

An initiative-switching matrix is shown in Figure 6. Each

cell represents a typical scenario during an HRI sequence.
The upper, left cell represents the case when both the
human and the robot are ready to give up the initiative to
the other agent (Ro-Ho). In this case the human decides
who will retain the initiative. The Robot seizes, Human
offers cell indicates the case where the robot seizes the
initiative while the Robot offers, Human seizes cell
represents the case in which the human seizes the
initiative. The ambivalent term refers to scenarios where
the agent can either give or take initiative, i.e., it is capable
of doing the task entirely on its own or with some
assistance. Hence if the human is ambivalent about taking
initiative and the robot seizes initiative, then the human
will relinquish initiative and vice versa. The same holds
for the robot ambivalence. If both the agents are
ambivalent then they can either share the initiative or the
human can assign initiative.

Discussion
The robots ability to interpret implicit human operator
states (emotional/affective state) is critical when
implementing a mixed-initiative interaction between
humans and robots. There are various emotional indicators
that can be exploited to identify affective states: facial
expressions, gestures, vocal intonation, physiology etc. We
focus on using physiological responses, as these are
generally involuntary and less dependent on culture,
gender, and age than the other emotional indicators. These
responses offer an opportunity to recognize emotions that

may be less intuitive for humans but more appropriate for

robots, which can acquire the physiological signals in realtime and employ various signal processing and pattern
recognition techniques to infer the underlying emotional
states. Recent innovations in affective computing (Picard
1997) and wearable computers have made it feasible to
process physiological signals using small, lightweight
biofeedback sensors that are non-invasive, comfortable,
unobtrusive, and fast enough for real-time applications.
The potential of physiological sensing for affect
recognition has already been demonstrated by the
pioneering work of Picard and her colleagues (Picard
1997, Healy & Picard 1998).
Currently we are collecting data from fifteen volunteer
participants as they engage in a series of cognitive tasks
designed to systematically and differentially evoke the
emotions targeted towards recognition (boredom, taskengagement, anxiety, and frustration). The data collected
will be used to develop a model of each individuals
physiological expression of the targeted emotions. This
will involve identifying, for each participant, the
physiological correlations and response patterns that
characterize the experience of each of the emotions.
Special emphasis is being placed on identifying
physiological markers and patterns that differentiate each
of the states from the others. The resulting models will
then provide the basis for the development of affect
recognition algorithms for each individual, which will also
be built using this initial data set.
Once the affective models for determining a humans
implicit state are determined, the MII architecture will be
incorporated in a human-robot collaboration task. In the
actual MII system, a mobile robot (Trilobot) will be used
as a test bed. The initial experiments will send the original
physiological data (originally collected using Biopac
System) to the robot as if it were being communicated in
real-time. The robot will receive this data and interpret the
humans affective state. The robot will employ other
sensors (infrared sensors, light sensors, motion detection
sensors, etc.) to sense its environment and formulate a
world model. The robot can also send messages to a
human operator via the HRI. The system will be tested
using human-robot collaboration applications such as
exploration, and search and rescue.

been applied to a large domain of problems including:

dialogue and discourse understanding, planning, humanmachine interaction, and human-robotic interaction.
Our work focuses on the development of affect-based
interaction between a human operator and a robot. We
have developed a preliminary mixed-initiative human
robot collaborative architecture that relies on affect-based
and standard human-robot interaction capabilities. Rileys
(Riley 1989) general model of MII has been applied to the
development of this architecture. In addition to the
architecture presentation, we have also presented some
affect-based constraints that were considered while
developing this architecture. Currently data collection is
being conducted to develop the affective models that will
be used for testing this architecture.
The future work includes designing HRI evaluations
wherein we can elicit a range of emotions from the
participants and test the speed and reliability of the robot's
MII architecture in real-time. Future work also consists of
expanding the range of tasks and contexts to which this
framework can be applied and increasing the reliability
and sophistication of the emotion recognition.

Conclusions

Chu-Carroll, J., & Brown, M.K. 1997. Initiative in Collaborative

Interactions Its Cues and Effects. 1997 AAAI Spring Symposium:
Computational Models for Mixed Initiative Interaction , Tech Report SS97-04, pp. 16-22.

Mixed-initiative interaction plays a crucial role in

developing collaborative HRI. The focus is to provide an
interaction that permits the human and robot the ability to
interact in a manner similar to human interaction. The
interaction focuses on which agent has the initiative and
how initiative transitions from one agent to another. MII is
not a new concept; Carbonell (Carbonell 1971) introduced
the term in 1971 in relation to Computer Assisted
Instruction. Throughout the years, the MII concept has

References
Anzai, Y. 1994. Human-Robot-Computer Interaction: a New Paradigm of
Research in Robotics. Advanced Robotics, 8(4): 357-369.
Bell, B., Franke, J., & Mendenhall, H. 2000. Leveraging Task Models for
Team Intent Inference. Proc. 2000 International Conference on Artificial
Intelligence.
Bradley, M., M., 2000. Emotion and Motovation. In J. T. Cacioppo, L G.
Tassinary, & G. G. Berntson (Eds.). Handbook of Psychophysiology, 2nd
Ed, pp. 602-642. New York: Cambridge University Press.
Bruemmer, D.J., Marble, J.L., Dudenhoeffer, D.D., Anderson, N.O. &
McKay, M.D. 2002. Mixed-Initiative Control for Remote
Characterization of Hazardous Environments. Proc. of the 36th Hawaii
International Conference on System Sciences.
Burnstein, M., Ferguson, G., & Allen, J. 2003. Integrating Agent-Based
Mixed-Initiative Control with an Existing Multi-Agent Planning System.
University of Rochester, Computer Science Department, Technical
Report 729.
Carbonell, J.R. 1971. Mixed-Initiative Man-Computer Instructional
Dialogues, BBN Report No. 1971, Job No. 11399, Bolt Beranek and
Newman, Inc.

Cinotti, G., Loi, G., & Pappalardo, M. 2001. Frequency Decomposition

and compounding of Ultrasound Medical Images With Wavelet Packets.
IEEE Trans. Med. Imagining, 20(8): 764-771.
DAloisi, D., Cesta A., & Brancaleoni, R. 1997. Mixed-Initiative Aspects
in an Agent-Based System. 1997 AAAI Spring Symposium:
Computational Models for Mixed Initiative Interaction , Tech Report SS97-04, pp. 30-36.

Donaldson, T. & Cohen, R. 1997. A Constraint Satisfaction Framework

for Managing Mixed-Initiative Discourse. 1997 AAAI Spring
Symposium: Computational Models for Mixed Initiative Interaction,
Tech Report SS-97-04, pp. 37-43.
Fleming, M. & Cohen, R. 1999. User Modeling in the Design of
Interactive Interface Agents. Proc. of the 7th International Conference on
User Modeling, pp. 66-76.
Frijda, N., H. 1986. The Emotions, chapter Physiology of Emotion, pages
124-175. Studies in Emotion and Social Interaction. Cambridge
University Press, Cambridge.
Goldman, R.P., Guerlain, S., Miller, C., & Musliner, D.J. 1997.
Integrated Task Representation for Indirect Interaction. 1997 AAAI
Spring Symposium: Computational Models for Mixed Initiative
Interaction, Technical Report SS-97-04.
Goodrich, M.A., Olsen, D.R., Crandall, J.W. & Palmer, T.J. 2001.
Experiments in Adjustable Autonomy. International Joint Conference on
Artificial Intelligence - Workshop Autonomy, Delegation, and Control:
Interacting with Autonomous Agents.
Guinn, C.I. 1998. An Analysis of Initiative Selection in Collaborative
Task-Oriented Discourse. User Modeling and User-Adapted Interaction,
8: 255-314.
Healy, J., & Picard, R. W., 1998. Digital Processing of Affective Signals.
Technical Reports #444, Vision and Modeling Group, Media Lab, MIT.

Psychophysiology, 2nd Ed, pp. 533-575.

University Press.

New York:

Cambridge

Penner, R.R., Nelson, K.S., & Soken, N.H. 1997. Facilitating Human
Interactions in Mixed Initiative Systems Through Dynamic Interaction
Generation. Proc. of the IEEE International Conference on Systems, Man
and Cybernetics, pp. 714-719.
Ramakrishnan, N., Capra, R., & Perez-Quinones, M. 2002. MixedInitiative Interaction = Mixed Computation. 2002 ACM SIGPLAN
Workshop on Partial Evaluation and Semantics-Based Program
Manipulation, pp. 119-130.
Picard, R. 1997. Affective Computing. The MIT Press, Cambridge,
Massachusetts.
Rani, P., Sarkar, N., Smith, C. & Kirby, L. 2004. Anxiety Detecting
Robotic Systems Towards Implicit Human-Robot Collaboration.
Robotica, 22(1): 85-95.
Rich, C., & Sidner, C.L. 1998. COLLAGENT: A Collaboration Manager
for Software Interface Agents. User Modeling and User-Adapted
Interaction, 8: 315-350.
Riley, V. 1989. A General Model of Mixed-Initiative Human-Machine
Systems. Proc. of the Human Factors Soceity33rd Annual Meeting, pp.
124-128.

Hearst, M.A. Ed., 1999. Trends & Controversies: Mixed-Initiative

Interaction. IEEE Intelligent Systems, 14(5): 14-23.

Riley, V. 1992. Human Factors Issues of Data Link: Application of A

Systems Analysis. Society of Aeronautic Engineers, Aerotech 92,
Anaheim, CA.

Horiguchi, Y., & Sawaragi, T. 2001. Naturalistic Human-Robot

Collaboration Mediated by Shared Communicational Modality in
Teleoperation System. Proc. of the Sixth International Computer Science
Conference of Active Media, pp. 24-35.

Roy, N., Baltus, G, Fox, D., Gemperle, F., Goetz, J., Hirsch, T.,
Margaritis, D., Montemerlo, M., Pineau, J., Schulte, J., & Thrun, S. 2000.
Towards Personal Service Robots for the Elderly. Workshop on
Interactive Robots and Entertainment.

Horvitz, E., 1999. Principles of Mixed-Imitative User Interfaces. Proc. of

the Conference on Human Factors in Computing Systems, pp.159-166.

Scherer, K. R. 1984. On the nature and function of emotion: A

component process approach. In K. R. Scherer & P. Ekman (Eds.),
Approaches to Emotion, pp. 293-317. Hillsdale, NJ, Erlbaum.

Ishizaki, M., Crocker, M., & Mellish, C. 1999. Exploring MixedInitiative Dialogue Using Computer Dialogue Simulation. User Modeling
and User-Adapted Interaction, 9L 79-91.
Kortenkamp, D., Bonasso, R.P., Ryan, D. & Schreckenghost, D. 1997.
Traded Control with Autonomous Robots as Mixed Initiative Interaction.
1997 AAAI Spring Symposium: Computational Models for Mixed
Initiative Interaction, Technical Report SS-97-04.
Lester, J.C., Stone, B.A., & Stelling, G.D. 1999. Lifelike Pedagogical
Agents for Mixed-Initiative Problem Solving in Constructivist Learning
Environments. User Modeling and User-Adapted Interaction, 9: 1-44.
Lemon, O., Bracy, A., Gruenstein, A., & Peters, S. 2001. The WITAS
Multi-Modal Dialogue System I. Proc. of the 7th European Conference
on Speech Communication and Technology, pp. 1559-1562.
Mitchell, S. 1997. An Architecture for Real-Time Mixed Initiative
Collaborative Planning. 1997 AAAI Spring Symposium: Computational
Models for Mixed Initiative Interaction, Tech Report SS-97-04, pp. 111113.
Murphy, R., Casper J.L., Micire, M.J., & Hyams, J. 2000. MixedInitiative Control of Multiple Heterogeneous Robots for Urban Search
and Rescue. University of South Florida, Center for Robot Assisted
Search and Rescue Technical Report, CRASAR-TR2000-11.
hman, A., Hamm, A., & Hugdahl, K. 2000. Cognition and the
autonomic nervous system: Orienting, anticipation, and conditioning. In
J. T. Cacioppo, L G. Tassinary, & G. G. Berntson (Eds.). Handbook of

Sherwood, R., Mishkin, A. Estlin, T., Chein, S., Backes, P., Norris, J.,
Cooper, B., Maxwell, S., & Rabideau, G. 2001. Autonomously
Generation Operations Sequences for a Mars Rover using AI-based
Planning. Proc. of the 2001 IEEE/RSJ International Conference on
Intelligent Robots and Systems, pp. 803-808.
Skinner, J.M. Unknown. Multi-Agent Systems and Mixed-Initiative
Intelligence.
LEF
Grant
report.
www.csc.com/aboutus/lef/mds67_off/uploads/skinner_mixed_initiative_a
gents.pdf.
Smith, C., A., & Lazarus, R., S. 1990. Emotion and adaptation. Handbook
of personality: Theory and research, L.A. Pervin (Ed.), pp. 609-637,
New York, Guilford.
Smith, C. A. 1989. Dimensions of appraisal and physiological response in
emotion. Journal of Personality and Social Psychology, 56(3): 339-353.
Wortman, B.C., Loftus, E. F., & Weaver, C. 1998. Psychology, McGrawHill, 5th edition.
Zheng, X.-Z., Tsuchiya, K., Sawaragi T., Osuka, K., Tsujita, K.,
Horiguchi, Y., & Aoi, S. 2004. Development of Human-Machine
Interface in Disaster-Purposed Search Robot Systems that Serve as
Surrogates for Human. Proc. of the IEEE International Conference on
Robotics and Automation, pp. 225-230.