PROCEEDINGS OF SPIE

SPIEDigitalLibrary.org/conference-proceedings-of-spie

Sensor Information Fusion And A

Robot Decisionmaking Methodology

Bonni Almand

Bonni Almand, "Sensor Information Fusion And A Robot Decisionmaking

Methodology," Proc. SPIE 0579, Intelligent Robots and Computer Vision IV,
(11 December 1985); doi: 10.1117/12.950831

Event: 1985 Cambridge Symposium, 1985, Cambridge, United States

Downloaded From: https://www.spiedigitallibrary.org/conference-proceedings-of-spie on 12 Sep 2023 Terms of Use: https://www.spiedigitallibrary.org/terms-of-use

 fusion and a robot decisionmaking methodology
Sensor information fusion methodology
Bonni Almand
Martin Marietta Denver Aerospace
Mail
Mail Stop 0571,
0571, P.O.
P.O. Box 179,
179, Denver,
Denver, Colorado
Colorado 80201
80201

Abstract
This paper is
This is a synopsis from
from a study
study of sensor
sensor data
data fusion
fusion and
and artificial
artificial intelligence
intelligence
(AI) aspects
(AI) aspects of
of blackboarding
blackboarding systems (or belief
systems (or belief structure
structure methodology).
methodology). The studystudy pro-
pro-
motes the design of a knowledge base for
motes for robot
robot motion control
control and
and aa decisionmaking
decisionmaking scheme
scheme
for aa robot
for robot working
working in an environment,
environment, using multiple sensors
sensors for
for guidance
guidance and
and control.
control.
Advanced manipulator control systems
systems require
require output
output processing
processing from
from several
several sensors,
sensors,
fusion
fusion of the sensor data,
data, and
and determination
determination of
of motions
motions that
that best
best adjust
adjust the
theend
end-effector
-effector
for an optimal
position for optimal motion
motion trajectory.
trajectory. A method to weigh the the data from
from the
the manipula-
manipula-
tor
tor guidance
guidance sensors
sensors and
and rules output processing will be proposed.
rules for sensor output proposed.
Blackboarding Systems
Systems for Data Fusion and Planning
Planning
Planning
Planning can be defined as as aa goal-
goal-oriented
oriented process
process of
of preparing
preparing a set of decisions
decisions from
from
for future
alternate options for future action.
action. With autonomous
autonomous planning, goal is
planning, an overall goal is prepro-
prepro-
grammed, but
grammed, but adaptive algorithms during execution produce subgoals
subgoals that
that may
may further
further be
be
subdivided
subdivided into
into primitive executable
executable operations.
operations. During each stage
stage of
of this
this process,
process, sen-
sen-
sory
sory input
input is
is obtained,
obtained, an
an existing
existing planner data base is is searched for
for plan extrapolation
extrapolation
and forecasting,
forecasting, and a decision is made based on data synthesis,
synthesis, modeling,
modeling, and
and deduction.
deduction.
diverse data from alternate source sensors
The diverse sensors often produce conflicting,
conflicting, incomplete,
incomplete, oror
ill-defined
ill- defined plans.
plans. Rules and inference techniques used for
Rules and for processing these
these data
data allow
allow
transformation
transformation from the initial
initial state
state to
to various
various tentative
tentative well-
well-defined
defined plan choices,
choices, and
then
then to the final goal
to the goal state.
state.
The
The decision
decision as to what
as to what plan to implement at a given time
time will depend
depend on
on such
such factors
factors
as
as requirements, time, reliability of the sensor data,
requirements, time, data, and complexity
complexity of
of the
the task.
task. Tools
for optimization of these criteria are based on heuristic evaluations,
for evaluations, and
and the
the actions
actions to
to
be taken are
be taken are determined while considering system constraints.
constraints. After policy
policy is
is established
established
based
based on decisions
decisions of the overall
overall system processor,
processor, the
the execution
execution of
of action
action is
is handled
handled by
by
a subsystem scheduler or
or controller.
controller.
The
The elements
elements of the
the control
control and
and decisionmaking processing lielie between the
the levels
levels ofof
sensor data
sensor data acquisition and performance of the indicative commands.
commands. To implement
implement aa rule
rule-
-
based control
control method,
method, a process instructions must
process of coordinating the system instructions must occur.
occur. There
are three basic
are three for this
basic levels for this control.
control. The top
top-level
-level controller
controller sets
sets the
the overall plan
based on data input,
input, and
and the
the middle
middle-level supervises the
-level subsystem supervises the low
low-level that
-level systems that
actually performs the detailed control.
control. The
The highest
highest level is where most of the automation
level is
processes
processes are
are implemented
implemented and symbolic heuristic
and symbolic used. The process of
heuristic techniques are used. of co-
co-
ordinating
ordinating the
the instructions
instructions from thethe top
top level
level involves
involves decisions that
that may assign
assign perform-
perform-
ance indices for
ance indices for end
end conditions
conditions and
and penalty functions
functions designed toto avoid certain
certain system
system
reactions. A library of performance history data data and
and state
state causal
causal relations
relations is
is updated
updated
continuously to minimize future
future operation
operation cost.
cost. This learning scheme for
This learning for continuous
continuous per-
formance
formance improvement
improvement and
and knowledge
knowledge design,
design, or knowledge engineering, called aa
engineering, may be called
system.1
blackboarding system.1

Current AI
Current AI knowledge engineering isis trying to
to produce rigorous
rigorous programs
programs that
that address
address
paradigms. The
certain paradigms. The methodologies
methodologies for solution require thatthat both the
the data
data and
and produc-
produc-
tion
tion rules
rules be
be represented
represented in
in aa form that
that will
will allow integration into
into aa global data
data base of
of
facts
facts and
and assumptions
assumptions as
as they
they occur
occur in the
the solution process.
process. The
The solution process
process imple-
ments a hierarchy of preferred actions and application strategies.
ments strategies. In In such systems, the
such systems,
blackboard
blackboard tracks events, and coordinates
tracks events, coordinates activity
activity without
without using
using aa predefined
predefined "hard-
"hard-coded"
coded"
settings may direct
procedure. Varying domain settings direct activities
activities inin differing
differing directions.
directions. This
type of
type of system
system maintains
maintains an
an attraction
attraction as
as aa dynamic
dynamic solution process where unexpected or
unplanned
unplanned events
events are
are tolerated
tolerated by
by allowing
allowing opportunistic intelligent plan gen-
opportunistic planning or intelligent gen-
eration based on assertions about
about current
current events.
events.
Blackboards
Blackboards are
are applicable
applicable to problems that cannot be represented strictly
strictly by
by an
an algo-
algo-
solved in
rithm or solved in aa straightforward
straightforward manner.
manner. For
For example,
example, a blackboarding approach isis ap-
ap-
propriate when a problem can be represented by multiple cooperative
cooperative information
information sources
sources
(blackboard
(blackboard specialists),
specialists), each
each supplying
supplying only
only incomplete
incomplete data
data pertaining
pertaining to the current
to the
problem state and none solely capable of validating a particular goal. goal. In this scenario,
scenario,
identifying the
a consensus identifying the next
next best
best action
action is
is desired.
desired.

436 //SPIE
SPIEVol
Vol.579
579Intelligent
IntelligentRobots
Robotsand
andComputer
ComputerVision
Vision (1985)
(1985)

Downloaded From: https://www.spiedigitallibrary.org/conference-proceedings-of-spie on 12 Sep 2023

Terms of Use: https://www.spiedigitallibrary.org/terms-of-use
 blackboard to
We can compare the blackboard
We aspects of
to aspects problem-solving
human problem
the human
of the experience. 2
-solving experience.2
often apply
Humans often
Humans an approach that takes advantage of current situations
apply an or is
situations or opportuni-
is opportuni-
stic.
stic. not necessarily
are not
Human plans are well-defined,
necessarily well- defined, butbut modifiable. actions
modifiable. Available actions
be added
may be
may or deleted,
added or emphasis may be reassigned as required.
and emphasis
deleted, and problem-
Individual problem
required. Individual -
procedures attempt
solving procedures
solving attempt toto structure beliefs
beliefs and change conflicting data. Humans also
conflicting data.
refer to
refer -domain specialists
inherent-domain
to multiple inherent reason about
and reason
specialists and evidence and
about evidence consequences.
and consequences.
There is
There is aa natural tendency to
natural tendency information and
protect against potentially upsetting information
to protect cat-
and cat-
results through incremental planning,
astrophic results
astrophic events, and
planning, rationalize events, fusing of
and fusing data.
of data.
structure of
The structure
The responses are determined by motivations and needs similar
human responses
of human similar to the
to the
goals in
goals in an AI planning problem. Memory can
an AI can be compared to
be compared the blackboard
to the know-
globalknow
blackboard global -
of experiences, and
base of
ledge base
ledge causal and
and causal Vertical branches model the
data. Vertical
relational data.
and relational the
levels of
levels instances of
abstraction, instances
of abstraction, and associated behavior.
experiences, and
of experiences, behavior. Horizontal net-net-
represents the
working represents
working role of various knowledge specialists that
the role importance to
assign importance
can assign
that can to
and identify
events and
events satisfy the
identify solution strategies that satisfy criteria in
the criteria their focus
in their atten-
of atten-
focus of
tion.
tion. Positive and
Positive assigned to
and negative evaluations may be assigned to each in memory
"node" in
each "node" at-
to at-
memory to
tach aa credibility
tach credibility toto future events and and deduce solutions. There
deduce solutions. is a means for
There is denial or
for denial or
proof by
proof contradiction, creation of sidetracks,
by contradiction, byproducts. Reinterpreta-
accidental byproducts.
sidetracks, and accidental
tion or
tion reevaluation is
or reevaluation is common in the human
in the problem-solving
human problem by using
processes by
-solving processes using an execu-
an execu-
tive function
tive polls aa list
that polls
function that and constructs
"worries" and
of "worries"
list of recommendations. In the case
constructs recommendations.
tests, priority is usually given to
indecisive tests,
of indecisive consistency in
to consistency rather than
outcome rather
in outcome tem-
than tem-
porary feature knowledge. This is indicative of
is indicative goal-determined
of aa goal- strategy.
determined strategy.
architecture, parallel
As a predecessor to blackboarding architecture, processing methods
parallel processing problem
for problem
methods for
solving assert
solving that different
assert that determinants can work toward different goals to
different determinants affect aa
to affect
common overall
common Heuristics in parallel
goal. Heuristics
overall system goal. parallel processors synchronization of
processors exploit synchronization of
activities through communication
activities networks. Parallel
communication networks. to aa
programs can be compared to
Parallel programs
breadth -first type
breadth-first of searching
type of algorithm.
searching algorithm.

problem solvers
similar method allows parallel problem
A similar where each
solvers where system processor
each system works
processor works
toward aa single
toward system task
single system concerns at
but concerns
task but individual network
at an individual node can
network node signals
involve signals
can involve
sources. Global
from multiple sources. Global coordination isis demonstrated when any subsystem chooses to
subsystem chooses to
negotiate with
negotiate another subsystem in
with another attempt to reach agreement about available
in an attempt op-
available op-
the system
node may wish to inform the
tions. A node work is
that work
system that complete or
partially complete
only partially
is only that
or that
availability of aa particular node is
availability is high, new task
and aa new
high, and initiated. Main task
be initiated.
may be
task may an-
task an-
nouncements are
nouncements eligibility specifications,
(1) eligibility
are (1) specifications, (2) abstractions, (3)
(2) task abstractions, (3) bids re-
for re-
bids for
work, and
quired work,
quired and (4) information. A method of
time information.
(4) expiration time quiescence (the
of quiescence process of
(the process of
detecting when an
detecting when ceased to
an activity has ceased progress) is
to progress) an asset
is an of some
asset of problem-
parallelproblem
some parallel -
systems.
solving systems.
In contrast,
In contrast, blackboarding assigns each knowledge
assigns each entry to
base entry
knowledge base problem-specific
to aa problem- specific
specialist. These entities are knowledgeable in
domain specialist or control specialist.
domain certain
in certain
areas of
areas and attend
problem and
the problem
of the attend to satisfying individual criterion, work in
criterion, but work conjunc-
in conjunc-
to attain a universal goal. Solution data are
tion to
tion categories:
are partitioned into several categories:
input data,
input current state
data, current description, interim results,
state description, solutions. Black-
results, and problem solutions. Black -
boarding systems
boarding systems implement stronger inference methods
stronger inference compared to
methods compared to parallel problem-solving
parallel problem -solving
and they
systems, and
systems, can explain their reasoning processes by keeping an
they can trail of
an audit trail the
of the
rules used to
rules used decision. For example,
to arrive at each decision. example, the expert system, approaches
system, MYCIN, approaches
aa blackboarding system because
blackboarding system because it rules. *-
incorporates backchaining of production rules.1
it incorporates
Figure 1
Figure 1 represents
represents a generic sensor chart. Problem
flow chart.
sensor blackboard flow - solving activities
Problem-solving activities
triggered by
are triggered
are the subsystem controllers that are counseled by the
by the problem domain
the problem spe-
domain spe-
cialists regarding potential
cialists regarding potential actions solution. The system sensor knowledge
for problem solution.
actions for
are part
sources are
sources of the
part of structure, whereas the processor "to
the permanent memory structure, "to do list
do list"
holds temporary
holds records for pending actions and may be modified based on
temporary records (1) action
on (1) credi-
action credi-
bility, (2) trigger function
bility, (2) action importance,
(3) action
function weight, (3) prior activity
(4) prior
importance, (4) outcomes,
activity outcomes,
(5) planning tactics,
(5) planning tactics, (6) rationale, and
(6) subsystem rationale, and (7) constraints.
(7) system constraints.
Activities on
Activities the left
on the side of
left side the "to
of the do list"
"to do are performed
list" are early in
performed early the processing
in the processing
and those
cycle and
cycle the right may be planned for
on the
those on later execution.
for later activities,
execution. All pending activities,
knowledge source
or knowledge
or records, are stores
source activity records, list for
stored on an accessible list evaluation
for evaluation
during the
during interpretation and
the interpretation decision process.
and decision remains
process.3 Processing policy usually remains
operative until
operative end of
until the end creation. A redefinition of current process plan
subgoal creation.
of aa subgoal
may be
generation may
generation from a process iteration
as an outcome from
be decided as plan), or
(refine plan),
iteration (refine deci-
or aa deci-
sion to
sion advance to
to advance may be
activity may
to another activity (chain). The strategy
made (chain).
be made usually
implemented usually
strategy implemented
prefers recent entries
prefers recent entries into the blackboard and favors with reliable,
favors activities with efficient
reliable, efficient
etiology.
etiology.

SP /EVol.
SPIE IntelligentRobots
579Intelligent
Vol579 ComputerVision
andComputer
Robotsand (1985)/ /437
Vision(1985) 437

Downloaded From: https://www.spiedigitallibrary.org/conference-proceedings-of-spie on 12 Sep 2023

Terms of Use: https://www.spiedigitallibrary.org/terms-of-use
 Datà Input
Data Input
Problem Domain
Problem Domain Event Detection
Clock Event Top -Level Abstraction
Top-Level Abstraction
Paradigms
Paradigms Representation
Information Representation
for Data
Sensor Data
for Sensor Feature Groups
Goals
Goals Clustering
Symbolic Domain
Symbolic Domain
User
User Relevance Selection
Relevance Selection
Meta Rules
Meta Rules
Trace Activities
Focus or Trace
Focus
Policies
Policies Option for
Option User Interaction/
for User Interaction/
High -Level Heuristics
High-Level Heuristics Sensor Query
Blackboard Query
Sensor Blackboard

i
Planes
Support Planes
Knowledge Sources Support
Edit
Edit
Chain- Search Techniques & Filtering
Search
Domain -
Domain Reevaluation,.
Reevaluation,_ Efficiency,
Probability, Efficiency,
Inference Constraints, Probability,
Refine
Refine Weighting
Importance, Weighting
Credibility, Importance,
Specialist Advice & Contract Net
& Contract Activities
Net Activities

Possible
Possible List
Invocable List
Invocable
Error
Error - Events
• Trigger Events
Store Detection
Detection Pattern &
• Pattern Identification
Problem Identification
& Problem
History List

Generation
Subgoal Generation
Subgoal
• Fire & Lookahead
Fire Rules & Cycling
Lookahead Cycling
Consequences,-"Expectations
• Consequences, Expectations
& Optimization
• Etiology, &
Considerations
Considerations

Knowledge Source
Record
Activation Record To-Do List
To-Do List
Performance
• Event Performance Plan Generation
• Strategy & Plan
• Learning • Back-Chaining, Forward-Chaining,
Back -Chaining, Forward -Chaining,
& Risk
• Probabilities & Risk & Gap-Filling
& Gap -Filling Solutions
Solutions
• Readjustments • Selection of Next Move
of Next Move

1.
Figure 1. flow.
Generic blackboard flow.
The blackboarding
The the importance
blackboarding system anticipates activity by allowing the of an
importance of activity
an activity
to be dynamic,
to be affected by
dynamic, affected by levels current process
the current
levels of abstraction above or below the level
process level
and possibly by events
events outside the the current
the realm of the notion of
process. This notion
current process. support-
of support -
data to
allows manipulation of data
plane knowledge allows metaplans by
create metaplans
to create rules to
causing rules
by causing flow
to flow
levels. Currently,
up and down through the blackboard abstraction levels. the most
Currently, the ap-
successful ap-
most successful
certainty factor
to evidential reasoning is to associate a certainty
proach to with both
factor with the left
both the and
left and
right hand
right sides of each rule and to perform conjecture analysis with
hand sides notification.
error notification.
with error
systems
Examples of data fusion and blackboarding systems
Examples
to create
attempt to
An attempt that generates
system that
create aa system control is
- independent blackboard control
domain-independent
generatesdomain is
founded by
project founded
BBl project
implemented in the BB1 B. Hayes
by B. -Roth. 4 It
Hayes-Roth. admits aa predefined pre-
It admits
user-defined
liminary control blackboard, aa user -defined domain-blackboard, some specific
domain -blackboard, some and
specific domain and
control knowledge sources
control sources. The intent of BB1
sources and a few generic knowledge sources. is to
BBl is to
construct dynamic
dynamic systems scheduling and
systems that control their own scheduling implement adaptive
and implement control
adaptive control
solving.
techniques for problem solving.
techniques

SPIEVol.
438 //SPIE 579Intelligent
Vol. 579 Robots and
IntelligentRobots Computer Vision (1985)
and Computer (1985)

Downloaded From: https://www.spiedigitallibrary.org/conference-proceedings-of-spie on 12 Sep 2023

Terms of Use: https://www.spiedigitallibrary.org/terms-of-use
 The
The HEARSAY-II
HEARSAY -IISpeech
Speech Understanding
Understanding System
System developed
developed by
by Lesser
Lesser and
and Erman in
in 1977
1977 was
was
actually the
actually the first
first successful sophisticated scheduler,
scheduler, oror blackboarding system.
system. ItIt was
designed
designed to
to implement
implement aa system that
that accepted continuous form of
continuous speech in the form of aa limited
limited
vocabulary
vocabulary from
from many
many cooperative speakers.'
speakers. * ItIt allowed
allowed some
some degree
degree of selection 9fof par-
ameters, segments,
ameters, segments, syllables,
syllables, words and phrases from
from among aa database
database of
of sources.
sources.-*
HEARSAY-III
A HEARSAY -III system
system has
has been
been implemented
implemented by
by Erman,
Erman, London, that improves
London, and Fickas that improves
the
the earlier
earlier version.^
version.4 It It shares some of the the same
same objectives with thethe BB1
BBl blackboarding
blackboarding
system
system (to
(to expand
expand the
the function
function of of control),
control), but it does
but it does not
not provide a strong control
control
theory. ItIt actually
actually defaults to to aa low
low-level
-level scheduler
scheduler that
that simply
simply schedules
schedules events.
A SU
SJ/X/Kproject,
project, invented
invented by
by Nii
Nii and
and Feigenbaum
Feigenbaum in
in 1978,
1978, represents a blackboarding
blackboarding sys-
sys-
tem
tem for
for analysis
analysis of
of continuous
continuous acoustic
acoustic signals
signals produced
produced by objects space,-'- This is
objects in space.' is
often referred
often referred toto as
as the
the "cocktail party problem'.
problem . Voice
Voice descriptions
descriptions are
are digitized
digitized and
and
frequencies
frequencies are
are plotted,
plotted, but no particular
particular model
model describing
describing the,
the motion
motion of sources can
can be
be
generated. TheThe best
best structure is is aa hierarchical
hierarchical real
real-time
-time search
search for a best hypothesis
from trends.
chosen from trends.
CRYSALIS 1979) attempted
CRYSALIS (Engelmore and Terry, 1979) attempted to
to infer
infer the
the structure
structure of
ofthree-
three-dimen-
dimen-
sional (3-D)
sional (3 -D)protein
protein molecules
molecules by
by using
using electron mapping derived
electron density mapping derived from
from xx-ray
-ray dif-
fraction
fraction patterns. The
The electron distributions
distributions allowed
allowed determination
determination ofof locations
locations of
of
individual atoms.
individual atoms. 11
The
The ILASP
HASP program
program developed for interpretation of sonar signals
developed for signals in
in real
real time
time (Nii
(Nii and
and
Feigenbaum) focused
Feigenbaum) focused on
on signal
signal-to-symbol
-to- symboltransformations
transformations by
by studying
studying harmonics in the
harmonics in the sig-
sig-
nals.^
nals. It implemented
It implemented an iterative control strategy
strategy for
for classifying
classifying events
events as
as simple,
simple,
expected, or
expected, or clock occurrences.

0PM computer simulation

The OPA simulation of of human
human multiple
multiple-task
-task sequence
sequence planning is
is aa program
program
created in 1979 by B.B. Hayes
Hayes-Roth,
-Roth, F.
F. Hayes-Roth,
Hayes -Roth, Rosenschein,
Rosenschein, and
and Cammarata.
Cammarata.55 The task
plan
plan developer
developer receives
receives as input the tasks desired,
as input desired, time
time constraints
constraints for
for actions,
actions, and
and lo-
lo-
cation necessary
cation necessary for
for task actions task sequence.
actions to form an optimal task sequence. ItIt identifies mutual-
ly
ly supportive
supportive partial
partial solutions
solutions along
along the
the road spatial
spatial environment with an incremental,
incremental,
top-down,
top -down,and
and bottom-up
bottom -updomain
domaincontrol
controlmechanism
mechanism by
by minimizing
minimizing angular
angular disparity and
and
optimizing
optimizing time
time to
to form an agenda of best tasks.
tasks. The
The blackboard
blackboard contains
contains aa metaplan
metaplan of
of
planning
planning intentions,
intentions, attributes
attributes of potential plans,
plans, and actual plans to to be
be executed.
executed. It
has aa knowledge
has knowledge base
base as
as aa world model of local tasks,
tasks, routes,
routes, and
and task
task relationships.'
relationships.1
Prototype for a robotic
robotic sensor
sensor blackboarding
blackboarding system
system
Robots
Robots generally acquire and process an an immense
immense amount
amount of
of data;
data; however,
however, only
only aa frac-
frac-
tion of the data isis useful.
useful. Some data are highly repetitive and and some
some are
are undependable.
undependable.
The usual
usual purpose of a robot sensor
sensor information
information processor
processor is
is to
to collect
collect time
time-dependent
-dependent
data
data that
that have
have the
the characteristics events. These data are then used to
characteristics of new events. to determine
determine
specific patterns
specific patterns or
or classes
classes of characteristics,
characteristics, translate thesethese characteristics
characteristics intointo
useful
useful information, transmit the
information, and transmit the most
most dependable,
dependable, accurate
accurate end
end-effector command.
-effector goal command.
This
This command
command may
may be
be derived
derived by weighting the input
input sensor
sensor data
data and combining
combining of of these
these data
data
to
to form
form aa unique plan command.
command. The
The notions of data fusion,
fusion, blackboarding,
blackboarding, andand decision
decision--
making
making are
are inseparable
inseparable in the robot planning world.
world. The robot control systemsystem must
must contain
contain
a blackboarding
blackboarding mode
mode for enabling decisions on what data to to retain
retain and
and how
how to
to process
process them
them
for
for future cases. However, the the decisionmaking
decisionmaking andand problem
problem-solving
-solving techniques in in robot
robot
control
control programming
programming implies
implies fuzzy boundaries for rule generation.
generation. Domain-specific
Domain -specific rules
may
may be
be implemented
implemented asas aa preliminary
preliminary means
means of
of accomplishing
accomplishing the
the robot
robot sensor
sensor data
data fusion
fusion
for
for a particular task.
task.
The issue
The issue of
of portability
portability from
from one
one robot
robot task
task application
application toto another
another raises
raises the
the question
question
of
of whether
whether the
the rule
rule base
base and
and heuristics general or dependent on the
heuristics are general the problem
problem domain.
domain.
For
For aa robot
robot blackboarding
blackboarding system to be truly portable,
portable, itit must span
span aa broad
broad range
range of
of ap-
ap-
sensor fusion
plications and sensor fusion techniques.
techniques. ItIt would
would be
be most
most appropriate to modularize the the
sensor information processing system
sensor system separately
separately from
from the
the problem-
problem-specific
specific heuristics and and
outputs. In In various
various robot
robot environment settings,
settings, different sensors
sensors may
may be
be used
used and
and alter-
alter-
nate
nate heuristics
heuristics may
may be
be required to encourage the correct processing rules rules to
to fire.
fire. The
robot
robot blackboarding
blackboarding specialist
specialist rules
rules and processes must be modifiable to to accommodate
accommodate the
the
work settings.
multitude of robot work settings.
The robot sensor
sensor-data
-data fusion
fusion system here is
system suggested here is predominately
predominately aa problem-
problem-specific
specific
description with some general aspects. An an example
example scenario,
scenario, consider
consider the situation in
the situation in
which a robot
robot manipulator must perform aa task
task in
in an
an environment
environment of
of obstacles
obstacles by
by using
using mul-
mul-
tiple sensors
tiple sensors for guidance and control
control of
of the
the end
end effector.
effector.. Distributed sensing systems
systems
are an extension of
of the
the distributed
distributed problem
problem-solving
-solving notion.
notion.' The data processing in in
these
these systems
systems is
is an
an attempt
attempt toto fuse together various sensor
sensor signals
signals to
to

SPIE
SPIE Vol.
Vol 579 Intelligent Robots and Computer
Computer Vision
Vision(1985)
(1985)// 439
439

Downloaded From: https://www.spiedigitallibrary.org/conference-proceedings-of-spie on 12 Sep 2023

Terms of Use: https://www.spiedigitallibrary.org/terms-of-use
 detect,
detect, classify, track objects.
classify, or track objects. A large amount of data may necessarily
necessarily be
be integrated
integrated
in such
in such aa system
system and
and therefore
therefore steps in the
steps in the processing hierarchy that lead
lead to
to an
an overall
overall
monitor
monitor node
node that
that transmits
transmits the
the appropriate integrated designation might be described
described . .

The
The lowest
lowest level
level in
in the
the hierarchy
hierarchy detects
detects signals
signals while recording,
recording, for example, frequency,
frequency,
time
time of
of detection,
detection, type
type of data,
data, strength of signal,
signal, and orientation of thethe detector.
detector.
Next,
Next, the
the sensor data are reduced by filtering out repetitive or or nonuseful
nonuseful data.
data. Finally,
methods
methods (such as as triangulation for range determination or or image
image convolution
convolution algorithms
algorithms for
for
vision
vision processing)
processing) are used to establish object positions and and attitudes,
attitudes, speeds,
speeds, and
and other
other
features so
features so that goal action
action can
can be
be rendered.
rendered. Changes
Changes in signal
signal detection or
or signal
signal con-
con-
tent
tent are
are easily
easily noticed and handled,
handled, and required focus attention cancan be
be accomplished
accomplished byby
isolating
isolating problems thatthat are
are readily
readily identified.
identified. ItIt should
should be
be noted that sensor processing
noted that
systems
systems require
require thethe transfer of complex information (e.g.,
(e.g., pixel images)
images) and
and imply
imply the
the
necessity
necessity for
for robust
robust comparison algorithm implementation.
implementation. Identification of of priorities
priorities
and
and aa means
means ofof data filtering is
data filtering is essential this case.
essential in this case. The notion of a contract net net is
is
ideal
ideal for
for this
this system because it improves
because it improves the communication between subprocesses
subprocesses byby pro-
pro-
two-way
viding two -way informative
informative responses
responses and deliberations
deliberations during
during time
time-critical situations.
-critical situations.

example, the
As a preliminary example, the Henderson
Henderson and
and Fai
Fai Multisensor
Multisensor Kernel
Kernel System
System (MKS)
(MKS) devel-
devel-
1983 identified a robot control work station
oped in 1983 station for
for inspection
inspection of
of assembly
assembly parts
parts that
that
was equipped with various manipulator arms, arms, hands,
hands, andand other
other sensors.
sensors. The distributed
distributed
sensor
sensor system
system integrated
integrated tactile
tactile data,
data, vision images
images output from
from aa TV
TV camera,
camera, and
and data
data
gathered
gathered from using
using the
the robot
robot dexterous
dexterous hands
hands to form a world model of of the
the objects
objects inin the
the
environment.'
environment.7 A method for for analyzing
analyzing ofof the
the 33-D
-D object
object data
data used
used aa spatial proximity
graph for specifying spatial relations between features features represented
represented inin the
the world.
world. Tech-
niques
niques in in range
range finding
finding and
and object
object recognition allowed analysis that that led
led to
to methods
methods for
for
manipulation. At
object manipulation. the lowest
At the lowest level
level of model representation, the the raw
raw sensory
sensory data
data
acquired and
were acquired and handled
handled with
with3 3-D models. The
-D models. The spatial
spatial proximity graph
graph consisted
consisted ofof
features
features extracted
extracted from
from images
images and
and derived engines. For example,
derived from sensor engines. example, object
object
structural
structural relations,
relations, like
like planar faces
faces on objects
objects were deciphered for for object
object recognition
recognition
and classification. An extension to smart sensors, sensors, or logical
logical sensors,
sensors, would
would allow
allow direct
direct
features (e.g.,
detection of object features (e.g., temperature,
temperature, compressibility,
compressibility, edge
edge detection,
detection, or or sur-
sur-
face
face normal detection). A trend toward a more abstract design design that
that integrated
integrated thethe input
input
data
data was
was accomplished
accomplished byby organizing
organizing the
the data into
into clusters. Finally, at
clusters. Finally, at the
the highest
highest
level,
level, aa coordination
coordination ofof control
control activity waswas implemented by assessing the the collective
collective data
data
sets
sets and
and generating
generating robot
robot controller commands,
commands, such as sensor" or
as "turn on a sensor" or "aim
"aim the
the
camera".
camera ". This system was reconfigurable to
This to incorporate
incorporate other
other sensors
sensors and
and extremely
extremely useful
useful
in
in assessing
assessing varying system situations.
situations. Application extensions
extensions could
could include
includereal
real-time
-time
processing
processing of other visual, auditory, or tactile sensor
visual, auditory, sensor systems.
systems. The use of of the
the multisen-
multisen-
sor
sor data
data integration
integration techniques
techniques allowed
allowed aa capability to demand less less data
data from
from the
the sensors
sensors
as processing continued
as continued and
and constraints
constraints took
took effect.
effect.
A rigorous probabilistic model isis needed
needed for
for the
the robot
robot planning
planning problem.
problem. Suggested
algorithms include a weighting method for combining imprecise
algorithms imprecise multiple
multiple sensor
sensor reports,
reports,
confidence
using con intervals as
idence intervals as defined
defined by
by the
theroot
root-mean-square
-mean -square error
error calculations in in sensor
sensor
analysis. in signal
Correlation in signal measurements andand probability
probability rules
rules are
are used
used to
to propo-
propo-
gate system assumptions. An example of rule probability that the the system
system could
could implement
implement
be:
might be:

If evidence E is true,
If true, then
then hypothesis
hypothesis Ii
II is
is true
true with
with probability P.
probability P.

means for specifying the

A means the statistical
statistical dependence
dependence of
of one
one evidential
evidential event
event in
in relation
relation
to
to another
another may
may also
also be
be helpful
helpful in identifying evidence and resultant relationships
in identifying relationships when
when
item is
more than one evidence item is considered.
considered. Errors
Errors in correlated event probability
probability esti-
esti-
of conflicting
mates and weighting of conflicting time
time-varying
-varying reports
reports assist interpolation to
assist in interpolation to aa
system goal.
reasonable resultant system goal.
The
The robot
robot planning problem
problem is
is represented
represented by
by aa goal-
goal-determined incorporating the
determined system incorporating the
following:
1)
1) A sequence-level
sequence -level planner
planner for
for determining
determining the
the next
next best
best motion for the manipulator,
and an operation-level
operation -level strategic
strategic planner
planner for
for identifying
identifying subgoal
subgoal focus behaviors.

2)
2) AA means
means for
for interrupting robot processes
processes to solve problems that
that arise,
arise, to
to make
make an-
an-
nouncements from knowledge-specialists
nouncements (sensors), and to
knowledge specialists (sensors), to set
set subgoal
subgoal motions
motions based
based on
on new-
new-
ly acquired sensor data.
data.
3) A mechanism for avoiding
3) avoiding of repetition by optimizing time
time and
and the
the motion
motion of
of the
the man-
man-
ipulator
ipulator arm,
arm, and
and an
an identification
identification of
of constraints
constraints for the manipulator arm
arm and
and sensor
sensor
scenario.

440 //SPIE
SPIEVol
Vol.579
579Intelligent
IntelligentRobots
Robotsand
andComputer
ComputerVision
Vision (1985)
(1985)

Downloaded From: https://www.spiedigitallibrary.org/conference-proceedings-of-spie on 12 Sep 2023

Terms of Use: https://www.spiedigitallibrary.org/terms-of-use
 4) A
4) modular system that
A modular designed for
is generally designed
that is diversity of
great diversity
for aa great tasks,
robot tasks,
of robot
and that
and can use
that can of sensor
types of
use many different types inputs for
sensor inputs guidance with
for guidance visual,
range, visual,
with range,
data.
and tactile data.

Figure 22 shows
Figure for robot motion control and commanding.
shows the approach designed for

World
World Model History List
Modular Heuristics
• Modular Recording Events
Planned Tasks
• Planned Tasks Etiology
Etiology
• Problem Domain Weighting
State
Edit of State
Representation
Sensor
Event 1
Signals
of Signals
Interpretation of
Sensor Weighting of Data
Sensor Data
of Sensor Goal
Acquire Goal
Event 2 Filtering
Priorities
Planner
Strategic Planner i
Sensor Node
Monitor Node
Event N New Expertise Reevaluation
Reevaluation
Supervision
Supervision
Subgoal Generation
Subgoal Error Analysis
• Force/Torque Graphing
Force /Torque Graphing Opportunistic
Exception Handling Planner
• Range Sensing Planner
Range Sensing
Spatial Proximity
• Spatial
Image Processing
• Image 1
System
Control System
Manipulator Servo
Manipulator Servo
Commands
Commands
Net
Contract Net
Knowledge Source Contract
Bids/Announcements
• Bids /Announcements i
• Problem
Problem Isolation Control System
Operation Planner
Operation Planner
Object Tracking
Scheduling/Tactical Planner
Scheduling /Tactical Planner Move Manipulator
Move
Perform Task
Constraints,
• Constraints,
Quiescence, Coordination
• Quiescence, Coordination Engage Object/
Engage
• User Query Release Object
Detailed Analysis
2.
Figure 2. Prototype sensor blackboard.
manipulator blackboard.
sensor manipulator
References
1. Nil,
1. H. Penny,
Nii, H. "An Introduction to
Penny, "An Engineering, Blackboard
Knowledge Engineering,
to Knowledge Model, and
Blackboard Model, and
Heuristic Programming
University Heuristic
AGE," Stanford University ReportNo.
Project, Report
Programming Project, No. 8080-29. 1980.
-29. March 1980.
2. Abelson, Robert P.
Abelson, Carroll, J.
P. and Carroll, Douglas, 'Computer
J. Douglas, Individual
"Computer Simulation of Individual
Belief Systems,"
Belief Systems," The American Scientist, Vol.
Behavioral Scientist,
American Behavioral No. 9,
8, No.
Vol. 8, pp. 2424-30
9, pp. 1965.
-30 1965.
Barbara, "A
Hayes -Roth, Barbara,
3. Hayes-Roth, "A Blackboard Model of
Blackboard Model of Control,"
Control," Stanford University Heuristic
University Heuristic
Report No.
Programming Project, Report HPP8383-38.
No. HPP -38. 1984.
August 1984.
4. Hayes Barbara, "BB1:
-Roth, Barbara,
Hayes-Roth, An Architecture
"BB1: An Architecture for for Blackboard Systems Control,
that Control,
Systems that
Explain, and
Explain, about Their
Learn about
and Learn Behavior," Stanford
Own Behavior,"
Their Own Heuristic Programming
University Heuristic
Stanford University Programming
No. 84
Report No.
Project, Report 1984.
-16. December 1984.
84-16.
6. Randall, and
Davis, Randall,
Davis, Smith, Reid
and Smith, "Negotiation as
G., "Negotiation
Reid G., Metaphor for
as aa Metaphor for Distributed
Distributed
Institute of
Solving," Massachusetts Institute
Problem Solving," Technology AI
of Technology No. 624.
Memo No.
AI Memo 1981.
624. May 1981.
Henderson, Thomas
7. Henderson, and Fai,
C., and
Thomas C., So, AA Multisensor
Wu So,
Fai, Wu Integration and
Multisensor Integration and Data
Data
Acquisition System,"
Acquisition Proceedings, pp.
IEEE Proceedings,
System," IEEE 274-279.
pp.274 -279. 1983
8. Rauch, Herbert E.,
Rauch, Herbert Concepts For
E., "Probability Concepts Expert System
An Expert
For An Data
Used For Data
System Used
Fusion," AI Magazine,
The AI
Fusion," The Magazine, Fall 1984, pp. 55
1984, pp. 55-60.
-60.

SPIE Vol. 579

SPIE Vol. 579 Intelligent
Intelligent Robots Computer Vision
Robots and Computer Vision (1985) 441
(1985) // 441

Downloaded From: https://www.spiedigitallibrary.org/conference-proceedings-of-spie on 12 Sep 2023

Terms of Use: https://www.spiedigitallibrary.org/terms-of-use