Introduction and Overview of

Data and Information Fusion

James Llinas
Research Professor, Director, Emeritus
Center for Multisource Information Fusion
University at Buffalo
llinas@buffalo.edu
 Basic through Advanced R&D In:
IF Based Situation and Threat Assessment Intelligence Analysis
Multiple-Sensor System Data Integration/Analysis
Multi-Intelligence Information Environments
COMINT, ELINT, OSINT, HUMINT, IMINT..
Graph Analysis Methodology and Systems
Applications:
Defense: C4ISR Support and Tactical Applications
Non-Defense: Disaster Consequence Management,
Critical Infrastructure Protection, Medical Informatics

Maritime Domain Awareness

Focus on High Level Information Fusion and Graph Analytics Situation and Threat Assessment
for Multiple Application Domains
Contents are CMIF Proprietary
 Integrated Sensor Selection and Routing Model
Sciences Sampling of Methods
Graph Theory
Maximizes UAV surveillance with consideration for fuel consumption

Surveillance benefit of sensor s at target j cant exceed the demand of sensor

Information Fusion Complex s at target j

Total number of each sensor s cant exceed the number available at the base

Situation Assessment Mathematical Conservation of sensor delivery

UAV sensor capacity

Threat Assessment Optimizations Route continuity

A target may not be visited by the same UAV multiple times

Resource Management All UAVs must begin at base location (location 0)

Information Theory
Route assignments cannot exceed total adjusted flight time

Sensor delivery may not take place unless a target is visited with the
appropriate sensor attached

Mathematical Optimization
Maintains the time that each target is visited and ensures this time falls
within the desired time window
Slide 28

Modeling and Simulation

Knowledge Representation and
Advanced
Reasoning (KRR)
Techniques In
Business/Engineering Linguistic Processing
Systems and Software
CMIF STEF System Main Window
Engineering
Product Lifecycle Management
Service Oriented Design
Rich Web Applications Modern
Complex Event Processing Graphical Message
s
Target
Graphs

Database Design Methods Click to match

single Target Click to match

Realtime Systems graph with a all Target graphs

selected with a selected
Message graph Message graph

Contents are CMIF Proprietary 3

Collaborative Decision Support:
CMIF Command and Control Lab
 Center for Multisource Information Fusion (CMIF)
Flexibility in Research and Development
Core Technology Research & Development
State University of New York at Buffalo, NY, USA
Top level research university and scientific staff focus is basic research and proof of concept experiments
In existence 20+ years (Average grant revenue ~US$8M)
Collaborative Partner for Technology Transition:
CUBRC, a not-for-profit defense R&D organization in Buffalo, New York, USA (~135 people, ~US$35M)
Cleared, experienced staff, facilities focus is development and transition

Core Technology R&D

Broad
UNIVERSITY at BUFFALO
Base
Of
Defense
R&D

Transition & Hardening

UB & CUBRC
 History of Information Fusion
Dates to early 80sfairly young in the sense of
technological historya maturing technology/field of
study
Driven by defense and intelligence needs
Originally as a data compression device to digest huge
amounts of sensed data as sensors advanced in capability (a
push requirement)
Later as an important element for decision support (a pull
requirement)
Matures to very broad range of application
Robotics, medicine, imagery/remote sensing, intelligent
transportation, conditioned-based maintenance, biometrics,
medicine, etc
 What is (Automated) Information Fusion?
Information fusion is an Information Process (Software) comprising:
FUNCTIONS:
Alignment
Association, correlation
Combination of data and information from
INPUTS:
Single and multiple sensors or sources to achieve
OUTPUTS:
Refined Estimates of :
parameters, characteristics, events, behaviors and relations
for/among observed entities in an observed field of view
It is sometimes implemented as a Fully Automatic process or as a Human-Aiding
process for Analysis and/or Decision Support
 Data Fusion: Definition

" Data Fusion is the process of combining data (or

information) for the purpose of estimating or
predicting some aspect of the world"

Steinberg, Bowman, and White, Revisions to the JDL Data Fusion Model, NSSDF 1998
 Most Simply--

Observation System
So that estimation
Multiple types of data Associated or algorithms (mathematical
Correlated to : techniques)or
--various types of automated reasoning
Real information --the same object methods (artificial
World --redundant or event intelligence techniques)
can produce better
--and complementary) or behavior estimates (than based on
any single type of data)

Multiple types of data Related to things To improve estimates about

those things
of interest

Observations Association Estimation

(Multiple) of Observations

These Basic Ideas are Transferable to Many Types of Problems

 Basic Role of Fusion

(Dynamic) Observational Common Referencing Estimates Dec-Mkg

Data
Real Means Analysis
States in the Alignment Association Of World States
(Streaming) etc
World

Process Refinement

Evaluation
Actions

Requirements
driven from here

One means to satisfy user information needs for decision/analysis support,

i.e., most frequently inserted to support human user
 Everyday Data Fusion
Multinodal Fusion
Sound
Augmented
Smell Sensing

Taste

Images

Touch
Pain

Balance
Temperature
Body Awareness
(Proprioception Robotic
Multisensor Fusion
 Sensor Fusion Exploits Sensor Commonalities
and Differences, Knowledge of Errors
Unknown Moving Object
DETECTION KINEMATICS CLASSIFICATION
COVERT
CONFIDENCE RANGE ANGLE CLASS TYPE
COVERAGE

RADAR FAIR POOR GOOD FAIR FAIR POOR

EO/IR FAIR FAIR POOR GOOD FAIR FAIR

C3I FAIR GOOD FAIR FAIR FAIR FAIR

SENSOR Data Data State

Alignment Association Estimation
FUSION

GOOD GOOD GOOD GOOD GOOD GOOD

Data Association Uses Overlapping Sensor Capabilities

so that State Estimation Can Exploit their Synergies
[12]
A Persistent Focus: Reduced Uncertainty

[13]
 Multisource Data (Evidence) Association
O Ti Tj Tk Tl Need some type of Mapping
mi that determines a good
mj way to allocate Obsvns
mm To Tracks
M Observations
From N Sensors Tracks T

STATE
STATE
ESTIMATION
Multiple DATA & STATE
ESTIMATION
Observations & & STATE
ESTIMATION
PREDICTION
ASSOCIATION ESTIMATION
&
PREDICTION
Multiple Entities &
PREDICTION
PREDICTION

Assigned Observations
Resulting from some Best way to decide
which Observations should be given to
each State Estimator
 What a Message looks like a Graphical Structure
An Observational Evidence Atom
--Not a Point Measurement--
An Observation (descriptionRepresentation of an Observation)
Synonyms
Includes judgments as
well as observations

Multiple
Relationships

Disconnected semantic
fragments

Generally all elements Some errors Quantified

have some type of
imperfection or error
Design of the Association Process for Linguistic (Msg) Inputs

Good
Effective Assignment
Semantic Solution &
Human
scoring Graph Merging
Observer 1
Hypotheses
Scored via Hypotheses
Hypotheses
Evaluation
Linguistic,Textual Self-generated Semantic
By high-dimensional
(Semantic) Inputs by node/arc Similarity
Scores that accnt Assignment problem
content
For uncertainty solution

Human
Observer 2 Pick a node/arc, Apply JVC or other
Search other graph Modern assignment
For associable elements Problem solution
(eg exploit ontology)

Smart
Interdependency with
Graph
Text, Semantic Operations
Search
 Fusion of Realtime Data and
A Priori Data Bases
TodayIncludes
Context --Sociocultural Info
--Social Media

Realtime
Intel Sources
Decision-Specific
Air Information
Surveillance Timely
Surface Situation
Accurate
Surveillance Weather
Consistent
Space Intelligence
Structured
Surveillance Coalition Forces
Logistics Integrated
Imagery Overlays
Terrain/Cultural Features
Decision Maker

A Priori and Realtime Fusion

Technology

DATA INFORMATION KNOWLEDGE UNDERSTANDING

 Data Fusion Functional Model
(Jt. Directors
OperationalofBenefits
Laboratories (JDL),
of Multiple 1993)
Sensor
Detection Aggregation Lethality
Data Fusion
Tracking
ID
INFORMATION
Behavior
Events
FUSION PROTOTYPE
Intent
Opportunity
JEM

Point and Level 0 Level 1 Level 2 Level 3 JWARN3

Standoff Sensors Multiple

Processing Reliability
ProcessingMethods:
Processing Processing
GCCS

Data Sources
Sub-object Data
SensorsAssociation &
Improved
Single-Object
--Combinatorial
SituationDetection
Threat/Impact
Optimization
Estimation Assessment Assessment
Estimation
Intel Sources --Linear/NL Estimation
Air Surveillance Extended
--Statistical Coverage
Surface Sensors --Knowledge-based
Distributed Level 4 (spatial and temporal)
Data Base
Standoff Sensors Processing --Control Theoretic
Management System
Space Sensors Improved
Adaptive Process Support Spatial
Database
Fusion
Database
Surveillance Refinement
Resolution
Sensor Mgmt
Level 0 Sub-Object Data AssociationProcess Mgmt
Robustness
& Estimation: pixel/signal level(Weather/visibility,
data association and characterization

Diverse Countermeasures)
Level 1 Object Refinement: observation-to-track association, continuous state estimation (e.g.
kinematics) and discrete state estimation (e.g. target type and ID) and prediction
Sensors Improved Detection
Level 2 Situation Refinement: object clustering and relational analysis, to include force
structure and cross force relations, communications, physical context, etc.
State Estimates ofImprovedReducedState Estimation
Uncertainty
And (Type,threat
Level 3 Impact Assessment: [Threat Refinement]:
Improved Location,
Accuracy Activity)
intent estimation, [event prediction],
consequence prediction, susceptibility and vulnerability assessment

Level 4: Process Refinement: adaptive search and processing (an element of resource management)
 Information Fusion: The Defense Context

Multiple Related to things To improve

Real types of of interest in the estimates about
World sensor data Real World those things

In the defense problem: Associated or Fusion (Estimation)

Non-cooperative, Correlated to the Techniques
Unfriendly
Deceptive
! same object or event
or behavior
 Todays IF Process Design Environment:
Information-space Motivation: Exploitation of all Information

Weather
Dynamic Real
World
Financial

Sensor Observations

Numbers Human Cultural

Observations Chat
Web Twitter
Semantic Label Language Political
HARD SOFT HARD & SOFT

Observational Data Contextual Information

A Priori Dynamic
L4 Knowledge Mgmt Modern Fusion Process World Model

Declarative Knowledge:
Learning Processes World State Estimates Ontologies
 The Soft Front-end Input

Unconstrained
Vocabulary
(Digitized)
(Possibly different
Trained Observer
languages)

Semantics

Computational Linguistics,
Untrained Observer Language
NLP
Processing

Automated
Text
Interview Extraction

Typical
~ RDF Triples (++) Atomic, Raw
Data Input
Bystander
 Source Characterization
Soft Data Hard Data
Calibration
(Truth)
Target
Real
World
Truth

Perceptual and Cognitive

Errors in observation
1

Error in oral expression

2

Error in audio capture 3

Error in audio -to -text 4

Conversion
Pd (Obs Params)
conversion
Error in text extraction 5
To Common Ref, To Common Ref,
Data Association Data Association
 Some Distinctions in Hard and Soft Observational Data
Data Hard Soft Remarks
Characteristic
Observation High Low Imputes requirements for adaptive, retrodiction-type
sampling rate processing (i.e. Out-of-Sequence Measurement type
processing), as well as agile Temporal Reasoning
Semantic Content Limited to specific, usually Can be conceptually Imputes requirements to design an automated Semantic
singular Entities broader than single Labeling process, coupled to a rich Domain Ontology
Entities
Limited to Entity Can include Judged Requires ability to associate and infer at multiple levels
Attributes Relationships of abstraction
Accuracy, Relatively high, good Broadly low accuracy in Imputes requirements for robust Common Referencing
Precision repeatability (Precision) attributes, high at the and Data Association
conceptual level

Totally distinct from Hard Sensors

Humans can also judge intangibles Philosophy: Relations not directly
--emotional state observablerequire reasoning over
properties of entities

This line of thought suggests that relations are the result of a process of some type of
comparison, ie [Brower, 2001], an act of reasoning.

Brower, J., (2001) "Relations without Polyadic Properties: Albert the Great on the Nature and Ontological Status of Relations." Archiv fr Geschichte der
Philosophie 83: 22557.
Counterinsurgency Problem Environment
MURI
Information Fusion
Technology

Soft Sensing Hard Sensing

COIN Decision Support
UNCLASSIFIED//FOUO

Not comprehensive
Counter-Insurgency Context *

24
Soft Operations Kinetic Operations
* Connable, B, Culture and COIN, www.citadel.edu/.../Connable,%20Culture%20and%20Counterinsurgency%20Brief.ppt
Some Remarks on Ontology and
Information Fusion
Dr. James Llinas
Research Professor, Director (Emeritus)
Center for Multisource Information Fusion
University at Buffalo and CUBRC
llinas@buffalo.edu

25
 Roles for Ontologies in IF Processes/Systems
Reasonably reliable a priori Declarative Knowledge
about some domain
In the face of domains for which reliable a priori
Procedural (dynamic) Knowledge is hard to specify
Weak Knowledge problems
As such, they provide a framework that connects
Entities and Relationships
Of fundamental concern for COIN, Ctr-Terrorism,
Irregular Warfare re social structures and militarily-
significant entity relations
The basic construct of a Situation or a Threat
and thus Level 2, 3 Fusion estimation
26
Complexities in Distributed and Networked Systems

In modern Distributed/Networked Systems there are No single

points of authority: These systems are collages of Legacy
systemsJoint/Multiservice systemsCoalition systems
Nodal Ontologies for Fusion/Situational Estimation, and
Communication-support Ontologies for Inter-Nodal
Communications/Data-sharing (eg JC3IEDM)
Harmonizing NLP Operations and Ontologies within and across
such systems
The issue of Uncertainty in Ontological specification:
Probabilistic and Non-Probabilistic Ontologies

Is there an Inescapable need for Semantic Mediation?

Mediator systems well-studied and developed*

Eg Gio Wiederhold (June 1, 1993). "Intelligent integration of information". ACM SIGMOD Record 22 (2)
(This was a major DARPA program) 27
 Semantic Complexity
Controlling Semantic Proliferation/Complexity:
Ontologies
Controlled Languages
Eg Battle Management Language
Eg Shade, U., et al, From Battle Management Language (BML) to Automatic
Information Fusion, Chapter in Information Fusion and Geographic Information
Systems, Lecture Notes in Geoinformation and Cartography,Popovich, V.V.;
Claramunt, C.; Devogele, Th.; Schrenk, M.; Korolenko, K. (Eds.), 2011, Springer
Understanding complexity drivers in text
Eg McDonald, D.D., Partially Saturated Referents as a Source of Complexity
in Semantic Interpretation, Proceedings of NLP Complexity Workshop:
Syntactic and Semantic Complexity in Natural Language Processing Systems,
2000
Measuring Semantic Complexity
Eg, Pollard, S and Biermann, A.W., A Measure of Semantic Complexity for
Natural Language Systems (2000) Proceedings of NLP Complexity Workshop:
Syntactic and Semantic Complexity in Natural Language Processing Systems,
2000

28
 The Association Problem
The Ontologically-specified World is controllablethe Real Data
World is not
While Ontologies can help in Fusion-based estimation and
inferencing problems, the mechanics of exploitation will involve
the associability of Real (uncontrolled) data to (controlled)
Entities and Relations in the Ontologies
Semantic similarity, metrics, degree (hops), etc
Efficient algorithmseg Cloud implementations
PhD-level research
There is also the issue of Coveragein poorly-
understood/known problems, how does one specify an
Ontology that has adequate coverage?
Issue of negative information

29
 Summary
Ontologies have a useful role in the design and
development of Information Fusion systems
Questions regarding issues of:
Authoritative control of semantics in distributed systems
Acceptable, optimal methods for mediation
Complexity of semantics
Understanding, measuring, controlling
Association of semantic terms and complex, high-
dimensional semantic structures
Seem to require further, continuing study to better
define best ways to employ ontological information
in complex, distributed, large-scale Information
Fusion systems and applications
30
 Unified Research on Network-based Hard and Soft
Information Fusion

A CMIF 5-year Multidisciplinary University

Research Initiative (MURI) Program
Funded by the Army Research Office; ~ $7M
UB/CMIF lead + Penn State + Tenn State

Soft/NLP/Ontology Lead: Prof Stu Shapiro, CSE

Building TRACTOR Soft front-end

31
32
33
34