Image Based Expert Systems

- Medical Perspective

Dr.S.Sridhar

Expert Systems
 Introduction
• Expert systems provide expert quality
advice, diagnoses and recommendations
• Designed to perform function of a human
expert
• Examples:
– Medical diagnosis - program takes place of a
doctor; given a set of symptoms the system
suggests a diagnosis and treatment
Expert Systems
 Some Observations
• IT is increasingly pervasive in healthcare delivery (and
• other complex applications)
• As more imaging and signal modalities become available
• communications become faster
• computers more powerful
• doctors are drowning in data

• what they want is information to guide patient

• management. - To do this, systems need to become smarter.

Expert Systems
 Intelligent Systems…
• Sense the World

– DSP, Image Processing, Statistical Data Collection

• Extract Information About the World

– Feature Extraction, Neural Network and Fuzzy Learning, Data Compression

• Recognize the State of the World

– Statistical & Neural Network Pattern Recognition, Detection Theory

• Predict the Behavior of the World

– Probability, Time Series Analysis, Dynamical Systems Theory

• Reason About the World and Plan Actions

– Artificial Intelligence, Decision Theory

• Act on, and Communicate with, the World

– Control Theory, Robotics,Expert Systems
Information & Communications Theory
 Expert Systems: Definitions
• Expert System
– “A system that uses human knowledge captured
in a computer to solve problems that ordinarily
require human expertise.”
• Efraim Turban and Jay Aronson
* Decision Support Systems / Decision
Making Systems / Knowledge Based
Systems / Intelligent Systems
Expert Systems
Applications of Expert Systems
PUFF:
Medical system
for diagnosis of
respiratory conditions

Expert Systems
 Applications of Expert Systems

MYCIN:
Medical system for
diagnosing blood disorders.
First used in 1979

Expert Systems
 Why use Expert Systems?
• Experts are not always
available. An expert system
can be used anywhere, any
time.
• Human experts are not 100%
reliable or consistent
• Experts may not be good at
explaining decisions
• Cost effective

Expert Systems
 Limitations /
Future Developments
• Difficult to elicitate the Knowledge
(Feigenbaum Knowledge Bottleneck)

• Certainty factors are not probabilities

• Acquiring the knowledge base is time

consuming
 Expert Systems on the Web
• http://www.aiinc.ca/demos/whale.html
• http://www.vanguardsw.com/
decisionscript/examples.htm
• http://www.exsys.com/demomain.html
• http://www.expertise2go.com/webesie/car
• http://www.expertise2go.com/webesie/wine

Expert Systems
 Problems with Expert Systems

• Limited domain
• Systems are not always up
to date, and don’t learn
• No “common sense”
• Experts needed to setup
and maintain system

Expert Systems
 Legal and Ethical Issues
• Who is responsible if the advice is wrong?
– The user?
– The domain expert?
– The knowledge engineer?
– The programmer of the expert system shell?
– The company selling the software?

Expert Systems
 Structure of Expert System

Knowledge
Base

Inference
User Interface
Engine

Working
Memory

Expert Systems
 Components of an Expert System

• The knowledge base is the collection of facts and

rules which describe all the knowledge about the
problem domain
• The inference engine is the part of the system that
chooses which facts and rules to apply when
trying to solve the user’s query
• The user interface is the part of the system which
takes in the user’s query in a readable form and
passes it to the inference engine. It then displays
the results to the user.
Expert Systems
 Expert Systems: Definition
• Explanation System
– A specialized inference engine that generates
explanations for the answers presented by the
system.

Expert Systems
 Expert System – Structure
Revisited
Knowledge Production
Base rules

Inference Recognise-
User Interface
Engine act cycle

Working Compared to
Memory production
rules

Expert Systems
 What Is Data Mining?

 Data mining:
 Extraction of interesting (non-trivial, implicit,
previously unknown and potentially useful)
information or patterns from data in large databases

Expert Systems
 Why DATA MINING?

• Huge amounts of data

• Electronic records of our decisions
– Choices in the supermarket
– Financial records
– Our comings and goings
• Data rich – but information poor
• Lying hidden in all this data is information!

Expert Systems
 Data vs. Information

• Society produces massive amounts of data

– business, science, medicine, economics, sports, …
• Potentially valuable resource
• Raw data is useless
– need techniques to automatically extract information
– Data: recorded facts
– Information: patterns underlying the data

Expert Systems
 Multidisciplinary Field
Database
Statistics
Technology

Machine
Learning
Data Mining Visualization

Artificial Other
Intelligence Disciplines
Expert Systems
 KDD Process

Database

Selection Data Training Data Model,

Transformation Preparation Data Mining Patterns

Evaluation,
Verification
Expert Systems
 Data Mining Tasks

• Exploratory Data Analysis

• Predictive Modeling: Classification and Regression
• Descriptive Modeling
– Cluster analysis/segmentation
• Discovering Patterns and Rules
– Association/Dependency rules
– Sequential patterns
– Temporal sequences
• Deviation detection
Expert Systems
 Are All the “Discovered” Patterns
Interesting?

• Interestingness measures: A pattern is interesting if

it is easily understood by humans, valid on new or
test data with some degree of certainty, potentially
useful, novel, or validates some hypothesis that a
user seeks to confirm

Expert Systems
Image Processing

Expert Systems
Examples of Medical Images

Expert Systems
 Special nature of medical images

• Derived from
– method of acquisition
– the subject whose images are being acquired
• Ability to provide information about the volume beneath the surface
– though surface imaging is used in some applications
• Image obtained for medical purposes almost exclusively probe the
otherwise invisible anatomy below the skin.
• Information may be from:
– 2D projection acquired by conventional radiography
– 2D slices of B-mode ultrasound
– full 3D mapping from CT, MRI, SPECT, PET and 3D ultrasound.

Expert Systems
X-ray Image

Expert Systems
 What good is it?
• A fracture of the middle finger is seen on
both views, though it is clearer on the view
on the left. This image can be used for
diagnosis - to distinguish between a sprain
and a fracture, and to choose a course of
treatment.

Expert Systems
 CT (Computed Tomography)

CT Image of plane through

liver and stomach Projection image
Expert Systems from CT scans
 What Good Is It?
• The set of CT images, from the heart down
to the coccyx, was used in planning surgery
for the alleviation of intestinal blockage.
• The surgery was successful (I’m still here).

Expert Systems
Functional Magnetic Resonance
Imaging

Plane 3

From http://www.fmri.org/
Picture naming task
Plane 6
Expert Systems
 What Good Is It?
• This set of images is part of research on brain
function (good for publication).
• Functional imaging is used prior to brain surgery,
to identify structures such as the motor areas that
should be avoided, and focal areas for epilepsy,
that should be resectioned.

Expert Systems
Ultrasound Imaging

Twin pregnancy during week 10

Expert Systems
 What Good Is It?
• This image allows a safe means for early
identification of a twin pregnancy.
• Obstetric ultrasonography can be used to
monitor high-risk pregnancies to allow
optimal treatment.
• Pre-natal scans are part of baby picture
albums.

Expert Systems
 Difficulties/specificities
• Radiology: perspective projection maps physical points into image
space
– but, detection and classification of objects is confounded to over-
and underlying tissue (not the case in general image processing).
• Tomography: 3D images bring both complication and simplifications
– 3D topography is more complex than 2D one.
– problem associated with perspective and occlusion are gone.
• Additional limitation to image quality:
– distortion and burring associated with relatively long acquisition
time (due to anatomical motion).
– reconstruction errors associated with noise, beam hardening etc.
• All these and others account for the differences between medical and
non medical approaches to processing and analysis.

Expert Systems
• Advantage of dealing with medical images:
– knowledge of what is and what is not normal human anatomy.
– selective enhancement of specific organs or objects via injection of
contrast-enhancing material.

• All these differences affect the way in which images are processed and
analysed.

• Validation of medical image processing and analysis techniques is also

a major part of medical application
– validating results is always important
– the scarcity of accurate and reliable independent standards
Expert Systems
 Processing and Analysis
• Medical image processing
– Deals with the development of problem specific approaches to
enhancement of raw medical data for the purposes of selective
visualisation as well as further analysis.

• Medical image analysis

– Concentrates on the development of techniques to supplement the
mostly qualitative and frequently subjective assessment of medical
images by human experts.
– Provides a variety of new information that is quantitative, objective
and reproducible

Expert Systems
 Table of Contents
• Image Mining – What is it?
• Feature Extraction
• Shape Detection
• Color Techniques
• Video Mining
• Facial Recognition
• Bioinformatics
Expert Systems
 Image Mining – What is it?
• Image Retrieval
• Image Classification
• Image Clustering
• Video Mining
• Applications
– Bioinformatics
– Geology/Earth Science
– Security
– …
Expert Systems
 Feature Extraction
• Identify major components of image
• Color
• Texture
• Shape
• Spatial relationships
• Feature Extraction & Image Processing
http://users.ecs.soton.ac.uk/msn/book/
• Feature Extraction Tutorial
http://facweb.cs.depaul.edu/research/vc/VC_Workshop/pr
esentations/pdf/daniela_tutorial2.pdf
Expert Systems
 Shape Detection
• Blob
http://elib.cs.berkeley.edu/blobworld/
• Boundary/Edge Detection
• Time Series – Eamonn Keogh
http://www.engr.smu.edu/~mhd/8337sp07/shapes.
ppt

Expert Systems
 Color Techniques
• Color Representations
RGB:
http://en.wikipedia.org/wiki/Rgb
HSV:
http://en.wikipedia.org/wiki/HSV_color_space
• Color Histogram
• Color Anglogram
http://www.cs.sunysb.edu/~rzhao/publications/VideoDB.p
df
Expert Systems
 Video Mining
• Boundaries between shots
• Movement between frames
• ANSES:
http://mmir.doc.ic.ac.uk/demos/anses.html

Expert Systems
 Ontologies
• Ontology
– Content theories, which have a general set of facts to be shared
– The main contribution is to identify specific classes of objects and
relationships in a specific domain
• Specification mechanism
– Interoperability
– Reuse
– Clarity
– Coherence
– Extensibility

Expert Systems
 Image Ontology

[Câmara et al., 2001]

• Physical Ontology  Method Ontology

– Describes the physical process  Transformation algorithms
of image generation physical level structural level
• Structural Ontology  Application Ontology
– Concerns geometric, functional  Describes the vocabulary related to a
and descriptive structures that generic domain
can be extracted  Task Ontology
Expert Systems
 Specializations of the App. Ontology
 Image Ontology
• Example:
– Application Ontology
• Forest and non-forest areas
– Task Ontology
• Cattle ranches
• Small farms
– Physical Ontology
• Statistical and morphological
properties of the image
– Structural Ontology
• Region structure: regular,
fishbone, corridor and so on
[Eymar Lopes, INPE]
– Method Ontology
• Algorithms and data structures to
extract regions Expert Systems
 Image Ontology

[Câmara et al., 2001]

• Semantic Mediator  Mapping between an instance of

– Relates Image Ontology to a concept on the application
Application Ontology domain and an instance of a
– Identify specific algorithms to concept on the structure domain
extract the desired structures  Matching
– Maps concepts from domain  Set of matchings in a temporal
ontology to extracted structures instance
Expert Systems
 Spatial configuration
 Proposal – 1st Phase
(for a specific deforestation pattern)
Graph Graphs and
Building the
Prototypical Segmentação
Segmentatione Segmented
Generation Structural
Images Rotulação
and Labeling Images Metrics
Ontology

Application

Semantic Mediation
Extraction of the
Application
Application representative graphs
Ontology
Generation Ontology (specialist)

Spatial

Graphs
Patterns Building the
Typology Application
Ontology of the
Expert Systems
Pattern
 Proposal – 2 nd Phase
(for a specific deforestation pattern)
Graph Graphs and
Building the
Images Segmentação
Segmentatione Segmented Structural
Rotulação Generation Metrics
and Labeling Images Ontology

Graphs Subgraph
of the Mining
Pattern

Spatial
Configurations

Expert Systems
 Application Domain
• Land Use and Cover Change
– Land use: purpose to which it’s employed (agriculture, ranching)

– Land cover: Physical status of its surface (forest, water)

– Changes bring environment, social and economics impacts

– The Amazon case: complexity, dimensions and demands involved

• Deforestation: 10.000.000 ha (1970’s)  59.000.000 ha (2000)

– Soil degradation, social conflicts, precarious urbanization

• Faster (and precise) the identification of such tendencies, higher the chances
of preventing, managing and reducing
Expert Systems their consequences
 Object Representation Formalism

• Graphs
– Mathematical abstraction employed in many problems
– Well known and researched formalism
– Represents objects and relationships in an natural way
• Spation Configurations may be approached using
graphs
– Graph inexact isomorphism

Expert Systems
 Building the Structural Ontology
• Extraction of areas (objects)
• Metrics generation
• perimeter area ratio
• fractal index (and so on)
• Graph mapping (objects, relationships, metrics)

Segmentação Graph Graphs and

Images Segmentatione Segmented
Rotulação Generation Metrics
and Labeling Images

Graphs and
Metrics

Expert Systems
Building the Structural Ontology

Application

Application
Ontology Application
Generation Ontology

Spatial
Patterns
Typology

Expert Systems Diffuse, Bidirectional, Fishbone patterns

[Mertens &Lambin, 1997; Escada, 2003 ]
 So much to talk about!
• What is medical image processing and analysis?
• Image segmentation
• Morphological methods
• Feature extraction
• Medical image interpretation
• Validation of image analysis techniques
• Computer-aided diagnosis in mammography
• Other medical applications
– Echocardiography
– Cardiac Image Analysis: motion and deformation
– Angiography and Intravascular Ultrasound
– Vascular Imaging and Analysis
– Pulmonary Imaging and Analysis
– Brain Image Analysis and Atlas Construction
– etc

Expert Systems
 Books

– Digital Image Processing Algorithms and Applications, I. Pitas, Wiley

– Digital Image Processing, Castleman, Prentice Hall.
– Digital Image Processing, Gonzales and Woods, Addison Wesley

– Foundations of Medical Imaging, Z.H. Cho, Joie P.Jones, Manbir Singh,

Wiley

– An Introduction to the principles of medical imaging, Chris C.N. Guy,

Dominic Ffytche, Imperial College Press.

– The essential physics of medical imaging, Jerrod T. Bushberg et al.,

Williams & Wilkins.
– IEEE Transactions on Medical Imaging.

Expert Systems
 Our approach

• The expert system is split into three simpler

expert systems:
– Data selection
– Image processings selection
– Interpretation
Graphic User Interface
 System Architecture
Image Data Rules
Selection Image Old From
Assistant Database Map Expert

New Segmented
SITI Interfaces Image
Image-Interpretation
Assistant

Image Function
Selection Assistant

New
Image Processing Library Map

JAI
 Image Data Selection Assistant
(IDSA)
• The challenge of selecting the right images
– A lot of sensors are available on the market
– The user may already own a huge amount of
images for different zones, different dates and
different sensors.
– The choice of the image depends of the task,
precision quality, and price, …
 Image Data Selection Assistant
(IDSA)

Recommend
images to buy

Min : 1998/01/01
Max: 1999/12/31
Best
images
Request from the user

No

Have
Rule-based system Yes
Image results in
(JESS)
Database database ?

Do a search in a SQL Filter results that

Database matches user requests
 Image Data Selection Assistant
(IDSA)
• Samples rules
– IF Theme(Forest) AND Have_Image(TM) AND
Have_Image(MSS)
=> Remove_Image(MSS)

TM image are more precise than MSS image

IF VeryHumidZone(true) OR WetSeason(true)
=> UseRadar(true) AND UseOptic(false)
For humid zones or during wet season, images may contain
clouds. Radar images are preferable over optical images

Expert Systems
 IDSA Architecture
Image User
Database Model

Production
Rules Questions
Graphical Interface Knowledge
about images Tutor

Image selection

Expert Systems
 Image Processing Assistant
(ASA)
• Before interpreting a map, several pre-
processing steps are needed
– Atmospheric corrections
– Image filtering
– Geometrics corrections
– Classification and segmentation
• For each step, many algorithms may exist,
with different strengths and virtues
depending on:
– Image type (sensor and resolution)
– Speed of processing
– Known parameters for the scene
– Background knowledgeExpert Systems
 Image Processing Assistant
(ASA)
• Multi-agent system
– Each algorithm is represented by an agent
– ASA acts as a broker that negotiates with agents
• The decision process
– The user chooses the type of image processing he wants to perform
– Agents that can process the request are activated
– The user gives preferences and known variables about the request
– Each agent makes a bid for the request
– ASA chooses the best offer from the agents and contacts the winner to
perform the image processing

Expert Systems
 Image Processing Assistant
Image to (ASA)
process
Implemented with MadKit

Image Processing Assistant

Agent 2

Agent 1
Agent n
…

Expert Systems
 Classified Image
Interpretation Assistant
• A segmented/classified image is a new map, but rarely accurate
– Noise filtering and image classification/segmentation, have a limited accuracy
• Human expertise is required to refine the output of the segmentation step
– Using an old map as reference and background knowledge
• CI2A assists in this refinement process
– Using a knowledge base about image interpretation: radiance, textural,
contextual, temporal, background knowledge (old map)
• CI2A verifies the correctness of segmented regions (forest cut, regeneration site,
water, …)

Expert Systems
Image Interpretation Assistant
New Satellite
image
Old map
Classification
Ancillary
Segmented data
image

Change detection

Changed areas
map

Image
description

Object analysis JESS (Java Expert System Shell)

Corrected
map
 Illustrations of production rules
• The reduction of a vegetation suggests a loss of
vegetation cover (with some degree of
confidence)
(defrule prob-recent-oldforest-ndvilow

(declare (salience 20))

?f <- (object (number ?number)

(oldclassName mixed|dense|open|shrub)
(ndvi ?ndvi))
(test (< ?ndvi ?*ndvilow*))
=>
(probClassSeg "store" ?number recent (* ?*conf-ndvi* 0.8)))
 Strengths

• Because it uses production rules:

– has easy explanation capabilities.
– Is modular:
• New rules can be added easily
• Debugged easily.
Expert Systems
Expert System

Expert Systems
 Ultrasound Image

Image Enhancement

Symptoms and Calculi Extraction

Clinical Test
Details of Patients
Distributed Analysis of Calculi
Data
Mining
Framework
Knowledge Base Inference Engine

User /
Diagnosis User Interface

Figure 1.1 Knowledge Based System

Expert Systems
 Performance Evaluation
Table 4.5 Performance Evaluation of the Expert System

Nature of the Case Total Score Mean Score

Intrarenal (kidney) Calculi Expert one: 200 80%
Expert two: 193 77.2%
(Ideal score is 250)
Ureteric and Bladder Calculi Expert one: 120 88.8%
Expert two: 117 86.6%
(Ideal score is 135)
Gallstones Expert one: 206 91.5%
Expert two: 205 91.1%
(Ideal score is 225)

Expert Systems
 Results of Data Mining
Table 5.4 Validation of classification rules

Image Features Renal Dataset Gall Dataset

Algorithm Dataset

Generated Correct Generated Correct Generated Correct

Ripper 6 4 1 0 3 1
Part 10 5 6 2 3 1
Ridor 19 8 3 1 4 1
Oner 3 1 3 0 3 3
Zeror 1 0 1 0 1 0
Prism 23 1 34 2 9 1
Id3 18 6 28 9 9 4
C4.5 12 5 6 3 5 4
Reptree 11 4 1 0 3 3
Random Tree 29 9 38 15 12 8
CN2 15 2 26 1 6 1
Expert Systems
Comparison Before and After data
mining

Expert Systems