Module 3: Meaning and Inference

Graphical Representations Module 3 Lecture 3

of Knowledge CSC 247/447 2021

 Knowledge Graphs
located-in location
JOE-BEAN-COFFEE ROCHESTER NEW-YORK
serves
COFFEE

❖ Knowledge graphs are popular resources these days

❖ termed coined in 2012 with “Google Knowledge Graph”
❖ Knowledge is expressed in terms of triples object-relation-
object
❖ e.g., DBPedia is a large crowd-sourced effort
The English version of the DBpedia knowledge base describes 4.58 million
things, out of which 4.22 million are classified in a consistent ontology, including
1,445,000 persons, 735,000 places (including 478,000 populated places),
411,000 creative works (including 123,000 music albums, 87,000 films and
19,000 video games), 241,000 organizations (including 58,000 companies and
49,000 educational institutions), 251,000 species and 6,000 diseases.
 What’s in DBpedia?
❖ Focus on quantity over quality!
❖ Almost exclusively about concrete objects in the world
❖ While a large amount of data, unclear how much is useful,
e.g., all the entries related to “The Bakery”
• The_Bakery wikiPageWikiLinkText "The Bakery".
• The_Bakery_(1990_film) wikiPageWikiLinkText "The Bakery".
• The_Bakery_(TV_series) wikiPageWikiLinkText "The Bakery".
• Arraute-Charritte caption "The Bakery".
• The_Bakery name "The Bakery".
• Chukchansi_Gold_Resort_&_Casino notableRestaurants "The Bakery".
• Louis_Szathmary restaurant "The Bakery".
• Fluorescent_Adolescent title "The Bakery".
• Pranked title "The Bakery".
• The_Bakery title "The Bakery".
❖ contains 172 million
•
•
Q7715368 name "The Bakery".
The_Bakery label "The Bakery".
triples with relation
• Q7715368 label "The Bakery". “Wikipagewikilink”
• The_Bakery name "The Bakery".
• Q7715368 name "The Bakery".
 DBPedia Ontology
❖ DBpedia has explicit classes in an ontology
❖ but the status of objects in relations is unclear

CATEGORY:DOG-TYPES

broader

CATEGORY:WORKING-DOGS

subject MAMMAL
type

BERNESE-MOUNTAIN-DOG is this an object of a class of objects??

 DBPedia Ontology
CATEGORY:UNITED-STATES
broader In what sense are cities a
… subclass of a country?

CATEGORY:MUNICIPALITIES_IN-THE-US

broader

CATEGORY:MUNICIPALITIES_IN-NEW-YORK

broader

CATEGORY:CITIES-IN-NEW-YORK LAKE_ONTARIO
wikipagewikilink CITY
wikipagewikilink
hypernym
ROCHESTER,_NEW_YORK
Here’s a concrete object - but what is it?
WordNet could be expressed as a knowledge graph

from Navigli, R. (2016). Chapter 20. ontologies. Mitkov, R. (Ed.), The Oxford handbook of computational linguistics. Oxford
University Press.
 Formalizing Knowledge Graphs
❖ Knowledge graphs are a propositional logic, with
constants and binary predicates
CATEGORY: CITIES-IN-NEW-YORK
subject
located-in located-in
JOE-BEAN-COFFEE ROCHESTER NEW-YORK
serves
COFFEE
❖ is the knowledge base
{LOCATED-IN(JOE-BEAN-COFFEE, ROCHESTER) &
SERVES(JOE-BEAN-COFFEE, COFFEE) &
LOCATED-IN(ROCHESTER, NEW-YORK),
SUBJECT(ROCHESTER, CAT:CITIES-IN-NEW-YORK)}
 Formalizing Knowledge Graphs
CATEGORY: CITIES-IN-NEW-YORK
subject
located-in located-in
JOE-BEAN-COFFEE ROCHESTER NEW-YORK
serves
COFFEE
{LOCATED-IN(JOE-BEAN-COFFEE, ROCHESTER) &
SERVES(JOE-BEAN-COFFEE, COFFEE) &
LOCATED-IN(ROCHESTER, NEW-YORK),
SUBJECT(ROCHESTER, CAT:CITIES-IN-NEW-YORK)}
❖ Could define inference rules
LOCATED-IN(?x, ?y) & LOCATED-IN(?y. ?z) =>
LOCATED-IN(?x, ?z)
 It’s Hard to capture much knowledge with
triples between constants
❖ We need to start writing sentence fragments as relation
names
❖ John is sitting under the couch

sitting-under
JOHN1 COUCH
LOST ANY COMMONALITY
BETWEEN THE
❖ John is sitting beside the couch PREDICATES

sitting-beside
JOHN1 COUCH

*ACTUAL RELATION NAMES IN THE VQA DATASET

 But It’s Hard to Represent Actual Situations
that can be described in language

❖ Hard to capture more than binary relations

❖ John is sitting between the couch and the chair
sitting-between
JOHN1 ?????

❖ Or indefinite statements
saw
❖ John saw a dog JOHN1 ????

❖ Or quantified statements
saw
❖ John saw many dogs JOHN1 ????

❖ Or commonsense knowledge
???
❖ Dogs bark DOG ????
We’ll address these issues today
 Towards a More Adequate Representation:
Semantic Nets
❖ Two types of nodes, representing
❖ predicate/sets (TYPE nodes)

❖ individuals (TOKEN NODES)

❖ Two key arc types

❖ subtype relations between types

❖ instance-of relations between an token and a type

ANIMAL
giving a semantics by mapping to logic
subtype
SUBTYPE(HUMAN, ANIMAL) means
HUMAN
subtype subtype ∀ x HUMAN(x) ⊃ ANIMAL(x)

FEMALE-HUMAN MALE-HUMAN INSTANCE-OF(MARY3, FEMALE-HUMAN) means

instance-of instance-of
FEMALE-HUMAN(MARY1)

MARY3 JOHN1
 Relations in Semantic Nets
❖ We could follow the strategy in Knowledge Graphs, i.e., introduce
new arc labels

ANIMAL
subtype Problems
HUMAN ❖ likes isn’t representable in the
subtype subtype ontology (e.g., we can’t say likes
FEMALE-HUMAN MALE-HUMAN is an event_of-experience)
❖ can’t attach additional
instance-of instance-of
information, e.g., Mary liked John
MARY3 likes JOHN1 in 2012
❖ can’t represent predicates with
arity greater than 2, e.g., He put
the cat in the house.
 Technical Problems and Philosophy
It turns out the solution to this problem lies in the writings of Donald Davidson in
the 1960’s
How many arguments does a predicate LIKE have?
❖ Maybe 2? Mary likes John
❖ Likes(Mary1, John1)

❖ Maybe 3? Mary liked John last year

❖ Likes(Mary1, John1, 2020)

❖ Maybe 4? Mary liked John in Mexico!

❖ Likes(Mary1, John1, Mexico) ????

❖ To have a clear semantics, we’d need 4

❖ Likes(Mary1, John1, <unspecified time>, Mexico)

❖ Many more with adverbials! Mary likes John a lot.

❖ Likes(Mary1, John1, <unspecified time>, <unspecified location>, INTENSE)

❖ Examples never end!

 Davidson’s Solution
Introduce objects such as relations and events as full members of the
ontology
e.g., LIKE is a unary predicate
Like(E1) == E1 is an instance of a LIKE event
❖ Mary likes John
❖ Likes(E1) & Liker(E1, Mary1) & Likee(E1, John1)

❖ Maybe 3? Mary liked John last year

❖ Likes(E1) & Liker(E1, Mary1) & Likee(E1, John1) & Time(E1, 2020)

❖ Maybe 4? Mary liked John in Mexico!

❖ Likes(E1) & Liker(E1, Mary1) & Likee(E1, John1) & Location(E1, Mexico)

❖ Many more with adverbials! Mary likes John a lot.

❖ Likes(E1) & Liker(E1, Mary1) & Likee(E1, John1) & Intense(E1)
 Representing Statements About Events
How many arguments does the predicate “BREAK” have?
❖ the window broke
❖ I broke the window
❖ I broke the window for john with a hammer
❖ I broke the window for john with a hammer in the morning
❖ I broke the window for john with a hammer in the morning
by the docks
❖ ….

BREAK(B1) & AFFECTED(B1, W1) & WINDOW(W1)

BREAK(B1) & AFFECTED(B1, W1) & WINDOW(W1) & AGENT(B1,ME1) & PERSON(ME1)

BREAK(B1) & AFFECTED(B1, W1) & WINDOW(W1) & AGENT(B1,ME1) &

INSTRUMENT(B1, H1) & HAMMER (H1) & BENEFICIARY(B1, JOHN1)
 … or in Semantic Network notation
BREAK
SPEAKER instance-of
WINDOW
B1 instance-of
instance-of HAMMER
agent

affected instrument instance-of

M1
H1
W1

BREAK(B1) & AFFECTED(B1, W1) & WINDOW(W1) & AGENT(B1,M1) & SPEAKER(M1)
& INSTRUMENT(B1, H1) & HAMMER (H1)
… or a slightly more expressive formulation
encode the type in the token node label

B1:BREAK
agent
instrument
affected
M1:SPEAKER
H1:HAMMER
W1:WINDOW

BREAK(B1) & AFFECTED(B1, W1) & WINDOW(W1) & AGENT(B1,M1) & SPEAKER(M1)
& INSTRUMENT(B1, H1) & HAMMER (H1)
 Typed Logics
❖ A typed logic allows quantified variables to be type
restricted
❖ e.g., all dogs like a bone
∀ d : DOG
∃ b : BONE
∃ e : LIKE-EVENT Liker(e, d) & Likee(e, b)

❖ logically equivalent to
∀ d . [ DOG(d) ⊃
∃ b [BONE (b) &
∃ e [LIKE-EVENT(e) &
Universal quantifiers use
implication and existential use
conjunction
Liker(e, d) & Likee(e, b)]]]
 Quantification our Semantic Network
“all dogs bark” ∀ d : DOG ∃ e :BARK-EVENT . Barker(e, d)
Note it couldn’t be all dogs & all bark events as not all dogs
are barkers in all backing events

??????:BARK-EVENT

barker

?X:DOG

We allow variables in node labels for universal quantification.

what do we do for the existential variable??
 Skolem to the rescue!
❖ Thoralf Skolem (1887-1963)
❖ many accomplishments in foundations of logic - we
look at but one small part
❖ The intuition is that we can transform a formula with an
existential variable in it to another formula where the
existential variable is replaced by a novel constant
ASSUME A KB WANT TO ADD A FACT

∀ x . Dog(x) ⊃ Barks(x) ∃ x Dog(x)

Happy(Fido1) substitute in a novel constant

Dog(Sk137)
What do we know about D137?
We know Dog(Sk137) and Barks(Sk137). We don’t know whether Happy(Sk137)
 Skolem to the rescue!
❖ Skolem proved the transformed set of facts is inferentially
equivalent to the original with the quantifier
❖ Important qualifications

QUANTIFIER SCOPING ISSUES LOGICAL OPERATOR ISSUES

¬ ∃ x Dog(x)
∀ x ∃ b . Likes(x, b) & Bone(b)

substitute in a Skolem function ¬Dog(SK12)

(i.e., ∃ x ¬Dog(x) !!!
∀ x Likes(x, Sk11(x)) & Bone(Sk11(x)) MUST FIRST “MOVE” NEGATION IN
¬ ∃ x Dog(x)
If we put constant in for b
is equivalent to
We’d have a formula equivalent to
∀x ¬ Dog(x)
∃ b ∀ x . Likes(x, b) & Bone(b) so there’s no existential at all!
Logical Equvalences Involving Negation
¬(A ∨ B) <-> (¬A & ¬B)
¬(A & B) <-> (¬A ∨ ¬B)
¬¬A <-> A
¬∀ x P(x) <-> ∃ x . ¬P(x)
¬ ∃ x P(x) <-> ∀ x . ¬P(x)
THIS DIRECTION MOVES NEGATION IN

This technique can convert any logical formula into one

where negation symbols only occur directly before an atomic
predicate
 Quantification

“all dogs bark” ∀ d : DOG ∃ e :BARK-EVENT Barker(e, d)

SK11(?X):BARK-EVENT

barker

?X:DOG
 What can be represented?
SK2(?X, ?Y):LIKES

arg0 arg1

?X:DOG ?Y:BONE

“all dogs like bones”

(interpreted as all dogs like all bones)
“There’s a food that not every dog likes”
∃ f :FOOD¬∀ d :DOG ∃ e : LIKES arg0(e, d) & arg1(e, f)
∃ f :FOOD ∃ d :DOG ∀ e :LIKES ¬[ arg0(e, d) & arg1(e, f)]
∃ f :FOOD ∃ d :DOG ∀ e :LIKES [¬arg0(e, d) ∨ ¬arg1(e, f)]
because of disjunction and negation, this is beyond the
expressive power of our network formalism!
 What can be expressed so far?
❖ Formulas that can be reduced to a form in typed logic
that contains no negation or disjunction!
❖ Note type restrictions have an implicit negation with
universal quantifiers
❖ “All dogs bark”
∀ d :DOG . BARK(d)
which is equivalent to ∀ d DOG(d) ⊃ BARK(d)
 Graph Matching as Inference
Base case: A graph X is entailed by another graph
Y if X is a subset of Y

A A
r2
r1 r2
C
B C Entails
r4
r3 r4
E
D E
 Graph Matching as Inference
Variables: A graph X is entailed by a another graph Y if
there is a substitution that satisfies the type restriction
that makes X a subset of Y
TYPE1
subtype ? A
T1 A
r2
subtype r2

T11 Entails C
?x:T1
{?x/C} r4
instance-of r3 r4
E
C D E
we can substitute C for
?x:T1 because C is a member of T11 which is
a subset of T1 (thus C satisfies the type restriction)
 Querying Graphs
We can query a semantic net with a pattern A by
finding a subgraph that instantiates A
NB: variables are interpreted existentially in queries
A
r2
A
?x:T1
r1 r2
r4

B C:T1 Query/Pattern E

r3 r4 “Is there a T1 such that…”

D E
QUERY SATISFIED
Network WITH SUBSTITUTION {?x/C}
 Selectional Restrictions
❖ Selectional Restrictions encode common sense knowledge
about what objects engage in what activities
❖ Two things we might say:
“all eating events “all things that eat are
have an animal as the animals”
∀ e : EAT
agent (the eater)” ∀ x : ANYTHING
AGENT(e, x) ⊃ ANIMAL(x)
?E:EAT

agent
Not expressible in our formalism!

SK1(?E):ANIMAL

∀ e: EAT
∃ x : ANIMAL AGENT(e, x)
 Selectional Restrictions: Which is better?
OPTION A OPTION B

“all eating events “all things that eat are

have an ANIMAL as agent” animals”
WHAT’S THE DIFFERENCE?
OPTION A SAYS ALL EAT EVENTS MUST HAVE
AN AGENT (WHICH IS AN ANIMAL), BUT THERE
MIGHT BE OTHER NON-ANIMAL AGENTS!

OPTION B SAYS THAT ANY AGENT OF AN

EAT EVENT MUST BE AN ANIMAL, BUT
THERE MIGHT NOT BE AN AGENT

PROBABLY WANT BOTH!

IF WE ASSERT EVENTS HAVE EXACTLY ONE AGENT, THESE BECOME EQUIVALENT!!
 Approximating Negation
❖ For every predicate P introduce another predicate !P
❖ We’d like !P to be equivalent to ¬P, but without disjunction we
don’t have the expressive power to assert such axioms
❖ But we can build something into the proof procedure. If we want
to know if a proposition P, then attempt to prove P and prove !P
and see what you get
Example: Does Fido bark?
ATTEMPT TWO QUERY GRAPHS

?X:BARK ?X:!BARK

agent agent

FIDO:DOG FIDO:DOG
YES! NO! DON’T KNOW!
 Inheritance in Semantic Nets
❖ To make examples clearer, we’ll use a simple abbreviation

INSTEAD OF WE WRITE

Sk(?x):CANFLY CANFLY
BIRD

arg0

?x:BIRD

“all birds can fly”

 Inheritance
❖ Perhaps the best known inference technique in semantic
networks is inheritance
❖ e.g., Given, “Birds Fly” and “Tweety is a Canary”

CANFLY
BIRD
subtype

CANARY
instance-of ENTAILS TWEETY FLIES
TWEETY
 Inheritance (2)
❖ some inheritance can be viewed as a form of logical
deduction, but consider we probably all agree that
❖ Birds fly
❖ Penguins are birds
BUT, PENGUINS DON’T FLY!!!!!!!!

INHERITANCE IS MORE INTUITIVELY DEFINED AS A

TYPICAL PROPERTY, OR A “DEFAULT”
“BIRDS FLY” === MEANS BIRDS FLY UNLESS WE HAVE
EVIDENCE TO THE CONTRARY
INTERESTING FORMALISMS FOR DEFINING THIS MORE
PRECISELY ARE DEFAULT LOGIC AND NON MONOTONIC
LOGIC, BUT WE WON’T HAVE TIME TO COVER THESE
 Inheritance (3)
a procedural approach to inheritance in semantic networks:
to determine if a property is inherited, start at the most specific
node and move up the hierarchy until you find the first answer

DOES PENNY1 FLY NO!

FLY
BIRDS DOES GEO1 FLY YES!
subtype
subtype
¬FLY
PENGUIN SWALLOW
instance-of
subtype
GEO1
EMPEROR PENGUINS
instance-of

PENNY
 Reminder:
Single Inheritance vs Multiple Inheritance
LIVING-ENTITY

ANIMAL GENDERED-BEING
subtype subtype

HUMAN subtype
FEMALE-BEING

subtype
subtype
subtype MALE-BEING

subtype

FEMALE-HUMAN MALE-HUMAN
instance-of instance-of

MARY JOHN1
 Multiple Inheritance

❖ Multiple inheritance complicates inheritance reasoning

❖ may lead to unresolvable contradictions!

FLY ¬FLY
WINGED-CREATURES LAND-CREATURES
subtype
subtype
PENGUIN
instance-of

PENNY DOES PENNY FLY?

 Summary
❖ There is great interest in capturing large amounts of
commonsense knowledge in graphical form
❖ Knowledge graphs a weakly expressive and often lack a
clear semantics
❖ but for some applications quantity is better than quality — just
need to knowing that object X is mentioned in the same
document with object Y (as in DBpedia) may be enough

❖ Semantic networks provide graphical representations

and can have a well-defined semantics
❖ And support some useful non-logical inference (e.g., inheritance
and semantic similarity)
 FRAMES
❖ Frames provide an alternative approach to structuring
knowledge, overlapping many of the issues in semantic nets
❖ A FRAME is a cluster of knowledge about a certain concept,
including
❖ Its roles, subparts and their properties
❖ Typical properties of the object and relationships between
subparts
❖ Any other information about the concept as needed
 The logic of FRAMES
predicate defining the frame type
variable for subcomponent of the frame

type of objects filling roles (selectional

restrictions)
relation of subcomponent to main type
relationship between subcomponents
relations to other frames
 The logic of FRAMES

at first glance, we just have an alternate notation for a logic

∀ p PC(p) ⊃ ∃ y . KEYBOARD(y) & PART-OF(y,p)

But Frames typically also encode default reasoning and support

reasoning facilitated by being able to cluster knowledge
Frames define clusters of related knowledge
e.g., FRAMES may capture temporal/CAUSAL knowledge

what must be true before act

what must be true after act

how the act is performed

 Frames can provide context for
understanding

RELEVANT CONTEXT IS BUY FRAME: B1

“I went to buy a computer” Object role of B1 is C1:COMPUTER
Buyer role of B1 is ME (the speaker!)

“The salesman was very rude” Seller role of B1

Definite description resolving in the current context (i.e., the BUY frame)
 Specialized Inference using Frames
Planning is a common activity of intelligent agents: given
the current state of the world, a desired future state (the
goal) and a set of possible actions (encode as frame
structures), how could the goal be achieved?

CURRENT STATE GOAL ACTION LIBRARY

Owns(John1, Car1) Owns(Ag1, Car1)

Owns(Ag1, $1000)
Value(Car1, $1000)
 Specialized Inference: Planning
GOAL
1. find action with Effect that matches goal
{?Buyer/Ag1, ?Object/Car1} Owns(Ag1, Car1)
2. create new state with instantiated
preconditions
BUY
INTERMEDIATE STATE

Owns(?Seller, Car1)
Owns(Ag1, ?Money) ACTION LIBRARY
INITIAL STATE Value(Car1, ?Money)

Owns(John1, Car1) 3. Instantiate to

Owns(Ag1, $1000) initial state:
Value(Car1, $1000)
{?Seller/John1,
?Money/$1000}
 The Plan
GOAL
INITIAL STATE
Owns(John1, Car1) Owns(Ag1, Car1)
Owns(Ag1, $1000) BUY [Buyer=Ag1, Object=Car1,
Value(Car1, $1000) Money=$1000]

TO PERFORM THIS PLAN, AGENT AG1 ACTION LIBRARY

WOULD PERFORM THE
DECOMPOSITION:

GIVE(AG1, JOHN1, $1000)

GIVE(JOHN1, AG1, CAR1)
 Thoughts on Planning
❖ We just showed how an agent could reason that they could achieve
the goal state by performing a single action (buying the car)
❖ Typically, plans may be much more complicated!

❖ Most planning requires more complex solutions involving performing

multiple actions
❖ To represent and reason about plans, we needed to extend the KB to
allow different states (or contexts) of the world to be captured
❖ Reasoning may then be performed within a context (i.e., in the context where I
own the car I am happy, in the context where I don’t I am sad)
❖ A key part of an intelligent agent is to act in ways that leave it in desirable states!

WE WILL SEE THAT REASONING ABOUT ONE’S OWN AND OTHER

AGENTS STATES IS CRITICAL FOR UNDERSTANDING LANGUAGE