Module : knowledge representation and reasoning

AI and KR
What is KR & R
 Knowledge and Reasoning
Knowledge and Reasoning:
humans are very good at acquiring new information by
combining raw knowledge, experience with reasoning.
AI-slogan: “Knowledge is power” (or “Data is power”?)

Examples:
Medical diagnosis --- physician diagnosing a patient
infers what disease, based on the knowledge he/she
acquired as a student, textbooks, prior cases

Common sense knowledge / reasoning ---

common everyday assumptions / inferences
e.g., “lecture starts at four” infer pm not am;
when traveling, I assume there is some way to get from the
airport to the hotel.
Knowledge
There are different ways of interpreting knowledge by philosophers,
scientists, psychologists.

“The fact or condition of knowing something with familiarity

gained through experience or association.” (Webster’s
Dictionary, 1988)(Knowing something via seeing, hearing,
touching, feeling, and tasting.)

 “The fact or condition of being aware of something”.

 (Ex. Sun is hot, balls are round, sky is blue,…)

What is the role of knowledge in intelligent behavior?

Knowledge and Intelligence
THE KNOWLEDGE-ENGINEERING PROCESS

Five major activities

1. acquisition,
2. representation,
3. validation,
4. inference,
5. explanation and justification
Knowledge acquisition
 Knowledge acquisition involves the
acquisition of knowledge from human
experts, books, documents, sensors, or
computer files. It may be meta-knowledge

8
Representation
How can we represent knowledge in a machine?

We need a language to represent domain knowledge

There must be a method to use this knowledge

It is the inference mechanism

-Syntax and semantics of a language is very important

Logic is one such language

Logic is a formal language

Logic is a language with concrete rules

Logic should not to be confused with logical reasoning

Logics are languages, reasoning is a process (may use logic)
Syntax and Semantics
 Syntax
 Rules for constructing legal sentences in the logic
 Which symbols we can use (English: letters, punctuation)
 How we are allowed to combine symbols
 Semantics
 How we interpret (read) sentences in the logic
 Assigns a meaning to each sentence
● There are different ways of representing
knowledge e.g.
− Logical Representation
− Propositional logic / Sentential Logic
− Predicate logic
− Production rule
− Network Representation
− Semantic networks
− Extended semantic net,
− Structured Representation- Frames
− Conceptual dependency etc.
Knowledge Validation / Verification

● Validate and verify (by test cases) until the

quality is acceptable

● Testing results are usually shown to a domain

expert(s) to verify the accuracy

13
Knowledge Inference
 This activity involves the design of
software to enable the computer to make
inferences based on the stored knowledge
and the specifics of a problem.

14
Explanation and Justification

 This step involves the design and

programming of an explanation capability

15
What is Logic?
􀁺 Reasoning about the validity of arguments.
􀁺 An argument is valid if its conclusions follow logically from its
premises – even if the argument doesn’t actually reflect the real
world:
Logical Operators
􀁺 And Λ
􀁺 Or V
􀁺 Not
􀁺 Implies → (if… then…)
􀁺 Iff ↔ (if and only if)
Translating between English and Logic

􀁺 Facts and rules need to be translated into logical notation.

For example:
􀁺 It is Raining and it is Thursday:
􀁺 RΛT
􀁺 R means “It is Raining”, T means “it is Thursday”.
Truth Tables

􀁺 Tables that show truth values for all possible inputs to a logical
operator.

􀁺 For example:

A truth table shows the semantics of a logical operator.

We can produce truth tables for complex logical expressions,
which show the overall value of the expression for all
Possible combinations of variables:
 Tautology
The expression A v ¬A is a tautology.

This means it is always true, regardless of the

value of A.

􀁺 A tautology is true under any interpretation.

􀁺 An expression which is false under any interpretation is
contradictory.
Equivalence

􀁺 Two expressions are equivalent if they always have the same

logical value under any interpretation:
􀁺A Λ B ≡ B Λ A

􀁺 Equivalences can be proven by examining truth tables.

Some Useful Equivalences

􀁺 A Λ A ≡ A : Idempotence law
􀁺 A v A ≡ A : Idempotence law

􀁺 A Λ (B Λ C) ≡ (A Λ B) Λ C : associative law
􀁺 A v (B v C) ≡ (A v B) v C : associative law

􀁺 A Λ (B v C) ≡ (A Λ B) v (A Λ C) : Distributive law
􀁺 A Λ (A v B) ≡ A : Absorption law
􀁺 A v (A Λ B) ≡ A
􀁺 A Λ true ≡A AΛ false ≡ false
􀁺 A v true ≡ true A v false ≡ A
 Propositional Logic
Most of the statements we make are proposition.

Ex 1: Hari is hardworking.

2. If Hari is hardworking and Hari is intelligent then Hari scores good mark.

Any proposition in propositional logic can have two values: True or False

It constitutes objects and relations or functions

Example for function: Hardworking(Hari)== Hari is hardworking

 Terminology of the Propositional Calculus: Syntax
Proposition symbols:
P, Q, R, P1, P2, ... , Q1, Q2, ..., R1, R2, ...

Atomic proposition: a statement that does not specifically

contain sub-statements.

P: “It is raining.”
Q: “Jack is having tea.”

Compound proposition: A statement formed from one or more

atomic propositions using logical connectives.

P v Q: It is raining and Jack is having tea.

Propositional Logic: Semantics

How we interpret (read) sentences in the logic

Assigns a meaning to each sentence
Representation in Propositional Logic

Questions

1. It rains in July

2. If it rains today and one doesn’t carry umbrella then

he will be drenched.
Answers
Simple Logic Representation

Representation with notation

1. It rains in July : P

2. If it rains today and one doesn’t carry umbrella then he will be drenched
P QR
Rules: To simplify propositional sentences

1. A valid sentence or tautology is a sentence that

is True under all interpretations.
2. An inconsistent sentence or contradiction is a
sentence that is False under all interpretations.
3. Implication
4. Equivalence
5. Entailment: P entails Q, written P |= Q, means
that whenever P is True, so is Q.
6. Consistent
7. Logical Equivalence / Identical
8. Validity – tautology is valid
34
Tautology
Implication Equivalence
Implication Equivalence: Contrapositive Law
 Logical equivalent
• Two sentences are logically equivalent if they have the same truth
value for every setting of their propositional variables
• All the equivalence law will come under this
Bi-conditional Equivalence
Is the following sentences a tautology?
Contradiction/Absurdity
Contradiction
 Contingency
• A proposition that is neither a tautology nor
a contradiction is called a contingency.
Exercise
 Let p be a proposition. Indicate whether
the propositions are:
(A) tautologies
(B) contradictions or
(C) contingencies.
p∧¬p
p∨¬p
p∨p
50
Exercise - Answer
 p∧¬p …..Contradiction

 p ∨ ¬ p …..Tautology

 p∨p …..Contingency

51
Prove that q∧¬(p→q) is a
contradiction without truth tables

52
Prove that q∧¬(p→q) is a
contradiction:

53
Validity
 If the conclusion column of truth table is FALSE while
every premise column is TRUE, then the argument is
INVALID. Otherwise, the argument is VALID

Example: Where
Validity
 If ducks sink, then ducks are made of
small rocks.
 Ducks do sink.

55
Validity
 If ducks sink, then ducks are made of
small rocks.
 Ducks do sink.
 Conclusion: Therefore, ducks are made of
small rocks.
D: ducks sink
R: ducks are made of small rocks

56
 Pre = (D->R) ˄ D Pre -> R

T T

F T

F T

F T
57
Inference rules

 Rules of inference are templates for

building valid arguments.

58
Rules of Inference

59
60
 Modus Ponens: Example

Example:
 “If I study well, then I will get good mark in
math.” p  q
 “I study well.” p

 Conclusion: q
 Therefore , “I will get good mark in Math.”
61
Modus Tollens: Example

Example:
 “If I study well, then I will get good mark in
math.” p  q
 “I will not study.” ¬q

 Conclusion: ¬p
 Therefore , “It will not get good mark in math .”
62
Hypothetical Syllogism / Transitivity

Example:
 “If I study well, then I will get good mark in math” p  q
 “If I get good mark in math, I will get good CGPA.” q  r

 Conclusion: p  r
 Therefore , “If I study well, then I will get good CGPA .”

63
Disjunctive Syllogism / Elimination

Example:
 “I will study Math or I will study AI.” p V q
 “I will not study Math.” ¬p

 Conclusion: p  r
 Therefore , “I will study AI.”

64
 Addition

Example:
 “I will study math.” p

 Conclusion: p V q
 Therefore , “I will study math or I will watch a
movie”

65
Simplification

Example:
 “I will study Math and I will study Physics.” p  q

 Conclusion: q
 Therefore , “I will study Math”

66
Conjunction

Example:
 “I will study Math”. p
 “I will study Physics.” q

 Conclusion: p  q
 Therefore , “I will study Math and I will study Physics”

67
 Resolution

Example:
 “I will study not math or I will study Physics”. ¬p V r
 “I will study math or I will study Chemistry”. p V q

 Conclusion: q V r
 Therefore , “I will study Chemistry or I will study
Physics”
68
Contradiction
 Sita falsely stated
“It was not a mountain” ¬p

Conclusion :
“ It was a mountain” p

69
Valid Arguments – Exercise1

70
Valid Arguments – Exercise2
With these hypotheses:
 “It is not sunny this afternoon and it is colder than yesterday.”
 “If we go swimming then it is sunny”
 “If we do not go swimming, then we will take a canoe trip.”
 “If we take a canoe trip, then we will be home by sunset.”

Using the inference rules, construct a valid

argument for the conclusion:
“We will be home by sunset.”

72
Proof
 p : “It is sunny this afternoon.”
 q : “It is colder than yesterday.”
 r : “We will go swimming.”
 s : “We will take a canoe trip.”
 t : “We will be home by sunset.”

Translation into propositional logic:

Hypotheses:
¬ p  q, r  p, ¬ r  s, s  t
Conclusion: t
73
 1. It is not sunny this afternoon and it is colder than yesterday.
2. If we go swimming then it is sunny.
3. If we do not go swimming then we will take a canoe trip.
4. If we take a canoe trip then we will be home by sunset.
5. We will be home by sunset

p It is sunny this afternoon 1. p  q

q It is colder than yesterday 2. r  p
r We go swimming
3. r  s
s We will take a canoe trip
t We will be home by sunset (the conclusion) 4. s  t
5. t

propositions hypotheses

74
Using the rules of inference to build arguments
1. p  q
p It is sunny this afternoon
q It is colder than yesterday
2. r  p
r We go swimming 3. r  s
s We will take a canoe trip 4. s  t
t We will be home by sunset (the conclusion)
5. t
Step
Step Reason
Reason
Reason
1. 
pp  qq Hypothesis
Hypothesis
Hypothesis
2. p Simplifica
Simplification
tionusing
using(1)
(1)
3. rp Hypothesis
Hypothesis
4. r Modus
Modustollens
tollens using
using(2)
(2)and
and(3)
(3)
5. r  s Hypothesis
6. s Modus ponens using (4) and (5)
7. s t Hypothesis
8. t Modus ponens using (6) and (7)
75
Predicate Logic
82
Predicate Logic or Quantifier Logic

There are many kinds of arguments which cannot be proven in

propositional logic and which require the notion of quantifiers.

Example:
All men are mortal

Socrates is a man

Therefore Socrates is mortal

All sentences are simple sentences. There are no propositional

connectives. But it’s a valid argument.
Predicate Logic or Quantifier Logic

Apart from propositional logic It's the internal structure of these

propositions that make the reasoning valid.

A more expressive logic is needed

Predicate logic (occasionally referred to as First-order
logic (FOL))

Internal structure of the sentences:

• The meaning of certain words or phrases such as “ALL”

• Certain properties of objects “mortal”
• Relations “is a”
Predicate Logic
 Objects
 E.g. people, houses, doreen, gord, william, 32
 Relations
Blue, oval, even, large
 Functions
 that hold among sets of objects
 E.g. father-of, best-friend
 Variables
 For statements about unidentified objects or general statements
 E.g. x, y, z, …
 Symbols: User provides
 Constant symbols, which represent individuals in the world
 Mary
 3
 Green

 Function symbols, which map individuals to

individuals
 father-of(Mary) = John
 color-of(Sky) = Blue
 Predicate symbols, which map individuals to truth
values
 greater(5,3)
 green(Grass)
 color(Grass, Green)
FOL Provides
• Variable symbols
– E.g., x, y
• Connectives
– Same as in PL: not (), and (), or (), implies (),
if and only if (biconditional )
• Quantifiers
– Universal x or (Ax)
– Existential x or (Ex)
Sentences are built from terms and atoms
 A term (denoting a real-world individual) is a constant symbol, a
variable symbol, or an n-place function of n terms.
x and f(x1, ..., xn) are terms, where each xi is a term.
A term with no variables is a ground term

 An atomic sentence is formed from objects and relations

e.g. brother(John, Jaison)

 A complex sentence is formed from atomic sentences connected by

the logical connectives:
P, PQ, PQ, PQ, PQ where P and Q are sentences

 A quantified sentence adds quantifiers  and 

 A well-formed formula (wff) is a sentence containing no “free”

variables. That is, all variables are “bound” by universal or existential
quantifiers.
(x)P(x,y) has x bound as a universally quantified variable, but y is free.
Predicate
 A predicate becomes a proposition when specific values
are assigned to the variables. P(x1, x2, ..., xn) is called a
predicate of n variables or n arguments.

 X lives in Y is predicate
 Mary lives in Austin.
 P(Mary, Austin)
 lives(Mary, Austin)

89
Propositional Vs Predicate Logic
 Propositional logic is logic that includes sentence
represented by letters (P,Q,R)and logical connectives, but
not quantifiers.
Eg: "Socrates is a Man“ represented as “P".

 In predicate logic, instead of single propositional symbol

denote the entire sentence we can create a predicate.
 We symbolize subject and predicate separately. "Socrates
is a Man" would be represented “m(s)“or Man (Socrates)

90
Propositional Vs Predicate Logic
 ƒ
Quantifiers: The important difference is that we can use
predicate logic to say something about a set of objects, by
introducing the universal quantifier ("∀"), the existential
quantifier ("∃") and variables ("x", "y" or "z").

 Universal quantifier—makes statements about every object. ƒ

 Existential quantifier—makes a statement about some
object in the universe without naming it.

 Eg: "Everything is green" as "∀G(x)" or "Something is blue"

as "∃B(x)".

91
Predicate Calculus - Exercise
 Today is wet
 John likes apples
 It rained on Tuesday
 Ben is a good cat.
 If it does not rain on Monday, Rama will go to
the mountains.
 Mary is a child who takes Coconut-crunchy

97
Predicate Calculus

 Properties: Wet(Today)
 Relations: likes(John, Apples)
 weather(Tuesday, rain)
/ rain(Tuesday)
 Good_cat (Ben)
 ¬ Weather (rain, Monday) → go (Rama, mountains)
 Child(Mary)  Takes(Mary, Coconut-crunchy )

98
Predicate Logic/ calculus
 All basketball players are tall.
 Some people like chocolates are tall.

99
Predicate Logic/ calculus
 All basketball players are tall.
∀x (basketball-players (x) → tall (x))
 Some people like chocolates.
∃ x likes ((x, chocolates)^tall(x))

100
The following statements associate with
persons
 a. Someone is sleeping
 b. No one is sleeping
 c. Everyone is sleeping
 d. Not everyone is sleeping

101
 There is some entity x such
1. Someone is sleeping that x is a person and x is
Let S = is sleeping, P = person sleeping

 ∃x (P(x) ∧ S(x))
2. No one is sleeping It is not the case that there is
Let S = is sleeping, P = person some entity x such that x is a
person and x is sleeping
 ¬∃x (P(x) ∧ S(x))
3. Everyone is sleeping
Let S = is sleeping, P = person For every entity x, if x is a
person, then x slept
 ∀x (P(x) → S(x))
4. Not everyone is sleeping
Let S = is sleeping, P = person:
It is not the case that for every
 ¬∀x (P(x) → S(x)) entity x, if x is a person, then x
slept

102
 Some girl liked some boy
∃x ∃y [ (GIRL(x) ∧ BOY(y)) ∧ LIKE(x,y) ]
 Every girl liked every boy
∀x ∀y [ (GIRL(x) ∧ BOY(y)) → LIKE(x,y) ]

103
 a. Some dragon is sleeping and twitching
∃x [D(x) ∧ (S(x) ∧ T(x))]
 b. No dragon is sleeping and twitching
¬∃x [D(x) ∧ (S(x) ∧ T(x))]
 c. Every dragon is sleeping and twitching
∀x [D(x) → (S(x) ∧ T(x))]
 d. Not every dragon is sleeping and
twitching
¬∀x [D(x) → (S(x) ∧ T(x))]
104
Exercises
 All saints were Hindus
 All Hindus were either loyal to Ravana or hated him
 Everyone is loyal to someone

105
Exercises
 All saints were Hindus
 All Hindus were either loyal to Ravana or hated him
 Everyone is loyal to someone

 ∀ x: saints(x) → hindus(x)
 ∀x: hindus(x) → loyal to (x, Ravana) ∨ hate (x,
Ravana)
 ∀x: ∃y: loyal to (x, y)

106
FOPL to sentences
Theorems of logical operations
Laws of Inference
Inference Rule
Note: Find the particular x that satisfies the condition and then eliminate “there exists”
Inference Rule
Combining rules of proposition and
quantified statements: Lifting

117
Arguments

 We have the two premises:

 “All men are mortal.”
 “Socrates is a man.”

And the conclusion:

 “Socrates is mortal.”

118
Express the Argument
 We can express the premises (above the line)
and the conclusion (below the line) in predicate
logic as an argument:

119
Unification -Unify algorithm takes two sentences and returns
a unifier/substitution that makes the two sentences look identical

 Suppose we have a query,

Knows(John,x): Whom does John know?

Find all the sentences in KB that unify with Knows(John,x)

Conversion to CNF
Exercise: Conversion to CNF
Forward chaining and Backward chaining: Inference Mechanism
Change the real worlds scenario to Rules. This set of Rules will be
there in knowledge base(KB).

Forward chaining: From the available set of rule in KB, start from
simple atomic sentences and infer a new fact using inference rules
like, Modus Ponen.

Forward chaining can be applied to only rule based system.

Rule based system is nothing but the KB with all the rule.

Normally rule will be: if cond then action

Query(Database) (P)

Inference Engine Result (Q)

KB(PQ)

Inference Rule/FC/BC
Forward chaining and Backward chaining: Inference Mechanism

Given set of rules in the KB. Apply forward chaining to infer Q

Given sentences:

Facts and Rules in the database

Bob is a buffalo
Pat is a pig
Steve is a slug

Inference by forward chaining: Faster(Bob, Steve)

Exercise - 1
Show that the premises:
 Some student in Section A of the course has not
read the book.
 Everyone in Section A of the course passed the
exam.

Conclusion
 Someone who passed the exam has not read
the book.
132
  A(x): “x is in Section A of the course"
 B(x): “x reads the book"
 P(x): “x passed the exam.“

Hypotheses:
 ∃ x(A(x) ^ ¬ B(x))
 ∀ x(A(x)  P(x)).
 Conclusion: ∃ x(P(x) ^ ¬ B(x)).

133
Semantic Networks
 A knowledge representation technique that
focuses on the relationships between objects

 A directed graph is used to represent a

semantic network or net

137
 Semantic Network
 The syntax of semantic net is simple. It is a network of
labeled nodes and links.
− It’s a directed graph with nodes corresponding to concepts, facts,
objects etc. and
− arcs showing relation or association between two concepts.
 The commonly used links in semantic net are of the
following types.
- isa  subclass of entity (e.g., child hospital is subclass of
hospital)
- Represented by rectangle
- inst  particular instance of a class (e.g., India is an instance of
country)
- Represented by rectangle
- prop  property link (e.g., property of dog is ‘bark)
- Represented by ellipse, and connected with dotted arrow

138
Representation of Knowledge in Sem
Net
“Every human, animal and bird is living
thing who breathe and eat. All birds can
fly. All man and woman are humans
who have two legs. Cat is an animal
and has a fur. All animals have skin and
can move. Giraffe is an animal who is
tall and has long legs. Parrot is a bird
and is green in color”.
139
Representation
Semantic Net
in Semantic Net
breathe, eat
Living_thing prop
isa isa
two legs isa fly
Human Animal Bird
isa isa inst inst inst
prop green
Man Woman Giraffe Cat Parrot
prop prop prop
inst fur
john skin, move tall, long legs

140
Module 5: Uncertainty and knowledge Reasoning
Uncertainty
Agents almost have never access to the whole truth about their
environment

Agents have to act under uncertainty.

Uncertainty:
uncertain input
uncertain knowledge
uncertain output
Why we need reasoning under uncertainty?

Consider the following rule:

We cant say that the above said rule is correct. “All patients with toothache
has cavity”. It is not correct. Some person may have gum disease or other
problems.

In order to make the rule true, we need to add almost many list of possible
causes. Hence trying FOL for domain like medical diagnosis fails for three
reasons.

- Specification becomes too large

- theoretical ignorance
- practical ignorance
When we are not able to set of rules exactly and completely then we will go for
probability.

Probability theory deals with uncertainty and assigns each sentences a

numerical degree of belief between 0 and 1.

Ex:
P(Weather = Sunny) = 0.1
P(Weather = Rain) = 0.7
P(Weather = Snow) = 0.2
Weather – random variable

P(Weather) = (0.1, 0.7, 0.2) – probability distribution

 Axioms of probability

1. All probability are between 0 and 1: 0  P(A)  1

1. P(true) = 1 and P(false) = 0

2. P(A  B) = P(A) + P(B) - P(A  B)

3. P(~A) = 1 – P(A)
Prior probability

Consider the random variables,

cavity={true, false}
weather={sunny, rain, cloudy, snow}

Prior or unconditional probability,

P(cavity=true)=0.1
P(weather=sunny)=0.72

Probability distribution gives values of all possible assignments:

P(weather)={0.72, 0.1, 0.08, 0.1}(normalized i.e, sums to 1)
Joint probability distribution

Joint probability distribution for a set of random variables gives the

probability of every atomic event (sample point) on those variables

P( Weather, Cavity) = a 4 X 2 matrix of values:

 Joint probability distribution
With two variables

With three variables

 Joint probability distribution

With four variables

 Joint probability distribution
With five variables
Probabilistic reasoning: inference by enumeration
Probabilistic reasoning: inference by enumeration
Bayes’ Rule

The condition probability of the occurrence of A

if event B occurs

P(A|B) = P(A  B) / P(B)

This can be written also as:

P(A  B) = P(A|B) * P(B)
P(A  B) = P(B|A) * P(A)

Hence P(B|A) = P(A|B) * P(B)

----------------
P(A)
But,
If events A and B are independent then,

P(A|B) = P(A), P(B|A) = P(B)

Hence,

P(A  B) = P(A) * P(B)

Conditional Probability and Product Rule
 Bayesian Belief Network

1. Represent dependencies among random variables

2. Give a short specification of conditional probability distribution

3. Many random variables are conditionally independent

4. Graphical representation

5. DAG – causal relationships among random variables

6. Allows inferences based on the network structure

Bayesian network
P(Cough, Fever , Smoke, Flu) =

P(Cough| ( Smoke^ Flu))* P(Fever|Flu)*P( Smoke)*

P(Flu)
Bayesian network: For Burglary
Calculate the probability that the alarm has sounded but neither a
burglary nor an earthquake has occurred and both Mary and John
call.

P(J, M, A, B, E ) =

P(J|A)* P(M|A)*P(A|(B E ))*P(B) *P(E)=

0.9 * 0.7 * 0.001 * 0.999 * 0.998 = 0.00062

P(J,  M,  A, B, E ) =

P(J, M, A)
Some Properties of BN

BN is a DAG

Nodes which have no parents have prior probabilities

All other nodes have conditional probabilities

A BN in which there is only one directed path between any pair of

nodes is a poly tree
Definition of AI

It is a branch of computer science

which deals with helping machines
finds solutions to complex
problems in a more human like
fashion.

This generally involve borrowing

characteristics from human
intelligence and applying them as
algorithms in a computer friendly
way
AI and ML
Machine Learning Types
Machine Learning Types
Types of Machine Learning and its Applications
Machine Learning Healthcare Applications
Machine Learning Algorithms
Regression
Classification
Clustering
Supervised Learning Algorithms
Logistic Regression

❖ Name is somewhat misleading. Really a technique for

classification, not regression.
– “Regression” comes from fact that we fit a
linear model to the feature space.

❖ Involves a more probabilistic view of classification.

Math behind Logistic Regression
Math behind Logistic Regression
Math behind Logistic Regression
 The Perceptron
By Rosenblatt in 1959

Processing Elements in NN
An ANN consists of perceptrons. Each of the perceptrons receives
inputs, processes inputs and delivers a single output.
 Perceptron

Summation Unit Thresholding Unit

x0=1
x1 w1
w0
w2
x2 o
n
. I= i=0 wi xi
. wn
1 if I> θ
. O=f(I)=
0 otherwise

xn
xi = input
wi =weight
O= Output

36
The Perceptron: Threshold Activation Function

Step Threshold

1 if I>θ
O=f(I)=
0 otherwise
Perceptron and Linearly Separable Problem

Perceptron can handle task which are linearly separable

Example: If the two different sets in 2D are linearly separable

Class1 Class2
Linearly Separable
XOR Problem: Not Linearly Separable

We could however construct multiple layers of perceptrons to get around

this problem.
Nonlinear Activation Functions

• Sigmoid Neuron unit function

1
O = f(I) =
1 + e −I
Sigmoid
K Nearest Neighbor
How does KNN algorithm work”
At k=3, we classify the new variable as square
At k=7, we classify the new variable as triangle
KNN Distance Metrics
Decision Tree

The algorithm can be summarized as :

1. At each stage (node), pick out the best feature as the test condition.

2. Now split the node into the possible outcomes (internal nodes).

3. Repeat the above steps till all the test conditions have been exhausted
into leaf nodes.
Decision Tree

For the right decision tree calculate entropy and information gain.
Decision Tree

Entropy calculation
Example Dataset
Decision Tree
Decision Tree
Decision Tree

Entropy for the whole dataset,

Entropy(Buy Computer)

9 9 5 5
Info( D) = I (9,5) = − log 2 ( ) − log 2 ( ) =0.940
14 14 14 14
 Decision Tree

Tree Construction
Entropy(Buy computer, Age)

age pi ni I(pi, ni)

young 2 3 0.971
middle 4 0 0
senior 3 2 0.971

5 4
Infoage ( D ) = I ( 2,3) + I ( 4,0)
14 14
5
+ I (3,2) = 0.694
14

Gain (age) = Info( D) − Infoage ( D) = 0.246

Similarly,

Gain(income) = 0.029
Gain( student ) = 0.151
Gain(credit _ rating ) = 0.048
Unsupervised Learning Algorithms
K-Means Clustering
K Means Algorithm
Let the Input: Data Points (V1,V2,…..Vn)

Step 1: Pick ‘K’ data points to the centroids randomly (Ck)

Step 2: Using Euclidean distance assign each data point to closest centroid Cj
arg min dist(Vi,Cj)

Step 3: For each cluster j = 1to k

3.1 find new centroid, Cj(new) = 1/nj Σvi

Step 4: Repeat step 2 and 3 until none of the cluster assignment change
K-Means Clustering Algorithm
K-Means Clustering Algorithm Cntd..
K-Means Clustering Algorithm Cntd..
K-Means Clustering Algorithm Cntd..
Choosing the right number of clusters in Kmeans Clustering

Elbow Method

Randomly select K=2 and compute SSE:

Repeat SSE for different value of K and plot it as given below.
From the above graph, it shows that as we increase the number of cluster,
the error got reduced. Find the elbow point.