Professional Documents
Culture Documents
Unit I - Basic Concepts of Algorithms
Unit I - Basic Concepts of Algorithms
Algorithms
Introduction
algorithm
Copyright © 2007 SSN College Of Engineering. CS1201- “Introduction to the Design & Analysis of Algorithms” 1-3
Notion of algorithm
problem
algorithm
Algorithmic solution
Copyright © 2007 SSN College Of Engineering. CS1201- “Introduction to the Design & Analysis of Algorithms” 1-4
Algorithms
I It is not depended on programming language, machine.
I Are mathematical entities, which can be thought of as
running on some sort of idealized computer with an infinite
random access memory
I Algorithm design is all about the mathematical theory
behind the design of good programs.
Copyright © 2007 SSN College Of Engineering. CS1201- “Introduction to the Design & Analysis of Algorithms” 1-5
Contd…
I Algorithmic is a branch of computer science that consists
of designing and analyzing computer algorithms
I The “design” pertain to
• The description of algorithm at an abstract level by means of a
pseudo language, and
• Proof of correctness that is, the algorithm solves the given problem
in all cases.
I The “analysis” deals with performance evaluation
(complexity analysis).
I Random Access Machine (RAM) model
Copyright © 2007 SSN College Of Engineering. CS1201- “Introduction to the Design & Analysis of Algorithms” 1-6
Why study algorithm design?
I Programming is a very complex task, and there are a number of aspects of
programming hat make it so complex. The first is that most programming
projects are very large, requiring the coordinated efforts of many people.
(This is the topic a course like software engineering.)
I The next is that many programming projects involve storing and accessing
large quantities of data efficiently. (This is the topic of courses on data
structures and databases.)
I The last is that many programming projects involve solving complex
computational problems, for which simplistic or naive solutions may not be
efficient enough. The complex problems may involve numerical data (the
subject of courses on numerical analysis), but often they involve discrete
data. This is where the topic of algorithm design and analysis is important.
I The focus of this course is on how to design good algorithms, and how to
analyze their efficiency. This is among the most basic aspects of good
programming.
Copyright © 2007 SSN College Of Engineering. CS1201- “Introduction to the Design & Analysis of Algorithms” 1-7
Contd…
I Algorithms help us to understand scalability.
I •Performance often draws the line between what is feasible and
what is impossible.
I •Algorithmic mathematics provides a languagefor talking
about program behavior.
I •Performance is the currencyof computing.
I •The lessons of program performance generalize to other
computing resources.
I •Speed is fun!
Copyright © 2007 SSN College Of Engineering. CS1201- “Introduction to the Design & Analysis of Algorithms” 1-8
Random Access Machine
I A Random Access Machine (RAM) consists of: . . .
• a fixed program input tape
• an unbounded memory
• a read-only input tape
• a write-only output tape Program
I Each memory register can hold an arbitrary integer (*)
...
I Each tape cell can hold a single symbol from a finite alphabet Σ Memory
output tape
. . .
I Instruction set:
• x ← y, x ← y {+, −, *, div, mod} z
• goto label
I Addressing modes:
• if y {<, ≤, =, ≥ ,> , ≠} z goto label • x may be direct or indirect
• x ← input, output ← y reference
• halt • y and z may be constants,
direct or indirect references
Copyright © 2007 SSN College Of Engineering. CS1201- “Introduction to the Design & Analysis of Algorithms” 1-9
Contd…
I Why analyze algorithms?
• evaluate algorithm performance
• compare different algorithms
I Analyze what about them?
• running time, memory usage, solution quality
• worst-case and “typical” case
I Computational complexity
• understanding the intrinsic difficulty of computational problems -
classifying problems according to difficulty
• algorithms provide upper bound
• to show problem is hard, must show that any algorithm to solve it
requires at least a given amount of resources
• transform problems to establish “equivalent” difficulty
Copyright © 2007 SSN College Of Engineering. CS1201- “Introduction to the Design & Analysis of Algorithms” 1-10
Example of computational problem:
sorting
I Statement of problem:
• Input: A sequence of n numbers <a1, a2, …, an>
I Algorithms:
• Selection sort
• Insertion sort
• Merge sort
• (many others)
Copyright © 2007 SSN College Of Engineering. CS1201- “Introduction to the Design & Analysis of Algorithms” 1-11
Selection Sort
I Input: array a[1],…,a[n]
I Algorithm:
for i=1 to n
swap a[i] with smallest of a[i],…a[n]
Copyright © 2007 SSN College Of Engineering. CS1201- “Introduction to the Design & Analysis of Algorithms” 1-13
Contd..
Copyright © 2007 SSN College Of Engineering. CS1201- “Introduction to the Design & Analysis of Algorithms” 1-14
Contd…
Copyright © 2007 SSN College Of Engineering. CS1201- “Introduction to the Design & Analysis of Algorithms” 1-15
Contd…
I Worst-case: (usually)
I Average-case: (sometimes)
I Best-case: (bogus)
Copyright © 2007 SSN College Of Engineering. CS1201- “Introduction to the Design & Analysis of Algorithms” 1-16
Contd…
Copyright © 2007 SSN College Of Engineering. CS1201- “Introduction to the Design & Analysis of Algorithms” 1-17
Contd…
Copyright © 2007 SSN College Of Engineering. CS1201- “Introduction to the Design & Analysis of Algorithms” 1-18
Contd…
Copyright © 2007 SSN College Of Engineering. CS1201- “Introduction to the Design & Analysis of Algorithms” 1-19
Contd…
Copyright © 2007 SSN College Of Engineering. CS1201- “Introduction to the Design & Analysis of Algorithms” 1-20
Some Well-known Computational
Problems
I Sorting
I Searching
I Shortest paths in a graph
I Minimum spanning tree
I Primality testing
I Traveling salesman problem
I Knapsack problem
I Chess
I Towers of Hanoi
I Program termination
Copyright © 2007 SSN College Of Engineering. CS1201- “Introduction to the Design & Analysis of Algorithms” 1-21
What is an algorithm?
I Recipe, process, method, technique, procedure, routine,…
with following requirements:
1. Finiteness
I terminates after a finite number of steps
2. Definiteness
I rigorously and unambiguously specified
3. Input
I valid inputs are clearly specified
4. Output
I can be proved to produce the correct output given a valid input
5. Effectiveness
I steps are sufficiently simple and basic
Copyright © 2007 SSN College Of Engineering. CS1201- “Introduction to the Design & Analysis of Algorithms” 1-22
Euclid’s Algorithm
Problem: Find gcd(m,n), the greatest common divisor of two
nonnegative, not both zero integers m and n
Examples: gcd(60,24) = 12, gcd(60,0) = 60, gcd(0,0) = ?
Copyright © 2007 SSN College Of Engineering. CS1201- “Introduction to the Design & Analysis of Algorithms” 1-23
Two descriptions of Euclid s
algorithm
Step 1 If n = 0, return m and stop; otherwise go to Step 2
Step 2 Divide m by n and assign the value fo the remainder to r
Step 3 Assign the value of n to m and the value of r to n. Go to
Step 1.
while n ≠ 0 do
r ← m mod n
m← n
n←r
return m
Copyright © 2007 SSN College Of Engineering. CS1201- “Introduction to the Design & Analysis of Algorithms” 1-24
Other methods for computing
gcd(m,n)
Consecutive integer checking algorithm
Step 1 Assign the value of min{m,n} to t
Step 2 Divide m by t. If the remainder is 0, go to Step 3;
otherwise, go to Step 4
Step 3 Divide n by t. If the remainder is 0, return t and stop;
otherwise, go to Step 4
Step 4 Decrease t by 1 and go to Step 2
Copyright © 2007 SSN College Of Engineering. CS1201- “Introduction to the Design & Analysis of Algorithms” 1-25
Other methods for gcd(m,n) [cont.]
Middle-school procedure
Step 1 Find the prime factorization of m
Step 2 Find the prime factorization of n
Step 3 Find all the common prime factors
Step 4 Compute the product of all the common prime factors
and return it as gcd(m,n)
Is this an algorithm?
Copyright © 2007 SSN College Of Engineering. CS1201- “Introduction to the Design & Analysis of Algorithms” 1-26
Sieve of Eratosthenes
Input: Integer n ≥ 2
Output: List of primes less than or equal to n
for p ← 2 to n do A[p] ← p
for p ← 2 to ⎣n⎦ do
if A[p] ≠ 0 //p hasn’t been previously eliminated from the list
j ← p* p
while j ≤ n do
A[j] ← 0 //mark element as eliminated
j←j+p
Example: 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20
Copyright © 2007 SSN College Of Engineering. CS1201- “Introduction to the Design & Analysis of Algorithms” 1-27
Why study algorithms?
I Theoretical importance
I Practical importance
Copyright © 2007 SSN College Of Engineering. CS1201- “Introduction to the Design & Analysis of Algorithms” 1-28
Basic Issues Related to
Algorithms
I How to design algorithms
I How to analyze algorithms
Copyright © 2007 SSN College Of Engineering. CS1201- “Introduction to the Design & Analysis of Algorithms” 1-29
Algorithm Problem solving
I Understand the problem
I Decide on:
• Computational Means
• Exact vs Approximate solving
• Data structures
• Algorithm design techniques
I Design an algorithm
I Prove correctness
I Analyse an algorithm
I Code an algorithm
Copyright © 2007 SSN College Of Engineering. CS1201- “Introduction to the Design & Analysis of Algorithms” 1-30
Understand the problem
I First important step
I What must be done rather than how to do it
I Input-instance of the problem
I Ascertaining the capability of computational device
I Generic one-processor, random-access-machine (RAM)-
instructions are executed one by one.
I computer resources- memory, BW, CPU measures
efficiency of algorithm.
Copyright © 2007 SSN College Of Engineering. CS1201- “Introduction to the Design & Analysis of Algorithms” 1-31
Choosing between exact and approximate problem solving
Copyright © 2007 SSN College Of Engineering. CS1201- “Introduction to the Design & Analysis of Algorithms” 1-32
Deciding on appropriate data
structures
I Algorithm + Data Structures = Program
Copyright © 2007 SSN College Of Engineering. CS1201- “Introduction to the Design & Analysis of Algorithms” 1-33
Algorithmic design technique
I An algorithm design technique is a general approach to
solving problems mathematically that is applicable to a
variety of problems from different areas of computing.
I Algorithm design technique provides guidance for
designing algorithms for new problems or problems for
which there is no satisfactory algorithm. It makes it
possible to classify algorithm according to underlying
design idea.
Copyright © 2007 SSN College Of Engineering. CS1201- “Introduction to the Design & Analysis of Algorithms” 1-34
Algorithm design
techniques/strategies
Copyright © 2007 SSN College Of Engineering. CS1201- “Introduction to the Design & Analysis of Algorithms” 1-35
Analysis of algorithms
I How good is the algorithm?
• time efficiency
• space efficiency
Copyright © 2007 SSN College Of Engineering. CS1201- “Introduction to the Design & Analysis of Algorithms” 1-36
Important problem types
I sorting
I searching
I string processing
I graph problems
I combinatorial problems
I geometric problems
I numerical problems
Copyright © 2007 SSN College Of Engineering. CS1201- “Introduction to the Design & Analysis of Algorithms” 1-37
Fundamental data structures
I list I graph
• array I tree
• string
I stack
I queue
I priority queue
Copyright © 2007 SSN College Of Engineering. CS1201- “Introduction to the Design & Analysis of Algorithms” 1-38
Analysis of algorithms
I Issues:
• correctness
• time efficiency
• space efficiency
• optimality
I Approaches:
• theoretical analysis
• empirical analysis
Copyright © 2007 SSN College Of Engineering. CS1201- “Introduction to the Design & Analysis of Algorithms” 1-39
y
efficiency
Time efficiency is analyzed by determining the number of
repetitions of the basic operation as a function of input size
T(n) ≈ copC(n)
running time execution time Number of times
for basic operation basic operation is
executed
Copyright © 2007 SSN College Of Engineering. CS1201- “Introduction to the Design & Analysis of Algorithms” 1-40
Input size and basic operation
examples
Visiting a vertex or
Typical graph problem #vertices and/or edges
traversing an edge
Copyright © 2007 SSN College Of Engineering. CS1201- “Introduction to the Design & Analysis of Algorithms” 1-41
p y
efficiency
I Select a specific (typical) sample of inputs
Copyright © 2007 SSN College Of Engineering. CS1201- “Introduction to the Design & Analysis of Algorithms” 1-42
, g ,
case
For some algorithms efficiency depends on form of input:
Copyright © 2007 SSN College Of Engineering. CS1201- “Introduction to the Design & Analysis of Algorithms” 1-43
Example: Sequential search
• Worst case
• Best case
Copyright © 2007 SSN College Of Engineering. CS1201- “Introduction to the Design & Analysis of Algorithms” 1-44
Types of formulas for basic operation s
count
I Exact formula
e.g., C(n) = n(n-1)/2
Copyright © 2007 SSN College Of Engineering. CS1201- “Introduction to the Design & Analysis of Algorithms” 1-45
Order of growth
I Most important: Order of growth within a constant multiple
as n→∞
I Example:
• How much faster will algorithm run on computer that is
twice as fast?
Copyright © 2007 SSN College Of Engineering. CS1201- “Introduction to the Design & Analysis of Algorithms” 1-46
Values of some important functions as
n→∞
Copyright © 2007 SSN College Of Engineering. CS1201- “Introduction to the Design & Analysis of Algorithms” 1-47
Asymptotic order of growth
A way of comparing functions that ignores constant factors and
small input sizes
Copyright © 2007 SSN College Of Engineering. CS1201- “Introduction to the Design & Analysis of Algorithms” 1-48
Big-oh
Copyright © 2007 SSN College Of Engineering. CS1201- “Introduction to the Design & Analysis of Algorithms” 1-49
Big-omega
Copyright © 2007 SSN College Of Engineering. CS1201- “Introduction to the Design & Analysis of Algorithms” 1-50
Big-theta
Copyright © 2007 SSN College Of Engineering. CS1201- “Introduction to the Design & Analysis of Algorithms” 1-51
g g g
definition
Definition: f(n) is in O(g(n)) if order of growth of f(n) ≤ order
of growth of g(n) (within constant multiple),
i.e., there exist positive constant c and non-negative integer
n0 such that
f(n) ≤ c g(n) for every n ≥ n0
Examples:
I 10n is O(n2)
I 5n+20 is O(n)
Copyright © 2007 SSN College Of Engineering. CS1201- “Introduction to the Design & Analysis of Algorithms” 1-52
p p y p
growth
I f(n) ∈ O(f(n))
Copyright © 2007 SSN College Of Engineering. CS1201- “Introduction to the Design & Analysis of Algorithms” 1-53
limits
Examples:
• 10n vs. n2
• n(n+1)/2 vs. n2
Copyright © 2007 SSN College Of Engineering. CS1201- “Introduction to the Design & Analysis of Algorithms” 1-54
p S g
formula
L’Hôpital’s rule: If limn→∞ f(n) = limn→∞ g(n) = ∞ and
the derivatives f´, g´ exist, then
Example: 2n vs. n!
Copyright © 2007 SSN College Of Engineering. CS1201- “Introduction to the Design & Analysis of Algorithms” 1-55
Orders of growth of some important
functions
I All logarithmic functions loga n belong to the same class
Θ(log n) no matter what the logarithm’s base a > 1 is
I order log n < order nα (α>0) < order an < order n! < order nn
Copyright © 2007 SSN College Of Engineering. CS1201- “Introduction to the Design & Analysis of Algorithms” 1-56
y p y
classes
1 constant
log n logarithmic
n linear
n log n n-log-n
n2 quadratic
n3 cubic
2n exponential
n! factorial
Copyright © 2007 SSN College Of Engineering. CS1201- “Introduction to the Design & Analysis of Algorithms” 1-57
y
algorithms
General Plan for Analysis
Copyright © 2007 SSN College Of Engineering. CS1201- “Introduction to the Design & Analysis of Algorithms” 1-58
Useful summation formulas and rules
Σl≤i≤u1 = 1+1+…+1 = u - l + 1
In particular, Σl≤i≤u1 = n - 1 + 1 = n ∈ Θ(n)
Copyright © 2007 SSN College Of Engineering. CS1201- “Introduction to the Design & Analysis of Algorithms” 1-59
Example 1: Maximum element
Copyright © 2007 SSN College Of Engineering. CS1201- “Introduction to the Design & Analysis of Algorithms” 1-60
p q
problem
Copyright © 2007 SSN College Of Engineering. CS1201- “Introduction to the Design & Analysis of Algorithms” 1-61
p
multiplication
Copyright © 2007 SSN College Of Engineering. CS1201- “Introduction to the Design & Analysis of Algorithms” 1-62
Example 4: Gaussian
elimination
Algorithm GaussianElimination(A[0..n-1,0..n])
//Implements Gaussian elimination of an n-by-(n+1) matrix A
for i ← 0 to n - 2 do
for j ← i + 1 to n - 1 do
for k ← i to n do
A[j,k] ← A[j,k] - A[i,k] ∗ A[j,i] / A[i,i]
Copyright © 2007 SSN College Of Engineering. CS1201- “Introduction to the Design & Analysis of Algorithms” 1-63
p g y
digits
Copyright © 2007 SSN College Of Engineering. CS1201- “Introduction to the Design & Analysis of Algorithms” 1-64
Copyright © 2007 SSN College Of Engineering. CS1201- “Introduction to the Design & Analysis of Algorithms” 1-66
Solving the recurrence for M(n)
M(n) = M(n-1) + 1, M(0) = 0
Copyright © 2007 SSN College Of Engineering. CS1201- “Introduction to the Design & Analysis of Algorithms” 1-67
p
Puzzle
1 3
Copyright © 2007 SSN College Of Engineering. CS1201- “Introduction to the Design & Analysis of Algorithms” 1-68
g
moves
M(n) = 2M(n-1) + 1, M(1) = 1
Copyright © 2007 SSN College Of Engineering. CS1201- “Introduction to the Design & Analysis of Algorithms” 1-69
Puzzle
n
n-1 n-1
1 1 1 1 1 1 1 1
Copyright © 2007 SSN College Of Engineering. CS1201- “Introduction to the Design & Analysis of Algorithms” 1-70
Example 3: Counting #bits
Copyright © 2007 SSN College Of Engineering. CS1201- “Introduction to the Design & Analysis of Algorithms” 1-71
Fibonacci numbers
The Fibonacci numbers:
0, 1, 1, 2, 3, 5, 8, 13, 21, …
Copyright © 2007 SSN College Of Engineering. CS1201- “Introduction to the Design & Analysis of Algorithms” 1-72
S g ( ) ( ) (
2) = 0
I Set up the characteristic equation (quadratic)
ar2 + br + c = 0
Copyright © 2007 SSN College Of Engineering. CS1201- “Introduction to the Design & Analysis of Algorithms” 1-73
numbers
F(n) = F(n-1) + F(n-2) or F(n) - F(n-1) - F(n-2) = 0
Characteristic equation:
Copyright © 2007 SSN College Of Engineering. CS1201- “Introduction to the Design & Analysis of Algorithms” 1-74
Computing Fibonacci numbers
1. Definition-based recursive algorithm
Copyright © 2007 SSN College Of Engineering. CS1201- “Introduction to the Design & Analysis of Algorithms” 1-75