Professional Documents
Culture Documents
Design and Analysis of Computer Algorithm Unit 1: Priya Gupta Asst. Prof. Department of Information Technology
Design and Analysis of Computer Algorithm Unit 1: Priya Gupta Asst. Prof. Department of Information Technology
Algorithm
UNIT 1
Priya Gupta
Asst. Prof.
Department of Information Technology
2
What is Algorithm?
• Algorithm
▫ is any well-defined computational procedure that
takes some value, or set of values, as input and
produces some value, or set of values, as output.
▫ is thus a sequence of computational steps that
transform the input into the output.
▫ is a tool for solving a well - specified
computational problem.
▫ Any special method of solving a certain kind of
problem (Webster Dictionary)
3
What is a program?
• A program is the expression of an algorithm in a
programming language
• a set of instructions which the computer will
follow to solve a problem
4
Some Application
• Study problems these techniques can be applied
to
▫ sorting
▫ data retrieval
▫ network routing
▫ Games
▫ etc
7
Complexity
• The complexity of an algorithm is simply the
amount of work the algorithm performs to
complete its task.
13
RAM model
• has one processor
• executes one instruction
at a time
• each instruction takes
"unit time“
• has fixed-size operands,
and
• has fixed size storage
(RAM and disk).
14
• Modularity • Reliability
• Correctness
• Maintainability
• Functionality
• Robustness
• User-friendliness
• Programmer time
• Simplicity
• Extensibility
15
PUZZLE(x)
while x != 1
if x is even
then x = x / 2
else x = 3x + 1
Sample run: 7, 22, 11, 34, 17, 52, 26, 13, 40, 20,
10, 5, 16, 8, 4, 2, 1
18
OUTPUT
instance
m:= a[1];
25, 90, 53, 23, 11, 34 for I:=2 to size of 11
input
if m > a[I] then
m:=a[I];
return s
m
Data-Structure
21
Define Problem
• Problem:
▫ Description of Input-Output relationship
• Algorithm:
▫ A sequence of computational step that transform the
input into the output.
• Data Structure:
▫ An organized method of storing and retrieving data.
• Our task:
▫ Given a problem, design a correct and good
algorithm that solves it.
22
Example Algorithm A
Problem: The input is a sequence of integers stored in array.
Output the minimum.
Algorithm A
23
Example Algorithm B
Computer Algorithm August 1, 2019
2. While n > 1
For i 1 to n /2
t2[ i ] min (t1 [ 2*i ], t1[ 2*i + 1] );
copy array t2 to t1;
n n/2;
3. Output t2[1];
24
Visualize Algorithm B
Computer Algorithm August 1, 2019
34 6 5 9 20 8 11 7
Loop 1
6 5 8 7
Loop 2
5 7
Loop 3
5
25
What do we need?
Correctness: Whether the algorithm computes
the correct solution for all instances
Measurement
parameterized by the
size of the input.
4
Td(n)
The algorihtms A,B,C are Tc (n)
implemented and run in
Running time
(second)
a PC. 2
Tb (n)
Algorithms D is
Ta (n)
implemented and run in
a supercomputer.
0
Let Tk( n ) be the 500 1000
amount of time taken by Input Size
the Algorithm
28
Methods of Proof
• Proof by Contradiction
▫ Assume a theorem is false; show that this assumption implies a
property known to be true is false -- therefore original hypothesis
must be true
• Proof by Counterexample
▫ Use a concrete example to show an inequality cannot hold
• Mathematical Induction
▫ Prove a trivial base case, assume true for k, then show hypothesis
is true for k+1
▫ Used to prove recursive algorithms
29
Review: Induction
• Suppose
▫ S(k) is true for fixed constant k
Often k = 0
▫ S(n) S(n+1) for all n >= k
• Then S(n) is true for all n >= k
30
Proof By Induction
• Claim:S(n) is true for all n >= k
• Basis:
▫ Show formula is true when n = k
• Inductive hypothesis:
▫ Assume formula is true for an arbitrary n
• Step:
▫ Show that formula is then true for n+1
31
Induction Example:
Gaussian Closed Form
• Prove 1 + 2 + 3 + … + n = n(n+1) / 2
▫ Basis:
If n = 0, then 0 = 0(0+1) / 2
▫ Inductive hypothesis:
Assume 1 + 2 + 3 + … + n = n(n+1) / 2
▫ Step (show true for n+1):
1 + 2 + … + n + n+1 = (1 + 2 + …+ n) + (n+1)
= n(n+1)/2 + n+1 = [n(n+1) + 2(n+1)]/2
= (n+1)(n+2)/2 = (n+1)(n+1 + 1) / 2
32
Induction Example:
Geometric Closed Form
• Prove a0 + a1 + … + an = (an+1 - 1)/(a - 1) for all a
1
▫ Basis: show that a0 = (a0+1 - 1)/(a - 1)
a0 = 1 = (a1 - 1)/(a - 1)
▫ Inductive hypothesis:
Assume a0 + a1 + … + an = (an+1 - 1)/(a - 1)
▫ Step (show true for n+1):
a0 + a1 + … + an+1 = a0 + a1 + … + an + an+1
= (an+1 - 1)/(a - 1) + an+1 = (an+1+1 - 1)/(a - 1)
33
Induction
• We’ve been using weak induction
• Strong induction also holds
▫ Basis: show S(0)
▫ Hypothesis: assume S(k) holds for arbitrary k <=
n
▫ Step: Show S(n+1) follows
• Another variation:
▫ Basis: show S(0), S(1)
▫ Hypothesis: assume S(n) and S(n+1) are true
▫ Step: show S(n+2) follows
34
Basic Recursion
• Base case: value for which function can be
evaluated without recursion
• Two fundamental rules
▫ Must always have a base case
▫ Each recursive call must be to a case that eventually leads toward
a base case
35
Recursion(1/2)
Problem: write an algorithm that will strip
digits from an integer and print them out one
by one
void print_out(int n)
{
if(n < 10)
print_digit(n); /*outputs single-digit to terminal*/
else {
print_out(n/10); /*print the quotient*/
print_digit(n%10); /*print the remainder*/
}
}
37
Recursion(2/2)
Prove by induction that the recursive printing
program works:
▫ basis: If n has one digit, then program is correct
▫ hypothesis: Print_out works for all numbers of k or fewer
digits
▫ case k+1: k+1 digits can be written as the first k digits
followed by the least significant digit
Recursion
• Don't need to know how recursion is being managed
• Recursion is expensive in terms of space
requirement; avoid recursion if simple loop will do
Important Question
• Is it always important to be on the most
preferred curve?
• How much better is one curve than another?
• How do we decide which curve a particular
algorithm lies on?
• How do we design algorithms that avoid being
on the bad curves?
43
Algorithm Analysis(1/5)
• Measures the efficiency of an algorithm or its
implementation as a program as the input size
becomes very large
• We evaluate a new algorithm by comparing its
performance with that of previous approaches
▫ Comparisons are asymtotic analyses of classes of
algorithms
• We usually analyze the time required for an
algorithm and the space required for a
datastructure
44
Asymptotic Performance
• In this course, we care most about asymptotic
performance
▫ How does the algorithm behave as the problem
size gets very large?
Running time
Memory/storage requirements
Bandwidth/power requirements/logic gates/etc.
49
Asymptotic Notation
• By now you should have an intuitive feel for
asymptotic (big-O) notation:
▫ What does O(n) running time mean? O(n2)?
O(n lg n)?
▫ How does asymptotic running time relate to
asymptotic memory usage?
• Our first task is to define this notation more
formally and completely
50
Analysis of Algorithms
• Analysis is performed with respect to a
computational model
• We will usually use a generic uniprocessor
random-access machine (RAM)
▫ All memory equally expensive to access
▫ No concurrent operations
▫ All reasonable instructions take unit time
Except, of course, function calls
▫ Constant word size
Unless we are explicitly manipulating bits
51
Input Size
• Time and space complexity
▫ This is generally a function of the input size
E.g., sorting, multiplication
▫ How we characterize input size depends:
Sorting: number of input items
Multiplication: total number of bits
Graph algorithms: number of nodes & edges
Etc
52
Running Time
• Number of primitive steps that are executed
▫ Except for time of executing a function call most
statements roughly require the same amount of
time
y=m*x+b
c = 5 / 9 * (t - 32 )
z = f(x) + g(y)
• We can be more exact if need be
53
Analysis
• Worst case
▫ Provides an upper bound on running time
▫ An absolute guarantee
• Average case
▫ Provides the expected running time
▫ Very useful, but treat with care: what is “average”?
Random (equally likely) inputs
Real-life inputs
54
subproblem 1 subproblem 2
of size n/2 of size n/2
a solution to a solution to
subproblem 1 subproblem 2
8 3 2 9 7 1 5 4
8 3 2 9 71 5 4
8 3 2 9 7 1 5 4 The non-recursive
version of Mergesort
starts from merging
3 8 2 9 1 7 4 5 single elements into
sorted pairs.
2 3 8 9 1 4 5 7
1 2 3 4 5 7 8 9
Analysis of Mergesort
T(n) = 2T(n/2) + Θ(n), T(1) = 0
A[i]p A[i]p
• Exchange the pivot with the last element in the first (i.e.,
) subarray — the pivot is now in its final position
• Sort the two subarrays recursively
Partitioning Algorithm
or i > r
or j = l
<
• Improvements:
▫ better pivot selection: median of three partitioning
▫ switch to insertion sort on small subfiles
▫ elimination of recursion
These combine to 20-25% improvement
TL TR
A = 12345678901357986429 B =
87654321284820912836
18 additions
Strassen observed [1969] that the product of two
Strassen’s Matrix Multiplication
matrices can be computed in general as follows:
M1 + M4 - M5 + M7 M3 + M5
=
M2 + M4 M1 + M3 - M2 + M6
Formulas for Strassen’s Algorithm
M1 = (A00 + A11) (B00 + B11)
Number of multiplications:
M(n) = 7M(n/2), M(1) = 1
Solution: M(n) = 7log 2n = nlog 27 ≈ n2.807 vs. n3 of
brute-force alg.
Technique
• The greedy method is a general algorithm design
paradigm, built on the following elements:
▫ configurations: different choices, collections, or values to find
▫ objective function: a score assigned to configurations, which
we want to either maximize or minimize
• It works best when applied to problems with the
greedy-choice property:
▫ a globally-optimal solution can always be found by a series of
local improvements from a starting configuration.
84
Making Change
• Problem: Accept n dollars, to return a collection of coins with a total
value of n dollars.
• Configuration: A collection of coins with a total value of n
• Objective function: Minimize number of coins returned.
• Greedy solution: Always return the largest coin you can
• Example 1: Coins are valued $.32, $.08, $.01
▫ Has the greedy-choice property, since no amount over $.32 can be made
with a minimum number of coins by omitting a $.32 coin (similarly for
amounts over $.08, but under $.32).
▫ For a certain amount (y) over a coin dimension (x), if we can reach it
using coins (<x) with a total n coins, then we can also use coin x with ≤ n
coins.
• Example 2: Coins are valued $.30, $.20, $.05, $.01
▫ Does not have greedy-choice property, since $.40 is best made with two
$.20’s, but the greedy solution will pick three coins (which ones?)
85
▫ Objective: maximize
b (x / w )
iS
i i i
x
▫ Constraint:
i W
iS
86
Example
• Given: A set S of n items, with each item i having
▫ bi - a positive benefit
▫ wi - a positive weight
• Goal: Choose items with maximum total benefit but with
weight at most W.
“knapsack”
Solution:
• 1 ml of 5
Items:
1 2 3 4 5
• 2 ml of 3
• 6 ml of 4
Weight: 4 ml 8 ml 2 ml 6 ml 1 ml • 1 ml of 2
Benefit: $12 $32 $40 $30 $50 10 ml
Value: 3 4 20 5 50
The Greedy Method
($ per ml)
87
Task Scheduling
• Given: a set T of n tasks, each having:
▫ A start time, si
▫ A finish time, fi (where si < fi)
• Goal: Perform all the tasks using a minimum number of
“machines.”
Machine 3
Machine 2
Machine 1
1 2 3 4 5 6 7 8 9
89
Example
• Given: a set T of n tasks, each having:
▫ A start time, si
▫ A finish time, fi (where si < fi)
▫ [1,4], [1,3], [2,5], [3,7], [4,7], [6,9], [7,8] (ordered by start)
• Goal: Perform all tasks on min. number of machines
Machine 3
Machine 2
Machine 1
1 2 3 4 5 6 7 8 9
Optimal 2-way Merge patterns and Huffman Codes:
Example. Suppose there are 3 sorted lists L1, L2, and L3, of sizes 30, 20,
and 10, respectively, which need to be merged into a combined sorted list,
but we can merge only two at a time. We intend to find an optimal merge
pattern which minimizes the total number of comparisons. For example,
we can merge L1 and L2, which uses 30 + 20 = 50 comparisons resulting in
a list of size 50. We can then merge this list with list L3, using another 50 +
10 = 60 comparisons, so the total number of comparisons is 50 + 60 = 110.
Alternatively, we can merge lists L2 and L3, using 20 + 10 = 30
comparisons, the resulting list (size 30) can then be merged with list L1, for
another 30 + 30 = 60 comparisons. So the total number of comparisons is
30 + 60 = 90. It doesn’t take long to see that this latter merge pattern is
the optimal one.
Binary Merge Trees: We can depict the merge patterns using a binary
tree, built from the leaf nodes (the initial lists) towards the root in which
each merge of two nodes creates a parent node whose size is the sum of the
sizes of the two children. For example, the two previous merge patterns
are depicted in the following two figures:
50 10 30 30
30 20 20 10
Merge L1 and L2, then with Merge L2 and L3, then with
L3 L1
merge cost = sum of all weighted external path lengths
Optimal Binary Merge Tree Algorithm:
Input: n leaf nodes each have an integer size, n 2.
Output: a binary tree with the given leaf nodes which
has a minimum total weighted external path lengths
Algorithm:
(1) create a min-heap T[1..n ] based on the n initial sizes. (2) while
(the heap size 2) do
(2.1) delete from the heap two smallest values, call
them a and b, create a parent node of size a + b
for the nodes corresponding to these two values
(2.2) insert the value (a + b) into the heap which
corresponds to the node created in Step (2.1)
When the algorithm terminates, there is a single value left in the heap
whose corresponding node is the root of the optimal binary merge tree.
The algorithm’s time complexity is O(n lgn) because Step (1) takes O(n)
time; Step (2) runs O(n) iterations, in which each iteration takes O(lgn)
time.
Example of the optimal merge tree algorithm:
2 3 5 7 9
Iteration 1: merge 2 and 3
into 5
10
Iteration 2:
5 5 16 Iteration 3: merge 7
merge 5 and
and 9 (chosen among
5 into 10
2 3 7 9 7, 9, and 10) into 16
2
6 Iteration 4: merge
16
10 10 and 16 into 26
5 5 7 9
Cost = 2*3 + 3*3 + 5*2 +
2 3 7*2 + 9*2 = 57.
Proof of optimality of the binary merge tree algorithm:
We use induction on n 2 to show that the binary merge tree is optimal in that
it gives the minimum total weighted external path lengths (among all possible
ways to merge the given leaf nodes into a binary tree).
(Basis) When n = 2. There is only one way to merge two
nodes. (Induction Hypothesis) Suppose the merge tree is
optimal when there are k leaf nodes, for some k 2.
(Induction) Consider (k + 1) leaf nodes. Call them
a1, a2, …, and ak+1. We may assume nodes a1, a2 are of the smallest values,
which are merged in the first step of the merge algorithm into node b. We call
the merge tree T, the part excluding a1, a2 T’ (see figure). Suppose an optimal
binary merge tree is S. We make two observations. (1) If node x of S is a
deepest internal node, we may swap its two children with nodes a1, a2 in S
without increasing the total weighted external path lengths. Thus, we may
assume tree S has a subtree S’ with leaf nodes x, a2, …, and ak+1.
(2) The tree S’ must be an optimal merge
tree for k nodes x, a2, …, and ak+1. By
induction hypothesis, tree S’ has a T S
total weighted external path lengths equal T’ S’
b x
to that of tree T’. Therefore, the total
weighted external path lengths of T equalsato 1 a a1 a
that of tree S, proving the optimality of T.
2 2
Huffman Codes:
Suppose we wish to save a text (ASCII) file on the disk or to transmit it
though a network using an encoding scheme that minimizes the number of
bits required. Without compression, characters are typically encoded by
their ASCII codes with 8 bits per character. We can do better if we have the
freedom to design our own encoding.
Example. Given a text file that uses only 5 different letters (a, e, i, s, t), the
space character, and the newline character. Since there are 7 different
characters, we could use 3 bits per character because that allows 8 bit
patterns ranging from 000 through 111 (so we still one pattern to spare).
The following table shows the encoding of characters, their frequencies, and
the size of encoded (compressed) file.
Character Frequency Code Total bits Code Total bits
a 10 000 001 30
30 e 15 01 30
001 45 i 10 24
12 010 36 00000 15
s 3 011 9 0001 16 11
t 4 100 12 26
space 13 101 39 00001 5
newline 1 110 3
146
Total 58 174
Fixed-length encoding Variable-length encoding
If we can use variable lengths for the codes, we can actually compress
more as shown in the above. However, the codes must satisfy the
property that no code is the prefix of another code; such code is called a
prefix code.
How to design an optimal prefix code (i.e., with minimum total
length) for a given file?
We can depict the codes for the given collection of characters using a
binary tree as follows: reading each code from left to right, we construct a
binary tree from the root following the left branch when encountering a
‘0’, right branch when encountering a ‘1’. We do this for all the codes by
constructing a single combined binary tree. For example,
0 1
0 0
0 0 1 0 1 0
1 1 1
Codes 001, 01, 10,
00000, 0001, 11, and
00001
Code Codes 001 Codes Note: each code terminates at
001 and 01 001, 01, a leaf node, by the prefix
and 10 property.
We note that the encoded file size is equal to the total weighted external
path lengths if we assign the frequency to each leaf node. For example,
‘e’
15 12 13 Total file size = 3*5 + 1*5 + 4*4 + 10*3 +
10 ‘i’ ‘’ 15*2 + 12*2 + 13*2 = 146, which is exactly
‘a’ the total weighted external path lengths.
4
‘t’
3 1
‘s’ ‘\n’
We also note that in an optimal prefix
code, each node in the tree has either x x
no children or has two. Thus, the
optimal binary merge tree algorithm y
finds the optimal code (Huffman code).
Merge x and y,
Node x has reducing total
only one child size
y
Greedy Strategies Applied to Graph problems:
We first review some notations and terms about graphs. A graph
consists of vertices (nodes) and edges (arcs, links), in which each edge
“connects” two vertices (not necessarily distinct). More formally, a
graph G = (V, E), where V and E denote the sets of vertices and edges,
respectively.
j
1 2 3 4
1 A 2-dimensional array
0 1 0 0
1 0 0 0 1 for the digraph, called
2 3 i 2 the adjacency matrix.
3 1 0 1 0
4 0 1 0 0
4
Sometimes, edges of a graph or digraph are given a positive weight or
cost value. In that case, the adjacency matrix can easily modified so
that T[i][j] = the weight of edge (i, j); 0 if there is no edge (i, j). Since
the adjacency matrix may contain many zeros (when the graph has
few edges, known as sparse), a space-efficient representation uses
linked lists representing the edges, known as the adjacency list
representation.
1 2
1
2 4
2 3 3 3 1
4 2
4 The adjacency lists for the digraph, which can
store edge weights by adding another field in the
list nodes.
The Minimum Spanning Tree (MST) Problem:
Given a weighted (undirected) graph G = (V, E), where each edge e has a
positive weight w(e). A spanning tree of G is a tree (connected graph
without cycles, or circuits) which has V as its vertex set, i.e., the tree
connects all vertices of the graph G. If |V| = n, then the tree has n – 1
edges (this is a fact which can be proved by induction). A minimum
spanning tree of G is a spanning tree that has the minimum total edge
weight.
1 1
3 6 3 6
8 A minimum spanning
2 3 2 3 tree (of 4 edges),
5 4 4 weight = 3 + 2 + 4 + 6
7 = 15.
5 4 5 4
2 2
A weighted graph of no parallel
edges or self-loops
Prim’s Algorithm for the Minimum Spanning Tree problem:
Create an array B[1..n] to store the nodes of the MST, and an array T[1..n –1]
to store the edges of the MST. Starting with node 1 (actually, any node can be
the starting node), put node 1 in B[1], find a node that is the closest (i.e., an
edge connected to node 1 that has the minimum weight, ties broken
arbitrarily). Put this node as B[2], and the edge as T[1]. Next look for a node
connected from either B[1] or B[2] that is the closest, store the node as B[3],
and the corresponding edge as T[2]. In general, in the kth iteration, look for a
node not already in B[1..k] that is the closest to any node in B[1..k]. Put this
node as B[k+1], the corresponding edge as T[k]. Repeat this process for n –1
iterations (k = 1 to n –1). This is a greedy strategy because in each iteration,
the algorithm looks for the minimum weight edge to include next while
maintaining the tree property (i.e., avoiding cycles). At the end there are
exactly n –1 edges without cycles, which must be a spanning tree.
Example: Prim’s MST Algorithm.