Design and Analysis of Algorithms: Course Code

Design and Analysis of
Algorithms
Course Code
Why Design and Analysis of Algorithm
• Theoretical importance
– the core of computer science
• Practical importance
– A practitioner’s toolkit of known algorithms
– Framework for designing and analyzing

algorithms for new problems
Why Design and Analysis of Algorithm
• David Harel in his book “Algorithm: the Spirit of
Computing”
“Algorithmics is more than a branch of computer

science. It is the core of computer science, and, in
all fairness, can be said to be relevant to most of
science, business, and technology.”
Algorithm
⚫Definition
⚫An algorithm is a sequence of unambiguous
instructions for solving a problem, i.e., for
obtaining a required output for any legitimate
input in a finite amount of time.
⚫An algorithm is a finite set of instructions that

accomplishes a particular task.
Algorithm
⚫Criteria
⚫Input: Zero or more quantities are externally
supplied
⚫Output: At least one quantity is produced
⚫definiteness: clear and unambiguous
⚫finiteness: terminate after a finite number of
steps
⚫effectiveness: instruction is basic enough to be
carried out
Notion of algorithm and problem
Problem
Algorithm
Input “Computer” Output

(or instance)
Fundamental of Algorithmic problem solving
Steps in designing and analyzing an algorithm
1. Understanding the Problem
2. Ascertaining the Capabilities of the Computational
Device
3. Choosing between Exact and Approximate Problem Solving

✔The next principal decision is to choose between solving the
problem exactly or solving it approximately.
Fundamental of Algorithmic problem solving
4. Deciding on appropriate Data Structures
5. Algorithm Design Techniques
6. Methods of Specifying an Algorithm
7. Proving an Algorithm’s Correctness
8. Analyzing an Algorithm
Time efficiency
Space efficiency
Simplicity and generality
9. Coding an algorithm
Fundamentals of Analysis of Algorithms
⚫The Analysis Framework
⚫ Time efficiency, also called time complexity, indicates how fast an
algorithm in question runs.
⚫Space efficiency, also called space complexity, refers to the amount of
memory units required by the algorithm in addition to the space
needed for its input and output.
1. Measuring an Input’s Size
“The choice of an appropriate size metric can be influenced by

operations of the algorithm in question.”
An input to an algorithm specifies an instance of the problem to be solved

⚫We need to identify most important operation of the

algorithm, called the basic operation
⚫The operation contributing the most to the total running

time, and compute the number of times the basic operation
is executed
⚫Basic operation is usually the most time-consuming
operation in the algorithm’s innermost loop.
⚫Thus, the established framework for the analysis of an

algorithm’s time efficiency suggests measuring it by
counting the number of times the algorithm’s basic
operation is executed on inputs of size n.
⚫Here is an important application. Let cop be the execution
time of an algorithm’s basic operation on a particular
computer, and
⚫let C(n) be the number of times this operation needs to be
executed for this algorithm. Then we can estimate the
running time T(n) of a program implementing this
algorithm on that computer is given by the formula
T (n) ≈ cop C(n)

⚫Orders of Growth
Algorithms that require an exponential number of operations are

practical
for solving only problems of very small sizes.
⚫We established that it is reasonable to measure an

algorithm’s efficiency as a function of a parameter
indicating the size of the algorithm’s input.
⚫But there are many algorithms for which running time

depends not only on an input size but also on the specifics
of a particular input.
⚫Worst-Case, Best-Case, and Average-Case Efficiencies

⚫Worst-Case
⚫The worst-case efficiency of an algorithm is its efficiency for the worst-

case input of size n, which is an input (or inputs) of size n for which the
algorithm runs the longest among all possible inputs of that size.
⚫The way to determine the worst-case efficiency of an algorithm is:

analyze the algorithm to see what kind of inputs yield the largest
value of the basic operation’s count C(n) among all possible inputs of
size n and then compute this worst-case value Cworst(n).
⚫Best-Case
⚫The best-case efficiency of an algorithm is its efficiency for the best-

case input of size n, which is an input (or inputs) of size n for
which the algorithm runs the fastest among all possible inputs of
that size.
⚫Accordingly, we can analyze the best case efficiency as follows.
⚫First, we determine the kind of inputs for which the count C(n) will be
the smallest among all possible inputs of size n.
(Note that the best case does not mean the smallest input; it means the
input of size n for which the algorithm runs the fastest.)
⚫Average-Case
⚫The average-case efficiency cannot be obtained by taking the average

of the worst-case and the best-case efficiencies.
⚫The direct approach for doing this involves –
⚫Take all possible inputs and calculate computing time for all of the
inputs. Sum all the calculated values and divide the sum by total
number of inputs
Important Problem Types/
Classification of the problem
⚫Sorting
⚫String Processing
⚫Searching
⚫Recurrences
⚫Shortest paths in a graph
⚫Minimum spanning tree
⚫Traveling salesman problem
⚫Knapsack problem
⚫Chess
⚫Towers of Hanoi
⚫Geometric and numerical problems
Algorithm design/Problem solving
strategies
⚫Brute force ⚫Greedy approach
⚫Divide and conquer ⚫Dynamic programming
⚫Decrease and conquer ⚫Backtracking and branch-and-

bound
⚫Transform and conquer
Asymptotic Notations
⚫The efficiency analysis framework concentrates on the
order of growth of an algorithm’s basic operation count
as the principal indicator of the algorithm’s efficiency.
⚫To compare and rank such orders of growth, computer

scientists use three notations:
1. O (big oh),
2. (big omega), and
3. (big theta).
O-notation
Mathematical Analysis of Nonrecursive
Algorithms
EXAMPLE 1 Consider the problem of finding the value of the largest element
in a list of n numbers. For simplicity, we assume that the list is implemented as
an array. The following is pseudocode of a standard algorithm for solving the
problem.
ALGORITHM MaxElement(A[0..n − 1])

//Determines the value of the largest element in a given array
//Input: An array A[0..n − 1] of real numbers
//Output: The value of the largest element in A
maxval ←A[0]
for i ←1 to n − 1 do
if A[i]>maxval
maxval←A[i]
return maxval
General Plan for Analyzing the Time
Efficiency of Non-Recursive Algorithms
⚫1. Decide on a parameter (or parameters) indicating an input’s size.
⚫2. Identify the algorithm’s basic operation. (As a rule, it is located in the
innermost loop.)
⚫3. Check whether the number of times the basic operation is executed
depends only on the size of an input. If it also depends on some additional
property, the worst-case, average-case, and, if necessary, best-case
efficiencies have to be investigated separately.
⚫4. Set up a sum expressing the number of times the algorithm’s basic
operation is executed.
⚫5. Using standard formulas and rules of sum manipulation, either find a
closed form formula for the count or, at the very least, establish its order of
growth.
Mathematical Analysis of Recursive
Algorithms
General Plan for Analyzing the Time
Efficiency of Recursive Algorithms
1. Decide on a parameter (or parameters) indicating an input’s
size.
2. Identify the algorithm’s basic operation.
3. Check whether the number of times the basic operation is
executed can vary on different inputs of the same size; if it
can, the worst-case, average-case, and best-case efficiencies
must be investigated separately.
4. Set up a recurrence relation, with an appropriate initial
condition, for the number of times the basic operation is
executed.
5. Solve the recurrence
Tower of Hanoi
Rules:
1. Only one disk can be moved among
the towers at any given time.
2. Only the "top" disk can be removed.
3. No large disk can sit over a small
disk.
M(n) = M(n − 1) + 1+ M(n − 1) for n > 1.
M(n) = 2M(n − 1) + 1 for n > 1,

M(1) = 1. ---- Initial condition
Thank You
Unit – 2 (a)
Divide and Conquer Approach

Greedy,Dynamic
Prepared By
SWEET SUBHASHREE
Asst. Prof., Dept. of IT, MIT-SOE
Divide and Conquer
• Recursive in structure
• Divide the problem into sub-problems that are similar to the
original but smaller in size
• Conquer the sub-problems by solving them recursively. If
they are small enough, just solve them in a straightforward
manner.
• Combine the solutions to create a solution to the original
problem
Prepared By - REETIKA KERKETTA 2

An Example: Merge Sort
Sorting Problem: Sort a sequence of n elements into non-
decreasing order.
• Divide: Divide the n-element sequence to be sorted into two

subsequences of n/2 elements each
• Conquer: Sort the two subsequences recursively using merge

sort.
• Combine: Merge the two sorted subsequences to produce the

sorted answer.

Merge Sort Procedure
7 2⏐9 4 → 2 4 7 9 Algorithm mergeSort(S, C)
Input sequence S
7⏐2 → 2 7 9⏐4 → 4 9 with n
elements,
7→7 2→2 9→9 4→4 comparator C
Output sequence S
🞂 Merge-sort on an input sequence S
with n elements consists of three sorted
steps: according to C
◦ Divide: partition S into two if S.size() > 1
sequences S1 and S2 of about n/2
elements each (S1, S2) ←
◦ Recur: recursively sort S1 and S2 partition(S, n/2)
◦ Conquer: merge S1 and S2 into a mergeSort(S1, C)
unique sorted sequence mergeSort(S2, C)
Prepared By - REETIKA KERKETTA
S ← merge(S1, S2)
4
Merge-Sort (A, p, r)
INPUT: a sequence of n numbers stored in array A
OUTPUT: an ordered sequence of n numbers
MergeSort (A, p, r) // sort A[p..r] by divide & conquer

1 if p < r
2 then q ← ⎣(p+r)/2⎦
3 MergeSort (A, p, q)
4 MergeSort (A, q+1, r)
5 Merge (A, p, q, r) // merges A[p..q] with A[q+1..r]

Procedure Merge Merge(A, p, q, r)
1 n1 ← q – p + 1
Input: Array containing sorted 2 n2 ← r – q
sub-arrays A[p..q] and A[q+1..r]. 3 for i ← 1 to n1
Output: Merged sorted sub-array 4 do L[i] ← A[p + i – 1]
in A[p..r]. 5 for j ← 1 to n2
6 do R[j] ← A[q + j]
7 L[n1+1] ← ∞
Merge-sort on an input sequence S 8 R[n2+1] ← ∞
with n elements consists of three
steps: 9 i←1
Divide: partition S into two 10 j←1
sequences S1 and S2 of about 11 for k ←p to r
n/2 elements each 12 do if L[i] ≤ R[j]
Recur: recursively sort S1 and 13 then A[k] ← L[i]
S2
14 i←i+1
Conquer: merge S1 and S2
into a unique sorted sequence 15 else A[k] ← R[j]
16 j←j+1
Quick Sort Algorithm
Given an array of n elements (e.g., integers):
• If array only contains one element, return
• Else
• pick one element to use as pivot.
• Partition elements into two sub-arrays:
• Elements less than or equal to pivot
• Elements greater than pivot
• Quick-sort two sub-arrays
• Return results

Partitioning Array
• Given a pivot, partition the elements of the array such that the
resulting array consists of:
1. One sub-array that contains elements >= pivot
2. Another sub-array that contains elements < pivot
• The sub-arrays are stored in the original data array.
Partitioning loops through, swapping elements below/above
pivot.

Randomizing Quick-sort
• Randomly permute the elements of the input array before
sorting
• OR ... modify the PARTITION procedure

• At each step of the algorithm we exchange element A[p]
with an element chosen at random from A[p…r]
• The pivot element x = A[p] is equally likely to be any one
of the r – p + 1 elements of the sub-array

Randomized Quick Sort
Alg. : RANDOMIZED-QUICKSORT(A, p, r)
if p < r
then q ← RANDOMIZED-PARTITION(A, p, r)
RANDOMIZED-QUICKSORT(A, p, q)
RANDOMIZED-QUICKSORT(A, q + 1, r)

Matrix multiplication
Square-matrix-multiplication(A,B)
{
n=A.rows
let c be a new n*n matrix
for i=1 to n
for j=1 to n
c[i,j]=0
for k=1 to n
c[i,j]=c[i,j]+a[i,k]*b[k,j]
return C;
}
11
Strassens’s Matrix Multiplication
• Strassen showed that 2x2 matrix multiplication can be
accomplished in 7 multiplication and 18 additions or
subtractions (2log27 =22.807).
• Divide-and conquer is a general algorithm design paradigm:
• Divide: divide the input data S in two or more disjoint subsets
S1, S2, …
• Recur: solve the sub problems recursively
• Conquer: combine the solutions for S1, S2, …, into a solution
for S
• The base case for the recursion are sub problems of constant size
• Analysis can be done using recurrence equations.
Strassens’s Matrix Multiplication
P1 = (A11+ A22)(B11+B22) C11 = P1 + P4 - P5 + P7

P2 = (A21 + A22) * B11 C12 = P3 + P5
P3 = A11 * (B12 - B22) C21 = P2 + P4
P4 = A22 * (B21 - B11) C22 = P1 + P3 - P2 + P6
P5 = (A11 + A12) * B22
P6 = (A21 - A11) * (B11 + B12)
P7 = (A12 - A22) * (B21 + B22)

Greedy Algorithms
1
A short list of categories
◼ Algorithm types we will consider include:
◼ Simple recursive algorithms
◼ Backtracking algorithms
◼ Divide and conquer algorithms
◼ Dynamic programming algorithms
◼ Greedy algorithms
◼ Branch and bound algorithms
◼ Brute force algorithms
◼ Randomized algorithms
2
2
Optimization problems
◼ An optimization problem is one in which you want

to find, not just a solution, but the best solution
◼ A “greedy algorithm” sometimes works well for
optimization problems
◼ A greedy algorithm works in phases. At each
phase:
◼ You take the best you can get right now, without regard
for future consequences
◼ You hope that by choosing a local optimum at each
step, you will end up at a global optimum
3
3
Example: Counting money
◼ Suppose you want to count out a certain amount of
money, using the fewest possible bills and coins
◼ A greedy algorithm would do this would be:
At each step, take the largest possible bill or coin
that does not overshoot
◼ Example: To make $6.39, you can choose:
◼ a $5 bill
◼ a $1 bill, to make $6
◼ a 25¢ coin, to make $6.25
◼ A 10¢ coin, to make $6.35
◼ four 1¢ coins, to make $6.39
◼ For US money, the greedy algorithm always gives
the optimum solution
4
4
A failure of the greedy algorithm
◼ In some (fictional) monetary system, “krons” come
in 1 kron, 7 kron, and 10 kron coins
◼ Using a greedy algorithm to count out 15 krons,
you would get
◼ A 10 kron piece
◼ Five 1 kron pieces, for a total of 15 krons
◼ This requires six coins
◼ A better solution would be to use two 7 kron pieces
and one 1 kron piece
◼ This only requires three coins
◼ The greedy algorithm results in a solution, but not
in an optimal solution
5
5
A scheduling problem
◼ You have to run nine jobs, with running times of 3, 5, 6, 10, 11,
14, 15, 18, and 20 minutes
◼ You have three processors on which you can run these jobs
◼ You decide to do the longest-running jobs first, on whatever
processor is available
P1 20 10 3
P2 18 11 6
P3 15 14 5
◼ Time to completion: 18 + 11 + 6 = 35 minutes

◼ This solution isn’t bad, but we might be able to do better
6
6
Another approach
◼ What would be the result if you ran the shortest job first?
◼ Again, the running times are 3, 5, 6, 10, 11, 14, 15, 18, and 20
minutes
P1 3 10 15
P2 5 11 18
P3 6 14 20
◼ That wasn’t such a good idea; time to completion is now
6 + 14 + 20 = 40 minutes
◼ Note, however, that the greedy algorithm itself is fast
◼ All we had to do at each stage was pick the minimum or maximum
7
7
An optimum solution
◼ Better solutions do exist:
P1 20 14
P2 18 11 5
P3 15 10 6 3
◼ This solution is clearly optimal (why?)
◼ Clearly, there are other optimal solutions (why?)
◼ How do we find such a solution?
◼ One way: Try all possible assignments of jobs to processors
◼ Unfortunately, this approach can take exponential time
8
8
Huffman encoding
◼ The Huffman encoding algorithm is a greedy algorithm
◼ You always pick the two smallest numbers to combine
◼ Average bits/char:
100 0.22*2 + 0.12*3 +
54 0.24*2 + 0.06*4 +
0.27*2 + 0.09*4
27 A=00
= 2.42
B=100
C=01 ◼ The Huffman
46 15
D=1010 algorithm finds an
E=11 optimal solution
22 12 24 6 27 9
F=1011
A B C D E F
9
9
Minimum spanning tree
◼ A minimum spanning tree is a least-cost subset of the edges of a
graph that connects all the nodes
◼ Start by picking any node and adding it to the tree
◼ Repeatedly: Pick any least-cost edge from a node in the tree to a
node not in the tree, and add the edge and new node to the tree
◼ Stop when all nodes have been added to the tree
4
6 ◼ The result is a least-cost
2 (3+3+2+2+2=12) spanning tree
4 If you think some other edge should be
1 5 ◼
3 2 in the spanning tree:

◼ Try adding that edge
3 3 2 3 Note that the edge is part of a cycle
3 ◼
4 ◼ To break the cycle, you must remove

2 4 the edge with the greatest cost
◼ This will be the edge you just added
10
10
Traveling salesman
◼ A salesman must visit every city (starting from city A), and wants
to cover the least possible distance
◼ He can revisit a city (and reuse a road) if necessary
◼ He does this by using a greedy algorithm: He goes to the next
nearest city from wherever he is
◼ From A he goes to B
A B C ◼ From B he goes to D
2 4
◼ This is not going to result in a
shortest path!
3 3
4 4 ◼ The best result he can get now
will be ABDBCE, at a cost of 16
◼ An actual least-cost path from A
D is ADBCE, at a cost of 14
E 11
11
Analysis
◼ A greedy algorithm typically makes (approximately) n choices
for a problem of size n
◼ (The first or last choice may be forced)
◼ Hence the expected running time is:
O(n * O(choice(n))), where choice(n) is making a choice
among n objects
◼ Counting: Must find largest useable coin from among k sizes of coin (k is
a constant), an O(k)=O(1) operation;
◼ Therefore, coin counting is (n)
◼ Huffman: Must sort n values before making n choices
◼ Therefore, Huffman is O(n log n) + O(n) = O(n log n)
◼ Minimum spanning tree: At each new node, must include new edges and
keep them sorted, which is O(n log n) overall
◼ Therefore, MST is O(n log n) + O(n) = O(n log n)
12
12
Other greedy algorithms
◼ Dijkstra’s algorithm for finding the shortest path in a
graph
◼ Always takes the shortest edge connecting a known node to an
unknown node
◼ Kruskal’s algorithm for finding a minimum-cost
spanning tree
◼ Always tries the lowest-cost remaining edge
◼ Prim’s algorithm for finding a minimum-cost spanning
tree
◼ Always takes the lowest-cost edge between nodes in the
spanning tree and nodes not yet in the spanning tree
13
13
14
14
15
15
16
16
17
17
18
18
19
19
20
20
21
21
22
22
23
23
The End
24
24
Dynamic Programming
Overview of Serial Dynamic Programming
• Dynamic programming (DP) is used to solve a wide

variety of discrete optimization problems such as
scheduling, string-editing, packaging, and inventory
management.
• Break problems into subproblems and combine their
solutions into solutions to larger problems.
• In contrast to divide-and-conquer, there may be
relationships across subproblems.
Dynamic Programming: Example
• Consider the problem of finding a shortest path between

a pair of vertices in an acyclic graph.
• An edge connecting node i to node j has cost c(i,j).
• The graph contains n nodes numbered 0,1,…, n-1, and
has an edge from node i to node j only if i < j. Node 0 is
source and node n-1 is the destination.
• Let f(x) be the cost of the shortest path from node 0 to
node x.
• A graph for which the shortest path between nodes 0

and 4 is to be computed.
Dynamic Programming
• The solution to a DP problem is typically expressed as a

minimum (or maximum) of possible alternate solutions.
• If r represents the cost of a solution composed of
subproblems x1, x2,…, xl, then r can be written as
Here, g is the composition function.

• If the optimal solution to each problem is determined by
composing optimal solutions to the subproblems and
selecting the minimum (or maximum), the formulation is
said to be a DP formulation.
The computation and composition of subproblem solutions

to solve problem f(x8).
Dynamic Programming
• The recursive DP equation is also called the functional

equation or optimization equation.
• In the equation for the shortest path problem the
composition function is f(j) + c(j,x). This contains a single
recursive term (f(j)). Such a formulation is called
monadic.
• If the RHS has multiple recursive terms, the DP
formulation is called polyadic.
Dynamic Programming
• The dependencies between subproblems can be

expressed as a graph.
• If the graph can be levelized (i.e., solutions to problems
at a level depend only on solutions to problems at the
previous level), the formulation is called serial, else it is
called non-serial.
• Based on these two criteria, we can classify DP
formulations into four categories - serial-monadic, serial-
polyadic, non-serial-monadic, non-serial-polyadic.
• This classification is useful since it identifies concurrency
and dependencies that guide parallel formulations.
0/1 Knapsack Problem
• We are given a knapsack of capacity c and a set of n objects

numbered 1,2,…,n. Each object i has weight wi and profit pi.
• Let v = [v1, v2,…, vn] be a solution vector in which vi = 0 if object i is
not in the knapsack, and vi = 1 if it is in the knapsack.
• The goal is to find a subset of objects to put into the knapsack so
that
(that is, the objects fit into the knapsack) and
is maximized (that is, the profit is maximized).

• The naive method is to consider all 2n possible subsets

of the n objects and choose the one that fits into the
knapsack and maximizes the profit.
• Let F[i,x] be the maximum profit for a knapsack of
capacity x using only objects {1,2,…,i}. The DP
formulation is:
• Construct a table F of size n x c in row-major order.

• Filling an entry in a row requires two entries from the
previous row: one from the same column and one from
the column offset by the weight of the object
corresponding to the row.
• Computing each entry takes constant time; the
sequential run time of this algorithm is Θ(nc).
• The formulation is serial-monadic.
Computing entries of table F for the 0/1 knapsack problem. The computation of
entry F[i,j] requires communication with processing elements containing
entries F[i-1,j] and F[i-1,j-wi].
• Using c processors in a PRAM, we can derive a simple

parallel algorithm that runs in O(n) time by partitioning
the columns across processors.
• In a distributed memory machine, in the jth iteration, for
computing F[j,r] at processing element Pr-1, F[j-1,r] is
available locally but F[j-1,r-wj] must be fetched.
• The communication operation is a circular shift and the
time is given by (ts + tw) log c. The total time is therefore
tc + (ts + tw) log c.
• Across all n iterations (rows), the parallel time is O(n log
c). Note that this is not cost optimal.
• Using p-processing elements, each processing element

computes c/p elements of the table in each iteration.
• The corresponding shift operation takes time (2ts + twc/p),
since the data block may be partitioned across two
processors, but the total volume of data is c/p.
• The corresponding parallel time is n(tcc/p + 2ts + twc/p),
or O(nc/p) (which is cost-optimal).
• Note that there is an upper bound on the efficiency of
this formulation.
Warshall Floyd's All-Pairs Shortest Path
• A PRAM formulation of this algorithm uses n2 processors

in a logical 2D mesh. Processor Pi,j computes the value
of dik,j for k=1,2,…,n in constant time.
• The parallel runtime is Θ(n) and it is cost-optimal.
• The algorithm can easily be adapted to practical
architectures, as discussed in our treatment of Graph
Algorithms.
Warshall Floyd's All-Pairs Shortest Path
• When multiplying a sequence of matrices, the order of

multiplication significantly impacts operation count.
• Let C[i,j] be the optimal cost of multiplying the matrices
Ai,…Aj.
• The chain of matrices can be expressed as a product of
two smaller chains, Ai,Ai+1,…,Ak and Ak+1,…,Aj.
• The chain Ai,Ai+1,…,Ak results in a matrix of dimensions
ri-1 x rk, and the chain Ak+1,…,Aj results in a matrix of
dimensions rk x rj.
• The cost of multiplying these two matrices is ri-1rkrj.
Optimal Matrix-chain multiplication Problem
• We have:
A nonserial polyadic DP formulation for finding an optimal matrix

parenthesization for a chain of four matrices. A square node
represents the optimal cost of multiplying a matrix chain. A circle
node represents a possible parenthesization.
• The goal of finding C[1,n] is accomplished in a bottom-up

fashion.
• Visualize this by thinking of filling in the C table
diagonally. Entries in diagonal l corresponds to the cost
of multiplying matrix chains of length l+1.
• The value of C[i,j] is computed as min{C[i,k] + C[k+1,j] +
ri-1rkrj}, where k can take values from i to j-1.
• Computing C[i,j] requires that we evaluate (j-i) terms and
select their minimum.
• The computation of each term takes time tc, and the
computation of C[i,j] takes time (j-i)tc. Each entry in
diagonal l can be computed in time ltc.
• The algorithm computes (n-1) chains of length two. This

takes time (n-1)tc; computing n-2 chains of length three
takes time (n-2)tc. In the final step, the algorithm
computes one chain of length n in time (n-1)tc.
• It follows that the serial time is Θ(n3).
The diagonal order of computation for the optimal matrix-

parenthesization problem.
Parallel Optimal Matrix-Parenthesization
Problem
• Consider a logical ring of processors. In step l, each processor computes a
single element belonging to the lth diagonal.
• On computing the assigned value of the element in table C, each processor
sends its value to all other processors using an all-to-all broadcast.
• The next value can then be computed locally.
• The total time required to compute the entries along diagonal l is ltc+tslog
n+tw(n-1).
• The corresponding parallel time is given by:
Parallel Optimal Matrix-Parenthesization
Problem
• When using p (<n) processors, each processor stores n/p nodes.
• The time taken for all-to-all broadcast of n/p words is
and the time to compute n/p entries of the table in the lth diagonal is
ltcn/p.
• This formulation can be improved to use up to n(n+1)/2 processors

using pipelining.
Discussion of Parallel Dynamic Programming
Algorithms
• By representing computation as a graph, we identify

three sources of parallelism: parallelism within nodes,
parallelism across nodes at a level, and pipelining nodes
across multiple levels. The first two are available in serial
formulations and the third one in non-serial formulations.
• Data locality is critical for performance. Different DP

formulations, by the very nature of the problem instance,
have different degrees of locality.
Optimal Binary Search Tree
1.Preface
• OBST is one special kind of advanced tree.
• It focus on how to reduce the cost of the search

of the BST.
• It may not have the lowest height !
• It needs 3 tables to record probabilities, cost,

and root.
2.Premise
• It has n keys (representation k1,k2,…,kn) in sorted order
(so that k1<k2<…<kn), and we wish to build a binary
search tree from these keys. For each ki ,we have a
probability pi that a search will be for ki.
• In contrast of, some searches may be for values not in ki,
and so we also have n+1 “dummy keys” d0,d1,…,dn
representating not in ki.
• In particular, d0 represents all values less than k1, and dn
represents all values greater than kn, and for i=1,2,…,n-
1, the dummy key di represents all values between ki and
ki+1.
＊The dummy keys are leaves (external nodes), and the
data keys mean internal nodes.
3.Formula & Prove
• The case of search are two situations, one is success,

and the other, without saying, is failure.
• We can get the first statement :

(i=1~n) ∑ pi + (i=0~n) ∑ qi = 1
Success Failure
• Because we have probabilities of searches for each key
and each dummy key, we can determine the expected
cost of a search in a given binary search tree T. Let us
assume that the actual cost of a search is the number of
nodes examined, i.e., the depth of the node found by the
search in T,plus1. Then the expected cost of a search in
T is : (The second statement)
• E[ search cost in T]
= (i=1~n) ∑ pi ．(depthT(ki)+1)
+ (i=0~n) ∑ qi ．(depthT(di)+1)
=1 + (i=1~n) ∑ pi ．depthT(ki)
+ (i=0~n) ∑ qi ．depthT(di)
Where depthT denotes a node’s depth in the tree T.
k2 k2
k1 k4 k1 k5
d0 d1
d0 d1 d5
k3 k5 k4
d2 d3 d4 d5 d4
k3
Figure (a)
i 0 1 2 3 4 5
d2 d3
pi 0.15 0.10 0.05 0.10 0.20

Figure (b)
qi 0.05 0.10 0.05 0.05 0.05 0.10

• By Figure (a), we can calculate the expected search cost node by node:
Cost=
Node# Depth probability cost
Probability *
k1 1 0.15 0.30 (Depth+1)
k2 0 0.10 0.10
k3 2 0.05 0.15
k4 1 0.10 0.20
K5 2 0.20 0.60
d0 2 0.05 0.15
d1 3 0.10 0.30
d2 3 0.05 0.20
d3 3 0.05 0.20
d4 3 0.05 0.20
d5 3 0.10 0.40
• And the total cost = (0.30 + 0.10 + 0.15 + 0.20 + 0.60 +
0.15 + 0.30 + 0.20 + 0.20 + 0.20 + 0.40 ) = 2.80
• So Figure (a) costs 2.80 ,on another, the Figure (b) costs
2.75, and that tree is really optimal.
• We can see the height of (b) is more than (a) , and the
key k5 has the greatest search probability of any key, yet
the root of the OBST shown is k2.(The lowest expected
cost of any BST with k5 at the root is 2.85)
Step1:The structure of an OBST
• To characterize the optimal substructure of OBST, we

start with an observation about subtrees. Consider any
subtree of a BST. It must contain keys in a contiguous
range ki,…,kj, for some 1≦i ≦j ≦n. In addition, a subtree
that contains keys ki,…,kj must also have as its leaves
the dummy keys di-1 ,…,dj.
• We need to use the optimal substructure to
show that we can construct an optimal solution
to the problem from optimal solutions to
subproblems. Given keys ki ,…, kj, one of these
keys, say kr (I ≦r ≦j), will be the root of an
optimal subtree containing these keys. The left
subtree of the root kr will contain the keys (ki ,…,
kr-1) and the dummy keys( di-1 ,…, dr-1), and the
right subtree will contain the keys (kr+1 ,…, kj)
and the dummy keys( dr ,…, dj). As long as we
examine all candidate roots kr, where I ≦r ≦j,
and we determine all optimal binary search trees
containing ki ,…, kr-1 and those containing kr+1
,…, kj , we are guaranteed that we will find an
OBST.
Step2: A recursive solution
• We are ready to define the value of an optimal

solution recursively. We pick our subproblem
domain as finding an OBST containing the keys
ki,…,kj, where i≧1, j ≦n, and j ≧ i-1. (It is when
j=i-1 that ther are no actual keys; we have just
the dummy key di-1.)
• Let us define e[i,j] as the expected cost of
searching an OBST containing the keys ki,…, kj.
Ultimately, we wish to compute e[1,n].
• The easy case occurs when j=i-1. Then we have just the
dummy key di-1. The expected search cost is e[i,i-1]= qi-1.
• When j≧1, we need to select a root krfrom among ki,…,kj
and then make an OBST with keys ki,…,kr-1 its left
subtree and an OBST with keys kr+1,…,kj its right
subtree. By the time, what happens to the expected
search cost of a subtree when it becomes a subtree of a
node? The answer is that the depth of each node in the
subtree increases by 1.
OPTIMAL—BST(p,q,n)
• For i 1 to n+1
do e[i,i-1] qi-1
do w[i,i-1] qi-1
For l 1 to n
do for i 1 to n-l +1
do j i+l-1
e[i,j] ∞
w[i,j] w[i,j-1]+pj+qj
For r i to j
do t e[i,r-1]+e[r+1,j]+w[i,j]
if t<e[i,j]
then e[i,j] t
root [i,j] r
Return e and root
e w
5 1
5 1
4 2.75 2
4 1.00 2
3 1.75 2.00 3 0.70 0.80
1.25 3 3
2 1.20 1.30 4 0.55
0.90 5 2 0.50 0.60 4
1 0.70 0.60 0.90
0.45 0.35
0.25 0.50 1 0.30 0.50 5
0 0.45 0.40 0.30 6
0 0.30 0.25 0.15 0.20 0.35 6
0.05 0.10 0.05 0.05 0.05 0.10
0.05 0.10 0.05 0.05 0.05 0.10
root
5 1
4 2 2
3 2 4 3
2 2 2 5 4
1 2 4 5 5
1
1 2 3 4 5
The tables e[i,j], w[i,j], and root [i,j]computed by Optimal-BST

Backtracking | Introduction
Backtracking is an algorithmic-technique for solving problems recursively by trying to build a

solution incrementally, one piece at a time, removing those solutions that fail to satisfy the
constraints of the problem at any point of time
There are three types of problems in backtracking –

1. Decision Problem – In this, we search for a feasible solution.
2. Optimization Problem – In this, we search for the best solution.
3. Enumeration Problem – In this, we find all feasible solutions.
4. Recursive backtracking solution.
void findSolutions(n, other params) :
if (found a solution) :
solutionsFound = solutionsFound + 1;
displaySolution();
if (solutionsFound >= solutionTarget) :
System.exit(0);
return
for (val = first to last) :

if (isValid(val, n)) :
applyValue(val, n);
findSolutions(n+1, other params);
removeValue(val, n);
2. Finding whether a solution exists or not
boolean findSolutions(n, other params) :
if (found a solution) :
displaySolution();
return true;
for (val = first to last) :

if (isValid(val, n)) :
applyValue(val, n);
if (findSolutions(n+1, other params))
return true;
removeValue(val, n);
return false;
Let us try to solve a standard Backtracking problem, N-Queen Problem.
The N Queen is the problem of placing N chess queens on an N×N chessboard so that no two
queens attack each other. For example, following is a solution for 4 Queen problem.
The expected output is a binary matrix which has 1s for the blocks where queens are placed.
For example, following is the output matrix for the above 4 queen solution.
{ 0, 1, 0, 0}
{ 0, 0, 0, 1}
{ 1, 0, 0, 0}
{ 0, 0, 1, 0}
Backtracking Algorithm: The idea is to place queens one by one in different columns,
starting from the leftmost column. When we place a queen in a column, we check for clashes
with already placed queens. In the current column, if we find a row for which there is no clash,
we mark this row and column as part of the solution. If we do not find such a row due to
clashes then we backtrack and return false.
1) Start in the leftmost column
2) If all queens are placed
return true
3) Try all rows in the current column. Do following for every tried row.
a) If the queen can be placed safely in this row then mark this row,
column] as part of the solution and recursively check if placing
queen here leads to a solution.
b) If placing the queen in [row, column] leads to a solution then return
true.
c) If placing queen doesn't lead to a solution then unmark this [row,
column] (Backtrack) and go to step (a) to try other rows.
3) If all rows have been tried and nothing worked, return false to trigger
backtracking.
Subset Sum | Backtracking

Subset sum problem is to find subset of elements that are selected from a given set whose sum adds
up to a given number K. We are considering the set contains non-negative values. It is assumed that
the input set is unique (no duplicates are presented
Exhaustive Search Algorithm for Subset Sum

One way to find subsets that sum to K is to consider all possible subsets. A power set contains all
those subsets generated from a given set. The size of such a power set is 2 N.
Backtracking Algorithm for Subset Sum
Using exhaustive search we consider all subsets irrespective of whether they satisfy
given constraints or not. Backtracking can be used to make a systematic consideration of the elements
to be selected.
Design and Analysis of
Algorithms
5. Coping with the Limitations of Algorithm Power
 Lower Bound Arguments , P, NP, NP- Complete and

NP Hard Problems. Backtracking , nQueen problem ,
Hamiltonian Circuit Problem , Subset Sum Problem.
Branch and Bound, Knapsack Problem , Travelling
Salesman Problem , Approximation Algorithms for
NP-Hard Problems: Travelling Salesman problem,
Knapsack problem.
6/9/2021
Computational Complexity
 The problems those can be solved in polynomial time

are called as P class problems.
 Example: searching, sorting
 A problem which cannot be solved in polynomial
time but can be verified in polynomial time is called
as Non Deterministic Polynomial or NP class
problems.
 Example: su-du-ku, travelling salesman problem,
prime factor, scheduling
6/9/2021
NP Class Problems
 NP class problems are hard to implement

 Easy to verify
 Exponential in nature
 Are verified in polynomial time
 Non-tractable problems
 Security problems are of type NP
6/9/2021
P-Class Problems
 P class problems are easy to implement

 Easy to verify
 Polynomial in nature
 Tractable problems
NP
6/9/2021
Is P = NP ?
 If P = NP is proved then
 Security domain is vulnerable to attacks
 Everything becomes more efficient such as transportation,

scheduling etc.
 If P  NP is proved then
 You can prove that there are some problems those can never
be solved
6/9/2021
Reduction
A B
Let A and B are two problems then problem A reduces

to problem B iff there is a way to solve A by
deterministic algorithm that solves B in polynomial
time.
➢ If A is reducible to B then A  B.
➢ If A is reducible to B and B in P then A in P.
➢ If A not in P implies B not in P.
6/9/2021
NP Hard
 A problem is NP hard if every problem in NP can be

polynomial reduced to it.
 Basically NP hard are optimization problems.
NP
NP Hard
6/9/2021
NP Complete
 A problem is called NP complete if it is in NP and it is

NP-Hard.
 The intersection of NP and NP hard is NP complete
problem.
 Basically NP complete are decision problems.
 Example: If problem A is a Decision problem and
problem B is Optimization problem then it is possible A
 B.
 Example : Knapsack decision problem can be converted into
knapsack optimization problem.
 All NP-Complete problems are NP-Hard but all NP-Hard
problems are not NP-Complete.
6/9/2021
Summary
Computational
complexity
Non deterministic
Polynomial(P)
polynomial(NP-Class)
NP- Hard NP-Complete
6/9/2021
Backtracking
 Backtracking can be defined as a general algorithmic

technique that considers searching every possible
combination in order to solve a computational problem.
 There are three types of problems in backtracking –
 Decision Problem – we search for a feasible solution.
 Optimization Problem – we search for the best solution.
 Enumeration Problem – we find all feasible solutions.
 In recursion, the function calls itself until it reaches a
base case. In backtracking, we use recursion to explore
all the possibilities until we get the best result for the
problem.
6/9/2021
Backtracking - In general
 Step 1 − if current_position is goal, return success

 Step 2 − else,
 Step 3 − if current_position is an end point, return
failed.
 Step 4 − else, if current_position is not end point,
explore and repeat above steps.
6/9/2021
Backtracking algorithm
(Given an instance of any computational problem P and data D corresponding to the instance,
all the constraints that need to be satisfied in order to solve the problem are represented by C.)
 The Algorithm begins to build up a solution, starting with an empty solution set . S = {}
 Add to S the first move that is still left (All possible moves are added to one by one). This now
creates a new sub-tree S in the search tree of the algorithm.
 Check if S + S satisfies each of the constraints in C.

 If Yes, then the sub-tree S is “eligible” to add more “children”.
 Else, the entire sub-tree S is useless, so recurs back to step 1 using argument S .
 In the event of “eligibility” of the newly formed sub-tree S, recurs back to step 1, using
argument S + S .
 If the check for S + S returns that it is a solution for the entire data D . Output and terminate
the program.
If not, then return that no solution is possible with the current S and hence discard it.
6/9/2021
N Queen Problem
In N-Queen problem, we are given an NxN chessboard and

we have to place n queens on the board in such a way that no
two queens attack each other.
A queen will attack another queen if it is placed in horizontal,
vertical or diagonal points in its way.
Expected output of 4 Queen’s Problem

{0 , 1 , 0 , 0}
{0 , 0 , 0 , 1}
{1 , 0 , 0 , 0}
{0 , 0 , 1 , 0}
6/9/2021
Backtracking algorithm for N queen’s problem
 Step 1 − Start from 1st position in the array.

 Step 2 − Place queens in the board and check. Do,
 Step 2.1 − After placing the queen, mark the position as a part of the
solution and then recursively check if this will lead to a solution.
 Step 2.2 − Now, if placing the queen doesn’t lead to a solution and
trackback and go to step (a) and place queens to other rows.
 Step 2.3 − If placing queen returns a lead to solution return TRUE.
 Step 3 − If all queens are placed return TRUE.
 Step 4 − If all rows are tried and no solution is found,
return FALSE.
6/9/2021
TSP nearest neighbourhood mehod
 Step-1
 1) Begin at any city and visit the nearest city.
 2) Then go to the unvisited city closest to the city
most recently used
 3)continue in this fashion until a tour is obtained.
 Step-2
 1) After applying this procedure repeat it by begining
of different city.
 2) Take best tour found.
6/9/2021
Example
A B C D E
A -- 132 217 164 58
B 132 -- 290 201 79
C 217 290 -- 113 303
D 164 201 113 -- 196
E 58 79 303 196 --
6/9/2021
Tours obtained
 1)A-E-B-D-C-A PATH COST TOTAL=668

 2)B-E-A-D-C-B PATH COST TOTAL=704
 3)C-D-A-E-B-C PATH COST TOTAL=704
 4)D-C-A-E-B-D PATH COST TOTAL=668
 5)E-A-B-D-C-E PATH COST TOTAL=807
 Hence A-E-B-D-C-A or D-C-A-E-B-D is the
minimum TSP
6/9/2021
Backtracking algorithm for N queen’s problem
 Step 1 − Start from 1st position in the array.

 Step 2 − Place queens in the board and check. Do,
 Step 2.1 − After placing the queen, mark the position as a part of the
solution and then recursively check if this will lead to a solution.
 Step 2.2 − Now, if placing the queen doesn’t lead to a solution and
trackback and go to step (a) and place queens to other rows.
 Step 2.3 − If placing queen returns a lead to solution return TRUE.
 Step 3 − If all queens are placed return TRUE.
 Step 4 − If all rows are tried and no solution is found,
return FALSE.
6/9/2021
Hamilton Circuit Problem
 A Hamiltonian cycle is a cycle that contains all

vertices in a graph exactly once.
 Example:
Possible Hamilton cycle

for the graph:
a-b-c-e-f-d-a
6/9/2021
Hamilton Circuit Problem
 Given a graph G = (V, E) we have to find the Hamiltonian Circuit using Backtracking approach.
 We start our search from any arbitrary vertex say 'a.' This vertex 'a' becomes the root of our
implicit tree.
 The first element of our partial solution is the first intermediate vertex of the Hamiltonian Cycle
that is to be constructed.
 The next adjacent vertex is selected by alphabetical order. If at any stage any arbitrary vertex
makes a cycle with any vertex other than vertex 'a' then we say that dead end is reached.
 In this case, we backtrack one step, and again the search begins by selecting another vertex and
backtrack the element from the partial; solution must be removed.
 The search using backtracking is successful if a Hamiltonian Cycle is obtained.
6/9/2021
Subset Sum Problem
6/9/2021
The Subset-sum Problem
 Problem statement
 given the set S = {x1, x2, x3, … xn } of positive integers and t, is
there a subset of S that adds up to t
 as an optimization problem, what subset of S adds up to the
greatest total <= t
 What we will show
 first we develop an exponential time algorithm to solve the
problem exactly
 we modify this algorithm to give us a fully polynomial time
approximation scheme that has a running time that is
polynomial in 1/e and n
The exponential algorithm
 Some preliminaries
 we use the notation S + x = { s+x : s  S}
 we have a merge lists algorithm that runs in time proportional to

the sum of the lengths of the two lists
Why it works
 Some notation
 Pi is the set of sums of all
subsets of values up to xi Pi = Pi −1  ( Pi −1 + xi )
 it can be shown that
 Proof and complexity

 by induction on i it can be shown that Li is a sorted list that
contains every element of Pi <= t
 since Li can be as long as 2i, the algorithm, in general, is
exponential
 polynomial time algorithms exist if t is polynomial in |S| or all
numbers in S are bounded by a polynomial in |S|
Trimming a List
 To trim by a factor d means

a term z can be represented by
a smaller term provided it is
“close enough”
The complexity is (m)

A fully-polynomial scheme
 The algorithm is similar to our exponential algorithm
except we now perform a trim step
An example
Branch and Bound
 Where backtracking uses a depth-first search with

pruning, the branch and bound algorithm uses a
breadth-first search with pruning
 Branch and bound uses a queue as an auxiliary data
structure
The Branch and Bound Algorithm
 Starting by considering the root node and applying a

lower-bounding and upper-bounding procedure to it
 If the bounds match, then an optimal solution has
been found and the algorithm is finished
 If they do not match, then algorithm runs on the child
nodes
Example:
The Traveling Salesman Problem
 Branch and bound can be used to solve the TSP

using a priority queue as an auxiliary data
structure
 An example is the problem with a directed graph
given by this adjacency matrix:
Traveling Salesman Problem
 The problem starts at vertex 1

 The initial bound for the minimum tour is the sum of
the minimum outgoing edges from each vertex
Vertex 1 min (14, 4, 10, 20) = 4

Vertex 2 min (14, 7, 8, 7) = 7
Vertex 3 min (4, 5, 7, 16) = 4
Vertex 4 min (11, 7, 9, 2) = 2
Vertex 5 min (18, 7, 17, 4) = 4
Bound = 21
 Next, the bound for the node for the partial tour
from 1 to 2 is calculated using the formula:
Bound = Length from 1 to 2 + sum of min outgoing edges for
vertices 2 to 5 = 14 + (7 + 4 + 2 + 4) = 31
 The node is added to the priority queue

 The node with the lowest bound is then removed
 This calculation for the bound for the node of the
partial tours is repeated on this node
 The process ends when the priority queue is empty
 The final results of this

example are in this tree:
 The accompanying
number for each node is
the order it was removed
in
Efficiency of Branch and Bound
 In many types of problems, branch and bound is

faster than branching, due to the use of a breadth-
first search instead of a depth-first search
 The worst case scenario is the same, as it will still
visit every node in the tree
Knapsack 0-1 Problem
 The goal is to maximize

the value of a
knapsack that can hold
at most W units (i.e. lbs or
kg) worth of goods from a
list of items I0, I1, … In-1.
 Each item has 2 attributes:

1) Value – let this be vi for item Ii
2) Weight – let this be wi for item Ii
 The difference between

this problem and the
fractional knapsack one is
that you CANNOT take a
fraction of an item.
 You can either take it or not.

 Hence the name Knapsack 0-1
problem.
 Brute Force
 The naïve way to solve this problem is to cycle through all 2n
subsets of the n items and pick the subset with a legal weight
that maximizes the value of the knapsack.
 We can come up with a dynamic programming algorithm that

will USUALLY do better than this brute force technique.
 As we did before we are going to solve the problem in
terms of sub-problems.
 So let’s try to do that…
 Our first attempt might be to characterize a sub-

problem as follows:
 Let Sk be the optimal subset of elements from {I0, I1, …, Ik}.
 What we find is that the optimal subset from the elements {I0, I1, …,
Ik+1} may not correspond to the optimal subset of elements from {I0,
I1, …, Ik} in any regular pattern.
 Basically, the solution to the optimization problem for Sk+1 might

NOT contain the optimal solution from problem Sk.
 Let’s illustrate that point with an example:

Item Weight Value
I0 3 10
I1 8 4
I2 9 9
I3 8 11
 The maximum weight the knapsack can hold is 20.
 The best set of items from {I0, I1, I2} is {I0, I1, I2}
 BUT the best set of items from {I0, I1, I2, I3} is {I0, I2, I3}.
 In this example, note that this optimal solution, {I0, I2, I3}, does
NOT build upon the previous optimal solution, {I0, I1, I2}.
 (Instead it build's upon the solution, {I0, I2}, which is really the optimal subset of
{I0, I1, I2} with weight 12 or less.)
Knapsack 0-1 problem
 So now we must re-work the way we build upon previous sub-
problems…
 Let B[k, w] represent the maximum total value of a subset Sk with
weight w.
 Our goal is to find B[n, W], where n is the total number of items and
W is the maximal weight the knapsack can carry.
 So our recursive formula for subproblems:

B[k, w] = B[k - 1,w], if wk > w
= max { B[k - 1,w], B[k - 1,w - wk] + vk}, otherwise
 In English, this means that the best subset of Sk that has total
weight w is:
1) The best subset of Sk-1 that has total weight w, or
2) The best subset of Sk-1 that has total weight w-wk plus the item k
Knapsack 0-1 Problem –
Recursive Formula
 The best subset of Sk that has the total weight w, either

contains item k or not.
 First case: wk > w

 Item k can’t be part of the solution! If it was the total weight would be >
w, which is unacceptable.
 Second case: wk ≤ w
 Then the item k can be in the solution, and we choose the case with
greater value.
Knapsack 0-1 Algorithm
for w = 0 to W { // Initialize 1st row to 0’s

B[0,w] = 0
}
for i = 1 to n { // Initialize 1st column to 0’s
B[i,0] = 0
}
for i = 1 to n {
for w = 0 to W {
if wi <= w { //item i can be in the solution
if vi + B[i-1,w-wi] > B[i-1,w]
B[i,w] = vi + B[i-1,w- wi]
else
B[i,w] = B[i-1,w]
}
else B[i,w] = B[i-1,w] // wi > w
}
}
 Let’s run our algorithm on the following data:

 n = 4 (# of elements)
 W = 5 (max weight)
 Elements (weight, value):
(2,3), (3,4), (4,5), (5,6)

Knapsack 0-1 Example
i/w 0 1 2 3 4 5
0 0 0 0 0 0 0
1 0
2 0
3 0
4 0
// Initialize the base cases

for w = 0 to W
B[0,w] = 0
for i = 1 to n
B[i,0] = 0
Items:
1: (2,3)
2: (3,4)
Knapsack 0-1 Example 3: (4,5)
4: (5,6)
i/w 0 1 2 3 4 5
0 0 0 0 0 0 0 i=1
1 0 0 vi = 3
2 0 wi = 2
3 0 w=1
4 0 w-wi = -1
if wi <= w //item i can be in the solution

B[i,w] = vi + B[i-1,w- wi]
else
B[i,w] = B[i-1,w]
Items:
1: (2,3)
2: (3,4)
4: (5,6)
i/w 0 1 2 3 4 5
0 0 0 0 0 0 0 i=1
1 0 0 3 vi = 3
2 0 wi = 2
3 0 w=2
4 0 w-wi = 0
if wi <= w //item i can be in the solution if wi <= w //item i can be in the solution
if vi + B[i-1,w-wi] > B[i-1,w] if vi + B[i-1,w-wi] > B[i-1,w]
B[i,w] = vi + B[i-1,w- wi] B[i,w] = vi + B[i-1,w- wi]
else else
B[i,w] = B[i-1,w] B[i,w] = B[i-1,w]
else B[i,w] = B[i-1,w] // wi > w else B[i,w] = B[i-1,w] // wi > w
Items:
1: (2,3)
2: (3,4)
4: (5,6)
i/w 0 1 2 3 4 5
0 0 0 0 0 0 0 i=1
1 0 0 3 3 vi = 3
2 0 wi = 2
3 0 w=3
4 0 w-wi = 1

B[i,w] = vi + B[i-1,w- wi]
else
B[i,w] = B[i-1,w]
Items:
1: (2,3)
2: (3,4)
4: (5,6)
i/w 0 1 2 3 4 5
0 0 0 0 0 0 0 i=1
1 0 0 3 3 3 vi = 3
2 0 wi = 2
3 0 w=4
4 0 w-wi = 2

B[i,w] = vi + B[i-1,w- wi]
else
B[i,w] = B[i-1,w]
Items:
1: (2,3)
2: (3,4)
4: (5,6)
i/w 0 1 2 3 4 5
0 0 0 0 0 0 0 i=1
1 0 0 3 3 3 3 vi = 3
2 0 wi = 2
3 0 w=5
4 0 w-wi = 3

B[i,w] = vi + B[i-1,w- wi]
else
B[i,w] = B[i-1,w]
Items:
1: (2,3)
2: (3,4)
4: (5,6)
i/w 0 1 2 3 4 5
0 0 0 0 0 0 0 i=2
1 0 0 3 3 3 3 vi = 4
2 0 0 wi = 3
3 0 w=1
4 0 w-wi = -2
else else
B[i,w] = B[i-1,w] B[i,w] = B[i-1,w]
Items:
1: (2,3)
2: (3,4)
4: (5,6)
i/w 0 1 2 3 4 5
0 0 0 0 0 0 0 i=2
1 0 0 3 3 3 3 vi = 4
2 0 0 3 wi = 3
3 0 w=2
4 0 w-wi = -1

B[i,w] = vi + B[i-1,w- wi]
else
B[i,w] = B[i-1,w]
Items:
1: (2,3)
2: (3,4)
4: (5,6)
i/w 0 1 2 3 4 5
0 0 0 0 0 0 0 i=2
1 0 0 3 3 3 3 vi = 4
2 0 0 3 4 wi = 3
3 0 w=3
4 0 w-wi = 0
else else
B[i,w] = B[i-1,w] B[i,w] = B[i-1,w]
Items:
1: (2,3)
2: (3,4)
4: (5,6)
i/w 0 1 2 3 4 5
0 0 0 0 0 0 0 i=2
1 0 0 3 3 3 3 vi = 4
2 0 0 3 4 4 wi = 3
3 0 w=4
4 0 w-wi = 1

B[i,w] = vi + B[i-1,w- wi]
else
B[i,w] = B[i-1,w]
Items:
1: (2,3)
2: (3,4)
4: (5,6)
i/w 0 1 2 3 4 5
0 0 0 0 0 0 0 i=2
1 0 0 3 3 3 3 vi = 4
2 0 0 3 4 4 7 wi = 3
3 0 w=5
4 0 w-wi = 2

B[i,w] = vi + B[i-1,w- wi]
else
B[i,w] = B[i-1,w]
Items:
1: (2,3)
2: (3,4)
4: (5,6)
i/w 0 1 2 3 4 5
0 0 0 0 0 0 0 i=3
1 0 0 3 3 3 3 vi = 5
2 0 0 3 4 4 7 wi = 4
3 0 0 3 4 w = 1..3
4 0 w-wi = -3..-1

else else
B[i,w] = B[i-1,w] B[i,w] = B[i-1,w]
Items:
1: (2,3)
2: (3,4)
4: (5,6)
i/w 0 1 2 3 4 5
0 0 0 0 0 0 0 i=3
1 0 0 3 3 3 3 vi = 5
2 0 0 3 4 4 7 wi = 4
3 0 0 3 4 5 w=4
4 0 w-wi = 0
else else
B[i,w] = B[i-1,w] B[i,w] = B[i-1,w]
Items:
1: (2,3)
2: (3,4)
4: (5,6)
i/w 0 1 2 3 4 5
0 0 0 0 0 0 0 i=3
1 0 0 3 3 3 3 vi = 5
2 0 0 3 4 4 7 wi = 4
3 0 0 3 4 5 7 w=5
4 0 w-wi = 1
else else
B[i,w] = B[i-1,w] B[i,w] = B[i-1,w]
Items:
1: (2,3)
2: (3,4)
4: (5,6)
i/w 0 1 2 3 4 5
0 0 0 0 0 0 0 i=4
1 0 0 3 3 3 3 vi = 6
2 0 0 3 4 4 7 wi = 5
3 0 0 3 4 5 7 w = 1..4
4 0 0 3 4 5 w-wi = -4..-1
if wi <= w //item i can be in the if wi <= w //item i can be in the solution

solution
B[i,w] = vi + B[i-1,w- wi]
B[i,w] = vi + B[i-1,w- wi]
else else
B[i,w] = B[i-1,w] B[i,w] = B[i-1,w]
Items:
1: (2,3)
2: (3,4)
4: (5,6)
i/w 0 1 2 3 4 5
0 0 0 0 0 0 0 i=4
1 0 0 3 3 3 3 vi = 6
2 0 0 3 4 4 7 wi = 5
3 0 0 3 4 5 7 w=5
4 0 0 3 4 5 7 w-wi = 0

else else
B[i,w] = B[i-1,w] B[i,w] = B[i-1,w]
Items:
1: (2,3)
2: (3,4)
4: (5,6)
i/w 0 1 2 3 4 5
0 0 0 0 0 0 0
1 0 0 3 3 3 3
2 0 0 3 4 4 7
3 0 0 3 4 5 7
4 0 0 3 4 5 7
We’re DONE!!
The max possible value that can be carried in this knapsack is $7
 This algorithm only finds the max possible value that

can be carried in the knapsack
 The value in B[n,W]
 To know the items that make this maximum value,

we need to trace back through the table.
Finding the Items
 Let i = n and k = W
if B[i, k] ≠ B[i-1, k] then
mark the ith item as in the knapsack
i = i-1, k = k-wi
else
i = i-1 // Assume the ith item is not in the knapsack
// Could it be in the optimally packed knapsack?
Items: Knapsack:
Knapsack 0-1 Algorithm 1: (2,3)
2: (3,4)
Finding the Items 3: (4,5)
4: (5,6)
i/w 0 1 2 3 4 5
0 0 0 0 0 0 0 i=4
1 0 0 3 3 3 3 k=5
2 0 0 3 4 4 7 vi = 6
wi = 5
3 0 0 3 4 5 7
B[i,k] = 7
4 0 0 3 4 5 7
B[i-1,k] = 7
i=n,k=W
while i, k > 0
i = i-1, k = k-wi
else
i = i-1
Items: Knapsack:
Knapsack 0-1 Algorithm 1: (2,3)
2: (3,4)
4: (5,6)
i/w 0 1 2 3 4 5
0 0 0 0 0 0 0 i=3
1 0 0 3 3 3 3 k=5
2 0 0 3 4 4 7 vi = 5
wi = 4
3 0 0 3 4 5 7
B[i,k] = 7
4 0 0 3 4 5 7
B[i-1,k] = 7
i=n,k=W
while i, k > 0
i = i-1, k = k-wi
else
i = i-1
Items: Knapsack:
Knapsack 0-1 Algorithm 1: (2,3) Item 2
2: (3,4)
4: (5,6)
i/w 0 1 2 3 4 5
0 0 0 0 0 0 0 i=2
1 0 0 3 3 3 3 k=5
2 0 0 3 4 4 7 vi = 4
wi = 3
3 0 0 3 4 5 7
B[i,k] = 7
4 0 0 3 4 5 7
B[i-1,k] = 3
k – wi = 2
i=n,k=W
while i, k > 0
i = i-1, k = k-wi
else
i = i-1
Items: Knapsack:
2: (3,4)
Item 1
4: (5,6)
i/w 0 1 2 3 4 5
0 0 0 0 0 0 0 i=1
1 0 0 3 3 3 3 k=2
2 0 0 3 4 4 7 vi = 3
wi = 2
3 0 0 3 4 5 7
B[i,k] = 3
4 0 0 3 4 5 7
B[i-1,k] = 0
k – wi = 0
i=n,k=W
while i, k > 0
i = i-1, k = k-wi
else
i = i-1
Items: Knapsack:
2: (3,4)
Item 1
4: (5,6)
i/w 0 1 2 3 4 5
0 0 0 0 0 0 0 i=1
1 0 0 3 3 3 3 k=2
2 0 0 3 4 4 7 vi = 3
wi = 2
3 0 0 3 4 5 7
B[i,k] = 3
4 0 0 3 4 5 7
B[i-1,k] = 0
k – wi = 0
k = 0, so we’re DONE!
The optimal knapsack should contain:

Item 1 and Item 2
Knapsack 0-1 Problem – Run Time
for w = 0 to W O(W)
B[0,w] = 0
for i = 1 to n O(n)
B[i,0] = 0
for i = 1 to n Repeat n times

for w = 0 to W O(W)
< the rest of the code >
What is the running time of this algorithm?

O(n*W)
Remember that the brute-force algorithm takes: O(2n)

Knapsack Problem
1) Fill out the dynamic

programming table for
the knapsack problem
to the right.
2) Trace back through the
table to find the items in
the knapsack.
Approximation algorithm
 Up to now, the best algorithm for solving an NP-

complete problem requires exponential time in the
worst case. It is too time-consuming.
 To reduce the time required for solving a problem,
we can relax the problem, and obtain a feasible
solution “close” to an optimal solution
Approximation Ratios
9-73
 Optimization Problems
 We have some problem instance x that has many feasible
“solutions”.
 We are trying to minimize (or maximize) some cost
function c(S) for a “solution” S to x.

For example,
 Š inding a minimum spanning tree of a graph
F
 Finding a smallest vertex cover of a graph
 Finding a smallest traveling salesperson tour in a graph
Š
Approximation Ratios
9-74
 An approximation produces a solution T
 Relative approximation ratio

 T is a k-approximation to the optimal solution OPT
if c(T)/c(OPT) < k (assuming a minimizing problem;
a maximization approximation would be the reverse)
 Absolute approximation ratio

 For example, chromatic number problem
 If the optimal solution of this instance is three and the
approximation T is four, then T is a 1-approximation to the optimal
solution.
The Euclidean traveling salesperson
problem (ETSP)
 The ETSP is to find a shortest closed path

through a set S of n points in the plane.
 The ETSP is NP-hard.
An approximation algorithm for ETSP
9-76
 Input: A set S of n points in the plane.

 Output: An approximate traveling salesperson
tour of S.
Step 1: Find a minimal spanning tree T of S.
Step 2: Find a minimal Euclidean weighted
matching M on the set of vertices of odd degrees
in T. Let G=M∪T.
Step 3: Find an Eulerian cycle of G and then
traverse it to find a Hamiltonian cycle as an
approximate tour of ETSP by bypassing all
previously visited vertices.
An example for ETSP algorithm
9-77
 Step1: Find a minimal spanning tree.

9-78
 Step2: Perform weighted matching. The number of

points with odd degrees must be even because
n
is even.  di = 2 E
i =1
9-79
 Step3: Construct the tour with an Eulerian
cycle and a Hamiltonian cycle.
9-80
 Time complexity: O(n3)

Step 1: O(nlogn)
Step 2: O(n3)
Step 3: O(n)
 How close the approximate solution to an

optimal solution?
 The approximate tour is within 3/2 of the optimal
one. (The approximate rate is 3/2.)
Proof of approximate rate
9-81
 optimal tour L: j1…i1j2…i2j3…i2m

{i1,i2,…,i2m}: the set of odd degree vertices in T.
2 matchings: M1={[i1,i2],[i3,i4],…,[i2m-1,i2m]}
M2={[i2,i3],[i4,i5],…,[i2m,i1]}
length(L) length(M1) + length(M2) (triangular inequality)
 2 length(M )
 length(M) 1/2 length(L )
G = T∪M
 length(T) + length(M)  length(L) + 1/2 length(L)
= 3/2 length(L)
The bottleneck traveling salesperson
problem (BTSP) 9-82
 Minimize the longest edge of a tour.

 This is a mini-max problem.
 This problem is NP-hard.
 The input data for this problem fulfill the following
assumptions:
 The graph is a complete graph.
 All edges obey the triangular inequality rule.
An algorithm for finding an optimal solution
9-83
Step1: Sort all edges in G = (V,E) into a

nondecresing sequence |e1||e2|…|em|. Let
G(ei) denote the subgraph obtained from G by
deleting all edges longer than ei.
Step2: i←1
Step3: If there exists a Hamiltonian cycle in
G(ei), then this cycle is the solution and stop.
Step4: i←i+1 . Go to Step 3.
An example for BTSP algorithm
9-84
 e.g.
◼ There is a Hamiltonian
cycle, A-B-D-C-E-F-G-A, in
G(BD).
◼ The optimal solution is 13.
Theorem for Hamiltonian cycles
9-85
 Def : The t-th power of G=(V,E), denoted as
Gt=(V,Et), is a graph that an edge (u,v)Et if
there is a path from u to v with at most t
edges in G.
 Theorem: If a graph G is bi-connected, then
G2 has a Hamiltonian cycle.
An example for the theorem
9-86
G2
A Hamiltonian cycle:
A-B-C-D-E-F-G-A
An approximation algorithm for BTSP
9-87
 Input: A complete graph G=(V,E) where all edges satisfy

triangular inequality.
 Output: A tour in G whose longest edges is not greater than
twice of the value of an optimal solution to the special
bottleneck traveling salesperson problem of G.
Step 1: Sort the edges into |e1||e2|…|em|.
Step 2: i := 1.
Step 3: If G(ei) is bi-connected, construct G(ei)2, find a
Hamiltonian cycle in G(ei)2 and return this as the output.
Step 4: i := i + 1. Go to Step 3.
An example
9-88
Add some more edges.

Then it becomes bi-
connected.
9-89
 A Hamiltonian cycle:
A-G-F-E-D-C-B-A.
 The longest edge: 16
 Time complexity:
polynomial time
How good is the solution ?
9-90
 The approximate solution is bounded by two

times an optimal solution.
 Reasoning:
A Hamiltonian cycle is bi-connected.
eop: the longest edge of an optimal solution
G(ei): the first bi-connected graph
|ei||eop|
The length of the longest edge in G(ei)22|ei|
(triangular inequality) 2|eop|
NP-completeness
9-91
 Theorem: If there is a polynomial

approximation algorithm which produces a
bound less than two, then NP=P.
(The Hamiltonian cycle decision problem
reduces to this problem.)
 Proof:
For an arbitrary graph G=(V,E), we expand G to a
complete graph Gc:
Cij = 1 if (i,j)  E
Cij = 2 if otherwise
(The definition of Cij satisfies the triangular inequality.)
9-92
Let V* denote the value of an optimal solution of the

bottleneck TSP of Gc.
V* = 1  G has a Hamiltonian cycle
Because there are only two kinds of edges, 1 and 2 in Gc, if

we can produce an approximate solution whose value is less
than 2V*, then we can also solve the Hamiltonian cycle
decision problem.
The bin packing problem
9-93
 n items a1, a2, …, an, 0 ai  1, 1  i  n, to

determine the minimum number of bins of unit
capacity to accommodate all n items.
 E.g. n = 5, {0.3, 0.5, 0.8, 0.2 0.4}
◼ The bin packing problem is NP-hard.

An approximation algorithm for the bin
packing problem
9-94
 An approximation algorithm:
(first-fit) place ai into the lowest-indexed bin
which can accommodate ai.
 Theorem: The number of bins used in the first-fit

algorithm is at most twice of the optimal solution.
Proof of the approximate rate
 Notations: 9-95
 S(ai): the size of ai
 OPT(I): the size of an optimal solution of an instance I
 FF(I): the size of bins in the first-fit algorithm
 C(Bi): the sum of the sizes of aj’s packed in bin Bi in the first-
fit algorithm
 OPT(I) 
C(Bi) + C(Bi+1
n
)1
 S ( ai )
m nonemptyi =1bins are used in FF:
C(B1)+C(B2)+…+C(Bm)  m/2
 FF(I) = m < 2 =2  2 OPT(I)
FF(I) < 2 OPT(I) m n
 C( B )
i  S (a )
i =1
i
i =1
Knapsack problem
9-96
 Fractional knapsack problem

 P
 0/1 knapsack problem

 NP-Complete
 Approximation
 PTAS
Fractional knapsack problem
4-
97
 n objects, each with a weight wi > 0

a profit pi > 0
capacity of knapsack: M
Maximize  pi xi
Subject to 1 i  n
0  xi  1, 1  i 1n w i xi  M
i n
The knapsack algorithm
4-
98
 The greedy algorithm:

Step 1: Sort pi/wi into nonincreasing order.
Step 2: Put the objects into the knapsack according
to the sorted sequence as possible as we can.
 e. g.
n = 3, M = 20, (p1, p2, p3) = (25, 24, 15)
(w1, w2, w3) = (18, 15, 10)
Sol: p1/w1 = 25/18 = 1.32
p2/w2 = 24/15 = 1.6
p3/w3 = 15/10 = 1.5
Optimal solution: x1 = 0, x2 = 1, x3 = 1/2
0/1 knapsack problem
3-
 Def: n objects, each with a weight wi > 0
99
a profit pi > 0
capacity of knapsack : M
Maximize pixi
1in
Subject to wixi  M
1in
xi = 0 or 1, 1 i n
 Decision version :
Given K,  pixi  K ?
1in
 Knapsack problem : 0  xi  1, 1 i n.

<Theorem> partition  0/1 knapsack decision
problem.
Polynomial-Time Approximation
9-
100 Schemes
 A problem L has a polynomial-time approximation

scheme (PTAS) if it has a polynomial-time
(1+ε)-approximation algorithm, for any fixed ε >0 (this
value can appear in the running time).
 0/1 Knapsack has a PTAS, with a running time that is

O(n^3 / ε).
Knapsack: PTAS
9-
101
 Intuition for approximation algorithm.

 Given a error ration ε, we calculate a threshold to classify
items
 BIG → enumeration ; SMALL →greedy
i 1 2 3 4 5 6 7 8
pi 90 61 50 33 29 23 15 13
wi 33 30 25 17 15 12 10 9
pi/wi 2.72 2.03 2.0 1.94 1.93 1.91 1.5 1.44
 In our case, T will be found to be 46.8 . Thus BIG = {1, 2, 3} and

SMALL = {4, 5, 6, 7, 8}.
Knapsack: PTAS
9-
102
 For the BIG, we try to enumerate all possible solutions.
 Solution 1:
 We select items 1 and 2. The sum of normalized profits
is 15. The corresponding sum of original profits is 90 +
61 = 151. The sum of weights is 63.
 Solution 2:
 We select items 1, 2, and 3. The sum of normalized
profits is 20. The corresponding sum of original profits
is 90 + 61 + 50 = 201. The sum of weights is 88.
Knapsack: PTAS
9-
103
 For the SMALL, we use greedy strategy to find a possible

solutions.
 Solution 1:
 For Solution 1, we can add items 4 and 6. The sum of profits will be
151 + 33 + 23 = 207.
 Solution 2:
 For Solution 2, we can not add any item from SMALL. Thus the sum
of profits is 201.
A bad example
9-
104
 A convex hull of n points in the plane can be

computed in O(nlogn) time in the worst case.
 An approximation algorithm:
Step1: Find the leftmost and rightmost points.
9-
105
Step2: Divide the points into K strips. Find the

highest and lowest points in each strip.
Step3: Apply the Graham scan to those highest
and lowest points to construct
9-
106
an approximate
convex hull. (The highest and lowest points
are already sorted by their x-coordinates.)
Time complexity
9-
107
 Time complexity: O(n+k)

Step 1: O(n)
Step 2: O(n)
Step 3: O(k)
How good is the solution ?
9-
108
 How far away the points outside are from the

approximate convex hull?
Answer: L/K.
 L: the distance between the leftmost and
rightmost points.
 Thank you
6/9/2021

Design and Analysis of Algorithms: Course Code

Uploaded by

Document Information

Original Description:

Original Title

Copyright

Available Formats

Share this document

Share or Embed Document

Sharing Options

Did you find this document useful?

Is this content inappropriate?

Copyright:

Available Formats

Design and Analysis of Algorithms: Course Code

Uploaded by

Copyright:

Available Formats

Design and Analysis of

– the core of computer science

– A practitioner’s toolkit of known algorithms

– Framework for designing and analyzing

“Algorithmics is more than a branch of computer

⚫An algorithm is a finite set of instructions that

Input “Computer” Output

3. Choosing between Exact and Approximate Problem Solving

7. Proving an Algorithm’s Correctness

“The choice of an appropriate size metric can be influenced by

An input to an algorithm specifies an instance of the problem to be solved

⚫We need to identify most important operation of the

⚫The operation contributing the most to the total running

⚫Thus, the established framework for the analysis of an

T (n) ≈ cop C(n)

Algorithms that require an exponential number of operations are

⚫We established that it is reasonable to measure an

⚫But there are many algorithms for which running time

⚫Worst-Case, Best-Case, and Average-Case Efficiencies

⚫The worst-case efficiency of an algorithm is its efficiency for the worst-

⚫The way to determine the worst-case efficiency of an algorithm is:

⚫The best-case efficiency of an algorithm is its efficiency for the best-

⚫The average-case efficiency cannot be obtained by taking the average

⚫The direct approach for doing this involves –

⚫Divide and conquer ⚫Dynamic programming

⚫Decrease and conquer ⚫Backtracking and branch-and-

⚫To compare and rank such orders of growth, computer

ALGORITHM MaxElement(A[0..n − 1])

M(n) = M(n − 1) + 1+ M(n − 1) for n > 1.

M(n) = 2M(n − 1) + 1 for n > 1,

Divide and Conquer Approach

Prepared By - REETIKA KERKETTA 2

• Divide: Divide the n-element sequence to be sorted into two

• Conquer: Sort the two subsequences recursively using merge

• Combine: Merge the two sorted subsequences to produce the

Prepared By - REETIKA KERKETTA 3

MergeSort (A, p, r) // sort A[p..r] by divide & conquer

Prepared By - REETIKA KERKETTA 5

• If array only contains one element, return

Prepared By - REETIKA KERKETTA 7

• The sub-arrays are stored in the original data array.

Partitioning loops through, swapping elements below/above

Prepared By - REETIKA KERKETTA 8

• OR ... modify the PARTITION procedure

Prepared By - REETIKA KERKETTA 9

Prepared By - REETIKA KERKETTA 10

P1 = (A11+ A22)(B11+B22) C11 = P1 + P4 - P5 + P7

Prepared By - REETIKA KERKETTA 13

◼ An optimization problem is one in which you want

◼ Time to completion: 18 + 11 + 6 = 35 minutes

3 2 in the spanning tree:

4 ◼ To break the cycle, you must remove

• Dynamic programming (DP) is used to solve a wide

• Consider the problem of finding a shortest path between

• A graph for which the shortest path between nodes 0

• The solution to a DP problem is typically expressed as a

Here, g is the composition function.

The computation and composition of subproblem solutions

• The recursive DP equation is also called the functional

• The dependencies between subproblems can be

• We are given a knapsack of capacity c and a set of n objects

(that is, the objects fit into the knapsack) and