ADVANCED ANALYSIS OF ALGORITHMS

(LECTURE-1)
SHAHBAZ AHMAD SAHI
RECOMMENDED BOOKS

◼ Book Title: Introduction to Algorithms

◼ Citation: Cormen et al. (2009)
◼ Author: Thomas. H. Cormen, Charles. E. Leiserson, Ronald. L. Rivest and Clifford Stein
◼ Edition: Third Edition
◼ ISBN: 978-0-262-53305-8
◼ Publisher: MIT Press and McGraw-Hill
GRADING POLICY

◼ Assignments 10 %
◼ Quizzes 05 %
◼ Final Paper 25%
◼ Mid-Term 20 %
◼ Final 40 %
COURSE CONTENTS

◼ Basic Concepts: Introduction, Asymptotic analysis, Growth of functions, Divide-and

conquer technique, Recurrence Relations, Randomized algorithms, Heuristic
algorithms, Brute Force approach, Dynamic algorithms, Greedy algorithms,
Backtracking
COURSE CONTENTS

◼ Graph Algorithms: Graph Theory, Optimization techniques, Searching algorithms, Minimal spanning
tree algorithms, Shortest Path, Maximum Flow Problem
◼ NP completeness
◼ Advance Techniques: Multithreaded/ Parallel Algorithms, Polynomials and FFT,
◼ Approximation Algorithms
◼ Number theoretic algorithms, RSA cryptosystems,
◼ Pattern matching, Computational Geometry,
◼ Image processing and streaming algorithms.
OBJECTIVE OF THIS COURSE

◼ Major objective of this course is:

◼ Design and analysis of modern algorithms
◼ Different variants
◼ Accuracy
◼ Efficiency
◼ Comparing efficiencies
◼ Motivation thinking new algorithms
◼ Advanced designing techniques
◼ Real world problems will be taken as examples
◼ To create feelings about usefulness of this course
EXPECTED RESULTS

◼ On successful completion, students will be able to

◼ Argue and prove correctness of algorithms
◼ Derive and solve mathematical models of problems
◼ Reasoning when an algorithm calls certain approach
◼ Integrating approaches in dynamic and greedy algorithms
◼ Use of graph theory in problems solving
◼ Advanced topics such as
◼ Computational geometry, number theory etc.

◼ Several other algorithms such as

◼ Pattern matching, NP completeness, approximate algorithms etc.
EXPECTED RESULTS

◼ On successful completion, students will be able to

◼ The students will also produce a publishable SCI/EI index journal research paper at the end of this course as a final course
assignment..
LECTURE - 1
INTRODUCTION (WHY, WHAT, HOW)
WHAT IS ALGORITHM?

◼ A computer algorithm is a detailed step-by-step method for solving a problem by using a computer.
◼ An algorithm is a sequence of unambiguous instructions for solving a problem in a finite amount of
time.
◼ More generally, an Algorithm is any well-defined computational procedure that takes collection of
elements as input and produces a collection of elements as output.

Input Algorithm output

POPULAR ALGORITHMS, FACTORS OF DEPENDENCE

◼ Most basic and popular algorithms are

◼ Sorting algorithms
◼ Searching algorithms
◼ Which algorithm is best?
◼ Mainly, it depends upon various factors, for example in case of sorting
◼ The number of items to be sorted
◼ The extent to which the items are already sorted
◼ Possible restrictions on the item values
◼ The kind of storage device to be used etc.
ONE PROBLEM, MANY ALGORITHMS

◼ Problem
◼ The statement of the problem specifies, in general terms, the desired input/output relationship.
◼ Algorithm
◼ The algorithm describes a specific computational procedure for achieving input/output relationship.
◼ Example
◼ One might need to sort a sequence of numbers into non-decreasing order.
◼ Algorithms
◼ Various algorithms e.g. merge sort, quick sort, heap sorts, radix sort, counting sort etc.
IMPORTANT DESIGNING TECHNIQUES

◼ Brute Force
◼ Straightforward, naïve/simple approach
◼ Mostly expensive
◼ Divide-and-Conquer
◼ Divide into smaller sub-problems
◼ Iterative Improvement
◼ Improve one change at a time
◼ Decrease-and-Conquer
◼ Decrease instance size
◼ Transform-and-Conquer
◼ Modify problem first and then solve it
◼ Space and Time Tradeoffs
◼ Use more space now to save time later
SOME OF THE IMPORTANT DESIGNING TECHNIQUES

◼ Greedy Approach
◼ Locally optimal decisions, can not change once made.
◼ Efficient
◼ Easy to implement
◼ The solution is expected to be optimal
◼ Every problem may not have greedy solution
◼ Dynamic programming
◼ Decompose into sub-problems like divide and conquer
◼ Sub-problems are dependant
◼ Record results of smaller sub-problems
◼ Re-use it for further occurrence
◼ Mostly reduces complexity exponential to polynomial
PROBLEM SOLVING PHASES

◼ Analysis
◼ How does system work?
◼ Breaking a system down to known components
◼ How components (processes) relate to each other
◼ Breaking a process down to known functions
◼ Synthesis
◼ Building tools
◼ Building functions with supporting tools
◼ Composing functions to form a process
◼ How components should be put together?
◼ Final solution
PROBLEM SOLVING PROCESS

◼ Problem
◼ Strategy
◼ Algorithm
◼ Input
◼ Output
◼ Steps
◼ Analysis
◼ Correctness
◼ Time & Space
◼ Optimality
◼ Implementation
◼ Verification
MODEL OF COMPUTATION (ASSUMPTIONS)

◼ Design assumption
◼ Level of abstraction which meets our requirements
◼ Neither more nor less e.g. [0, 1] infinite continuous interval
◼ Analysis independent of the variations in
◼ Machine
◼ Operating system
◼ Programming languages
◼ Compiler etc.
◼ Low-level details will not be considered
◼ Our model will be an abstraction of a standard generic single-processor machine, called a random access machine or
RAM.
MODEL OF COMPUTATION (ASSUMPTIONS)

◼ A RAM is assumed to be an idealized machine

◼ Infinitely large random-access memory
◼ Instructions execute sequentially
◼ Every instruction is in fact a basic operation on two values in the machines memory which takes unit time.
◼ These might be characters or integers.
◼ Example of basic operations include
◼ Assigning a value to a variable
◼ Arithmetic operation (+, - , × , /) on integers
◼ Performing any comparison e.g. a < b
◼ Boolean operations
◼ Accessing an element of an array.
MODEL OF COMPUTATION (ASSUMPTIONS)

◼ In theoretical analysis, computational complexity

◼ Estimated in asymptotic sense, i.e.
◼ Estimating for large inputs
◼ Big O, Omega, Theta etc. notations are used to compute the complexity
◼ Asymptotic notations are used because different implementations of algorithm may differ in efficiency
◼ Efficiencies of two given algorithm are related
◼ By a constant multiplicative factor
◼ Called hidden constant.
DRAWBACKS IN MODEL OF COMPUTATION

◼ First poor assumption

◼ We assumed that each basic operation takes constant time, i.e. model allows
◼ Adding
◼ Multiplying
◼ Comparing etc.

◼ two numbers of any length in constant time

◼ Addition of two numbers takes a unit time!
◼ Not good because numbers may be arbitrarily

◼ Addition and multiplication both take unit time!

◼ Again very bad assumption
MODEL OF COMPUTATION NOT SO BAD

◼ Finally what about Our Model?

◼ But with all these weaknesses, our model is not so bad because we have to give the
◼ Comparison not the absolute analysis of any algorithm.
◼ We have to deal with large inputs not with the small size
◼ Model seems to work well describing computational power of modern nonparallel machines
◼ Can we do Exact Measure of Efficiency ?
◼ Exact, not asymptotic, measure of efficiency can be sometimes computed but it usually requires certain
assumptions concerning implementation
DESIGN AND ANALYSIS

◼ In order to understand design and analysis procedure we will do an

example
SELECTION PROBLEM

◼ Suppose you want to purchase a Laptop

◼ You want to pick fastest Laptop
◼ But fast Laptops are expensive
◼ You cannot decide which one is more important price or speed
SELECTION PROBLEM

◼ Definitely do not want a Laptop if there is a another Laptop that is both fast and cheaper
◼ We say that fast cheap Laptops “dominates” the slow expensive
◼ So given a list of Laptops we want those that are not dominated by the other
CRITERION FOR SELECTION

◼ Two criterion for selection:

◼ Speed
◼ Price
MATHEMATICAL MODEL

◼ P is in two dimensional space and its coordinates are P=(p.x , p.y)

◼ (x,y)
◼ x is speed of Laptop
◼ y is negation of price
◼ We can say about y
◼ High value of y mean cheap Laptop and low y means Laptop is expensive
◼ A point p is said to be dominated by a point q if p.x <= q.x and p.y <= q.y
◼ In an array of points (Laptops) P={p1,p2,p3,….,pn} a maximal point is not dominated by any other point
EXAMPLE

◼ Given a set of point P={p1,p2,p3,….,pn } output the maximal points

◼ Those points of P such that pi is not dominated by any other point
BRUTE-FORCE ALGORITHM

Maximal(int n, Point p[1……n])

for i = 1 to n
maximal = true
for j = 1 to n
if( i != j ) and (p[i].x <= p [j]. x) and (p [i] .y <= p [j].y)
maximal = false;
break;
if (maximal == true)
then output p [i].x , p[i]. y
RUNNING TIME ANALYSIS

◼ We can observe that:

◼ The running time depends upon the input size.
◼ Different input of the same size may result different time
RUNNING TIME ANALYSIS

◼ Count the number of steps of the pseudo code that are executed
◼ Or count the number of times an element of p is accessed
◼ Or number of comparisons that are performed
ANALYSIS OF SELECTION PROBLEM

◼ Input size is n

◼ We will count how number of steps

BRUTE-FORCE ALGORITHM

Maximal(int n, Point p[1……n])

n times
for i = 1 to n
1 time Worst Case
maximal = true
n times Analysis
for j = 1 to n
if( i != j ) and (p[i].x <= p [j]. x) and (p [i] .y <= p [j].y)
maximal = false; 5 times
break;
if (maximal == true)
2 times
then output p [i].x , p[i]. y
ANALYSIS OF SELECTION PROBLEM

◼ The outer loop runs n times

◼ For each iteration. The inner loop runs n times in the worst case
◼ P is accessed four times in the if statement
◼ The output statement accesses p 2 times
BRUTE-FORCE ALGORITHM

Maximal(int n, Point p[1……n])

n times
for i = 1 to n
1 time
maximal = true
n times
for j = 1 to n
if( i != j ) and (p[i].x <= p [j]. x) and (p [i] .y <= p [j].y)
maximal = false; 5 times
break;
if (maximal == true)
2 times
then output p [i].x , p[i]. y
ANALYSIS OF SELECTION PROBLEM

◼ In the worst case every point is maximal so every point is output

◼ Worst-case time is:
ANALYSIS OF SELECTION PROBLEM
ANALYSIS OF SELECTION PROBLEM

• Suppose n=50 and

• We use a computer whose speed is
10000000 instructions per second
• Then this computer requires
COMPLEXITY OF THE ALGORITHMS

◼ Algorithm complexity is the work done by the algorithm there are following
complexities:
◼ Worst case time
◼ Average case time
◼ Best case time
◼ Expected time
GROWTH OF FUNCTIONS

Input Size Time of T(n)

50 0.01265
100 0.0503
500 1.2515
1000 5.003
5000 125.015
10000 500.03
100000 50000.3
1000000 5000003
IMPORTANT

◼ We will almost always work with worst case time.

◼ Average case time is most difficult to compute
◼ Worst case refer to the upper limit on the running time
ANALYSIS OF SELECTION PROBLEM

T(n) is Asymptotically equivalent to n2

GROWTH OF FUNCTIONS

◼ At what rate the functions will grow if we increase the input size approximately
towards infinite
HOW FUNCTIONS GROW
Algorithm 1 2 3 4
33n 46nlgn 13n2 3.4n3 2n
Input(n) Solution time

10 .0003s .0015s .0013 .0043 .001s

100 .003s .03s .13s 3.4s 4.1016y

1000 .033s.45s 13s .94h

10000 .33s 6.1s 22m 39d
100000 3.3s 1.3m 1.5d 108y
 BIG THETA Θ

◼ Given a function g(n), we define Θ(g(n)) to be a set of functions that are asymptotically equivalent to
g(n)
◼ For example functions:
◼ 4n2
◼ 8n2+2n-3
◼ n2/5+10log n
◼ All are asymptotically equivalent Θ(n2)
FORMAL DEFINITION

◼ Θ(g(n))={ f(n): there exist positive constants c1, c2 and n0 such that
◼ 0 ≤ c1g(n) ≤ f(n) ≤ c2g(n)
◼ for all n ≥ n0
◼ }
GRAPHICALLY
ASYMPTOTIC NOTATION

◼ This is written as “f(n) = Θ(g(n)) “ or

◼ “f(n) ε Θ(g(n)) “
◼ That is f(n) and g(n) are asymptotically equivalent
◼ This means that they have essentially same growth rate
EXAMPLE ASYMPTOTIC ANALYSIS

◼ Algorithm 3 Cost
◼ sum = 0; c1
◼ for(i=0; i<N; i++) c2
◼ for(j=0; j<N; j++) c2
◼ sum += arr[i][j]; c3
◼ ------------
◼ c1 + c2 x (N+1) + c2 x N x (N+1) + c3 x N2 = O(N2)

48
Ο-NOTATION AND Ω-NOTATION

◼ The definition of Θ–notation relies on proving both lower and upper asymptotic bounds
◼ Sometime we only interested in proving one bound or the other
◼ Ο-notation used to define the upper bound
◼ Ω -notation used to define the lower bound
Ο-NOTATION(BIG-OH)

◼ Ο(g(n))={f(n): there exist positive constants c and n0 such that

◼ 0 ≤ f(n) ≤ cg(n)
◼ for all n ≥ n0
◼ }
Ω-NOTATION(BIG-OMEGA)

◼ Ω(g(n))={f(n): there exist positive constants c and n0 such that

◼ 0 ≤ cg(n) ≤ f(n)
◼ for all n ≥ n0
◼ }
ASYMPTOTIC NOTATIONS

◼ Θ(g(n)) : 0 ≤ c1g(n) ≤ f(n) ≤ c2g(n)

◼ Ω(g(n)) : 0 ≤ cg(n) ≤ f(n)

◼ Ο(g(n)) : 0 ≤ f(n) ≤ cg(n)

GRAPHICALY
FAMOUS ASYMPTOTIC ORDERS

◼ Θ(1) constant time

◼ Θ(lg n)
◼ Θ(n)
◼ Θ(nlg n) best sorting algorithms
◼ Θ(n2) and Θ(n3) good only in small values
◼ Θ(2n) Exponential time
◼ Θ(n!), Θ(nn) only acceptable n<=20