Analysis and Design of algorithms

Books


Fundamentals of Computer algorithms

Horowitz, Sahani and Rajasekaran

Introduction to Algorithms
Coremen, Leiserson

The Design and Analysis of Computer Algorithms

Aho, Hopcroft and Ullman
2

ALGORITHM

A finite set of instructions which if followed accomplish a particular task. In addition every algorithm must satisfy following criteria:

1. Input: zero or more quantities externally supplied 2. Output: at least one quantity is produced 3. Definiteness: Each instruction must be clear and unambiguous. 4. Finiteness: In all cases algorithm must terminate after finite number of steps. 5. Effectiveness: each instruction must be sufficiently basic.
4

Efficiency of Algorithms Two algorithms on two systems Algorithm A1 50 n lg n Algorithm A2 2 n2

A2
Super computer 108 ins/sec

A1

P.C 106 ins /sec

For n = 106 Time taken by Super Computer = 2.(106)2/ 108 = 20,000 sec Time taken by P .C. = 50 . 106 lg 106 / 106 = 1,000 sec

Thus by using a fast algorithm , the personal computer gives results 20 times faster than the result given by super computer using a slow algorithm. Thus a good algorithm is like a sharp knife, it does exactly what it is supposed to do with a minimum amount of effort.

Pseudocode
Pseudocode is an English language like representation of the code required for an algorithm. It is partly English, partly structured code. The English part provides a relaxed syntax that is easy to read. The code part consists of an extended version of the basic algorithmic constructs-sequence, 8 selection and iteration.

Sequence, selection, loop

A sequence is a series of statements that do not alter the execution path within an algorithm. Statements such as assign and add are sequence statements. A call to another algorithm is also considered a sequence statement. Selection statements evaluate one or more alternatives. Paths are followed based on its result.
9

 The typical selection statement is the two way selection if (condition) action 1 else action 2. The part of the loop are identified by indentation. Loop iterates a block of code. It closely resembles the while loop. It is a pretest loop.

10

1 2

3 4 5 6

7 8

i= 0 Loop(all data are read) 1 i=i+1 2 read numbers into array[i] 3 sum = sum + number Average = sum / I Print (average) J=0 Loop (j < i) 1 j = j+ 1 2 dev = array[j] average 3 print (array [ j] . Dev) Return End deviation
11

Linear loop
1 2 l=1 Loop(I <= 1000)
1 2 application code I=I + 1

The body of the loop is repeated 1000 times. 1 2 I=1 Loop (I <= n) 1 Application code 2 I=I+2 For this the code time is proportional to n

Logarithm Loops
Multiply loops loops 1 I=1 2 Loop (I < n)
1 application code 2 I = l*2

Divide 1 I=n 2 loop( l>= 1)

1 application code 2 I=I/2

F(n) = [log n]

F(n) = [log n]

Nested loop- linear logarithmic

1 I=1 2 loop(I <= n)

1 2 j=1 loop( j < = n) 1 application code 2 j = j *2

3 I=I+1 F(n ) = [n log n]

Dependent Quadratic
I=1 loop ( I < = n) 1 j=1 2 loop( j < = l) 1 application code 2 j=j+1 3 I=I + 1 no of iterations in the body of the inner loop is 1 + 2 + 3 + 4 + + 10 = 55 I.e. n(n +1)/2 On an average = ( n+1/)2 thus total no of iterations = n (n+1)/2 1 2

Quadratic
1 l=1 2 Loop (I < = n)
1 j=1 2 Loop( j < = l) 1 application code 2 j = j+1 3 I = j+1 F(n) = n2

Algorithm Analysis

Complexity
Some questions to answer:  How fast can we solve a problem?  There may be many algorithms for a given problem. Which algorithm to use?  What are the classical algorithm design techniques ?  Are there problems inherently difficult to solve?
18

How do we express the complexity of algorithm? Resources : Time and Space Complexity lower bounds for problems. Complexity classes P, NP etc.

19

Analysis vs. Design

Analysis: predict the cost of an algorithm in terms of resources and performance Design: design algorithms which minimize the cost

Machine model: Generic Random Access Machine (RAM) Executes operations sequentially Set of primitive operations:
Arithmetic. Logical, Comparisons, Function calls

Simplifying assumption: all ops cost 1 unit

Eliminates dependence on the speed of our computer, otherwise impossible to verify and to compare

Review of Complexity
Most of the primary sorting algorithms run on different space and time complexity. Time Complexity is defined to be the time the computer takes to run a program (or algorithm in our case). Space complexity is defined to be the amount of memory the computer needs to run a program. (Note:-What is Running Time and it is expressed as function of the size of its input.)

L1.22

Time Complexity
Real Time:
To analyze the real time complexity of a program we need to determine two numbers for each statement in it: amount of time a single statement will take.

No. of times it is executed. Product of these two, will be the total time taken by the statement.

First no. depends upon the machine and compiler used , hence the real time complexity is machine dependent.

 To make analysis machine independent it is assumed that every instruction takes the same constant amount of time for execution. Hence the determination of time complexity of a program is the matter of summing the frequency counts of all the statements.

Frequency count

Kinds of analysis
Worst-case: (usually) T(n) = maximum time of algorithm on any input of size n. Average-case: (sometimes) T(n) = expected time of algorithm over all inputs of size n. Need assumption of statistical distribution of inputs. Best-case: (NEVER) Cheat with a slow algorithm that works fast on some input.

L1.26

INSERTION SORT
Insertion sort keeps making the left side of the array sorted until the whole array is sorted. Real life example:
An example of an insertion sort occurs in everyday life while playing cards. To sort the cards in your hand you extract a card, shift the remaining cards, and then insert the extracted card in the correct place.
L1.27

Insertion sort
INSERTION-SORT (A, n) A[1 . . n] for j 2 to n do key A[ j] i j1 while i > 0 and A[i] > key do A[i+1] A[i] i i1 A[i+1] = key i j
key sorted
L1.28

pseudocode

1
A:

Example of insertion sort

8 2 4 9 3 6

L1.29

Example of insertion sort

8 2 4 9 3 6

L1.30

Example of insertion sort

8 2 2 8 4 9 3 3 6 6 4 9

L1.31

Example of insertion sort

8 2 2 8 4 9 3 3 6 6 4 9

L1.32

Example of insertion sort

8 2 2 2 8 4 4 9 3 3 3 6 6 6 4 9

L1.33

Example of insertion sort

8 2 2 2 8 4 4 9 3 3 3 6 6 6 4 9

L1.34

Example of insertion sort

8 2 2 2 2 8 4 4 9 3 3 3 3 6 6 6 6 4 9

8 8

L1.35

Example of insertion sort

8 2 2 2 2 8 4 4 9 3 3 3 3 6 6 6 6 4 9

8 8

L1.36

Example of insertion sort

8 2 2 2 2 2 8 4 4 9 3 3 3 3 9 6 6 6 6 6 4 9

8 8 4

L1.37

Example of insertion sort

8 2 2 2 2 2 8 4 4 9 3 3 3 3 9 6 6 6 6 6 4 9

8 8 4

L1.38

Example of insertion sort

8 2 2 2 2 2 2 8 4 4 9 3 3 3 3 9 6 6 6 6 6 9 done 4 9

8 8 4

3 3

8 6

L1.39

Selection sort
How does it work:
first find the smallest in the array and exchange it with the element in the first position, then find the second smallest element and exchange it with the element in the second position, and continue in this way until the entire array is sorted. How does it sort the list in a non increasing order?

Selection sort is:

The simplest sorting techniques. a good algorithm to sort a small number of elements an incremental algorithm induction method

Selection sort is Inefficient for large lists. Incremental algorithms process the input elements one-byone and maintain the solution for the elements processed so far. L1.40

Selection Sort Algorithm

nput: An array A[1..n] of n elements. Output: A[1..n] sorted in nondecreasing order. 1. for i n 1 to n - 1 2. k n i 3. for j n i +1 to n {Find the i th smallest element.} 4. if A [ j ] < A[ k] then k n j 5. end for 6. if k { i then 7. temp nA[ i ] (interchange A[ i ] and A[ k ] ) 8. A[ i] n A[ k] 9. A[k ] n temp 10. end for
L1.41

Selection Sort in Action

Analysis of Algorithms

Algorithm analysis: quantify the performance of the algorithm, i.e., the amount of time and space taken in terms of n. T(n) is the total number of accesses made from the beginning of selection_sort until the end. selection_sort itself simply calls swap and find_min_index as i goes from 1 to n-1

n 1

T (n) ! find  min  element  swap

i !1
= n-1 + n-2 + n-3 + + 1 = n(n-1)/2

Or = (n - i) = n (n - 1) / 2 - O(n2)
L1.43

Binary search
sorted sequence : (search 9) 1 4 5 7 9 10 12 15 step 1 o step 2 o step 3 o best case: 1 step = O(1) worst case: (log2 n+1) steps = O(log n) average case: O(log n) steps

Growth Rate of an Algorithm (Order of Growth)

We often want to compare the performance of algorithms When doing so we generally want to know how they perform when the problem size (n) is large Since cost functions are complex, and may be difficult to compute, we approximate them using Asymptotic notation

Example of a Cost Function

Cost Function: tA(n) = n2 + 20n + 100
Which term dominates?

It depends on the size of n

n = 2, tA(n) = 4 + 40 + 100
The constant, 100, is the dominating term

n = 10, tA(n) = 100 + 200 + 100

20n is the dominating term

n = 100, tA(n) = 10,000 + 2,000 + 100

n2 is the dominating term

n = 1000, tA(n) = 1,000,000 + 20,000 + 100

n2 is the dominating term

Asymptotic notation Introduction

Exact counting of operations is often difficult (and tedious), even for simple algorithms Often, exact counts are not useful due to other factors, e.g. the language/machine used, or the implementation of the algorithm (different types of operations do not take the same time anyway) Asymptotic-notation is a mathematical language for evaluating the running-time (and memory usage) of algorithms.

Big O Notation
O notation approximates the cost function of an algorithm
The approximation is usually good enough, especially when considering the efficiency of algorithm as n gets very large Allows us to estimate rate of function growth

Instead of computing the entire cost function we only need to count the number of times that an algorithm executes its barometer instruction(s)
The instruction that is executed the most number of times in an algorithm (the highest order term)

The general idea is

when using Big-O notation, rather than giving a precise figure of the cost function using a specific data size n express the behaviour of the algorithm as its data size n grows very large so ignore
lower order terms and constants

O Notation Examples
All these expressions are O(n):
n, 3n, 61n + 5, 22n 5,

All these expressions are O(n2):

n2, 9 n2, 18 n2+ 4n 53,

All these expressions are O(n log n):

n(log n), 5n(log 99n), 18 + (4n 2)(log (5n + 3)),

Asymptotic Notation (cont.)

Note: Even though it is correct to say 7n - 3 is O(n3), a better statement is 7n - 3 is O(n), that is, one should make the approximation as tight as possible Simple Rule: Drop lower order terms and constant factors

7n-3 is O(n) 8n2log n + 5n2 + n is O(n2log n)

Worst/ best/average cases

Worst case is the longest running time for any input of size n O-notation represents upper bound I.e. an upper bound for worst case. Best case is time taken for some input data set that results in best possible performance.You cannot do better. This is lower bound. Average case is the average performance

The Big-Oh Notation:

given functions f(n) and g(n), we say that f(n) is O(g(n) ) if and only if there are positive constants c and n0 such that f(n) c g(n) for n n0

Example
For functions f(n) and g(n) (to the right) there are positive constants c and n0 such that: f(n) c g(n)
f(n) = 2n + 6

c g(n) ! 4n

for n n0
conclusion: 2n+6 is O(n).
n g(n) ! n

Asymptotic Notation (terminology)

Special classes of algorithms: constant O(1)

logarithmic: linear: quadratic: polynomial: exponential:

O(log n) O(n) O(n2) O(nk), k 1 O(an), a > 1

EXAMPLE Consider 1/3 n2 5n The dominating term is n2 Therefore it should be of O(n2) Given a positive constant c , a positive integer n0 to be found such that 1/3 n2 - 5 n e c n2

Asymptotic Analysis of The Running Time

Use the Big-Oh notation to express the number of primitive operations executed as a function of the input size. Comparing the asymptotic running time -an algorithm that runs in O(n) time is better than one that runs in O(n2) time -similarly, O(log n) is better than O(n) -hierarchy of functions: log n << n << n2 << n3 << 2n

Categories of algorithm efficiency

Efficiency Constant Logarithmic Linear Linear logarithmic Quadratic Polynomial Exponential Factorial Big O O(1) O(log n) O(n) O(n log n) O(n2) O(nk) O(cn) O(n!)

5-notation
For function g(n), 5(g(n)) is given by:

5(g(n)) = {f(n):  +ve constants c1, c2, and n0 such that 0 e c1g(n) e f(n) e c2g(n), n u n0 } Intuitively: Set of all functions that have the same rate of growth as g(n). g(n) is an asymptotically tight bound for f(n).

O -notation
For function g(n), O(g(n)) is given by: O(g(n)) = {f(n):  +ve constants c and n0 such that 0 e f(n) e cg(n), n u n0 }
Intuitively: Set of all functions whose rate of growth is the same as or lower than that of g(n).

g(n) is an asymptotic upper bound for f(n).

f(n) = 5(g(n)) f(n) = O(g(n)). 5(g(n)) O(g(n)).

; -notation
For function g(n), ;(g(n)) is given by: ;(g(n)) = {f(n):  +ve constants c and n0 such that 0 e cg(n) e f(n), n u n0 }

Intuitively: Set of all functions whose rate of growth is the same as or higher than that of g(n).
g(n) is an asymptotic lower bound for f(n)
f(n) = 5(g(n)) f(n) = ;(g(n)). 5(g(n)) ;(g(n)).

Relations Between 5, O, ;

Practical Complexity
250

f(n) = n f(n) = log(n) f(n) = n log(n) f(n) = n^2 f(n) = n^3 f(n) = 2^n

0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20