(> Design and Analysis of Algorithms Module - 1 Prepared By: Smruti Smaraki Sarangi Asst. Professor IMS Unison University, Dehradunig Algorithm: e An algorithm is a process to solve a problem manually in sequence with finite number of steps. e It is a set of rules that must be followed when solving a specific problem. e It is a well-defined computational procedure which takes some value or set of values as input and generates some set of value as output. e So, an algorithm is defined as a finite sequence of computational steps, that transforms to given input into the output for a given problem.e An algorithm is considered to be correct, if for every input instance, it generates the correct output and gets terminated. e So a correct algorithm solves a given computational problem and gives the desired output. e The main objectives of algorithm is = To solve a problem manually in sequence with finite no. of steps = For designing an algorithm, we need to construct an efficient solution for a problem.‘Why we read Algorithm: e We read the algorithm. = For testing the good programmer in 2 levels. That is: i. Macro-level (Implementation of algorithm like s/w) ii. Micro-level (Algorithm part like h/w) e For designing an algorithm, we have to learn to solve computational problem in economics. ¢ By applying mathematical algebra, we get computational logic.Algorithm Paradigm: It includes 4 steps. That is: = Design of algorithm = Algorithm validation * Analysis of algorithms = Algorithm testing 1. Design of algorithm: = Various designing techniques are available which yield good and useful algorithm. = These techniques are not only applicable to only computer science, but also to other areas, such as operation research and electrical engineering.= The techniques are: divide and conquer, incremental approach, dynamic programming etc. By studying this we can formulate good algorithm. 2. Algorithm validation: = Algorithm validation checks the algorithm result for all legal set of input. = After designing, it is necessary to check the algorithm, whether it computes the correct and desired result or not for all possible legal set of input. = Here the algorithm is not converted into the program. But after showing the validity of the method, a program is written. This is known as “program providing” or “program verification”. = Here we check the program output for all possible set of input.= It requires that, each statement should be precisely defined and all basic operations can be correctly provided. 3. Analysis of algorithms: = The analysis of algorithm focuses on time complexity or space complexity. i ‘A ue ee | = The amount of memory needed by program to run to completion is referred to as space complexity. = The amount of time needed by an algorithm to run to completion is referred to as time complexity. = For an algorithm time complexity depends upon the size of the input, thus is a function of input size ‘n’” = Usually, we deal with the best case time, average case time and worst case time for an algorithm.a = The minimum amount of time that an algorithm | requires for an input size ‘n’, is referred to as Best ve) Case Time Complexity. i = Average Case Time Complexity is the execution of an [al algorithm having typical input data of size ‘n’ = The maximum amount of time needed by an algorithm for an input size ‘n’ is referred to as Worst Case Time Complexity. 4. Algorithm testing: = This phase involves testing of a program. It consists of two phases. That is: Debugging and Performance Measurement. = Debugging is the process of finding and correcting the cause at variance with the desired and observed behaviors.= Debugging can only point to the presence of errors, but not their absence. = The Performance Measurement or Profiling precise by described the correct program execution for all possible data sets and it takes time and space to compute results. NOTES: e While designing and analyzing an algorithm, two fundamental issue to be considered. That is: |. Correctness of the algorithm 2. Efficiency of the algorithm e While designing the algorithm, it should be clear, simple and should be unambiguous. e The characteristics of algorithm is: finiteness, definiteness, efficiency, input and output.A ; . ( | Analysis of Algorithms: Woy S| e Analysis of algorithms depend upon various factors, such \ as memory, communication bandwidth or computer hardware. But the most often used is the computational time that an algorithm requires for completing the given task. ead _¢ As algorithms are machine and language independent these | are the only important, durable and original parts of computer science. Thus, we will do all our design and implementation for the RAM model of computation. e In RAM model, all instructions are executed sequentially one after another with concurrent operations. In performing simple operations like addition, subtraction, assignment etc. model takes | step.A call to a subroutine and loops are not single step operation. Instead each memory access takes exactly one step. By counting the number of steps the running time of an algorithm is reassured. The analysis of an algorithm focuses on the time and space complexity. The space complexity refers to the amount of memory required by an algorithm to run completion. Time complexity is a function of input size ‘n’. It is referred to as the amount of time required by an algorithm to run to completion. Perhaps different time can arise for the same algorithm, we usually refer best case, average case, worst case complexity.___» Worst Case __ Average Case No. of I _____/ Best Case Steps — — — N 1 2 1. Worst-Case Time Complexity: = The worst-case time complexity is a function defined by the maximum amount of time needed by an algorithm for an input size ‘n’. Thus, it is the function defined by the maximum no. of steps taken on any instance of size ‘n’ = A worst case estimate is normally computed, because it provides an upper bound for all inputs including particularly the bad ones.2. Average-Case Time Complexity: = The average case time complexity is the execution of an algorithm having typical input data of size ‘n’, thus if the function by the average no. of steps taken on any instance of size ‘n’ = Average-case analysis does not provide the upper- bound and it is difficult to compute. 3. Best-Case Time Complexity: = The best-case time complexity is the maximum amount of time that an algorithm requires for an input of size ‘n’, thus it is the function defined by the minimum no. of steps taken on any instance of size nm. = All this time complexities define a numerical function time ~ size.(| Calculation of Running Time: e There are several ways to estimate the running time of a program. If 2 programs are expected to take similar times, probably the best way to decide which is faster is to code them both up and run them. e Generally there are several algorithmic ideas and we would like to estimate the bad ones early. e So, an analysis is usually required furthermore the ability to do an analysis usually provides insight into designing efficient algorithm. e The analysis also generally pin points, which are worth coding carefully./ \ al ¢ To simplify the analysis, we will adopt the conversion that ( | there are no particular units of time. Thus, we throw away Ned low ordered terms. _ __¢ So, what we are essentially doing is computing a big oh (O) running time. Since big oh (O) is an upper bound, we must be careful never to underestimate the running time of the program. e In effect, the answer provided is a guarantee that the program will terminate within a certain time period. The program may stop earlier than this, but never later.(| Analyzing the control _ Structure: — e Sequencing: | = Let ‘P,’ and ‘P,’ be two fragments of an algorithm they | may be single instructions or complicated sub- algorithms. Let ‘t,’ and ‘t,’ be the times taken by ‘P,’ and ‘P,’ respectively. These times may depend on various parameters such as the instance size. = The sequencing root says that the time required computing P,, P;. i.e.: 1 P, then P, is simply t, + t, by maximum rule, the time will be O(max(t,, t,)). = E.g.: i) t; = O(n), t, = 6(n’). So, the computational time is: t, = O(n’). ii) if t, = O(n), t, = O(n?) => t, = O(n’). iii) if t, = O(n), t, = O(n’) => t, = O(n’).(| Analyzing the control _ Structure: — e Sequencing: | = Let ‘P,’ and ‘P,’ be two fragments of an algorithm they | may be single instructions or complicated sub- algorithms. Let ‘t,’ and ‘t,’ be the times taken by ‘P,’ and ‘P,’ respectively. These times may depend on various parameters such as the instance size. = The sequencing root says that the time required computing P,, P;. i.e.: 1 P, then P, is simply t, + t, by maximum rule, the time will be O(max(t,, t,)). = E.g.: i) t; = O(n), t, = 6(n’). So, the computational time is: t, = O(n’). ii) if t, = O(n), t, = O(n?) => t, = O(n’). iii) if t, = O(n), t, = O(n’) => t, = O(n’).= Add the time individual statements. The maximum is the one that count. { K | e If then else: Ry = Again consider P, and P, be the parts of an algorithm, with computation time t, and t, respectively. Now ‘P,’ is compared only when the given condition is true. Otherwise for the false condition ‘P,’ is computed. Thus, the total time is according to the conditional rule ‘if then else’. = According to the maximum rule this computation time is: max(t,, t,). E.g.: i) Suppose P, = t, = O(n), P; = t, = O(n?) => T(n) = O(n”). ii) t, = O(n’), t, = O(n?) => T(n) = O(n?) or O(n?) => O(n?)f _ ¢ For Loop: \ | = It is to be noted that P(i) is computed for each iteration { A_| from i — 1 to m. If the value of ‘m’ is zero, then we are M4 considering that ‘m’ does not generate any error instead — the loop is terminated without doing anything. E.g.: fori — 1 tom { PQ) } = If P(i) takes any constant time ‘t’, for its computation then for ‘m’ iterations, the total time for the loop is simply ‘mt’. Here, we are not considering the loop control. As we know that for loop can be expressed as:while(i < m) ( P@) isitl } = The test condition, the assignment instruction and sequencing operation (goto: implicit in while loop) are considered at unit cost for simplicity. Suppose if all these operations are bounded by ‘c’ then the computation time for the loop is bundled above by: T(n) T= 246E.g.: fori — 1 tom { sum <— sum + t[i] } Total time = Y7Ey t; = Liby OC) = BCL, 1) = (m) = If the algorithm consists of nested for loop, then the total time is: for i< 1 tom { forj —1 tom { PG) } mm =>Tay= 2 Lt i=lj=lSh ¢ While Loop: / i | = The while loops are difficult to analyze in comparison to Vent for loops, as in these there is no obvious method which | determines how many times we shall have to repeat the | loops. = The simple technique for analyzing the loops is to firstly determine functions of variables involve whose value decreases each time. Secondly for determining the loop it is necessary that this value must be a positive integer. = By keeping the track of how many times the values of function decreases, one can obtain the no. of repetition of the loop. The other approach for analyzing while loops is to treat them. E.g.: while(m > 0) {m<-m-l} => Time T(n) = 6(m)= The rules are: 1. Sequencing: Add the time of the individual statements. The maximum is the one that count. Alternative Structures: Time for testing the condition + the maximum time taken by any of the alternative paths. Loops: Execution time of a loop is at most the execution time of the statements of the body (including the condition tests). Nested Loops: Analyze them as inside out. Sub-programs: Analyze them as separate algorithms and substitute the time whenever necessary. Recursive Sub-programs: Generally the running time can be expressed as a recurrence relation, with solution growth rate of execution time.ee «| Z | Asymptotic Notation: e The notation, which we use to describe the asymptotic running time of an algorithm are defined in terms of functions, whose domains are the set of natural numbers and real numbers. = The natural number set is denoted as: N = {0, 1, 2, ...} = The positive integer set is denoted as: N* = {1, 2, 3, ...} = Real number set is denoted as R. = Positive real number set is denoted as R*. = Non-negative real number set is denoted as R*. Such notations are convenient for describing the worst case running time function T(n), which is usually defined only on integer input sizes.e The different types of notations are: \ > ® Big oh (O) notation { p> = Small oh (0) notation eed = Theta (6) notation = Omega (Q) notation = Small omega () notation 1. Big Oh (O) Notation: = The upper bound for the function is provided by Big Oh (O) notation. We can say, the running time of an algorithm is O(g(n)), if whenever input size is equal to or exceeds, some threshold ‘no’, its running time can be bounded by some positive constant ‘c’ time g(n).= Let f(n) and g(n) are two functions from set of natural numbers to set of non-negative real numbers and f(n) is said to be O(g(n)). = That is: f(n) = O(g(n)), iff there exist a natural number ‘no’ and a positive constant c > 0, such that f(n) < c(g(n)), for all n> np. c(g(n)) Running f(n) time Ny Input sizeExamples: 1. f(n) = 2n? + 7n - 10, n =5,c =3. => f(n) = O(g(n)), where g(n) = n? f(n) < c(g(n)) => 2n? + 7In— 10 <3 x n? =>2x254+7x5-10<3x25 => 50+ 35-10<75 => 75< 75. So, itis in O(g(n)) = O(n’). 2. f(n) = 2n? + 7n- 10, n = 4, ¢ = 3, g(n) =n? => f(n) < c(g(n)) => 2n? + 7n—- 10 <3 xr? =>2x 16+7x4-10<3x 16 => 32+ 28-10 <48 => 50 < 48. So, it is not in O(g(n)).3. f(n) = 2n? + 7n — 10, n = 6, c = 3, g(n) =n? => f(n) < c(g(n)) => 2n? + 7In- 10 <3 x n? => 2x 36+7x6-10<3x 36 => 72 + 42-10 < 108 => 104 < 108. So, it is in O(g(n)) = O(n’). 2. Small Oh (0) Notation: = The functions in small oh (0) notation are the smaller function in Big oh (O) notation. We use small oh (0) notation to denote an upper bound that is not asymptotically tight. = This notation defined as: f(n) = o(g(n)), iff there exist any positive constant c > 0 and nO > 0, such that f(n) < c(g(n)), for all n> no. = The definition of O-notation and o-notation are similar.{ | = The main difference is that in f(n) = O(g(n)), the bound f(n) < c(g(n)) holds for some constant c > 0, but in f(n) = ol o(g(n)), the bound f(n) < c(g(n)) hold for all constant aw c>0. = In this notation, the function f(n) becomes insignificant relative to g(n) as ‘n’ approaches infinity. That is: lim £@ = 0. 3. Big omega (Q) Notation: = The lower bound for the function is provided by Big Omega () Notation. = We can say, the running time of an algorithm Q(g(n)), if whenever input size is equal to or exceeds some thresholds value ‘ny’, its running time can be denoted by some positive constant ‘c’ times g(n).= Let f(n) and g(n) are 2 functions from set of natural numbers to set of non-negative real numbers and f(n) said to be Q(g(n)). = That is, f(n) = Q(g(n)) iff there exist a natural number ‘no’ and a constant c > 0, such that f(n) > c(g(n)), for all n= Ny. fi Running - time c(g(n)) No Input size(A, powee i Lf) =n? +3nt+4,n=1,c=1. ati. | => f(n) = Q(g(n)), where g(n) = n2 os f(n) > e(g(n)) => n? + 3n + 4 > cn? ed =>1l+3x1l+421xl=>821 => f(n) = Q(g(n)) = Q(n’). (proved) 4. Small omega (@) Notation: = For a given function g(n), we denoted it as w(g(n)), where ‘q’ notation are the larger functions of Big omega (Q) notation. = We use a notation to denote a lower bound that is not asymptotically tight. = We define this notation as: f(n) = w(g(n)), there exist some positive constant c > 0 and ng > 0, such that f(n) > c(g(n)), for all n> no. f@) _ = The relation f(n) = w(g(n)) implies that Jim ow am =| 5. Theta (0) Notation: \ > = For a given function g(n), we denoted it as 6(g(n)), the { Pd set of functions f(n) = O(g(n)), if there exist some Rw constant ¢;, C) and ng, such that: c,(g(n)) < f(n) < c,(g(n)), for all n> no. = For all values of ‘n’ to the right of ‘ng’, the values of f(n) lies at or above c,(g(n)) and at or below c,(g(n)). = In other words, for all n > no, the function f(n) is equal to g(n) within a constant factor. = We say that g(n) is an asymptotically tight bound for f(n).= The definition of 0(g(n)) requires that every member f(n) € O(g(n)) be asymptotically, non-negative. That is: f(n) be non-negative whenever ‘n’ is sufficiently large. = So f(n) = 0(g(n)), which implies that: f(r) lim —— = constant c. noo g(n) c(g(n)) f(n) c,(g(m)) Running time No Input sizeAsymptotic Notation Properties: 1. Reflexivity: i. f(n) = 6(f()) ii, fn) = Of) iii. f(n) = Q(f(n)) 2. Symmetric: i, £(m) = 6(g(n)) iff g(m) = 84M) 3. Reflexivity: i f(m) = O(g(n)) iff g(n) = Q(E(n)) ii, f(n) = o(g(n)) iff g(n) = w(fn))Vn NG 4. Transitivity: vw i; ii. iii. iv. v. f(n) = 0(g(n)) and g(n) = O(h(n)) => f(n) = O(h(n)) f(n) = O(g(n)) and g(n) = O(h(n)) => fn) = O(h(n)) f(n) = Q(F(n) and g(n) = Q(h(n)) => fn) = Q(a(n)) f(n) = o(g(n)) and g(n) = o(h(n)) => f(n) = o(h(n)) f(n) = w(f(n)) and g(n) = w(h(n)) => f(n) = w(h(n)) Some Important Formula: i. iti. For any two functions f(n) and g(n), we have: f(n) = 0(g(n) iff f(n) = O(g(n)) and fn) = Q(g(n)) Tf f(n) = O(g(n)) and f(n) = Q(g(n)) => fn) = O(g(n)) If T,(n) = O(f(n)) and T, = O(g(n)), then a. T,(n) + T,(n) = O[max (f(n), g(n))] b. T\@) * T,(m) = offi) * g(n)] iv. f(n) and g(n) are two asymptotic non-negative functions, then max(f(n), g(n)) = @(f(n) + g(n)).6. Some Important Formula: i. Ign = log,n (binary) ii, Inn = log.n (natural) iii. 1g*n = (1gn)* (exponential) iv. Iglgn =1g(gn) (composition) v. a =blog,a vi. log,(ab) = log.a + log.b vii. log,a"= nlog,a vill. log.a/b = log. a/log.b ix. log,(I/a) = -log,a x. log,a = I/log,b xi. alog,c =clog,a xii. logn < nlogn O. So,n!=1*2*3* ....*n. n! = 2an(n/e)(1 + ~~ O(n) [Stirling’s Approximation] n! = O(n") n! = @(2") 1g(n!) = 0(mlgn) Ig*n = min{i > 0; Ig(i) n< 1} Ig*2 =1 Ig*4 =2 1g*16=3 1g*65536 = 4 Ig*(295536) = 5Problems: | 1. Show that for any real constant ‘a’ and ‘b’, where b > 0. Ans: (n +a)’ = @(n), Here f(n) = (n + a)? and g(n) = n?. . if lim £® = We know that, f(n) = 0(g(n)), if Jim, a(n) constant. ce qe, (Nta)e =>(n+a)>=0(n»), if Jim, fate? = constant. {n(1 + a/n)}* => lim {nb + a/n)}> n> => lim n> => lim (1 + a/n) => 1' = 1 = constant. 00 => (n + a)’ = O(n) (proved)2. Prove that 2"*+1= O(2") Ans: Here f(n) = 2"*! and g(n) = 2°. ~ fm) _ y, 2att Dn 2 Jinn Soo = HR, = Lim => f(n) = O(g(n)). We know that, f(n) = 0(g(n)) there exist f(n) < c(g(n)), for all n= np. According to O-notation, f(n) = O(g(n)) iff there exist positive constant ‘c’ and ‘nO’ and either f(n) < c(g(n)) and for all n > no. => 2"*+!= O(2") (proved) 3. Prove that 27" = @(2") Ans: Here f(n) = 27" and g(n) = 2". im 2 = jim 2 = tim 22 dim oop ~ A oe = Jim Ge => 2?" = «(2"), which is a w-notation. (Proved) = 2=constant = 0 => f(n) = w(g(n))a 4. Show that 5n?= o(n3) Ans: Here f(n) = 5n? and g(n) = n°. ffm) Sn? 5 te 3 jim om Jim 4 = lims = 0. So, it is in small oh notation. So => 5n? = o(n*) (proved). 5. Show that 2" = o(n?) Ans: Here f(n) = 2" and g(n) = n?. n30g(N) ~— n-00 o(n’) is proved. So, it is small oh notation. 6. Show that n?/2 = o(n) Ans: Here f(n) = n?/2 and g(n) =n. £m) n/2 rn 4 . . . . n lim —~ = lim —= lim > *-= lim >=0 noo g(n) nso noo 2 nN noo2 So, f(n) = w(g(n)) (proved).7. Theorem: Let f(n) = ay + ayn + ajn? +... + a,n™, then prove that f(n) = 0”). Ans: Here f(n) = ay + ayn + a,n? +... + a,n™ and g(n) = n™. fon) _ According to 8-notation, Jim, aa) => f(n) = 6(g(n)) ay tan +ajn? +... +annm => lim noo n™ . n(a/n™+ an/n™ an?/n™ ... +an) => lim —?—_1_ _ + ns nl => a, = constant c => f(n) = 0(n™) (proved)MM» 8. Let f(n) = 7n3 + 5n? + 4n + 2. Prove f(n) = 0(n?). Ans: Here f(n) = 7n? + 5n? + 4n + 2 and g(n) =n’. According to 8-notation, lim on = constant n>0 Tn} + 5n? + 4n +2 nm => Jim (7+ 5/n+ 4/n? + 2/n3) = 7 = constant => f(n) = 0(n*) (proved) 9. Prove lg(n!) = 0(nlgn). Ans: n">n! => f(n) = 0(g(n)) => jim, => Ign" > Ign! => nlgn > Ign! => Ign! Ign! < 1 —nlgn => Ign! = O(nlgn) --- (i) Where c, =1. Now to show that Ign! = Q(nlgn). That is there exist some constant ‘c’ and ‘ny’, such that 0 < cnlgn < Ign!=> Ign n° Ign! = Q(nlgn) --- (ii) From equation (i) and (ii), lgn! = 0(nlgn) (proved). 10. Prove n! = O(n"). ! Ans: We have to show that, Jim, = =0 n! = V2an(n/e)(1 + O(1/n)) (According to Stirling’s Approximation) > lim Veminl + ocl/n)) n> Let g(n) = 1/n, f(n) = 0(g(n)) => f(n) < ¢,(g(n)) = ¢).1/n => f(n) = O(1/n) < c/n But, 6(1/n) < c).1/n.| => lim me lim Y2an(1 + oy/n) c/n) no 1" ~ n-s00 e => li noo e => V2 lim (Nn/en + lim ¢,/Vn.en) n> n> _ V2n(vn/en + ¢,/vn.en) Mm . I ot => \2n Jim Gay q + Nine® — | (since lim G = 9) - 1 => V2n lim Gag—7 +) _ => V2n/2 lim or -1l=vn/2*0=0. ! Here, lim = = 0 =>n! =o0(n") proved. noRecurrence: e Arecurrence is an equation or inequality that describes a function in terms of its value on small inputs. e The running time of the recursive algorithm can be obtained by a recurrence. ¢ To solve the recurrence relation means to obtain a function defined on the natural numbers that satisfies the recurrence.as Recurrence Relation: e Arecurrence relation (RR) is defined as for a sequence {a,} is an equation that expresses ‘a,’ in terms of one or more previous elements ap, aj.. a,., of sequence for all n > no without any base cases. e E.g.: i) for instance, consider a recurrence relation t, = 2t, — 1. if ‘c’ is a constant, then any function of the form c2" is a solution to the above recurrence relation. ¢ By considering, mathematical induction, we have induction part, if t,—1=c2"- 1. _ ¢ If we have the initial condition t) = 5, then the only choice for the constant ‘c’ is to have value 5, so as to give the correct initial value. Thus on the basis part of the proof, we have t, = 5.2".It does not matter in which order the basis and induction are established, what matters is that both have been verified to be correct. Hence the solution for the recurrence is t, = 5.2". e E.g.: ii) the worst-case running time T(n) of the MERGE- SORT procedure could be described by the recurrence. Tin) =J 0(1), ifn=1 2T(n/2) + O(n), if n> 1. Whose solution was claimed to be T(n) = 0(nlgn)Recurrence Equation Method: e We solve the recurrence equation by the following methods. That is: i. Substitution method. ii. Iterative method ill. Master method iv. Recurrence or Recursion Tree 1. Substitution Method: =In this method, first we guess a solution and use mathematical induction to find the constant and show that the solution works.= The substitution method can be used to establish either =a] upper or lower bounds on a recurrence. < E.g.: T(n) = 2T( Lyy/24 ) + n --- (1) is a recurrence \ ] relation. We guess the solution T(n) = O(nlgn), where 7 g(n) = nlgn is the solution. That is: f(n) = T(n) < enlgn --- (2). Then we have to prove that, this solution is true, by using mathematical induction. From equation-1, T(n) = 2T( Ly/2d jy+n => T(n) < 2c( bn/24) Ig¢ bn/24) +n => T(n) < 2c. n/2 Ign/2 + n = cn(Ign — 1g2) +n => cnign—cn+n. So, T(n) < cnlgn — n(c — 1) => T(n) < cnlgn for c >1 = Now, using mathematical induction, Tn) = T(n-1) + 1.= Forn=1, if T(1) = 1, then 1 1gl =0 => T(1) is not true for n= 1. But let T(1) is true. For n = 2, if T(2) = 2T(1) + 2 =4, then 4c>2. Hence, T(n) < cnlgn, c > | is true for n = 2 => T(2) is true. Similarly, T(3), T(4) ... is true => T(k) is true. => T(k + 1) is true by using Tk. => Tin) =T(n- 1) + 1, g(a) =n = Of) Tin) =T(n- 1) +1 =T(n-2)+141 =T(int+3)+14+14+1 Let k= T(n—k) + k. Letn—k =0 (base value). =>n=k=>Ty+n=n+1= Let T, = | (base case) Td) =1,n-k=1=>k=n-1l=>Tw@)=14+n-l=n = So, it is true for Ln/24 and it is true for n. If it is true for n, then it is true for 3. If it is true for 3, then it is true for 6 and 7 and if it is true for 6, then it is true for 12 and 13. = Hence, we conclude that for n >c, where c > 2. So, T(n) = O(nign) is a solution of T(n) = 2T( 'n/24) +n. = Substitution Method is of two types. That is : i. Backward Substitution ii. Forward Substitutioni. Backward Substitution Method: = Question: T(n + 1) = 2T(n). Solve this recurrence relation using substitution method, using backward substitution. Ans: T(n + 1) = 2T(n), let base value T(0) = 1 => T(n + 1) = 2(2T(™m — 1)) = 22T(n— 1) (1* term) = 2?(2T(n — 3)) = 2°T(n — 2) (2™ term) For k" term = 2** !T(n—k) Letn-k=0=>k=n. So for k"" term, it is 2°+!T(Q) = 2"+!.1 = 29+! => T(n + 1) = 2"*!=> T(n) = 2" (Ans)Se "Now to prove this backward substitution using \ | mathematical induction is: T(n) = 2" We) Let for n = 1 is true => T(1) = 2! =2 i Let for n=n is true => T(n) = 2" 7s We have to prove, T(n + 1) = 2"*! => T(n¢ 1) =2.T(n) =2.2" = 29+! => T(n + 1) = 2"*! (proved) ii. Forward Substitution Method: = Question: T(n + 1) = 2T(n). Solve this recurrence relation using forward substitution method. = Ans: T(n + 1) = 2T(n), => T(1) =2T(0)=2 (n=0 and T(0) = 1) => T(2) = 2T(1) =2 x2 =2? (n= 1 and T(1) = 2) => T(3) = 2T(2) = 23= For k, T(k) = 2* (n =k and T(k — 1) = 2*-! Putting n =k => T(n) = 2" => T(n+ 1) =2"*! = To prove this forward substitution method, we have to solve by the mathematical induction process. = That is: T(n) = 2". Let n= | is true => T(1) = 2! =2 Let n =n is true => T(n) = 2" ® We have to prove, T(n + 1) = 2"+! => T(n + 1) =2.T(n) = 2. 2" = 2°*! [T) = 2°] => T(n +1) = 2"*! (proved)Problems on Substitution Method: 1. T(n) = 2T(n —- 1) + 1, initial value/base value T(0) = 1 using forward substitution. e Ans: T(n) = 2T(n — 1) + 1, where T(0) = 1 =>T(1)=2T(0)+1=2x1+1=3=22-1 T(2) =2T(/) +1=2x341=7=2-1 T(3) = 2T2)+1=2x7+1=15=2'-1 T(k) = 2k+1-1 => T(n) =2"+!-1 =>T(n+ 1)=2"*?-1=2Tm) +1[ e Now, by using mathematical induction, T(n) =2"+!- 1. Letn=1 => T(1) =2!*!-1. | ¢ Similarly, let ‘n’ is true => T(n) = 2"*!- 1 ew We have to prove, T(n + 1) = 2T(n) + 1 = 2(2"*!—-1) +1 => 2.204124] =20+2-] => T(n+ 1) = 2"*?— 1 (proved) 2. Consider the recurrence T(n) = 3T(n/2) + n, n > 1 with initial condition T(0) = 0, obtain the solution for the above recurrence. e Ans: T(n) = 3T(n/2) + n, n> land T(0) =0 Let forn = 1, TU) = 3T(1/2) + 1 =1 n=2, T(2)=3T(1)+2=3x1+2=5 n=2?=4, T(4) = 3T(2) +3 =333 x 1+2)+2? =3?x14+3x24+27=94+6+4=19fO> n=23 = 8, T(8)=65=33 x 1+3°x24+3x2?7+23 n=2+= 16, T(8) = 211 =34x14+35x24+3?x2?4+3x 23424 (We take the values represented in the powers of 2). ¢ From the above computation, we guess the solution is: T(n) = 3"! 2041 So, for n = 0, T(O) = 3° — 2° = 0, is satisfy the initial condition. So the statement is correct6 for n = 1. Let us assume for some k > 0 then: T(2k) = 329 + 3-12! 4 3K-222 4 + 3x D1 4 30x 2k = DK (2/3)! = 3k+!_ ake! ¢ So it can be observed that, T(k) = 3k+!— 2+! (if ‘n’ is not power of 2). By this, we guess as the solution is correct. Thus, T(n) = 38*!— 20+!3. Consider the recurrence T(n) = 1, n = 1 and 2T( Ly/24 ), n > 1. We have to find an asymptotic bound on T(n). © T(n)=1,n=1 and 2T(bn/24),n>1 e For the above recurrence, we guess that it satisfies O(nlgn). Thus, for this we have to show that there exists a constant ‘c’, such that T(m T(n) <2 bLn/24 log Ln2t+n => T(n) < c2logn — cnlog2 + n => T(n) < cnlogn — (clog2 — 1)" => T(n) < cnlogn, for all c > I/log2 e By mathematical induction, we can check T(2) = 4 and T3)=5S./ \ A | — | | e As for n= 1, T(n) < cnlogn yields 0. Thus, the inductive proof T(n) < cnlogn for constant c > 1 is completed by choosing ‘c’ large enough such that: T(2) < c2log2 and T(2) < c3log3. Since, the above relation holds for c > 2. Thus, T(n) < cnlogn holds true. e Thus, our guess as the solution T(n) = O(nlogn) is correct. 4. Consider the recurrence T(n) = 2T( La/2t+ 16) + n. We have to show that it is asymptotically bound on O(nlogn). © T(n)= 27( bn/24 + 16) +n. For, T(n) = O(nlogn), we have to show that for some constant, Tn) < cnlogn => T(n) < c2( bn/24 + 16) log( 'n/24+ 16) +n = cnlog(n/2) + 32 + n—cnlog2 +n =cnlogn—cn+32+n| j = cnlogn — (c — 1)n + 32 a’ = cnlogn — b < cnlogn (if c > 1, b is constant) Thus, T(n) = O(nlogn) ~~ _| 5. Consider the recurrence T(n) = 2T( Ly/2d )+n, | we have to show that it is asymptotically bound on O(nlogn). © Tin) = 2T( Ln/24 ) + n. For, T(n) = Q(inlogn), we have to show that for some constant ‘c’, T(n) = cnlogn => T(n) <¢2( bn/24 jlog( Ln/24) +n => T(n) = cnlog(n/2) + n — cnlog2 + n = cnlogn — cnlog2 +n =cnlogn—cn+n => T(n) =cnlogn for c = 1. Thus T(n) = Q(nlogn)La 6. Consider the recurrence T(n) = 2T(‘n/2 ) +1, we have to show that it is asymptotically bound \ on O(logn). © T(n) = 2T( Lat ) + 1. For, T(n) = O(logn), we have to show that for some constant ‘c’ T(n) < cnlogn => T(n) < clog( tn/24) + 1 = clogn — clog? + 1 => T(n) 1. Thus T(n) = O(logn) 2. Iterative Method: = An iterative method is the method, where the recurrence relation is solved by considering 3 steps. That is: I. Step 1: expand the recurrence Il. Step 2: express is as a summation (>)) of terms, dependent only on ‘n’ and the initial condition. III. Step 3: evaluate the summation (>). RasProblems on Iteration Method: 1. Tm) =0,n = 0 - (i) (initial condition) and =c+T(n—1),n>0- (ii) =>T(n)=T(n-1)+c =T(n- 2)+c+c=>T(n— 2) +2c =T(n-3)+c+c+¢c=>T(n—3) + 3c =Tan-4)+ce+c+c0+c=>T(n—-4) +4c For k term, T(n—k) +c +c¢+C... k times =Tin-k) +d, From the base case, n-k = 0 =>n=k => T(n) = TO) + Dh, c =0+cn=>T(n) =cn.2. Tin) =0,n=0- (i) (initial condition) and =T(n-1) +n, n>0-(ii) =>T(n)=T(n-1)+n Tnn—1)=T(n—2)+n-1 => T(n)=T(n-2)+n-14+n T(n-2)=Tin-3)+n-2 => T(n) =T(n—-3)+n-—2+n-Il+n For kt term, T(n—k) +n-k-1+....+n-L+n =>T(n)=Tin-k) + Ein -i From the base case, n -k = 0 =>n=k =>T(n)=T0)+ Degn —i=04+ Tein -i =>T(n=Tegn -iWe know that, Y7_, i =n (n+ 1) /2 Tin) = EM in -i =ntn—-1lt+n—-24...414+04+¢1 => T(n) = n(n + 1)/2-1 3. Tm) =c, n= | —(i) (initial condition) and = 2T(n/2) +c, n> 1 - (ii) => T(n) = 2T(n/2) + = 2T(2T(n/2?) + c) +c = 2?T(n/22) + 2c +€ = 2? (2T(n/23) +c) + 2c + = 23T(n/23) + 27c + 2!c + 2% For k® term, 2'T(n/2*) + 24- le +... + 2!e + 2% => n/2k=1 => n= 2k=>k=log,nThat is, 2logn + 2logn - I/e +..... + 2!c + 2% = 2log,n = nlog,2 =n! = n (since, alog,c = clog,a) =2'T(n/2") +¢ DE, 2! -1 (since, a+ ar+ar?+... + ar™=a(rm*!— 1)/r— 1) = 2T(n/24) + (2k = 1/2- 1) = 2T(n/24) + c(2*— 1) Let k = logn, = 2lognT(n/2logn) +c2'2"—c =nT(1)+cen-—c =2cn-—c¢ =c(2n- 1)4. T(n) = 2T(bn/34)4n Expanding the above terms, we get T(n) =n + 2/3n + 4T(n/9) =n + 2/3n + 4/9n + 8T(n/27) It is to be noticed that, we can meet the boundary condition, where (n/3') < 1. ie.: after performing rlog,nq expansions. Thus, Tin) = ny,LYE" (2/3)'n + 2logsnO(1) T(n) = (T(n- 3) + 1) +2=T(n—3) +3 For kth term, T(n) = T(n—- k) +k When k=n-1 => T(n—k) =T(1) = 0(1) Thus, T(n) = 0(1) + (n— 1) = O(n). Hence, T(n) = @(n) 6. Ti) =T(/2) +n, T(1) = 0(1) Expanding the above terms T(n/2) = T(n/4) + n/2 Thus, T(n) = T(n/4) + n/2 +n T(n/4) = T(n/8) + n/4 => T(n) = T(n/8) + n/44+n/2+nT(n/8) = T(n/16) + n/8 => T(n) = T(n/16) + 0/8 + 0/4 + n/2 +n T(n/16) = T(n/32) + n/16 => T(n) = T(n/32) + 0/16 + 1/8 + 1/4 + 1/2 + For kth term, T(n) = T(n/25) + Df) It can be observed that the recursion stops, when we get to T(1). This happens when n/2* = 1, That is n = 2‘ => k = logn Thus, T(n) = 0(1) + Be 'g 4) <0) + LoS) < 0(1) + 2n = 6(n) Hence, T(n) = 6(n)7. T(n) =3T('n/44 +n Expanding the above terms, we get Tan) = 3T( Ln/16 )) + (4n/44) +n = 33T( Ln/t644 ))) + (bn/164) + (bn/4t) +n =n43(bn/44) + 9¢ Lote) + 2TT( bn/1644 yy The recursion stops, when n/4i < 1, which implies n < 4! => log,n =i Thus, T(n) =n + 3 bLn/44 +... + 3' bn/4id + 3log,n.6(1) T(n) < (n+ 3n/4 + 9n/16 + ... + 3log,n)0(1) < ny 0(3/4)*+ O(nlog,3) {as 3log,n = nlog,3} 1,b>1. =In the above recurrence, the problem of size ‘n’ is divided into ‘a’ sub-problems each of size ‘n/b’. = Each sub-problem of size ‘n/b’ can be solved recursively in time T(n/b). = The cost of dividing or splitting the problem and combine the solutions or result is described by the function f(n). = Here the size is interpreted as ‘n/p or pn/bq . The Tin) can be bounded asymptotically by the following 3 cases.1. CASE |: if f(n) = O(n'’2,4-*) for some constant € > 0 then T(n) = 0(n!’s,") 2. CASE II: if f(n) = 0(n'’s,*) then T(n) = 0(n'°s,*. logn) 3. CASE II: if f(n) = Q(n'’s,**°) for some constant € > 0 if af(n/b) < cf(n), for some constant c > 0 and c < 1 and n= sufficiently very large, then T(n) = 0(f(n)) NOTES: ¢ Ifthe recurrence is of following form i.e.: T(n) = aT(n/b) + cn4, n> no e Then the solution of the recurrence is, Tin) =| O(n), if a< (b)4 6(n“logn), if a = (b)¢ O(n'°2,2), if a> (b)4e E.g.: T(n) = 3T(n") + log,n Let us assume m = log,n => n = 3". Thus, n!? = 393 => T(3™) = 3T(3™) + m. Again, consider s(m) = T(3™). We have s(m) = 3s(m/3) + m. Using master method, s(m) € 0(mlogm) => T(n) € O(log,n(log log,n)) (Ans) Let T(n) = 4T(n/2) + nlogn. If f(n) = logarithmic part or polynomial part, then master method did not work or can’t be apply. So for the solution we have to apply substitution or iterative method.Here a = 4, b = 2, f(n) = nlogn, n!°s,* = n*. Here we can’t compare n? with nlogn. So we can’t say that those two terms are equal or greater than or less than. So we go for iterative or substitution method. Tn) = 2T(3n/2) + 3. Here a = 2, b = 2/3, f(n) = 3. Now nlog,@ = nlos,,2, It can’t be solved in master method, because logarithmic function can’t take fractional value. Tt always took integer form. So this type of problem is solved by either iterative or substitution method.Problems on Master Method: . T(n) = 3T(n/2) + 2. Here a = 3, b = 2, f(n) = n*. Now, nleg,4 = nlee,3 = n! 585, But f(n) = n”, so it will be in case-3, where € > 0. =>e=.415 => f(n) = Q(n'"s,4+ 9 => T(n) = O(f(m)) => T(n) = 0(n?) and af(n/b) < cf(n) => 3 * f(n/2) < cn. => 3n?/4 < cn? = 3/4 0 and c < 1). So, af(n/b) < cf(n) is satisfied. 2. T(n) = 4T(n/2) + n’. Here a = 4, b = 2, f(n) = n’. Now, nles,# = nlee,4 = nlee,2? = piles? = p24) = 2, (since, log,2 = 1). So it satisfied case — 2. i.e.: f(n) = (n!"2,) => f(n) = O(n?) => T(n) = O(nl’e,*. logn) => T(n) = 0(n’logn) (Ans)T(n) = 2T(n/2) + n. Here a = 2, b = 2, f(n) = n. Now, nlee,a = nls? = n = f(n). So it satisfies case — 2. That is: f(n) = O(nl’s,*) = n => T(n) = O(n'’s,*. logn) = O(mlogn). T(n) = 16T(n/4) + n. Here a = 16, b = 4, f(n) = n. Now, nlee,a = ples 16 = ples 4? = ples, 4 = n?*! = n?. It satisfies case — 1. That is f(n) = O(n!"s,2-*) =>n=O(n?-*) =O(n?-!) => n= O(n) (since, € = | and e > 0) => T(n) = 0(n!°s,2) => T(n) = @(n2) (Ans) T(n) = 2T(n/2) + n — 1. Here a = 2, b = 2, f(n) =n—- 1. Now, n'°s,* = nl®s,? = n. Since, f(n) =n — | that does not belongs to O(n'*,"~°). So case — | does not apply. But as f(n) € 0(n). So according to case — 2, we have T(n) = O(nlogn) (Ans). Tn) = TB3n/4) + 1 and T(1) = 0(1). We have to find its asymptotic bound. Using the Master method we have, a= 1,b =4/3 and f(n) = 1. Now, n'ee,* = nlee,,.! = n° = 1. So case —2 applies. Since 1 = 0(1). So T(n) = O(logn) . Tn) = 4T(n/2) + n. Here a = 4, b = 2, f(n) = n. Now, nes, = nbes,4 = nles,2? = n2les,? = n?. Since, f(n) = n. So it satisfies case — 1. That is f(n) = O(n!°8,*~*) => n= O(n? ~!) => n = O(n). (since € > 0 and € = 1). Thus T(n) = 0(n'°s,*) => T(n) = O(n’) (Ans). Tn) = 4T(n/2) + n°. Here a = 4, b = 2, f(n) = n°. Now, nes.2 = nles,4 = nles,2? = n2les,? = n?. Since, f(n) = n°. So it satisfies case — 3. That is f(n) = Q(~nl’s,*+*) = O(n? +!) => n = Q(n3). Thus, T(n) = 0(n%) and af(n/b) < cf(n). => 4f(n/2) < cn? => 4f(n°/8) < cn? => 4/8n3 < cn? => 1/2n3 = cn} => c = 1/2. (since c > 0 and c < 1). So, af(n/b) < cf(n) is satisfied.4. Recursion Tree Method: = Recursion Tree Method is pictorial representation of an iteration method, which is in the form of a tree, where at each levels, nodes are expanded. It is used to keep track of the size of the remaining arguments in the recurrence and the non-recursive costs. In a recursion tree, each node represents the cost of a single sub-problem. We add the cost within each level of the tree to obtain a set of pre-level cost and then we add all the levels of costs to determine the total cost of all levels of recursion. = In general, T(n) = aT(n/b) + f(n)f(n) f(nib) fnfb) fb) ~~ aftav) AN OTR OTT™ f(a/b?) f(n/b?)f(n/b?)—_f(1n/b?) f(n/b2) f(n/b?) f(n/b?) f(n/b?) f(n/b?) ~~ wfn/b?) AN FCM”) FC b®) F(a /b8) oer reeeee seers anf(n/b") Ta) => T(n) = aT(n/b) + f(n) =>T(n/b) = aT(n/b?) + f(n/b)Theorem: 1. Let a > 1 and b > 1 be constants. Let f(n) be a non- negative function defined on exact powers of ‘b’. Define T(n), be an exact power of ‘b’, by the recurrence. T(n) = [0(1),n=1 {arn + f(n), if n = bi, i=+ve integer => Tin) = 0(nP58) + D2" lf) Since, alog,n = nlog,a @(n!°e,*) = Total cost for the leafs, yiegen = sum over of all levels a'f(n/b') = cost per level(| Problems on Recursion 1 Tree: oS 1. T(n) = T(n/3) + T(2n/3) + n. The recursion tree for this is: <> a Cus) Canis) 0 C9) C209) Can) Gani) —> sststansessesanenseesnsaseansnsescsesnesssseansnscssusensceasnncesenest Total = O(nlogn) logs on => T(n) =n+n+n+..... log, n times = O(nlogn).2. Tin) = 2T(n/2) + n?. The recursion tree for this is: => P Total = O(n?) log.n So, the above recurrence has the solution T(n) = 6(n2).3. T(n) = 4T(n/2) + n. The recursion tree for this is: We have, n+ 2n+4n+... + logn times =n(1+2+4..... logn times) =n(2log,2 — D2 — 1) =n? —n= O(n?) => T(n) = 6(n’)4. T(n) = 3T(n/4) + n. The recursion tree for this is: nf n/4 4 ~~ 3/4 e a/i6) (/16)(n/16) — (n/16) (n/16) (n/16) (0/16) (n/16) (0/16) ~ 9n/16 Td) Td) Td) n‘log> leaves. ——> => T(n) = O(n4log3) + DH0a"— 'G yin => T(n) < O(nlog3) + DZo@)n => T(n) < O(n*log3) + (1/(1 — 34)" => T(n) < 0(n4log3) + (1/4") => T(n) € O(n). Here, we have the linear worst case complexity.5. Solve the factorial with recursion tree with recurrence relation. Factorial: The term ‘n’ factorial indicates the product of the positive integers from | to n inclusive and is denoted by n!. Factorial of number (n) is a recursive manner is defined by the recurrence relation is: Factorial(n) = Tn) =| 1,n=0 T(n—1)*1,n>0 The algorithm for the factorial is: fact(n) i if n= then return 1 else if n > 0 return n * fact(n — 1)From the recurrence relation, Tm)= I,n=0 | ene Delia Put,n=1,2,....N Td) =TQO) + 1, T(2)=TO) +1....Tm) =Tm-1)+1 Using recurrence tree, the solution is: => T(n)=T(n- 1) +1 + T(in-1)=T(n—2)+1 + T(in-2)=TMm-3)+1.... + T(2)=TI)+1 + T(I)=TO)+1 Ta) =TO)+1+14+1+....4+n=>T(n)=TO)+n => T(n)=n+1If, T(1) = 1 is given, then we have to calculate upto T(2)=T() +1 =>Tin=TU)+1t+14+14+...4+n-1 =>T(n)=Td)+n-1l=1+n-1l=n=>T(n)=n Ta) 1 Tm-l aN 1 T-2) 1 T(n-3) wi 1 T(O) : Calculate if for ‘n’ value n(1) =n6. Solve the Fibonacci series with recursion tree with recurrence relation. Fibonacci Series: Fibonacci Series is a series of positive integer in manner that the next term of a series is the addition of previous 2 terms. i.e.: 0, 1, 1, 2, 3, 5, 8, 13, 21. The algorithm for Fibonacci series is: Fibseq(n) { ifn=0 then return 0 else ifn =1 then return | else ifn>1 then return (Fibseq(n — 1) + Fibseq(n — 2))The Fibonacci sequence in recursive manner is defined by the recurrence relation is: Tin) = /0, ifn =0 Lifn=1 Tin— 1) * T(n— 2), ifn > 1. From the recurrence relation, Put, n= 2, 3,.... n. So, T(2) = T@) + T(1), T(3) = T(1) + T(2) .... Ti) = Tin - 1) + T(n- 2)Tin) Tin— 1) Tn 2) Tin—2) T(n—-3) Tin—3) Tin —4) TO) TA) TO) Ta) TO) Td) TO) TA) => T(n) =T(n— 1) +T(n—2) + Tm-l)=Tm-2)+TMm-3) + Tm-2)=Tm-3)+Tm—4).... + T3)=TC)+TQ) + T(2)=T@)+T() T(n) = TO) + T(n— 1) = Tm — 1) => T(n) = T(n- 1), , g Binary Search Algorithm: \ ¢ Binary_search(af[], n, x) ed begin low <— 1, high — n, j =0; while (low < high AND j =0) begin mid — “(low + highy/24 ; if (x = a[mid]) j< mid; else if (x 2i-' log,j — 1 < log.n < log,j (Taking logarithmic value) =>j-Ijj= Uogn! +1 So, the time complexity is: O(logn)(Insertion Sort Algorithm: i\ e Insertion_sort(a[], n) begin for (i= 2 ton) begin j — i; temp < afi]; while(j > 1 AND alj— 1] > temp) begin alj] — alj— 1): joj-b end while a[j] < temp; end for endInsertion sort Analysis: ¢ Insertion_sort(a[], n) Sno Code Cost times 1 | begin 0 1 2 | for (i=2 ton) G 7 3 | begin 0 n-1 4 |jei CG n-1 5__| temp < afi] G al 6 | while > 1 AND alj—1]>temp) begin | C, hak 7 | alil—alj— 1) Cs | Uje2G-! 8 |j—oj-t Co | Lpe2tj-1 9 | end while O | Shet-l 10 | alj] — temp G all 11 | end for ° a. 12, | end 9 1© Total time = > cost x time =0+Cn+0+C(n—-1)+C(n- 1+ CdR + CoQjbet; —14+Cdfi2t; —1+0+C(n—- 1+ 0+0 = Cn+CGm-1)+Cn—1) +C,(n— 1) + CQ jze ty + CsDfLy tj — 1+ CoLj2 6 —1 = (C, + Cy+ Cy + Cyn — (C+ Cyt Cz) + CyEPe ty +Cs Dia (t; — 1) +C, Dita (t; -— YD =Cgn— Cy + CaQfbg t+ CsXfL2 tt) — Csbfb2 1+ Cod jez t) — CoBjE2 1 = Cyn ~ Cy + (Cyt Cs + Co)Xjb2 tj — CsXjb2 1 - Codjke 1 = Cgn— Cy + Cydia t) — Con + Cs — Con + Cg=Cgn — Con — Cn— Cy + Cs + C6 + Cyd jn2 6; = n(Cg — Cs — Cy) + (Cs + Cg — Cy)+ Croft =Cy + Cyn + Ciodfe2 tj = Cy, + Cyn + Cypnta + 1/2 — Cy = Cy, — Cyp + Cyn + Cyon?/2 + Cjon/2 =Cy3 + Cyn + C,n?/2 + Cygn/2 = Cy3 + Cyn + Cygn/2 + Cygn7/2 = Cyn?/2 + C,yn/2 + C,3 = An? + Bn + C= O(n?) © So, the best case run time of insertion sort is: T(n) = Cin + C,(n— 1) + C,(n- 1) + C(n- 1) =(C,+C,+ C,+ Cyn —(C, + C3 4+ C,) © The running time, can be expressed as ‘a, + b’ for constant ‘a’ and ‘b’ that depend on the statement costs ‘C;’, it is thus a linear function of ‘n’ j=2 Fj© The worst case run time of insertion sort: Tin) = Cin + C,(n— 1) + Cm — 1) + C,((n(n + 1) — 1/2) + Cy((n(n = 1)/2) + C,((n(n — 1/6) + Co = 1) = (C,/2 + C5/2 + C,/2)n? + (C, + C) + C3 + C,/2 - C,/2 +C,/2+C;) e The worst case running time can be expressed as: an? + bn +c for constants a, b, and c and again depend on the statement costs ‘C,’, it is thus a quadratic function of én? nBubble Sort Algorithm: e This algorithm sorts the element of an array ‘A’, having elements in ascending or increasing order. e Step 1: Initialization (p = pass counter, E = count the no. of exchanges, | = unsorted element) © Step 2: loop, Repeat through step 4, while (p < n- 1) Set E — 0: initializing exchange variables e Step 3: Comparison loop Repeat fori 1, ... 1-1 if (A[i] > A[i + 1]) then set A[i] — A[i + 1] : Exchanging values setE O(n’)Analysis Design Technique ¢ Given a problem, the algorithm is largely influenced by the choice of data structure. With a chosen data structure one can develop a no. of different algorithm for set problem. ¢ The I* intuitive algorithm may not be the best one far as memory and time efficiency is concerned. There are some general techniques for development of algorithms. Those are: = Divide and Conquer = Greedy Strategy = Dynamic Programming = Back Tracking = Branch and Bound ° oS(| Divide and Conquer: ; e The Divide and conquer method includes 3 steps. That are: 1. Step 1: Divide the problem into no. of sub-problems. 2. Step 2: Conquer the sub-problem by solving them recursively, only if the problem sizes are small enough to be solved in a straight forward manner, otherwise step-1 is executed. 1) 3. Step 3: Combine the solutions obtained by sub- problems and create a final solution to the original problem. = Example: Merge Sort, Quick Sort and Heap Sort a. Merge Sort: |. Step 1: The whole list is divided into two sub-lists of n/2 elements each for sorting.\ Sh / t c 4 | |. Step 2: Sort the sub-list recursively using merge sort. 2. Step 3: Now merge the two sorted sub-lists to generate the sorted answer. = For accomplishing the whole task, we are using two procedures ‘Merge Sort’ and ‘Merge’. The procedure ‘Merge’ is used for combining the sub-lists. = The analysis is the part of the Merge Sort is solved by recursion tree method. Merge Sort Algorithm: Merge_Sort(A, p, r) il. iil. ifp Total Cost = cnlogn and Total Time = nlogn => T(n) = 0(nlogn) Example: Sort the elements 2, 4, 5, 7, 1, 2, 3, 6 using merge sort. Ans: 2, 4,5, 7, 1, 2, 3, 6. Now, the array is: pel | 2 3 |4a@| 5 6 7 | 8 2 4 5 7 1 2 3 6fc p=lLr=8 q= 414+8y2t= l4st=4 Merge_sort(A, p, q) = Merge_sort(A, 1, 4) Merge_sort(A, q + 1, r) = Merge_sort(A, 5, 8) Merge(A, p, q, T) n,=q-p+1=4,n,=r—q=8-4=4 jand R[1 ...n,+ 1] Create Arrays L[1 ... ny + => L[I to 5] and R[1 to 5] 273 ]4 1] 273 Li2]4]s[7]o R[1| 2] 3 fori = lton, =1to4 or j= lton,=1to4 Lfi] = Alp +i-1] RU] =Alq +i] L{l] =A[l] =2 R[1] =A[5] =1 L[2] =A[2] =4 R[2] = A[6] =2 L[3] = A[3] =5 R[3] = A[7] =3L[4] =A[4] =7 R[4] =A[8] =6 L[5] =00 R[5] =20 Now, 17273 )4 5 7 Al2][4][5 [7171 3 273 7475 1J/2]3]4]5 Li2]4][5s [7] R/[1]2]3 [6] i J ¢ Nowie1,j<1.Fork=ptor=1 to8. If L[i] < Rj], then A[k] — R[i] and i=i+ 1 else A[k] = R[j] andj=j +1. ¢ Here, Lfi] = L[1] = 2 and R{j] = R[1] = 1 => L[i] < Rj] (false). ¢ So, A[k] = R{j] => A[1] = R[1] = 1 andj =j+1=1+1=2.1 2 3 5 6 7 8 A 1 4 5 1 2 3 6 k 2 3 4 5 1 2 3 4 5 L 2 5 7 oO R 1 2 3 6 oO i j e Nowi=1,j=2,k=ptor=2to8. © L{i] < Rj] => LE] < R22] = 2 <2 (true). ¢ So, A[k] = L[i] => A[2] = L[1] =2 andi=i+ 1=1+1=2. Now,a J ¢ Nowi=2,j=2,k=ptor=3to8. ¢ L{i] < RG] > L[2] < R[2] = 4 2 (false). ¢ So, A[k] = R[j] => A[3] = R[2] = 2 andj =j + 1=2+1=3. Now, 1 2 3 5 6 7 8 A I ie 2 1 2 3 6 1 3) 4 {5 1} 2)3)4)]5 L 2 4 5 ~ R 1 2 3 6 oO¢ Nowi=2,j=3,k=ptor=4to8. © L{i] L[2] < R[3] => 4 $3 (false). ¢ So, A[k] = Rij] => A[4] = R[3] = 3 andj =j + 1=3+1=4. Now, 1]2]3]4 Li2];4{s]7]Je]/ri[if2]{3]ele i j e Nowi=2,j=4,k=ptor=5to8. e Lfi] < R[j] = L[2] < R[4] = 4 < 6 (true). e So, A[k] = L[i] => A[S] = L[2]= 4 and i =i+ 1= 3. wn Nu w wn172 ]3]4 6]7]8 Ati /2{[2/3]4f/21]3]6 k 2[3]4]5 1]2]3]4 rl2f4{/sl{7l/#|][r{1/2]3]e i J ¢ Nowi=3,j=4,k=ptor=6to8. ° L{i] s RU] => LB] s R[4] => 5 <6 (true). e So, A[k] = L[i] => A[6] = L[3]=5 and i =i+ 1=4. Now,1 3 5 1{/2/3]4]5 L] 2 5 0 R]/1]2];3]6]o i J ¢ Nowi=4,j=4,k=ptor=7 to8. © L{i] < Rfj] => L[4] < R[4] = 7 <6 (false). e So, A[k] = RIj] => A[7] = RI4]= 6 and j =j + 1=5. Now, 1 2/3 ]4]5 ] 6 8 Al 1 2 | 2 4 | 5 6 k 1 3)4]5 1{/2/3]4]5 L{/2]4]5]7]|o R[1][2]3]6]o° Nowi=4,j=5,k=ptor=8to8. © Li] < Rij] => L[4] < R[5] = 7 <~ (true). ° So, A[k] = L[i] => A[8] = L[4]=7 andi =i+ I=5. Now, 1 2 3 4 6 8 A 1 2 3 4 5 7 1 2 3 4 2 1 2 3 5 L 2 4 5 7 ~ R 1 2 3 i ¢ Now the elements are sorted using merge sort. e That is: 1, 2, 2, 3, 4,5, 6, 7.The representation of tree structure is: 1 2 2 3 4 5 6 7 2 4 5 7 2 4 5 7 2 s| [4 7 1 3 2 6 Van ferge ergo ka 5 2 4 7 1 3 2 T 6 e The recurrence relation is: Tm) e ifn a[down], then down++. While pivot < a[up], then up--. wn If the position of down < position of up, then swap the value of up and down. Then again the conditions 1 and 2 etc. are performed. 4. Tf position of down < position of up is false then swap up the value with pivot value. 5. Then after the first pivot element is selected then the array is divided into 2 parts and again these 2 parts are again sorted.Quick Sort Algorithm: Quick_Sort(A, p, r) i. ifp T(n) = @(nlogn). [Applying master method]. = E.g.: T(n) = T(n— 1) + TO) + O(n) => T(n) =T(n-— 1) + 0(n) => T(n- 1) = T(n— 2) + @(n- 1) II. Worst Case Analysis: = It occurs when the partitioning routine or algorithm or pseudo-code produces one sub-problem with n — 1 elements and one with zero elements.=Let us assume that this unbalanced partitioning arises in each recursive call. The partitioning costs, 0(n) times, since T(1) = 0(1) and the recurrence for the running time is: "E.g.: T(n) = T(n— 1) + TO) + O(n) => T(n) = T(n — 1) + O(n) => T(n) = 6(n?) [Using substitution method]. IIL. Average Case Analysis: =In average case analysis, the array is partitioned by choosing any random number. In this case, at each level some of the partitions are well balanced while some are fairly unbalanced. Let us assume that the partition of array to be q : 1, the recurrence so obtained is: T(n) = T9n/10) + T(n/10) + n => T(n) = O(nlogn). = Each step is about ‘n’ and (9/10)'. n = 1. => log(10/9) . logn steps.n/10 9n/10 —————__n AOS n/100 9n/100 = 9n/100 81n/100 — n IN NN OA seen O(nlogn)| ¢. Heap Sort: \ va = The heap sort is accomplished by using 2 other function, { MS] that is: ad i. Build-Heap: For maintaining a max heap ii. Max Heapify: For fixing the heap = The heap is created when we input an array of ‘n’ element where ‘n’ represents the length of the array ‘A’. ie.: n =length[A]. | Algorithm for Build a heap: Build_Max_Heap(A) i. heapsize[A] ii. for i length [A}/2 down to 1 iii. do Max_Heapify(A, i)Algorithm for Max_Heapify: Max_Heapify(A, i) i. 1 left(i) ii, re right(i) iii. if |< heapsize[A] and A[l] > A[i] iv. then largest —1 v. else largest —i vi. ifr A[largest] vii. then largest —r viii. if largest 4 i ix. then exchange A[i] — A[largest] x. Max_Heapify(A, largest)| Algorithm for Heapsort: \ cA Heapsort(A) A | i. Build_Max_Heap(A) | ii. fori — length [A] down to 2 | iii, do exchange A[I] © Afi] iv. heapsize[A] —[A]-1 y. Max_Heapify(A, 1) _—— for Heapsort: I. Running Time of Max_Heapify: = The running time of Max_Heapify on a sub-tree of size n’ rooted at given node (i) is 0(1) time to fix up the relationship among the elements A[i], Al[left(i)] and Al[right(i)] + the time to run Max_Heapify on a sub-tree rooted at one of the children of node ‘i’.IL. Il. = The children’s sub-tree is having size 2n/3. So the worst case occurs when the last row of the tree is exactly half full. “If ‘n’ is the heapsize, then T(n) < T(2n/3) + O0(1) that implies T(n) = O(ogn) [Using Master Method]. Time required by Max_Heapify, when called on a node of height ‘b’ is O(h). Running time of Build_Max_Heap: =The running time of Max_Heapify is O(logn). Here heapify is invoked n/2 times, thus the time complexity of Build-Max-Heap is O(nlogn) and takes time O(n). Running time for Heapsort: = The heap sort procedure takes time O(nlogn). Since the call to Build-Max-Heap takes time @(n) and each of the n— 1 call to Max_Heapify takes O(logn). So the total running time for Heapsort is O(nlogn).a Lower Bound of Sorting: K | e Before going to lower bound for sorting, some important concepts are explained below. That are: = «Internal sorting: It refers to the sorting operation performed over a list which is stored in a primary memory. = External sorting: When a list is stored in a file is accommodated in the secondary memory, the sorting technique is referred to external sorting. = In-place: The sorting algorithm is in-place, only if a constant number of data elements of an input array are ever stored is required and hence it is possible to sort a large list without the need of additional working language.= Stable: A sorting algorithm can be divided into 2 main classes. That is: in-place and stable. A sorting algorithm is stable, in which the two elements that are equal remain in the same relative position after performing the sorting. In a comparison sort, we use only comparisons between elements to gain order information about an input sequence that is given two element a, and a, We perform one of the tests a; < a;, a; S a;, aj = a;, a; 2 a;, or a, > a, to determine their relative order. Here a; = =a is useless and the comparisons a; < a, a, > aj, a, < aj, a, > a, are all equivalent in that they yield identical information ‘about the relative order ‘a,’ and ‘a;’. We therefore assume that all comparisons have the from as Sa,.