Algorithms

Iterative Algorithms

I Basic construction for repeating the same set of steps

I Essential for general computation

I while , for , repeat

GENERIC-ITERATION1
1 Initialization
2 repeat
3 Progression
4 until Stopping-Condition

GENERIC-ITERATION2
1 Initialization
2 while not Stopping-Condition
3 Progression

GENERIC-ITERATION3
1 Initialization
2 if not Stopping-Condition
3 Progression
4 else Go To 2

Example

FINDMAX1(A)
1 max ← −∞ // Initialize max
2 i←1
3 while i ≤ A.length
4 if A[i] > max // Update max if necessary
5 max ← A[i]
6 i ←i+1
7 return (max )

FINDMAX2(A)
1 max ← −∞ // Initialize max
2 for i ← 1 to A.length
3 if A[i] > max // Update max if necessary
4 max ← A[i]
5 return (max )

Loop Invariants
I Loop invariants help us understand why an algorithm is correct.

A loop invariant is a predicate such that:

1. Initialization of the loop:

It is true, given the precondition, after the Initialization, that is,
prior to the first iteration of the loop
2. Maintenance of the loop:
If it is true before an iteration of the loop, it remains true after the
execution of the Progression (body of the loop) and hence before
the next iteration
3. Termination of the loop:
The loop terminates, and when it terminates, the invariant along
with the Stopping-Condition give the postcondition of the loop, a
useful property that helps show that the algorithm is correct.
introductory example: the Euclidian division 9

precondition postcondition
of the loop of the loop

and
initialization

establishes
stopping
invariant
condition

and
not

progression
preserves

action situation entailment

Fig. 3.1 – Articulation invariant/stopping condition/progression/initialization.

Example

FINDMAX(A)
1 max ← −∞
An introductory example: the Euclidian division
2 for i ← 1 to A.length
consider
3 the if
classical
A[i] > problem
max of
// the Euclidian
Update max division. We are interested in the calc
if necessary
on of the quotient of the “integer” division of A by B (A 2 N, B 2 N1 ), knowing that w
4 max ← A[i]
not provided with a division operation. The specification of the program to design
en by5 thereturn
pair: (max )

condition: (A 2 N) and (B 2 N1 ).
Loop Invariant:
stcondition: q is the quotient and r is the remainder of the division of A by B.
At the start of each iteration of the for loop of lines 2-4, the variable
In order
max to
is make
equalthe postcondition
to the maximum“usable”
entry in(this point willAbe
the subarray [1 developed
: i − 1] later), it
ormulated on the basis of the definition of the Euclidian division by:

(A = B · q + r) and (0 6 r < B).

Sorting
m this expression, we take as invariant:

(A = B · q + r) and (0 6 r)
Sorting
d as stopping condition: r < B. The actions related to the progression and the initializ
n remain
Input: Anexpressed,
to be array A ofas well as
length n the termination expression. As to the progressio
uffices to increment q of 1 and to decrement r of the value of B to restore the invaria
hich is possible since the stopping condition is not met, hence r > B). The invariant
Output:
ablished from the A ordered copy Bbyofthe
precondition array A, that0is,tofor
theassignments: some
q and A permutation
to r. Concerning th
mination
i0 , i1 ,expression,
. . . , in−1 ofit0,can
1, . be n − 1, it holds
. . ,observed that the expression (A - q · B) is equal to
that
er the initialization takes place and diminishes of B whilst being positive after each ste
(1) A[j] = B[ij ] for j = 0, . . . , n − 1
(2) B[0] ≤ B[1] ≤ · · · ≤ B[n − 1]

Insertion-Sort

I Insertion sort is one of the simplest methods for sorting an array A

I The idea is to iteratively sort the initial subarray adding one more
element in each iteration

I In the ith step, the next element A[i] is inserted into the previous
subarray A[1 . . i − 1] which is already sorted

I The method is not very efficient, its running time is O(n2 ) while
faster algorithms sort in time O(n log n).

I We present first a recursive version and then an iterative version of

the algorithm.

Recursive version

INSERTION-SORT(A, n)
1 if n = 1 return
2 INSERTION-SORT(A, n − 1)
3 INSERTLAST(A, n)

INSERTLAST(A, n)
1 // Insert A[n] into the sorted sequence A[1 . . n − 1]
2 if n = 1 return
3 if A[n − 1] > A[n]
4 A[n − 1] ↔ A[n]
5 INSERTLAST(A, n − 1)

Iterative Version

I The iterative version has two nested loops

I it starts with the initial element trivially sorted by itself

I each iteration of the outer loop (for) adds the next element

I the inner loop (while) corresponds to the insertion.

INSERTION-SORT(A)
1 for j ← 2 to A. length
2 // Insert A[j] into the sorted sequence A[1 . . j − 1]
3 i ←j−1
4 while i > 0 and A[i] > A[i + 1]
5 A[i] ↔ A[i + 1]
6 i ←i−1

I The next version holds the element to be inserted in the variable

key while the place where is to be inserted is found.

INSERTION-SORT(A)
1 for j ← 2 to A. length
2 key ← A[j]
3 // Insert A[j] into the sorted sequence A[1 . . j − 1]
4 i ←j−1
5 while i > 0 and A[i] > key
6 A[i + 1] ← A[i]
7 i ←i−1
8 A[i + 1] ← key

Invariant for the outer for-loop in lines 1-8.

At the start of each iteration of the for loop of lines 1-8, the subarray
A[1 . . j − 1] consists of the elements originally in A[1 . . j − 1] but in sorted
order.

Initialization:
I In the first iteration, j = 2 and then A[1 . . j − 1] = A[1] which is the
original A[1] and it is trivially sorted.

Maintenance:
I Steps 2 places the next item A[j] in the temporary variable key and
step 3 initializes i to j − 1.
I From the termination analysis for the inner while loop in lines 5-7
the sorted subarray A[1 . . i] is unmodified and all its elements are
smaller than or equal than key, and
I the sorted subarray A[i + 1 . . j − 1] has been shifted one position to
A[i + 2 . . j] and all of its elements are larger than key.

1 i i+1 j−1 j

1 i i+1 i+2 j

I Thus, step 8 places key which contains the original A[j] into the
correct place so that now the subarray A[1 . . j] consists of the elements
originally in A[1 . . j] but in sorted order.
Termination: The loop terminates when j takes the value A. length + 1.
By the invariant then A is the initial array but sorted.

Invariant for the inner while-loop in lines 5-7.

I From the invariant of the outer for loop, before the execution of the
inner while loop, the subarray A[1 . . j − 1] consists of the elements
originally in A[1 . . j − 1] in sorted order and the subarray A[j . . n] has
not been modified.

I From the instructions 2 and 4 before the while key contains the
original element A[j] and i has been initialized to j − 1.

INSERTION-SORT(A)
1 for j ← 2 to A. length
2 key ← A[j]
3 // Insert A[j] into the sorted sequence A[1 . . j − 1]
4 i ←j−1
5 while i > 0 and A[i] > key
6 A[i + 1] ← A[i]
7 i ←i−1
8 A[i + 1] ← key

Invariant for the inner while-loop in lines 5-7.

At the start of each iteration of the while loop of lines 5-7, the sorted
subarray A[1 . . i] is unmodified and the sorted subarray A[i + 1 . . j − 1]
has been shifted one position to A[i + 2 . . j] and all of its elements are
larger than key.

1 i i+1 j−1 j

1 i i+1 i+2 j

Initialization:
In the first iteration, i = j − 1 and then A[1 . . i] is A[1 . . j − 1] and
A[i + 1 . . j − 1] is an empty subarray. Thus, the invariant holds.

Maintenance:
In an iteration of the loop the conditions i > 0 and A[i] > key hold and
in lines 6-7 A[i] is shifted to position A[i + 1] and i is decremented by 1,
and so the invariant holds at the beginning of the next iteration.

Termination:
The loop terminates when either i = 0 or A[i] ≤ key.
I If i = 0 then the initial A[1 . . j − 1] subarray has been shifted to
A[2 . . j] and all of its elements are larger than key and so the correct
sorted position for key is A[1] as done in line 8.
I If A[i] ≤ key then the correct sorted position for key is A[i + 1] as
done in line 8.

Merge-Sort

I Application of divide-and-conquer method

I The running time is O(n log n)

Divide: The array is split into two halves, the left and right parts.
Conquer: Each half is sorted recursively
Combine: The two sorted halves are merged into a single sorted array
Basis of recursion: The recursion "bottoms out” –it reaches the base
case – when the subarray to be sorted has just 1 element

Pseudocode

MERGE-SORT(A, p, r )
1 if p ≥ r // zero or one element?
2 return
3 q ← b(p + r )/2c // midpoint of A[p . . r ]
4 MERGE-SORT(A, p, q) // recursively sort A[p . . q]
5 MERGE-SORT(A, q + 1, r ) // recursively sort A[q + 1 . . r ]
6 MERGE(A, p, q, r ) // Merge A[p . . q] and A[q + 1 . . r ] into A[p . . r ]

MERGE(A, p, q, r )
1 nL ← q − p + 1 // length of A[p : q]
2 nR ← r − q // length of A[q + 1 : r ]
3 let L[0 : nL − 1] and R[0 : nR − 1] be new arrays
4 for i ← 0 to nL − 1 // copy A[p : q] into L[0 : nL − 1]
5 L[i] ← A[p + i]
6 for j ← 0 to nR − 1 // copy A[q + 1 : r ] into R[0 : nR − 1]
7 R[j] ← A[q + 1 + j]
8 i←0 // i indexes smallest remaining element in L
9 j←0 // j indexes smallest remaining element in L
10 k←p // k indexes location in A to fill
11 while i < nL and j < nR
12 if L[i] ≤ R[j] //
13 A[k] ← L[i] // As long as each of the arrays L and R
14 i ←i+1 // contains an unmerged element, copy
15 else A[k] ← R[j] // the smallest unmerged element back
16 j ←j+1 // into A[p : r ].
17 k ←k+1 //
18 while i < nL
19 A[k] ← L[i] //
20 i ←i+1 // Having gone through one of L and R
21 k ←k+1 // entirely, copy the remainder of the other
22 while j < nR // to the end of A[p : r ].
23 A[k] ← R[j] //
24 j ←j+1 //
25 k ←k+1

I The while loop of lines 11-17 repeatedly identifies the smallest value
in L and R that has yet to be copied back into A[p : r ] and copies it back
in
I The index k gives the position of A that is being filled in, and the
indices i and j give the positions in L and R, respectively, of the smallest
remaining values

I Eventually, either all of L or all of R is copied back into A[p : r ], and

this loop terminates.

Merging Example

Merge-Sort recursion unfolding

Now, the recurrence equation is

T (n) = 2T (n/2) + Cn

for n > 1, with base case T (1) = D.

Let us verify that T (n) = O(n log n).

Iteration the recurrence equation:

T (n) = 2T (n/2) + Cn
= 2(2T ((n/2)/2) + C(n/2)) + Cn
= 22 T (n/22 ) + C2n
= 22 (2T ((n/22 )/2) + C(n/22 )) + C2n
= 23 T (n/23 ) + C3n
..
.
= 2i T (n/2i ) + Cin
..
.
= 2k T (n/2k ) + Ckn
= nT (1) + Ckn
= Dn + Cn log2 n
= O(n log n).

Quick-Sort

I Application of divide-and-conquer method

I The running time is O(n log n)

Divide: A pivot is selected and the array is split into two parts, the
entries smaller than or equal to the pivot and the entries larger than
the pivot
Conquer: Each half is sorted recursively
Combine: The two sorted parts are simply concatenated
Basis of recursion: The recursion "bottoms out” –it reaches the base
case – when the subarray to be sorted has just 1 element

Pseudocode

QUICK-SORT(A, p, r )
1 if p < r
2 q ← PARTITION(A, p, r )
3 QUICK-SORT(A, p, q)
4 QUICK-SORT(A, q + 1, r )

PARTITION(A, p, r )
1 pivot ← A[r ]
2 i ←p−1
3 for j ← p to r − 1
4 if A[j] ≤ pivot
5 i ←i+1
6 A[i] ↔ A[j]
7 A[i + 1] ↔ A[r ]
8 return i + 1

Invariant

Maintenance

Running Time

The recurrence for the average running time is T (n) is

n
1X
T (n) = (n − 1) + (T (k − 1) + T (n − k)) ,
n k=1

with T (1) = 0. This can be simplified to

n −1
2X
T (n) = (n − 1) + T (k).
n k=1

A possible approach is simply to verify using induction on n that

T (n) ≤ Cn log n for some constant C > 0.
The base case n = 1 is clear since T (1) = 0.
Then
n −1
2C X
T (n) ≤ (n − 1) + k log k
n k=1
 2 
2C n log n n2 log e
≤ (n − 1) + · −
n 2 4
= Cn log n + ((n − 1) − (C log e/2)n)
≤ Cn log n,
P −1 n2 log n 2
if C ≥ 2/ log e (we have used the bound nk=1 k log k ≤ 2
− n log
4
e

which can be obtained by upper bounding the sum by an integral).

Selection-Sort

I Iteratively selects the minimum of the remaining subarray A[i . . n]

and places it in the correct position A[i]

I The running time is O(n2 )

1 for i ← 1 to A.length − 1
2 // find the min element in the unsorted A[i . . A.length]
3 jMin ← i
4 for j ← i + 1 to A.length
5 if A[j] < A[jMin]
6 jMin ← j
7 // place min element in A[i]
8 if jMin 6= i
9 A[i] ↔ A[jMin]

Heap-Sort

Binary Search

BINARYSEARCH(A[p . . q], key )

1 left ← p, right ← q
2 while left ≤ right
3 mid ← bleft + right)/2c
4 if key = A[mid]
5 return (mid)
6 elseif key > A[mid]
7 left ← mid + 1
8 else right ← mid − 1
9 return (-1)

Correctness of BINARYSEARCH:
Each iteration of the while loop of lines 2–8
either finds key in A[mid] and returns the index mid in line 5 or
it reduces the search subarray from A[left . . right] to the “half”
subarray A[left0 . . right0 ] which may contain key:
it is A[left . . mid − 1] if key < A[mid] (line 8) or
A[mid + 1 . . right] if key > A[mid] (line 7).
The discarded half cannot contain key.
Also, because of the choice of mid, we have that
right − left + 1
 
0 0
right − left + 1 ≤ , (∗)
2
with equality in the worst case; so if key is not found then eventually
the condition left ≤ right no longer holds and so the while loop is
exited, and −1 returned in line 9.
Number of times the array is halved: According to (∗), a subarray of
length n is reduced to bn/2c in the worst case. If

2k−1 ≤ n < 2k

then, dividing by 2, jnk

k −2
2 ≤ < 2 k −1
2
and so, after k − 1 times,
jj j n k kk
2 ≤ ···
0
· · · < 21
2
and there is still one more halving. So the total of halvings is k. From

k − 1 ≤ log n < k,

we have that k = blog nc + 1.