Advanced

Programming
Techniques
Lecture 3
 Efficiency
• In computer science, often the question is
not how to solve a problem, but how to
solve a problem well
• Many sorting algorithms are well-known;
the problem is not to find a way to sort
words, but to find a way to efficiently sort
words.
• We are interested in solutions that
optimize the use of computing and human
resources
 Efficiency
• If it's possible to solve a problem by using a brute force technique,
such as trying out all possible combinations of solutions (for
instance, sorting a group of words by trying all possible orderings
until you find one that is in order), then why is it necessary to find
a better approach?
• if you had a fast enough computer, maybe it wouldn't be. But as it
stands, we do not have access to computers fast enough.
• For instance, if you were to try out all possible orderings of 100
words, that would require 100! (100 factorial) orders of words.
• That's a number with a 158 digits; were you to compute
1,000,000,000 possibilities per second, you would still be left with
the need for over 1x10^149 seconds, which is longer than the
expected life of the universe.
• Clearly, having a more efficient algorithm to sort words would be
handy; and, of course, there are many that can sort 100 words
within the life of the universe.
 Resources
• Resources relevant to execution are
– Processor
– internal memory
• Algorithms should be economical in
the use of resources
• No general recipe for efficiency
• Avoid redundant computations
 Example
• Common mistake is to recalculate
expressions in a loop
x :=0;
for i:= 1 to n do
begin
x:=x+0.01;
y:=(a*a*a+c)*x*x+b*b*x
writeln(‘x = ‘,x,’y = ‘,y);
end;
 Example
• Redundant computations are avoided by
precomputing
a3c := a*a*a+c;
b2 := b*b;
x := 0;
for i:= 1 to n do
begin
x:=x+0.01;
y:=(a3c)*x*x+b2*x
writeln(‘x = ‘,x,’y = ‘,y);
end;
• Eliminate redundancies in the innermost
loops of computations
 Array Processing
• Inefficiencies creep into array
processing:
• Example
p:=1;
for i:= 2 to n do
if(a[i]>a[p]) then p:=i;
max:=a[p];

• In the comparison a[i] > a[p], two

memory references are required for
each array element
 Example
p:=1;
max:=a[1];
for i:= 2 to n do
if(a[i]>max) then
begin
max:=a[i];
p:=i;
end;
• The conditional test using max requires
only one memory reference instruction
• Use of max also makes it easier to identify
the task to be performed
 Late Termination
• More tests done than required
• Example: Searching for an item even after
it is known that the element cannot occur
in an array
• Searching for MOORE after MORRIS is
futile
while search_name<>current and not eof do
get next name from list
while search_name > current and not eof do
get next name from list
test if current =search_name
 Example
• Bubblesort algorithm
• The inner loop that drives the exchange
mechanism always goes through the
length of the array
for i:= 1 to n-1
for j:= 1 to n-1
if(a[i]>a[j+1]) then exchange(a[j],a[j+1]);
• Aftr the ith iteration, the last i values will
an array will be in sorted order
• Loop should not proceed beyond n-I
for i:= 1 to n-1
for j:= 1 to n-i
if(a[i]>a[j+1]) then exchange(a[j],a[j+1]);
 Output Conditions
• The algorithm may establish the output
conditions before the general conditions for
termination have been met
• Bubblesort may be used on data that is almost
sorted
• Desirable to terminate the loop as soon as it is
established that the elements are sorted:
sorted = false
while not sorted do
sorted=true
for all unsorted adjacent pairs do
if the current pair not in order
exchange the elements
set sorted to false
 Tradeoffs
• precomputing saves processor time
while using up more storage
• Implementation using minimum
loops
• Let one loop do one task
 What is a good algorithm?
• Attributes of good algorithms:
– Simple, powerful and general solutions
– Easily understood
– Easily modified
– Correct in the given context
– Understood at different levels
– Economical in the use of CPU, storage and peripherals
– Well documented
– Independent of implementations
– May be used as a subprocedure for other problems
– Pleasing and satisfactory design
 Efficiency
• the efficiency of an algorithm is expressed as how long
it runs in relation to its input.
• Usually we refer to the length of input as n;
• so, for the sorting example, the efficiency is roughly
n!.
• it's possible to come up with the correct order early on
in the attempt -- for instance, if the words are already
partially ordered, it's unlikely that the algorithm would
have to try all n! combinations.
• Often we refer to the average efficiency, would be in
this case n!/2.
• But because the division by two is nearly insignificant
as n grows larger (half of 2 billion is, for instance, still
a very large number), we usually ignore constant
terms (unless the constant term is zero).
 Computational Cost
n log2n nlog2n n2 n3 2n

2 1 2 4 8 4

10 3.322 33.22 102 103 >103

102 6.644 664.4 104 106 >>1025

103 9.966 9966.0 106 109 >>1025

0

104 13.287 132,87 108 1012 >>1025

7 00
 Complexity
• Now that we can describe any algorithm's
efficiency in terms of its input length (assuming
we know how to compute the efficiency), we can
compare algorithms based on their "order".
• Here, "order" refers to the mathematical method
used to compare the efficiency -- for instance,
n^2 is "order of n squared," and n! is "order of n
factorial."
• It should be obvious that an order of n^2
algorithm is much less efficient than an algorithm
of order n. But not all orders are polynomial --
we've already seen n!, and some are order of log
n, or order 2^n.
 Order notation
• Standard notation to represent
functions
• Bounds for computing time
• O-notation
• If the dominant mechanism is
executed cn2 times for constant c,
and the problem size n, the
algorithm is said to have an order n2
complexity
 Asymptotic Complexity
• A function g(n) is O(f(n)) if there is a
constant c for which the relationship
g(n )≤ cf(n)
Holds for all positive and finite n.
lim g(n)/f(n)=c where c is not equal to 0.
n∞

• g(n)=3n2+6n+3 and
lim 3n2+6n+3 =3
n∞ n2

• This algorithm has an asymptotic complexity of O(n2)

 Big-O Notation
• This notation, known as big-O notation, is a
typical way of describing algorithmic efficiency;
note that big-O notation typically does not call for
inclusion of constants.
• Also, if you are determining the order of an
algorithm and the order turns out to be the sum
of several terms, you will typically express the
efficiency as only the term with the highest order.
• For instance, if you have an algorithm with
efficiency
n2 + n, then it is an algorithm of order O(n2).
 Example
for(x=0; x<n; x++) {
min = x;
for( y=x; y<n; y++) {
if(array[y]<array[min]) min=y;
}
temp = array[x];
array[x] = array[min];
array[min] = temp;
}
• We compute that the order of the outer loop (for(int x = 0; ..)) is
O(n);
• then, we compute that the order of the inner loop is roughly O(n)
• Note that even though its efficiency varies based on the value of
x, the average efficiency is n/2, and we ignore the constant, so
it's O(n).
• After multiplying together the order of the outer and the inner
loop, we have O(n2).
 Worst and Average Cases
• Two measures of performance
– Worst and average cases
• Worst case for a give problem of size
n corresponds to the maximum
complexity encountered among all
problems of size n.
• Choose a set of input conditions that
force the algorithm to make the least
possible progress towards its goal.
 Average case
• An algorithm that linearly searches an ordered list
of elements for some value x.
• In the worst case, it will be necessary to examine
all n values in the list
• Average case: it is assumed that all possible
points of termination are equally likely. The
probability that x is found at position 1 is 1/n and
at position 2 is 1/n, and so on.
• Average search cost=(1+2+3+4+5)*(1/5)=3 for
n=5
• Note that 1+2+3+..n=n(n+1)/2
• Average cost=(n+1)/2
 Examples
O(1)
An algorithm that runs the same no matter what the input. For
instance, an algorithm that always returns the same value regardless
of input could be considered an algorithm of efficiency O(1).
O(logn)
Algorithms based on binary trees are often O(logn). This is because a
perfectly balanced binary search tree has logn layers, and to search for
any element in a binary search tree requires traversing a single node
on each layer.
The binary search algorithm is another example of a O(log n)
algorithm. In a binary search, one is searching through an ordered
array and, beginning in the middle of the remaining space to be
searched, whether to search the top or the bottom half. You can divide
the array in half only logn times before you reach a single element,
which is the element being searched for, assuming it is in the array.
 Examples
O(n)
Algorithms of efficiency n require only one pass over an entire input. For
instance, a linear search algorithm, which searches an array by checking
each element in turn, is O(n). Often, accessing an element in a linked list
is O(n) because linked lists do not support random access.
O(nlogn)
Often, good sorting algorithms are roughly O(nlogn). An example of an
algorithm with this efficiency is merge sort, which breaks up an array into
two halves, sorts those two halves by recursively calling itself on them,
and then merging the result back into a single array. Because it splits the
array in half each time, the outer loop has an efficiency of logn, and for
each "level" of the array that has been split up (when the array is in two
halves, then in quarters, and so forth), it will have to merge together all of
the elements, an operations that has order of n.
 Examples
O(n2)
A fairly reasonable efficiency, still in the polynomial time range,
the typical examples for this order come from sorting algorithms,
such as the selection sort example on the previous page.
O(2n)
The most important non-polynomial efficiency is this exponential
time increase. Many important problems can only be solved by
algorithms with this (or worse) efficiency. One example is
factoring large numbers expressed in binary; the only known way
is by trial and error, and a naive approach would involve dividing
every number less than the number being factored into that
number until one divided in evenly. For every increase of a single
digit, it would require twice as many tests.