Theory of Complexity

28.6.20

1. Motivation

In the previous chapter Computability computable and non-computable functions

and problems that can be solved by a computer program or cannot be solved by a
computer program were considered. The time or effort of a program or a problem
solution was not discussed at all. In this chapter only computable functions both
total and partial are subjects of the consideration.
The question here is how much effort these functions create or how much effort it
needs to solve a problem. Function means here both a given mathematical func-
tion implemented in a programming language and also a function in the definition
of a programming language. Effort means time and/or memory space needed to
execute a function.
Different Functions in a programming language that solve the same problem can
be compared. Problems can be analyzed how much effort their solution requires
independent from an implementation in a programming language. Also different
problems can be compared.

In total the theory of complexity deals with

 computable problems, i.e. a problem can be solved using a computer
 time / effort needed to solve a problem (e.g. steps a Turing Machine performs,
number of operator call in C)
 memory space needed to solve a problem (e.g. tape size needed in the Turing
Program)
 comparison of different problems regarding effort
 classification of problems regarding effort
 comparison of different functions to solve one problem regarding effort

2. First example
A well-known example for the consideration of effort to solve a problem is the
travelling salesman problem:
A salesman living in Deggendorf visits several customers in different cities and
tries to find the best (e.g. shortest or fastest) way to visit all of them. The following
picture shows the (fictional) highway connections between these cities (without
the distances).
Is this problem solvable?
What approach can be taken to find the best way?
 Obviously the problem can easily be solved in a naïve way. The salesman starts
in Deggendorf. From Deggendorf Berlin, München and Hamburg can be reached.
From Berlin Hamburg and Regensburg and Deggendorf can be reached and so
on. It all combinations are tried out the best way can be found. The problem is the
number of possible combinations that is increasing “dramatically” with the number
of cities and connections between them. If the situation gets more complex, i.e.
more cities and different ways between them, the number of possible ways to visit
all customers makes an analytical solution impossible. The reason is here the
exponentially increasing runtime.

The question if the traveling salesman problem can be solved with an effort
“below exponential” is not yet answered.
3. Theoretical Approach

This chapter gives some ideas how to bring the problem of complexity in a formal
context.

Definition:
Let f :  be a total function over the natural numbers. ( including 0)
The class TIME(f (n)) is the set of all languages A where a deterministic multi-tape
Turing Machine M exists with A = T(M) (i.e. M comes to an end with an input out
of A) and time(M(x))  f (|x|)  words x.

(timeM(x) is the number of calculation steps performed by M with input x, |x| is the
length of x)

This definition allows to classify the effort to solve a problem1 using a Turing
Machine. (A more practical definition is given with the classes O(f(n)) below.)

The class of problems causing polynomial effort can be defined now:

Definition:
P := { A |  Turing Machine M and a polynom p with T(M) = A and time M(x)  p(|x|))

1
„language can be replaced by „problem“
 Intuitively spoken, P contains all problems that can be solved by “efficient“
algorithms (efficient = max. polynomial effort).

A more practical definition to measure effort needed to solve a problems or

needed to execute an algorithm is the O-notation.

Definition:
 Let f, g :    be functions ( including 0)
 One writes f(n)  O(g(n)) if and only if there is a positive constant c such that
for all sufficiently large values of n, the value of f(n) is at most c multiplied by
the value of g(n).
 f(n)  O(g(n))   c ,  n0  : f(n)  c * g(n)  n  n0
Remark: The ‘O‘ stands here for “Order“ (in German “Ordnung“)

This still rather formal definition can be applied to problems and algorithms in a
more simplified way where c is set to 1 and g is a rather simple function.

For the relation between polynomial effort and exponential effort the following
(mathematical) relation is important:

let l  , q , q > 1  nl  O(qn)

In terms of the definition above, this means that for a given natural number l there
is a n0  and a natural number c so that nl ≤ c * qn  n  n0

Spoken in words, this means that any exponential function (basis > 1) grows faster than
any polynomial if the exponent n exceeds a limit value.

Example:
• Let l be 5 and q be 2 then n5 ≤ 2n for n  25

Using the above O-notation definition both problems and algorithms can be
analyzed related to their runtime. The analysis does not provide a concrete
runtime (e.g. runtime in seconds2) but describes how the runtime changes with
changes in the input size.

Examples:
 sorting algorithms: quicksort  O(n log(n)) (in „normal cases“), bubblesort
 O(n2)
 There is no known algorithm to solve the Travelling-Salesman-Problem
with polynomial runtime
 The Travelling-Salesman-Problem can be solve with algorithms in
exponential runtime, i.e.  O(cn), with c > 1.

2
A concrete runtime would make sense for an existing computer but not for a theoretical consideration.
n describes here the „size“ of the input

The following “O-classes” are often considered:

 constant runtime: O(1) resp. O(k)

 logarithmic effort O(log(n))
 linear runtime: O(n)
 quadratic runtime O(n2)
 cubic runtime: O(n3)
 polynomial runtime: O(nk) for a constant k
 exponential runtime: O(2n) resp. O(qn) for q > 1
The following table shows the relation between different classes:

runtime Steps for n = Input that can

100 be handled
with 1000000
steps

O(1) 1 1000000

O(n) 100 1000000

O(n2) 10000 1000

O(n3) 1000000 100

O(2n)  1,27 * 1030  20

The table shows the effort for “exponential problems” increases “dramatically”
with the input size and cannot be handled any more in a reasonable time. E.g.
to solve the Travelling-Salesman-Problem with 20 cities takes 1000000 steps.

Back to the theory of complexity: A problem is in P if and only if it can be

solved by a computer (e.g. a Turing Machine) that executes a polynomial
number of steps, i.e. the computer performs nk steps, i.e. O(nk) for an input
size n. The complexity classes are mainly independent from the computer
model. Using a Turing Machine or While- resp. Goto-programs on any
computer does not change the complexity classes.

In the chapter with the theory of Turing Machines there was a short
presentation about non-deterministic Turing Machines. This definition can also
be used to define a problem class in the theory of complexity:

Definition:
 Now we define the class of all languages that are accepted by
nondeterministic Turing Machines with polynomial runtime limitation:
let f:    ( including 0) a total function.
The class NTIME(f (n)) consists of all languages A for that a non-deterministic
multi-tape Turing Machine M exists with A = T(M) and ntimeM(x)  f(|x|) for all
words x.

ntimeM(x) is defined as minimum of the length of accepting calculations of M

with input x, if x  T(M) or as 0 if x  T(M)

Analog to the above definition of TIME(f(n)) the definition NTIME(f(n)) can be

used to define complexity classes:

Definition:
Complexity class NP

NP :=  NTIME(p(x))
p polynomial

Remark:
obviously P  NP
but it is unknown P  NP

Another example for a well-known problem is the “SAT”-problem:

Decide if a boolean formula f(a, b, …) can be satisfied (therefore SAT

problem), i.e. decide if a set of values exists for all variables (a, b, …) that lead
to the boolean formula f resulting to value true3.
E.g.
 a  b, a  b, a, a  b  c are satisfiable
 a  a, a  a  false are not satisfiable

The SAT-problem can be solved with exponential effort, but it can be decided
with polynomial effort if a set of values satisfies the boolean formula, i.e. SAT
is element of NP (it is even NP-complete).

The class NP contains (intuitively) all problems, for that a possible solution can
be guessed and the correctness of the guessing can be determined with
polynomial effort.
(the name NP comes from nondeterministic polynomial. This means that a
nondeterministic Turing Machine can guess a possible result and find out with
polynomial effort, if this is a solution or not)

This leads to the question if there are problems where it is more complex to
find a solution than to evaluate if a possible solution is correct, i.e. P  NP ?

3
in this case the formula is satisfiable, therefor the name SAT
Reducability

In the theory of computability we tried to „reduce“ a problem A (with unknown

computability) to another problem B (with known computability) to find out the
computability of A

This concept can be adapted to complexity to make statements in the

following form: If a problem B can be solved, then another problem A can be
solved and this takes only a polynomial additional effort.

Formal definition of Polynomial Reducibility:

Let A and B be two languages, A  *, B  *. A is called polynomial reducible
to B (A p B) if there is a total, computable, deterministic function f : *  *, f
can be calculated with polynomial effort and for all x  *

x  A  f(x)  B

If A p B then
 B  P  A  P (i.e. if B can be calculated with polynomial effort that A can
be calculated with polynomial effort)
 B  NP  A  NP (i.e. if B is in the class of the most complex problem then
A is in the same class)

Proof of „if A p B then B  P  A  P“

A Turing Machine TMA that accepts A can be constructed from a Turing
Machine TMB that accepts B and a Turing Machine TMf that calculates f.

The picture below shows the principle:

TMB has polynomial runtime p(|x|), f has polynomial runtime q(|x|), this means
that TMA has polynomial runtime p(|x|) + q(|x|), as p and q are polynoms.

The most complex problems are the NP-complete problems (NP-vollständigen

Probleme). These are the problems that can be reduced to NP-problems.

Definition NP-hard, NP-complete:

A language A is called NP-hard (NP-schwer), if for all languages L  NP L p A
 A language A is called NP-complete (NP-vollständig), if A is NP-hard and A 
NP
(this means that a NP-complete language is at least as complex as any other
problem in NP)

Remark: Here also the word “language” can be replaced by the word
“problem”.

The concept of NP-Completeness in (theoretical) computer science important

because many practical problems are NP-complete
 Exact solutions can be calculated but needs a too big effort
 Approximate solutions must be calculated instead with less effort.

Instead of …

tell your boss

Examples of NP-complete problems

 SAT problem
 Directed Hamiltonian Path (gerichteter Hamilton Kreis) (DHP)
(Is there a path through a directed graph visiting all vertexes only once)
 Undirected Hamiltonian Path (ungerichteter Hamilton Kreis) (UHP)
 Graph Coloring (Färbbarkeit von Graphen)
(e.g. is it possible to color the vertices of a graph in a way that no neighbor
vertex have the same color, with a given number of colors)
 Bin Packaging
(can a given number of objects be packed in a given number of containers
without overloading a container)
Examples of non NP-complete problems or problems with unknown status
 Prime number (can be solved with polynomial effort)
 Prime factorization (unknown)
 Isomorphism of graphs (unknown)
Summary of this part of the course:
 Decidable problems can be classified (in terms of time and space)
 In particular there are problems that can be solved in polynomial time (P) and
problems where possible solutions can be checked on polynomial time (NP)
 The most complex problems are in the class NP-complete (no polynomial
solution known)