Algorithms Useful Combinatorial Optimization Algorithms *not in exam

Professor Chien-Mo James Li Graduate Institute of Electronics Engineering National Taiwan University
Algorithms NTUEE

Outline
Introduction Exponential time Algorithms
Exhaustive search / Branch and Bound Mathematical Programming Randomized Algorithms Simulated annealing Genetic algorithm Heuristic Tseng and Siewiorek Algorithm (minimum clique partition) Quine-McCluskey Algorithm (minimum set covering)

Algorithms

NTUEE

Combinatorial Optimization
Combinatorial Optimization
Find an optimal solution from a finite set of objects instances and solutions are discrete exhaustive search is often not feasible Continuous optimization Input instance and the solution set can be continuous

Examples of Combinatorial Optimization

minimum spanning tree problem (polynomial time) traveling salesman problem (NP-hard) Applications of Combinatorial Optimization Electronic Design Automation Control Operational research Artificial intelligence .
NTUEE

Algorithms

Decision Problems vs. Optimization Problems

Decision problems: find answers to yes/no questions
MST: Given a graph G=(V, E) and a bound K, is there a spanning tree with a cost at most K? TSP: Given cities, distance between each pair of cities, and a bound B, is there a route that starts and ends at a given city, visits every city exactly once, and has total distance at most B ? Optimization problems: find a legal answer to minimize cost MST: Given a graph G=(V, E), find the cost of a minimum spanning tree of G TSP: Given a set of cities and that distance between each pair of cities, find the distance of a minimum route starts and ends at a given city and visits every city exactly once

Algorithms

NTUEE

NP-complete and NP-Hard

A problem L is NPC if both conditions are true
L NP 2. L' P L for every L' NP A problem L is NP-hard if this condition is true 1. L' P L for every L' NP NP-hard problems are more difficult than NPC Example: TSP decision problem is NPC, optimization problem is NP-hard
1.

Algorithms

NTUEE

[Wikipedia]

Coping with NPC, NP-hard Problems

Approximation algorithms:

Guarantee to be a fixed percentage away from the optimum Exponential algorithms Branch and Bound/Exhaustive search (Feasible only when the problem size is small) Mathematical programming: Integer Linear programming , 0/1 ILP Local search: Simulated annealing (hill climbing), genetic algorithms, etc Randomized algorithms: Make use of a randomizer for operation Heuristics: No formal guarantee of performance
Others: Pseudo-polynomial time algorithms: e.g. Dynamic programming for 0-1 Knapsack problem Probabilistic algorithms: Assume some probabilistic distribution of the instances. Restriction: Work on some special cases of the original problem.

Algorithms

NTUEE

Outline
Introduction Exponential time Algorithms
Exhaustive search / Branch and Bound Mathematical Programming Randomized Algorithms Simulated annealing Genetic algorithm Heuristic

Algorithms

NTUEE

Exhaustive search
General principle
Systematically assign values to unspecified variables until a single point in search space is identified , or An implicit constraints makes it impossible to continue backtracking Example: TSP X means backtracking

Algorithms

NTUEE

Branch and Bound

General principle
Estimate the cost lower bound Kill partial solutions higher than the lowest cost Example: TSP Use MST as cost lower bound E.g. A B C F = 22; MST of {DEFA} = 9 22+9 > 27 Killed

Algorithms

NTUEE

Exhaustive Search vs. Branch and Bound

exhaustive: 72 nodes!

B&B: only 27 nodes

Algorithms NTUEE
10

10

Pseudo Code of B&B

Disadvantage:

Finding lower bound is not easy all the time Still too many branches when problem is large

Algorithms

NTUEE

11

11

Outline
Introduction Exponential time Algorithms
Exhaustive search / Branch and Bound Mathematical Programming Randomized Algorithms Simulated annealing Genetic algorithm Heuristic

Algorithms

NTUEE

12

Linear Programming
General principle
Convert problems into a mathematical LP problem Canonical form of LP AXb X0 Integer Linear Programming (ILP) Variables are restricted to integers 0-1 ILP Solutions are restricted to 0,1 Why ILP useful for optimization? Problem independent ILP solvers are widely available Runs fast in most cases (although worst case not polynomial)

Algorithms

NTUEE

13

13

ILP Example : TSP (optimization version)

Variables x1 x12
Xi = 1 means the edge is traveled Xi = 0 means the edge is not traveled Minimize cost of travel

Subject to constraints

w(e ) x
i i =1

12

Every vertex has exactly two edges traveled

v1 : x1 + x 2 + x 3 + x 4 = 2 v 2 : x1 + x 5 + x 6 + x 7 = 2 v 3 : x 2 + x 5 + x8 + x 9 = 2 v 4 : x 4 + x 7 + x10 + x11 = 2 v 5 : x 3 + x8 + x10 + x12 = 2 v 6 : x 6 + x 9 + x11 + x12 = 2

are these constraints enough?

NTUEE

Algorithms

14

14

TSP: Subtour Exclusion Constraints(1)

Multiple disjoint tour should be prevented

Add more constraints to avoid multiple disjoint tour

{v1, v 2, v 3}{v 4, v 5, v 6} : x 3 + x 4 + x 6 + x 7 + x 8 + x 9 2 {v1, v 3, v 5}{v 2, v 4, v 6} : x1 + x 4 + x 5 + x 9 + x10 + x12 2 {v1, v 2, v 4}{v 3, v 5, v 6} : x 2 + x 3 + x 5 + x 6 + x10 + x11 2
Algorithms NTUEE
15

15

TSP: Subtour Exclusion Constraints(2)

Type I: for a non-trivial subset S of vertex {1, 2, ... n} Type II:

Type I&II: number of constraints grows exponentially with n Type III: add aux variables ui = (2, 3, ...n)

every vertex i has one unique number ui

Number of constraints: n2
NTUEE

Algorithms

16

Outline
Introduction Exponential time Algorithms
Exhaustive search / Branch and Bound Mathematical Programming Randomized Algorithms Simulated annealing Genetic algorithm Heuristic Quine-McCluskey for SOP minimization

Algorithms

NTUEE

17

Local Search
General principle
Search only the neighbors of the current solution Move to the neighbor with lower cost than the current solution Problem Can be trapped in local minimum

Algorithms

NTUEE

18

Simulated Annealing (SA)

General Principle: mimic material cooling process
Search the neighbors of current solution A good move means the cost decreases always accepted A bad move means the cost increases Accepted with probability exp (-cost/T) Analogy Energy = cost function Temperature = controlling parameter T Advantage: can escape local minimum Uphill climbing is possible

Algorithms

NTUEE

19

Pseudo Code of SA

Algorithms

NTUEE

20

Tabu Search
General Principle: avoid staying in the taboo solutions

Keep a list of visited solutions : taboo list Always move to a new solution other than the taboo list even the new solution is poor than the current solution

Algorithms

NTUEE

21

Genetic Algorithms (GA)

General Principle

Survival of the fittest (s) Keep a group of feasible solutions population Parent population generates the child population Keep only the best children

Algorithms

NTUEE

22

Important steps in GA
Cross over: Two feasible solutions generate their child by switching
chromosomes

Mutation: some chromosomes can change by probability

Algorithms

NTUEE

23

Pseudo Code of GA

Algorithms

NTUEE

24

Outline
Introduction Exponential time Algorithms
Exhaustive search / Branch and Bound Mathematical Programming Randomized Algorithms Simulated annealing Genetic algorithm Heuristic Tseng and Siewiorek Algorithm (minimum clique partition) Quine-McCluskey Algorithm (minimum set covering)

* Heuristic refers to experience-based techniques for problem solving, where the exhaustive search is impractical. [Wikipedia]

Algorithms

NTUEE

25

Test Cube Compaction Problem

Problem: given a set of test cubes {1, 0, X} Output: a minimum set of compacted test cubes Two test cubes are compatible iff
no position is 0 in one cube and 1 in the other cube Compatible test cubes can be merged into one test cube merged test cubes has X elements only where both cubes are X Example: a set of 4 test cubes 0xx10 t0 t4+t5, t3+t6, t0+ t1+t2, t7 0xx1x t1 Any better solution?
t2 t3 t4 t5 t6 t7
Algorithms NTUEE

0x01x 01xx0 x0xx0 1xxxx x1x00 11xx0

26

Compatibility Graph
Compatibility graph G(V, E)
vertex vi represents a test cube ti edge eij between vi and vj means two test cubes are compatible test cubes in a clique can be merged into one test cube Example
t0 t1 t2 t3 t4 t5 t6 0xx10 0xx1x 0x01x 01xx0 x0xx0 1xxxx x1x00 11xx0

This is a minimum clique partition problem

t7

NP-hard

Algorithms

NTUEE

27

Tseng and Siewiorek Algorithm

GKC = compatibility graph in Kth iteration Vc = set of vertex in GKC Ec = set of edges in GKC

keep those edges that do not touch vi and vj

keep edges of common neighbors

Q: why remove other edges?

Algorithms NTUEE

28

T & S Example
combine vertices step by step select two vertices of maximum common neighbors supervertices is a set of vertices

e.g., combine v0 v1 v0,1 supervertices

t0,1,2,3 t4 t5, 6, 7
Algorithms NTUEE

01010 x0xx0 11x00

29

Outline
Introduction Exponential time Algorithms
Exhaustive search / Branch and Bound Mathematical Programming Randomized Algorithms Simulated annealing Genetic algorithm Heuristic Tseng and Siewiorek Algorithm (maximum clique ) Quine-McCluskey Algorithm (minimum set covering)

Algorithms

NTUEE

30

Set-Covering Problem
Given a finite set X, and a family F of subsets of X subset S F covers its elements
X=U S
SF

Set covering problem: find a minimum number of subset C F that

cover all X
X=U S
SC

Optimization problem
minimum size of C Decision problem exist a covering C size k? exercise: prove set covering is NPC (Hint: use vertex covering) Applications of Set-Covering find minimum of people (C) that perform all tasks (X) minimum 2-level combinational circuit design
NTUEE

Algorithms

31

Greedy-Set-Cover
Each iteration chooses the subset that covers most elements

running time O(|X||F| min(|X|,|F|))

example: |X|=12, |F|=6

greedy solution {S1, S4, S5, S6} size =4 optimal solution C={S3, S4, S5} size=3 (Corollary 35.5) GREEDY-SET-COVER is a polynomial time (ln|X|+1)approximation algorithm
NTUEE

Algorithms

32

SOP Minimization * not in exam

Two-level AND-OR Combinational Circuit Sum of product (SOP)
f = acd + bcd + bcd + acd Disjunctive normal form (DNF) f=(ac d)(bc d)(bc d)(ac d) Very important EDA problem: Given f, find minimize number of gates if gates are the same, minimize number of inputs

Algorithms

NTUEE

33

Quine-McCluskey Algorithm
Invented in 1956, one of the earliest EDA algorithms Two-step process

1. Generate all Prime Implicants (PI) PI = product term that cannot be combined with other terms 2. Select minimum number of PI
set-covering

Algorithms

NTUEE

34

Prime Implicant Generation (1/5)

f(abcd)=m(4,5,6,8,9,10,13)+d(0,7,15)

at least ONE of the following terms must be covered abcd + abcd + abcd + abcd + abcd + abcd + abcd called On-set abcd+abcd+abcd may or may not be covered ab a called Dont-care set cd 00 01 11 10 None of the others can be covered called Off-set 00 0 X 1 1
01

0 0 0

1 X 1 b

1 X 0

1 d 0 1

11

c
10

Algorithms

NTUEE

35

Prime Implicant Generation (2/5)

Implicant Table
Column I

zero 1 one 1

0000 0100 1000 0101

two 1

0110 1001 1010 0111 1101 1111

NTUEE

three 1 four 1
Algorithms

36

Prime Implicant Generation (3/5)

Implicant Table
Column I Column II

0000 | 0100 | 1000 | 0101 | 0110 | 1001 | 1010 | 0111 1101

0-00 -000 01001-0 10010-0 01-1 -101 0111-01 -111 11-1

NTUEE

| |

- are dont cares

1111 |
Algorithms

37

Prime Implicant Generation (4/5)

Implicant Table
Column I Column II Column III

0000 | 0100 | 1000 | 0101 | 0110 | 1001 | 1010 | 0111 | 1101 | 1111 |
Algorithms

0-00 * -000 *

01-- * -1-1 *

010- | 01-0 | 100- * 10-0 * 01-1 | -101 | 011- | 1-01 * -111 | 11-1 |
NTUEE

7 Prime Implicants: 0-00 = a'c'd' 100- = ab'c' 1-01 = ac'd -1-1 = bd -000 = b'c'd' 10-0 = ab'd' 01- - = a'b

38

Karnaugh Map
PI on Karnaugh map

see logic design textbook

ab cd 00 00 01 11

a
10

X 0 0 0

1 1 X 1 b

0 1 X 0

1 1 d 0 1

01

11

c
10

7 Prime Implicants : 0-00 = a'c'd' 100- = ab'c' 1-01 = ac'd -1-1 = bd -000 = b'c'd' 10-0 = ab'd' 01-- = a'b

Algorithms

NTUEE

39

PI Table
4 0,4 (0-00) 0,8 (-000) 8,9 (100-) 8,10 (10-0) 9,13 (1-01) 4,5,6,7 (01- -) 5,7,13,15 (-1-1) 5 6 8 9 10 13

rows = prime implicants columns = ON-set elements X" means ON-set element is covered by the prime implicant
Algorithms NTUEE

40

Essential Rows and Distinguished Column

4 0,4 (0-00) 0,8 (-000) 5 6 8 9 10 13

essential rows

8,9 (100-) 8,10 (10-0) 9,13 (1-01) 4,5,6,7 (01- -)

5,7,13,15 (-1-1)

If column has a single X, it is distinguished. The implicant associated with the row is essential. Essential rows must appear in the minimum cover .
Algorithms NTUEE

distinguished columns 41

Select Essential Rows

4 0,4 (0-00) 0,8 (-000) 8,9 (100-) 8,10 (10-0) 9,13 (1-01) 4,5,6,7 (01- -) 5,7,13,15 (-1-1) 5 6 8 9 10 13

Eliminate all columns covered by essential rows

Algorithms

NTUEE

42

Final Result
4 0,4 (0-00) 0,8 (-000) 8,9 (100-) 8,10 (10-0) 9,13 (1-01) 4,5,6,7 (01- -) 5,7,13,15 (-1-1) 5 6 8 9 10 13

Find minimum set of rows that cover the remaining columns f = ab'd' + ac'd + a'b
Algorithms NTUEE

43

FFT
Can Quine-McCluskey find optimal solution for all cases?
obviously not ... set-covering is NPC Give an example where PI table has neither essential rows nor distinguished columns Quine-McCluskey has trouble finding optimal solution

Algorithms

NTUEE

44