Lecture 2 - Combinatorial Optimization, Problem Complexity, When To Use Heuristics

University of Zagreb, Croatia
Lectures 2
Heuristic Optimization Methods:
Combinatorial optimization; complexity

theory; when and why to use heuristics
Slides prepared by Nina Skorin-Kapov
Academic year 2020/2021

Graduate Studies Programme
Combinatorial optimization
 Combinatorial optimization problems:

 Discrete decision variables
 Finite search space
 Objective function can be anything ((non)linear,
nonanalytic, …)
 Many real-world problems fall into this category
Department of Telecommunications 2
Examples of classical combinatorial
optimization problems
 Travelling salesman problem (TSP)

 Satisfiability problem (SAT)
 Knapsack problem
 Graph coloring (GC)
 Steiner tree problem (STP)
 Bin packing (BP)
Traveling Salesman Problem (TSP)
 Given n cities and their mutual distances, find

the shortest tour which visits each city once,
and only once (Hamiltonian cycle), and returns
to the original starting point.
 (n-1)! possible tours
 (n-1)!/2 possible tours if the orientation doesn’t matter
A 2 B
Some possible tours:
A-B-C-D, distance=9
3 4 1 3
A-B-D-C, distance=10
C D
2
Satisfiability problem (SAT)
 SAT: given a Boolean expression, can the

variables be assigned in such a way as to
make the formula evaluate to TRUE
 Decision problem
 Example SAT:
x1  ( x1  x2 )  ( x1  x3 )
 The Maximum Satisfiability problem (MAX-
SAT) is finding an assignment which gives the
maximum number of clauses of a given
Boolean expression that can be satisfied.
 Optimization problem
Knapsack problem
 Given are n objects and a knapsack of size C

units.
 Each object has value vi and size ci.
 Objective: Find the subset of objects which fits
in the knapsack and have the maximal total
value
v2=6 c2=5
C=9
v1=8 c1=9 v3=3 c3=3
v2=6 c2=5
v4=2 c4=1
v3=3 c3=3
v4=2 c4=1 Total value: v2+v3+v4=11
Total capacity: c2+c3+c4=9
Graph coloring (GC)
 Given a graph, color its nodes with the

minimum number of colors such that no two
adjacent nodes share the same color
5 2
colors = 3
4 3
Minimum Steiner Tree Problem
• Steiner tree: Given a graph G=(V,E) and a subset of

nodes D in V, a Steiner tree is a connected and acyclic
subgraph of G which spans all nodes in D
• A minimum Steiner tree is such a subgraph of minimum
weight in a weighted graph
(3) e
b
(4)
(3) (1) (3)
h
a d (2) f
(1) (2)
(4) (1) (2)
c
(4)
g
Bin packing (BP)
 Pack n objects of size ci , i=1,...,n into the

minimum number of identical bins of capacity C
 Geometric bin packing: items are of arbitrary shape
 Applications: packing commercials into breaks;
storage; packing boxes into trucks,...
...
Bin packing (BP)
 Pack n objects of size ci , i=1,...,n into the

minimum number of identical bins of capacity C
 Geometric bin packing: items are of arbitrary shape
 Applications: packing commercials into breaks;
storage; packing boxes into trucks,...
2 bins
needed
...
Combinatorial explosion
 A finite search space implies that an optimal

solution can surely be found through
exhaustive search (brute-force)
 Problem: combinatorial explosion!
TSP (Travelling Salesman Problem)

cities (n) Size of solution Relative time for
space (n!) brute force
20 20! ≈ 2.5*10^18 1 hour
21 21! ≈ 5.1*10^19 20 hours
22 22! ≈ 1.2*10^21 17.5 days
25 25! ≈ 1.5.*10^25 6 centuries
Complexity theory
 Algorithm complexity
 Problem complexity
Algorithm complexity
 Computational complexity: the number of steps

needed to solve a problem of size n
 Worst-case analysis; scalability issues
 Gives an asymptotic bound on the step count
 Does not specify the practical run-time for a given
instance
 Big O-notation
 An algorithm has complexity f(n)=O(g(n)) if there
exist postive constants n0 and c such that
n  n0 , f (n)  c  g (n)
 Big Ω-notation; Big Θ-notation
Polynomial time algorithm
 An algorithm is polynomial if its complexity is

O(p(n)), where p(n) is a polynomial function of n
p(n) = ak n k +  + a j n j +  + a1n + a0
O(nk)
ak  0; a j  0, 1  k  k − 1
 Example:
 Dijkstra’s shortest path algorithm
 O(n2)
 Floyd’s all-pairs shortest path algorithm

 O(n3)
Exponential time algorithm
 An algorithm is exponential if its complexity is

O(cn), where c is constant c  , c  1
 Example:
 Exhaustive search for SAT
 O(2n)
 Exhaustive search for TSP

 O(n!)
Polynomial vs. Exponential Growth
Complexity Problem size

n=10 n=20 n=30 n=40 n=50
Exponential Polynomial
O(n) 0.01 ms 0.02 ms 0.03 ms 0.04 ms 0.05 ms

growth
O(n2) 0.1 ms 0.4 ms 0.9 ms 1.6 ms 2.5 ms
O(n5) 0.1 s 3.2 s 24.3 s 1.7 min 5.2 min
O(2n) 0.001 s 1.0 s 17.9 min 12.7 days 35.7 yrs
O(3n) 2x108
growth
0.059 s 58.0 min 6.5 yrs 3855

centuries centuries
Values from: M Garey and D. Johnson, “Computers and Intractability: A Guide to the Theory of
NP-completeness”, W.H. Freeman and Co. Publishers, NY, 1979
Problem Complexity
 Problems are classified according to how

difficult they are to solve
 The complexity of a problem is equivalent to

the complexity of the best algorithm solving it
 Tractable (“easy”) problems – there exists a
polynomial time algorithm to solve it
 Intractable (“difficult”) problems –no polynomial time
algorithm is known which solves it
Decision problems
 Complexity problems deal with decision problems

- the solution is yes/no
 Every optimization problem can be reduced to a
decision problem
 TSP
 “Is there a Hamiltonian tour shorter than T?”
 Knapsack problem
 “Is
there a subset of objects whose total size is less than C
and value greater than V?”
 Bin Packing
 “Do the objects fit in less than B bins?”
 SAT
 “Isthere a combination of values that makes the logical
expression true?”
Complexity classes
 Class P: decision problems that can be solved

by a deterministic machine in polynomial time
(i.e., worst-case complexity is bounded by a
polynomial function)
 Example: MST, Shortest path problem, max flow
network
 Class NP: decision problems that can be
solved by a nondeterministic machine in
polynomial time
A proposed solution can be verified (checked) in
polynomial time
 Example: all P problems, TSP, MST, …
Complexity classes
 Class NP-complete: NP decision problem X is

NP-complete if all other problems in NP can be
reduced to X in polynomial time;
 Ifa polynomial time algorithm is found for an NP-
complete problem, it can solve all problems in NP,
then P=NP
 Is P = NP or P  NP ??? Open question
P
OR P = NP = NP-complete
NP NP -
complete
Complexity classes
 Class NP-hard: “at least as hard as NP-

complete problems”
 An algorithm for solving it can also solve NP-
complete problems but an algorithm which solves an
NP-complete problem does not necessarily solve it
 Can be any type: decision, search or optimization
problems
 Alloptimization problems whose associated decision
problems are NP-complete are NP-hard
P = NP =
NP - P
NP -hard
complete
OR NP-complete
NP NP -hard
NP-hard problems
 Most real-world optimization problems;

usually solved sub-optimally using heuristics
 Classical examples:
Flow-shop and job scheduling problems
Generalized assignment problem, location facility
Data clustering, graph partitioning, graph coloring
Vehicle routing problem, set covering, Steiner
tree problem
Knapsack problem, bin packing
Summary: Why are some problems
difficult to solve?
 The solution space is too big for exhaustive

search
 NP-complete/hard; combinatorial explosion
 The problem is too complicated to facilitate an
answer
 Must use simplified models
 The evaluation function is noisy or changes
with time
 It is so heavily constrained that even one
feasible solution is difficult to find
Optimization methods
Exact Approximate
A*
methods methods
Simplex-based Constraint
for LP programming Approximation
algorithms
Heuristic
Branch and Bound, Dynamic
Cut , Price (B&B, programming algorithms
B&C, B&P)
Improvement (meta)heuristics
Constructive heuristics
• Local search
• Greedy algorithms
•Nature inspired
• Tabu search, Genetic
Hybrid methods
algorithms, Simmulated
• GRASP Annealing, Ant colony
•Problem specific
heuristics
Exact methods:
 Simplex for LP problems – if variables are

continuous
 Branch and Bound and A*: based on implicit
enumeration of all solutions;
 ‘divide and conquer’ concept
 Dynamic programming: ‘bottom-up’ approach
 Recursive division of the problem
 Constraint programming: uses tree search and
logical implications
Approximate algorithms
Approximate algorithms can be divided into

approximation algorithms and heuristic
algorithms
 Approximation algorithms: provide provable

solution quality and provable run-time bounds
 Heuristic algorithms: do not guarantee

optimality of the solution or define how close it
is to the optimal; usually give ‘good’ solutions
quickly
Approximation algorithms
 ε-approximation algorithm: its time complexity is

polynomial and the obtained solution s is
s    s, if   1 where s* is the global
  s  s, if   1 optimum (minimization);
ε can be constant or a function of the input size
 Example: Bin packing
 First Fit Decreasing (FFD) – Θ(n log n)
 (11/9)s*+1
 First Fit
 (17/10)s*+2
Approximation algorithms
 Goal of designing approximation algorithms:

find tight worst-case bounds
 Gives knowledge on the difficulty of the
problem
 Problem dependent
 In practice, often attain approximations too far
from the global optimum and may thus not be
applicable
Heuristic algorithms
 Don’t guarantee how far we are from the

optimum
 How to evaluate the quality of the solutions ?
 Analytical lower/upper bounds
 Compare with opimal solutions for small
test cases
 Average case analysis
 Empirical testing
 Difficult to find a representative sample
 Hard to compare
Heuristic algorithms
Heuristics can be constructive or improvement (or

hybrid)
 Constructive: evaluate partial solutions;

terminate when a whole solution is constructed
 Ex: greedy algorithms
 Improvement: iterative; evaluate complete

solutions
 Metaheuristics – general ‘templates’ used to guide
the design of heuristics for specific problems
Metaheuristics
 ‘meta’ – Greek for ‘upper layer methodology’

 Term ‘metaheuristic’ – 1986, F. Glover
 Guiding strategies for designing heuristics
 Define how to explore the search space
 Can be applied to any optimization problem
 Often mimic natural phenomena

 evolution– Genetic algorithms (GA)
 memory – Tabu Search (TS)
 annealing – Simulated Annealing (SA)
 ants – Ant Colony Optimization (ACO)
 bees, wasps, immune system, …
When to use heuristics?
 Easy problems (class P) but with excessively

large instances
 Easy (or difficult) problems with real-time
constraints
 Difficult problems (NP-hard, NP-complete) of
moderate and large size
 Non-analytical models of optimization problems
 Problems with uncertainty or robustness issues
Which optimization method to use?
 Problem type:
 Design problems: long-term decision making,
financial investments
 Solution quality very important; Search time can be months
 Planning problems: medium-term decision making,
network planning
 Solution quality important; Search time can be hours/days
 Control problems: short-term decision making,
operational problem; online problems (routing)
 Search time critical; Solution as good as can be with limited
time
Trade off: solution quality vs. search time
Design
problems
Solution quality
Planning
problems NOTE:
Doesn’t
necessarily
Control grow linearly
problems
Search time
Which optimization method to apply?
 The trade-off between search time and solution

quality depends on the problem and the
method used but in general:
 Exact algorithms are usually used for smaller-sized
instances for planning and design (and for small
online problems)
 Greedy algorithms are usually fast and used for

online problems (and planning problems if they give
high quality solutions)
 Iterative improvement heuristics often require many

iterations and are best for planning problems (and
difficult design problems)
Create a new heuristic or use existing?
 If the problem can be reduced to a classical

NP-hard optimization problem, it may be easier
to apply the State-of-the-Art methods for that
problem or similar approaches
 Examples:
routing → Steiner tree problem
Multicast
Wavelength assignment → Graph coloring

Lecture 2 - Combinatorial Optimization, Problem Complexity, When To Use Heuristics

Uploaded by

Document Information

Original Title

Copyright

Available Formats

Share this document

Share or Embed Document

Sharing Options

Did you find this document useful?

Is this content inappropriate?

Copyright:

Available Formats

Lecture 2 - Combinatorial Optimization, Problem Complexity, When To Use Heuristics

Uploaded by

Copyright:

Available Formats

University of Zagreb, Croatia

Combinatorial optimization; complexity

Slides prepared by Nina Skorin-Kapov

Academic year 2020/2021

 Combinatorial optimization problems:

 Many real-world problems fall into this category

 Travelling salesman problem (TSP)

 Given n cities and their mutual distances, find

 SAT: given a Boolean expression, can the

 Given are n objects and a knapsack of size C

 Given a graph, color its nodes with the

• Steiner tree: Given a graph G=(V,E) and a subset of

 Pack n objects of size ci , i=1,...,n into the

 Pack n objects of size ci , i=1,...,n into the

 A finite search space implies that an optimal

TSP (Travelling Salesman Problem)

 Computational complexity: the number of steps

 An algorithm is polynomial if its complexity is

 Floyd’s all-pairs shortest path algorithm

 An algorithm is exponential if its complexity is

 Exhaustive search for TSP

Complexity Problem size

O(n) 0.01 ms 0.02 ms 0.03 ms 0.04 ms 0.05 ms

O(n2) 0.1 ms 0.4 ms 0.9 ms 1.6 ms 2.5 ms

O(n5) 0.1 s 3.2 s 24.3 s 1.7 min 5.2 min

O(2n) 0.001 s 1.0 s 17.9 min 12.7 days 35.7 yrs

0.059 s 58.0 min 6.5 yrs 3855

 Problems are classified according to how

 The complexity of a problem is equivalent to

 Complexity problems deal with decision problems

 Class P: decision problems that can be solved

 Class NP-complete: NP decision problem X is

 Class NP-hard: “at least as hard as NP-

 Most real-world optimization problems;

 The solution space is too big for exhaustive

 Simplex for LP problems – if variables are

Approximate algorithms can be divided into

 Approximation algorithms: provide provable

 Heuristic algorithms: do not guarantee

 ε-approximation algorithm: its time complexity is

 Goal of designing approximation algorithms:

 Don’t guarantee how far we are from the

Heuristics can be constructive or improvement (or

 Constructive: evaluate partial solutions;

 Improvement: iterative; evaluate complete

 ‘meta’ – Greek for ‘upper layer methodology’

 Often mimic natural phenomena

 Easy problems (class P) but with excessively

 The trade-off between search time and solution

 Greedy algorithms are usually fast and used for

 Iterative improvement heuristics often require many

 If the problem can be reduced to a classical

You might also like