MAP Estimation Algorithms in

Computer Vision - Part II

M. Pawan Kumar, University of Oxford

Pushmeet Kohli, Microsoft Research

Example: Image Segmentation
E: {0,1}n → R
E(x) = ∑ ci xi + ∑ cij xi(1-xj) 0 → fg
1 → bg
i i,j
n = number of pixels

Image (D)
 Example: Image Segmentation
E: {0,1}n → R
E(x) = ∑ ci xi + ∑ cij xi(1-xj) 0 → fg
1 → bg
i i,j
n = number of pixels

Unary Cost
(ci)
Dark (negative) Bright (positive)
Example: Image Segmentation
E: {0,1}n → R
E(x) = ∑ ci xi + ∑ cij xi(1-xj) 0 → fg
1 → bg
i i,j
n = number of pixels

Discontinuity
Cost (cij)
 Example: Image Segmentation
E: {0,1}n → R
E(x) = ∑ ci xi + ∑ cij xi(1-xj) 0 → fg
1 → bg
i i,j
n = number of pixels

x* = arg min E(x)

x

How to minimize
E(x)?
Global Minimum (x*)
Outline of the Tutorial

The st-mincut problem

Connection between st-mincut

and energy minimization?

What problems can we solve

using st-mincut?

st-mincut based Move algorithms

Recent Advances and Open Problems

Outline of the Tutorial

The st-mincut problem

Connection between st-mincut

and energy minimization?

What problems can we solve

using st-mincut?

st-mincut based Move algorithms

Recent Advances and Open Problems

 The st-Mincut Problem

Source

2 9 Graph (V, E, C)
1 Vertices V = {v1, v2 ... vn}
v1 v2 Edges E = {(v1, v2) ....}

2 Costs C = {c(1, 2) ....}

5 4

Sink
 The st-Mincut Problem
What is a st-cut?

Source

2 9
1
v1 v2
2
5 4

Sink
 The st-Mincut Problem
What is a st-cut?
An st-cut (S,T) divides the nodes
between source and sink.
Source

2 9 What is the cost of a st-cut?

1 Sum of cost of all edges
v1 v2 going from S to T
2
5 4

Sink

5 + 2 + 9 = 16
 The st-Mincut Problem
What is a st-cut?
An st-cut (S,T) divides the nodes
between source and sink.
Source

2 9 What is the cost of a st-cut?

1 Sum of cost of all edges
v1 v2 going from S to T
2
5 4
What is the st-mincut?
Sink st-cut with the
minimum cost

2 + 1 + 4 = 7
 How to compute the st-mincut?

Solve the dual maximum flow problem

Compute the maximum flow between
Source Source and Sink
Constraints
2 9
Edges: Flow < Capacity
1
v1 v2 Nodes: Flow in = Flow out
2
5 4
Min-cut\Max-flow Theorem
Sink
In every network, the maximum flow
equals the cost of the st-mincut
 Maxflow Algorithms

Flow = 0
Augmenting Path Based
Algorithms
Source

2 9 1. Find path from source to sink

with positive capacity
1
v1 v2
2. Push maximum possible flow
2 through this path
5 4

3. Repeat until no path can be

Sink
found

Algorithms assume non-negative capacity

 Maxflow Algorithms

Flow = 0
Augmenting Path Based
Algorithms
Source

2 9 1. Find path from source to sink

with positive capacity
1
v1 v2
2. Push maximum possible flow
2 through this path
5 4

3. Repeat until no path can be

Sink
found

Algorithms assume non-negative capacity

 Maxflow Algorithms

Flow = 0 + 2
Augmenting Path Based
Algorithms
Source

2-2 9 1. Find path from source to sink

with positive capacity
1
v1 v2
2. Push maximum possible flow
2 through this path
4
5-2
3. Repeat until no path can be
Sink
found

Algorithms assume non-negative capacity

 Maxflow Algorithms

Flow = 2
Augmenting Path Based
Algorithms
Source

0 9 1. Find path from source to sink

with positive capacity
1
v1 v2
2. Push maximum possible flow
2 through this path
3 4

3. Repeat until no path can be

Sink
found

Algorithms assume non-negative capacity

 Maxflow Algorithms

Flow = 2
Augmenting Path Based
Algorithms
Source

0 9 1. Find path from source to sink

with positive capacity
1
v1 v2
2. Push maximum possible flow
2 through this path
3 4

3. Repeat until no path can be

Sink
found

Algorithms assume non-negative capacity

 Maxflow Algorithms

Flow = 2
Augmenting Path Based
Algorithms
Source

0 9 1. Find path from source to sink

with positive capacity
1
v1 v2
2. Push maximum possible flow
2 through this path
3 4

3. Repeat until no path can be

Sink
found

Algorithms assume non-negative capacity

 Maxflow Algorithms

Flow = 2 + 4
Augmenting Path Based
Algorithms
Source

0 5 1. Find path from source to sink

with positive capacity
1
v1 v2
2. Push maximum possible flow
2 through this path
3 0

3. Repeat until no path can be

Sink
found

Algorithms assume non-negative capacity

 Maxflow Algorithms

Flow = 6
Augmenting Path Based
Algorithms
Source

0 5 1. Find path from source to sink

with positive capacity
1
v1 v2
2. Push maximum possible flow
2 through this path
3 0

3. Repeat until no path can be

Sink
found

Algorithms assume non-negative capacity

 Maxflow Algorithms

Flow = 6
Augmenting Path Based
Algorithms
Source

0 5 1. Find path from source to sink

with positive capacity
1
v1 v2
2. Push maximum possible flow
2 through this path
3 0

3. Repeat until no path can be

Sink
found

Algorithms assume non-negative capacity

 Maxflow Algorithms

Flow = 6 + 1
Augmenting Path Based
Algorithms
Source

0 4 1. Find path from source to sink

with positive capacity
1-1
v1 v2
2. Push maximum possible flow
2+1 through this path
2 0

3. Repeat until no path can be

Sink
found

Algorithms assume non-negative capacity

 Maxflow Algorithms

Flow = 7
Augmenting Path Based
Algorithms
Source

0 4 1. Find path from source to sink

with positive capacity
0
v1 v2
2. Push maximum possible flow
3 through this path
2 0

3. Repeat until no path can be

Sink
found

Algorithms assume non-negative capacity

 Maxflow Algorithms

Flow = 7
Augmenting Path Based
Algorithms
Source

0 4 1. Find path from source to sink

with positive capacity
0
v1 v2
2. Push maximum possible flow
3 through this path
2 0

3. Repeat until no path can be

Sink
found

Algorithms assume non-negative capacity

History of Maxflow Algorithms
n: #nodes
Augmenting Path and Push-Relabel
m: #edges
U: maximum
edge weight

Algorithms
assume non-
negative edge
weights

[Slide credit: Andrew Goldberg]

History of Maxflow Algorithms
n: #nodes
Augmenting Path and Push-Relabel
m: #edges
U: maximum
edge weight

Algorithms
assume non-
negative edge
weights

[Slide credit: Andrew Goldberg]

Augmenting Path based Algorithms

Ford Fulkerson: Choose any augmenting path

Source

1000 1000
1
a1 a2

0
1000 1000
Sink
Augmenting Path based Algorithms

Ford Fulkerson: Choose any augmenting path

Source

1000 1000
1
a1 a2

0 Bad
1000 1000 Augmenting
Paths
Sink
Augmenting Path based Algorithms

Ford Fulkerson: Choose any augmenting path

Source

1000 1000
1
a1 a2

0 Bad
1000 1000 Augmenting
Path
Sink
Augmenting Path based Algorithms

Ford Fulkerson: Choose any augmenting path

Source

999 1000
0
a1 a2

1
1000 999
Sink
Augmenting Path based Algorithms
n: #nodes

Ford Fulkerson: Choose any augmenting path m: #edges

Source

999 1000
0
a1 a2

1
1000 999
Sink

We will have to perform 2000 augmentations!

Worst case complexity: O (m x Total_Flow)
(Pseudo-polynomial bound: depends on flow)
Augmenting Path based Algorithms
n: #nodes

Dinic: Choose shortest augmenting path m: #edges

Source

1000 1000
1
a1 a2

0
1000 1000
Sink

Worst case Complexity: O (m n2)

 Maxflow in Computer Vision

• Specialized algorithms for vision

problems
– Grid graphs
– Low connectivity (m ~ O(n))

• Dual search tree augmenting path

algorithm
[Boykov and Kolmogorov PAMI 2004]
• Finds approximate shortest augmenting
paths efficiently
• High worst-case time complexity
• Empirically outperforms other
algorithms on vision problems
 Maxflow in Computer Vision

• Specialized algorithms for vision

problems
– Grid graphs
– Low connectivity (m ~ O(n))

• Dual search tree augmenting path

algorithm
[Boykov and Kolmogorov PAMI 2004]
• Finds approximate shortest augmenting
paths efficiently
• High worst-case time complexity
• Empirically outperforms other
algorithms on vision problems
• Efficient code available on the web
http://www.adastral.ucl.ac.uk/~vladkolm/software.html
Outline of the Tutorial

The st-mincut problem

Connection between st-mincut

and energy minimization?

What problems can we solve

using st-mincut?

st-mincut based Move algorithms

Recent Advances and Open Problems

St-mincut and Energy Minimization
S st-mincut
Minimizing a Qudratic
Pseudoboolean
function E(x)
T

Functions of boolean variables Pseudoboolean?

E: {0,1}n → R

E(x) = ∑ ci xi + ∑ cij xi(1-xj) cij≥0

i i,j

Polynomial time st-mincut algorithms

require non-negative edge weights
 So how does this work?

Construct a graph such that:

1. Any st-cut corresponds to an assignment of x
2. The cost of the cut is equal to the energy of
x : E(x)

S st-mincut

E(x)

T
Solution
 Graph Construction

E(a1,a2)

Source (0)

a1 a2

Sink (1)
 Graph Construction

E(a1,a2) = 2a1

Source (0)

a1 a2

Sink (1)
 Graph Construction

E(a1,a2) = 2a1 + 5ā1

Source (0)

a1 a2

5
Sink (1)
 Graph Construction

E(a1,a2) = 2a1 + 5ā1+ 9a2 + 4ā2

Source (0)

2 9

a1 a2

5 4
Sink (1)
 Graph Construction

E(a1,a2) = 2a1 + 5ā1+ 9a2 + 4ā2 + 2a1ā2

Source (0)

2 9

a1 a2

2
5 4
Sink (1)
 Graph Construction

E(a1,a2) = 2a1 + 5ā1+ 9a2 + 4ā2 + 2a1ā2 + ā1a2

Source (0)

2 9
1
a1 a2

2
5 4
Sink (1)
 Graph Construction

E(a1,a2) = 2a1 + 5ā1+ 9a2 + 4ā2 + 2a1ā2 + ā1a2

Source (0)

2 9
1
a1 a2

2
5 4
Sink (1)
 Graph Construction

E(a1,a2) = 2a1 + 5ā1+ 9a2 + 4ā2 + 2a1ā2 + ā1a2

Source (0)

2 9 Cost of cut = 11

1
a1 a2 a1 = 1 a2 = 1

2
E (1,1) = 11
5 4
Sink (1)
 Graph Construction

E(a1,a2) = 2a1 + 5ā1+ 9a2 + 4ā2 + 2a1ā2 + ā1a2

Source (0)

2 9 st-mincut cost = 8

1
a1 a2 a1 = 1 a2 = 0

2
E (1,0) = 8
5 4
Sink (1)
Energy Function Reparameterization

Two functions E1 and E2 are reparameterizations if

E1 (x) = E2 (x) for all x

For instance:
a1 ā1 1+ 2a1 + 3ā1 3 + ā1
E1 (a1) = 1+ 2a1 + 3ā1
0 1 4 4
E2 (a1) = 3 + ā1
1 0 3 3
 Flow and Reparametrization

E(a1,a2) = 2a1 + 5ā1+ 9a2 + 4ā2 + 2a1ā2 + ā1a2

Source (0)

2 9
1
a1 a2

2
5 4
Sink (1)
 Flow and Reparametrization

E(a1,a2) = 2a1 + 5ā1+ 9a2 + 4ā2 + 2a1ā2 + ā1a2

Source (0)

2 9
2a1 + 5ā1
1
a1 a2
= 2(a1+ā1) + 3ā1
2
= 2 + 3ā1
5 4
Sink (1)
 Flow and Reparametrization

E(a1,a2) = 2 + 3ā1+ 9a2 + 4ā2 + 2a1ā2 + ā1a2

Source (0)

0 9
2a1 + 5ā1
1
a1 a2
= 2(a1+ā1) + 3ā1
2
= 2 + 3ā1
3 4
Sink (1)
 Flow and Reparametrization

E(a1,a2) = 2 + 3ā1+ 9a2 + 4ā2 + 2a1ā2 + ā1a2

Source (1)

0 9
9a2 + 4ā2
1
a1 a2
= 4(a2+ā2) + 5ā2
2
= 4 + 5ā2
3 4
Sink (0)
 Flow and Reparametrization

E(a1,a2) = 2 + 3ā1+ 5a2 + 4 + 2a1ā2 + ā1a2

Source (1)

0 5
9a2 + 4ā2
1
a1 a2
= 4(a2+ā2) + 5ā2
2
= 4 + 5ā2
3 0
Sink (0)
 Flow and Reparametrization

E(a1,a2) = 6 + 3ā1+ 5a2 + 2a1ā2 + ā1a2

Source (1)

0 5
1
a1 a2

2
3 0
Sink (0)
 Flow and Reparametrization

E(a1,a2) = 6 + 3ā1+ 5a2 + 2a1ā2 + ā1a2

Source (1)

0 5
1
a1 a2

2
3 0
Sink (0)
 Flow and Reparametrization

E(a1,a2) = 6 + 3ā1+ 5a2 + 2a1ā2 + ā1a2

3ā1+ 5a2 + 2a1ā2

= 2(ā1+a2+a1ā2) +ā1+3a2
Source (1)
= 2(1+ā1a2) +ā1+3a2
0 5
F1 = ā1+a2+a1ā2
1
a1 a2 F2 = 1+ā1a2

2 a1 a2 F1 F2
3 0 0 0 1 1
0 1 2 2
Sink (0) 1 0 1 1
1 1 1 1
 Flow and Reparametrization

E(a1,a2) = 8 + ā1+ 3a2 + 3ā1a2

3ā1+ 5a2 + 2a1ā2

= 2(ā1+a2+a1ā2) +ā1+3a2
Source (1)
= 2(1+ā1a2) +ā1+3a2
0 3
F1 = ā1+a2+a1ā2
3
a1 a2 F2 = 1+ā1a2

0 a1 a2 F1 F2
1 0 0 0 1 1
0 1 2 2
Sink (0) 1 0 1 1
1 1 1 1
 Flow and Reparametrization

E(a1,a2) = 8 + ā1+ 3a2 + 3ā1a2

Source (1)

0 3
3 No more
a1 a2
augmenting paths
0 possible
1 0
Sink (0)
 Flow and Reparametrization

E(a1,a2) = 8 + ā1+ 3a2 + 3ā1a2 Residual Graph

(positive coefficients)

Source (1)
Total Flow
bound on the 0 3
optimal solution
3
a1 a2

0
1 0
Sink (0)
Inference of the optimal solution becomes
trivial because the bound is tight
 Flow and Reparametrization

E(a1,a2) = 8 + ā1+ 3a2 + 3ā1a2 Residual Graph

(positive coefficients)

Source (1)
Total Flow
bound on the 0 3 st-mincut cost = 8
optimal solution
3
a1 a2 a1 = 1 a2 = 0

0
E (1,0) = 8
1 0
Sink (0)
Inference of the optimal solution becomes
trivial because the bound is tight
 Example: Image Segmentation
E: {0,1}n → R
E(x) = ∑ ci xi + ∑ cij xi(1-xj) 0 → fg
1 → bg
i i,j

x* = arg min E(x)

x

How to minimize
E(x)?
Global Minimum (x*)
 How does the code look like?

Graph *g;

For all pixels p

Source (0)
/* Add a node to the graph */
nodeID(p) = g->add_node();

/* Set cost of terminal edges */

set_weights(nodeID(p), fgCost(p), bgCost(p));

end

for all adjacent pixels p,q

add_weights(nodeID(p), nodeID(q), cost);
end

g->compute_maxflow();

label_p = g->is_connected_to_source(nodeID(p)); Sink (1)

// is the label of pixel p (0 or 1)
 How does the code look like?

Graph *g;

For all pixels p

Source (0)
/* Add a node to the graph */
nodeID(p) = g->add_node();

/* Set cost of terminal edges */ bgCost(a1) bgCost(a2)

set_weights(nodeID(p), fgCost(p), bgCost(p));

end
a1 a2
for all adjacent pixels p,q
add_weights(nodeID(p), nodeID(q), cost);
end
fgCost(a1) fgCost(a2)
g->compute_maxflow();

label_p = g->is_connected_to_source(nodeID(p)); Sink (1)

// is the label of pixel p (0 or 1)
 How does the code look like?

Graph *g;

For all pixels p

Source (0)
/* Add a node to the graph */
nodeID(p) = g->add_node();

/* Set cost of terminal edges */ bgCost(a1) bgCost(a2)

set_weights(nodeID(p), fgCost(p), bgCost(p));
cost(p,q)
end
a1 a2
for all adjacent pixels p,q
add_weights(nodeID(p), nodeID(q), cost(p,q));
end
fgCost(a1) fgCost(a2)
g->compute_maxflow();

label_p = g->is_connected_to_source(nodeID(p)); Sink (1)

// is the label of pixel p (0 or 1)
 How does the code look like?

Graph *g;

For all pixels p

Source (0)
/* Add a node to the graph */
nodeID(p) = g->add_node();

/* Set cost of terminal edges */ bgCost(a1) bgCost(a2)

set_weights(nodeID(p), fgCost(p), bgCost(p));
cost(p,q)
end
a1 a2
for all adjacent pixels p,q
add_weights(nodeID(p), nodeID(q), cost(p,q));
end
fgCost(a1) fgCost(a2)
g->compute_maxflow();

label_p = g->is_connected_to_source(nodeID(p)); Sink (1)

// is the label of pixel p (0 or 1)
a1 = bg a2 = fg
 Image Segmentation in Video
s = 0 st-cut
n-links

E(x) w pq
x*

t =1

Image Flow Global Optimum

 Image Segmentation in Video

Image

Global
Flow Optimum
 Dynamic Energy Minimization
minimize
EA SA

Can we do Recycling
better? Solutions

EB SB
computationally
expensive operation

Boykov & Jolly ICCV’01, Kohli & Torr (ICCV05, PAMI07)

 Dynamic Energy Minimization
minimize
EA SA
Reuse flow

differences
A and B between EB* Simpler
energy
similar A and B
cheaper
operation

EB SB
computationally 3 – 100000
Reparametrization
expensive operation time speedup!

Kohli& &Torr
Boykov & Jolly ICCV’01, Kohli Torr(ICCV05,
(ICCV05,PAMI07)
PAMI07)
 Dynamic Energy Minimization
Original Energy
E(a1,a2) = 2a1 + 5ā1+ 9a2 + 4ā2 + 2a1ā2 + ā1a2
Reparametrized Energy
E(a1,a2) = 8 + ā1+ 3a2 + 3ā1a2

New Energy
E(a1,a2) = 2a1 + 5ā1+ 9a2 + 4ā2 + 7a1ā2 + ā1a2
New Reparametrized Energy
E(a1,a2) = 8 + ā1+ 3a2 + 3ā1a2 + 5a1ā2

Kohli& &Torr
Boykov & Jolly ICCV’01, Kohli Torr(ICCV05,
(ICCV05,PAMI07)
PAMI07)
Outline of the Tutorial

The st-mincut problem

Connection between st-mincut

and energy minimization?

What problems can we solve

using st-mincut?

st-mincut based Move algorithms

Recent Advances and Open Problems

 Minimizing Energy Functions

• General Energy Functions

– NP-hard to minimize MAXCUT NP-Hard
– Only approximate minimization
possible Submodular
Functions

• Easy energy functions

– Solvable in polynomial time
– Submodular ~ O(n6)
Functions defined on trees

Space of Function
Minimization Problems
 Submodular Set Functions
Let E= {a1,a2, .... an} be a set
2|E| = #subsets of E

Set function f  2|E|  ℝ

 Submodular Set Functions
Let E= {a1,a2, .... an} be a set
2|E| = #subsets of E

Set function f  2|E|  ℝ is submodular if

f(A) + f(B)  f(AB) + f(AB) for all A,B  E

E
A B

Important Property
Sum of two submodular functions is submodular
 Minimizing Submodular Functions

• Minimizing general submodular functions

– O(n5 Q + n6) where Q is function evaluation time
[Orlin, IPCO 2007]

• Symmetric submodular functions

– E (x) = E (1 - x)
– O(n3) [Queyranne 1998]

• Quadratic pseudoboolean
– Can be transformed to st-mincut
– One node per variable [ O(n3) complexity]
– Very low empirical running time
 Submodular Pseudoboolean Functions
Function defined over boolean vectors x = {x1,x2, .... xn}

Definition:
• All functions for one boolean variable (f: {0,1} -> R) are submodular

• A function of two boolean variables (f: {0,1}2 -> R) is submodular if

f(0,1) + f(1,0)  f(0,0) + f(1,1)

• A general pseudoboolean function f  2n  ℝ is submodular if all

its projections fp are submodular i.e.

fp(0,1) + fp (1,0)  fp (0,0) + fp (1,1)

Quadratic Submodular Pseudoboolean
Functions

E(x) = ∑ θi (xi) + ∑ θij (xi,xj)

i i,j

For all ij θij(0,1) + θij (1,0)  θij (0,0) + θij (1,1)

Quadratic Submodular Pseudoboolean
Functions

E(x) = ∑ θi (xi) + ∑ θij (xi,xj)

i i,j

For all ij θij(0,1) + θij (1,0)  θij (0,0) + θij (1,1)

Equivalent (transformable)

E(x) = ∑ ci xi + ∑ cij xi(1-xj) cij≥0

i i,j

i.e. All submodular QPBFs are st-mincut solvable

 How are they equivalent?
A = θij (0,0) B = θij(0,1) C = θij (1,0) D = θij (1,1)

xj
0 1 0 1 0 1 0 1
B+C
0 A B 0 0 0 0 0 D-C 0 0
-A-D
xi = A + + +
1 C D 1 C-A C-A 1 0 D-C 1 0 0

if x1=1 add if x2 = 1 add

C-A D-C

θij (xi,xj) = θij(0,0)

+ (θij(1,0)-θij(0,0)) xi + (θij(1,0)-θij(0,0)) xj
+ (θij(1,0) + θij(0,1) - θij(0,0) - θij(1,1)) (1-xi) xj

B+C-A-D  0 is true from the submodularity of θij

 How are they equivalent?
A = θij (0,0) B = θij(0,1) C = θij (1,0) D = θij (1,1)

xj
0 1 0 1 0 1 0 1
B+C
0 A B 0 0 0 0 0 D-C 0 0
-A-D
xi = A + + +
1 C D 1 C-A C-A 1 0 D-C 1 0 0

if x1=1 add if x2 = 1 add

C-A D-C

θij (xi,xj) = θij(0,0)

+ (θij(1,0)-θij(0,0)) xi + (θij(1,0)-θij(0,0)) xj
+ (θij(1,0) + θij(0,1) - θij(0,0) - θij(1,1)) (1-xi) xj

B+C-A-D  0 is true from the submodularity of θij

 How are they equivalent?
A = θij (0,0) B = θij(0,1) C = θij (1,0) D = θij (1,1)

xj
0 1 0 1 0 1 0 1
B+C
0 A B 0 0 0 0 0 D-C 0 0
-A-D
xi = A + + +
1 C D 1 C-A C-A 1 0 D-C 1 0 0

if x1=1 add if x2 = 1 add

C-A D-C

θij (xi,xj) = θij(0,0)

+ (θij(1,0)-θij(0,0)) xi + (θij(1,0)-θij(0,0)) xj
+ (θij(1,0) + θij(0,1) - θij(0,0) - θij(1,1)) (1-xi) xj

B+C-A-D  0 is true from the submodularity of θij

 How are they equivalent?
A = θij (0,0) B = θij(0,1) C = θij (1,0) D = θij (1,1)

xj
0 1 0 1 0 1 0 1
B+C
0 A B 0 0 0 0 0 D-C 0 0
-A-D
xi = A + + +
1 C D 1 C-A C-A 1 0 D-C 1 0 0

if x1=1 add if x2 = 1 add

C-A D-C

θij (xi,xj) = θij(0,0)

+ (θij(1,0)-θij(0,0)) xi + (θij(1,0)-θij(0,0)) xj
+ (θij(1,0) + θij(0,1) - θij(0,0) - θij(1,1)) (1-xi) xj

B+C-A-D  0 is true from the submodularity of θij

 How are they equivalent?
A = θij (0,0) B = θij(0,1) C = θij (1,0) D = θij (1,1)

xj
0 1 0 1 0 1 0 1
B+C
0 A B 0 0 0 0 0 D-C 0 0
-A-D
xi = A + + +
1 C D 1 C-A C-A 1 0 D-C 1 0 0

if x1=1 add if x2 = 1 add

C-A D-C

θij (xi,xj) = θij(0,0)

+ (θij(1,0)-θij(0,0)) xi + (θij(1,0)-θij(0,0)) xj
+ (θij(1,0) + θij(0,1) - θij(0,0) - θij(1,1)) (1-xi) xj

B+C-A-D  0 is true from the submodularity of θij

Quadratic Submodular Pseudoboolean
Functions
x in {0,1}n

E(x) = ∑ θi (xi) + ∑ θij (xi,xj)

i i,j

For all ij θij(0,1) + θij (1,0)  θij (0,0) + θij (1,1)

Equivalent (transformable)

st-mincut
S

T
 Minimizing Non-Submodular Functions

E(x) = ∑ θi (xi) + ∑ θij (xi,xj)

i i,j

θij(0,1) + θij (1,0) ≤θ ij (0,0) + θij (1,1) for some ij

• Minimizing general non-submodular functions is

NP-hard.

• Commonly used method is to solve a relaxation

of the problem

[Slide credit: Carsten Rother]

Minimization using Roof-dual Relaxation

unary
E ({ x p })    p ( x p )

 pq (0,0)   pq (1,1)   pq (0,1)   pq (1,0)

   pq ( x p , xq )
pairwise submodular
~
   pq ( x p , xq ) ~ ~ ~ ~
 pq (0,0)   pq (1,1)   pq (0,1)   pq (1,0)
pairwise nonsubmodular

[Slide credit: Carsten Rother]

Minimization using Roof-dual Relaxation
Double number of variables: x p  x p , x p ( x p  1  x p )

E ({ x p })    p ( x p ) E ' ({ x p },{x p })  
 p ( x p )   p (1  x p )
2

   pq ( x p , xq )  pq ( x p , xq )   pq (1  x p ,1  xq )

2
~ ~
~  pq ( x p ,1  xq )   pq (1  x p , xq )
   pq ( x p , xq ) 
2

[Slide credit: Carsten Rother]

 Minimization using Roof-dual Relaxation
Double number of variables: x p  x p , x p ( x p  1  x p )

E ({ x p })    p ( x p ) E ' ({ x p },{x p })  
 p ( x p )   p (1  x p )
2

   pq ( x p , xq )  pq ( x p , xq )   pq (1  x p ,1  xq )

2
~ ~
~  pq ( x p ,1  xq )   pq (1  x p , xq )
   pq ( x p , xq ) 
2

~
f pq ( x p , xq )   pq ( x p ,1  xq )

~ ~ ~ ~
 pq (0,0)   pq (1,1)   pq (0,1)   pq (1,0) f pq (0,1)  f pq (1,0)  f pq (0,0)  f pq (1,1)
Non- submodular Submodular
Minimization using Roof-dual Relaxation
Double number of variables: x p  x p , x p ( x p  1  x p )

E ({ x p })    p ( x p ) E ' ({ x p },{x p })  
 p ( x p )   p (1  x p )
2

   pq ( x p , xq )  pq ( x p , xq )   pq (1  x p ,1  xq )

2
~ ~
~  pq ( x p ,1  xq )   pq (1  x p , xq )
   pq ( x p , xq ) 
2

Property of the problem:

•
• is submodular ! Ignore x p  1  x p
(solvable using st-mincut)
Minimization using Roof-dual Relaxation
Double number of variables: x p  x p , x p ( x p  1  x p )

E ({ x p })    p ( x p ) E ' ({ x p },{x p })  
 p ( x p )   p (1  x p )
2

   pq ( x p , xq )  pq ( x p , xq )   pq (1  x p ,1  xq )

2
~ ~
~  pq ( x p ,1  xq )   pq (1  x p , xq )
   pq ( x p , xq ) 
2

Property of the solution:

xp  1 xp x p is the optimal label

 Recap

• Exact minimization of Submodular QBFs

using graph cuts.

• Obtaining partially optimal solutions of non-

submodular QBFs using graph cuts.
 But ...
• Need higher order energy functions to
model image structure
– Field of experts [Roth and Black]
• Many problems in computer vision involve
multiple labels

E(x) = ∑ θi (xi) + ∑ θij (xi,xj) + ∑ θc (xc)

i i,j c

x ϵ Labels L = {l1, l2, … , lk} Clique c  V

Transforming problems in QBFs

Higher order Quadratic

Pseudoboolean Pseudoboolean
Functions Functions

Multi-label Pseudoboolean
Functions Functions
Transforming problems in QBFs

Higher order Quadratic

Pseudoboolean Pseudoboolean
Functions Functions

Multi-label Pseudoboolean
Functions Functions
 Higher order to Quadratic
• Simple Example using Auxiliary variables

f(x) = { 0 if all xi = 0
C1 otherwise
x ϵ L = {0,1}n

min f(x) = min C1a + C1 ∑ ā xi

x x,a ϵ {0,1}
Higher Order Quadratic Submodular
Submodular Function Function

∑xi = 0 a=0 (ā=1) f(x) = 0

∑xi ≥ 1 a=1 (ā=0) f(x) = C1
 Higher order to Quadratic

min f(x) = min C1a + C1 ∑ ā xi

x x,a ϵ {0,1}
Higher Order Quadratic Submodular
Submodular Function Function

C1∑xi

C1

1 2 3
∑xi
 Higher order to Quadratic

min f(x) = min C1a + C1 ∑ ā xi

x x,a ϵ {0,1}
Higher Order Quadratic Submodular
Submodular Function Function

C1∑xi

a=0 a=1
Lower envelop
of concave
C1 functions is
concave

1 2 3
∑xi
 Higher order to Quadratic

min f(x)
x
= min f1 (x)a + f2(x)ā
x,a ϵ {0,1}
Higher Order Quadratic Submodular
Submodular Function Function

f2(x)
a=1
f1(x) Lower envelop
of concave
functions is
concave

1 2 3
∑xi
 Higher order to Quadratic

min f(x)
x
= min f1 (x)a + f2(x)ā
x,a ϵ {0,1}
Higher Order Quadratic Submodular
Submodular Function Function

f2(x)

a=0 a=1
f1(x) Lower envelop
of concave
functions is
concave

1 2 3
∑xi
Transforming problems in QBFs

Higher order Quadratic

Pseudoboolean Pseudoboolean
Functions Functions

Multi-label Pseudoboolean
Functions Functions
 Multi-label to Pseudo-boolean
So what is the problem?
Em (y1,y2, ..., yn) Eb (x1,x2, ..., xm)

yi ϵ L = {l1, l2, … , lk} xi ϵ L = {0,1}

Multi-label Problem Binary label Problem

such that:
Let Y and X be the set of feasible solutions, then
1. For each binary solution x ϵ X with finite energy there
exists exactly one multi-label solution y ϵ Y
-> One-One encoding function T:X->Y

2. arg min Em(y) = T(arg min Eb(x))

 Multi-label to Pseudo-boolean
• Popular encoding scheme
[Roy and Cox ’98, Ishikawa ’03, Schlesinger & Flach ’06]
 Multi-label to Pseudo-boolean
• Popular encoding scheme
[Roy and Cox ’98, Ishikawa ’03, Schlesinger & Flach ’06]

Ishikawa’s result:

E(y) = ∑ θi (yi) + ∑ θij (yi,yj)

i i,j

y ϵ Labels L = {l1, l2, … , lk}

Convex
θij (yi,yj) = g(|yi-yj|) Function

g(|yi-yj|)
|yi-yj|
 Multi-label to Pseudo-boolean
• Popular encoding scheme
[Roy and Cox ’98, Ishikawa ’03, Schlesinger & Flach ’06]

Schlesinger & Flach ’06:

E(y) = ∑ θi (yi) + ∑ θij (yi,yj)

i i,j

y ϵ Labels L = {l1, l2, … , lk}

θij(li+1,lj) + θij (li,lj+1)  θij (li,lj) + θij (li+1,lj+1)

Covers all Submodular

multi-label functions
More general than Ishikawa
 Multi-label to Pseudo-boolean
Problems

• Applicability
– Only solves restricted class of energy functions
– Cannot handle Potts model potentials

• Computational Cost
– Very high computational cost
– Problem size = |Variables| x |Labels|
– Gray level image denoising (1 Mpixel image)
(~2.5 x 108 graph nodes)
Outline of the Tutorial

The st-mincut problem

Connection between st-mincut

and energy minimization?

What problems can we solve

using st-mincut?

st-mincut based Move algorithms

Recent Advances and Open Problems

 St-mincut based Move algorithms

E(x) = ∑ θi (xi) + ∑ θij (xi,xj)

i i,j

x ϵ Labels L = {l1, l2, … , lk}

• Commonly used for solving non-submodular

multi-label problems
• Extremely efficient and produce good
solutions
• Not Exact: Produce local optima
Energy Move Making Algorithms

Solution Space
 Move Making Algorithms
Current Solution
Search
Neighbourhood
Optimal Move
Energy

Solution Space
 Computing the Optimal Move
Current Solution
Search
Neighbourhood
Optimal Move
xc
Key Property
Energy

(t)

Move Space

Solution Space

Bigger move • Better solutions

space • Finding the optimal move hard
 Moves using Graph Cuts
Expansion and Swap move algorithms
[Boykov Veksler and Zabih, PAMI 2001]
• Makes a series of changes to the solution (moves)
• Each move results in a solution with smaller energy

Current Solution
Search
Neighbourhood

Move Space (t) : 2N N Number of

Variables
L Number of
Space of Solutions (x) : LN Labels
 Moves using Graph Cuts
Expansion and Swap move algorithms
[Boykov Veksler and Zabih, PAMI 2001]
• Makes a series of changes to the solution (moves)
• Each move results in a solution with smaller energy

Current Solution

Move to new Construct a move How to

solution function minimize move
functions?
Minimize move function
to get optimal move
 General Binary Moves

x = t x1 + (1- t) x2
New Current Second
solution Solution solution

Em(t) = E(t x1 + (1- t) x2)

Minimize over move variables t to get the
optimal move

Move energy is a submodular QPBF

(Exact Minimization Possible)

Boykov, Veksler and Zabih, PAMI 2001

 Swap Move
• Variables labeled α, β can swap their labels

[Boykov, Veksler, Zabih]

 Swap Move
• Variables labeled α, β can swap their labels
Tree
Ground
Swap Sky, House House
Sky

[Boykov, Veksler, Zabih]

 Swap Move
• Variables labeled α, β can swap their labels

• Move energy is submodular if:

– Unary Potentials: Arbitrary
– Pairwise potentials: Semimetric

θij (la,lb) ≥ 0
θij (la,lb) = 0 a = b

Examples: Potts model, Truncated Convex

[Boykov, Veksler, Zabih]

 Expansion Move
• Variables take label a or retain current label

[Boykov, Veksler, Zabih]

 Expansion Move
• Variables take label a or retain current label
Tree
Ground
House
Status: InitializeSky
Expand Ground
House
with Tree Sky

[Boykov, Veksler, Zabih]

 Expansion Move
• Variables take label a or retain current label

• Move energy is submodular if:

– Unary Potentials: Arbitrary
– Pairwise potentials: Metric Semi metric
+
Triangle
θij (la,lb) + θij (lb,lc) ≥ θij (la,lc) Inequality

Examples: Potts model, Truncated linear

Cannot solve truncated quadratic

[Boykov, Veksler, Zabih]

 General Binary Moves

x = t x1 + (1-t) x2
New First Second
solution solution solution

Minimize over move variables t

Move Type First Second Guarantee

Solution Solution
Expansion Old solution All alpha Metric
Fusion Any solution Any solution 

Move functions can be non-submodular!!

 Solving Continuous Problems using
Fusion Move

x = t x1 + (1-t) x2
x1, x2 can be continuous Optical Flow
Example

Solution
from
Method 2 x2

Solution
from F Final
Method 1 x1 x Solution

(Lempitsky et al. CVPR08, Woodford et al. CVPR08)

 Range Moves
• Move variables can be multi-label

x = (t ==1) x1 + (t==2) x2 +… +(t==k) xk

• Optimal move found out by using the Ishikawa

• Useful for minimizing energies with truncated convex
pairwise potentials

T
θij (yi,yj) = min(|yi-yj|,T)
θij (yi,yj)
|yi-yj|

O. Veksler, CVPR 2007

 Move Algorithms for Solving Higher
Order Energies
E(x) = ∑ θi (xi) + ∑ θij (xi,xj) + ∑ θc (xc)
i i,j c

x ϵ Labels L = {l1, l2, … , lk} Clique c  V

• Higher order functions give rise to higher order move

energies

• Move energies for certain classes of higher order

energies can be transformed to QPBFs.

[Kohli, Kumar and Torr, CVPR07] [Kohli, Ladicky and Torr, CVPR08]
Outline of the Tutorial

The st-mincut problem

Connection between st-mincut

and energy minimization?

What problems can we solve

using st-mincut?

st-mincut based Move algorithms

Recent Advances and Open Problems

 Solving Mixed Programming Problems
x – binary image segmentation (xi ∊ {0,1})
ω – non-local parameter (lives in some large set Ω)

E(x,ω) = C(ω) + ∑ θi (ω, xi) + ∑ θij (ω,xi,xj)

i i,j
unary pairwise
constant
potentials potentials
≥0

ω θi (ω, xi)
Pose Shape
Prior
Stickman Rough Shape
Model Prior
 Open Problems

Characterization of Problems Solvable using

st-mincut

• What functions can be transformed to

submodular QBFs?

Submodular
Functions

st-mincut
Equivalent
 Minimizing General Higher Order
Functions

• We saw how simple higher order potentials

can be solved

• How more sophisticated higher order

potentials can be solved?
 Summary
Exact
Transformation

Labelling (global optimum) Submodular Quadratic

Pseudoboolean Function
Problem Or Relaxed
transformation
(partially optimal)

st-mincut
S

Sub-problem

T
Move making algorithms
Thanks. Questions?
 Use of Higher order Potentials
E(x1,x2,x3) = θ12 (x1,x2) + θ23 (x2,x3)

θij (xi,xj) =
{ 0 if xi=xj
C otherwise

8
E(6,6,6) = 0 + 0 = 0
7
Disparity 6
Labels
5

P1 P2 P3
Pixels

Stereo - Woodford et al. CVPR 2008

 Use of Higher order Potentials
E(x1,x2,x3) = θ12 (x1,x2) + θ23 (x2,x3)

θij (xi,xj) =
{ 0 if xi=xj
C otherwise

8
E(6,6,6) = 0 + 0 = 0
7
Disparity 6
E(6,7,7) = 1 + 0 = 1
Labels
5

P1 P2 P3
Pixels

Stereo - Woodford et al. CVPR 2008

 Use of Higher order Potentials
E(x1,x2,x3) = θ12 (x1,x2) + θ23 (x2,x3)

θij (xi,xj) =
{ 0 if xi=xj
C otherwise

8
E(6,6,6) = 0 + 0 = 0
7
Disparity 6
E(6,7,7) = 1 + 0 = 1
Labels E(6,7,8) = 1 + 1 = 2
5

Pairwise potential
P1 P2 P3
penalize slanted
Pixels planar surfaces

Stereo - Woodford et al. CVPR 2008

 Computing the Optimal Move
Current Solution
Search
Neighbourhood
Optimal Move
xc Transformation
T function
E(x) (t)

x
T(xc, t) = xn = xc + t
 Computing the Optimal Move
Current Solution
Search
Neighbourhood
Optimal Move
xc Transformation
T function
E(x) (t)
Em Move Energy

x
T(xc, t) = xn = xc + t
Em(t) = E(T(xc, t))
 Computing the Optimal Move
Current Solution
Search
Neighbourhood
Optimal Move
xc Transformation
T function
E(x) (t)
Em Move Energy

x
T(xc, t) = xn = xc + t
minimize
Em(t) = E(T(xc, t)) t*
Optimal Move