What Energy Functions Can

be Minimized Using
Graph Cuts?
Shai Bagon
Advanced Topics in Computer Vision
June 2010

What is an Energy Function?

Image
For a given
Segmentation:
problem:
Useful Energy function:
1. Good solution Low energy
2.suggested
Tractablesolution
Can be minimized

E
a number
-20
237

Families of Functions or Outline

F2 submodular
Non submodular
F3
Beyond F3

Foreground Selection
Let
yi color of ith pixel
xi {0,1} BG/FG labels (variables)

Given BG/FG scribbles:

Pr(xi|yi)=How likely each pixel to be FG/BG
Pr(xm|xn)=Adjacent pixels should have same label
yi

F energy:
E(x)=iEi(xi)+ijEij(xi,xj)
2

xi
xm

xn

Submodular
Known concept from set-functions:
f x f y f x y f x y x, y S
f 1,0 f 0,1 f 1,1 f 0,0

S 2

E(x) = i Ei(xi) + ij Eij (xi, xj), xi {0,1}

xj

xi

Eij(xi,xj):

What does it
mean?
B+C-A-D 0

F2 submodular
How to
Minimize?
E(x) = i Ei(xi) + ij Eij (xi, xj), xi {0,1}
Local beliefs:

Prior knowledge:

Data term

Smoothness term

Graph Partitioning
V EE ww )
A weighted graph G=( V
Special Nodes: s t
s-t cut: S V, T V , s S, t T

S T , S T V
Cost of a cut: Cut ) S , T ( wij
iS , jT

Nice property: 1:1 mapping

s-t cut {0,1}|V|-2

wij

Graph Partitioning - Energy

C-A

E(x) = i Ei(xi) + ij Eij (xi, xj)

D-C
s

Ej(1)

B+C-A-D
j

i
Eij(xi,xj)
xj

xi

Ei(0)

=A+

C-A C-A

0 D-C
0 D-C

0 B+C-A-D
0

Graph Partitioning - Energy

E(x) = i Ei(xi) + ij Eij (xi, xj)

C-A

D-C
s

Ej(1)

B+C-A-Dassignment
st cut binary
i

0 vS

cut
xv cost energy of assignment
Ei(0)
1 v T
t
min cut Energy min.
cut S , T A wij
iS , jT

A E j 1 Ei 0 D C B C A D
E x
B=Eij(0,1)

Recap
F2 submodular:
E(x) = i Ei(xi) + ij Eij (xi, xj)
Eij(1,0)+Eij(0,1)Eij(0,0)+Eij(1,1)

Mapping from energy to graph partition

Min Energy = computing min-cut
Global optimum in poly time
for submodular functions!

Next
Multi-label F2
E(x)=i Ei(xi) + ij Eij(xi,xj) s.t. xi {1,,L}
Fusion moves: solving binary sub-problems
Applications to stereo, stitching, segmentation
Solve Binary problem:

xi=0

xi=1

Current
labeling

=
suggested
labeling

Fusion

Alpha
expansion

Stereo matching

see http://vision.middlebury.edu/stereo/

Input:

Pairwise MRF
[Boykov et al. 01]

Ground
truth
slide by Carsten Rother, ICCV09

Panoramic stitching

slide by Carsten Rother, ICCV09

Panoramic stitching

slide by Pushmeet Kohli, ICCV09

AutoCollage

[Rother et. al. Siggraph

http://research.microsoft.com/en-us/um/cambridge/projects/autocollage/

Next
Multi-label F2
E(x)=i Ei(xi) + ij Eij(xi,xj) s.t. xi {1,,L}
Fusion moves: solving binary sub-problems
Applications to stereo, stitching, segmentation

Non-submodular
Beyond pair-wise interactions: F3

Merging Regions
regions (Ncuts)

input image

edge prob.
j
i
pi

weak edge

pi prob. of boundary being edge

min:
GOAL: Find labeling
xi{0,1}
strong
edgethat max:
Pr x
p1i pi log1 pi
pilog
i: xi 1

i: x i 0

i: xi 0

Taking -log

Merging Regions
x

log p log1 p log p log p

i

i: xi 1

log pi
i

i: xi 0

i: xi 0

log1 p log p
i

i: xi 0

i:xi 0

pi Adding and
1 xi C wi xi
C log
1 pisubtracting

the i
i
same number
pi 1
likely to be an edge
2
1 1 pi likey to be merged
pilog
2
i: x 0

wi 0
pi
wi log
x1pi : wlog
p
i 0 i
def

i: xi 1

Merging Regions
Solving for edges:
arg min x C i wi xi
Consistency constraints:
No dangling edge
J
x1

x2

x3

EJ

xi

wi

1 longer
E J 1 x1No
x2 x3
x2 x3 1 x1 x3
1 x1 pair-wise:
F3

Minimization trick
1 x1 x2 x3 min z 1 x1 x2 x3 2
z 0 ,1

xi min z i 1 xi K 1
i 1

z 0 ,1

Freedman D., Turek MW, Graph cuts with many pixel interactions:
theory and applications to shape modeling. Image Vision Computing 2010

Merging Regions
The resulting energy:
E x

min z n wn xn
l ,m ,nJ zlmn xl xm xn 1 1 xl xn

+ Pair-wise
- Non submodular!

Quadratic Pseudo-Boolean Optimization

s
j

i
i

Kolmogorov V., Carsten R., Minimizing non-submodular functions with graph cuts a review.
PAMI07

Quadratic Pseudo-Boolean Optimization

+ All edges with positive capacities
- No constraint i 1 i
Labeling rule: 0 if i S , i T

yi 1 if i T , i S
otherwise

partial labeling

s
i

j
t

Quadratic Pseudo-Boolean Optimization

Properties of partial labeling y:
1. Let z=FUSE(y,x)
E(z)E(x)
2. y is subset of optimal y*
y is complete:
1. E submodular
2. Exists flipping
(inference in trees)

s
i

j
t

QBPO - Probing
QPBO:

Probe Node p:
0

What can we say about

variables?
r -> is always 0
s -> is always equal to q
t -> is 0 when q = 1
slide by Pushmeet Kohli, ICCV09

QBPO - Probing
Probe nodes in an order until energy
unchanged
Simplified energy preserves global
optimality and (sometimes) gives the global
minimum

slide by Pushmeet Kohli, ICCV09

Merging Regions
Result using QPBO-P:
input image

regions (Ncuts)

Result

Recap
F3 and more
Minimization trick

Non submodular
QPBO approx. partial labeling

Beyond F3

[Kohli et. al. CVPR 07, 08, PAMI 08, IJCV 09]

Image Segmentation
E(X) =

ci xi + dij |xi-xj|
i

Image

n = number of
pixels
E: {0,1}n R
0 fg, 1bg

i,j

Unary Cost

Segmentation

[Boykov and Jolly 01] [Blake et al. 04] [Rother et al.`04]

Pn Potts Potentials

Patch
Dictionary
(Tree)
h(Xp) =

0
if xi = 0, i p
Cmax otherwise

Cmax 0

[slide credits: Koh

Pn Potts Potentials
n = number of
pixels
E: {0,1}n R
0 fg, 1bg

E(X) =

h(Xp) =

ci xi + dij |xi-xj|
i

i,j

0
if xi = 0, i p
Cmax otherwise

hp (Xp)

[slide credits: Koh

Image Segmentation
n = number of
pixels
E: {0,1}n R
0 fg, 1bg

E(X) =

ci xi + dij |xi-xj|
i

Image

i,j

Pairwise
Segmentation

hp (Xp)

Final Segmentation

[slide credits: Koh

Application:
Recognition and Segmentation

Image

One
superpixelizatio
n

Unaries only Pairwise CRF only

TextonBoost [Shotton et al. 06]
[Shotton et al. 06]

another
superpixelization

Pn Potts

from [Kohli et al.

Robust(soft) Pn Potts model

h(xp) =

0 if xi = 0, i p
f(xp) otherwise
p

Pn Potts

p
Robust Pn
Potts

from [Kohli et al.

Application:
Recognition and Segmentation

Image

Unaries only Pairwise CRF only

TextonBoost [Shotton et al. 06]
[Shotton et al. 06]

another
One
supersuperpixelizatio pixelization
n

Pn Potts

robust Pn Potts robust Pn Potts

(different f)

From [Kohli et al.

Same idea for surface-based stereo

]Bleyer 10[

One input
image

Ground
truth depth

Stereo with
hardsegmentation

This approach gets best result on

Middlebury Teddy image-pair:

Stereo with
robust Pn Potts

How is it done
Most general binary
function:

H (X) = F ( xi ) concave
H (X)

xi

The transformation is to a submodular pair-wise

MRF, hence optimization globally optimal

[slide credits: Koh

Higher order to Quadratic

Start with Pn Potts model:
f(x) =

min f(x)
x

Higher Order
Function

xi = 0
xi > 0

{
=

0 if all xi = 0
C1 otherwise

x {0,1}n

min C1a + C1 (1-a) xi

x,a {0,1}

Quadratic Submodular
Function

f(x) = 0

a=0

f(x) = C1

a=1

[slide credits: Koh

Higher order to Quadratic

=

min f(x)
x

min C1a + C1 (1-a)

x,a {0,1}

Higher Order Function

xi

Quadratic Submodular
Function

C1xi

C1

xi

[slide credits: Koh

Higher order to Quadratic

=

min f(x)
x

min C1a + C1 (1-a)

x,a {0,1}

Higher Order
Submodular Function

xi

Quadratic Submodular
Function

C1xi
a=1

a=0

Lower envelope
of concave
functions is
concave

C1

xi

[slide credits: Koh

Summary
Submodular F2
F3 and beyond: minimization trick
Non submodular
QPBO(P)

Beyond F3 Robust HOP

f2(x)

a=1

a=0

f1(x)

xi

s
i

j
t