2

Simulated Annealing Algorithm

(simulated annealing SA)

(annealing process)

(Simulated Annealing SA) 1983 Kirkpatrick

[Kirkpatrick et al., 1983]
SA

(global optimum)

(local optimum)

SA

45


Simulated Annealing Algorithm

SA FSA (Fast Simulated An-

nealing) ADA (Adaptive Simulated Annealing)

SA [Kirkpatrick et al., 1983][Rutenbar, 1989][Hajek, 1985]
(iterative improvement search)

SA

SA
()
SA
(probabilistic algorithm)

2.1

Simulated Annealing Algorithm

ri E(ri )
T (Boltzmann probability)

P r[E(ri )] = eE(ri )/kB T

(2.1)

kB

SA SA

2.1 SA 4
( Metropolis Monte
Carlo ) [Ham and Kostanic, 2001]

(random generator) move

(objective function)
(annealing schedule)

- Simulated Annealing Algorithm

1. x x

2. T T
()
3. xp = x + x x
4. f = f (xp ) f (x)
46

2.1
Simulated Annealing Algorithm

2.1:
5. xp

(
P r(x xp ) =

1
ef /T

f < 0
f 0

(2.2)

T f 0 (uniform
distribution) [0, 1] P r(x xp ) > xp
x

6.

7. T T ()

SA
- Annealing Schedule
SA ( -

T ) move

47


Simulated Annealing Algorithm

(Markov chain) [Isaacson and Madsen, 1976]
[Geman and Geman, 1984]
T (k) =

T (0)
k = 1, 2, . . .
log (1 + k)

(2.3)

k T (k) k T (0) (

) SA
f (x) k
T (k) = T (k 1)

(2.4)

1 (exponential
cooling scheme ECS) = 0.95
(linear cooling scheme LCS)
T (k) = T (k 1) T

(2.5)

T
LCS
- Initial Temperature

T (0)

0.8 80%
T (0) f

- Final Temperature

SA

SA

2.2

SA

SA

SA
48

2.2

2.2.1

SA [Kirkpatrick et al., 1983] () "Metropolis

Monte Carlo integration" [Metropolis et al., 1953] -

2.3

1/log(k + 1) ( 1 log ) k

simulated annealing

2.2.2

Boltzmann Annealing

(Boltzmann Annealing BA [Szu and

Hartley, 1987]) 2.6 Ek k () Ek+1
k () E
(E = Ek+1 Ek )
P r[E]

e(Ek+1 /T )
e(Ek+1 /T ) +e(Ek /T )
1
1+e(E/T )
(E/T )

=
e

(2.6)

3
P r[E] -
T (k) - k T
gT (r) - (probability density)

r = {ri , i = 1, . . . , M } T M (
dimension )

SA

2.3 BA
(Boltzmann distribution)

ln k0
(2.7)
Tk = T0
ln k
T "" k "" k0 k

Tk+1 = Tk T0

ln k0
k(ln k)2

(2.8)

Tk+1
Tk

= cTk , 0 < c < 1

= T0 e(c1)k

(2.9)

49


Simulated Annealing Algorithm

BA
SA

2.2.3

Fast Annealing

[Szu and Hartley, 1987] SA (Cauchy

distribution)

Tk =

T0
k

(2.10)

SA (Fast Annealing FA)

BA

2.2.4

Adaptive Simulated Annealing

(Adaptive Simulated Annealing ASA [Ingber, 1993])

Ingber
(Very Fast Simulated Reannealing VFSR [Ingber, 1989])
ASA
VFSR
ASA SA

SA

(Boltzmann
Annealing BA [Szu and Hartley, 1987]) (Fast Annealing FA [Szu and
Hartley, 1987])
ASA rki i ( D) k

rki [Ai , Bi ]
(2.11)
Ti i
Ti (k) = T0i eci k

1/D

(2.12)

ASA SA
ci
Tf i
ci

= T0i emi kf = eni

= mi eni /D

(2.13)

mi ni ASA
SA SA

SA
SA
50

2.2

2.2: peaks SA

 2.1 SA
peaks ( 2.2)

2.3-() () SA T (0) = 20
0.1 T (k) = T (k 1) = 0.99

75 SA

2.3-() () SA
T (0) = 60

2.4 = 0.95
SA

SA

SA


SA
51


Simulated Annealing Algorithm

() T (0) = 20 = 0.99

()

() T (0) = 60 = 0.99

()

2.3: peaks

2.3

SA:

[Xu et al., 2007] ()

SA (face recognition) 2
(2D Principal Component Analysis 2DPCA)

2.3.1

2DPCA

PCA
2DPCA 2 PCA
1 2DPCA PCA

{X1 , X2 , . . . , XN } N ( 2.5 2DPCA

52

2.3 SA:

1
1

0
0

1
1

3
3

0
x

20

40

60

80

100

120

140

120

140

() T (0) = 20 = 0.95

()

2
1

2
0
1

2
0

3
4

5
2

3
3

0
x

20

40

60

80

100

() T (0) = 60 = 0.95

()

2.4: peaks

2.5: ( [Xu et al., 2007])

(covariance matrix)
C

= E[(X E(X))T (X E(X))]

PN
T (Xi X)

= N1 i=1 (Xi X)

(2.14)

N
Xi i h w X
2DPCA A h w x
w A x y
53


Simulated Annealing Algorithm

h
y = Ax

(2.15)

2DPCA x (projection vector) ( A)

(projected vector)

(2.16)

J(w) = tr(Sw )

Sw
tr(Sw ) (trace) Sw Sw
Sw

= E[y E(y)]T [y E(y)]

= E[(X E(X))w]T [(X E(X))w]

(2.17)

(2.18)

J(w) = wT Cw

(optimal projection vector) w1 , w2 , . . . , wd

(orthonormal eigenvector) C (eigenvalue) d

(feature vector) Yi = [yi1 , yi2 , . . . , yid ]

yik = Xi wk k = 1, 2, . . . , d (
)

Yt = [yt1 , yt2 , . . . , ytd ]

2.3.2

2DPCA

2DPCA
Frobenius
dF (Yi , Yj ) =

"

d X
m
X

(yik (l) yjk (l))2

k=1 l=1

#1/2

(2.19)

Yang
d
X

kyik yjk k2

(2.20)

|(Yi Yj )T (Yi Yj )|

(2.21)

dY (Yi , Yj ) =

k=1

Volume
dV (Yi , Yj ) =

SA
54

2.3 SA:

2.6: 2DPCA SA

2.3.3

SA

N {X1 , X2 , . . . , XN } Xi

i m n Xi 2DPCA

Yi = [yi1 , yi2 , . . . , yid ] i = 1, 2, . . . , N yik , k = 1, 2, . . . , d m

Xi Xj Yi Yj

kyik yjk k2 , k = 1, 2, . . . , d Xi
Xj ( Yi Yj )

SA
SA wk , k = 1, 2, . . . , d

d(Yi , Yj ) =

d
X

k+1

wk kyik yjk k2 /

d
X

wk

(2.22)

k=1

yik , k = 1, 2, . . . , d

Yi wk

SA 2DPCA SA
2.6 SA

55


Simulated Annealing Algorithm

2DPCA SA
1. Tmax , lmax , kmax , T (l), w0 , N (x), T = Tmax , w = w0 w = (w1 , . . . , wd )T
w0 d

2. w w = N (w) N ()

3. w w 2.22
w w c = c(w ) c(w)

4. c < 0 w = w ec/T > rand w = w rand

5. SA T = T (l)

SA
(2.23)

w = (w1 , . . . , wd )T
w
dSA (Yi , Yj ) =

d
X

wk kyik

yjk k2 /

k=1

d
X

wk

(2.24)

k=1

Yt

Yi Yt

(2.25)

c = arg mini dSA (Yt , Yi )

c = arg mini

d
X

wk kyik ytk k2 /

k=1

c [1, 2, . . . , N ] Yt c

2.3.4

d
X

wk

(2.26)

k=1

SA
2.7 2.1
SA
(Tmax )

500 SA (kmax ) 2.8

SA

56

2.4

2.7: ( [Xu et al., 2007])

2.1:

Frobenius
Yang
Volume
SA

2.4

4,7,8
3
2,3
3

(%)
95.5
96.0
97.0
97.5

SA
SA (gradient)
SA

() SA
SA

SA

SA

57


Simulated Annealing Algorithm

2.8: ( [Xu et al., 2007])

58

2.4

2.1. Rastrigin f (x, y) = 20 + x2 + y 2 10(cos 2x + cos 2y) 2.9

(local minima) (global minima)

SA SA

2.9: Rastrigin
2.2. SA (clustering data) (x1 , x2 )

100
SA

2.3. SA (curve fitting) gauss3.mat

MATLAB ( 2.10)

xb1 2

f (x) = a0 eb0 x + a1 e

xb2 2

+ a2 e

2.4. SA
SA SA

SA
(perceptron) 2 XOR

2.5. peaks

2.6. XO 3 3 SA
(state machine)

59


Simulated Annealing Algorithm

2.10:
2.7. SA [Sundermann and Lemahieu, 1995]

Positron Emission Tomography (PET)

SA Salt & Pepper ( 2.11 )

I n
I
X
n
F =
[Iij
Iij ]2
i,j

SA ( )

SA

2.11: SA
2.8. [Xu et al., 2007] SA

SA
SA

60

J. A. Anderson and E. Rosenfeld, editors. Neurocomputing: Foundation of Research. M.I.T. Press,

Cambridge, MA, 1988.
S. Geman and D. Geman. Stochastic relaxation, gibbs distributions, and the bayesian restoration of
images. IEEE Transactions on Pattern Analysis and Machine Intelligence, PAMI-6:721--741, 1984.
B. Hajek. A tutorial survey of theory and applications of simulated annealing. In Proceedings of the
24th IEEE Conference on Decision and Control, volume 2, pages 755--760, 1985.
Fredric M. Ham and Ivica Kostanic. Principle of Neurocomputing for Science & Engineering, chapter 5,
pages 209--215. McGraw-Hill, 2001.
L. Ingber. Very fast simulated re-annealing. In Mathl. Comput. Modelling, volume 12, pages 967--973,
1989.
L. Ingber, 1993. URL ftp.alumni.caltech.edu:/pub/ingber/ASA-shar,ASA-shar.Z,ASA.
tar.Z,ASA.tar.gz,ASA.zip.
D. L. Isaacson and R. W. Madsen. Markov Chains: Theory and Applications. New York: Wiley, 1976.
S. Kirkpatrick, C. D. Gelatt Jr., and M. P. Vecchi. Optimization by simulated annealing. Science, 220:
671--680, 1983. Reprinted in 1988, Anderson and Rosenfeld [1988], pp. 554-567.
N. Metropolis, A.W. Rosenbluth, M.N. Rosenbluth, A.H. Teller, and F. Teller. Equation of state calculations by fast computing machines. In J. Chem. Phys., volume 21, pages 1087--1092, 1953.
R. A. Rutenbar. Simulated annealing algorithms: An overview. IEEE Circuits and Devices Magazine,
pages 19--26, January 1989.
Erik Sundermann and Ignace L. Lemahieu. Pet image reconstruction using simulated annealing.
In Murray H. Loew, editor, Proceedings of SPIE, volume 2434 of Medical Imaging 1995: Image
Processing, pages 378--386, May 1995.
H. Szu and R. Hartley. Fast simulated annealing. In Phys. Lett., volume A 122, pages 157--162, 1987.
Z. Xu, L. Wang, and L. Yang. A simulated annealing and 2dpca based method for face recognition.
In 3rd International IEEE Conference on Signal-Image Technologies and Internet-Based System,
pages 777--782, 2007.
61

62