Professional Documents
Culture Documents
Chapter 6
OPTIMIZATION
1
Methods
1 Function Values Only (grid search)
Chapter 6
4 Quasi-Newton methods
2
Grid Search
• Fungsi dihitung di setiap titik pada grid
• Nilai ekstremum ditemukan pada satu titik
Chapter 6
3
Gradient Method
Chapter 6
k
(1) Calculate a search direction s
4
Gradient Method: Steepest
Descent (Search Direction)
k k
s f ( x ) Don’t need to normalize
*
H ( x ) is positive definite for a minimum.
5
Gradient Method:
Step Length
Chapter 6
How to pick
• analytically
• numerically
6
Chapter 6
7
8
Chapter 6
9
Analytical Method
How does one minimize a function in a search direction
using an analytical method?
k k k 1 k k k 1 k k k
f (x s ) f (x ) f ( x ) T f ( x )( x ) ( x )T H ( x )( x )
2
k k
df ( x s ) k k k k k
0 T f ( x )(s ) (s )T H ( x )(s )
d
Solve for
k k
T f ( x )(s ) (6.9)
k k k
(s )T H ( x )(s )
12
Chapter 6
13
Chapter 6
14
Gradient Method:
Termination Criteria
f(x) f(x)
Chapter 6
x x
Big change in f(x) but little change Big change in x but little change
in x. Code will stop if x is sole criterion. in f(x). Code will stop if x is sole criterion.
For minimization you can use up to three criteria for termination:
(1) f ( x k ) f ( x k 1 ) except when f ( x k ) 0
1
k
f (x ) then use f ( x k ) f ( x k 1 ) 2
xi k 1 xi k except when x k 0
(2) 3
xi k
then use x k 1 x k 4
(3) f ( x k ) 5 or si k 6
15
Gradient Method:
Conjugate Search Directions
Improvement over gradient method for general quadratic functions
Basis for many NLP techniques
Two search directions are conjugate relative to Q if
( s i )T Q ( s j ) 0
16
Chapter 6
17
Conjugate Gradient Method
Step 1. At x 0 calculate f (x 0 ). Let
s 0 f ( x 0 )
Step 2. Savef ( x 0 ) and compute
x1 x 0 0 s 0
by minimizing f(x) with respect to in the s0 direction (i.e., carry out a unidimensional search for 0).
Chapter 6
f ( x 0 )f ( x 0 )
For the kth iteration the relation is
k 1 k 1 T f ( x k 1 )f ( x k 1 )
s f ( x )s k
(6.6)
T f ( x k )f ( x k )
For a quadratic function it can be shown that these successive search directions are conjugate.
After n iterations (k = n), the quadratic function is minimized. For a nonquadratic function,
the procedure cycles again with xn+1 becoming x0.
Step 4. Test for convergence to the minimum of f(x). If convergence is not attained, return to step 3.
Step n. Terminate the algorithm when f ( x ) is less than some prescribed tolerance.
k
18
Chapter 6
19
Chapter 6
20
Chapter 6
21
Chapter 6
22
Minimize f ( x1 3) 9( x2 5) using the method of conjugate gradients with
2 2
1
In vector notation, x 0
1
4
f x0
72
Chapter 6
3.554 4 3.564
s1 0.00244
0.197 72 0.022
and
1.223 1 3.564
x2 0.022
5.011
One dimensional Search
25
Chapter 6
26
s H f ( x ) f ( x ) k
k 1 k 1 k
T
(s ) f ( x ) f ( x ) H / k
k T k 1 k1
k k 1
Using definition of conjugate directions, (s )T Hs =0,
T
f ( x k 1
) f ( x ) H H f ( x ) k s 0
1
k k 1 k
Chapter 6
k k 1
f T ( x )f ( x )0
k 1 k
and f T ( x )s 0, and solving for the weighting factor:
k 1 k 1
T f ( x )f ( x )
k
k k
T f ( x )f ( x )
k 1 k 1 k
s f ( x ) k s
27
Newton Method: Linear vs.
Quadratic Approximation of f(x)
k k k 1 k k k
f ( x ) f ( x ) ( x x )T f ( x ) ( x x )T H ( x )( x x )
2
k k k
x x x k s
Chapter 6
29
Chapter 6
30
Chapter 6
31
Chapter 6
32
Marquardt’s Method
1
If H ( x ) or H ( x ) is not always positive definite, make it
positive definite.
1 1
Let H ( x ) H ( x ) I ; similar for H( x )
is a positive constant large enough to shift all the
Chapter 6
negative eigenvalues of H ( x ).
Example
0
At the start of the search, H( x ) is evaluated at x and
Not positive definite
0 1 2
found to be H ( x ) as the eigenvalues
2 1 are e1 3, e2 1
0
Modify H ( x ) to be ( 2)
Positive definite as the
1 2 2
H
eigenvalues are e1 5, e2 1
2 1 2
is adjusted as search proceeds. 33
Step 1
0
Pick x the starting point.
Let convergence criterion
Step 2
Chapter 6
Set k 0. Let 0 10 3
Step 3
k
Calculate f ( x )
Step 4
34
Step 5
1
Calculate s( x ) - H I f ( x )
k k k k k
Step 6
k 1 k k
Calculate x x s( x )
Chapter 6
Step 7
k 1 k
Is f( x ) f ( x )? If yes, go to step 8. If no, go to step 9.
Step 8
1 k
Set k 1 and k k 1. Go to step 3
4
Step 9
Set k 2 k . Go to step 5
35
Secant Methods
Recall for one dimensional search the secant method
only uses values of f(x) and f ′(x).
1
f ( x ) f ( x )
k p
k 1
x x k
f ( x k
)
x x
k p
Chapter 6
37
• Probably the best update formula is the BFGS update
(Broyden – Fletcher – Goldfarb – Shanno) – ca. 1970
38
Chapter 6
39
Chapter 6
40
Chapter 6
41
Chapter 6
42