You are on page 1of 18

Lecture 10: descent methods

• • • • •

Generic descent algorithm Generalization to multiple dimensions Problems of descent methods, possible improvements Fixes Local minima

Gradient descent (reminder)
Minimum of a function is found by following the slope of the function

f
f(x) guess

f(m) m

x

1

Gradient descent (illustration) f f(x) guess next step f(m) m x Gradient descent (illustration) f f(x) guess next step new gradient f(m) m x 2 .

Gradient descent (illustration) f f(x) guess next step f(m) m x Gradient descent (illustration) f f(x) guess f(m) stop m x 3 .

Gradient descent: algorithm Start with a point (guess) Repeat Determine a descent direction Choose a step Update Until stopping criterion is satisfied guess Gradient descent: algorithm Start with a point (guess) Repeat Determine a descent direction Choose a step Update Until stopping criterion is satisfied Direction: downhill 4 .

Gradient descent: algorithm Start with a point (guess) Repeat Determine a descent direction Choose a step Update Until stopping criterion is satisfied step Gradient descent: algorithm Start with a point (guess) Repeat Determine a descent direction Choose a step Update Until stopping criterion is satisfied Now you are here 5 .

Gradient descent: algorithm Start with a point (guess) Repeat Determine a descent direction Choose a step Update Until stopping criterion is satisfied Stop when “close” from minimum Gradient descent: algorithm Start with a point (guess) Repeat Determine a descent direction Choose a step Update guess = x direction = -f’(x) step = h > 0 x:=x–hf’(x) f’(x)~0 Until stopping criterion is satisfied 6 .

Example of 2D gradient: pic of the MATLAB demo Illustration of the gradient in 2D Example of 2D gradient: pic of the MATLAB demo Illustration of the gradient in 2D 7 .

Example of 2D gradient: pic of the MATLAB demo Illustration of the gradient in 2D Example of 2D gradient: pic of the MATLAB demo Definition of the gradient in 2D This is just a genaralization of the derivative in two dimensions. 8 . This can be generalized to any dimension.

Example of 2D gradient: pic of the MATLAB demo Illustration of the gradient in 2D Example of 2D gradient: pic of the MATLAB demo Gradient descent works in 2D 9 .

Generalization to multiple dimensions Start with a point (guess) Repeat Determine a descent direction Choose a step Update Until stopping criterion is satisfied guess Generalization to multiple dimensions Start with a point (guess) Repeat Determine a descent direction Choose a step Update Until stopping criterion is satisfied Direction: downhill 10 .

Generalization to multiple dimensions Start with a point (guess) Repeat Determine a descent direction Choose a step Update Until stopping criterion is satisfied step Generalization to multiple dimensions Start with a point (guess) Repeat Determine a descent direction Choose a step Update Until stopping criterion is satisfied Now you are here 11 .

Generalization to multiple dimensions Start with a point (guess) Repeat Determine a descent direction Choose a step Update Until stopping criterion is satisfied Stop when “close” from minimum Generalization to multiple dimensions Start with a point (guess) Repeat Determine a descent direction Choose a step Update guess = x direction = -f(x) step = h > 0 x:=x–h Vf’(x) Until stopping criterion is satisfied Vf’(x)~0 12 .

you need to compute the gradient of its price: Stock 1 13 . For the descent method. Example of 2D gradient: MATLAB demo The cost to buy a portfolio is: Stock N Stock i Stock 2 If you want to minimize the price to buy your portfolio. and by in N dimensions. f’(x) can be replaced by In two dimensions.Multiple dimensions Everything that you have seen with derivatives can be generalized with the gradient.

large steps: potentially bad results f f(x) guess f(m) stop m Too many steps: takes too long to converge x Problem 1: choice of the step When updating the current computation: .Problem 1: choice of the step When updating the current computation: .large steps: potentially bad results f f(x) Next point (went too far) f(m) m Current point x 14 .small steps: inefficient .small steps: inefficient .

2004 ] 15 . Notes. Boyd.Problem 2: « ping pong effect » [S. Boyd. Notes. L. Stanford Univ. Convex Convex Optimization lect. L. Convex Convex Optimization lect. Stanford Univ. Vandenberghe. Vandenberghe. 2004 ] Problem 2: « ping pong effect » [S.

L. Convex Convex Optimization lect. Prof.Problem 2: (other norm dependent issues) [S. 2004 ] Problem 3: stopping criterion Intuitive criterion: In multiple dimensions: Or equivalently Rarely used in practice. More about this in EE227A (convex optimization. Stanford Univ. Notes. Vandenberghe. El Ghaoui). L. Boyd. 16 .

Newton steps Fundamental problem of the method: local minima Local minima: pic of the MATLAB demo The iterations of the algorithm converge to a local minimum 17 .Normalized steepest descent .Backtracking line search .Line search methods. in particular .Fixes Several methods exist to address this problem .Exact line search .

Local minima: pic of the MATLAB demo View of the algorithm is « myopic » 18 .