Scribd Logo
Upload

Welcome to Scribd!

  • Optimization Algorithms: Gradient Descent With Momentum

    Uploaded by

    Subhadip Sinha
    62 views3 pages
    This document discusses gradient descent with momentum for optimization algorithms in deep learning. It explains that momentum works by adding a fraction β of the previous gradient to the current gradient to smooth out oscillations during weight updates. The document provides an example implementation that computes the weighted average of current and previous gradients to update the weights and biases on each iteration.

    Original Description:

    Original Title

    06_gradient-descent-with-momentum_C2W2L05

    Copyright

    © © All Rights Reserved

    Available Formats

    PPTX, PDF, TXT or read online from Scribd

    Did you find this document useful?

    Is this content inappropriate?

    Report this Document
    This document discusses gradient descent with momentum for optimization algorithms in deep learning. It explains that momentum works by adding a fraction β of the previous gradient to the current gradient to smooth out oscillations during weight updates. The document provides an example implementation that computes the weighted average of current and previous gradients to update the weights and biases on each iteration.

    Copyright:

    © All Rights Reserved
    Download as PPTX, PDF, TXT or read online from Scribd
    Flag for inappropriate content
    62 views3 pages

    Optimization Algorithms: Gradient Descent With Momentum

    Uploaded by

    Subhadip Sinha
    This document discusses gradient descent with momentum for optimization algorithms in deep learning. It explains that momentum works by adding a fraction β of the previous gradient to the current gradient to smooth out oscillations during weight updates. The document provides an example implementation that computes the weighted average of current and previous gradients to update the weights and biases on each iteration.

    Copyright:

    © All Rights Reserved
    Download as PPTX, PDF, TXT or read online from Scribd
    Flag for inappropriate content
    of 3

    Optimization

    Algorithms

    Gradient descent
    deeplearning.ai
    with momentum
    Gradient descent example

    deeplearning.ai
    Andrew Ng
    Implementation details
    Oniteration 𝑡 :
     

    Compute on the current mini-batch


     

     
    𝑣 𝑑𝑊 = 𝛽 𝑣 𝑑𝑊 + ( 1− 𝛽 ) 𝑑𝑊
     
    𝑣 𝑑𝑏 =𝛽 𝑣 𝑑𝑏 + ( 1− 𝛽 ) 𝑑𝑏
     
    𝑊 =𝑊 − 𝛼 𝑣 𝑑𝑊 ,  𝑏=𝑏 − 𝛼 𝑣 𝑑𝑏

     
    Hyperparameters:  
    𝛽=0.9
    Andrew Ng

    You might also like