Scribd Logo
Upload

Welcome to Scribd!

  • Deep Learning - Lecture 6

    Uploaded by

    Myristica At
    4 views11 pages
    This document discusses techniques for addressing overfitting in deep neural networks, including regularization methods like L1 and L2 regularization and dropout. It begins by defining overfitting and contrasting it with underfitting and normal fitting. Next, it explains that overfitting occurs in deep neural networks due to their large number of parameters, continuous gradient updating, and the scale sensitiveness of cross entropy loss. The document then introduces regularization as a way to reduce model complexity and prevent overfitting by reducing weights for uninformative features. It proceeds to explain L1 and L2 regularization, providing illustrations, and then defines dropout and its implementation in PyTorch.

    Original Description:

    Copyright

    © © All Rights Reserved

    Available Formats

    PDF, TXT or read online from Scribd

    Did you find this document useful?

    Is this content inappropriate?

    Report this Document
    This document discusses techniques for addressing overfitting in deep neural networks, including regularization methods like L1 and L2 regularization and dropout. It begins by defining overfitting and contrasting it with underfitting and normal fitting. Next, it explains that overfitting occurs in deep neural networks due to their large number of parameters, continuous gradient updating, and the scale sensitiveness of cross entropy loss. The document then introduces regularization as a way to reduce model complexity and prevent overfitting by reducing weights for uninformative features. It proceeds to explain L1 and L2 regularization, providing illustrations, and then defines dropout and its implementation in PyTorch.

    Copyright:

    © All Rights Reserved
    Download as PDF, TXT or read online from Scribd
    Flag for inappropriate content
    4 views11 pages

    Deep Learning - Lecture 6

    Uploaded by

    Myristica At
    This document discusses techniques for addressing overfitting in deep neural networks, including regularization methods like L1 and L2 regularization and dropout. It begins by defining overfitting and contrasting it with underfitting and normal fitting. Next, it explains that overfitting occurs in deep neural networks due to their large number of parameters, continuous gradient updating, and the scale sensitiveness of cross entropy loss. The document then introduces regularization as a way to reduce model complexity and prevent overfitting by reducing weights for uninformative features. It proceeds to explain L1 and L2 regularization, providing illustrations, and then defines dropout and its implementation in PyTorch.

    Copyright:

    © All Rights Reserved
    Download as PDF, TXT or read online from Scribd
    Flag for inappropriate content
    of 11

    SIF 910: Deep Learning

    Lecture 6: Modification and Ext. to a Feed Forward Neural Net (1)

    Chandra Kusuma Dewa


    2021/2022
    Outlines

    ● Overfitting in deep neural network training


    ● Solving overfitting problem with regularization
    ● Regularization techniques
    ○ L1 Regularization
    ○ L2 Regularization
    ○ Dropout
    What is Overfitting?

    Image source: https://arxiv.org/pdf/1901.06566.pdf


    Underfit v.s. Normal v.s. Overfit

    Image source: http://mlwiki.org/index.php/Overfitting


    The main cause of the overfitting problem...

    According to Salman and Liu (2019):

    ● Deep neural networks are prone to


    overfitting because of the large
    number of parameters to be
    learned,
    ● Continuous gradient updating, and
    ● Scale sensitiveness of cross
    entropy loss

    Image source: https://ai.googleblog.com/2019/08/exploring-weight-agnostic-neural.html


    How to prevent the problem? - Regularization

    We can make the neural


    networks less complex by
    reducing the value of the
    weights for the uninformative
    features of the dataset. In other
    words, we regularize the model
    using the following methods:
    ● L1 Regularization
    ● L2 Regularization
    ● Dropout
    ● etc

    Image source: https://www.analyticsvidhya.com/blog/2018/04/fundamentals-deep-learning-regularization-techniques/


    L1 Regularization

    Image source: https://androidkt.com/how-to-add-l1-l2-regularization-in-pytorch-loss-function/


    L2 Regularization

    Image source: https://androidkt.com/how-to-add-l1-l2-regularization-in-pytorch-loss-function/


    L1 and L2 Regularization in PyTorch

    Image source: https://androidkt.com/how-to-add-l1-l2-regularization-in-pytorch-loss-function/


    Dropout

    Image source: https://jmlr.org/papers/volume15/srivastava14a/srivastava14a.pdf


    Dropout in PyTorch

    You might also like