Scribd Logo
Upload

Welcome to Scribd!

  • Activations

    Uploaded by

    nitin garg
    13 views8 pages
    The document discusses different activation functions used in neural networks - ReLU, Tanh, Sigmoid, Softmax, GeLU. It provides the pros and cons of each function. The key points are: ReLU is default for hidden layers, Sigmoid for binary classification output, Softmax for multi-class classification output, and GeLU or Tanh can also be used. Thumb rules suggest starting with ReLU and GeLU for hidden layers and choosing the appropriate activation for the output based on the problem type.

    Original Description:

    Activation functions

    Original Title

    activations

    Copyright

    © © All Rights Reserved

    Available Formats

    PDF, TXT or read online from Scribd

    Did you find this document useful?

    Is this content inappropriate?

    Report this Document
    The document discusses different activation functions used in neural networks - ReLU, Tanh, Sigmoid, Softmax, GeLU. It provides the pros and cons of each function. The key points are: ReLU is default for hidden layers, Sigmoid for binary classification output, Softmax for multi-class classification output, and GeLU or Tanh can also be used. Thumb rules suggest starting with ReLU and GeLU for hidden layers and choosing the appropriate activation for the output based on the problem type.

    Copyright:

    © All Rights Reserved
    Download as PDF, TXT or read online from Scribd
    Flag for inappropriate content
    13 views8 pages

    Activations

    Uploaded by

    nitin garg
    The document discusses different activation functions used in neural networks - ReLU, Tanh, Sigmoid, Softmax, GeLU. It provides the pros and cons of each function. The key points are: ReLU is default for hidden layers, Sigmoid for binary classification output, Softmax for multi-class classification output, and GeLU or Tanh can also be used. Thumb rules suggest starting with ReLU and GeLU for hidden layers and choosing the appropriate activation for the output based on the problem type.

    Copyright:

    © All Rights Reserved
    Download as PDF, TXT or read online from Scribd
    Flag for inappropriate content
    of 8

    ACTIVATION FUNCTIONS

    WHICH ONE TO USE

    swipe right
    ReLU

    0 when x <= 0; x when x > 0. 0 when x <0; 1 when x > 0

    PROs CONs
    • Solves vanishing gradient • Neurons with -ve input die
    • Computationally Efficient • Sensitive to initialization
    • Faster Convergence • Not differentiable at 0
    • Default for hidden layers

    swipe right
    Tanh

    Range of Tanh : [-1,1]

    Range of Deriv. Tanh: (0,1]

    PROs CONs
    • Zero-centered range (better for • Vanishing gradient problem
    optimization) • Computationally expensive
    • Smooth gradient • Not suitable for deeper networks

    swipe right
    Sigmoid

    Range of Sigmoid : [0,1]

    Range of Deriv. Sigmoid : (0,0.25]

    PROs CONs
    • Output suitable for binary • Vanishing gradient problem
    classification • Computationally expensive
    • Used for multi-label • Too much compression
    classification
    • Smooth gradient

    swipe right
    Softmax

    Range : (0,1)

    PROs CONs
    • Interpretable as likelihood of a • Doesn't work for multi-label
    class classification
    • works well with categorical • Vulnerable to Imbalance
    cross entropy (CCE) loss datasets
    • Optimal for multi-class • Instable to large input values
    classification (Overflow errors)

    swipe right
    GeLU

    Range similiar to ReLU

    PROs CONs
    • Smooth gradient • Computationally expensive that
    • Dynamic gating makes network ReLU
    adaptable • Reduced Interpretibility
    • Used in SoTA transformer
    models (GPT, BERT, SAM)
    Thumb rules

    • Start with ReLU for hidden layers, then GeLU


    and Tanh
    • If binary classification use Sigmoid for output
    • If multi-class, use Softmax for output
    • For transformer-based models, start with
    GeLU
    References
    https://towardsdatascience.com/why-rectified-linear-unit-relu-in
    -deep-learning-and-the-best-practice-to-use-it-with-tensorflow-e
    9880933b7ef

    https://medium.com/@omkar.nallagoni/activation-functions-wit
    h-derivative-and-python-code-sigmoid-vs-tanh-vs-relu-44d23915
    c1f4

    https://www.researchgate.net/figure/The-Softmax-activation-fun
    ction-and-its-derivative_fig5_373474238 [accessed 1 Mar, 2024]

    https://www.cs.cmu.edu/~bhiksha/courses/deeplearning/Spring
    .2019/archive-f19/www-bak11-22-2019/document/note/hwnotes
    /HW1p1.html.backup

    https://www.researchgate.net/figure/The-GELU-function-and-its
    -derivative-with-respect-to-x_fig2_373051870 [accessed 2 Mar,
    2024]

    You might also like