Scribd Logo
Upload

Welcome to Scribd!

  • Regularization: The Problem of Overfitting

    Uploaded by

    Shubham Sharma
    20 views23 pages
    Regularization helps address the problem of overfitting in machine learning models. It works by adding parameters to the cost function that reduce the magnitude of the learned model's parameters. This encourages simpler models that are less complex and therefore less likely to overfit. Regularized linear and logistic regression models minimize a cost function that is the sum of the original cost plus terms proportional to the magnitude of the parameters weighted by lambda. This has the effect of shrinking parameter values except those most critical for predicting the target variable.

    Original Description:

    Original Title

    01_The_Problem_of_Overfitting_10_min.pptx

    Copyright

    © © All Rights Reserved

    Available Formats

    PPTX, PDF, TXT or read online from Scribd

    Did you find this document useful?

    Is this content inappropriate?

    Report this Document
    Regularization helps address the problem of overfitting in machine learning models. It works by adding parameters to the cost function that reduce the magnitude of the learned model's parameters. This encourages simpler models that are less complex and therefore less likely to overfit. Regularized linear and logistic regression models minimize a cost function that is the sum of the original cost plus terms proportional to the magnitude of the parameters weighted by lambda. This has the effect of shrinking parameter values except those most critical for predicting the target variable.

    Copyright:

    © All Rights Reserved
    Download as PPTX, PDF, TXT or read online from Scribd
    Flag for inappropriate content
    20 views23 pages

    Regularization: The Problem of Overfitting

    Uploaded by

    Shubham Sharma
    Regularization helps address the problem of overfitting in machine learning models. It works by adding parameters to the cost function that reduce the magnitude of the learned model's parameters. This encourages simpler models that are less complex and therefore less likely to overfit. Regularized linear and logistic regression models minimize a cost function that is the sum of the original cost plus terms proportional to the magnitude of the parameters weighted by lambda. This has the effect of shrinking parameter values except those most critical for predicting the target variable.

    Copyright:

    © All Rights Reserved
    Download as PPTX, PDF, TXT or read online from Scribd
    Flag for inappropriate content
    of 23

    Regularization

    The problem of
    overfitting
    Machine Learning
    Example: Linear regression (housing prices)
    Price

    Price

    Price
    Size Size Size

    Overfitting: If we have too many features, the learned hypothesis


    may fit the training set very well ( ), but fail
    to generalize to new examples (predict prices on new examples).
    Andrew Ng
    Example: Logistic regression

    x2 x2 x2

    x1 x1 x1

    ( = sigmoid function)

    Andrew Ng
    Addressing overfitting:
    size of house

    Price
    no. of bedrooms
    no. of floors
    age of house
    average income in neighborhood Size
    kitchen size

    Andrew Ng
    Addressing overfitting:

    Options:
    1. Reduce number of features.
    ― Manually select which features to keep.
    ― Model selection algorithm (later in course).
    2. Regularization.
    ― Keep all the features, but reduce magnitude/values of
    parameters .
    ― Works well when we have a lot of features, each of
    which contributes a bit to predicting .

    Andrew Ng
    Regularization
    Cost function

    Machine Learning
    Intuition

    Price
    Price

    Size of house Size of house

    Suppose we penalize and make , really small.

    Andrew Ng
    Regularization.

    Small values for parameters


    ― “Simpler” hypothesis
    ― Less prone to overfitting
    Housing:
    ― Features:
    ― Parameters:

    Andrew Ng
    Regularization.

    Price

    Size of house

    Andrew Ng
    In regularized linear regression, we choose to minimize

    What if is set to an extremely large value (perhaps for too large


    for our problem, say )?
    Price

    Size of house

    Andrew Ng
    Regularization
    Regularized linear
    regression
    Machine Learning
    Regularized linear regression
    Gradient descent
    Repeat

    Andrew Ng
    Normal equation

    Andrew Ng
    Non-invertibility (optional/advanced).
    Suppose ,
    (#examples) (#features)

    If ,

    Andrew Ng
    Regularization
    Regularized
    logistic regression
    Machine Learning
    Regularized logistic regression.

    x2

    x1
    Cost function:

    Andrew Ng
    Gradient descent
    Repeat

    Andrew Ng
    Advanced optimization
    function [jVal, gradient] = costFunction(theta)
    jVal = [ code to compute ];

    gradient(1) = [code to compute ];

    gradient(2) = [code to compute ];

    gradient(3) = [code to compute ];

    gradient(n+1) = [ code to compute ];


    Andrew Ng

    You might also like