Scribd Logo
Upload

Welcome to Scribd!

  • 0 1 Intro Notations

    Uploaded by

    mahdi nematshahi
    6 views6 pages
    The document discusses notations used in supervised discriminative learning. It notes that labeled training data contains inputs x and corresponding classifications or regressions y. A neural network is parameterized by weights θ and represented as a function fθ that maps inputs to outputs. The network is trained discriminatively by minimizing a loss function on the training set or per example, plus a regularization term. For regression networks specifically, mean squared error is commonly used as the loss function with L2 regularization.

    Original Description:

    Original Title

    0_1_intro_Notations

    Copyright

    © © All Rights Reserved

    Available Formats

    PDF, TXT or read online from Scribd

    Did you find this document useful?

    Is this content inappropriate?

    Report this Document
    The document discusses notations used in supervised discriminative learning. It notes that labeled training data contains inputs x and corresponding classifications or regressions y. A neural network is parameterized by weights θ and represented as a function fθ that maps inputs to outputs. The network is trained discriminatively by minimizing a loss function on the training set or per example, plus a regularization term. For regression networks specifically, mean squared error is commonly used as the loss function with L2 regularization.

    Copyright:

    © All Rights Reserved
    Download as PDF, TXT or read online from Scribd
    Flag for inappropriate content
    6 views6 pages

    0 1 Intro Notations

    Uploaded by

    mahdi nematshahi
    The document discusses notations used in supervised discriminative learning. It notes that labeled training data contains inputs x and corresponding classifications or regressions y. A neural network is parameterized by weights θ and represented as a function fθ that maps inputs to outputs. The network is trained discriminatively by minimizing a loss function on the training set or per example, plus a regularization term. For regression networks specifically, mean squared error is commonly used as the loss function with L2 regularization.

    Copyright:

    © All Rights Reserved
    Download as PDF, TXT or read online from Scribd
    Flag for inappropriate content
    of 6

    Notations 

    (supervised discriminative learning)
    Notation 

    Supervised Learning

    Labelled training data:        𝒟 𝐱 ,𝑦 :

    • input 𝐱∈𝒳 ℝ
    • classification: 𝑦∈𝒴 0,1, … , 𝑘 1
    • regression: 𝑦∈𝒴 ℝ

    We say, a neural network is parametrized by  𝜽 ∈ Θ ℝ (its weights) 
    and is represented by 𝑓𝜽 : 𝒳 → 𝒴

    9/4/2022 KTH ‐ DD2412 2


    Notation

    A neural network function

    y 𝑓𝜽 𝐱

    𝐱 𝑦

    9/4/2022 KTH ‐ DD2412 3


    Notation

    Training a deep network discriminatively

    𝜽∗ argmin𝜽 ℒ 𝒟 loss function on the whole training set

    𝜽∗ argmin𝜽 𝑙 𝑓𝜽 𝐱 , 𝑦 loss function per training example

    𝜽∗ argmin𝜽 𝑙 𝑓𝜽 𝐱 , 𝑦 Ω 𝜽 a regularization term

    9/4/2022 KTH ‐ DD2412 4


    Notation

    Example: Deep Regression Networks

    𝜽∗ argmin𝜽 𝑙 𝑓𝜽 𝐱 , 𝑦 Ω 𝜽

    mean squared error (MSE)

    𝜽∗ argmin𝜽 𝑓𝜽 𝐱 𝑦 𝜽

    L2 regularization (similar to weight decay)

    9/4/2022 KTH ‐ DD2412 5


    Reflection/Discussion point 

    What do the following two 
    statements mean?
    • Standard deep (regression) networks give a 
    point estimate prediction 
    i.e.,  𝑦 𝑎𝑟𝑔𝑚𝑎𝑥 𝑃 𝑦|𝐱

    • Standard deep network weights are a point 
    3 minutes
    estimate of the parameter distribution 
    i.e.,  𝜽∗ 𝑎𝑟𝑔𝑚𝑎𝑥𝜽 𝑃 𝜽 𝒟

    • What is the alternative?
    6

    You might also like