Scribd Logo
Upload

Welcome to Scribd!

  • Neuron Model: Sigmoidal Function

    Uploaded by

    Pooja Patwari
    3 views15 pages
    1) The document describes the basics of neural networks including the sigmoid activation function, backpropagation learning algorithm, average squared error cost function, and weight update rule. 2) The backpropagation algorithm adjusts weights to minimize average squared error by propagating error signals backward from output to hidden layers using the gradient descent method. 3) Weight updates are calculated using the delta rule, which takes a step in the direction opposite the gradient of the error function with respect to the weights. Hidden neuron gradients are determined recursively in terms of downstream neurons.

    Original Description:

    Bilinear Classifier Excercise

    Original Title

    BP2

    Copyright

    © © All Rights Reserved

    Available Formats

    PDF, TXT or read online from Scribd

    Did you find this document useful?

    Is this content inappropriate?

    Report this Document
    1) The document describes the basics of neural networks including the sigmoid activation function, backpropagation learning algorithm, average squared error cost function, and weight update rule. 2) The backpropagation algorithm adjusts weights to minimize average squared error by propagating error signals backward from output to hidden layers using the gradient descent method. 3) Weight updates are calculated using the delta rule, which takes a step in the direction opposite the gradient of the error function with respect to the weights. Hidden neuron gradients are determined recursively in terms of downstream neurons.

    Copyright:

    © All Rights Reserved
    Download as PDF, TXT or read online from Scribd
    Flag for inappropriate content
    3 views15 pages

    Neuron Model: Sigmoidal Function

    Uploaded by

    Pooja Patwari
    1) The document describes the basics of neural networks including the sigmoid activation function, backpropagation learning algorithm, average squared error cost function, and weight update rule. 2) The backpropagation algorithm adjusts weights to minimize average squared error by propagating error signals backward from output to hidden layers using the gradient descent method. 3) Weight updates are calculated using the delta rule, which takes a step in the direction opposite the gradient of the error function with respect to the weights. Hidden neuron gradients are determined recursively in terms of downstream neurons.

    Copyright:

    © All Rights Reserved
    Download as PDF, TXT or read online from Scribd
    Flag for inappropriate content
    of 15

    NEURON MODEL

    • Sigmoidal Function

    Increasing a

    -10 -8 -6 -4 -2 2 4 6 8 10

    • induced field of neuron j


    • Most common form of activation function
    • a →∞ ⇒ ϕ → threshold function
    • Differentiable
    1
    LEARNING ALGORITHM

    • Back-propagation algorithm
    Function signals
    Forward Step

    Error signals
    Backward Step

    • It adjusts the weights of the NN in order to


    minimize the average squared error.
    2
    Average Squared Error
    • Error signal of output neuron j at presentation of n-th
    training example:
    C: Set of
    • Total energy at time n: neurons
    in output
    • Average squared error: layer
    N: size of
    • Measure of learning training set
    performance:

    • Goal: Adjust weights of NN to minimize EAV

    3
    Notation

    Error at output of neuron j


    Output of neuron j
    Induced local
    field of neuron j

    4
    Weight Update Rule
    Update rule is based on the gradient descent method
    take a step in the direction yielding the maximum
    decrease of E

    Step in direction opposite to the gradient

    With weight associated to the link from neuron i


    to neuron j
    5
    The derivation

    6
    Definition of the Local
    Gradient of neuron j

    Local Gradient

    We obtain

    because

    7
    Update Rule
    • We obtain

    because

    8
    Compute local gradient of
    neuron j

    • The key factor is the calculation of ej


    • There are two cases:
    – Case 1): j is a output neuron
    – Case 2): j is a hidden neuron

    9
    Error ej of output neuron
    • Case 1: j output neuron

    Then

    10
    Local gradient of hidden
    neuron
    • Case 2: j hidden neuron

    • the local gradient for neuron j is recursively


    determined in terms of the local gradients of
    all neurons to which neuron j is directly
    connected

    11
    Use the Chain Rule

    from

    We obtain

    12
    Local Gradient of hidden
    neuron j
    Hence

    w1j e1 Signal-flow
    δ ϕ’(v1) graph of
    δ ϕ’(vj) back-propag
    wkj 1
    ek ation error
    j
    δ ϕ’(vk)
    wm j signals to
    k neuron j
    em
    δ ϕ’(vm)
    m
    13
    Delta Rule
    • Delta rule Δwji = ηδj yi

    IF j output node

    IF j hidden node

    C: Set of neurons in the layer following the one


    containing j

    14
    Local Gradient of neurons

    a>0

    if j hidden node

    If j output node

    15

    You might also like