MADEIRA

INTERNATIONAL
WORKSHOP
IN MACHINE
LEARNING

Premium sponsor: Gold sponsor: Bronze sponsor:

2022 1
 CONTENTS

 Problems with RNN Recap

 Gated Recurrent Unit
 Difference between LSTMs and GRUs
 GRUs in Practice

. 2
 PROBLEMS WITH RNN

 Unable to learn long term dependencies.

 E.g. The cat who already ate a bunch of food, was full.
The cat who … was full.

. 3
 PROBLEMS WITH RNN

 Vanishing gradient problem.

Loss(pred, ground truth)

. 4
 GATED RECURRENT UNIT (GRU)

 Three main components:

 Hidden state-responsible for preserving long term dependencies.
 Update Gate-For deciding when to update the memory cell.
 Reset Gate-relevancy of the previous memory cell (state=current
hiddens state).

. 5
GRU (UPDATE AND RESET GATES)

6
GRU (CANDIDATE HIDDEN STATE)

7
GRU (FULL WORKFLOW)

𝑐𝑎𝑛𝑑𝑖𝑑𝑎𝑡𝑒 ℎ𝑖𝑑𝑑𝑒𝑛 𝑠𝑡𝑎𝑡𝑒, 𝑖𝑓 𝑢𝑝𝑑𝑎𝑡𝑒 𝑔𝑎𝑡𝑒 = 1

𝑐𝑢𝑟𝑟𝑒𝑛𝑡 ℎ𝑖𝑑𝑑𝑒𝑛 𝑠𝑡𝑎𝑡𝑒 = ቊ
𝑝𝑟𝑒𝑣𝑖𝑜𝑢𝑠 ℎ𝑖𝑑𝑑𝑒𝑛 𝑠𝑡𝑎𝑡𝑒, 𝑖𝑓 𝑢𝑝𝑑𝑎𝑡𝑒 𝑔𝑎𝑡𝑒 = 0
8
 GRU AND LSTM COMPARISON

LSTM GRU

Three gates: input, forget, Two gates: update and reset

. and output 9
 GATE ANALOGIES (GRU, LSTMS)

LSTM GRU

• Input Gate • Input Gate (hidden state=cell state)

• Forget Gate • Forget Gate (1-update gate)

• Output Gate
• Output gate (Not required)
(Cell state=current hidden state)

. 10
GATED RECURRENT
UNIT IN PRACTICE

11
Thank You

12