Course Information

Course Name Deep Learning Seminar

Semester Winter 2016/17
Location & Time University of Haifa
Education Building, Class 463, Tuesdays 10:15-11:45.
Lecturer Dr. Amir Adler
E-mail adleram@cs.technion.ac.il
Syllabus (1) Review of Deep Learning applications and architectures:
fully connected networks, convolutional networks (CNN),
recurrent networks (RNN), Stochastic Gradient Descent,
Back-Propagation, and Regularization Techniques.
(2) Students Lectures (40 minutes each), based on state-of-the-
art papers.
Pre-requisites: Machine Learning or Algorithms
Grading: 85% Lecture + 15% Class Presence & Participation

1
 Course Plan

Topic Week

Introduction to Deep Learning: (1) Review of computer vison, natural language 1

processing, and speech processing applications. (2) Evolution of neural networks
technology, and milestones during past 10 years,
Machine Learning concepts review: supervised, unsupervised, semi-supervised 2
learning, representation learning, reinforcement learning.
Fully connected and convolutional networks in detail. Multi-layer perceptron, 3
non-linear activation units, SoftMax layer, convolutional layer, max-pooling,
average-pooling.
Stochastic Gradient Descent (SGD) and loss functions (mean-squared error, 4
negative log likelihood). SGD with momentum.
The Back-Propagation algorithm 5
Recurrent Neural Networks (RNN): Basic RNN structure, characters prediction 6
and image captioning with RNNs, exploding and vanishing gradients problem,
Long Short Term Memory (LSTM), Gated Recurrent Unit (GRU).
Students Lectures 7
Students Lectures 8
Students Lectures 9
Students Lectures 10
Students Lectures 11
Students Lectures 12
Students Lectures 13
Students Lectures 14

Text Book

1. Ian Goodfellow and Yoshua Bengio and Aaron Courville, "Deep Learning", MIT Press
2016. Available online: http://www.deeplearningbook.org/

2
Lectures

No. Paper Name Conference Link

/Journal
1 Beyond short snippets deep CVPR 2015
networks for video classification
large scale video classification with CVPR 2014
convolutional neural networks
https://arxiv.org/abs/1503.08909
http://vision.stanford.edu/pdf/karpat
hy14.pdf

2 Deep Residual Learning for Image CVPR 2016 https://arxiv.org/abs/1512.03385

Recognition
Identity Mappings in Deep ECCV 2016 https://arxiv.org/abs/1603.05027
Residual Networks
3 SSD: Single Shot MultiBox ECCV 2016 https://arxiv.org/abs/1512.02325
Detector
Faster R-CNN: Towards Real-Time NIPS 2015 https://arxiv.org/abs/1506.01497
Object Detection with Region
Proposal Networks
4 Rich feature hierarchies for CVPR 2014 https://arxiv.org/abs/1311.2524
accurate object detection and
semantic segmentation
OverFeat: Integrated Recognition, ICLR 2014 https://arxiv.org/pdf/1312.6229.pdf
Localization and Detection using
Convolutional Networks
5 Show, Attend and Tell: Neural ICML 2014 https://arxiv.org/pdf/1502.03044v3.
Image Caption Generation with pdf
Visual Attention
Unifying Visual-Semantic NIPS 2014 https://arxiv.org/abs/1411.2539
Embeddings with Multimodal
Neural Language Models
6 Mastering the game of Go with Nature 2016 http://www.nature.com/nature/journ
deep neural networks and tree al/v529/n7587/full/nature16961.ht
search (by Google DeepMind) ml

Human level control through deep Nature 2015 http://www.nature.com/nature/journ

reinforcement learning (by Google al/v518/n7540/full/nature14236.ht
DeepMind) ml

7 Learning Hierarchical Features for PAMI 2013 http://yann.lecun.com/exdb/publis/p

Scene Labeling
Indoor Semantic Segmentation
using depth information
8 Understanding the difficulty of
training deep feedforward neural
3
df/farabet-pami-13.pdf
ICLR 2013 https://arxiv.org/abs/1301.3572
AISTATS http://jmlr.org/proceedings/papers/v
2010 9/glorot10a/glorot10a.pdf
 networks
Batch Normalization: Accelerating
Deep Network Training by
Reducing
Internal Covariate Shift
9 Learning spatiotemporal features
with 3d convolutional networks
sequence to sequence video to text
10 Very Deep Convolutional Networks ICLR 2015
for Large-Scale Image Rcognition
Going Deeper With Convolutions CVPR 2015
11 Object Contour Detection with a
Fully Convolutional Encoder-
Decoder Network
Learning a deep convolutional
network for image super-resolution
12 Explain Images with Multimodal
Recurrent Neural Networks
Long-term Recurrent Convolutional
Networks for Visual Recognition
and Description
13 Zero-Shot Learning Through Cross-
Modal Transfer
DeViSE: A Deep Visual-Semantic
Embedding Model
14 Video Super-Resolution With
Convolutional Neural Networks
Deep multi-scale video prediction
beyond mean square error
15 Learning with a Wasserstein Loss
Fast and Robust Earth Mover’s
Distances (background paper)
ICML 2015 http://jmlr.org/proceedings/papers/v
37/ioffe15.pdf
ICCV 2015 https://arxiv.org/pdf/1412.0767.pdf
CVPR 2015 https://arxiv.org/abs/1505.00487
https://arxiv.org/pdf/1409.1556.pdf
https://arxiv.org/abs/1409.4842
CVPR 2016 http://www.cv-
foundation.org/openaccess/content_
cvpr_2016/papers/Yang_Object_Co
ntour_Detection_CVPR_2016_pap
er.pdf
ECCV 2014 http://citeseerx.ist.psu.edu/viewdoc/
download?doi=10.1.1.642.1999&re
p=rep1&type=pdf
NIPS 2014 https://arxiv.org/pdf/1410.1090.pdf
Workshop
CVPR 2015 http://www.icsi.berkeley.edu/pubs/
vision/longtermrecurrent15.pdf
NIPS 2013 http://nlp.stanford.edu/pubs/Socher
GanjooManningNg_NIPS2013.pdf
NIPS 2013 http://static.googleusercontent.com/
media/research.google.com/en//pub
s/archive/41473.pdf
IEEE Trans. http://ivpl.eecs.northwestern.edu/sit
on Comp. es/default/files/07444187.pdf
Imaging
2016
ICLR 2016 https://arxiv.org/pdf/1511.05440v6.
pdf
NIPS 2015 https://arxiv.org/pdf/1506.05439v3.
pdf
ICCV 2009 http://www.ariel.ac.il/sites/ofirpele/
publications/ICCV2009.pdf