Deep Learning:

Trends, successes, and opportunities

Abdel-rahman Mohamed
Microsoft Research

Talk@Cairo University 4/22/17

 Artificial Intelligence (AI): Where we are
AI: From research papers to top newspapers

3
AI: Huge potential predicted by major consulting firms

4
AI economic potential: Time to double GDP
(Accenture’s 2016 report on AI)

5
AI: A national priority – USA

6
AI: A national priority – USA

7
AI: A national priority – Canada

8
What is AI ?

• AI is a set of technologies that can:

o Sense: E.g. Automatic Speech Recognition, Computer Vision
o Comprehend: E.g. Natural Language Processing (NLP) applications
o Act: E.g. Recommendation systems, Robotics
o Learn from experience: E.g. knowledge base construction

• Artificial General Intelligence (AGI) / Full AI: Could successfully

perform any intellectual task that a human being can do:
o Solve multiple problems using the same model/agent, e.g. Atari
o Generalize to unseen events/situations.
o ….
9
AI impact on our lives: Automation

10
AI impact on our lives: Automation (Mckinsey Global Institute)

11
AI impact on our lives: Augmenting human abilities

12
 Deep Learning: The workhorse of AI
Deep Neural Networks (DNNs): The second appearance!

• Why:
o Multiple layers of distributed representation.
o Shared representation across large number of classes.
o Marginalize unwanted information in the inputs.
o Benefit from adding more data.
• Impressive successes over a diverse set of tasks:
o Traditional tasks: ASR, MT, NLP, CV
o Games: Atari, GO, Poker, Doom
o Many more
• Only one set of skills to work in all these tasks!
• Lots of commercial products Input features
14
Why now? Intersection of many trends in other fields

• Data: Unprecedented amount of stored data

• Computational power: Cheap FLOPS (floating-point operations per
second)
o CPUs, GPUs, CUDA, Cloud computing
• Open Source code: Free individual and industrial scale codebases
o Github, Tensorflow, Torch, Caffe, Theano, CNTK, Kaldi
• Educational resources: Availability of free high quality educational
resources
o Arxiv, Coursera, edx, Udacity

15
 Neural Nets: A crash course
Forward pass

Sigmoid:

RELU:

Softmax:
Input features
17
Objective functions
• Sum-of-squares error:

• Cross entropy (CE) error (multiple classes):

• CE (multiple independent attributes):

Input features
18
Backward pass

• NNs are directed graphs:

o In the forward pass we sum over all incoming arrows.
o In the backward pass we sum over all outgoing arrows.
• Both passes are implemented by matrix multiplication and
elementwise operations. 19
Optimization: Bag of tricks!

• Mini-batch Gradient descent

• Momentum

• Proper weight initialization

• Learning rate annealing

Pictures from “NNs for pattern recognition” by
• AdaGrad, AdaDelta, RMSProp, Adam Chris Bishop 20
Convolutional Neural Networks (CNNs)

• Problem: Feedforward NNs aren’t efficient for translated inputs.

• Solution: Search the input space for patterns, hence Convolutional.
• The pooling operation ensures invariance to input shifts.
• VGG net: Go deep with small kernel sizes.
• ResNet: Go even deeper with skip connections.

Pictures from Stanford cs231n 21

Recurrent Neural Networks (RNNs)

• Problem: How to handle sequential depencies?

• Solution: Process individual input time-steps + hidden-hidden
weights, hence Recurrent.
• Problem: Vanishing gradients over time due large sequence lengths.
• Solution: Use hidden units with read/write controls to hold information

22
How kids learn to play football?

• Supervised learning ?
o For every leg move, do they get information on how good or bad it was?
o For each data point, there is a corresponding target output to be predicted.
• Unsupervised learning ?
o By watching games, they are able to learn interesting moves and apply them?
o Agents are capable of discovering important patterns, as well as generating
them.
• Reinforcement learning ?
o They get coach feedback at the end of the day?
o Agents act to maximize the expected overall return, i.e. positive feedback.
• All of the above ? 23
Deep Learning challenges

• Interpretability
• Bag of tricks: Initialization, curriculum, regularization, hyper-
parameter tuning
• Computational cost
• Fooling network
• No convergence or performance guarantees

24
AI challenges

• Unsupervised learning
• Reinforcement learning
• Generalizing to unseen events
• Acquiring new knowledge from environment
• Data vs. user privacy
• Representing semantics: From symbolic transduction to semantics
understanding

25
Thank you

26
Types of Careers in AI
• RSDE
• Data scientist
• Applied researcher
• Researcher
• high performance compute / cloud engineer

27
Three direction to move forward
• Four components needed for AI:
• Data
• Toolkits/code
• Papers/education
• Compute
• Have a cloud service for University students and startup companies
• AI Education – offering mentorship for students on what to study, peer
mentoring etc
• Searching for directions where AI could be of benefit for helping local
businesses, social problems etc.

28