Advanced Machine
Learning (AML)
Outline
• 12 weeks of teaching split into the following groups:
• Introduction to Machine learning/Deep Learning
• Regression, logistic regression, Multi-class classification
• Convolutional Neural Networks
• Time series prediction
• Fault identification / Anomaly detection
Marco Star
• Postdoc at IME
• Mechatronics
• Office: block A, 4th floor,
• Email: mstar@sdu.dk
Tools for the course
• Theory based on online courses and books (Machine
Learning and Deep Learning Specialisation from Stanford
University)
• Books:
• Dive into Deep Learning,
• Understanding Deep Learning
• Practice: Based on various resources & assignments from the
aforementioned
• Use of python for programming
• Use of jupyter notebooks for assignments
Resources • Free books and resources:
• https://d2l.ai/
• https://udlbook.github.io/udlbo
ok/
• https://www.bishopbook.com/
• Online courses:
• Machine learning courses
on Coursera by Andrew Ng,
edX etc.
• https://course.fast.ai/
Practice Sessions
• After lecture practice sessions
• As a general rule you can treat the problems like problems you would need to solve for a job
• Hence, feel free to Google and look things up online, but remember ultimately you need to
understand and be able to justify the decisions you made to find the solution
Current applications (google results)
Machine Learning
Necessary for machine learning:
• Input data points
• Expected output
• A way to measure how good is the
algorithm performing
Supervised learning
Define a mapping from input to output
Learn this mapping from paired input/output data examples
Regression
• Univariate regression problem (one output, real value)
• Fully connected network
Text classification
• Binary classification problem (two discrete classes)
• Transformer network
Music genre classification
• Multiclass classification problem (discrete classes, >2 possible values)
• Recurrent neural network (RNN)
Image classification
• Multiclass classification problem (discrete classes, >2 possible classes)
• Convolutional network
What is a supervised learning model?
An equation relating input (age) to output (height)
Search through family of possible equations to find one that fits training data well
What is a supervised learning model?
Deep neural networks are just a very flexible family of equations
Fitting deep neural networks (optimizing their parameters) = “Deep Learning”
Terms
Regression = continuous numbers as output
Classification = discrete classes as output
Two class and multiclass classification treated
differently
Univariate = one output
Multivariate = more than one output
Translation
Image captioning
Image generation from text
What do these examples have in common?
Very complex relationship between input and output
Sometimes may be many possible valid answers
But outputs (and sometimes inputs) obey rules
Language obeys Natural images also
grammatical rules have “rules”
Unsupervised Learning
Learning about a dataset without labels
Clustering
Finding outliers
Generating new examples
Filling in missing data
DeepCluster: Deep Clustering for Unsupervised Learning of Visual Features (Caron et al., 2018)
DeepCluster: Deep Clustering for Unsupervised Learning of Visual Features (Caron et al., 2018)
Unsupervised Learning
Learning about a dataset without labels
e.g., clustering
Generative models can create examples
e.g., generative adversarial networks
Unsupervised Learning
Learning about a dataset without labels
e.g., clustering
Generative models can create examples
e.g., generative adversarial networks
PGMs learn distribution over data
e.g., variational autoencoders,
e.g., normalizing flows,
e.g., diffusion models
Generative models
https://thesecatsdonotexist.com/
Latent variables
Why should this work?
Interpolation
Interpolation
Image from: Kingma, D.P., & Welling, M. (2013). Auto-Encoding Variational Bayes. CoRR, abs/1312.6114.
Conditional synthesis
Note about Interpolation
• Machine learning is pretty much just fancy interpolation
• For example, training a model to predict height based on a person's age,
Height
Age
Note about Interpolation
• Machine learning is pretty much just fancy interpolation
• For example, training a model to predict height based on a person's age,
Height
Age
Note about Interpolation
• Machine learning is pretty much just fancy interpolation
• For example, training a model to predict height based on a person's age,
Height
Age
Supervised learning overview
Supervised learning model = mapping from one or more inputs to one or more
outputs
Model is a family of equations
blank
Computing the inputs from the outputs = inference
Model also includes parameters
Parameters affect outcome of equation
Loss function
Training dataset of I pairs of input/output examples:
Loss function or cost function measures how bad model is:
or for short:
Loss function
Training dataset of I pairs of input/output examples:
Loss function or cost function measures how bad model is:
or for short:
Returns a scalar that is smaller
when model maps inputs to
outputs better
Training
Loss function:
Returns a scalar that is smaller
when model maps inputs to
outputs better
Find the parameters that minimize the loss:
Testing
To test the model, run on a separate test dataset of input / output
pairs
See how well it generalizes to new data
Housing price
prediction
Example: 1D Linear regression model
Model:
Parameters
y-offset
slope
Example: 1D Linear regression model
Model:
Parameters
y-offset
slope
Example: 1D Linear regression model
Model:
Parameters
y-offset
slope
Example: 1D Linear regression model
Model:
Parameters
y-offset
slope
Example: 1D Linear regression training data
Example: 1D Linear regression training data
Loss function:
“Least squares loss
function”
Example: 1D Linear regression loss function
Loss function:
“Least squares loss
function”
Example: 1D Linear regression loss function
Loss function:
“Least squares loss
function”
Example: 1D Linear regression loss function
Loss function:
“Least squares loss
function”
Example: 1D Linear regression loss function
Loss function:
“Least squares loss
function”
Example: 1D Linear regression loss function
Example: 1D Linear regression loss function
Example: 1D Linear regression loss function
Example: 1D Linear regression loss function
Example: 1D Linear regression training
Example: 1D Linear regression training
Example: 1D Linear regression training
Example: 1D Linear regression training
Example: 1D Linear regression training
This technique is known as gradient descent
Content credits
Understanding Deep Learning slides
Current frameworks
Practical part
Installing Conda 1. Download the latest install for your architecture
https://docs.conda.io/en/latest/miniconda.html
If conda is installed correctly (might need a logout and
login, or restart), you should be able to see the output
when typing conda into your terminal.
To list the currently installed environments just type
conda env list
It should currently just show the “base” environment
installed.
Installing Jupyter Notebook (default)
conda install -c conda-forge notebook
Installing Jupyter Lab conda install -c conda-forge nb_conda_kernels
Installing Jupyter Lab
conda install -c conda-forge jupyterlab
Installing Jupyter Notebook extensions
Nice to have:Jupyter Notebook extensions, which support a lot of
autocompletion, additional information, and in general things that make
your life easier. A good default setting is included with the following
install command:
pip install jupyter_contrib_nbextensions
add the pip package manager to the base:
conda install pip
Start Jupiter lab:
jupyter lab
Installing necessary
packages Packages can easily be installed later through command line. For example,
the numpy package can be installed like this:
conda install -c conda-forge numpy
or
pip install numpy
pip install matplotlib
Open and run the
jupyter notebooks 1. Background mathematics
2. Supervised linear regression
in Week1.zip
