SUPERVISED LEARNING
PART 1
We will consider two types of learning, supervised and
unsupervised learning. Before even explaining each of those
terms, what do we mean by learning in our introductory
artificial intelligence course? Here, learning refers to acquiring
knowledge from data with the objective of making better
decisions than without using that data. By decisions, we can
understand for example managerial decisions like “should I
invest in this project?” but also platform automated decisions
like “What is shown to you on the landing page of Netflix?”.
With our society, that is our personal lives, businesses,
governments being highly digitalized, unprecedented
amounts of data are generated and stored in real time. These
massive amounts of data, we can call this effectively Big Data,
can be used for improved decision making.
Of course, having access to such data is interesting but
it becomes only useful if we can identify clearly the
relevant data sources and then quickly learn from it
to make decisions. Yes, you might already be thinking
that this sounds complicated. In fact; massive datasets
and swift decision making will be possible only thanks
to the processing power of computers. Highly scalable
computational resources and full digitalization of our society
are the main reason for the so-called artificial intelligence
revolution. Ok, now we have computing power and data, how
can you learn to make better decisions?
Example: Lyn & Paris real estate
The objective of this video is to understand the principles
of supervised learning. The best way to start is with an
example. Lyn has just been accepted to the prestigious
master in management program at Essec Business School.
Lyn currently lives in Brussels with her parents and
accommodation becomes a pressing issue. Since she plans
to stay longer in France than the duration of her studies,
living and working in France is her childhood dream, she
considers buying an apartment in Paris. Lyn knows that the
Paris real estate market is different from the one in Brussels,
but such a statement will not help her making an informed
decision.
Luckily, Lyn followed a bachelor’s degree in computer
science at Louvain University with a major in data science.
Time to put her academic knowledge in practice. First, what
is the question she would like to answer? Well, she would
JEROEN ROMBOUTS - PROFESSOR OF ECONOMETRICS AND STATISTICS
like to understand what type of apartments in Paris are
for sale on the market, and what explains the asking price
differences of these apartments.
So the question is: what determines the asking price of an
apartment in Paris? To answer this question in order to make
an informed decision of buying an apartment, Lyn needs
to build a dataset. What is a dataset? At this stage, we can
imagine a matrix with N lines or observations and P columns or
variables. Yes you can think of an Excel or Google sheet in fact.
The target variable
Supervised learning is characterized by the fact that in the
dataset one of the P variables is a target variable. The target
variable can also be called response or dependent variable.
In Lyn’s case, the target variable is the asking price for a
specific apartment in Paris. In other cases, it could be sales
of a product, buy-not buy of a product or service, your face
being recognized or not to unlock your smartphone, etc. As
you can see, the possible target variable can be any variable
that is of interest as long as it is well defined or labeled. In
current applications, supervised learning problems are the
most common ones.
What do we want to do with the target variable? We want
to predict it with the remaining available P-1 available
variables in the dataset. The latter variables are often called
feature or independent variables, or just shortly features.
Unfortunately, not all feature variables are relevant to predict
the target variable, and some important ones might be even
missing in the dataset. We will need to search for those
features which are related to the target variable. Note that we
deliberately write that we wish to predict the target variable,
and not to explain it. At this stage, we would like to find an
algorithm that predicts well, irrespective of the fact that
any causal relationship exists. In this sense, we can call the
supervised learning algorithms prediction machines.
Ok, Lyn has a supervised learning problem with asking
price as a target variable. The features are the variables
that potentially influence the apartment asking price. For
example, the number of bedrooms and bathrooms, the size
in squared meters, age of the building, etc. Lyn decides
to activate her web scraping skills and extracts 1000
apartments for sale on a popular real estate website. With
2 CORE MODULE I - FOUNDATIONAL TECHNOLOGIES OF AI
modern data science open source software packages this is
trivial to do. Just Google for it. The resulting dataset has not
only the features we discussed before, but also contains the
apartment floor (not all apartment buildings have elevators
in Paris…), heating system, is there a balcony, district in
Paris.
The supervised learning algorithm will connect the feature
variables to the target variable. We will omit the details how
this can be formalized mathematically, but it is important
to know that there are many ways to do this and you cannot
really tell in advance which method is best. The supervised
learning algorithms you should have heard about and which
are available in standard data science software packages are:
Linear regression, logistic regression, decision tree, random
forest, and neural network.
Split the dataset
An interesting characteristic of supervised learning is that
the original dataset can be split into training and test
datasets. The training dataset is used for estimating or
training the algorithm. This means that the parameters
or decision rules that describe the algorithm are chosen
such that a loss function is minimized. This loss function is
directly related to the prediction error the algorithm makes
on the training dataset. Optimization of this loss function is
done by means of a data science software program such as
Python.
What happens when we train the algorithm on the entire
dataset without splitting it? Then we run the risk of
overfitting which means that the algorithm will produce very
small prediction errors on the existing data but is bound to
make large errors when new data is fed into the algorithm.
JEROEN ROMBOUTS - PROFESSOR OF ECONOMETRICS AND STATISTICS
So in practice, we split the dataset in two parts. The part
of the original dataset that has not been used to train is
called the test dataset. Using only the feature variables,
predictions are made with the fitted algorithm. Doing this,
we can contrast the predictions with the true target variable
values from the test dataset and compute prediction errors
on the test dataset. Then the best algorithm is the one with
the smallest prediction errors. Once the best algorithm is
identified, it can be used for truly out of sample prediction.
That is, on new observations for which only the features are
observed.
Lyn, decides to use 800 apartments to train different
candidate algorithm and then to test them on the remaining
200 apartments. It turns out that linear regression works
best on this data, which she particularly appreciates since
linear regressions are fairly simple to interpret. In terms of
prediction, Lyn can now assess for example the asking price
for an apartment in the 9th district in Paris with one room,
30 squared meters, and a balcony. And she can do that even
before visiting the apartment with a real estate agent, so that
she can evaluate if the current asking price is in line with the
market.
3 CORE MODULE I - FOUNDATIONAL TECHNOLOGIES OF AI
 SUPERVISED LEARNING
PART 2
Let me now give more precise details on the different
supervised learning algorithms:
linear regression
Linear regression can be applied when the target variable
takes infinitely many possible values. The asking price
for an apartment in Lyn’s case but other examples are:
your electricity consumption sampled every 15 minutes
with smart meters, or sales for different grocery stores of
a supermarket chain. A linear regression will explain the
expected value of the target variable as a function of the
feature variables. The parameters are extremely easy and
fast to estimate even in the case of large datasets, and
they can be typically directly interpreted as sensitivities of
each feature on the target variable. So everything is easy to
interpret, which is the reason why this model is extremely
popular. In terms of predictions, the linear regression model
is easy to predict given observed features. For example,
give me the characteristics of a new to open grocery store
of a supermarket chain and I will tell you the expected
amount of sales. Regression methods are new? Not at all, it
was invented in the 1800s by Gauss and Legendre, and in
particular Galton in 1877.
Logistic regression
Logistic regression type models can be applied when the
target variable takes limited possible values, the most
standard case being two values. For example, related to
conversion rates, a product is bought or not on a website. Or
related to churn, you stay client or not at your telco company.
The model parameters are relatively easy to estimate, and
importance of each feature to the probability of observing one
of the outcomes can be computed. In terms of predictions,
the logistic model allows classifying the target variable when
only the features are observed. For example, given your socio
demographic characteristics, I will obtain the probability that
you will buy a product online. If the probability is high, there
is no need pushing a promotion to you.
JEROEN ROMBOUTS - PROFESSOR OF ECONOMETRICS AND STATISTICS
The previous two types of models allow identifying the
significant feature variables that drive the target variable.
This is what we call attribution, that is we can tell which
features explain the target variable. On top of that, they
allow to quantify what is the expected change in the target
variable when one of the features change. Given their
construction, the above models are called smooth.
The next methods that I will explain you now are pure
prediction models. This means that the impact of the
features on the target is often not easy to assess since they
are not smooth. The only purpose is maximize prediction
performance.
Decision trees
Decision trees fall in the category of what we call
nonparametric methods. There are no parameters to
estimate, only cutoff thresholds. They apply both to
quantitative and qualitative target variables. Decision trees
are easy to interpret, since they basically can be graphically
thought of as a tree with branches. In the simplest case of
binary splits, each branch splits in two according to a value
to exceed for one of the feature variables. As usual with
easy to interpret algorithms, the machinery in the back is
quite advanced. I will spare you the details in this video.
Just imagine that to construct the first branch of the tree
you need to consider all the features and all possible cutoff
values, quite a computationally heavy task. I can testify that
you can start using your computer as home heating when
the number of observations and features is large.
Random forests
Random forests are, well as the term alludes to, re-
applying many decision trees to the same problem. Each of
those trees are constructed using a bootstrap resampling
technique and uses only a random subset of the features.
In case of a classification problem, the final prediction is
determined by the majority vote over all trees. This method
is relatively recently proposed by Breiman in 2001.
Where is artificial intelligence? In the narrow sense, AI
is to be interpreted as intelligence reproduced by neural
4 CORE MODULE I - FOUNDATIONAL TECHNOLOGIES OF AI
networks or its extensions. It can be used in both regression
or classification problems. That means AI needs a target
variable in order to train the deep neural networks. The
training of neural networks requires tuning often hundreds
of thousands parameters and therefore a lot of data are
necessary to obtain small prediction error rates. Think about
image recognition that is trained on massive sets of pictures,
and then performs amazingly well most of the time, but
can at some point make embarrassing mistakes. In terms
of machinery in the back, you can imagine a sophisticated
logistic regression algorithm.

To sum up, we have reviewed the most important supervised

learning algorithms. The training and test datasets allows
to gauge performance among different algorithms. The best
one will necessarily always be superior and therefore need
continuous monitoring. In fact, we always need to watch
out for concept drift. Concept drift means that the data
generating mechanism changes over time.

Lyn has scrapped a Paris real estate website at the beginning

of the COVID pandemic to get her dataset. However, many
people working in Paris have decided to live elsewhere in
France to have more space while teleworking. Some workers
come only one or two days a week to Paris and find it more
flexible to stay in a hotel. The demand for apartments might
therefore go down, and therefore asking prices as well. In
such a situation, Lyn’s prediction model has the risk of
systematically overpricing the apartment prices. How can
we solve such a problem? Exactly, get new data and do
the supervised learning exercise again to determine which
algorithm works best.

JEROEN ROMBOUTS - PROFESSOR OF ECONOMETRICS AND STATISTICS 5 CORE MODULE I - FOUNDATIONAL TECHNOLOGIES OF AI