NOTES UNIT 3 FOML

UNIT-III UNSUPERVISED LEARNING AND TREE MODELS

Unsupervised Learning Techniques: Clustering: K-Means - Limits of K-Means - Using
Clustering for Image Segmentation - Using Clustering for Pre-processing - Using
Clustering for Semi-Supervised Learning - DBSCAN - Other Clustering Algorithms.
Ensemble learning and Random Forests: Voting Classifiers - Bagging and Pasting -
Random Forests - Extra-Trees - Feature Importance – Boosting AdaBoost - Gradient
Boosting

Unsupervised Learning Techniques

Note: This is introduction. So read what is needed from syllabus. But complete clustering
will be there in this section.

As the name suggests, unsupervised learning is a machine learning technique in which

models are not supervised using training dataset. Instead, models itself find the hidden
patterns and insights from the given data. It can be compared to learning which takes place in
the human brain while learning new things. It can be defined as:Unsupervised learning is a
type of machine learning in which models are trained using unlabelled dataset and are
allowed to act on that data without any supervision.

Unsupervised learning cannot be directly applied to a regression or classification problem

because unlike supervised learning, we have the input data but no corresponding output data.
The goal of unsupervised learning is to find the underlying structure of dataset, group
that data according to similarities, and represent that dataset in a compressed format.

Example: Suppose the unsupervised learning algorithm is given an input dataset containing
images of different types of cats and dogs. The algorithm is never trained upon the given
dataset, which means it does not have any idea about the features of the dataset. The task of
the unsupervised learning algorithm is to identify the image features on their own.
Unsupervised learning algorithm will perform this task by clustering the image dataset into
the groups according to similarities between images.

Why use Unsupervised Learning?

Below are some main reasons which describe the importance of Unsupervised Learning:
o Unsupervised learning is helpful for finding useful insights from the data.
o Unsupervised learning is much similar as a human learns to think by their own
experiences, which makes it closer to the real AI.
o Unsupervised learning works on unlabeled and uncategorized data which make
unsupervised learning more important.
o In real-world, we do not always have input data with the corresponding output so to
solve such cases, we need unsupervised learning.

Working of Unsupervised Learning

Working of unsupervised learning can be understood by the below diagram:

NOTES UNIT 3 FOML

Here, we have taken an unlabelled input data, which means it is not categorized and
corresponding outputs are also not given. Now, this unlabelled input data is fed to the
machine learning model in order to train it. Firstly, it will interpret the raw data to find the
hidden patterns from the data and then will apply suitable algorithms such as k-means
clustering, Decision tree, etc.Once it applies the suitable algorithm, the algorithm divides the
data objects into groups according to the similarities and difference between the objects.

Types of Unsupervised Learning Algorithm

The unsupervised learning algorithm can be further categorized into two types of problems:

o Clustering: Clustering is a method of grouping the objects into clusters such that
objects with most similarities remains into a group and has less or no similarities with
the objects of another group. Cluster analysis finds the commonalities between the
data objects and categorizes them as per the presence and absence of those
commonalities.
o Association: An association rule is an unsupervised learning method which is used
for finding the relationships between variables in the large database. It determines the
set of items that occurs together in the dataset. Association rule makes marketing
strategy more effective. Such as people who buy X item (suppose a bread) are also
NOTES UNIT 3 FOML

tend to purchase Y (Butter/Jam) item. A typical example of Association rule is Market

Basket Analysis.

Advantages of Unsupervised Learning

 Unsupervised learning is used for more complex tasks as compared to supervised
learning because, in unsupervised learning, we don't have labelled input data.
 Unsupervised learning is preferable as it is easy to get unlabelled data in comparison
to labelled data.

Disadvantages of Unsupervised Learning

 Unsupervised learning is intrinsically more difficult than supervised learning as it
does not have corresponding output.
 The result of the unsupervised learning algorithm might be less accurate as input data
is not labelled, and algorithms do not know the exact output in advance.
Clustering
K-Means Clustering
K-Means clustering algorithm is defined as an unsupervised learning method having an
iterative process in which the dataset are grouped into k number of predefined non-
overlapping clusters or subgroups, making the inner points of the cluster as similar as
possible while trying to keep the clusters at distinct space it allocates the data points to a
cluster so that the sum of the squared distance between the clusters centroid and the data
point is at a minimum, at this position the centroid of the cluster is the arithmetic mean of the
data points that are in the clusters.

Understanding K- Means Clustering Algorithm

This algorithm is an iterative algorithm that partitions the dataset according to their features
into K number of predefined non- overlapping distinct clusters or subgroups. It makes the
data points of inter clusters as similar as possible and also tries to keep the clusters as far as
possible. It allocates the data points to a cluster if the sum of the squared distance between the
cluster’s centroid and the data points is at a minimum, where the cluster’s centroid is the
arithmetic mean of the data points that are in the cluster. A less variation in the cluster results
in similar or homogeneous data points within the cluster.

How does K- Means Clustering Algorithm Works?

K- Means Clustering Algorithm needs the following inputs:

 K = number of subgroups or clusters

 Sample or Training Set = {x1, x2, x3,………xn}
Now let us assume we have a data set that is unlabeled, and we need to divide it into clusters.
NOTES UNIT 3 FOML

Now we need to find the number of clusters.

This can be done by two methods:

 Elbow Method
 Purpose Method
Elbow Method
In this method, a curve is drawn between “within the sum of squares” (WSS) and the number
of clusters. The curve plotted resembles a human arm. It is called the elbow method because
the point of the elbow in the curve gives us the optimum number of clusters. In the graph or
curve, after the elbow point, the value of WSS changes very slowly, so the elbow point must
be considered to give the final value of the number of clusters.

Purpose-Based
In this method, the data is divided based on different metrics, and after then it is judged how
well it performed for that case. For example, the arrangement of the shirts in the men’s
clothing department in a mall is done on the criteria of the sizes. It can be done on the basis
of price and the brands also. The best suitable would be chosen to give the optimal number of
NOTES UNIT 3 FOML

clusters, i.e. the value of K.Now lets us get back to our given data set above. We can
calculate the number of clusters, i.e. the value of K, by using any of the above methods.

How to use the Above Methods?

Now let us see the execution process:
Step 1: Initialization
Firstly, initialize any random points called the centroids of the cluster. While initializing, you
must take care that the centroids of the cluster must be less than the number of training data
points. This algorithm is an iterative algorithm; hence the next two steps are performed
iteratively.

Step 2: Cluster Assignment

After initialization, all data points are traversed, and the distance between all the centroids
and the data points are calculated. Now the clusters would be formed depending upon the
minimum distance from the centroids. In this example, the data is divided into two clusters.
NOTES UNIT 3 FOML

Step 3: Moving Centroid

As the clusters formed in the above step are not optimized, so we need to form optimized
clusters. For this, we need to move the centroids iteratively to a new location. Take data
points of one cluster, compute their average and then move the centroid of that cluster to this
new location. Repeat the same step for all other clusters.

Step 4: Optimization
The above two steps are done iteratively until the centroids stop moving, i.e. they do not
change their positions anymore and have become static. Once this is done, the k- means
algorithm is termed to be converged.
Step 5: Convergence
Now, this algorithm has converged, and distinct clusters are formed and clearly visible. This
algorithm can give different results depending on how the clusters were initialized in the first
step.

Applications of K- Means Clustering Algorithm

NOTES UNIT 3 FOML

 Market segmentation
 Document clustering
 Image segmentation
 Image compression
 Vector quantization
 Cluster analysis
 Feature learning or dictionary learning
 Identifying crime-prone areas
 Insurance fraud detection
 Public transport data analysis
 Clustering of IT assets
 Customer segmentation
 Identifying Cancerous data
 Used in search engines
 Drug Activity Prediction

Advantages of K- Means Clustering Algorithm

 It is fast
 Robust
 Easy to understand
 Comparatively efficient
 If data sets are distinct, then gives the best results
 Produce tighter clusters
 When centroids are recomputed, the cluster changes.
 Flexible
 Easy to interpret
 Better computational cost
 Enhances Accuracy
 Works better with spherical clusters

Disadvantages of K- Means Clustering Algorithm

 Needs prior specification for the number of cluster centres.

 If there are two highly overlapping data, then it cannot be distinguished and cannot
tell that there are two clusters
 With the different representations of the data, the results achieved are also different
 Euclidean distance can unequally weigh the factors
 It gives the local optima of the squared error function
 Sometimes choosing the centroids randomly cannot give fruitful results
 It can be used only if the meaning is defined
 Cannot handle outliers and noisy data
 Do not work for the non-linear data set
 Lacks consistency
 Sensitive to scale
 If very large data sets are encountered, then the computer may crash.
 Prediction issues

KMeans Clustering
NOTES UNIT 3 FOML

K-means is a centroid-based algorithm, or a distance-based algorithm, where we calculate the

distances to assign a point to a cluster. In K-Means, each cluster is associated with a centroid.
The main objective of the K-Means algorithm is to minimize the sum of distances between
the points and their respective cluster centroid.
Step 1: Choose the number of clusters k. The first step in k-means is to pick the number of
clusters, k.
Step 2: Select k random points from the data as centroids. Next, we randomly select the
centroid for each cluster. Let’s say we want to have 2 clusters, so k is equal to 2 here. We
then randomly select the centroid.

Step 3: Assign all the points to the closest cluster centroid. Once we have initialized the
centroids, we assign each point to the closest cluster centroid.

Step 4: Recompute the centroids of newly formed clusters. Now, once we have assigned all
of the points to either cluster, the next step is to compute the centroids of newly formed
clusters.
Step 5: Repeat steps 3 and 4
The step of computing the centroid and assigning all the points to the cluster based on their
distance from the centroid is a single iteration. But wait – when should we stop this process?
It can’t run till eternity, right?
Stopping Criteria for K-Means Clustering
There are essentially three stopping criteria that can be adopted to stop the K-means
algorithm:
1. Centroids of newly formed clusters do not change
2. Points remain in the same cluster
3. Maximum number of iterations are reached
 Clustering: the goal is to group similar instances together into clusters. This is a great
tool for data analysis, customer segmentation, recommender systems, search engines,
image segmentation, semi-supervised learning, dimensionality reduction, and more.
 Anomaly detection: the objective is to learn what “normal” data looks like, and use
this to detect abnormal instances, such as defective items on a production line or a
new trend in a time series.
 Density estimation: this is the task of estimating the probability density function
(PDF) of the random process that generated the dataset. This is commonly used for
anomaly detection: instances located in very low-density regions are likely to be
anomalies. It is also useful for data analysis and visualization.
APPLICATIONS
NOTES UNIT 3 FOML

 For customer segmentation: you can cluster your customers based on their pur‐
chases, their activity on your website, and so on. This is useful to understand who
your customers are and what they need, so you can adapt your products and
marketing campaigns to each segment. For example, this can be useful in recom‐
mender systems to suggest content that other users in the same cluster enjoyed.
 For data analysis: when analyzing a new dataset, it is often useful to first discover
clusters of similar instances, as it is often easier to analyze clusters separately.
 As a dimensionality reduction technique: once a dataset has been clustered, it is
usually possible to measure each instance’s affinity with each cluster (affinity is
any measure of how well an instance fits into a cluster). Each instance’s feature
vector x can then be replaced with the vector of its cluster affinities. If there are k
clusters, then this vector is k dimensional. This is typically much lower
dimensional than the original feature vector, but it can preserve enough
information for further processing.
 For anomaly detection (also called outlier detection): any instance that has a low
affinity to all the clusters is likely to be an anomaly. For example, if you have
clus‐ tered the users of your website based on their behavior, you can detect users
with unusual behavior, such as an unusual number of requests per second, and so
on. Anomaly detection is particularly useful in detecting defects in manufacturing,
or for fraud detection.
 For semi-supervised learning: if you only have a few labels, you could perform
clustering and propagate the labels to all the instances in the same cluster. This
can greatly increase the amount of labels available for a subsequent supervised
learning algorithm, and thus improve its performance.
 For search engines: for example, some search engines let you search for images
that are similar to a reference image. To build such a system, you would first
apply a clustering algorithm to all the images in your database: similar images
would end up in the same cluster. Then when a user provides a reference image,
all you need to do is to find this image’s cluster using the trained clustering model,
and you can then simply return all the images from this cluster.
 To segment an image: by clustering pixels according to their color, then replacing
each pixel’s color with the mean color of its cluster, it is possible to reduce the
number of different colors in the image considerably. This technique is used in
many object detection and tracking systems, as it makes it easier to detect the
contour of each object.
CODE
from sklearn.cluster import KMeans
k=5
kmeans = KMeans(n_clusters=k)
y_pred = kmeans.fit_predict(X)
>>> y_pred array([4, 0, 1, ..., 2, 1, 0], dtype=int32)
>>> y_pred is kmeans.labels_ True
copy of the labels of the instances it was trained on, available via the labels_
>>> kmeans.cluster_centers_ array([[-2.80389616, 1.80117999], [ 0.20876306, 2.25551336],
[-2.79290307, 2.79641063], [-1.46679593, 2.28585348], [-2.80037642, 1.30082566]])
cluster_centers_ cluster centres after running the model over the data

PREDICTION OVER NEW DATA

>>> X_new = np.array([[0, 2], [3, 2], [-3, 3], [-3, 2.5]])
>>> kmeans.predict(X_new)
NOTES UNIT 3 FOML

FINE TUNING

Random assignment of Cluster centroids can be avoided. Instead initialization can be

done.
If you happen to know approximately where the centroids should be (e.g., if you ran another
clustering algorithm earlier), then you can set the init hyperparameter to a NumPy array
containing the list of centroids, and set n_init to 1:
good_init = np.array([[-3, 3], [-3, 2], [-3, 1], [-1, 2], [0, 2]])
kmeans = KMeans(n_clusters=5, init=good_init, n_init=1)

init – initialializing centroids

n_init = number of iterations

Model Inertia
this is the mean squared distance between each instance and its closest
centroid.
>>> kmeans.inertia_
>>> kmeans.score(X) – Negative of the inertia

MiniBatch KMeans
Instead of using the full dataset at each iteration, the algo‐ rithm is capable of using mini-
batches, moving the centroids just slightly at each iter‐ ation. This speeds up the algorithm
typically by a factor of 3 or 4 and makes it possible to cluster huge datasets that do not fit in
memory. Scikit-Learn implements this algorithm in the MiniBatchKMeans class. You can
just use this class like the KMeans class:

from sklearn.cluster import MiniBatchKMeans

minibatch_kmeans = MiniBatchKMeans(n_clusters=5)
minibatch_kmeans.fit(X)

Finding the Optimal Number of Clusters

A more precise approach (but also more computationally expensive) is to use the silhouette
score, which is the mean silhouette coe‡cient over all the instances.
An instance’s silhouette coefficient is equal to (b – a) / max(a, b) where a is the mean
distance to the other instances in the same cluster (it is the mean intra-cluster distance), and b
is the mean nearest-cluster distance, that is the mean distance to the instances of the next
closest cluster (defined as the one that minimizes b, excluding the instance’s own cluster).
The silhouette coefficient can vary between -1 and +1:
 a coefficient close to +1 means that the instance is well inside its own cluster
and far from other clusters
 while a coefficient close to 0 means that it is close to a cluster boundary,
 and finally a coefficient close to -1 means that the instance may have been
assigned to the wrong cluster.

>>> from sklearn.metrics import silhouette_score

>>> silhouette_score(X, kmeans.labels_) 0.655517642572828

APPLICATION / USAGE IN IMAGE CLUSTERING

It groups the pixels with similar colors together as a cluster.
NOTES UNIT 3 FOML

>>> from matplotlib.image import imread # you could also use `imageio.imread()`
>>> image = imread(os.path.join("images","clustering","ladybug.png"))
>>> image.shape (533, 800, 3)
X = image.reshape(-1, 3)
kmeans = KMeans(n_clusters=8).fit(X)
segmented_img = kmeans.cluster_centers_[kmeans.labels_]
segmented_img = segmented_img.reshape(image.shape)

USING CLUSTERING FOR PREPROCESSING

Clustering can be an efficient approach to dimensionality reduction, in particular as a

preprocessing step before a supervised learning algorithm. For example, let’s tackle the digits
dataset which is a simple MNIST-like dataset containing 1,797 grayscale 8×8 images
representing digits 0 to 9. First, let’s load the dataset:
Logistic Regression Pre-Process by Grid Search hyper
clustering parameter n_clusters
from sklearn.datasets import from sklearn.pipeline from sklearn.model_selection
load_digits import Pipeline import GridSearchCV
X_digits, y_digits = pipeline = Pipeline([
load_digits(return_X_y=True) ("kmeans", param_grid =
KMeans(n_clusters=5 dict(kmeans__n_clusters=rang
#Now, let’s split it into a training 0)), ("log_reg", e(2, 100))
set and a test set: LogisticRegression()),
]) grid_clf =
from sklearn.model_selection GridSearchCV(pipeline,
import train_test_split X_train, pipeline.fit(X_train, param_grid, cv=3, verbose=2)
X_test, y_train, y_test = y_train)
train_test_split(X_digits, y_digits) grid_clf.fit(X_train, y_train)
#Next, let’s fit a Logistic #Now let’s evaluate
Regression model: this classification #Let’s look at best value for k,
pipeline: and the performance of the
from sklearn.linear_model import >>> resulting pipeline:
LogisticRegression pipeline.score(X_test,
y_test) >>> grid_clf.best_params_
NOTES UNIT 3 FOML

log_reg = 0.9822222222222222 {'kmeans__n_clusters': 90}

LogisticRegression(random_state
=42) >>> grid_clf.score(X_test,
y_test) 0.9844444444444445
log_reg.fit(X_train, y_train)
Let’s evaluate its accuracy on the
test set:
>>> log_reg.score(X_test, y_test)
0.9666666666666667

USING CLUSTERING FOR SEMI-SUPERVISED LEARNING

Another use case for clustering is in semi-supervised learning, when we have plenty of
unlabeled instances and very few labeled instances. Let’s train a logistic regression model on
a sample of 50 labeled instances from the digits dataset:

n_labeled = 50
log_reg = LogisticRegression()
log_reg.fit(X_train[:n_labeled], y_train[:n_labeled])

What is the performance of this model on the test set?

>>> log_reg.score(X_test, y_test) 0.8266666666666667

The accuracy is just 82.7%: it should come as no surprise that this is much lower than earlier,
when we trained the model on the full training set. Let’s see how we can do better. First, let’s
cluster the training set into 50 clusters, then for each cluster let’s find the image closest to the
centroid. We will call these images the representative images:
k = 50
kmeans = KMeans(n_clusters=k)
X_digits_dist = kmeans.fit_transform(X_train)
representative_digit_idx = np.argmin(X_digits_dist, axis=0) X_representative_digits =
X_train[representative_digit_idx]

Now let’s look at each image and manually label it:

y_representative_digits = np.array([4, 8, 0, 6, 8, 3, ..., 7, 6, 2, 3, 1, 1])

Now we have a dataset with just 50 labeled instances, but instead of being completely
random instances, each of them is a representative image of its cluster. Let’s see if the
performance is any better:
>>> log_reg = LogisticRegression()
>>> log_reg.fit(X_representative_digits, y_representative_digits)
>>> log_reg.score(X_test, y_test) 0.9244444444444444

Wow! We jumped from 82.7% accuracy to 92.4%,

But perhaps we can go one step further: Let’s see what happens if we only propagate the
what if we propagated the labels to all the labels to the 20% of the instances that are
other instances in the same cluster? This is closest to the centroids:
called label propagation:
NOTES UNIT 3 FOML

DBSCAN

DBSCAN Clustering analysis

It is basically an Unsupervised learning method that divides the data points into a number of
specific batches or groups, such that the data points in the same groups have similar
properties and data points in different groups have different properties in some sense. It
comprises many different methods based on differential evolution.
E.g. K-Means (distance between points), Affinity propagation (graph distance), Mean-shift
(distance between points), DBSCAN (distance between nearest points), Gaussian mixtures
(Mahalanobis distance to centers), Spectral clustering (graph distance) etc.
Fundamentally, all clustering methods use the same approach i.e. first we calculate
similarities and then we use it to cluster the data points into groups or batches. Here we will
focus on Density-based spatial clustering of applications with noise (DBSCAN) clustering
method.
Clusters are dense regions in the data space, separated by regions of the lower density of
points. The DBSCAN algorithm is based on this intuitive notion of “clusters” and “noise”.
The key idea is that for each point of a cluster, the neighborhood of a given radius has to
contain at least a minimum number of points.
NOTES UNIT 3 FOML

Why DBSCAN?
Partitioning methods (K-means, PAM clustering) and hierarchical clustering work for finding
spherical-shaped clusters or convex clusters. In other words, they are suitable only for
compact and well-separated clusters. Moreover, they are also severely affected by the
presence of noise and outliers in the data.
Real life data may contain irregularities, like:
1. Clusters can be of arbitrary shape such as those shown in the figure below.
2. Data may contain noise.

The figure below shows a data set containing nonconvex clusters and outliers/noises. Given
such data, k-means algorithm has difficulties for identifying these clusters with arbitrary
shapes.
DBSCAN algorithm requires two parameters:
1. eps : It defines the neighborhood around a data point i.e. if the distance between
two points is lower or equal to ‘eps’ then they are considered as neighbors. If the
eps value is chosen too small then large part of the data will be considered as
outliers. If it is chosen very large then the clusters will merge and majority of the
data points will be in the same clusters. One way to find the eps value is based on
the k-distance graph.
2. MinPts: Minimum number of neighbors (data points) within eps radius. Larger the
dataset, the larger value of MinPts must be chosen. As a general rule, the
NOTES UNIT 3 FOML

minimum MinPts can be derived from the number of dimensions D in the dataset
as, MinPts>= D+1. The minimum value of MinPts must be chosen at least 3.

In this algorithm, we have 3 types of data points.

Core Point: A point is a core point if it has more than MinPts points within eps.
Border Point: A point which has fewer than MinPts within eps but it is in the neighborhood
of a core point.
Noise or outlier: A point which is not a core point or border point.

DBSCAN algorithm can be abstracted in the following steps:

1. Find all the neighbor points within eps and identify the core points or visited with
more than MinPtsneighbors.
2. For each core point if it is not already assigned to a cluster, create a new cluster.
3. Find recursively all its density connected points and assign them to the same
cluster as the core point.
A point a and b are said to be density connected if there exist a point c which has a
sufficient number of points in its neighbors and both the points a and b are within
the eps distance. This is a chaining process. So, if b is neighbor of c, c is neighbor
of d, d is neighbor of e, which in turn is neighbor of a implies that b is neighbor
of a.
4. Iterate through the remaining unvisited points in the dataset. Those points that do
not belong to any cluster are noise.

Below is the DBSCAN clustering algorithm in pseudocode:

DBSCAN(dataset, eps, MinPts){
# cluster index
C=1
for each unvisited point p in dataset {
mark p as visited
# findneighbors
Neighbors N = find the neighboring points of p
if |N|>=MinPts:
N = N U N'
if p' is not a member of any cluster:
add p' to cluster C
}

 For each instance, the algorithm counts how many instances are located within a small
distance ε (epsilon) from it. This region is called the instance’s εneighborhood.
NOTES UNIT 3 FOML

 If an instance has at least min_samples instances in its ε-neighborhood (includ‐ ing

itself), then it is considered a core instance. In other words, core instances are those
that are located in dense regions.
 All instances in the neighborhood of a core instance belong to the same cluster. This
may include other core instances, therefore a long sequence of neighboring core
instances forms a single cluster
 Any instance that is not a core instance and does not have one in its neighbor‐ hood is
considered an anomaly

DBSCAN has no predict so KNN is used instead

NOTES UNIT 3 FOML

COMPUTATIONAL COMPLEXITY
S.No Model Complexity
1 LinearSVC O(m × n)
2 SVC O(m2xn) or O(m3 xn)
3 CART O(log2 (m))
4 DBSCAN O(m log m)
5 KMeans O(m2)

Ensemble learning and Random Forests

Ensemble Learning

Definition: If you aggregate the predictions of a group of predictors (such as classifiers or

regressors), you will often get better predictions than with the best individual predictor. A
group of predictors is called an ensemble; thus, this technique is called Ensemble Learning,
and an Ensemble Learning algorithm is called an Ensemble method. Ensemble methods,
including bagging, boosting, stacking, and a few others.
NOTES UNIT 3 FOML

Voting Classifier:
A Voting Classifier is a machine learning model that trains on an ensemble of numerous
models and predicts an output (class) based on their highest probability of chosen class as the
output.

It simply aggregates the findings of each classifier passed into Voting Classifier and predicts
the output class based on the highest majority of voting. The idea is instead of creating
separate dedicated models and finding the accuracy for each them, we create a single model
which trains by these models and predicts output based on their combined majority of voting
for each output class.

Voting Classifier supports two types of votings.

Hard Voting: In hard voting, the predicted output class is a class with the highest majority of
votes i.e the class which had the highest probability of being predicted by each of the
classifiers. Suppose three classifiers predicted the output class(A, A, B), so here the majority
predicted A as output. Hence A will be the final prediction.
NOTES UNIT 3 FOML

PARAMETERS:
predict_proba() method - tell Scikit-Learn to predict the class with the
highest class probability, averaged over all the individual classifiers. This is called soft
voting.
It often achieves higher performance than hard voting because it gives more
weight to highly confident votes. All you need to do is replace voting="hard" with
voting="soft" and ensure that all classifiers can estimate class probabilities.

This is not the case of the SVC class by default, so you need to set its probability hyperpara‐
meter to True (this will make the SVC class use cross-validation to estimate class prob‐
abilities, slowing down training, and it will add a predict_proba() method).

CODE:

from sklearn.ensemble import RandomForestClassifier

from sklearn.ensemble import VotingClassifier
from sklearn.linear_model import LogisticRegression
from sklearn.svm import SVC
log_clf = LogisticRegression()
rnd_clf = RandomForestClassifier()
svm_clf = SVC()
voting_clf = VotingClassifier(
estimators=[('lr', log_clf), ('rf', rnd_clf), ('svc', svm_clf)],
voting='hard')
voting_clf.fit(X_train, y_train)
Let’s look at each classifier’s accuracy on the test set:
>>> from sklearn.metrics import accuracy_score
>>> for clf in (log_clf, rnd_clf, svm_clf, voting_clf):
... clf.fit(X_train, y_train)
NOTES UNIT 3 FOML

... y_pred = clf.predict(X_test)

... print(clf.__class__.__name__, accuracy_score(y_test, y_pred))
...

LogisticRegression 0.864
RandomForestClassifier 0.896
SVC 0.888
VotingClassifier 0.904

Soft Voting: In soft voting, the output class is the prediction based on the average of
probability given to that class. Suppose given some input to three models, the prediction
probability for class A = (0.30, 0.47, 0.53) and B = (0.20, 0.32, 0.40). So the average for
class A is 0.4333 and B is 0.3067, the winner is clearly class A because it had the highest
probability averaged by each classifier.

Bagging and Pasting

Bagging and pasting are techniques that are used in order to create varied subsets of the
training data. The subsets produced by these techniques are then used to train the predictors
of an ensemble.

Definition:
Bagging: One way to get a diverse set of classifiers is to use very different training
algorithms, as just discussed. Another approach is to use the same training algorithm for
every predictor, but to train them on different random subsets of the training set. When
sampling is performed with replacement, this method is called bagging1 (short for
bootstrap aggregating2).
Pasting: When sampling is performed without replacement, it is called
pasting.
NOTES UNIT 3 FOML

Once all predictors are trained, the ensemble can make a prediction for a new
instance by simply aggregating the predictions of all predictors. The aggregation
function is typically the statistical mode (i.e., the most frequent prediction, just like a
hard voting classifier) for classification, or the average for regression. Each individual
predictor has a higher bias than if it were trained on the original training set, but
aggregation reduces both bias and variance.4 Generally, the net result is that the
ensemble has a similar bias but a lower variance than a single predictor trained on the
original training set.

As you can see in Figure 7-4, predictors can all be trained in parallel, via different CPU cores
or even different servers. Similarly, predictions can be made in parallel.
This is one of the reasons why bagging and pasting are such popular methods: they
scale very well.

ATTRIBUTES:
BaggingClassifier class (or BaggingRegressor for regression). The following code trains an
ensemble of 500 Decision Tree classifiers,5 each trained on 100 training instances ran‐
domly sampled from the training set with replacement (this is an example of bagging,
but if you want to use pasting instead, just set bootstrap=False).

The BaggingClassifier automatically performs soft voting instead of hard voting if the base
classifier can estimate class probabilities (i.e., if it has a predict_proba() method), which is
the case with Decision Trees classifiers.

Njobs: The n_jobs param‐ eter tells Scikit-Learn the number of CPU cores to use for training
and predictions (–1 tells Scikit-Learn to use all available cores):

Max_samples: max_samples can alternatively be set to a float between 0.0 and 1.0, in which
case the max number of instances to sample is equal to the size of the training set times
max_samples

CODE:

from sklearn.ensemble import BaggingClassifier

from sklearn.tree import DecisionTreeClassifier
bag_clf = BaggingClassifier(
DecisionTreeClassifier(), n_estimators=500,
max_samples=100, bootstrap=True, n_jobs=-1)
bag_clf.fit(X_train, y_train)
y_pred = bag_clf.predict(X_test)
NOTES UNIT 3 FOML

Figure 7-5 compares the decision boundary of a single Decision Tree with the decision
boundary of a bagging ensemble of 500 trees (from the preceding code), both trained on the
moons dataset. As you can see, the ensemble’s predictions will likely generalize much better
than the single Decision Tree’s predictions: the ensemble has a comparable bias but a smaller
variance (it makes roughly the same number of errors on the training set, but the decision
boundary is less irregular).

Bootstrapping introduces a bit more diversity in the subsets that each predictor is
trained on, so bagging ends up with a slightly higher bias than pasting, but this also
means that predictors end up being less correlated so the ensemble’s variance is
reduced. Overall, bagging often results in better models, which explains why it is gen‐
erally preferred. However, if you have spare time and CPU power you can use
crossvalidation to evaluate both bagging and pasting and select the one that works best.

OOB – Out of Bag Evaluation

Definition:
With bagging, some instances may be sampled several times for any given predictor,
while others may not be sampled at all. By default a BaggingClassifier samples m
training instances with replacement (bootstrap=True), where m is the size of the
training set. This means that only about 63% of the training instances are sampled on
average for each predictor. The remaining 37% of the training instances that are not
sampled are called out-of-bag (oob) instances. Note that they are not the same 37%
for all predictors.

you can set oob_score=True when creating a BaggingClassifier to request an automatic oob
evaluation after training. The following code demonstrates this. The resulting evaluation
score is available through the oob_score_ variable:

>>> bag_clf = BaggingClassifier(

... DecisionTreeClassifier(), n_estimators=500,
... bootstrap=True, n_jobs=-1, oob_score=True)
...
>>> bag_clf.fit(X_train, y_train)
>>> bag_clf.oob_score_
NOTES UNIT 3 FOML

0.90133333333333332

According to this oob evaluation, this BaggingClassifier is likely to achieve about

90.1% accuracy on the test set. Let’s verify this:
>>> from sklearn.metrics import accuracy_score
>>> y_pred = bag_clf.predict(X_test)
>>> accuracy_score(y_test, y_pred)
0.91200000000000003
We get 91.2% accuracy on the test set—close enough!

Random Patches and Random Subspaces

Sampling both training instances and features is called the Random Patches method.Keeping
all training instances (i.e., bootstrap=False and max_sam ples=1.0) but sampling features
(i.e., bootstrap_features=True and/or max_features smaller than 1.0) is called the Random
Subspaces method. Sampling features results in even more predictor diversity, trading a bit
more bias for a lower variance.

Random Forests

A Random Forest is an ensemble of Decision Trees, generally trained via the bagging method
(or sometimes pasting), typically with max_samples set to the size of the training set. Instead
of building a BaggingClassifier and pass‐ ing it a DecisionTreeClassifier, you can instead use
the RandomForestClassifier class, which is more convenient and optimized for Decision
Trees10 (similarly, there is a RandomForestRegressor class for regression tasks). The
following code trains a Random Forest classifier with 500 trees (each limited to maximum 16
nodes), using all available CPU cores:

from sklearn.ensemble import RandomForestClassifier

rnd_clf = RandomForestClassifier(n_estimators=500, max_leaf_nodes=16, n_jobs=-1)
rnd_clf.fit(X_train, y_train)
y_pred_rf = rnd_clf.predict(X_test)

PROCEDURE:

Random Forests
Random forest is an ensemble model using bagging as the ensemble method and decision tree
as the individual model.
Let’s take a closer look at the magic🔮 of the randomness:
NOTES UNIT 3 FOML

Step 1: Select n (e.g. 1000) random subsets from the training set
Step 2: Train n (e.g. 1000) decision trees
 one random subset is used to train one decision tree
 the optimal splits for each decision tree are based on a random subset of features (e.g. 10
features in total, randomly select 5 out of 10 features to split)
Step 3: Each individual tree predicts the records/candidates in the test set, independently.
Step 4: Make the final prediction
For each candidate in the test set, Random Forest uses the class (e.g. cat or dog) with
the majority vote as this candidate’s final prediction.
Of course, our 1000 trees are the parliament here.

NOTE:

The following BaggingClassifier is

roughly equivalent to the previous RandomForestClassifier:
bag_clf = BaggingClassifier(
DecisionTreeClassifier(splitter="random", max_leaf_nodes=16),
n_estimators=500, max_samples=1.0, bootstrap=True, n_jobs=-1)

The BaggingClassifier class remains useful if you want a bag of something other than
Decision Trees.
There are a few notable exceptions: splitter is absent (forced to "random"), presort is absent
(forced to False), max_samples is absent (forced to 1.0), and base_estimator is absent (forced
to DecisionTreeClassifier with the provided hyperparameters

Extra-Trees

A forest of such extremely random trees is simply called an Extremely Randomized

Trees ensemble12 (or Extra-Trees for short). Once again, this trades more bias for a
lower variance. It also makes Extra-Trees much faster to train than regular Random
Forests since finding the best possible threshold for each feature at every node is one
of the most time-consuming tasks of growing a tree. Supported by sklearn
ExtraTreesClassifier class.

Feature Importance
NOTES UNIT 3 FOML

You can access the result using the feature_importances_ variable. For example, the
following code trains a RandomForestClassifier on the iris dataset (introduced in Chapter 4)
and outputs each feature’s importance. It seems that the most important features are the petal
length (44%) and width (42%), while sepal length and width are rather unimportant in
comparison (11% and 2%, respectively).
>>> from sklearn.datasets import load_iris
>>> iris = load_iris()
>>> rnd_clf = RandomForestClassifier(n_estimators=500, n_jobs=-1)
>>> rnd_clf.fit(iris["data"], iris["target"])
>>> for name, score in zip(iris["feature_names"], rnd_clf.feature_importances_):
... print(name, score)
...
sepal length (cm) 0.112492250999
sepal width (cm) 0.0231192882825
petal length (cm) 0.441030464364
petal width (cm) 0.423357996355

BOOSTING

Boosting (originally called hypothesis boosting) refers to any Ensemble method that
can combine several weak learners into a strong learner. The general idea of most
boosting methods is to train predictors sequentially, each trying to correct its prede‐
cessor. There are many boosting methods available, but by far the most popular are
AdaBoost13 (short for Adaptive Boosting) and Gradient Boosting. Let’s start with Ada‐
Boost.

AdaBoost
One way for a new predictor to correct its predecessor is to pay a bit more attention
NOTES UNIT 3 FOML

to the training instances that the predecessor underfitted. This results in new predic‐
tors focusing more and more on the hard cases. This is the technique used by Ada‐
Boost.

For example, to build an AdaBoost classifier, a first base classifier (such as a Decision
Tree) is trained and used to make predictions on the training set. The relative weight
of misclassified training instances is then increased. A second classifier is trained
using the updated weights and again it makes predictions on the training set, weights
are updated, and so on (see Figure 7-7)

Figure 7-8 shows the decision boundaries of five consecutive predictors on the
moons dataset (in this example, each predictor is a highly regularized SVM classifier
with an RBF kernel14). The first classifier gets many instances wrong, so their weights get
boosted. The second classifier therefore does a better job on these instances, and
so on. The plot on the right represents the same sequence of predictors except that
the learning rate is halved (i.e., the misclassified instance weights are boosted half as
much at every iteration). As you can see, this sequential learning technique has some
similarities with Gradient Descent, except that instead of tweaking a single predictor’s
parameters to minimize a cost function, AdaBoost adds predictors to the ensemble,
gradually making it better.
Once all predictors are trained, the ensemble makes predictions very much like bag‐
ging or pasting, except that predictors have different weights depending on their
overall accuracy on the weighted training set.

PROCEDURE
AdaBoost (Adaptive Boosting)
AdaBoost is a boosting ensemble model and works especially well with the decision tree.
Boosting model’s key is learning from the previous mistakes, e.g. misclassification data
points.
AdaBoost learns from the mistakes by increasing the weight of misclassified data points.
NOTES UNIT 3 FOML

AdaBoost adapts.

Step 0: Initialize the weights of data points. if the training set has 100 data points, then each
point’s initial weight should be 1/100 = 0.01.
Step 1: Train a decision tree
Step 2: Calculate the weighted error rate (e) of the decision tree. The weighted error rate
(e) is just how many wrong predictions out of total and you treat the wrong predictions
differently based on its data point’s weight. The higher the weight, the more the
corresponding error will be weighted during the calculation of the (e).
Step 3: Calculate this decision tree’s weight in the ensemble
the weight of this tree = learning rate * log( (1 — e) / e)
 the higher weighted error rate of a tree, 😫, the less decision power the tree will be given
during the later voting
 the lower weighted error rate of a tree, 😃, the higher decision power the tree will be given
during the later voting
Step 4: Update weights of wrongly classified points

the weight of each data point =

 if the model got this data point correct, the weight stays the same
 if the model got this data point wrong, the new weight of this point = old weight *
np.exp(weight of this tree)
Note: The higher the weight of the tree (more accurate this tree performs), the more boost
(importance) the misclassified data point by this tree will get. The weights of the data points
are normalized after all the misclassified points are updated.
Step 5: Repeat Step 1(until the number of trees we set to train is reached)
Step 6: Make the final prediction
The AdaBoost makes a new prediction by adding up the weight (of each tree) multiply the
prediction (of each tree). Obviously, the tree with higher weight will have more power of
influence the final decision.
NOTES UNIT 3 FOML

CODE:
from sklearn.ensemble import AdaBoostClassifier
ada_clf = AdaBoostClassifier(
DecisionTreeClassifier(max_depth=1), n_estimators=200,
algorithm="SAMME.R", learning_rate=0.5)
ada_clf.fit(X_train, y_train)

GRADIENT BOOSTING
Gradient Boosting works by sequentially adding predictors to an ensemble, each one
correcting its predecessor. However, instead of tweaking the instance weights at every
iteration like AdaBoost does, this method tries to fit the new predictor to the residual
errors made by the previous predictor.
Let’s go through a simple regression example using Decision Trees as the base predic‐
tors (of course Gradient Boosting also works great with regression tasks). This is
called Gradient Tree Boosting, or Gradient Boosted Regression Trees (GBRT). First, let’s
fit a DecisionTreeRegressor to the training set (for example, a noisy quadratic train‐
ing set):

PROCEDURE:

Gradient Boosting
Gradient boosting is another boosting model. Remember, boosting model’s key is learning
from the previous mistakes.
Gradient Boosting learns from the mistake — residual error directly, rather than update the
weights of data points.
Let’s illustrate how Gradient Boost learns.
NOTES UNIT 3 FOML

Step 1: Train a decision tree

Step 2: Apply the decision tree just trained to predict
Step 3: Calculate the residual of this decision tree, Save residual errors as the new y
Step 4: Repeat Step 1 (until the number of trees we set to train is reached)
Step 5: Make the final prediction
The Gradient Boosting makes a new prediction by simply adding up the predictions (of all
trees).

from sklearn.tree import DecisionTreeRegressor

tree_reg1 = DecisionTreeRegressor(max_depth=2)
tree_reg1.fit(X, y)
Now train a second DecisionTreeRegressor on the residual errors made by the first
predictor:
y2 = y - tree_reg1.predict(X)
tree_reg2 = DecisionTreeRegressor(max_depth=2)
tree_reg2.fit(X, y2)
Then we train a third regressor on the residual errors made by the second predictor:
y3 = y2 - tree_reg2.predict(X)
tree_reg3 = DecisionTreeRegressor(max_depth=2)
tree_reg3.fit(X, y3)
Now we have an ensemble containing three trees. It can make predictions on a new
instance simply by adding up the predictions of all the trees:
y_pred = sum(tree.predict(X_new) for tree in (tree_reg1, tree_reg2, tree_reg3))
NOTES UNIT 3 FOML

A simpler way to train GBRT ensembles is to use Scikit-Learn’s GradientBoostingRegressor

class. Much like the RandomForestRegressor class, it has hyperparameters to control the
growth of Decision Trees (e.g., max_depth, min_samples_leaf, and so on), as well as
hyperparameters to control the ensemble training, such as the number of trees (n_estimators).
The following code creates the same ensemble as the previous one:

from sklearn.ensemble import GradientBoostingRegressor

gbrt = GradientBoostingRegressor(max_depth=2, n_estimators=3, learning_rate=1.0)
gbrt.fit(X, y)

The learning_rate hyperparameter scales the contribution of each tree. If you set it
to a low value, such as 0.1, you will need more trees in the ensemble to fit the train‐
ing set, but the predictions will usually generalize better. This is a regularization tech‐
nique called shrinkage. In order to find the optimal number of trees, you can use early
stopping

import numpy as np
from sklearn.model_selection import train_test_split
from sklearn.metrics import mean_squared_error
X_train, X_val, y_train, y_val = train_test_split(X, y)
gbrt = GradientBoostingRegressor(max_depth=2, n_estimators=120)
gbrt.fit(X_train, y_train)
errors = [mean_squared_error(y_val, y_pred)
for y_pred in gbrt.staged_predict(X_val)]
bst_n_estimators = np.argmin(errors)
gbrt_best = GradientBoostingRegressor(max_depth=2,n_estimators=bst_n_estimators)
gbrt_best.fit(X_train, y_train)

STACKING

Stacking is based on a simple idea: instead of using trivial functions

(such as hard voting) to aggregate the predictions of all predictors in an ensemble,
why don’t we train a model to perform this aggregation?
NOTES UNIT 3 FOML

Next, the first layer predictors are used to make predictions on the second (held-out)
set (see Figure 7-14). This ensures that the predictions are “clean,” since the predictors
never saw these instances during training. Now for each instance in the hold-out set there are three
predicted values. We can create a new training set using these predic‐
ted values as input features (which makes this new training set three-dimensional),
and keeping the target values. The blender is trained on this new training set, so it
learns to predict the target value given the first layer’s predictions.

It is actually possible to train several different blenders this way (e.g., one using Lin‐
ear Regression, another using Random Forest Regression, and so on): we get a whole
layer of blenders. The trick is to split the training set into three subsets: the first one is
used to train the first layer, the second one is used to create the training set used to
train the second layer (using predictions made by the predictors of the first layer),
and the third one is used to create the training set to train the third layer (using pre‐
dictions made by the predictors of the second layer). Once this is done, we can make
NOTES UNIT 3 FOML

a prediction for a new instance by going through each layer sequentially, as shown in
Figure 7-15.

PROCEDURE:

 Split training data sets into n-folds using the RepeatedStratifiedKFold as this is the
most common approach to preparing training datasets for meta-models.
 Now the base model is fitted with the first fold, which is n-1, and it will make
predictions for the nth folds.
 The prediction made in the above step is added to the x1_train list.
 Repeat steps 2 & 3 for remaining n-1folds, so it will give x1_train array of size n,
 Now, the model is trained on all the n parts, which will make predictions for the
sample data.
 Add this prediction to the y1_test list.
 In the same way, we can find x2_train, y2_test, x3_train, and y3_test by using Model
2 and 3 for training, respectively, to get Level 2 predictions.
 Now train the Meta model on level 1 prediction, where these predictions will be used
as features for the model.
 Finally, Meta learners can now be used to make a prediction on test data in the
stacking model.