Prof.
Pedram Jahangiry
Machine Learning
Model Model Type and use case
Type
Linear - Parametric
Linear regression Used for regression only
Linear - Parametric
Polynomial regression
Used for regression only
Linear - Parametric
Penalized regression:
Ridge, LASSO and Elastic
Net Used for regression only
Linear - Parametric
Logistic regression
Used for classification only
Non-linear - Non-parametric
KNN Used for both regression and
classification
Supervised
Kernel basis (non-linear)
Linear SVM is parametric
Kernel SVM is non-parametric
SVM
Used for both regression and
classification
Tree-based (non-linear)
Non-parametric
Decision Trees
Used for both regression and
classification
Ensemble method (non-linear)
Non-parametric
Random Forest
Used for both regression and
classification
Ensemble method (non-linear)
Non-parametric
Boosting (XGboost)
Used for both regression and
classification
Utah State University: Huntsman School of Business
Description
Finds the "best fit" through all the data points. - very fast training because of closed form solution
Extending linear regression model to capture
non-linearities
Linear method that penalizes irrelevant
features using regularization
L1 regularization: LASSO
L2 regularization: Ridge
combining L1 and L2: Elastic net
Basically the adaptation of linear regression
to classification problems.
Make prediction for a new observation by
finding similarities (“nearness”) between it
and its k nearest neighbors in the existing
dataset.
Uses a kernel to transform the feature space
to linearly separable boundaries
- Progressively divide data sets into smaller
data groups based on a descriptive feature ,
until they reach sets that are small enough to
be described by some label
- Many trees are created on bootstrapped data - Avoid overfitting by reducing the model variance.
and combined using averaging.
- Implements boosting to build decision trees - Efficient handling of missing data
of weak prediction models and generalizes
using a loss function.
Pros Cons Hyperparameters
- highly interpretable (giving significancy results) - validity of linear regression assumptions none
- cannot capture complex relationships
- no hyperparameter tuning required
- interpretable for low values of d (giving significancy
- need to choose the right polynomial degree d: degree of polynomial
results)
- Can capture polynomial relationships - notorious tail behavior (sensitive to outliers)
- Can be used for feature selection (reducing the penalty (how much to penalize the
- Requires feature scaling
dimension of the feature space) parameters)
L1 ratio: ration between L1 and L2
- interpretable
regularization
- the same as penalized regression if
- probabilitsitc model (the outputs are probabilities) - validity of linear regression assumptions
regularization is used
- highly interpretable (giving significancy results) - sensitive to extreme values
- easy to understand - cannot capture complex relationships
- fast and efficient
- Intuitive and simple - Choice of K - K value
- Easy to implement for multi class problem - Slow (memory based approach) - distance metrics
- Few parameters/hyper parameters - Curse of dimensionality
- No assumption (non parametric) - Hard to interpret
- Requires feature scaling
- Not good with multiple categorical features
- SVM can be memory efficient! uses only a subset of
- Requires feature scaling - Kernel: linear, rbf, poly, ...
the training data (support vectors)
- Can handle non linear data sets - No probability outcome! - C: Cost of misclassification
- Can handle high dimensional spaces (even when - Gamma (for rbf): how far the
- Does not perform well with noisy data
D>N) influence reach
- Linear SVM are not very sensitive to overfitting (soft
- Limited interpretability (specially for Kernel SVM) - d (for poly): degree of polynomial
margin; regularization)
- Memory intensive: Long training time when we
- Can have high accuracy (even compared to NN)
have large data sets.
- Sensitive to noisy data. It can overfit noisy data.
- Max tree depth, min samples per leaf
-Easy to interpret and visualize Small variations in data can result in the
(node), min samples split
different decision tree
- Can easily handle categorical data without the need to
- Can lead to overfitting - Cost complexity alpha
create dummy variables
- Can easily capture Non linear patterns - Poor level of predictive accuracy - Criterion: gini/entropy/ ...
- Can handle data in its raw form (no preprocessing
needed ).
- No assumption (non parametric)
- Can handle colinearity efficiently
All the advantages of Decision Trees + - no interpretability DTs parameters +
- Typically more accurate - complexity - m: subset of features
- many hyper parameters - B: number of bootstrapped trees
- very flexible and parallelizable! - slow on large data sets
- No data preprocessing (no feature scaling)
- Great with high dimensionality
All the advantages of Random Forests + - no interpretability RF parameters +
- Regularization for avoiding overfitting - many hyper parameters - Regularization terms
- In-built cross validation capability
- Cache awareness and out-of-core computing
- Tree pruning using depth-first approach
- Parallelized tree building
 Prof. Pedram Jahangiry
Utah State University: Huntsman School of Business

Machine Learning
Model Model Type and use case Description Pros Cons Hyperparameters
Type
- Reducing the number of features to the most relevant
- Hard to interpret None
predictors is very useful in general.
Non- Parametirc
- Dimension reduction facilitates the data visualization
- Requires feature scaling
in two or three dimensions.
Principal components are vectors that define a
- Before training another supervised or unsupervised
new coordinate system in which the first axis
Principle Component learning model, it can be performed as part of EDA to
goes in the direction of the highest variance in
identify patterns and detect correlations .
Unsupervised

Analysis (PCA) the data. The second PC is orthogonal to PC1

Used for dimension reduction
and etc.
- Machine learning models are quicker to train , tend to
reduce overfitting (by avoiding the curse of
dimensionality), and are easier to interpret if provided
with lower dimensional datasets.

Cab be both Parametric and Non-
Parametirc
K-Means
Used for clustering the data
- Uses a measure of similarity to detect
groups within data set: K means is an
algorithm that repeatedly partitions
observations into a fixed, pre-specified
number of clusters
- Simple to understand - Need to choose k before running the algorithm K: Number of clusters
- The k means algorithm is fast and works well on very
- Requires feature scaling - Distance metrics
large datasets
- Can help visualize the data and facilitate detecting
- Poor performance with clusters of irregular shapes
trends or outliers.
- Not applicable for categorical data
- Unable to handle noisy data

Cab be both Parametric and Non-
Parametirc
Hierarchical clustering
Used for clustering the data
- The optimal number of clusters can be obrained by - The choice of distance metrics and linkage
- Uses a measure of similarity to detect - Distance metrics
the model itself, methods can be tricky
groups within data set: Hierarchical clustering
is an iterative procedure used to build a
hierarchy of clusters - Requires feature scaling - Linkage methods