MCA 206 Data Analytics

NIT Kurukshetra

Mathematical Foundations of Machine Learning

NIT Kurukshetra

Introduction to key terms: Definition of ML(TOM Mitchell), Attributes, Samples, Target Attribute,
Selection of Attribute (Ethical and computational perspective), Curse of Dimensionality, Generalizability,
Overfitting, Probability Distribution, Validation data, Types of machine learning problems viz.
Supervised, Unsupervised, Reinforcement, Semi-Supervised

Case Study: K-Nearest Neighbour, Model parameters, Parameter Tuning using cross-validation, and
Accuracy as quantification of goodness of model for a given problem

Concept Learning: Concept, Hypothesis, Hypothesis Space and representation, Modelling Concept
Learning as a Search problem, hypotheses ordering relationship within hypothesis space, Inductive
Learning Hypothesis, Maximally Specific/Maximally General Hypothesis, Find S Algorithm, Version Space
and Candidate Elimination Algorithm, List then Eliminate Algorithm, Representation for version space
including most general and specific boundary hypotheses. Discussion of Algorithms over Convergence,
Inductive bias, Unbiased learner and its futility [ID3 from TOM Mitchell]

Linear Algebra: Vectors, Length, Magnitude, difference vector, orthogonal vector, distance between two
vectors, unit vectors, orthogonal, orthonormal vectors, basis vectors, orthonormal basis, importance of
orthonormal basis, span, coordinates, subspace.

Interpreting matrix for data science, Understanding matrix multiplication, linear combination of vectors,
Solving System of linear equations through Gauss Elimination, Row echelon form, basic and free
variables, the impact of exchanging of rows and columns of a matrix equation, Reduced Row echelon
form, Gauss Jordan method, particular and homogeneous solution, Matrix inverse, Rank, pivots,
truncation error, propagation of error on addition, multiplication and other algebraic operations, ill-
conditioned matrix, Nullity, linear (in)dependence, identifying linear relationship among variables

Identifying unique solutions, no and infinite solutions in a system of linear equations, full row rank, full
column rank, solving in all the cases for a unique solution with optimization perspective, Ordinary Least
Squares method.

Conversion of a multi-variable function to a matrix form, Multi-variable differential Calculus, Gradient,

Hessian.
Vector equation of a line in 2D, plane in 3D, projection of a vector onto a subspace, projection on an
orthonormal basis, contours, halfspaces,
Matrix as linear transformation operator, Reflection, Projection, Rotation, scaling Operator,
Eigenvectors, Eigenvalues, Positive Definite Matrix, Semi-Positive Definite Matrix, Null Space and
Column Space with Eigenvectors, Algebraic and Geometric Multiplicity of Eigenvectors, [ Diagonalization
of Symmetric Matrix, Principal Component Analysis, SVD]

Probability: Random Phenomena, Probability measure, random variables, probability distribution,

probability density and mass function, cumulative distribution function, normal, uniform, binomial and
Bernoulli mass function, probability interpretation from data, Independent and Exclusive Events,
Conditional Probability, case study Monty Hall problem, joint and marginal distributions, Bayes
probability, Naive Bayes and Maximum Likelihood as an example, moments of PDF, mean, variance.
[Sample Statistics and Hypothesis Testing and Model Assessment]

Optimization:
Motivation, Objective function, Decision variables, constraints, types of optimization problem. Solving
unconstrained univariate and multivariate problems, first-order necessary conditions, second-order
sufficiency condition, Steepest descent method of iterative optimization, learning rate, and convergence
criteria for iterative optimisation.
[Single Neuron Learning, Perceptron Learning, the inability of a single neuron, and perceptrons. A brief
on Sum of Products and Product of Sums to represent any boolean function, and universal power of one
hidden layer neural network, ANN, Back Propagation, Activation function etc]

Constrained optimisation with equality constraints, Lagrangian, first-order conditions, Constrained

optimisation with inequality constraints, Lagrangian function, first-order conditions, Karush Khun Tucker
conditions, Sufficient condition, SVM

Performance Measures and Loss functions: Regression-Root mean squared error, average error,
coefficient of determination, Similarity- Karl Pearson’s, Spearman, Kendal Correlation coefficients,
Classification- Accuracy, precision, recall, f-score, Sensitivity, Specificity, Balanced Accuracy, Prevalence,
Positive Predicted Value, Negative Predicted Value, Detection Rate, Detection prevalence, Kappa Statistics,
Receiver Operating Characteristic Curve, Area under ROC curve [Clustering-Dunn Index, Silhouette Index]

Data Analysis Techniques and case studies:

Data Preprocessing (imputation), the Understanding cycle of solution of data analysis problems, Logistic
regression, Ordinary Least Squares, K-mean clustering

*[items included in square bracket are not included in exam]

Courses Followed
 ● Data Science for Engineers - Course (NPTEL)
● https://www.youtube.com/playlist?list=PLeThw3BkDjNL0HILrl4yZMvex-4K6MmL5 (complete playlist)
● https://www.youtube.com/channel/UCB2p-jaoolkv0h22m4I9l9Q/playlists (Stanford)

Books Followed

● Linear Algebra And Its Applications by Gilbert Strang

● Machine Learning (McGraw-Hill Series in Computer Science) Reviews & Ratings - Amazon.in by Tom
Mitchell

Recommended Channels

● For Probability, Hypothesis Testing, Model Assessment etc zedstatistics - YouTube

● For Optimization First Order Methods - YouTube
● Machine Learning 10-601: Lectures by Tom Mitchell