Vishal

Batch 10
17SCSE101611
AI Assignment

Q1. Explain the Linear Descriminat analysis algorithm.

Ans: Linear Discriminant Analysis or Normal Discriminant Analysis or Discriminant
Function Analysis is a dimensionality reduction technique which is commonly used for
the supervised classification problems. It is used for modeling differences in groups i.e.
separating two or more classes. It is used to project the features in higher dimension
space into a lower dimension space.
For example, we have two classes and we need to separate them efficiently. Classes
can have multiple features. Using only a single feature to classify them may result in
some overlapping as shown in the below figure. So, we will keep on increasing the
number of features for proper classification.
Two criteria are used by LDA to create a new axis:
1. Maximize the distance between means of the two classes.
2. Minimize the variation within each class.

Q2. Discuss the the Naive Bayes algorithm.

Ans: It is a classification technique based on Bayes’ Theorem with an assumption of
independence among predictors. In simple terms, a Naive Bayes classifier assumes
that the presence of a particular feature in a class is unrelated to the presence of any
other feature.

For example, a fruit may be considered to be an apple if it is red, round, and about 3
inches in diameter. Even if these features depend on each other or upon the existence
of the other features, all of these properties independently contribute to the probability
that this fruit is an apple and that is why it is known as ‘Naive’.

Naive Bayes model is easy to build and particularly useful for very large data sets.
Along with simplicity, Naive Bayes is known to outperform even highly sophisticated
classification methods.

Bayes theorem provides a way of calculating posterior probability P(c|x) from P(c), P(x)
and P(x|c). Look at the equation below:
Above,

 P(c|x) is the posterior probability of class (c, target) given predictor (x, attributes).

 P(c) is the prior probability of class.
 P(x|c) is the likelihood which is the probability of predictor given class.
 P(x) is the prior probability of predictor.

Q3. Implementing the Decision Tree algorithm from scratch.

Ans: Decision trees are among the most powerful Machine Learning tools available
today and are used in a wide variety of real-world applications from Ad click predictions
at Facebook¹ to Ranking of Airbnb experiences. Yet they are intuitive, easy to interpret
— and easy to implement.

import
numpy
as np

class DecisionTreeClassifier:
def __init__(self, max_depth=None):
self.max_depth = max_depth

def _best_split(self, X, y):

m = y.size
if m <= 1:
return None, None

# Count of each class in the current node.

 num_parent = [np.sum(y == c) for c in range(self.n_classes_)]

# Gini of current node.

best_gini = 1.0 - sum((n / m) ** 2 for n in num_parent)
best_idx, best_thr = None, None

# Loop through all features.

for idx in range(self.n_features_):
# Sort data along selected feature.
thresholds, classes = zip(*sorted(zip(X[:, idx], y)))

# We could actually split the node according to each feature/threshold

pair
# and count the resulting population for each class in the children, but
# instead we compute them in an iterative fashion, making this for loop
# linear rather than quadratic.
num_left = [0] * self.n_classes_
num_right = num_parent.copy()
for i in range(1, m): # possible split positions
c = classes[i - 1]
num_left[c] += 1
num_right[c] -= 1
gini_left = 1.0 - sum(
(num_left[x] / i) ** 2 for x in range(self.n_classes_)
)
gini_right = 1.0 - sum(
(num_right[x] / (m - i)) ** 2 for x in range(self.n_classes_)
)

# The Gini impurity of a split is the weighted average of the Gini

# impurity of the children.
gini = (i * gini_left + (m - i) * gini_right) / m

# The following condition is to make sure we don't try to split two

# points with identical values for that feature, as it is impossible
# (both have to end up on the same side of a split).
if thresholds[i] == thresholds[i - 1]:
continue
 if gini < best_gini:
best_gini = gini
best_idx = idx
best_thr = (thresholds[i] + thresholds[i - 1]) / 2 # midpoint

return best_idx, best_thr