1. Describe why SVMs offer more accurate results than Logistic Regression. - 2.
2. Express multiple one way repeated measure ANOVA on a two way 9. How can the initial number of clusters for k-means algorithm be
SVM try to maximize the margin between the closest support vectors whereas
logistic regression maximize the posterior class probability. - LR is used for
solving Classification problems, while SVM model is used for both
Classification and regression. - SVM is deterministic while LR is probabilistic. -
LR is vulnerable to overfitting, while the risk of overfitting is less in SVM.
2. Explain about Probability Distribution and Entropy.
- Probability Distributions A probability distribution is a statistical function
that describes all the possible values and probabilities for a random variable
within a given range. This range will be bound by the minimum and maximum
possible values, but where the possible value would be plotted on the
probability distribution will be determined by a number of factors like mean
(average), standard deviation, skewness, and kurtosis. Two types are :
Discrete Probability Distributions and Continuous Probability Distributions. -
Entropy Entropy measures the amount of surprise and data present in a
variable. In information theory, a random variable’s entropy reflects the
average uncertainty level in its possible outcomes. Events with higher
uncertainty have higher entropy.
3. Identify the difference between Active Learning and Reinforcement
Learning. Explain it with suitable examples and diagram.
Active learning is based on the concept, if a learning algorithm can choose the
data it wants to learn from, it can perform better than traditional methods
with substantially less data for training. So it's kind of semi-supervised
machine learning. Query Output World Active Learner Classifier/Model
Response Reinforcement Learning is a type of machine learning technique
that enables an agent to learn in an interactive environment by trial and error
using feedback from its own actions and experiences. It is based on rewards
and punishments mechanism which can be both active and passive.
Environment Next State Reward/Penalty Action Agent
4. Express Multiple one-way ANOVA on a two-way device.
- A one-way ANOVA is primarily designed to enable the equality testing
between three or more means. A two-way ANOVA is designed to assess the
interrelationship of two independent variables on a dependent variable. - A
one-way ANOVA only involves one factor or independent variable, whereas
there are two independent variables in a two-way ANOVA. - In a one-way
ANOVA, the one factor or independent variable analysed has three or more
categorical groups. A two-way ANOVA instead compares multiple groups of
two factors. - One-way ANOVA need to satisfy only two principles of design of
experiments, i.e., replication and randomization. As opposed to two-way
ANOVA, which meets all three principles of design of experiments which are
replication, randomization and local control.
5. Why you mean to take Big Data?
Big Data is a massive amount of data sets that cannot be stored, processed,
or analysed using traditional tools. There are millions of data sources that
generate data at a very rapid rate. These data sources are present across the
world. Some of the largest sources of data are social media platforms and
networks. Let’s use Facebook as an example—it generates more than 500
terabytes of data every day. This data includes pictures, videos, messages,
and more. Data also exists in different formats, like structured data, semi-
structured data, and unstructured data. For example, in a regular Excel sheet,
data is classified as structured data
6. Write the role of Activation Function in Neural Networks.
The activation function in Neural Networks takes an input 'x' multiplied by a
weight 'w'. Bias allows you to shift the activation function by adding a
constant (i.e. the given bias) to the input. Bias in Neural Networks can be
thought of as analogous to the role of a constant in a linear function, whereby
the line is effectively transposed by the constant value. With no bias, the
input to the activation function is 'x' multiplied by the connection weight 'w0'.
In a scenario with bias, the input to the activation function is 'x' times the
connection weight 'w0' plus the bias times the connection weight for the bias
'w1'. This has the effect of shifting the activation function by a constant
amount (b * w1).
7. Compare and construct the relationship between Clustering and Centroid.
Several approaches to clustering exist. Each approach is best suited to a
particular data distribution. Focusing on centroid-based clustering using k-
means : Centroid-based clustering organizes the data into non-hierarchical
clusters, in contrast to hierarchical clustering defined below. k-means is the
most widely-used centroid-based clustering algorithm. Centroid-based
algorithms are efficient but sensitive to initial conditions and outliers.
8. Write short note on Deep Learning.
Deep learning is a branch of machine learning which is based on artificial
neural networks. It is capable of learning complex patterns and relationships
within data. In deep learning, we don’t need to explicitly program everything.
It has become increasingly popular in recent years due to the advances in
processing power and the availability of large datasets. Because it is based on
artificial neural networks (ANNs) also known as deep neural networks (DNNs).
These neural networks are inspired by the structure and function of the
human brain’s biological neurons, and they are designed to learn from large
amounts of data.
9. Explain ANOVA. ANOVA stands for Analysis of Variance.
It is a statistical method used to analyze the differences between the means
of two or more groups or treatments. It is often used to determine whether
there are any statistically significant differences between the means of
different groups. ANOVA compares the variation between group means to the
variation within the groups. If the variation between group means is
significantly larger than the variation within groups, it suggests a significant
difference between the means of the groups.
10. Explain Quadratic Determinant Analysis.
Quadratic Discrimination is the general form of Bayesian discrimination.
Discriminant analysis is used to determine which variables discriminate
between two or more naturally occurring groups. Difference from LDA is it
relaxed the assumption that the mean and covariance of all the classes were
equal. Working : QDA is a variant of LDA in which an individual covariance
matrix is estimated for every class of observations. QDA is particularly useful
if there is prior knowledge that individual classes exhibit distinct covariances.
QDA assumes that observation of each class is drawn from a normal
distribution (similar to linear discriminant analysis). QDA assumes that each
class has its own covariance matrix (different from linear discriminant
analysis) The advantage of quadratic discriminant analysis over linear
discriminant analysis and linear regression is that when the decision
boundaries are linear.
11. Describe Probability Distribution in details
A probability distribution is a mathematical function that defines the
likelihood of different outcomes or values of a variable. This function is
commonly represented by a graph or probability table, and it provides the
probabilities of various possible results of an experiment or random
phenomenon based on the sample space and the probabilities of events.
Probability distributions are fundamental in probability theory and statistics
for analyzing data and making predictions. Probability distributions enable us
to analyze data and draw meaningful conclusions by describing the likelihood
of different outcomes or events. In statistical analysis, these distributions
play a pivotal role in parameter estimation, hypothesis testing, and data
inference. They also find extensive use in risk assessment.
13. What is the role of Statistical Model in data analytics?
Statistical modelling is the process of applying statistical analysis to a
dataset. A statistical model is a mathematical representation (or
mathematical model) of observed data. When data analysts apply various
statistical models to the data they are investigating, they are able to
understand and interpret the information more strategically. Rather than
sifting through the raw data, this practice allows them to identify
relationships between variables, make predictions about future sets of data,
and visualize that data so that non-analysts and stakeholders can consume
and leverage it.
14. Justify why SVM is so fast?
SVM performs and generalized well on the out of sample data. Due to this as
it performs well on out of generalization sample data SVM proves itself to be
fast as the sure fact says that in SVM for the classification of one sample , the
kernel function is evaluated and performed for each and every support
vectors.
15. Main characteristics of Data Analytics.
Data analytics is the process of examining, cleaning, transforming, and
interpreting data to extract valuable insights and support decision-making.
main characteristics of data analytics are: ● Data collection from various
sources. ● Data cleaning and preprocessing for accuracy. ● Descriptive,
diagnostic, predictive, and prescriptive analytics. ● Data visualization for
better understanding. ● Use of machine learning and AI. ● Handling big data
and real-time analytics. ● Emphasis on data security and privacy. ● An
iterative process to refine insights. ● Domain-specific and fosters a data-
driven culture Class Test I
design?
- A one-way ANOVA is primarily designed to enable the equality testing
between three or more means. A two-way ANOVA is designed to assess
the interrelationship of two independent variables on a dependent
variable. - A one-way ANOVA only involves one factor or independent
variable, whereas there are two independent variables in a two-way
ANOVA. - In a one-way ANOVA, the one factor or independent variable
analysed has three or more categorical groups. A two-way ANOVA
instead compares multiple groups of two factors. - One-way ANOVA
need to satisfy only two principles of design of experiments, i.e.,
replication and randomization.
3. Classify overfitting and underfitting and how to combat them?
Overfitting : Overfitting occurs when the model tries to cover all the
data points, or more than the required data points present in the given
dataset. Because of this, the model starts caching noise and inaccurate
values present in the dataset, and all these factors reduce the
efficiency and accuracy of the model. The overfitted model has low
bias and high variance. Some ways by which the occurrence of
overfitting can be reduced are : Cross-Validation, Training with more
data, Removing features, Early stopping the training, Regularization,
etc. Underfitting : Underfitting occurs when the model is not able to
capture the underlying trend of the data. To avoid overfitting, the fed
of training data is stopped earlier due to which the model is not able to
learn enough from the training data, and causes underfitting. Hence it
reduces the accuracy and produces unreliable predictions.
4. Describe why SVM is more accurate than Logistic Regression?
- SVM try to maximize the margin between the closest support vectors
whereas logistic regression maximize the posterior class probability. -
LR is used for solving Classification problems, while SVM model is used
for both Classification and regression. - SVM is deterministic while LR
is probabilistic. - LR is vulnerable to overfitting, while the risk of
overfitting is less in SVM.
5. Write the best practice for big data analytics.
Some of the best practices in big data analytics are : - Defining clear
objectives. - Knowing which data is important and which is not. -
Assuring data quality. - Committing to proper data labelling. - Choosing
proper data storage locations. - Managing data lifecycle. - Simplifying
procedures for backup. - Implementing security measures. - Ensuring
scalable infrastructure. - Arranging data audits at regular basis.
6. Write the role of activation function in neural network.
The activation function in Neural Networks takes an input 'x' multiplied
by a weight 'w'. Bias allows you to shift the activation function by
adding a constant (i.e. the given bias) to the input. Bias in Neural
Networks can be thought of as analogous to the role of a constant in a
linear function, whereby the line is effectively transposed by the
constant value. With no bias, the input to the activation function is 'x'
multiplied by the connection weight
7. Describe Descriptive Statistics.
Descriptive statistics refers to a branch of statistics that involves
summarizing, organizing, and presenting data meaningfully and
concisely. It focuses on describing and analyzing a dataset's main
features and characteristics without making any generalizations or
inferences to a larger population. The primary goal of descriptive
statistics is to provide a clear and concise summary of the data,
enabling researchers or analysts to gain insights and understand
patterns, trends, and distributions within the dataset. This summary
typically includes measures such as central tendency (e.g., mean,
median, mode), dispersion (e.g., range, variance, standard deviation),
and shape of the distribution (e.g., skewness, kurtosis).
8. Applications of Clustering.
The clustering technique can be widely used in various tasks. Some
most common uses of this technique are : o Market Segmentation o
Statistical data analysis o Social network analysis o Image
segmentation o Anomaly detection, etc. Apart from these general
usages, it is used by the Amazon in its recommendation system to
provide the recommendations as per the past search of products.
Netflix also uses this technique to recommend the movies and web-
series to its users as per the watch history. The below diagram explains
the working of the clustering algorithm. We can see the different fruits
are divided into several groups with similar properties.
17. Explain why Big Data Analytics is helpful in Business Reserve.
Explain the steps to be followed to deploy a Big Data solution.
- Improved Accuracy : Big data analytics enables businesses to make
decisions based on facts and evidence rather than intuition or
guesswork. By analyzing large volumes of data, patterns and trends
that may not be apparent at a smaller scale can be identified. - Real-
time Insights : In the fast-paced business environment, real-time
insights are essential for timely decision making. Big data analytics
allows organizations to process and analyze data in realtime or near
real-time, enabling them to respond quickly to emerging trends,
market shifts, and customer demands. - Customer Understanding :
Understanding customers is vital for tailoring products, services, and
marketing strategies. Big data analytics provides a holistic view of
customer behavior, preferences, and needs by analyzing multiple data
sources, such as online interactions, social media sentiment, purchase
history, and demographic information. This knowledge enables
businesses to personalize their offerings.
18. Short Note on Association Rule Mining and Deep Learning.
- Association rule mining finds interesting associations and
relationships among large sets of data items. This rule shows how
frequently an itemset occurs in a transaction. A typical example is a
Market Based Analysis. It is one of the key techniques used by large
relations to show associations between items. It allows retailers to
identify relationships between the items that people buy together
frequently. Given a set of transactions, we can find rules that will
predict the occurrence of an item based on the occurrences of other
items in the transaction. - Deep learning is a branch of machine
learning which is based on artificial neural networks. It is capable of
learning complex patterns and relationships within data. In deep
learning, we don’t need to explicitly program everything. It has become
increasingly popular in recent years due to the advances in processing
power and the availability of large datasets. Because it is based on
artificial neural networks (ANNs) also known as deep neural networks
(DNNs). These neural networks are inspired by the structure and
function of the human brain’s biological neurons, and they are
designed to learn from large amounts of data.
16. Write the role of activation function in neural network.
The activation function in Neural Networks takes an input 'x' multiplied
by a weight 'w'. Bias allows you to shift the activation function by
adding a constant (i.e. the given bias) to the input. Bias in Neural
Networks can be thought of as analogous to the role of a constant in a
linear function, whereby the line is effectively transposed by the
constant value.
20. Write big data analysis helpful in increasing business revenue?
Explain the steps to be followed to deploy a big data solution.
Improved Accuracy : Big data analytics enables businesses to make
decisions based on facts and evidence rather than intuition or
guesswork. By analyzing large volumes of data, patterns and trends
that may not be apparent at a smaller scale can be identified.
Real-time Insights : In the fast-paced business environment, real-time
insights are essential for timely decision making. Big data analytics
allows organizations to process and analyze data in realtime or near
real-time, enabling them to respond quickly to emerging trends,
market shifts, and customer demands.
Customer Understanding : Understanding customers is vital for
tailoring products, services, and marketing strategies. Big data
analytics provides a holistic view of customer behavior, preferences,
and needs by analyzing multiple data sources, such as online
interactions, social media sentiment, purchase history, and
demographic information. This knowledge enables businesses to
personalize their offerings, deliver targeted marketing campaigns, and
enhance the overall customer experience.
Competitive Advantage : In today’s competitive landscape, gaining an
edge over rivals is crucial. Big data analytics helps companies uncover
insights that can differentiate them from competitors. By identifying
market trends, consumer preferences, and emerging opportunities,
businesses can develop innovative products, optimize pricing
strategies, and deliver superior customer service.
Risk Management : Big data analytics plays a vital role in risk
management by identifying potential risks and predicting future
outcomes. By analyzing historical data and using predictive modelling
techniques, businesses can identify potential threats, fraud patterns,
and anomalies. This empowers organizations to take proactive
measures to mitigate risks, improve security, and safeguard their
operations, reputation, and financial well-being.
estimated? Give example.
Elbow Method : It is one of the most popular ways to find the optimal
number of clusters. This method uses the concept of WCSS (Within
Cluster Sum of Squares) value, which defines the total variations within
a cluster. Gap Statistic Method : It compares the total within intra-
cluster variation for different values of k with their expected values
under null reference distribution of the data. The estimate of the
optimal clusters will be value that maximize the gap statistic (i.e., that
yields the largest gap statistic). This means that the clustering
structure is far away from the random uniform distribution of points.
Silhouette Approach : It measures the quality of a clustering, i.e., it
determines how well each object lies within its cluster. A high average
silhouette width indicates a good clustering. Average silhouette
method computes the average silhouette of observations for different
values of k. The optimal number of clusters k is the one that maximize
the average silhouette over a range of possible values for k.
10. Describe cleansing and what are the best ways to practice data
cleansing?
Data Cleansing, also called Data Scrubbing is the first step of data
preparation. It is a process of finding out and correcting or removing
incorrect, incomplete, inaccurate, or irrelevant data in the dataset. If
data is incorrect, outcomes and algorithms are unreliable, though they
may look correct. The best practices for Data Cleaning are : - Develop a
data quality strategy. - Correct data at the point of entry. - Validate the
accuracy of data. - Manage removing duplicate data.
11. Define the Complexity theory of Map Reduce. What is the reduce
size of Map Reduce?
In the context of MapReduce, complexity theory refers to the analysis
of the computational complexity of algorithms and problems when
using the MapReduce programming model. Time Complexity : Map
Phase : If n is the number of input records and m is the size of the input
data, the time complexity of the Map phase is typically O(n+m). Shuffle
and Sort : The time complexity depends on the efficiency of the
shuffling mechanism and the size of the data being shuffled. Reduce
Phase : If r is the number of reducer nodes and k is the number of
unique keys, the time complexity of the Reduce phase is often O(r+k).
The Reducer Of Map-Reduce is consist of mainly 3 processes/phases:
Shuffle : Shuffling helps to carry data from the Mapper to the required
Reducer. With the help of HTTP, the framework calls for applicable
partition of the output in all Mappers. Sort : In this phase, the output of
the mapper that is actually the key-value pairs will be sorted on the
basis of its key value. Reduce : Once shuffling and sorting will be done
the Reducer combines the obtained result and perform the
computation operation as per the requirement.
OutputCollector.collect() property is used for writing the output to the
HDFS. Keep remembering that the output of the Reducer will not be
sorted.
12. Challenges of Conventional System.
Fundamental challenges - Storage - Processing - Security - Finding and
Fixing Data Quality Issues - Evaluating and Selecting Big Data
Technologies - Data Validation - Scaling Big Data Systems
13. What are the different hierarchical methods for cluster analysis?
Hierarchical clustering refers to an unsupervised learning procedure
that determines successive clusters based on previously defined
clusters. It works via grouping data into a tree of clusters. Hierarchical
clustering stats by treating each data points as an individual cluster.
There are two methods of Hierarchical Clustering : - Agglomerative
Clustering : Agglomerative clustering is a bottom-up approach. It starts
clustering by treating the individual data points as a single cluster then
it is merged continuously based on similarity until it forms one big
cluster containing all objects. It is good at identifying small clusters. -
Divisive Clustering : Divisive clustering works just the opposite of
agglomerative clustering. It starts by considering all the data points
into a big single cluster and later on splitting them into smaller
heterogeneous clusters continuously until all data points are in their
own cluster. Thus, they are good at identifying large clusters. It follows
a top-down approach and is more efficient than agglomerative
clustering. But, due to its complexity in implementation, it doesn’t
have any predefined implementation in any of the major machine
learning frameworks.
14. State the Quadratic Determinant Analysis.
Quadratic Discrimination is the general form of Bayesian
discrimination. Discriminant analysis is used to determine which
variables discriminate between two or more naturally occurring
groups. Difference from LDA is it relaxed the assumption that the mean
and covariance of all the classes were equal. Working : QDA is a variant
of LDA in which an individual covariance matrix is estimated for every
class of observations. QDA is particularly useful if there is prior
knowledge that individual classes exhibit distinct covariances. QDA
assumes that observation of each class is drawn from a normal
distribution (similar to linear discriminant analysis). QDA assumes that
each class has its own covariance matrix (different from linear
discriminant analysis) The advantage of quadratic discriminant
analysis over linear discriminant analysis and linear regression is that
when the decision boundaries are linear, the linear discriminant
analysis and logistic regression will perform well.
15. What is the role of statistical model in Data Analytics?
Statistical modelling is the process of applying statistical analysis to a
dataset. A statistical model is a mathematical representation (or
mathematical model) of observed data. When data analysts apply
various statistical models to the data they are investigating, they are
able to understand and interpret the information more strategically.
Rather than sifting through the raw data, this practice allows them to
identify relationships between variables, make predictions about
future sets of data, and visualize that data so that non-analysts and
stakeholders can consume and leverage it.
16. Explain how HADOOP is related to Big Data?
What are the features of HADOOP? HADOOP is an open source, Java
based framework used for storing and processing big data. The data is
stored on inexpensive commodity servers that run as clusters. Its
distributed file system enables concurrent processing and fault
tolerance. HDFS - Hadoop comes with a distributed file system called
the Hadoop Distributed File System (HDFS) which was designed for Big
Data processing. - It attempts to enable storage of large files, by
distributing the data among a pool of data nodes. - It holds very large
amount of data and provides easier access. - It is highly fault tolerant
and designed using low-cost hardware.
14. Can you express multiple one way repeated measure ANOVA on a
two way design?
- A one-way ANOVA is primarily designed to enable the equality testing
between three or more means. A two-way ANOVA is designed to assess
the interrelationship of two independent variables on a dependent
variable. - A one-way ANOVA only involves one factor or independent
variable, whereas there are two independent variables in a two-way
ANOVA. - In a one-way ANOVA, the one factor or independent variable
analysed has three or more categorical groups. A two-way ANOVA
instead compares multiple groups of two factors. - One-way ANOVA
need to satisfy only two principles of design of experiments, i.e.,
replication and randomization. As opposed to two-way ANOVA, which
meets all three principles of design of experiments which are
replication, randomization and local control.
15. Group the main characteristics of big data. Why you need to take
big data?
Five Vs of Big Data : Volume - The name Big Data itself is related to an
enormous size. Big Data is a vast 'volumes' of data generated from
many sources daily, such as business processes, machines, social
media platforms, networks, human interactions, and many more.
Variety - Big Data can be structured, unstructured, and semi-
structured that are being collected from different sources. Data will
only be collected from databases and sheets in the past, But these
days the data will comes in array forms, that are PDFs, Emails, audios,
photos, videos, etc. Veracity - Veracity means how much the data is
reliable. It has many ways to filter or translate the data. Veracity is the
process of being able to handle and manage data efficiently. Value -
Value is an essential characteristic of big data. It is valuable and
reliable data that are used for storing, processing, and analysing.
Velocity - Velocity creates the speed by which the data is created in
real-time. It contains the linking of incoming data sets speeds, rate of
change. The primary aspect of Big Data is to provide demanding data
rapidly.