MODULE 1
How much data does google handle?
- Google now processes over 40,000
search queries every second on
average, which translate to over 3.5
billion searches per day and 1.2 trillion
searches per year worldwide.
- Google currently processes over 20
petabytes of data per day through an
average of 100,000 MapReduce jobs
spread across its massive computing
clusters.
- The average MapReduce job ran across
approximately 400 machines in
September 2007, crunching
approximately 11,000 machine years in
a single month.
Why do we summarize data?
The main objective of summarizing data is to
easily identify the characteristics of the data to
be processed.
Summarizing your data will enable you to
classify the normal values of your data and
uncover what are the odd values present on
your dataset.
What is raw data?
Raw data pertains to the collected data before
it’s processed or ranked.
Qualitative raw data
Suppose you are tasked to gather the ages (in
years) of 50 students in a university.
Categorical raw data
Another example is gathering the student status
of the same 50 students.
Statistical Analysis
Statistics is defined as the science of collecting,
analyzing, presenting, and interpreting data, as
well as of making decisions based on such
analysis.
Types of Statistics
• Descriptive statistics is consists of different
techniques for organizing, displaying, and
describing data by using labels, graph, and
summary measures.
• Inferential statistics is consists of methods
that use sample results to help make decisions
or predictions about a population.
Underlying your dataset
A data set is a collection of observations on one
or more variables. (collection of data from
excel, database, row in a data mining)
A variable is a characteristic under study that
assures different values for different elements.
(age, name etc)
Observation or measurement pertains to a
value of a variable.(age 10 or 20 etc)
Basic Terms
In the above table, the variables are the
customer ID, first name, last name, address, and
age. The data set are all the transactions from
customer 1 to 5. Each transaction can also be
called as measurement or observation.
What are the different types of variables
available?
 help of summary statistics and graphical
representations.
Quantitative Variables

It pertains to a variable that can be measured

numerically. A data collected on a quantitative
variable are called quantitative data.
Quantitative variables are divided into two
types: discrete variables and continuous
variables.
Measures of Central Tendencies
A basic step in exploring your data is getting a
“typical value” for each feature (variable): an
estimate of where most of the data are located
(i.e. their central tendency).

Discrete Variables

Discrete variable is a variable whose values are

countable. In other words, a discrete variable
can assume only certain values. (ex. Age,
number of siblings and pertains to specific
values)

Continuous Variables

These are data that can take on any interval

values. Also called as float, interval, numeric.

Qualitative Variables Arithmetic Mean

Qualitative or categorical variable pertains to
the variables that values couldn’t be measured.
Ordinal Variables
Categorical data that has an explicit ordering.
Also called as ordered factor.
The arithmetic mean is calculated by adding
together the values in the sample. The sum is
then divided by the number of items in the
sample (the replication).

Binary Variables

A special case of categorical with just two

categories. (0/1, True or False) Also called as
dichotomous, logical, indicator.

Qualitative vs Quantitative
Mean

The median is the middle value, taken when you

arrange your numbers in order (rank).

When calculating for median, one must

remember the following:

• Arrange the data values of the given data set

in increasing order. (from smallest to largest)
Exploratory Data Analysis (EDA)
• Find the value that divides the ranked data set
It refers to the critical process of performing in two equal parts.
initial investigations on the data so as to
discover patterns, to spot anomalies, to test Mode
hypothesis and to check assumptions with the
It is the most frequent value in a sample. It is Interpreting Standard Deviation
calculated by working out how many there are
• A small standard deviation means that the
of each value in your sample. The one with the
values in a statistical data set are close to the
highest frequency is the mode. It is possible to
mean of the data set, on average
get tied frequencies, in which case you report
both values. • A large standard deviation means that the
values in the data set are farther away from the
Types of Modes
mean, on average.
1. Unimodal – the distribution is said to be
• The standard deviation can never be a
unimodal if it has only one value with highest
negative number.
number of occurences.
• The smallest possible value for the standard
2. Bimodal – the data set contains two modes.
deviation is 0
3. Multimodal – if the distribution has more
• The standard deviation is affected by outliers
than two modes.
(extremely low or extremely high numbers in
Measures of Dispersion the data set). That’s because the standard
deviation is based on the distance from the
A second dimension, variability, also referred to
mean.
as dispersion, measures whether the data
values are tightly clustered or spread out. • The standard deviation has the same units as
the original data.

Exploring the Data Distribution

Two ways to explore distribution

1. Graphically – using frequency histograms or

tally plots draws a picture of the sample shape.
2. Shape statistics – such as skewness and
kurtosis.

Range Histogram

The difference between the largest and the A plot of the frequency table with the bins on
smallest value in a data set. the x-axis and the count (or pro‐ portion) on the
y-axis.
𝑟𝑎𝑛𝑔𝑒 = 𝑙𝑎𝑟𝑔𝑒𝑠𝑡 𝑣𝑎𝑙𝑢𝑒 − 𝑠𝑚𝑎𝑙𝑙𝑒𝑠𝑡 𝑣𝑎𝑙𝑢e

Standard Deviation

The standard deviation is used when the data

are normally distributed. You can think of it as a
sort of “average deviation” from the mean.

The general formula for calculating standard

deviation looks like the following:

Understanding Distribution

• If a histogram is symmetric, then it can be said

that the values for the mean, median, and
mode are the equal.
 Shapes Statistics

1. Skewness – a measure of how central the

average is in the distribution.

2. Kurtosis – a measure of how pointy the

distribution

Skewness

The skewness of a sample is a measure of how

• If a histogram and frequency distribution is
central the average is in relation to the overall
skewed to the right it means that the value of
spread of values.
the mean is the largest among the three and
mode has the lowest value. Positively Skewed

A positive value indicates that the average is

skewed to the left, that is, there is a long “tail”
of more positive values.

• If a histogram and frequency distribution are

skewed to the left, the value of the mean is the
smallest and mode has the largest value.

Negatively Skewed

A negative value indicates that the average is

skewed to the right, that is, there is a long “tail”
of more positive values.

Boxplot

A plot introduced by Tukey as a quick way to

visualize the distribution of data.

STATISTICAL ANALYSIS

Consider this scenario: A new advertisement for

Tom and Jerry’s rash guard introduced in May
of last year resulted to a 30% increase in ice
cream sales for the following three months.
Thus, the advertisement was effective.

Problem: In reality, rash guard sales typically
increases in summer months. Thus, there is a
“history effect”.
What is statistics?
Statistics is the science concerned with
developing and studying methods for collecting,
analyzing, interpreting and presenting empirical
data.
• Rely upon the calculation of numbers
• Rely upon how the numbers are chosen and
how statistics are interpreted
Types of Statistics
• Descriptive Statistics - describing and
summarizing data sets using pictures and
statistical quantities.
• Inferential Statistics - analyzing data sets and
drawing conclusions from them
• Probability - the study of chance events
governed by rules (or laws)
What is hypothesis testing?
Hypothesis testing is a statistical procedure for
testing whether chance is a plausible
explanation of an experimental finding.
What is null hypothesis?
Null Hypothesis is usually denoted by H0 and
the alternative hypothesis is denoted by H1.
The null hypothesis is a statement that is
assumed to be true at the beginning of an
analysis.
Suppose you are trying to test a certain case,
initially, the person being questioned is not
guilty. This initial verdict is the null hypothesis.
The contrary of this claim is the alternative
hypothesis.
Claim: Coke claims that each bottle of their soft
drink is filled at 8 oz. of all the time.
𝐻0: 𝑒𝑎𝑐ℎ 𝑏𝑜𝑡𝑡𝑙𝑒 𝑖𝑠 𝑓𝑖𝑙𝑙𝑒𝑑 𝑤𝑖𝑡ℎ 8 𝑜𝑧 𝑜𝑓 𝑠𝑜𝑑𝑎
𝑙𝑒𝑡 𝜇 𝑖𝑠 𝑡ℎ𝑒 𝑎𝑣𝑒𝑟𝑎𝑔𝑒 𝑜𝑓 𝑜𝑓 𝑠𝑜𝑑𝑎 𝑐𝑜𝑛𝑡𝑒𝑛𝑡 𝐻0:
𝜇 = 8 oz
𝐻1: 𝑒𝑎𝑐ℎ 𝑏𝑜𝑡𝑡𝑙𝑒 𝑖𝑠 𝑓𝑖𝑙𝑙𝑒𝑑 𝑤𝑖𝑡ℎ 𝑙𝑒𝑠𝑠 𝑡ℎ𝑎𝑛 8𝑜𝑧
𝑜𝑓 𝑠𝑜𝑑𝑎 𝑙𝑒𝑡 𝜇 𝑖𝑠 𝑡ℎ𝑒 𝑎𝑣𝑒𝑟𝑎𝑔𝑒 𝑜𝑓 𝑜𝑓 𝑠𝑜𝑑𝑎
𝑐𝑜𝑛𝑡𝑒𝑛𝑡 𝐻0: 𝜇 < 8 oz
Conducting
Student’s T-test
Student’s T-test
In statistics, student’s t-test is a method
developed by William Sealy Gosset used in
testing hypothesis about the mean of a small
sample drawn from a normally distributed
population when the population standard is
unknown.
Scenario
You want to know if there’s a difference with
the height of crops from two fields.
𝐻0:There’s no statistically significant difference
between the samples.

Identify Critical Value

If the t value < critical value = don’t reject the

null hypothesis

If the t value > critical value = reject the null

hypothesis

Significance Levels (Alpha)

A significance level, also known as alpha or α, is

an evidentiary standard that a researcher sets
before the study. It defines how strongly the
sample evidence must contradict the null
hypothesis before you can reject the null
hypothesis for the entire population.

Confidence Intervals

The table shows the equivalent confidence

levels and levels of significance.

𝐻0:There’s no statistically significant difference

between the samples.

Identify p value

If the p value > critical value = don’t reject the

null hypothesis

If the p value < critical value = reject the null

hypothesis

𝑑𝑒𝑔𝑟𝑒𝑒𝑠 𝑜𝑓 𝑓𝑟𝑒𝑒𝑑𝑜𝑚 = 𝑛1 + 𝑛2 – 2

• 𝑑𝑓 = 16 + 16 − 2 = 30
• 𝑑𝑓 = 30
Since the value for T-test is 0.026
• Since the t value is greater than 0.025, we
reject the hypothesis.
Assumptions in T-test
• independent observations
• normally distributed data for each group
• equal variances for each group
One-way ANOVA
What is ANOVA?
Analysis of variance (ANOVA) is a statistical
technique used to compare the means of two or
more groups of observations or treatments. For
this type of problem, you have the following:
• a continuous dependent variable, or response
variable
• a discrete independent variable, also called a
predictor or explanatory variable.
One-way ANOVA
Use analysis of variance to test for differences
between population means.
Research Questions for One-way ANOVA
• Do accountants, on average, earn more than
teachers?
• Do people treated with one of two new drugs
have higher average T-cell counts than people
in the control group?
• Do people spend different amounts
depending on which type of credit card they
have?
Assumptions in ANOVA
• Observations are independent.
• Errors are normally distributed.
• All groups have equal response variances.
Correlation
What is correlation?
• Exploratory data analysis in many modeling
projects (whether in data science or in research)
involves examining correlation among
predictors, and between predictors and a target
variable.
• Variables X and Y (each with measured data)
are said to be positively correlated if high values
of X go with high values of Y, and low values of
X go with low values of Y.
Correlation Coefficient
• A metric that measures the extent to which
numeric variables are associated with one
another (ranges from -1 to +1).
• Scatterplot is a plot in which the x-axis is the
value of one variable, and the y-axis the value of
another.
Assumptions in Correlation
• The correlation coefficient measures the
extent to which two variables are associated
with one another.
• When high values of v1 go with high values of
v2, v1 and v2 are positively associated
• When high values of v1 are associated with
low values of v2, v1 and v2 are negatively
associated
• The correlation coefficient is a standardized
metric so that it always ranges from -1 (perfect
negative correlation) to +1 (perfect positive
correlation)
• 0 indicates no correlation, but be aware that
random arrangements of data will produce both
positive and negative values for the correlation
coefficient just by chance
Types of Correlation
SUPPLEMENTARY MATERIAL 1
Every data analysis requires data. Data can be in
different forms such as images, text, videos, and
etc, that are usually gathered from different
data sources. Organizations store data in
different ways, some through data warehouses,
traditional RDBMS, or even through cloud. With
the voluminous amount of data that an
organization processes each day, the dilemma
on how to start data analysis emerges.
How do we start performing an analysis?
First and foremost, know your data.
To understand your organization’s data, there
are numerous techniques that can be used. In
module 1, the most common techniques will be
identified.
What is raw data?
Raw data pertains to the collected data before
it’s processed or ranked. Suppose you are
tasked to gather the ages (in years) of 50
students in a university.
The table below is an example of quantitative
raw data.
Another example is gathering the student status
of the same 50 students. Now we will have an
example of categorical raw data which is
presented in the table below.
The examples above can also be called
ungrouped data. An ungrouped data set
contains information on each member of a
sample or population individually.
Raw data can be summarized through charts,
dashboards, tables, and numbers. The most
common way to describe raw data is through
the frequency distributions.
A frequency distribution shows how the
frequencies are distributed over various
categories.
Below is a frequency table that summarized the
survey conducted by Gallup Poll about the
Worries About Not Having Enough Money to
Pay Normal Monthly Bills.
A frequency distribution of a qualitative variable
enumerates all the categories and the number
of instances that belong to each category.
Let’s transform the following responses into a
frequency table in order to interpret the data
better.
Is it easier to understand through frequency
table, isn’t it? In order to study data, in this
module we will be using statistics.
Statistics is defined as the science of collecting,
analyzing, presenting, and interpreting data, as
well as of making decisions based on such
analysis.
Since statistics is a broad body of knowledge, it
is divided into two areas: descriptive statistics
and inferential statistics.
What is descriptive statistics?
Descriptive statistics is consists of different
techniques for organizing, displaying, and
describing data by using labels, graph, and
summary measures.
What is inferential statistics?
It is consist of methods that use sample results
to help make decisions or predictions about a
population.
Basic terms
A variable is a characteristic under study that
assures different values for different elements.
Observation or measurement pertains to a
value of a variable.
A data set is a collection of observations on one
or more variables.
In the above table, the variables are the
customer ID, first name, last name, address, and
age. The data set are all the transactions from
customer 1 to 5. Each transaction can also be
called as measurement or observation.
Types of Variables
1. Quantitative Variables It pertains to a
variable that can be measured numerically. A
data collected on a quantitative variable are
called quantitative data.
Examples: Income, height, gross sales, price of a
home, number of cars owned.
Quantitative variables are divided into two
types: discrete variables and continuous
variables.
• Discrete variable is a variable whose values
are countable. In other words, a discrete
variable can assume only certain values.
• Continuous variables are variables that can
assume any numerical value or interval.
2. Qualitative or Categorical Variables
It pertains to the variables that values couldn’t
be measured.

Population Vs Sample
A population is consist of all elements –
individuals, items, or objects, whose
characteristics are being studied.
A sample is a portion of the population selected
for study.
Visual example of Population vs Sample
Let’s proceed to the different ways to group and
study data.
Aside from using the frequency table, we can
also make use of the different graphs that are
commonly used to visually present data.
The first one is a bar graph. A bar graph is made
of bars whose heights represent the frequencies
of respective categories. One type of bar graph
is called pareto chart.
Pareto chart is a bar graph where in the bars
are arranged based on their heights. It is
arranged in descending order (largest to
smallest).
Another way to present data is through pie
chart. A pie chart is a circle divided into portions
that represent frequencies or percent ages of a
population.
To graph grouped data, we can use of the
following methods:
A group data can be presented using
histograms. Histograms can be drawn for a
frequency distribution. A histogram is a graph
which classes are marked on horizontal axis and
the frequencies, relative frequencies, or
percentages are marked on the vertical axis.
The above histogram shows the percentage
distribution of annual car insurance premiums
in 50 states. The data used to make this
distribution and histogram are based on
estimates made by insure.com.
Understanding Frequency Distribution Curve
Knowing the meaning for each curve in
histogram would be helpful to interpret a
dataset. A histogram can be:
1. Symmetric
2. Skewed
3. Uniform rectangular
A symmetric histogram is the type of histogram
which both sides are equal.
 SUPPLEMENTARY MATERIAL 2

Examples of statistical analysis application in

real life:

If a histogram doesn’t have equal sides, it is said
to be skewed.
• A skewed-to-the-right histogram has longer
tail on the right side.
1. Manufacturers use statistics to weave quality
into beautiful fabrics, to bring lift to the airline
industry and to help guitarists make beautiful
music.
2. Researchers keep children healthy by using
statistics to analyze data from the production of
viral vaccines, which ensures consistency and
safety.

3. Communication companies use statistics to

• A skewed-to-the-left histogram has longer tail
on the left.
optimize network resources, improve service
and reduce customer churn by gaining greater
insight into subscriber requirements.
4. Government agencies around the world rely
on statistics for a clear understanding of their
countries, their businesses and their people.

Understanding the measures of central

tendencies:

Measures of central tendencies are useful in

If the histogram has equal values then it would
be considered as uniform or rectangular
histogram.
identifying the middle value for histograms and
frequency distribution. Methods used to
calculate for the measures central tendencies
can determine the typical values that can be
found in your data.

Measures of Tendencies

1. Mean
What is mean?
2. Median
Mean is also called as arithmetic mean which
3. Modes pertains to the sum of all the values over the
number of items added.
Measures of Dispersion

1. Range

2. Variance

3. Standard Deviation

What is median?
The median is the middle value, taken when you
arrange your numbers in order (rank). This
measure of the average does not depend on the
shape of the data.
When calculating for median, one must
remember the following:
1. Arrange the data values of the given data set
in increasing order. (from smallest to largest)
2. Find the value that divides the ranked data
set in two equal parts.
Example:
Find the median for 2014 compensation:
Step 1: Arrange the values in increasing order
16.2 16.9 19.3 19.3 19.6 21.0 22.2 22.5 28.7
33.7 42.1
Step 2: Identify the center of the data sets.
16.2 16.9 19.3 19.3 19.6 21.0 22.2 22.5 28.7
33.7 42.1
For this example, the middle value is 21.0,
therefore it is the median of the data values.
There are instances where the data values are
even, in this case, the two middle numbers are
gathered and divided into two.
See the example below: The following data
describes the cell phone minutes used last
month by 12 randomly selected customers:
230, 2053, 160, 397, 510, 380, 263, 3864, 184,
201, 326, 721
Here, we need to arrange the data values first.
It will give us:
160 184 201 230 263 326 380 397 510 721 2053
3864
If we observe our data, we have no central
value, to compute for the median, we need to
identify the values that divide the data into two
equal parts. In our case we have 326 and 380.
We can only have one median per data set, so
the median would be calculated using:
What is mode?
Mode is the value that occurs the most frequent
in the given dataset.
Let’s identify the most frequent values in the
following example:
77 82 74 81 79 84 74 78
In this data set, the value 74 appears twice.
Therefore, our mode is 74.
When a dataset has only one value that repeat
the most then the distribution can be called as
unimodal. If the data in the distribution has two
values that repeat the most, then it is called
bimodal. If there are more than two modes in a
dataset, it is said to be multimodal.
Relationships among mean, median, and mode:
• If a histogram is symmetric, then it can be
said that the values for the mean, median, and
mode are the equal.
 What is standard deviation?

This method is the most commonly used

measure of dispersion. The value of standard
deviation tells how closely the values of a data
set are clustered around the mean.

The things one must remember when dealing

• If a histogram and frequency distribution is
skewed to the right it means that the value of
the mean is the largest among the three and
mode has the lowest value. If the mean is the
largest, it means that the dataset is sensitive to
outliers that occur in the right.
with standard deviation are the ff:
• A lower value of standard deviation indicates
that the data set are spread over a smaller
range around the mean.
• A larger value of the standard deviation
indicates that the values of that data set are
spread over a relatively larger range around the
mean.

• The standard deviation can be obtained by

taking the positive square root of the variance.

• If a histogram and frequency distribution are

skewed to the left, the value of the mean is the
smallest and mode has the largest value. In this
scenario, the left tail contains the outliers.

Measures of Dispersion

If an analyst wants to know how disperse a

dataset is, the methods of calculating the
measures of dispersion can be used. Measures
of dispersion can help in determining how
spread the data values are.

What is range?

Range is the simplest method to compute when

measuring the dispersion of data. The range can
be obtained by subtracting the smallest value in
the dataset from the largest value.
FORMATIVES 11. If the histogram is skewed right, the mean is
greater than the median.
1. If a histogram and frequency distribution are
skewed to the left, the value of the means is the - True
largest.
12. Identify whether the statement below is an
- False example of

2. Which of the following is an example of a) positive correlation

categorical raw data?
b) negative correlation
- Collected subjects offered
c) no correlation
3. You conduct a survey where the respondents
The more one eats, the less hunger one will
could choose from good, better, best, excellent.
have.
What type of variable should contain this type
of data? - Positive correlation
- Interval 14. Boxplot is a plot in which the x-axis is the
value of one variable, and the y-axis the value of
4. Student height is a categorical variable.
another.
- True
- False
5. Quantitative variable pertains to the variables
15.0 indicates no correlation, but be aware that
that values couldn’t be measured.
random arrangement of data will produce both
- False positive and negative values for the correlation
coefficient just by chance

6. The data below is bimodal.
Data: 77 82 74 81 79 84 74 82
- True
- True
16. The alternative hypothesis or research
hypothesis Ha represents an alternative claim
about the value of the parameter.

8. What is the median of the following data? - True

17. Identify whether the statement below is an

example of

a) positive correlation

b) negative correlation

c) no correlation

As one increases in age, often one’s agility

- 24
decreases.
9. People’s age is an example of continuous
- Negative correlation
variable
18. Find the median: 15, 5, 9, 18, 22, 25, 5
- True
- 12
10. It is a portion of the population selected for
study. (Use lowercase for your answer) 19. Observe the histogram below. Based on it,
how many students were greater than or equal
- Sample
to 60 inches tall?

- 11
20. The height of three volleyball players is
210 cm, 220 cm and 191 cm. What is their
average height?
- 207
20/20
1. The standard deviation can never be a
negative number
- True
2. It refers to the critical process of
performing initial investigations on the data
so as to discover patterns, to spot anomalies,
to test hypothesis and to check assumptions
with the help of summary statistics and
graphical representations
- Exploratory Data Analysis
3. A variable is a characteristic under study
that assures different values for different
elements.
- True
4. The data that is collected before being
processed is called statistical data.
- False
5. The smallest possible value for the
standard deviation is 1.
- False
6. Observation or measurement pertains
to a value of a variable.
- True
7. Customer gender is an example of ordinal
data.
- False
8. If the distribution has more than two
modes, it is called multimodal.
- True
9. It is a measure of how central the average
is in relation to the overall spread of values
- Skewness
10. A special case of categorical with just
two categories is called logical.
- True
11. Identify whether the statement
below is a null hypothesis or alternative
hypothesis:
Contrary to popular belief, people can
see through walls.
- alternative hypothesis
12. The height of three volleyball players is
210 cm, 220 cm and 191 cm. What is their
average height?
- 207
13. 0 indicates no correlation, but be aware
that random arrangements of data will
produce both positive and negative values
for the correlation coefficient just by chance
- True
14. Inferential statistics is about describing
and summarizing data sets using pictures
and statistical quantities
- False
15. Identify whether the statement
below is an example of
a) positive correlation
b) negative correlation
c) no correlation
 
The more you exercise your muscles,
the stronger they get.
 - positive correlation

16. Inferential Statistics is about

analyzing data sets and drawing
conclusions from them
- True
17. A residual is the difference between
the observed value of the response and
the predicted value of the response
variable.
- True
18. The smallest possible value for the
standard deviation is 0.
- True
19. When conducting two-tailed T-test,
the data is normally distributed.
- True
20. When x and y are positively
correlated, as the value of x increases,
the value of y tends to increase as well.
- True
 - Instagram’s algorithm

MODULE 2

- Weather forecast
- Singapore’s supply and demand
gap on a typical day
Data Mining
is defined as a process used to extract usable
insights from a larger set of any raw data.

It implies analysing data patterns in large

batches of data using one or more software.

Key Features of Data Mining

• Automatic pattern predictions based on trend

- Microsoft’s voice recognition and behavior analysis.

• Prediction based on likely outcomes.

• Creation of decision-oriented information.

• Focus on large data sets and databases for

analysis.

• Clustering based on finding and visually

documented groups of facts not previously
known.

- Gmail’s spam
Key Terms

• A data set may have attribute variables and

target variable(s).

• The values of the attribute variables are used

to determine the values of the target
variable(s).

• Attribute variables and target variables may

also be called as independent variables and
dependent variables, respectively, to reflect
that the values of the target variables depend
on the values of the attribute variables Data Patterns learned in data mining

• Classification and prediction patterns

• Cluster and association patterns

• Data reduction patterns

• Outliers and anomaly patterns

• Sequential and temporal patterns

Data-mining approaches can be separated into

two categories

Two major approaches in data mining

1. Supervised learning – the desired output is

Independent variables known.

2. Unsupervised learning – it is used against

data that has no historical labels.

Data Analytics Lifecycle

• Big data analysis differs from traditional data

analysis primarily due to the volume, velocity,
and variety characteristics of data being
processed.

What is big data?

Normal data comes from your traditional

relational database system

Big data comes from different data sources.
Data are usually in petabytes, or terabytes
Characteristics of Big Data
1 Volume - big data starts as low as 1 terabyte
and it has no upper limit
2 Velocity - big data enters an average system
at velocity ranging between 30 Kilobytes (KB)
p/sec to as much as 30 gigabytes (GB) per sec
3 Variety - big data is composed of
unstructured, semi structured, structured
dataset
How can you ensure the success of an analytics
projects?
PEOPLE
First, you need the right people.
Key Roles for successful Analytics Project
• Business User – understands the domain area
• Project Sponsor – provides requirements
(monetary, equipment, etc.)
• Project manager – ensures meeting
objectives.
• Business Intelligence Analyst – provides
business domain expertise based on deep
understanding of the data
• Database Administrator (DBA) – creates DB
environment
• Data Engineer – provides technical skills, assist
data management and extraction, supports
analytic sandbox
• Data Scientist – provides analytic techniques
data and modeling
Different Analytical Life Cycle
Data Life Cycle according to SAS
1st Phase: Ask
Whether you’re a data scientist developing a
new model to reduce churn, or a business
executive wanting to improve the customer
experience, this phase defines what your
business needs to know.
2nd Phase: Prepare
This phase is both critical to success and
frustratingly timeconsuming. You have data
sitting in database, on desktops or in Hadoop,
plus you want to capture live-streaming data.
3rd Phase: Explore
In this phase, you’ll search for relationships,
trends and patterns to gain a deeper
understanding of your data.
You’ll also develop and test hypotheses through
rapid prototyping in an iterative process.
4th Phase: Model
This is the phase where you code your own
models using R or Python or an interactive
predictive software like SAS.
Organizations must be able to identify the
appropriate algorithm to analyze the available
data.
5th Phase: Implement
After validating the accuracy of the model
generated during the previous phase, it must be
implemented in the organization.
Models are usually deployed by automating
common manual tasks.
6th Phase: Act
In this phase, we enable two types of decisions:
operational that are automated, and strategic
decisions where individuals make a long term
impact.
7th Phase: Evaluate
Organizations must always monitor the
performance of the model generated.
When the performance starts to degrade below
the acceptance level, the model can be
recalibrated or replaced with a new model.
8th Phase: Ask again
The marketplace changes. Your business
changes. And that’s why your analytics process
occasionally needs to change.
What is KKD?
KKD
The term Knowledge Discovery in Databases, or
KDD for short, refers to the broad process of
finding knowledge in data, and emphasizes the
"high-level" application of particular data
mining methods.
It is of interest to researchers in machine
learning, pattern recognition, databases,
statistics, artificial intelligence, knowledge
acquisition for expert systems, and data
visualization.
Steps of the KDD Process
Understanding each step
1. Developing an understanding of
- Data Selection
• the application domain
• the relevant prior knowledge
• the goals of the end-user
2. Creating a target data set:
selecting a data set or focusing on a subset of
variables, or data samples, on which discovery is
to be performed.
3. Data Cleaning and preprocessing
• Removal of noise or outliers
• Collecting necessary information to model or
account for noise
• Strategies for handling missing data fields
• Accounting for time sequence and known
changes
4. Data reduction and projection.
• Finding useful features to represent the data
depending on the goal of the task
5. Choosing the data mining task.
• Deciding whether the goal of the KDD process
is classification, regression, clustering, etc.
6. Choosing the data mining algorithm
• Selecting method(s) to be used for searching
for patterns in the data
• Deciding which models and parameters may
be appropriate
• Matching a particular data mining method
with the overall criteria of the KDD Process
7. Data mining.
• Searching for patterns of interest in particular
representation form or a set of such
representations as classification rules or tress,
regression, clustering, and so forth.
8. Interpreting mined patterns.
9. Consolidating discovered knowledge.
What is CRISP-DM?
CRISP-DM
The CRISP-DM methodology is described in
terms of a hierarchical process model,
consisting of sets of tasks described at four
levels of abstraction (from general to specific):
phase, generic task, specialized task, and
process instance
1st Phase: Business Underdstanding
This initial phase focuses on understanding the
project objectives and requirements from a
business perspective, then converting this
knowledge into a data mining problem
definition and a preliminary plan designed to
achieve the objectives.
2nd Phase: Data Underdstanding
The data understanding phase starts with initial
data collection and proceeds with activities that
enable you to become familiar with the data,
identify data quality problems, discover first
insights into the data, and/or detect interesting
subsets to form hypotheses regarding hidden
information.
3rd Phase: Data Preparation
The data preparation phase covers all activities
needed to construct the final dataset (data that
will be fed into the modeling tool(s)) from the
initial raw data.
Data preparation tasks are likely to be
performed multiple times and not in any
prescribed order. Tasks include table, record,
and attribute selection, as well as
transformation and cleaning of data for
modeling tools.
4th Phase: Modeling
In this phase, various modeling techniques are
selected and applied, and their parameters are
calibrated to optimal values.
Typically, there are several techniques for the
same data mining problem type. Some
techniques have specific requirements on the
form of data. Therefore, going back to the data
preparation phase is often necessary.
5th Phase: Evaluation
Before proceeding to final deployment of the
model, it is important to thoroughly evaluate it
and review the steps executed to create it, to
be certain the model properly achieves the
business objectives.
A key objective is to determine if there is some
important business issue that has not been
sufficiently considered. At the end of this phase,
a decision on the use of the data mining results
should be reached.
6th Phase: Deployment
Depending on the requirements, the
deployment phase can be as simple as
generating a report or as complex as
implementing a repeatable data mining process
across the enterprise.
In many cases, it is the customer, not the data • Priority Filtering
analyst, who carries out the deployment steps.
• Spam Filtering

• Text Processing
SUBTOPIC 1 INTRODUCTION TO DATA MINING
• Speech Recognition
What is Data Mining
• Face Recognition
Data mining aims at discovering useful data
patterns from massive amounts of data.

Artificial Intelligence

AI is the broad science of mimicking human

abilities.

AI is the science of training machines to perform

human tasks.

Machine Learning
Aspects of Business Analytics
Arthur Samuel (1959). Machine Learning: Field
of study that gives computers the ability to
learn without being explicitly programmed.

Tom Mitchell (1998) Well-posed Learning

Problem: A computer program is said to learn
from experience E with respect to some task T
and some performance measure P, if its
performance on T, as measured by P, improves
with experience E.

Suppose your email program watches which

emails you do or do not mark as spam, and Impact of Analytics
based on that learns how to better filter spam.
What is the task T in this setting?

• Classifying emails as spam or not spam. T

• Watching you label emails as spam or not

spam. E

• The number (or fraction) of emails correctly

classified as spam/not spam. P

• None of the above—this is not a machine

learning problem.
Data Mining Trends

• Ongoing research to ensure that analysts have

Standard Examples of Machine Learning access to modern techniques which are robust
and scalable
• Smart tagging
• Innovative computational implementations of
• Product Recommendations
existing analytical methods
• Creative applications of existing methods to Major Types of data patterns
solve new and different problems
Classification and Prediction
• Integration of methods from multiple
Allow us to classify or predict values of target
disciplines to provide targeted solutions
variables from values of attribute variables.
The role of big data in data mining
Cluster and Association

Cluster patterns give groups of similar data

records such that data records in one group are
similar but have larger differences from data
records in another group.

Association patterns are established based on

co-occurrences of items in data records

Data reduction patterns

Specific Applications
Data reduction patterns look for a small number
• Attrition/Churn prediction of variables that can be used to represent a
data set with a much larger number of variables
• Propensity to buy/avail of a product or service
Outliers and Anomaly Patterns
• Cross-sell or up-sell probability
Outliers and anomalies are data points that
• Next-best offer differ largely from the norm of data.
• Time-to-event modeling Sequential and Temporal Patterns
• Fraud detection Sequential and temporal patterns reveal
• Revenue/profit predictions patterns in a sequence of data points.

• Probability of default in credit risk assessment If the sequence is defined by the time over
which data points are observed, we call the
sequence of data points as a time series.
Data Mining can be performed in the ff data Data Mining Techniques
types:
- Classification
• Relational databases - Clustering
• Data warehouses - Regression
- Outer
• Advanced DB and information repositories - Sequential Patterns
- Prediction
• Object-oriented and object-relational
- Association Rules
databases
Challenges
• Transactional and Spatial databases
• Skilled Experts are needed to formulate the
• Heterogenous and legacy databases
data mining queries.
• Multimedia and streaming database
• Overfitting: Due to small size training
• Text databases database, a model may not fit future states.

• Text mining and web mining • Data mining needs large databases which
sometimes are difficult to manage
• Business practices may need to be modified to
determine to use the information uncovered.
• If the data set is not diverse, data mining
results may not be accurate.
• The data mining techniques are not accurate,
and so it can cause serious consequences in
certain conditions.

• Integration information needed from Industries that utilize data mining

heterogeneous databases and global
information systems could be complex ❑ Communications

❑ Insurance

Advantages of data mining ❑ Education

• Data mining technique helps companies to get ❑ Manufacturing

knowledgebased information.
❑ Banking
• Data mining helps organizations to make the
profitable adjustments in operation and ❑ Retail
production. ❑ Service providers
• The data mining is a cost-effective and ❑ E-commerce
efficient solution compared to other statistical
data applications. ❑ Super markets

• Data mining helps with the decision-making ❑ Crime

process.
❑ Bioinformatics
• Facilitates automated prediction of trends and
behaviors as well as automated discovery of
hidden patterns. SUBTOPIC 2 SUPERVISED AND UNSUPERVISED
• It can be implemented in new systems as well LEARNING
as existing platforms SUPERVISED LEARNING
• It is the speedy process which makes it easy • the desired output is known.
for the users to analyze huge amount of data in
less time. • also known as predictive modeling

• uses patterns to predict the values of the label

on additional unlabeled data
Disadvantages of data mining
• used in applications where historical data
• There are chances of companies may sell predicts likely future events
useful information of their customers to other
companies for money. For example, American Example Problem: House price prediction
Express has sold credit card purchases of their
customers to the other companies.

• Many data mining analytics software is

difficult to operate and requires advance
training to work on.

• Different data mining tools work in different

manners due to different algorithms employed Use of data in Supervised Learning
in their design. Therefore, the selection of
correct data mining tool is a very difficult task.
We can use the abundance of data to guard
against the potential for overfitting by
decomposing the data set into partitions:

Training dataset: Consists of the data used to

build the candidate models.

Test dataset: The data set to which the final

model should be applied to estimate this
model’s effectiveness when applied to data that
have not been used to build or select the
model.

• If there is only one dataset, it may be

partitioned into a training and test sets.

• The basic assumption is that the training and

test sets are produced by independent sampling
from an infinite population. • Partition a data set of observations into
increasingly smaller and more homogeneous
subsets.

• At each iteration, a subset of observations is

split into two new subsets based on the values
of a single variable.

• Series of questions that successively narrow

down observations into smaller and smaller
groups of decreasing impurity.
Supervised Learning example techniques

Classification Classification Tree

Weather Conditions for Playing Tennis
Logistic Regression
Attempts to classify a categorical outcome (y = 0
or 1) as a linear function of explanatory
variables.
UNSUPERVISED LEARNING
• used against data that has no historical labels
• the goal is to explore the data and find some
structure within
• There is no right or wrong answer
Self-organizing maps
Unupervised Learning example techniques
Nearest-neighbor mapping
k-nearest neighbors (k-NN): This method can be
used either to classify an outcome category or
predict a continuous outcome.
• k-NN uses the k most similar observations
from the training set, where similarity is
typically measured with Euclidean distance.
• When k-NN is used as a classification method,
a new observation is classified as Class 1 if the
percentage of it k nearest neighbors in Class 1 is
greater than or equal to a specified cut-off value
(e.g. 0.5).
• When k-NN is used as a prediction method, a
new observation’s outcome value is predicted
to be the average of the outcome values of its
k-nearest neighbors.
K-Nearest neighbor
To classify an outcome, the training set is
searched for the one that is “most like” it. This
is an example of “instancebased” learning. It is
“rote learning”, the simplest form of learning.
Clustering
A definition of clustering could be “the process
of organising objects into groups whose
members are similar in some way”.
Software Demo (Weka)
What is Weka?
Weka is tried and tested open source machine
learning software that can be accessed through
a graphical user interface, standard terminal
applications, or a Java API.
Logistic Regression in Weka
• Open diabetes dataset.
• Click Classifier Tab
• Choose Classifier: functions>Logistic
• Use Test Options: Use Training Set
• Press Start • Right-click result from Result List for options

• Choose Visualize tree

Trees in Weka

• Open weather dataset.

Ordinal data can also be called as ordered
factor.
- False

In this phase you’ll also develop and test

hypotheses through rapid prototyping in an
iterative process.
- Explore

It is a comprehensive data mining

methodology and process model that
provides anyone – from novice to data
mining experts – with a complete blueprint
• Click Classify Tab for conducting a data mining project.

• Choose Classifier: trees>J48 - CRISP-DM

• Use Test Options: Use Training Set CRISP-DM stands for

____________________________.
•Press Start
- Cross-industry standard process for
data mining

It is a role that is responsible for collecting,

analyzing, and interpreting large amount of
data.
- Data Scientist

It is the aspect of business analytics that

finds patterns in unstructured data like social
media or survey tools which could uncover
insights about consumer sentiment
 - Data Mining
One of the benefits of data mining is
overfitting.
- False
It is the phase of CRISP-DM where analysts
review the steps executed.
- Evaluation
A key objective is to determine if there is
some important business issue that has not
been sufficiently considered.
- True
Identify the third step in CRISP-DM.
- Data Preparation
In this approach in data mining, input
variables and output variables will be given.
- Supervised Learning
Self-organizing maps are example of
supervised learning.
- True
Logistic regression is classified as supervised
learning.
- True
Association rule algorithm is an example of
what approach in data mining?
- Unsupervised Learning
Weka is tried and tested open source
machine learning software that can be
accessed through a graphical user interface,
standard terminal applications, or a Java API.
- True
Unsupervised machine learning finds all kind
of unknown patterns in data
- True
All data is unlabeled and the algorithms learn
to inherent structure from the input data.
This statement pertains to ___________.
- Supervised Learning
Random forest is an example of supervised
learning.
- True
Unsupervised methods help you to find
features which can be useful for
categorization.
- True
All data is labeled and the algorithms learn
to predict the output from the input data.
This statement pertains to ___________. (Use
lowercase for your answer)
- Unsupervised Learning
17/20
This role provides the funding when doing
analytical project.
- Project Sponsor
It is a role that is responsible for collecting,
analyzing, and interpreting large amount of
data.
- Data Scientist
Based on the analytical life cycle defined by
SAS, this phase has two types of decisions:
operational and strategic.
- Act
Interpreting mined patterns concludes the
KDD process.
- True
It is the aspect of business analytics that
finds patterns in unstructured data like social
media or survey tools which could uncover
insights about consumer sentiment.
- Text analytics
This is the first step in KDD process.
- Data Selection
In this phase, you’ll search for relationships,
trends and patterns to gain a deeper
understanding of your data.
- Explore
Which of the following is not included on Which of the following is not considered as
the analytical life cycle defined by SAS? data mining technique?
- Integration - Kurtosis

These are values that lie away from the bulk
of the data.
- Outliers
CRISP-DM stands for
____________________________.
- Cross-industry standard process for
data mining
All data is labeled and the algorithms learn
to predict the output from the input data.
This statement pertains to ___________. (Use
lowercase for your answer)
- supervised
Supposed your email program watches
which emails you do or do not mark as spam,
and based on that learns how to better filter
spam. What is the performance measure P in
this setting?
- The number of email correctly
classified as spam/not spam
Self-organizing maps are example of
supervised learning.
- False

It makes the computation of multi-layer

neural network feasible.
- Deep Learning

Unsupervised methods help you to find

features which can be useful for
categorization.
- False

Logistic regression is classified as supervised

learning.
- True

Supposed you want to train a machine to

help you predict how long it will take you to
drive home from your workplace. What type
of data mining approach should you use?
- Supervised learning
One of the challenges of data mining is It
can be implemented in new systems as well
as existing platforms.
- False
One of the advantages of data mining is that
there are chances of companies may sell
useful information of their customers to
other companies for money.
- False
MODULE 3 INTRODUCTION TO REGRESSION
ANALYSIS
What is REGRESSION ANALYSIS?
A technique of studying the dependence of one
variable (called dependent variable), on one or
more variables (called explanatory variable),
with a view to estimate or predict the average
value of the dependent variables in terms of the
known or fixed values of the independent
variables.
 • where a and b are parameters in the
regression model.

Parameters

• Dependent variable or response: Variable

being predicted

• Independent variables or predictor variables:

Variables being used to predict the value of the
When do you use regression?
dependent variable.

• Simple regression: A regression analysis

Reason to use regression? involving one independent variable and one
dependent variable.
• Estimate the relationship that exists, on the
average, between the dependent variable and • In statistical notation:
the explanatory variable
y = dependent variable
• Determine the effect of each of the
x = independent variable
explanatory variables on the dependent
variable, controlling the effects of all other Types of Regression Analysis
explanatory variables
- Linear Regression
• Predict the value of dependent variable for a - Multiple Linear
given value of the explanatory variable - Nonlinear Regression
Understanding regression model based on the Who uses regression?
concept of slope
• Data analysts
• The mathematical of slope is similar to
regression model. • Market researchers

Y=mx+b • Professors

• And when using the slope intercept formula, • Data scientists

we focus on the two constants (numbers) m and
b.
• m describes the slope or steepness of the line,
whereas
• b represents the y-intercept or the point
where the graph crosses the y-axis.
Advantages of using regression?
• It indicates the strength of impact of multiple
independent variables on a dependent variable
• It indicates the significant relationships
between dependent variable and independent
variable.

Regression Model Regression Pitfalls

• The situation using the regression model is Overfitting

analogous to that of the interviewers, except
instead of using interviewers, predictions are
made by performing a linear transformation of
the predictor variable.
• The prediction takes the form
It pertains to the accuracy of the provisional
model is not as high on the test set as it is on
the training set, often because the provisional
model is overfitting on the training set

y = ax +b

Excluding Important Predictor Variables
The linear association between two variables
ignoring other relevant variables can differ both
in magnitude and direction from the association
that controls for other relevant variables.
Extrapolation
It refers to estimates and predictions of the
target variable made using the regression
equation with values of the predictor variable
outside of the range of the values of x in the
data set
Missing Values
Missing data has the potential to adversely
affect a regression analysis by reducing the total
usable sample size.
Power and sample size
In small datasets, a lack of observations can
lead to poorly estimated models with large
standard errors.
SUBTOPIC 1 OVERVIEW OF REGRESSION
Linear Regression
It is a model that tests the relationship between
a dependent variable and a single independent
variable. It can also be described using the
following expression:
𝑦 = 𝑎 + 𝑏𝑋 + 𝜖
It is a model that tests the relationship between
a dependent variable and a single independent
variable. It can also be described using the
following expression:
Where

• y – dependent variable

• X – independent (explanatory) variable

• a – intercept

• b – slope

• 𝜖 – residual (error)

Linear model assumptions

1. The dependent and independent variables Step 1

show a linear relationship between the slope
and the intercept.
2. The independent variable is not random.
3. The value of the residual (error) is zero.
4. The value of the residual (error) is constant
across all observations.
Identify the dependent and independent
variables.
For this example, we are trying to predict the
house price. Therefore, it is the independent
variable.

5. The value of the residual (error) is not

correlated across all observations.

6. The residual (error) values follow the normal

distribution.

Example problem:

If you want to know the strength of relationship

between House Price and Square feet, you can
use regression.

Identify the dependent and independent

variables.

Since we want to predict house price based on

square feet, it is now our dependent variable.
House Price Data
 Substitute the values of the coefficients to the
formula mentioned previously.

Value of X variable 1 from the results provided

in Step 3.

Step 5

Insert the value for sq. ft. to predict for the

house price.

𝒉𝒐𝒖𝒔𝒆 𝒑𝒓𝒊𝒄𝒆 = 𝟗𝟖. 𝟐𝟓 + 𝟎. 𝟏𝟎𝟗𝟖(𝒔𝒒. 𝒇𝒕. )

Step 2

Run regression analysis on the data using any

system that offers statistical analysis. For this
example, we can use Microsoft Excel with the
help of Data Analysis Tool pack.
Multiple Linear

Multiple linear regression analysis is essentially

similar to the simple linear model, with the
exception that multiple independent variables
are used in the model. The mathematical
representation of multiple linear regression is:
Step 3
𝑦 = 𝑎 + 𝑏𝑋1 + 𝑐𝑋2 + 𝑑𝑋3 + 𝜖
Analyze the results. Take note of the values for
coefficients. Interpretation

• Interpretation of slope coefficient βj :

Represents the change in the mean value of the
dependent variable y that corresponds to a one
unit increase in the independent variable xj ,
holding the values of all other independent
variables in the model constant.

• The multiple regression equation that

describes how the mean

Step 4

Substitute the values of the coefficients to the Multiple Linear Regression Model
formula mentioned previously.
• Estimated multiple regression equation:
𝒉𝒐𝒖𝒔𝒆 𝒑𝒓𝒊𝒄𝒆 = 𝟗𝟖. 𝟐𝟓 + 𝟎. 𝟏𝟎𝟗𝟖(𝒔𝒒. 𝒇𝒕. )

Substitute the values of the coefficients to the

formula mentioned previously.
• The least squares method is used to develop
the estimated multiple regression equation:

value of the Intercept from the results in step 3.

The Estimation Process for Multiple
Regression

Nonlinear Regression

Nonlinear regression is a regression in which

the dependent or criterion variables are
modeled as a non-linear function of model
parameters and one or more independent
variables.

Nonlinear regression can be model in several

equations similar to the one below:

Regression Analysis

Regression is a technique used for forecasting,

time series modeling and finding the casual
effect between the variables.

Why use regression?

1) Prediction of a target variable (forecasting).

2) Modeling the relationships between the

dependent variable and the explanatory
variable.

3) Testing hypothesis.
SUBTOPIC 2 LINEAR REGRESSION

- The whole process of linear regression • = Point estimator of E(y|x)

is based on the fact that there exists a • b0 = Estimated y-intercept
relation between the independent
variables and dependent variable. • b1 = Estimated slope

Simple Linear Regression The graph of the estimated simple linear

regression equation is called the estimated
The whole process of linear regression is based regression line.
on the fact that there exists a relation between
the independent variables and dependent
variable. Possible Linear Regression Graphs
Simple Linear Regression Model

y = β0 + β1 x + ε

Parameters: The characteristics of the

population, β0 and β1

Random variable - Error term, ε

- The error term accounts for the The Estimation Process in Simple Linear
variability in y that cannot be explained Regression
by the linear relationship between x
and y.

Simple Linear Regression

Regression equation: The equation that

describes how the expected value of y, denoted
E(y), is related to x.

Regression equation for simple linear

regression:

E(y|x) = β0 + β1x

• E(y|x) = expected value of y for a given value

of x

• β0 = y-intercept of the regression line

• β1 = slope

The graph of the simple linear regression

equation is a straight line
Simple Linear Regression Model
Least Square Method
Least squares method: A procedure for using
sample data to find the estimated regression
equation.

Estimated simple linear regression equation: • Here, we will determine the values of b0 and
b1 .

• Interpretation of b0 and b1 :
• The slope b1 is the estimated change in the
mean of the dependent variable y that is
associated with a one unit increase in the
independent variable x.
• The y-intercept b0 is the estimated value of the
dependent variable y when the independent
variable x is equal to 0.
Miles Traveled and Travel Time (in Hours) for
Ten Butler Trucking Company Driving
Assignments
Scatter Chart of Miles Traveled and Travel
Time in Hours for Sample of Ten Butler
Trucking Company Driving Assignments
Least Squares Method
Least squares method equation:
• y𝑖 = observed value of the dependent
variable for the ith observation
• = predicted value of the dependent
variable for the ith observation
• n = total number of observations
ith residual: The error made using the
regression model to estimate the mean value
of the dependent variable for the ith
observation.
• Denoted as
• Hence,
• Least squares estimates of the regression
parameters:
Slope equation
y-intercept equation
• = value of the independent variable for
the ith observation
• = value of the dependent variable for the
ith observation
• = mean value for the independent
variable
• = mean value for the dependent variable
• n = total number of observations
For the Butler Trucking Company data in
Table 4.1:
• Estimated slope of b1 = 0.0678
• y-intercept of b0 = 1.2739
• The estimated simple linear regression
model:
 • Experimental region: The range of values of
the independent variables in the data used to
estimate the model.

• The regression model is valid only over this

region.

• Extrapolation: Prediction of the value of the

dependent variable outside the experimental
region.

• It is risky Multiple Linear

Predicted Travel Time in Hours and the • It is a statistical technique that uses several
Residuals for Ten Butler Trucking Company independent variables to predict the dependent
Driving Assignments variable.

• It is the extension of linear regression.

Regression Question 1

Could the explanatory variables such as number

of cigarettes, cholesterol level, and exercise
predict the Coronary Heart Disease (CHD)
mortality?

Identifying variables
Scatter Chart of Miles Traveled and Travel
Time in Hours for Butler Trucking Company
Driving Assignments with Regression Line
Superimposed

Research Question 2

Does the number of years of psychological

study and number of years of counseling
experience predict clinical psychologists’
Linear Regression in Excel
effectiveness in treating mental illness?

Identifying Variables
Difference Correlation Results

Linear Regression only deals with one

independent variable.

Multiple linear Regression with more than one

independent variable.

Value of R square

Case Study

You want to know the effect of violence, stress,

social support on internalizing behavior.

Details about the study

• Participants were children aging from 8 to 12 Test for overall significance

• Lived in high-violence areas, USA
• Shows if there is a linear relationship between
• Hypothesis: violence and stress lead to all of the X variables taken together and Y
internalizing behavior, whereas social support
• Hypothesis:
would reduce internalizing behavior.

Parameters

• Predictors Results of Significance

• Degree of witnessing violence

• Measure of life stress

• Measure of social support

• Outcome

• Internalizing behavior (i.e. depression,

anxiety)
Substitute the values in the linear equation

In order to test the hypothesis, you can use the

following formula:

Interpretations

• Slopes for Witness and Stress are positive, but

slope for Social Support is negative

• If you had subjects with identical stress and

social support, a one unit increase in Witness
would produce .038 unit increase in
internalizing symptoms

• If Witness = 20, Stress = 5 and SocSupport 35,

then we would predict that the internalizing
symptoms would be .012.

SUBTOPIC 3 LOGISTIC REGRESSION
Predicting Categorical Outcome
Logistic Regression
It is a statistical technique used to develop
predictive models with categorical dependent
variables having dichotomous or binary
outcomes.
Similar to linear regression, the logistic
regression models the relationship between the
dependent variable and one or more
independent variables.
Graph of Linear Regression
Graph of Logistic Regression
Logistic Regression Assumptions
• Logistic regression measures the relationship
between the categorical dependent variable
and one or more independent variables by
estimating probabilities using a logistic function,
which is the cumulative logistic distribution.
• There are other variants of logistic regression
that focus on modeling a categorical variable
with three or more levels, say X, Y, and Z and a
few others.
Concept of Probability
Unlike linear regression, logistic regression
model is concern for the log of odds ratio or the
probability of the event to happen. Everything
starts with the concept of probability.
Example Scenario
Let's say that the probability of success of some
event is 0.8. Then the probability of failure is 1 –
0.8 = 0.2. The odds of success are defined as the
ratio of the probability of success over the
probability of failure.
Logistic Regression Equation
In our example, the odds of success are 0.8/0.2
= 4. That is to say that the odds of success are 4
to 1. If the probability of success is 0.5, that is, a
50-50 percent chance, then the odds of success
are 1 to 1. In our example, the odds of success
are 0.8/0.2 = 4. That is to say that the odds of
success are 4 to 1. If the probability of success is
0.5, that is, a 50-50 percent chance, then the
odds of success are 1 to 1.
To predict the probability of the event to
happen, we can further solve the preceding
equation as follows:
Maximum-likelihood estimation
It is a method of estimating the parameters of a
statistical model with given data. The method of
maximum likelihood selects the set of values of
the model parameters that maximizes the
likelihood function, that is, it maximizes the
“agreement” of the selected model with the
observed data.
Building Logistic Regression Model
You can perform logistic regression using R by
using the glm() function. The family =
"binomial" command tells R to use the glm
function to fit a logistic regression model. (The
glm() function can fit other models too; we'll
look into this later.)
Coefficients results
Interpreting Results
• A positive estimate indicates that, for every
unit increase of the respective independent
variable, there is a corresponding increase in
the log of odds ratio and the other way for a
negative estimate.
• A long with the independent variables, we
also see 'Intercept'. Intercept is the log of odds
of the event (Good or Bad Quality) when we
have all the categorical predictors having a
value as 0.
• We can see the standard error, z value, and p-
value along with an asterisk indication to easily
identify significance.
• We then determine whether the estimate is
truly far away from 0. If the standard error of
the estimate is small, then relatively small
values of the estimate can reject the null
hypothesis.
• If the standard error is large, then the
estimate should also be large enough to reject
the null hypothesis.
Testing the significance
• To test the significance, we use the 'Wald Z
Statistic' to measure how many standard
deviations the estimate is away from 0.
• The significance of the estimate can be
determined if the probability of the event
happening by chance is less than 5%.
How do we assess the goodness of fit or
accuracy of the model?
Two ways to validate model accuracy
What is confusion matrix?
A confusion matrix is a table used to analyze the
performance of a model (classification). Each
column of the matrix represents the instances
in a predicted class while each row represents
the instances in an actual class or vice versa.
What is ROC Curve?
The Receiver Operating Characteristic (ROC)
curve is a standard technique to summarize
classification model performance over a range
of trade-offs between true positive (TP) and
false positive (FP) error rates. The ROC curve is
a plot of sensitivity (the ability of the model to
predict an event correctly) versus 1-specificity
for the possible cutoff classification probability
values.
Validate model using confusion matrix
Let’s assume that we acquire the following
results after running logistic regression to our
sample data set:
Columns in confusion matrix
• True Positive (TP): When it is predicted as
TRUE and is actually TRUE
• False Positive (FP): When it is predicted as
TRUE and is actually FALSE
• True Negative (TN): When it is predicted as
FALSE and is actually FALSE
• False Negative (FN): When it is predicted as
TRUE and is actually FALSE
Substitute the values
The preceding confusion matrix is overlaid with
the nomenclature and is showcased here:
Confusion Matrix Formula
An exhaustive list of metrics that are usually
computed from the confusion matrix to aid in
interpreting the goodness of fit for the
classification model are as follows:
Interpreting ROC Curve
Interpreting the ROC curve is again
straightforward. The ROC curve visually helps us
understand how our model compares with a
random prediction. The random prediction will
always have a 50% chance of predicting
correctly; after comparing with this model, we
can understand how much better is our model.
The diagonal line indicates the accuracy of
random predictions and the lift from the
diagonal line towards the left upper corner
indicates how much improvement our model
has in comparison to the random predictions.
Models having a higher lift from the diagonals
are considered to be more accurate models.
FORMATIVE 25/30
It is a model that tests the relationship
between a dependent variable and a single
independent variable.
- Linear regression
Supposed you want to train a machine to
help you predict how long it will take you to
drive home from your workplace using
regression. What type of data mining
approach should you use?
- Supervised Learning
In small datasets, a lack of observations can
lead to poorly estimated models with large
standard errors.
- True
Multiple linear regression is classified as
supervised learning.
- True
It is computed as the ratio of the two odds.
- Odds ratio
Multilinear regression is a regression in
which the dependent or criterion variables
are modeled as a non-linear function of
model parameters and one or more
independent variables.
- False
Regression is a technique used for
forecasting, time series modeling and finding
the casual effect between the variables.
- False
It is used to compare two nested generalized
linear models.
- Likelihood ratio test
The lower is the AUC value, the worse is the
model predictive accuracy.
- True
Which of the following is not a reason when - Explanatory variable
to use regression?
- When you aim to know the products
The formula below describes
that are usually bought together.

It is a regression in which the dependent or

criterion variables are modeled as a non-
linear function of model parameters and one
- Multiple linear regression
or more independent variables.
Regression is a technique used for
- Nonlinear Regression
forecasting, time series modeling, and
It is essentially similar to the simple linear finding the casual effect between the
model, with the exception that multiple variables.
independent variables are used in the model.
- False
- Multiple Regression
It is the tool pack in Microsoft Excel that can
There should not be any multicollinearity be downloaded to perform linear regression.
between the independent variables in the
- Data Analysis Tool Pack
model, and all independent variables should
be independent to each other A group of middle school students wants to
know if they can use height to predict age.
- True
The explanatory variable is height.
If a predictor variable x is found to be highly
- True
significant we would conclude that:
Multiple linear regression can be expressed
- a change in x causes a change in y
(changes in x are associated to changes
in y) as 
Overfitted data can significantly lose the - False
predictive ability due to an erratic response
to noise whereas underfitted will lack the The _____________ are defined as the ratio of
accuracy to account for the variability in the probability of success over the
response in its entirety. probability of failure. (Use lowercase for
your answer).
- True
- odds ratio
Given the results for multiple linear
regression below. Identify the estimated Multiple linear regression is classified as
regression equation. unsupervised learning.

- False

The formula above is used to model

nonlinear regression.

- exam score = 5.56*hours + - False

prep_exams*-0.60 + 67.67
It is a model that tests the relationship
It is the variable being manipulated by between a dependent variable and a single
researchers. independent variable.
 - Linear Regression
When it is predicted as TRUE and is actually
TRUE.
- True Positive
If a predictor variable x is found to be highly
significant we would conclude that: a change
in y causes a change in x.
- False (changes in x are associated to
changes in y)
When it is predicted as FALSE and is actually
FALSE.
- False Negative
Response variable is the variable being
manipulated by researchers.
- False
FORMATIVE 25/30
A linear regression analysis was
conducted to predict the company sales.
Below are the results. Identify the linear
regression equation assuming that x is
the parameter for advertising.
- sales = 23.42x + intercept
The higher is the AUC value, the better is
the model predictive accuracy.
-True
The explanatory variable is the variable
being predicted.
-False
Logistic regression is classified as
supervised learning.
-True
If a predictor variable x is found to be highly
significant we would conclude that:
-changes in x are associated to changes in
y
It is the variable being predicted.
-Dependent variable
In logistic regression, it is that the target
variable must be discrete and mostly binary
or dichotomous.
-True
It is the tool pack in Microsoft Excel that can
be downloaded to perform linear regression.
-Data Analysis Tool Pak
It is essentially similar to the simple linear
model, with the exception that multiple
independent variables are used in the
model.
-Multiple regression
It pertains to the accuracy of the provisional
model is not as high on the test set as it is
on the training set. (Use UPPERCASE for
your answer)
-OVERFITTING
Extrapolation of the range of values of the
independent variables in the data used to
estimate the model.
-True
In logistic regression, it is that the target
variable must be discrete and mostly binary
or dichotomous.
-True
It is the tool pack in Microsoft Excel that can
be downloaded to perform linear regression.
-Data Analysis Tool Pack
Multiple linear regression is classified as
unsupervised learning.
-False
A researcher believes that the origin of the
beans used to make a cup of coffee affects
hyperactivity. He wants to compare coffee
from three different regions: Africa, South
America, and Mexico. What is the response
variable on this study?
-Hyperactivity level
The graph of the simple linear regression
equation is a straight line
-True
Given the results for multiple linear
regression below. Identify the estimated
regression equation.
-True Negative
Logistic regression measures the
relationship between the ____________
dependent variable and one or more
independent variables. (Use lowercase
for your answer)
categorical
-categorical
A group of middle school students wants to
know if they can use height to predict age.
The explanatory variable is height.
-True
Which of the following evaluation metrics
can’t be applied in the case of logistic
regression output to compare with the
target?
-Mean-Squared-Error

-exam score = 5.56*hours + prep_exams*-

0.60 + 67.67

There should not be any multicollinearity

between the independent variables in the
model, and all independent variables should
be independent to each other
-True
Logistic regression is classified as one of
the examples of unsupervised learning.
-False
It is essentially similar to the simple linear
model, with the exception that multiple
independent variables are used in the
model.
-Multiple Regression
It is essentially similar to the simple linear
model, with the exception that multiple
independent variables are used in the
model.
-Multiple linear regression
Which of the following methods do we use
to best fit the data in Logistic Regression?
-Maximum Likelihood
What type of relationship is shown in the
graph below?

The graph of the simple linear regression

equation is a straight line.
-True
Multiple linear regression can be expressed
as y=a+bX+ ϵ

-False
When it is predicted as FALSE and is
actually FALSE.
 2. Overfitted data can significantly lose
the predictive ability due to an erratic
response to noise whereas
underfitted will lack the accuracy to
account for the variability in
response in its entirety.
 True

3. A group of middle school students

wants to know if they can use height
to predict age. What is the response
variable in this study
 Height
 The type of school
 Age
(https://online.stat.psu.edu/stat200/le
sson/1/1.1/1.1.2#:~:text=A%20group
%20of%20middle%20school,use
%20height%20to%20predict
%20age.&text=The%20students
%20want%20to%20use,the
%20response%20variable%20is
%20age.)
-Undefined
 The number of students

A regression line is used for all of the 4. Multilinear regression is a regression

following except one. Which one is not a in which the dependent or criterion
valid use of a regression line? variables are modeled as a non-
linear function of model parameters
-to determine if a change in X causes a and one or more independent
change in Y. variables.
 False

5. It is the variable being manipulated

by researchers.
 Explanatory Variable
 Factor
 Response Variable
 Dependent Variable

6. Multiple linear regression is

24/30 classified as supervised learning.
 True
1. Regression is a famous supervised
learning technique.
 True
7. Research question: Do fourth
graders tend to be taller than the
third graders?
This is an observational study. The
researcher wants to use grade level to
explain differences in height. What is the
response variable in this study?
 Gender
 Age
 Grade level
 Height
8. If p value for model fit is less than
0.5, then signify that our full
model fits significantly better than
our reduced model.
 False
9. It is used to compare two nested
generalized linear models.
 Odds ratio
 Hosmer-lemeshow test
 Likelihood ratio test
 ROC Curve
10. In linear models, the dependent
and independent variables show
a linear relationship between the
slope and the intercept
 True
11. Supposed you want to train a
machine to help you predict how
long it will take you to drive home
from your workplace. What type
of data mining approach should
you use
 Unsupervised Learning
 Supervised learning
12. A researcher believes that the
origin of the beans used to make
a cup of coffee affects
hyperactivity. He wants to
compare coffee from three
different regions: Africa, South
y=a+bX_1+cX_2+ 〖dX〗_3+ ϵ
America, and Mexico. What is the
explanatory variable in this
study?
 The taste of the coffee
 The cups of coffee taken
 Origin of the coffee
 Hyperactivity level
13. 1The value of the residual (error) is
not correlated across all
observations.
 True
14. There should not be any
multicollinearity between the
independent variables in the model,
and all independent variables should
be independent to each other
 False
15. It is the variable being predicted.
 Explanatory variable
 Dependent variable
 Independent variable
 Factor
16. The graph of the simple linear
regression equation is a straight line
 True
17. It is the tool pack in Microsoft Excel
that can be downloaded to perform
linear regression.
 Data Analysis Tool Pack
 Test Mining Tool Pack
 Regression Tool Pack
 Data Mining Tool Pack
18. It is a point that lies far away from
the rest.
 Outliers
 Dummy variables
 Residual
19. The formula below describes
  Linear Regression  Nonlinear regression
 Multiple Linear Regression
 Nonlinear Regression
26. Logistic regression is classified as
supervised learning.
20. Logistic regression is classified as
 True
one of the examples of unsupervised
learning.
 False
27. In the equation

21. It is a table used to analyze the

y=a+bX+ ϵy=a+bX+ ϵ
performance of a model A and b are considered as independent
(classification). (Use lowercase) variables.
 confusion matrix
 False

22. The family = ______________

command tells R to use the glm 28. Shown below is a scatterplot of Y
function to fit a logistic regression versus X. Which of the following is
model. (Use lowercase for your the approximate value for R2?
answer to supply the code)  99.5%
 logit regression  50%
23. It is a model that tests the  25%
relationship between a dependent  0%
variable and a single independent
variable.
 Linear Regression
 Least squared method
 Multiple regression
 Nonlinear regression

24. y=a+bX_1+cX_2+ 〖dX〗_
3+ ϵy=a+bX_1+cX_2+ 〖dX〗_3+
ϵ
The formula above is used to model
nonlinear regression.
 False

29. Nonlinear is the extension of linear

25. It is a regression in which the regression
dependent or criterion variables  False
are modeled as a non-linear
30. A group of middle school students
function of model parameters and
wants to know if they can use height to
one or more independent
predict age. The response variable in
variables.
this study is age.
 Linear Regression  True
 Least squared method
 Multiple regression
ADDITIONAL FORMATIVE independent variables are used in the model.
Multiple Regression
It is essentially similar to the simple
The value of the residual (error) is not
linear model, with the exception that
correlated across all observations. True
multiple independent variables are used
in the model. Multiple regression Given the results for multiple linear
regression below. Predict the exam score if a
Which of the following is not a reason student spent hours studying and had 2 prep
when to use regression? When you aim exams taken..
to know the products that are usually
bought together.
It pertains to the accuracy of the
105.39
provisional model is not as high on the
test set as it is on the training set. (Use Logistic regression is classified as supervised
UPPERCASE for your answer) learning. True
OVERFITTING In logistic regression, it is that the target
variable must be discrete and mostly binary
It is used to explain the variability in the
or dichotomous. True
response variable. response variable
It can be utilized to assess the strength of
It is used to measure the binary or the relationship between variables and for
dichotomous classifier performance modeling the future relationship between
visually and Area Under Curve (AUC) is them Regression Analysis
used to quantify the model performance. If p value for model fit is less than 0.5, then
ROC Curve signify that our full model fits significantly
better than our reduced model. False
Missing data has the potential to
adversely affect a regression analysis by Factor is the variable being predicted. False
reducing the total usable sample size. It is a model that tests the relationship
True between a dependent variable and a
single independent variable. Linear
In regression, the value of the residual Regression
(error) is one. True False
What type of relationship is shown in the
The value of the residual (error) is not graph below? Negative Linear Relationship
correlated across all observations. True
It is the variable of primary interest.
response variable
If p value for model fit is less than 0.5, then
signify that our full model fits significantly
better than our reduced model. False
It is used to explain the variability in the
response variable parameter
The value of the residual (error) is one false

The value of the residual (error) is correlated

across all observations. False It is essentially similar to the simple linear
model, with the exception that multiple
It is essentially similar to the simple linear independent variables are used in the model.
model, with the exception that multiple Multiple linear regression
Logistic regression is used to predict
continuous target variable. True
Response variable is the variable being
manipulated by researcher. False
Logistic regression is classified as
supervised learning. True
A researcher believes that the origin of
the beans used to make a cup of coffee
affects hyperactivity. He wants to
compare coffee from three different
regions: Africa, South America, and
Mexico. What is the explanatory variable
in this study? Origin of the coffee
The lower is the AUC value, the worse is the
model predictive accuracy. True
Regression is a famous supervised learning
technique. True
In regression, the value of the residual
(error) is one. False
How will you express the equation of a
regression analysis where you aim to predict
the value of y based on x. (Use lowercase
and avoid so much space) y=b0+b1*x1
Regression is a technique used for
forecasting, time series modeling, and
finding the causal effect between the
variables. False
Supposed you want to train a machine to
help you predict how long it will take you to
drive home from your workplace using
regression. What type of data mining
approach should you use? Supervised
Learning
Multilinear regression is a regression in
which the dependent or criterion variables
are modeled as a non-linear function of
model parameters and one or more
independent variables. True
Multilinear regression is a regression in
which the dependent or criterion variables
are modeled as a non-linear function of
model parameters and one or more
independent variables. False
If a predictor variable x is found to be highly
significant we would conclude that: changes
in x are associated to changes in y
It is used to develop the estimated multiple
regression. Multiple Regression
The graph for a nonlinear relationship is
often a straight line false
It is essentially similar to the simple linear
model, with the exception that multiple
independent variables are used in the model.
Multiple Regression.
 In linear regression, it is that the target
variable must be discrete and mostly binary
or dichotomous. True
What type of relationship is shown in the
graph above? No relationship
The value of the residual (error) is not
correlated across all observations. True
Extrapolation of the range of values of the
independent variables in the data used to
estimate the model. True
A group of middle school students wants to
know if they can use height to predict age.
What is the explanatory variable in this
study? height
Supposed you want to train a machine to
help you predict how long it will take you to
drive home from your workplace. What type
of data mining approach should you use?
Supervised Learning
Multiple linear regression is classified as
unsupervised learning. False
It is a point that lies far away from the rest.
Outliers
When it is predicted as TRUE and is actually
TRUE. True Positive
Two variables are correlated if there is a
linear association between them. If not, the
variables are uncorrelated. True
Multiple linear regression is classified as
unsupervised learning. False
What type of relationship is shown in the
graph below? Negative Linear Relationship

Logistic regression is used to predict
continuous target variable. True
You predicted negative and it’s false. False
Negative
In ROC Curve, models having a higher lift
from the diagonals are considered to be
more accurate models. True
y=a+bX_1+cX_2+ 〖dX〗_3+ ϵy=
a+bX_1+cX_2+ 〖dX〗_3+ ϵ
The formula above is used to model
nonlinear regression. False
Logistic regression is classified as
supervised learning. True
It is essentially similar to the simple
linear model, with the exception that
multiple independent variables are used
in the model. Multiple regression
The lower is the AUC value, the worse is
the model predictive accuracy. True
Multiple linear regression is classified as
supervised learning. True
It is the variable of primary interest.
Parameter
A group of middle school students wants to
know if they can use height to predict age.
What is the response variable in this study?
Age
If p value for model fit is less than 0.5, then
signify that our full model fits significantly
better than our reduced model. False
Supposed you want to train a machine to
help you predict how long it will take
you to drive home from your workplace
using regression. What type of data
mining approach should you use?
Supervised Learning
In the equation y=a+bX+ ϵ What is
considered as a dependent parameter/s?
y
the graph for a nonlinear relationship is
often a straight line. True
In linear models, the dependent and
independent variables show a linear
relationship between the slope and the
intercept true
The formula below describes
y=a+bX_1+cX_2+ 〖dX〗_3+ ϵy=
a+bX_1+cX_2+ 〖dX〗_3+ ϵ
Multiple linear regression
A group of middle school students wants
to know if they can use height to predict
age. What is the response variable in this
study? Age
It is a regression in which the dependent or
criterion variables are modeled as a non-
linear function of model parameters and one
or more independent variables. Nonlinear
Regression
Slope is the point that lies far away from
the rest. True
Nonlinear is the extension of linear
regression. False
In logistic regression, it is that the target
variable must be discrete and mostly binary
or dichotomous. True
Given the results for multiple linear
regression below. Predict the exam score
if a student spent
10 hours in studying and had 4 prep
exams taken.  
120.98
It can be utilized to assess the strength of
the relationship between variables and for
modeling the future relationship between
them. Regression Analysis
Logistic regression is classified as one of
the examples of unsupervised learning.
False
Research question: Do fourth graders
tend to be taller than third graders?
This is an observational study. The
researcher wants to use grade level to
explain differences in height. What is the
explanatory variable on this study? grade
level
The family = ______________ command tells
R to use the glm function to fit a logistic
regression model. (Use lowercase for your
answer to supply the code) glm()
What type of relationship is shown in the
graph below? No relationship
It is a regression in which the dependent or
criterion variables are modeled as a non-
linear function of model parameters and one
or more independent variables. Nonlinear
Regression
Which of the following evaluation metrics
can’t be applied in the case of logistic
regression output to compare with the
target? Mean-Squared-Error
Nonlinear is the extension of linear
regression. False
Which of the following methods do we use
to best fit the data in Logistic Regression?
Maximum Likelihood
When it is predicted as TRUE and is actually
TRUE. True Positive
Residual is a point that lies far away from
the rest true
You predicted negative and it’s false. False
Negative.
How do we assess the goodness of fit or
accuracy of the model in logistic regression?
(Use lowercase for your answer) roc curve
The graph for a nonlinear relationship is
often a straight line. True
How will you express the equation of a
regression analysis where you aim to predict
the value of y based on x. Y=a+bX
It is used to explain the variability in the
response variable parameter
Research question: Do fourth graders
tend to be taller than the third graders?
This is an observational study. The
researcher wants to use grade level to
explain differences in height. What is the
response variable in this study? Height
It can be utilized to assess the strength
of the relationship between variables
and for modeling the future relationship
between them. Regression Analysis
It is used to develop the estimated
multiple regression. Multiple Regression
It refers to estimates and predictions of
the target variable made using the
regression equation with values of the
predictor variable outside of the range of
the values of x in the data set. (Use
UPPERCASE for your answer)
PREDICTION
The formula below describes
y= (b_1+ b_2 x_1+b_3 x_2+b_4
x_3)/(1+ b_5 x_1+b_6 x_2+ b_7
x_3 )
Nonlinear regression
A researcher believes that the origin of the
beans used to make a cup of coffee affects
hyperactivity. He wants to compare coffee
from three different regions: Africa, South
America, and Mexico. What is the response
variable in this study? Hyperactivity level
The graph of the simple linear regression
equation is a straight line true
In the equation
option is true for such a case? odds will
be 1
It is essentially similar to the simple
linear model, with the exception that
multiple independent variables are used
in the model. Multiple linear regression
It is a regression in which the dependent
or criterion variables are modeled as a
non-linear function of model parameters
and one or more independent variables
Multiple Regression

y=a+bX+ ϵy=a+bX+ ϵ Shown below is a scatterplot of Y versus

X. Which of the following is the
What is considered as the parameter/s? approximate value for R2? 99.5%
a,b

In logistic regression, it is that the target

variable must be discrete and mostly
binary or dichotomous. True
It is used to explain the variability in the
response variable parameter
Logistic regression is classified as one of
the examples of unsupervised learning.
False
Regression is a technique used for
forecasting, time series modeling, and
finding the casual effect between the
variables. True
If a predictor variable x is found to be
There should not be any multicollinearity highly significant we would conclude
between the independent variables in that: a change in y causes a change in x.
the model, and all independent variables false
should be independent to each other
Regression is a famous supervised
true
learning technique. True
It is the range of values of the
exam score = 5.56*hours + prep_exams*-0.60
independent variables in the data used
+ 67.67
to estimate the model. Experimental region
It is a statistical technique that uses several
The value of the residual (error) is independent variables to predict the
correlated across all observations. False dependent variable. Multiple linear
Two variables are correlated if there is a regression
linear association between them. If not, It is the tool pack in Microsoft Excel that can
the variables are uncorrelated. True be downloaded to perform linear regression.
Data Analysis Tool Pack
Suppose you have been given a fair coin
and you want to find out the odds of Compute for the accuracy of the model
getting heads. Which of the following depicted in the confusion matrix below;
TP = 20 
TN = 27
FP = 18
FN = 25
52%
It is considered as the log of odds of the
event. (Use lowercase for your answer)
intercept
Supposed your model was able to predict
that a certain student fails the certification
exam and she actually is not. False Negative
Two variables are correlated if there is a
linear association between them. If not, the
variables are uncorrelated. True
MODULE 4 TIME SERIES AND FORECASTING
Time series analysis was introduced by Box and
Jenkins (1976) to model and analyze time series
data with autocorrelation.
Time series data consist of data observations
over time.
Application of Time Series
• Predicting stock prices
• Airline fares
• Labor force size
• Unemployment data
• Natural gas price
Jollibee: Forecasted Daily Closing Price
U.S Regular Gasoline Price Prediction
Key Ideas in Forecasting
• The objective of time series analysis is to
uncover a pattern in the time series and then
extrapolate the pattern into the future.

• The forecast is based solely on past values of

the variable and/or on past forecast errors.

• Modern data-collection technologies have

enabled individuals, businesses, and
Sample Scenario: Gasoline Sales Time Series
government agencies to collect vast amounts of
Plot after obtaining the Contract with the
data that may be used for causal forecasting.
Vermont State Police
• Time series: A sequence of observations on a
variable measured at successive points in time
or over successive periods of time.

• The measurements may be taken every

hour, day, week, month, year, or any other
regular interval. The pattern of the data is an
important to understand the series’ past
behavior.

• If the behavior of the times series data of

the past is expected to continue in the
Time Series Patterns
future, we can use it to guide us in selecting
an appropriate forecasting method. • Trend pattern: Gradual shifts or movements
to relatively higher or lower values over a
Sample Scenario: Gasoline Sales Time Series
longer period of time.

• A trend is usually the result of long-term

factors such as population increases or
decreases, shifting demographic
characteristics of the population, improving
technology, changes in the competitive
landscape, and/or changes in consumer
preferences.

Sample Scenario: Gasoline Sales Time Series Bicycle Sales Time Series
Plot

Bicycle Sales Time Series Plot

Sample Scenario: Gasoline Sales Time Series

after obtaining the Contract with the Vermont
State Police
Cholesterol Drug Revenue Time Series
Cholesterol Drug Revenue Time Series Plot
Time Series Patterns
• Seasonal pattern: Recurring patterns over
successive periods of time.
• Example: A manufacturer of swimming
pools expects low sales activity in the fall and
winter months, with peak sales in the spring
and summer months to occur every year.
• Time series plot not only exhibits a seasonal
pattern over a one-year period but also for
less than one year in duration.
• Example: daily traffic volume shows
within-the-day “seasonal” behavior
Umbrella Sales Time Series
Quality Smartphone Sales Time Series Plot
Forecasting Methodologies
 Moving Average
 Exponential Smoothing
SUBTOPIC 1 OVERVIEW OF TIME SERIES
ANALYSIS
• Time series data is a sequence of observations
collected from a process with equally spaced
periods of time.
• It establish relation between “cause” and
“effect”
• One variable is “Time” which is considered as
the independent variable and the second is
“Data” also known as the dependent variable.
Application of Time Series in Real-Life
Forecast Population
Germany Population from 2010-2019
Understanding the time series data
Time series data is expected to have two
variables: time and data
What are other examples of time series data?
1. Daily data on sales
2. Monthly inventory
3. Daily customers
4. Monthly interest rates, cost
5. Monthly unemployment rates
6. Weekly measures of money supply
7. Daily closing prices of stock indexes, and
soon.
Plot time series data
Benefits of Time Series Analysis
Through Time Series Analysis, businessmen can
predict about the changes in economy.
Furthermore, it could also help in determining
the following:
• Stock Market analysis
• Risk analysis and evaluation
• Census analysis
• Budgetary analysis
• Inventory studies
• Sales Forecasting
Components of Time Series
Secular Trend
The increase or decrease in the movements of a
time series is called secular trend. A time series
data may show upward trend or downward
trend for a period of years and this may be due
to factors like:
• Increase of population
• Change in technological progress
• Large scale shift in consumers demands

Seasonal variation

It is a short-term fluctuation in a time series

which occur periodically in a year.

Examples: Forecast Error

• More woolen clothes are sold in winter than
in the season of summer
• Each year more ice creams are sold in summer
and very little in Winter season
Measures to determine how well a particular
forecasting method is able to reproduce the
time series data that are already available

Cyclical Variations Forecast Error: Difference between the actual

and the forecasted values for period t.
These are recurrent upward or downward
movements in a time series but the period of
cycle is greater than a year.

Mean Forecast Error: Mean or average of the

forecast errors.

Irregular Variations Forecast Error Cont.

These are fluctuations in the time series that • Measures to determine how well a particular
are short in duration, erratic in nature and forecasting method is able to reproduce the
follow no regularity in the occurrence pattern. time series data that are already available.

Forecasting Methodologies Mean Absolute Error (MAE): Measure of

forecast accuracy that avoids the problem of
 Moving Average positive and negative forecast errors offsetting
 Exponential Smoothing one another.
Application available to perform TSA

 Tableau
 Excel
 SAS 9.4
Mean Squared Error (MSE): measure that avoids
 SAP Analytics
the problem of positive and negative errors
Forecast Accuracy offsetting each other is obtained by computing
the average of the squared forecast errors.
Computing Forecasts and Measures of Forecast
Accuracy using the most recent Value as the
Forecast for the next Period.
Computing Forecasts and Measures of Forecast Business Forecasting
Accuracy using the Average of all the Historical
Airline Passengers 1990-2004
Data as the Forecast for the next Period

Univariate Time Series Models
Univariate time series models are models used
when the dependent variable is a single time
series.
Multivariate Time Series Models
Multivariate time series models are used when
there are multiple dependent variables. In
addition to depending on their own past values,
each series may depend on past and present
values of the other series.
Modeling U.S. gross domestic product, inflation,
and unemployment together as endogenous
variables is an example of a multivariate time
series model.
Why use time series analysis?
Advantages
The airline data illustrate seasonality, because
there are seasonal patterns with respect to
when people fly. For example, more people fly
in August than in February. The data illustrate
trend, because the number of passengers is
increasing from year to year. The data illustrate
the effect of events, because the events of
September 11, 2001, had a significant negative
impact on airline travel.
Consider the above data restricted to years
1994 through 1997. This restricted period
allows you to focus on trend and seasonality
without having to worry about the effects of
external events. To validate data preparation
and to formulate forecast models for a time
series, a time plot of the data is produced.
Model Diagnostic Statistics

• Reliability – time series uses collected

historical data over a period of time.

• Seasonal patterns – since TSA deals with

periodic data, it is easy to predict seasonal
pattern.

• Estimation of trends – the graph of TSA makes

it easy to visualize the increase or decrease in
sales, production, etc.

• Growth – through depicting patterns, TSA
helps in measuring the financial growth.
Supplementary
When the forecast value is derived using a
linear regression model, the square of Pearson’s
correlation coefficient is equivalent to the
popular 2 R (R-Square) statistic. In practice,
Pearson’s correlation and average error are
seldom used as primary accuracy diagnostics.
These measures will be used in this work to
motivate the accuracy measures that are
commonly used.
R-Square is never appropriate for evaluating
time series models. Some sources redefine R-
Square so that it is more appropriate for time
series modeling. The above definition can
actually lead to negative values if the model is
nonlinear, which is a contradiction for a squared
quantity based on non-complex numbers. The
Time Series Forecasting System provides some
alternative R-Square measures.
Simulating a Retrospective Study
1. Divide the time series data into two
segments. The fit sample is used to derive a
forecast model. The holdout sample will be
used to evaluate forecast accuracy.
2. Derive a set of candidate models.
3. Calculate the chosen model accuracy statistic
by forecasting the holdout sample.
4. Pick the model with the best accuracy
statistic.
Before obtaining the data, as stated above, you
must select one of the accuracy statistics based
on the business problem. In marketing and
inventory management, MAPE is commonly
chosen. RMSE is commonly used in all areas of
business. R-Square is also very common, but as
you have seen, R-Square should never be used
to evaluate forecast models.
The simulated retrospective study simulates
what would have been experienced if the
forecast model had been implemented on the
date of the first series value in the holdout
sample.
The use of a simulated retrospective study to
pick a model is known as honest assessment.
Unfortunately, honest assessment is not
presented in many modern forecasting
textbooks. Because many textbook data sets are
small, honest assessment is often not possible.
In your work, you may discover that you are
rarely able to carry out a simulated
retrospective study because of data limitations.
Rules of Thumb
 At least four time points are required
for every parameter to be estimated in
a model.
 Anything above the minimum series
length can be used to create a holdout
sample.
 Holdout samples should rarely contain
over 25% of the series.
Summary of Data Used for Forecast Model
Building
Fit Sample
 Used to estimate model parameters for
accuracy evaluation
 Used to forecast values in holdout
sample
Holdout Sample
 Used to evaluate model accuracy
 Simulates retrospective study
Full = Fit + Holdout data is used to fit
deployment model
For model deployment or for generating
forecasts for decision making, the chosen model
is refit to the entire data set.
Three Environments for Forecasting Using SAS
Software*
 The SAS Windowing Environment
 The Time Series Forecasting System
 SAS Enterprise Guide
Additional Supplementary
A time series is simply a series of data points
ordered in time. In a time series, time is often
the independent variable and the goal is usually
to make a forecast for the future.
Autocorrelation
Informally, autocorrelation is the similarity
between observations as a function of the time
lag between them.
Seasonality
factor. Of course, this is useful if you notice
seasonality in your time series.
Seasonal autoregressive integraded moving
average model (SARIMA)

Seasonality refers to periodic fluctuations. For SARIMA is actually the combination of simpler

example, electricity consumption is high during models to make a complex model that can
the day and low during night, or online sales model time series exhibiting non-stationary
increase during Christmas before slowing down properties and seasonality.
again.

Stationarity

Stationarity is an important characteristic of

time series. A time series is said to be stationary
if its statistical properties do not change over
time. In other words, it has constant mean and
variance, and covariance is independent of time.

Modelling time series

There are many ways to model a time series in
order to make predictions. Here, I will present:
 moving average
 exponential smoothing
 ARIMA
Moving average
The moving average model is probably the most
naive approach to time series modelling. This
model simply states that the next observation is
the mean of all past observations.
Exponential smoothing
Exponential smoothing uses a similar logic to
moving average, but this time, a
different decreasing weight  is assigned to each
observations. In other words, less importance  is
given to observations as we move further from
the present.
Double exponential smoothing
SUBTOPIC 2 MOVING AVERAGE
Concept of averaging methods
• If a time series is generated by a constant
process subject to random error, then mean is a
useful statistic and can be used as a forecast for
the next period.
• Averaging methods are suitable for stationary
time series data where the series is in
equilibrium around a constant value ( the
underlying mean) with a constant variance over
time.
What is moving average?
• It is a technique that calculates the overall
trend in sales volume from historical data of the
company.
• This technique is well-known when
forecasting short-term trends.

Double exponential smoothing is used when

there is a trend in the time series. In that case,
we use this technique, which is simply a Sample scenario
recursive use of exponential smoothing twice.

Tripe exponential smoothing

This method extends double exponential

smoothing, by adding a seasonal smoothing

Applying Moving average
The mean sales for the first five years (2003-
2007) is calculated by finding the mean from
the first five years.
• Compute for the second subset of 5 years
(2004-2008) 𝑎𝑣𝑒 𝑜𝑓 𝑠𝑢𝑏𝑠𝑒𝑡2 = $6𝑀 + $5𝑀 +
$8𝑀 + $9𝑀 + $5𝑀 5 𝑎𝑣𝑒 𝑜𝑓 𝑠𝑢𝑏𝑠𝑒𝑡 2 = $6.6𝑀
• Get the average of the third subset (2005-
2009) 𝑎𝑣𝑒 𝑜𝑓 𝑠𝑢𝑏𝑠𝑒𝑡3 = $5𝑀 + $8𝑀 + $9𝑀 +
$5𝑀 + $4𝑀 5 𝑎𝑣𝑒 𝑜𝑓 𝑠𝑢𝑏𝑠𝑒𝑡3 = $6.2𝑀
• Continue calculating each five-year average
until you reach 2009-2013. This gives you a
series of points that you can plot a chart for
moving averages.
Averaging methods
• The Mean
• Uses the average of all the historical data as
the forecast
• When new data becomes available , the
forecast for time t+2 is the new mean including
the previously observed data plus this new
observation.
• This method is appropriate when there is no
noticeable trend or seasonality.
Assumptions on Moving Average
• The moving average for time period t is the
mean of the “k” most recent observations.
• The constant number k is specified at the
outset.
• The smaller the number k, the more weight is
given to recent periods.
• The greater the number k, the less weight is
given to more recent periods.
• A large k is desirable when there are wide,
infrequent fluctuations in the series.
• A small k is most desirable when there are
sudden shifts in the level of series.
• For quarterly data, a four-quarter moving
average, MA(4), eliminates or averages out
seasonal effects.
Reminders
• For monthly data, a 12-month moving
average, MA(12), eliminate or averages out
seasonal effect.
• Equal weights are assigned to each
observation used in the average.
• Each new data point is included in the average
as it becomes available, and the oldest data
point is discarded.
Moving average formula
A moving average of order k, MA(k) is the value
of k consecutive observations
K is the number of terms in the moving average.
The moving average model does not handle
trend or seasonality very well although it can do
better than the total mean.
Example: Weekly Department Store Sales
The weekly sales figures (in millions of dollars)
presented in the following table are used by a
major department store to determine the need
for temporary sales personnel.
Graph of Stores weekly sales
Using moving average
• Use a three-week moving average (k=3) for
the department store sales to forecast for the
week 24 and 26.
• The forecast error is
Forecasted value
• The forecast for the week 26 is
Results
RMSE = 0.63
 A simple moving average (SMA) is the
simplest type of moving average.
SUBTOPIC 3 EXPONENTIAL SMOOTHING
• This method provides an exponentially
weighted moving average of all previously
observed values.
• Appropriate for data with no predictable
upward or downward trend.
• The aim is to estimate the current level and
use it as a forecast of future value.
Simple Exponential Smoothing Method
Formally, the exponential smoothing equation
is
Where:
= 𝒇𝒐𝒓𝒆𝒄𝒂𝒔𝒕 𝒇𝒐𝒓 𝒕𝒉𝒆 𝒏𝒆𝒙𝒕 𝒑𝒆𝒓𝒊𝒐d
= 𝒔𝒎𝒐𝒐𝒕𝒉𝒊𝒏𝒈 𝒄𝒐𝒏𝒔𝒕𝒂𝒏𝒕

= 𝒐𝒃𝒔𝒆𝒓𝒗𝒆𝒅 𝒗𝒂𝒍𝒖𝒆 𝒐𝒇 𝒔𝒆𝒓𝒊𝒆𝒔 𝒊𝒏
𝒑𝒆𝒓𝒊𝒐𝒅 t
= 𝒐𝒍𝒅 𝒇𝒐𝒓𝒆𝒄𝒂𝒔𝒕 𝒇𝒐𝒓 𝒑𝒆𝒓𝒊𝒐𝒅 t
The forecast is 𝑭𝒕+𝟏 based on weighting the
most recent observation 𝒚𝒕 with a weight 𝜶 and
weighting the most recent forecast Ft with a
weight of 1- 𝜶.
• The implication of exponential smoothing can
be better seen if the previous equation is
expanded by replacing Ft with its components
as follows:
• If this substitution process is repeated by
replacing Ft-1 by its components, Ft-2 by its
components, and so on the result is:
• Therefore, Ft+1 is the weighted moving
average of all past observations.
The following table shows the weights assigned
to past observations for = 0.2, 0.4, 0.6, 0.8, 0.9
• The exponential smoothing equation
rewritten in the following form elucidate the
role of weighting factor 𝛼.
• Exponential smoothing forecast is the old
forecast plus an adjustment for the error that
occurred in the last forecast.
Assumptions
• The value of smoothing constant 𝛼 must be
between 0 and 1
• 𝛼 can not be equal to 0 or 1.
• If stable predictions with smoothed random
variation is desired then a small value of 𝛼 is
desire.
• If a rapid response to a real change in the
pattern of observations is desired, a large value
of 𝛼 is appropriate.
• To estimate 𝛼, Forecasts are computed for 𝛼
equal to .1, .2, .3, …, .9 and the sum of squared
forecast error is computed for each.
• The value of 𝛼 with the smallest RMSE is
chosen for use in producing the future
forecasts.
Simple Exponential Smoothing Method
• To start the algorithm, we need F1 because
• Since F1 is not known, we can
• Set the first estimate equal to the first
observation.
• Use the average of the first five or six
observations for the initial smoothed value.
Holt’s Exponential Smoothing
• Holt’s two parameter exponential smoothing
method is an extension of simple exponential
smoothing.
• It adds a growth factor (or trend factor) to the
smoothing equation as a way of adjusting for
the trend.
• Three equations and two smoothing constants
are used in the model.
• The exponentially smoothed series or
current level estimate.
• The trend estimate.
 • Forecast m periods into the future. Examination of the plot

• A non-stationary time series data.

• Seasonal variation seems to exist.

• Sales for the first and fourth quarter are larger

Variables in Holt’s ES
than other quarters.
• Lt = Estimate of the level of the series at time t

•  = smoothing constant for the data.

• yt = new observation or actual value of series

in period t.

•  = smoothing constant for trend estimate

• bt = estimate of the slope of the series at time

t

• m = periods to be forecast into the future.

Example: Quarterly sales of saws for Acme tool

Holt’s Exponential Smoothing
company
• The weight  and  can be selected
• The plot of the Acme data shows that there
subjectively or by minimizing a measure of
might be trending in the data therefore we will
forecast error such as RMSE.
try Holt’s model to produce forecasts.
• Large weights result in more rapid changes in
• We need two initial values
the component.
• The first smoothed value for L1
• Small weights result in less rapid changes.
• The initial trend value b1.
Example: Quarterly sales of saws for Acme tool
company • We will use the first observation for the
estimate of the smoothed value L1, and the
• The following table shows the sales of saws
initial trend value b1 = 0.
for the Acme tool Company.
• We will use 𝛼 = .3 and 𝛼 =.1.
• These are quarterly sales From 1994 through
2000.
• RMSE for this application is: 𝛼 = .3 and 𝛼 = .1 Compute for the MSE
RMSE = 155.5
Actual
• The plot also showed the possibility of
seasonal variation that needs to be 20
investigated.
100 abs(20 – 30) ^ 2

FORMATIVES
A trend is usually the result of long-term
factors such as population increases or
decreases, shifting demographic
characteristics of the population,
improving technology, changes in the
competitive landscape, and/or changes
in consumer preferences. True
What does the graph below illustrate? an
increasing trend only
Compute for the MSE: -9 85.5 81 9
Actual
81
81 abs(81 – 90) ^ 2
MSE stands for the mean standard error.
False
The reproduction of crops is highly
dependent on this component in time
series. Seasonal variations
Which of the following describes an
unpredictable, rare event that appears in the
time series? Irregular variations

Mean absolute error is the mean or 88.67 dalwa sagot ? 88.67

average of the forecast errors. False Moving average is suitable for dealing with a
time series that has short-term trends. True
Qualitative methods can be used when
past information about the variable Compute for the MAD based on the
being forecast is available. False following values:3
Trend projection is an example of a time
series. True Error |Error|

Compute for the forecasted error: 3

5 5
Actual
39 -3 3
4 4
The formula below describes MAD
Another measure is mean standard error
(MSE), which is a measure of the average
size of the prediction errors in squared units.
True
It is a model that tests the relationship
between a dependent variable and a single
independent variable. Linear Regression
The moving average method uses a
weighted average of the observed value.
False
Which of the following describes the overall
tendency of a time series? Trend
Its objective is to uncover a pattern in the
time series and then extrapolate the pattern
into the future. Time Series Analysis
If a time series is generated by a constant
process subject to random error, then
median is a useful statistic. True
The value of α with the smallest RMSE is
chosen for use in producing future forecasts.
True
One measure of forecasting accuracy is the
mean accuracy deviation (MAD). As the
name suggests, it is a measure of the
average size of the prediction errors. To
estimate the change in Y for a one-unit
change in X. false
Based on the given values below, what is the
value of F3 if the value of the alpha is 0.2?
40 (F1&F1 = 39 TO GET 40
F3=0.2(44)+0.8(39.99))
It pertains to the gradual shifts or
movements to relatively higher or lower
values over a longer period of time.
Trend Pattern
Another measure is mean standard error
(MSE), which is a measure of the average
size of the prediction errors in squared units.
True
The forecast is Ft+1 based on weighting the
most recent observation yt with a weight 
alpha and weighting the most recent
forecast Ft with a weight of 1-alpha . true
Supply the missing values given for the
attribute salary. 19, 275 18, 625 18, 650
19,525
salary
16000
12000
17500
29000
If elections are held every 6 years in a
country, then the existing government may
allow wages to increase the year before the
election to make the people more likely to
vote for them. The statement is an example
of cyclical variation. True
Trend series a sequence of observations on
a variable measured at successive points in
time or over successive periods of time. True
The classical multiplicative time series model
indicates that the forecast is the product of
which of the following terms?
- trend, cyclical, and irregular
components
Which of the following statements is correct
regarding moving averages?
- The choice of the number of
periods impacts the performance
of the moving average forecast
The method of moving averages is used for
which of the following purposes?
- smooth the time series
When exponential smoothing is used as a
forecasting method, which of the following is
used as the forecast for the next period?
- smoothed value in the current time
period
Which of the following is a valid weight for
exponential smoothing?
- 0.5
Which of the following indicates the purpose for
using the least squares method on time series
data?
- identifying the trend variations
An autoregressive forecast includes which of
the following terms?
- All of the above
Which of the following indicates a guideline for
selecting a forecast model?
- All of the above
To assess the adequacy of a forecasting model,
which of the following methods is used?
- Mean absolute deviation
Which of the following indicates the calculation
of an index number?
- (Pi/Pbase)100
How many degrees of freedom are used for the
t test to determine the significance of the
highest order autoregressive term?
- n – 2p – 1
22/30
Each year, more sweaters are sold in
winter and very few in the summer
season. This is an example of a
______________.
- Seasonal variations
It is a measure that avoids the problem of
positive and negative errors offsetting each
other is obtained by computing the average
of the squared forecast errors.
- Mean Absolute Error
Which of the following research scenarios
would time-series analysis be best for?
- Measuring the time it takes for
something to happen based on a given
number of variables
Compute for the Mean Absolute Deviation:
- 4
A trend is usually the result of long-term
factors such as population increases or
decreases, shifting demographic
characteristics of the population,
improving technology, changes in the
competitive landscape, and/or changes
in consumer preferences.
- True
It pertains to the gradual shifts or
movements to relatively higher or lower
values over a longer period of time
- Trend Pattern
It is a measure of forecast accuracy that
avoids the problem of positive and negative
forecast errors offsetting one another.
 - Mean Absolute Error

Compute for the MSE

- 100

Spike is the predictable change in something

based on the season.

- False

It pertains to the difference between the

actual and the forecasted values for period t

- Forecast Error

A time series data usually has two variables

namely transaction and item. Deviation and error are used
- False interchangeably in time series

- False

Which of the following statements best

describes the cyclical component?

- It represents periodic fluctuations that

recur within the business cycle

Compute for the 3 Moving Average based

on the table below: Compute for the MSE based on the
following values:

-
The formula below describes

- MAPE (d nascreenshot shuta)

Compute for the MAD based on the

following value:
- 16.67?

MAPE stands for mean absolute predicted

error

- False

Compute for the MAD:

 Exponential Smoothing is an obvious
extension the moving average method

- True (False)
- 6
Given the following values for age, what is
It is the variable being manipulated by the problem with the data?
researchers.

- Explanatory variable

MAPE stands for mean absolute predicted

error

- False

Moving average is suitable for dealing with a

time series that has short-term trends.

- True

A trend is usually the result of long-term

factors such as population increases or
decreases, shifting demographic
characteristics of the population, improving
technology, changes in the competitive
landscape, and/or changes in consumer
preferences

- True

Trend series a sequence of observations on

a variable measured at successive points in
time or over successive periods of time

- True

Data Inconsistency?
A heterogenous data set is a data set whose
data records have the same target value. A time series data usually has two variables
namely transaction and item.
- False
- False
The sales of the mini electric fans that
Louise sells varies every season. This is an
example of a seasonal effect to a time series.
24/20
- True Regression is a famous supervised learning
technique.
One measure of forecasting accuracy is the
- True
mean accuracy deviation (MAD). As the
name suggests, it is a measure of the Forecasting methods can be classified as
average size of the prediction errors. To qualitative and quantitative
estimate the change in Y for a one-unit
change in X. - True

- False MSE stands for the mean standard error.

 - False
It pertains to the gradual shifts or
movements to relatively higher or lower
values over a longer period of time.
- Trend Pattern
Compute for the MAPE:
The classical multiplicative time series model
indicates that the forecast is the product of
which of the following terms?
- trend, cyclical, and irregular
components
The MAD for the following values is 63.67.

- -7.69

It is a measure of forecast accuracy that

avoids the problem of positive and negative
forecast errors offsetting one another

- Mean Absolute Error

A trend is usually the result of long-term

factors such as population increases or
decreases, shifting demographic
- False
characteristics of the population, improving Time series is an example of unsupervised
technology, changes in the competitive learning.
landscape, and/or changes in consumer
preferences. - False

- True Averaging methods are suitable for

stationary time series data
These are fluctuations in the time series that
are short in duration, erratic in nature - True

- Irregular variations The formula below describes

What component affects the time series

analysis when a sudden volcanic eruption
happens?

- Irregular variations

What does the graph below illustrate?

- MAPE
an increasing trend only
The choice of the number of periods impacts
the performance of the moving average
The seasonal component represents periodic forecast.
fluctuations that recur within the business
- True
cycle.
It is the variable being manipulated by
- False researchers.
MSE stands for ________.
- Explanatory variable
- Mean Squared Error

The formula below describes

 - True

Exponential Smoothing is an obvious

extension of the five- moving average
method

- False

Averaging methods are suitable for

stationary time series data
- MSE - True
The equation below illustrates how to Supply the missing values given for the
compute the mean absolute deviation. attribute salary.

- False

Averaging methods are suitable for

nonstationary time series data.

- False

Which of the following describes an

unpredictable, rare event that appears in the
time series?

- Irregular variations
Trend series a sequence of observations on
a variable measured at successive points in
time or over successive periods of time
- True
Exponential Smoothing is an obvious
extension the moving average method.
- 18,650
With moving average the idea is that the
most recent observations will usually
provide the best guide as to the future, so
we want a weighting scheme that has
decreasing weights as the observations get
older (exponential smoothing)
False

- False

The value of α with the smallest RMSE is

chosen for use in producing future forecasts.

- True

The forecast is Ft+1 based on weighting the

most recent observation yt with a weight  MODULE 5 OVERVIEW OF
alpha and weighting the most recent ASSOCIATION ANALYSIS
forecast Ft with a weight of 1-alpha .
What are patterns?
Patterns are set of items, subsequences,
or substructures that occur frequently
together (or strongly correlated) in a
data set Patterns represent intrinsic and
importance properties of datasets
When do you use Pattern Discovery?
❑ What products were often purchased
together?
❑ What are the subsequent purchases
after buying an iPad?
❑ What code segments likely contain
copy-and-paste bugs?
❑ What word sequences likely form
phrases in this corpus?
Concept of Market Basket Analysis
Market basket analysis is like an
imaginary basket used by retailers to
check the combination of two or more
items that the customers are likely to
buy
“Two-thirds of what we buy in the
supermarket we had no intention of
buying,” - Paco Underhill, author of
Why We Buy: The Science of Shopping
Association Rule is the foundation of
several recommender systems
Amazon
Spotify
Possible actions based on Association rule
Barbie Doll -> Candy
1. Pull them closer together in the store.
2. Put them far apart in the store.
3. Package candy bars with the dolls
4. Package Barbie + candy + poorly selling item
5. Raise the price on one, and lower it on the
other
6. Offer Barbie accessories for proofs of
purchase
7. Do not advertise candy and Barbie together
8. Offer candies in the shape of a Barbie doll
Process of Rule Selection
Generate all rules that meet specified support &
confidence
• Find frequent item sets (those with sufficient
support – see above)
• From these item sets, generate rules with
sufficient confidence
Rule Interpretation
Lift ratio shows how effective the rule is in
finding consequents (useful if finding particular
consequents is important)
Confidence shows the rate at which
consequents will be found (useful in learning
costs of promotion)
Support measures overall impact
Rule sets (XLMiner)
Challenges
 Random data can generate apparently
interesting association rules
 The more rules you produce, the
greater this danger
 Rules based on large numbers of
records are less subject to this danger
SUBTOPIC 1 PATTERN DISCOVERY
What is the essence of data mining?
“...the discovery of interesting, unexpected, or
valuable structures in large data sets.” - David
Hand
“If you’ve got terabytes of data, and you’re
relying on data mining to find interesting things
in there for you, you’ve lost before you’ve even
begun.” - Herb Edelstein
What is pattern discovery?
According to SAS, it is one of the broad
categories of analytical methods associated
with data mining. It is a process of uncovering
patterns from massive data sets.
Pattern Discovery Caution
• Poor data quality (hindi enough yung data or
walang pattern, pinipilit lang)
• Opportunity
• Interventions (irregular variations)
• Separability (different dataset, di naka sync
yung databse)
• Obviousness (trivial rules, logical rules, like
kapag umorder ng chicken matik may kasamang
coke)
• Non-stationarity (inconsistent yung sales)
Applications of Pattern Discovery
• Data reduction
• Novelty detection (mga loyalty cards)
• Profiling (purchasing score, range ng binibili
nla, if employed or student)
• Market basket analysis (what are the two
products that are bought together, combination
product x and y)
• Sequence analysis (What is the next product
na kukunin mo after you purchase one)
Pattern discovery tools
Pattern discovery key ideas
• Patterns are not known
• But data which are believed to possess
patterns are given
• Examples:
• Clustering: grouping similar samples into
clusters
• Associative rule mining: discover certain
features that often appear together in data
Pattern Finding
• Patterns are known beforehand and are
observed/described by:
• Explicit samples
• Similar samples (usually)
• Modeling approaches
• Build a model for each pattern
• Find the best fit model for new data
• Usually require training using observed
samples
Sequential Pattern

Sequential patterns: shopping sequences,

medical treatments, natural disasters, weblog
click streams, programming execution
sequences, DNA, protein, etc.

Pattern discovery techniques

1. Clustering - Clustering aims at dividing the

data set into groups (clusters) where the inter-
cluster similarities are minimized while the
similarities within each cluster are maximized

2. Association Rule mining - Association rule

discovery on usage data results in finding
groups of pages that are commonly accessed.
3. Sequential pattern mining
4. Classification
5. Decision Trees
What is confidence?
Confidence defines the likeliness of occurrence
of consequent on the cart given that the cart
already has the antecedents. Confidence is
estimated using the following:

6. Naïve Bayes Classifier

Technically, confidence is the conditional

What is support? probability of occurrence of consequent given
the antecedent.
The strength of the association is measured by
the support and confidence of the rule. Calculating for the confidence

Support is estimated using the following:

The support for the rule A -> B is the probability

that the two item sets occur together

Calculating for the Support

Support = if then, overall probability na bibilhin

yung product together, how frequenty the if
and then relationship appears

Confidence = ano yung probability na kukunin

yung consequent given the antecedent
Rule Interpretation
Lift ratio shows how effective the rule is in
finding consequents (useful if finding particular
consequents is important)
Confidence shows the rate at which
consequents will be found (useful in learning
costs of promotion)
Support measures overall impact
SUBTOPIC 2 ASSOCIATION RULE
Market basket analysis (also known as
association rule discovery or affinity analysis) is
a popular data mining method. In the simplest
situation, the data consists of two variables: a
transaction and an item.
Forbes (Palmeri 1997) reported that a major
retailer determined that customers who buy
Barbie dolls have a 60% likelihood of buying one
of three types of candy bars.
Application of Association Rule
• Market Basket Analysis: given a database of
customer transactions, where each transaction
is a set of items the goal is to find groups of
items which are frequently purchased together.
• Telecommunication (each customer is a
transaction containing the set of phone calls)
• Credit Cards/ Banking Services (each
card/account is a transaction containing the set
of customer’s payments)
• Medical Treatments (each patient is
represented as a transaction containing the
ordered set of diseases)
• Basketball-Game Analysis (each game is
represented as a transaction containing the
ordered set of ball passes)
Advantages/Disadvantages
• Advantages:
– Uses large itemset property.
– Easily parallelized
– Easy to implement.
• Disadvantages:
– Assumes transaction database is memory
resident.
– Requires many database scans.
Key Ideas
• a set of all the
items
• Transaction T: a set of items such that T I
• Transaction Database D: a set of transactions
• A transaction T  I contains a set X  I of
some items, if X  T
• An Association Rule: is an implication of the
form X  Y, where X, Y  I
• A set of items is referred as an itemset. A
itemset that contains k items is a k-itemset.
• The support s of an itemset X is the
percentage of transactions in the transaction
database D that contain X.
• The support of the rule X  Y in the
transaction database D is the support of the
items set X  Y in D.
• The confidence of the rule X  Y in the
transaction database D is the ratio of the
number of transactions in D that contain X  Y
to the number of transactions that contain X in
D.
Sample Scenario
Sample dataset
Finding the association rules
Variables in Association Rule
• Given:
― a set I of all the items;
― a database D of transactions;
― minimum support s;
― minimum confidence c;
• Find:
― all association rules X  Y with a minimum
support s and confidence c.
Problem Decomposition
• Find all sets of items that have minimum
support (frequent itemsets)
• Use the frequent itemsets to generate the
desired rules
If the minimum support is 50%, then
{Shoes,Jacket} is the only 2- itemset that
satisfies the minimum support.
If the minimum confidence is 50%, then the
only two rules generated from this 2-itemset,
that have confidence greater than 50%, are:
Shoes  Jacket Support=50%, Confidence=66%
Jacket  Shoes Support=50%,
Confidence=100%
Terms
“IF” part = antecedent
“THEN” part = consequent
“Item set” = the items (e.g., products)
comprising the antecedent or consequent
• Antecedent and consequent are disjoint (i.e.,
have no items in common)
Example: Phone Faceplates
Many rules are possible
For example: Transaction 1 supports several
rules, such as
• “If red, then white” (“If a red faceplate is
purchased, then so is a white one”)
• “If white, then red”
• “If red and white, then green”
• + several more
Confidence
Example rule
{red, white} > {green} with confidence = 2/4 =
50%
• [(support {red, white, green})/(support {red,
white})]
{red, green} > {white} with confidence = 2/2 =
100%
• [(support {red, white, green})/(support {red,
green})]
Plus 4 more with confidence of 100%, 33%, 29%
& 100%
If confidence criterion is 70%, report only rules
2, 3 and 6
Sample Association Problem
Task 1: Compute for the Support

What is the support for 𝑚𝑖𝑙𝑘 -> 𝑎𝑝𝑝𝑙𝑒𝑠?

Association Rule Mining

Task 2: Compute for the Confidence

What is the confidence of {𝑎𝑝𝑝𝑙𝑒𝑠 → 𝑏𝑎𝑛𝑎𝑛𝑎}?

1. It is intended to select the “best”

subset of predictors. Variables
Selection
2. Write the Support formula for the
following expression;
Task 3: Compute for the lift ratio
A => B
What is the lift ratio of {𝑎𝑝𝑝𝑙𝑒𝑠 → 𝑏𝑎𝑛𝑎𝑛𝑎}?
(transactions that contain every

item A and B)/(all transactions)

3. Affinity analysis is a data mining

method that usually consists of two
variables a transaction and an item.
[TRUE]
4. Which of the following is not an
application of a sequential pattern?
Identifying fake news]
5. Which of the following is not an
application of pattern discovery?
None of pattern discovery
6. It shows how effective the rule is in
finding consequents. Lift ratio
7. Trend series a sequence of
observation on a variable measured
at successive points in time or over
successive periods of time. False
8. Observe the table below and
compute for the confidence of
Beer->diaper.

0.43
10. Observe the table below and
compute for the support of
Beer=>peanut 4/7
11. Observe the table below and
compute for the support of
SD Card => Phone case

None of the choices

9. Observe the table below and
compute for the confidence of
Diaper => peanut

0.3
12. Observe the table below and
compute for the lift ratio of

1.5
13. When it comes to association
analysis, the more rules you
produce, the greater the risk is.
True
14. Supposed you want to solve a time
series problem where a rapid
response to a real change in the
pattern of observations is desired,
which among the following is the
ideal value for your alpha? 0.8
15. Which of the following is not an
advantage of using association rule?
Assumes transaction database is
memory resident.
16. A trend is usually the result of long-
term factors such as population
increases or decreases, shifting
demographic characteristics of
population, improving technology,
changes in the competitive
landscape, and/or changes in
consumer preferences. True
17. It measures the overall impact.
Support
18. Clustering aims to discover certain
features that often appear together
in data.False
19. It is another type of association
analysis that involves using
sequence data. Association Rule
20. Observe the table below and
compute for the support of
SD Card => Phone case
0.3
1. Segmentation is a data mining
method that usually consists of
two variables a transaction and
an item. False
2. It is a useful tool for data
reduction, such as choosing the
best variables or cluster
components for
analysis.Variable Clustering
3. It controls for the support
( frequency) of consequent
while calculating the
conditional probability of
occurrence of {Y} given {X}. Lift
ratio
4. Association rule mining is about
grouping similar samples into
clusters. False
5. It is the process of discovering
useful patterns and trends in
large data sets.[data mining]
6. observe the table below and
compute for the lift ratio of
egg -> Peanut
 given the antecedent.
Confidence
10. Input validation helps to lesson
what type of anomaly?
Insertion anomaly
11. observe the table below and
compute for the confidence of
Phone -> SD card

0.93
7. observe the table below and
compute for the confidence of
Airpods->powerbank

0.5
12. Lift ratio shows how effective
the rule is in finding
consequents. True
13. observe the table below and
compute for the lift ratio of

0.4

8. observe the table below and

support for Fries -> Burger

0.6
9. It is the conditional probability
of occurrence of consequent 0.1
The objective of clustering is to
uncover a pattern in the time
series and then extrapolate the
pattern into the future. False

Observe the table below and

compute for the lift ratio of egg→
peanut.

- 4/7

Which of the following is not an

application of pattern discovery?
- None of the choices

Pattern discovery can also be applied

to data profiling.

- False

Observe the table below and compute

for the confidence of egg→ beer

- 0.93

Observe the table below and

compute for the support for beer⇒
peanut

- None of the choices

Observe the table below and compute

for the confidence of beer→diaper.
 Observe the table below and compute for the
support for diaper⇒ peanut

- 1/5 or 0.2

Observe the table below and compute for the

confidence of egg→peanut
- 0.43

It is the conditional probability of

occurrence of consequent given the
antecedent

- Confidence

Observe table below and compute for the

support for powerbank⇒charger

- 4/5

Lift ration shows how effective the rule is in

finding consequents.

- True

It pertains to how likely item Y is purchased

when item X is purchased, expressed as {X
-> Y}. (Use lowercase for your answer)

- confidence
 19/20

- 0.1

It pertains to how likely item Y is purchased

when item X is purchased, expressed as {X
-> Y}. (Use lowercase for your answer)

- confidence

This measure defines the likeliness of

occurrence of consequent on the cart given
that the cart already has the antecedents.

- Confidence

It pertains to how popular an itemset is, as

measured by the proportion of transactions
in which an itemset appears. (Use lowercase
for your answer)

- lift ratio - 1
It measures the overall impact. Its objective is to uncover a pattern in the
time series and then extrapolate the pattern
- Support
into the future.
Which of the following is not an application
- Time Series
of pattern discovery?

- None of the choices

Observe the table below and compute for
Observe the table below and compute for the
the support for milk⇒ egg.
lift ratio of phone case→ SD Card
 It pertains to how likely item Y is
purchased when item X is purchased
while controlling for how popular item Y
is. (Use lowercase for you answer)
- lift ratio

These are set of items, subsequences, or

substructures that occur frequently together
(or strongly correlated) in the data set.

- Patterns

Market Basket Analysis creates If-Then

scenario rules.

- True

Which of the following is not an application

- 0.14 of pattern discovery?
Observe the table below and compute for - None of the choices
the confidence of beer→diaper.

Observe the table below and compute for

the lift ratio of egg→ peanut.

- 0.40

Lift ration shows how effective the rule

- 0.93
is in finding consequents.

It is a useful tool for data reduction, such

- True
as choosing the best variables or cluster
components for analysis. (Use lowercase
for your answer) variable clustering
pattern discoveryare also part of the Observe the table below and compute for
pattern discovery. the support for airpods⇒ charger

- False

Clustering aims to discover certain features

that often appear together in data.

- False

It measures the overall impact.

- Support

Observe the table below and compute for

the support for SD Card⇒ phone case

- 0.4

Its objective is to uncover a pattern in the

time series and then extrapolate the pattern
into the future. - Time Series

Input validation helps to lessen what

type of anomaly? Insertion anomaly

It shows how effective the rule is in

finding consequents. Lift ratio

Lift ratio shows how effective the rule is

in finding consequents. True
- 0.3
This measure defines the likeliness of
occurrence of consequent on the cart
Which of the following is not an application given that the cart already has the
of pattern discovery? antecedents.

- None of the choices Confidence


F5 16/20 19/20
1. The objective of clustering is to
uncover a pattern in the time series
and then extrapolate the pattern into
the future. False true
2. These are set of items,
subsequences, or substructures that
occur frequently together (or
strongly correlated) in the data set.
Pattern
3. When it comes to association
analysis, the more rules you produce,
the greater the risk is. True
4. Observe the table below and
compute for the support
for diaper⇒ peanut
Transaction ID Items
Transaction A Beer, Peanut, Egg
Beer, Milk, Peanut,
Transaction B
Diaper
Transaction C Milk, Diaper, Egg
Transaction D Peanut, Egg, Diaper
Transaction E Beer, Peanut, Egg
Transaction F Egg, Beer, Peanut
Transaction G Beer, Diaper, Peanut
1.75 (3/7 = .43 Umulit sa number19)
5. It controls for the support
(frequency) of consequent while
calculating the conditional
probability of occurrence of {Y} given
{X}. Lift ratio
6. It pertains to how popular an itemset
is, as measured by the proportion of
transactions in which an itemset
appears. Support
7. It shows how effective the rule is in
finding consequents. Lift ratio
8. Observe the table below and
compute for the lift ratio
of diaper→ milk.
Transaction ID Items
Transaction A Beer, Peanut, Egg
Beer, Milk, Peanut,
Transaction B
Diaper
Transaction C Milk, Diaper, Egg
Transaction D Peanut, Egg, Diaper
Transaction E Beer, Peanut, Egg
Transaction F Egg, Beer
Transaction G Beer, Diaper, Peanut
1.5 0.29/0.29*0.57=1.75
9. Supposed you want to solve a
time series problem where a rapid
response to a real change in the
pattern of observations is desired,
which among the following is the
ideal value for your alpha ? 0.2
(Choices : 0.2 1 0 0.8)
10. Association rule mining is about
grouping similar samples into
clusters. False
11. It is the conditional probability of
occurrence of consequent given the
antecedent. Confidence
12. It is the conditional probability of
occurrence of consequent given the
antecedent. Confidence
13. Observe the table below and
compute for confidence of phone
case→ SD Card
Transaction ID Items Transaction D Peanut, Egg, Diaper

airpods, charger, Transaction E Beer, Peanut, Egg

1
powerbank
Transaction F Egg, Beer
2 powerbank, phone case
Transaction G Beer, Diaper, Peanut
airpods, phone case,
3 4/7.
charger
16. Market Basket Analysis creates If-
4 phone case, SD Card Then scenario rules. True
17. It shows how effective the rule is in
finding consequents. Lift ratio
5 SD Card, charger, airpods 18. It is another type of association
analysis that involves using sequence
data. Sequence Analysis
SD Card, phonecase,
6
powerbank

Powerbank, phonecase, SD
7
Card

8 airpods, powerbank

19. Observe the table below and

9 phone case, airpods compute for the support
for diaper⇒peanut
10 charger, SD Card, airpods
Transaction ID Items
0.5 (0.3/0.6 = 0.5)
14. It pertains to how popular an itemset Transaction A Beer, Peanut, Egg
is, as measured by the proportion of
transactions in which an itemset
appears. (Use lowercase for your Beer, Milk, Peanut,
Transaction B
answer) support Diaper
15. Observe the table below and
compute for the support Transaction C Milk, Diaper, Egg
for beer⇒ peanut

Transaction ID Items Transaction D Peanut, Egg, Diaper

Transaction A Beer, Peanut, Egg Transaction E Beer, Peanut, Egg

Beer, Milk, Peanut, Transaction F Egg, Beer, Peanut

Transaction B
Diaper
Transaction G Beer, Diaper, Peanut
Transaction C Milk, Diaper, Egg
0.43 ( 3/7 = .43 Inulit nya lang nasa number 4)
 20. It shows how effective the rule is in
finding consequents. Lift ratio
MODULE 6
What is data preparation?
Data preparation is the method of transforming
raw data into a form suitable for modeling. It is
also known as data preprocessing.
Why do we need to preprocess the data?
Missing values
Having null values in your data set could affect
the accuracy of the model.
Outliers
When your data has outliers, it could affect the
distribution of your data.
• Inconsistent data
• Improperly formatted data
• Limited features
• The need for techniques such as feature
engineering
Ways to Preprocess Data
Data cleaning
Data cleaning (or data cleansing) routines
attempt to fill in missing values, smooth out
noise while identifying outliers, and correct
inconsistencies in the data. In this section, you
will study basic methods for data cleaning.
Data Integration and Transformation
There are a number of issues to consider during
data integration. Schema integration and object
matching can be tricky. This is referred to as the
entity identification problem.
Data Reduction
Data reduction techniques can be applied to
obtain a reduced representation of the data set
that is much smaller in volume, yet closely
maintains the integrity of the original data.
Data Preprocessing Techniques
Variables Selection How to handle missing data?

Observe minimizing garbage in, garbage out Missing data is a problem that continues to
(GIGO)! plague data analysis methods. Let’s examine the
cars data set.
Procedures

1. Backward-selection

2. Forward-selection

SUBTOPIC 1 DATA ANOMALIES A common method of “handling” missing values

What are anomalies?
Anomalies are problems that can occur in
poorly planned, unnormalized databases where
all the data is stored in one table (a flat-file
database). Anomalies are caused when there is
too much redundancy in the database's
information
Database Anomalies
• Insertion anomaly - happen when inserting
vital data into the database is not possible
because other data is not already there.
• Update Anomalies - happen when the person
charged with the task of keeping all the records
current and accurate, is asked, for example, to
change an employee’s title due to a promotion.
• If the data is stored redundantly in the same
table, and the person misses any of them, then
there will be multiple titles associated with the
employee. The end user has no way of knowing
which is the correct title.
is simply to omit the records or fields with
missing values from the analysis. However, this
action would make our data bias.
Common criteria in handling null values
1. Replace the missing value with some
constant, specified by the analyst
2. Replace the missing value with the field mean
(for numeric variables) or the mode (for
categorical variables).
3. Replace the missing values with a value
generated at random from the observed
distribution of the variable.
4. Replace the missing values with imputed
values based on the other characteristics of the
record.
Applications

• Deletion Anomalies - happen when the

deletion of unwanted information causes
desired information to be deleted as well. 1 ) Replacing missing field values with user
-defined constants.
• For example, if a single database record
contains information about a particular product
along with information about a salesperson for
the company and the salesperson quits, then
information about the product is deleted along
with salesperson information.
2) Replacing missing field values with means or
modes.
How to handle outliers? • Including categorical inputs in the model can
cause quasi-complete separation.
Use graphical methods to identify outliers
How to handle noisy data?

Handling Noisy Data

1. Binning: Binning methods smooth a sorted

data value by consulting its “neighbor-hood,”
that is, the values around it.

2. Regression: Data can be smoothed by fitting

the data to a function, such as with regression.
Linear regression involves finding the “best” line
to fit two attributes (or variables), so that one
Histogram of vehicle weights: can you find the
attribute can be used to predict the other.
outlier?
3. Clustering: Outliers may be detected by
clustering, where similar values are organized
into groups, or “clusters.” Intuitively, values
that fall outside of the set of clusters maybe
considered outliers

Dealing with categorical inputs

Solutions to the problems of categorical inputs

include the following choices:

Variable Clustering • Use the categorical input as a link to other

data sets.
Mortgage Balance
Credit Card Balance • Collapse the categories based on the number
of observations in a category.
Number of Checks
Checking Deposits • Collapse the categories based on the
Teller Visits reduction in the chi-square test of association
between the categorical input and the target.
Age
• Use smoothed weight of evidence coding to
convert the categorical input to a continuous
input.

Anomaly detection

Anomaly detection (aka outlier analysis) is a

How to handle categorical data?
Dealing with categorical inputs
• When a categorical input has many levels,
expanding the input into dummy variables can
greatly increase the dimension of the input
space.
step in data mining that identifies data points,
events, and/or observations that deviate from a
dataset’s normal behavior.
Anomalous data can indicate critical incidents,
such as a technical glitch, or potential
opportunities, for instance a change in
consumer behavior.
Observe the graph
This graph shows an anomalous drop detected
in time series data. The anomaly is the yellow
part of the line extending far below the blue
shaded area, which is the normal range for this
metric.
Three main categories of business data
anomalies
1. Global Outliers
Also known as point anomalies, these outliers
exist far outside the entirety of a data set.
2. Contextual Outliers
Also called conditional outliers, these anomalies
have values that significantly deviate from the
other data points that exist in the same context.
3. Collective outliers
When a subset of data points within a set is
anomalous to the entire dataset, those values
are called collective outliers. In this category,
individual values aren’t anomalous globally or
contextually.
Why does your company needs anomaly
detection?
Reasons to detect anomalies
1. Anomaly detection for application
performance
2. Anomaly detection for product quality
3. Anomaly detection for user experience
SUPPLEMENTARY MATERIAL 1
Why Preprocess the Data?
Supposing that you are a manager of one
company and have been charged with analyzing
the company’s data with respect to the sales at
your branch. You immediately set out to
perform this task. You carefully inspect the
company’s database and data warehouse,
identifying and selecting the attributes or
dimensions to be included in your analysis, such
as item, price, and units sold.
You notice that several of the attributes for
various tuples have no recorded value. For your
analysis, you would like to include information
as to whether each item purchased was
advertised as on sale, yet you discover that this
information has not been recorded.
Furthermore, users of your database sys-tem
have reported errors, unusual values, and
inconsistencies in the data recorded for some
transactions.
Incomplete, noisy, and inconsistent data are
commonplace properties of large real-world
databases and data warehouses. Incomplete
data can occur for a number of rea-sons.
Attributes of interest may not always be
available, such as customer information for
sales transaction data.
There are many possible reasons for noisy data
(having incorrect attribute values). The data
collection instruments used may be faulty.
There may have been human or computer
errors occurring at data entry. Errors in data
transmission can also occur.
There may be technology limitations, such as
limited buffer size for coordinating synchronized
data transfer and consumption. Incorrect data
may also result from inconsistencies in naming
conventions or data codes used, or inconsistent
formats for input fields, such as date. Duplicate
tuples also require data cleaning.
Data cleaning routines work to “clean” the data
by filling in missing values, smoothing noisy
data, identifying or removing outliers, and
resolving inconsistencies. If users believe the
data are dirty, they are unlikely to trust the
results of any data mining that has been applied
to it.
Suppose that you would like to include data
from multiple sources in your analysis. This
would involve integrating multiple databases,
data cubes, or files, that is data integration. Yet
some attributes representing a given concept
may have different names in different
databases, causing inconsistencies and
redundancies.
Furthermore, it would be useful for your
analysis to obtain aggregate information as to
the sales per customer region—something that
is not part of any precomputed data cube in
your data warehouse. You soon realize that
data transformation operations, such as
normalization and aggregation, are additional
data preprocessing procedures that would
contribute toward the success of the mining
process.
Data reduction obtains a reduced
representation of the data set that is much
smaller in volume, yet produces the same (or
almost the same) analytical results. There are a
number of strategies for data reduction. These
include data aggregation (e.g., building a data
cube), attribute subset selection (e.g., removing
irrelevant attributes through correlation
analysis), dimensionality reduction (e.g., using
encoding schemes such as minimum length
encoding or wavelets), and numerosity
reduction (e.g., “replacing” the data by
alternative, smaller representations such as
clusters or parametric models).
Descriptive Data Summarization
For data preprocessing to be successful, it is
essential to have an overall picture of your data.
Descriptive data summarization techniques can
be used to identify the typical properties of your
data and highlight which data values should be
treated as noise or outliers.
The descriptive statistics are of great help in
understanding the distribution of the data. Such
measures have been studied extensively in the
statistical literature. From the data mining point
of view, we need to examine how they can be
computed efficiently in large databases.
Measuring the Central Tendency
The most common and most effective numerical
measure of the “center” of a set of data is the
(arithmetic) mean.
Different Measures of Central Tendency
1. Distributive Measure
2. Algebraic Measure
3. Holistic Measure
Measuring the Dispersion of Data
The degree to which numerical data tend to
spread is called the dispersion, or variance of
the data. The most common measures of data
dispersion are range, the five number summary
(based on quartiles), the interquartile range,
and the standard deviation. Boxplots can be
plotted based on the five-number summary and
are a useful tool for identifying outliers.
Graphic Displays of Basic Descriptive Data
Summaries
Aside from the bar charts, pie charts, and line
graphs used in most statistical or graphical data
presentation software packages, there are other
popular types of graphs for the display of data
summaries and distributions. These include
histograms, quantile plots, q-q plots, scatter
plots, and loess curves. Such graphs are very
helpful for the visual inspection of your data.
Data Cleaning
Data cleaning (or data cleansing) routines
attempt to fill in missing values, smooth out
noise while identifying outliers, and correct
inconsistencies in the data. In this section, you
will study basic methods for data cleaning.
Missing Values
These are the methods in filling in the missing
value for the attributes:
1. Ignore the tuple This is usually done when
the class label is missing (assuming the mining
task involves classification).
2. Fill in the missing value manually In general,
this approach is time-consuming and may not
be feasible given a large data set with many
missing values.
3. Use a global constant to fill in the missing
value Replace all missing attribute values by the
same constant, such as a label like “Unknown”
or −∞.
4. Use the attribute mean to fill in the missing
value Use this value to replace the missing value
for income.
5. Use the attribute mean for all samples
belonging to the same class as the given tuple
For example, if classifying customers according
to credit risk, replace the missing value with the
average income value for customers in the
same credit risk category as that of the given
tuple.
6. Use the most probable value to fill in the
missing value This may be determined with
regression, inference-based tools using a
Bayesian formalism, or decision tree induction.
Data Cleaning as a Process
The first step in data cleaning as a process is
discrepancy detection. Discrepancies can be
caused by several factors, including poorly
designed data entry forms that have many
optional fields, human error in data entry,
deliberate errors (e.g., respondents not wanting
to divulge information about themselves), and
data decay (e.g., outdated addresses).
As a starting point, use any knowledge you may
already have regarding properties of the data.
Such knowledge or “data about data” is
referred to as metadata. Field overloading is
another source of errors that typically results
when developers squeeze new attribute
definitions into unused (bit) portions of already
defined attributes (e.g., using an unused bit of
an attribute whose value range uses only, say,
31 out of 32 bits).
The data should also be examined regarding
unique rules, consecutive rules, and null rules.
A unique rule says that each value of the given
attribute must be different from all other values
for that attribute.
A consecutive rule says that there can be no
missing values between the lowest and highest
values for the attribute, and that all values must
also be unique (e.g., as in check numbers).
A null rule specifies the use of blanks, question
marks, special characters, or other strings that
may indicate the null condition (e.g., where a
value for a given attribute is not available), and
how such values should be handled.
There are a number of different commercial
tools that can aid in the step of discrepancy
detection.
Data scrubbing tools use simple domain
knowledge (e.g., knowledge of postal addresses,
and spellchecking) to detect errors and make
corrections in the data. These tools rely on
parsing and fuzzy matching techniques when
cleaning data from multiple sources.
Data auditing tools find discrepancies by
analyzing the data to discover rules and
relationships, and detecting data that violate
such conditions. They are variants of data
mining tools. Commercial tools can assist in the
data transformation step.
Data migration tools allow simple
transformations to be specified, such as to
replace the string “gender” by “sex”.
ETL (extraction/transformation/loading) tools
allow users to specify transforms through a
graphical user interface (GUI). These tools
typically support only a restricted set of
transforms so that, often, we may also choose
to write custom scripts for this step of the data
cleaning process.
Data Integration
There are a number of issues to consider during
data integration. Schema integration and object
matching can be tricky. This is referred to as the
entity identification problem. Redundancy is
another important issue.
An attribute (such as annual revenue, for
instance) may be redundant if it can be
“derived” from another attribute or set of
attributes. Some redundancies can be detected
by correlation analysis. Given two attributes,
such analysis can measure how strongly one
attribute implies the other, based on the
available data.
Data Transformation
In data transformation, the data are
transformed or consolidated into forms
appropriate for mining. Data transformation can
involve the following:
• Smoothing, which works to remove noise
from the data. Such techniques include binning,
regression, and clustering.
• Aggregation, where summary or aggregation
operations are applied to the data. For example,
the daily sales data may be aggregated so as to
compute monthly and annual total amounts.
• Generalization of the data, where low-level or
“primitive” (raw) data are replaced by higher-
level concepts through the use of concept
hierarchies.
• Normalization, where the attribute data are
scaled so as to fall within a small specified
range, such as −1.0 to 1.0, or 0.0 to 1.0.
• Attribute construction (or feature
construction), where new attributes are
constructed and added from the given set of
attributes to help the mining process.
Data Reduction
Data reduction techniques can be applied to
obtain a reduced representation of the data set
that is much smaller in volume, yet closely
maintains the integrity of the original data. That
is, mining on the reduced data set should be
more efficient yet produce the same (or almost
the same) analytical results. Strategies for data
reduction include the following:
1. Data cube aggregation, where aggregation
operations are applied to the data in the
construction of a data cube.
2. Attribute subset selection, where irrelevant,
weakly relevant, or redundant attributes or
dimensions may be detected and removed.
3. Dimensionality reduction, where encoding
mechanisms are used to reduce the dataset
size.
4. Numerosity reduction, where the data are
replaced or estimated by alternative, smaller
data representations such as parametric models
(which need store only the model parameters
instead of the actual data) or nonparametric
methods such as clustering, sampling, and the
use of histograms.
5. Discretization and concept hierarchy
generation, where raw data values for
attributes are replaced by ranges or higher
conceptual levels. Data discretization is a form
of numerosity reduction that is very useful for
the automatic generation of concept
hierarchies.
SUBTOPIC 2 VARIABLE SELECTION What if you have 2k models to choose from?

Variable selection is intended to select the

“best” subset of predictors.

Why do we need to perform variable selection?

Why variable selection?

• We want to explain the data in the simplest

way — redundant predictors should be
removed.

• Unnecessary predictors will add noise to the

estimation of other quantities that we are Forward Selection
interested in. Degrees of freedom will be
wasted

• Collinearity is caused by having too many

variables trying to do the same job.

• Cost: if the model is to be used for prediction,

we can save time and/or money by not
measuring redundant predictors.

Prior to variable selection

1. Identify outliers and influential points -

maybe exclude them at least temporarily

2. Add in any transformations of the variables

that seem appropriate

Forward Selection

The forward selection procedure starts with no

variables in the model.

Steps Forward Selection

1. For the first variable to enter the model,

select the predictor most highly correlated with
the target.

2. For each remaining variable, compute the

sequential Fstatistic for that variable, given the
variables already in the model.

3. For the variable selected in step 2, test for

the significance of the sequential F-statistic. If Backward Selection
the resulting model is not significant, then stop,
This is the simplest of all variable selection
and report the current model without adding
procedures and can be easily implemented
the variable from step 2. Otherwise, add the
without special software. In situations where
variable from step 2 into the model and return
there is a complex hierarchy, backward
to step 2.
elimination can be run manually while taking
account of what variables are eligible for
removal.

Steps of Backward-selection

1. Perform the regression on the full model;

that is, using all available variables. For
example, perhaps the full model has four
variables, x1, x2, x3, x4.

2. For each variable in the current model,

compute the partial F-statistic. In the first pass
through the algorithm, these would be F(x1|x2, Scatter plots of MPG with each predictor. Some
x3, x4), F(x2|x1, x3, x4), F(x3|x1, x2, x4), and non-linearity
F(x4|x1, x2, x3). Select the variable with the
smallest partial F-statistic. Denote this value
Fmin.

3. Test for the significance of Fmin. If Fmin is

not significant, then remove the variable
associated with Fmin from the model, and
return to step 2. If Fmin is significant, then stop
the algorithm and report the current model

Scatter plots of ln MPG with each predictor

(including ln HP). Improved linearity.

Stepwise Procedure

The stepwise procedure represents a

modification of the forward selection
procedure. A variable that has been entered
into the model early in the forward selection
process may turn out to be nonsignificant, once
other variables have been entered into the
model.
The following can be done to treat
unsatisfactory response except: Returning to
the field

A good rule of thumb in having a right AGD

amount of data is to have 10 records for
every predictor value. True AGD
Given are the following records for the These outliers exist far outside the entirety
attribute rating. What is the problem of a data set. Global outliers
with the data? Data Inconsistency
These are also known as point anomalies.
Global outliers
application_rating
Anomaly detection can cause a bad user
experience. True
1
Anomaly detection is also known as outlier
analysis. True
2
It is a best practice to divide your dataset
into train and test dataset. True
a
It fits and performs variable selection on an
ordinary least square regression predictive
b
model. Linear regression selection

It is a useful tool for data reduction, such as

c
choosing the best variables or cluster
components for analysis. (Use lowercase for
3 your answer) variable clustering

The simplest of all variable selection

procedures is stepwise procedure. FALSE
Supply the missing value. AGD
Backward selection starts with all the
variables. True
Degree
It is about estimating the value for the target
variable except that the target variable is
SMBA
categorical rather than numeric. Prediction

WMA Prior to variable selection, one must identify

outliers and influential points - maybe
exclude them at least temporarily. True
AGD
Variable clustering is about grouping the
attributes with similarities. True
AGD
Resampling refers to the process of sampling
at random and with replacement from a data
WMA set. True

Estimation is about estimating the value for

WMA the target variable except that the target
variable is categorical rather than numeric.
- True
The figure below illustrates the first step Enumerate at least one of the two(2) types of
in doing backward selection. variables transformation commonly used in
machine learning

Data preparation affects:

False
 The objectives of the research
Forward selection is the opposite of  The quality of the research
stepwise selection. False
 The research approach
Formative 6  The sample size

Forward selection is the opposite of stepwise It is a manipulation of scale values to ensure

selection. comparability with other scales:

 True  Scale transformation

Estimation is about estimating the value for the
target variable except that the target variable is
categorical rather than numeric.
 True
Linearity is caused by having too many variables
trying to do the same job.
It is a best practice to divide your dataset into
train and test dataset.
 True
This happens when the deletion of unwanted
information causes desired information to be
deleted as well.

 False  Insertion anomaly

 Deletion anomaly
It is about estimating the value for the target
variable except that the target variable is The following are techniques to treat missing
categorical rather than numeric. values except:

 Estimation  Substitute an imputed value

 Returning to the field
It is the process of transforming the existing
 Substitute an imputed value
features into a lower-dimensional space,
 Case wise deletion
typically generating new features that are
composites of the existing features. Clustering can also detect outliers.

 Feature extraction  True

Postcoding process is necessary for: Supply the missing values given for the attribute
city.
 Structure questions
City
A good rule of thumb in having a right amount
of data is to have 10 records for every predictor Makati
value.
Caloocan
 True
Caloocan
Forward selection is the simplest variable
Makati
selection model.
Caloocan
 False
 Caloocan
  Makati Age
 Quezon
16
 None of the choices
27
These anomalies have values that significantly
deviate from the other data points that exist in -8990
the same context.
19
 Contextual outliers
15
When there’s a missing value for a categorical
18
variable, it is ideal to supply it by computing for
the average of the data values available. Answer:Data Inconsistency
 True Anomaly detection can cause a bad user
experience.
Outlier analysis can provide good product
quality.  False
 True You can use histogram to detect outliers.
A review of the questionnaires is essential in  True
order to:
When a subset points within a set is anomalous
 Select the data analysis strategy to the entire data set, those values are:
 Find new insights
Collective outliers
 Increase the quality of the data
 Increase accuracy and precise of the These anomalies have values that significantly
collected data deviate from the other data points that exist in
the same context.
Feature selection maps the original feature
space to a new feature space with lower  Contextual outliers
dimensions by combining the original feature
space. These are problems that can occur in poorly
planed, un-normalized databases where all the
True data is stored in one table (a flat-file database).
False  Anomalies
This happens when inserting vital data into the
database is not possible because other data is
not already there.

 Insertion anomaly

Unnecessary predictors will add noise to the

estimation of other quantities that we are
interested in.
16/20
 True
A homogenous data set is a data set whose
You can also use regression when handling
data records have the same target value.
noisy data.
True
True
Supply for the missing values.
Given the following values for age, what is the
problem with the data?
 This happens when the deletion of
unwanted information causes desired
information to be deleted as well.
Deletion anomaly

Instead of using the real number for age

attribute, you categorized the age as the
following:

Young = 12 – 17
Adult   = 18 -34
Old     = 35 – 60
What kind of data preparation was
practiced? Data Cleaning
It is the process of integrating multiple
- 19.6 databases, data cubes, or files.  data
It is a manipulation of scale values to ensure integration
comparability with variables with other
scales:
Scale transformation

Supply the missing value in the given data

below. These are problems that can occur in
poorly planned, un-normalized databases
where all the data is stored in one table
(a flat-file database). Anomalies
You can also use regression when
handling noisy data. True
The procedure starts with an empty set
of features [reduced set]. Forward
Selection

It is the simplest of all variable selection

procedures and can be easily
implemented without special software
(Use lowercase for your answer)
Backward Selection
The forward selection procedure starts with
no variables in the model. True

- 88.2 Estimation is about estimating the value

for the target variable except that the
target variable is categorical rather than
It is a best practice to divide your numeric.
dataset into train and test dataset. True
True
The figure below illustrates the first step
in doing backward selection. False
It is intended to select the “best” subset
of predictors.  (Use lowercase for your
answer)
Variables Selection

Enumerate at least one of the two (2)

types of variables transformation
commonly used in machine learning:
(Use lowercase for your answer)
categorical variables?
numerical variables?
Forward selection is the opposite of
stepwise selection. False
The figure below illustrates the basic steps
for what type of variable selection method?

Backward

Prior to variable selection, one must

identify outliers and influential points -
maybe exclude them at least
temporarily. True