Professional Documents
Culture Documents
Nisheeth
Distinctions Between Parameters and
Statistics
Parameters Statistics
Vary No Yes
Calculated No Yes
The Bayesian’s universe
Generates
World Data
Model
Magic
Prior Lklhd
Inference Inference
Params
The statistician’s universe
Sample
Population Data
Statistics
Stats
Approximate
Params
“The Reverend Bayes published posthumously. The field of statistics
would greatly improve if all Bayesians were to follow his example”
Sampling Distributions of a Mean
x ~ N , SE x
where SE x
n
Hypothesis Testing
• Is also called significance testing
• Tests a claim about a parameter using
evidence (data in a sample
• The technique is introduced by considering a
one-sample z test
• The procedure is broken into four steps
• Each element of the procedure must be
understood
Hypothesis Testing Steps
A. Null and alternative hypotheses
B. Test statistic
C. P-value and interpretation
D. Significance level (optional)
Null and Alternative Hypotheses
• Convert the research question to null and
alternative hypotheses
• The null hypothesis (H0) is a claim of “no
difference in the population”
• The alternative hypothesis (Ha) claims “H0 is
false”
• Collect data and seek evidence against H0 as a
way of bolstering Ha (deduction)
Illustrative Example: “Body Weight”
Reasoning
x ~ N 170,5
Sampling distribution of xbar
under H0: µ = 170 for n = 64
Test Statistic
This is an example of a one-sample test of a
mean when σ is known. Use this statistic to
test the problem:
x 0
z stat
SE x
where 0 population mean assuming H 0 is true
and SE x
n
Illustrative Example: z statistic
• For the illustrative example, μ0 = 170
• We know σ = 40
• Take a random sample of n = 64. Therefore
40
SE x 5
n 64
• If we found a sample mean of 173, then
x 0 173 170
zstat 0.60
SE x 5
Illustrative Example: z statistic
If we found a sample mean of 185, then
x 0 185 170
zstat 3.00
SE x 5
What is a test statistic?
𝑧
1 𝑧2
𝑝 𝑥 ≤ 𝑥 |𝑁 𝑥|𝜇, 𝜎 2 = 𝑝 𝑧 ≤ 𝑧|𝑁(𝑧|0,1) = exp − 𝑑𝑧
2𝜋 2
−∞
p-value
• The P-value answer the question: What is the
probability of the observed test statistic or one more
extreme when H0 is true?
• This corresponds to the AUC in the tail of the
Standard Normal distribution beyond the zstat.
• Convert z statistics to P-value :
For Ha: μ > μ0 P = Pr(Z > zstat) = right-tail beyond zstat
For Ha: μ < μ0 P = Pr(Z < zstat) = left tail beyond zstat
For Ha: μ μ0 P = 2 × one-tailed P-value
One-sided P-value for zstat of 0.6
One-sided P-value for zstat of 3.0
Two-Sided P-Value
• One-sided Ha
AUC in tail beyond
zstat
• Two-sided Ha
consider potential
deviations in both Examples: If one-sided P
directions = 0.0010, then two-sided
double the one- P = 2 × 0.0010 = 0.0020.
sided P-value If one-sided P = 0.2743,
then two-sided P = 2 ×
0.2743 = 0.5486.
Interpretation
• P-value answer the question: What is the
probability of the observed test statistic …
when H0 is true?
• Thus, smaller and smaller P-values provide
stronger and stronger evidence against H0
• Small P-value strong evidence
α-Level (Used in some situations)
• Let α ≡ probability of
erroneously rejecting H0
• Set α threshold (e.g., let α = .10,
.05, or whatever)
• Reject H0 when P ≤ α
• Retain H0 when P > α
• Example: Set α = .10. Find P =
0.27 retain H0
• Example: Set α = .01. Find P =
.001 reject H0
(Summary) One-Sample z Test
A. Hypothesis statements
H0: µ = µ0 vs.
Ha: µ ≠ µ0 (two-sided) or
Ha: µ < µ0 (left-sided) or
Ha: µ > µ0 (right-sided)
B. Test statistic
x 0
z stat where SE x
SE x n
C. P-value: convert z to p value
D. Significance statement depending on α
Another example: IQ testing
• Let X represent Weschler Adult Intelligence
scores (WAIS)
• Typically, X ~ N(100, 15)
• Take i.i.d. samples of n = 9 from a population
• Data {116, 128, 125, 119, 89, 99, 105, 116,
118}
• Calculate sample mean = 112.8
• Does sample mean provide strong evidence that
population mean μ > 100?
Example
A. Hypotheses:
H0: µ = 100 versus
Ha: µ > 100 (one-sided)
Ha: µ ≠ 100 (two-sided)
B. Test statistic:
15
SE x 5
n 9
x 0 112.8 100
zstat 2.56
SE x 5
C. P-value: P = Pr(Z ≥ 2.56) = 0.0052
𝑥− 𝜇 𝑛
𝑇𝑛−1 (𝑥 ) = 2
1 2
𝑆 𝑆 = 𝑥𝑖 − 𝑥
𝑛 𝑛−1
𝑖=1
Can you say why?
(Summary) T Test
A. Hypothesis statements
H0: µ = µ0 vs.
Ha: µ ≠ µ0 (two-sided) or
Ha: µ < µ0 (left-sided) or
Ha: µ > µ0 (right-sided)
B. Calculate Test statistic
𝑛
𝑥− 𝜇 1
𝑇𝑛−1 (𝑥 ) = 𝑆2 = 𝑥𝑖 − 𝑥 2
𝑆 where,
𝑛−1
𝑛 𝑖=1
Pooled
Unpooled
Test statistic construction
• Proportions as random variables
• What does the sampling distribution of the
mean look like?
• What will the test statistic look like?
Different set of tests for variance
• Sampling distribution for variance looks very
different
• Hypothesis testing logic stays the same
– Hypothesize
– Calculate
– Interpret
(Summary) Statistical hypothesis
testing
A. Hypothesis statements
H0: µ = µ0 vs.
Ha: µ ≠ µ0 (two-sided) or
Ha: µ < µ0 (left-sided) or
Ha: µ > µ0 (right-sided)
B. Calculate Test statistic
C. P-value: convert test statistic to p value
D. Significance statement depending on α
PRACTICAL CONSIDERATIONS
Multiple comparisons
• Let’s say we’re doing IQ testing department-
wise
– 7 departments
– α = 0.05
– Run C(7,2) = 21 pairwise T-tests on the data
– CSE > EE
• How to interpret?
• Bonferroni correction for FWER α* =
α/#comparisons
P-hacking
Reading error bars
Reading error bars
Truth
Decision H0 true H0 false
Retain H0 Correct retention Type II error
Reject H0 Type I error Correct rejection
40
𝑥𝑐𝑟𝑖𝑡 = 170 + 1.96 × = 189.6
16
Z statistic
• Very important
– Science experiments
– Costly data collection
p-value
Review: statistics
• The language of statistics
– Describes a universe where we sample datasets from
a population
• Interesting properties are proved for sampling
distributions of parameter estimates
• Statistical hypothesis testing
– Helps us decide if a sample belongs to a population
• A priori calculation of important statistical
properties can help design better studies
– Power, sample size, effect size