You are on page 1of 100

Unit 14: Nonparametric

Statistical Methods

Statistics 571: Statistical Methods


Ramón V. León

7/26/2004 Unit 14 - Stat 571 - Ramón V. León 1


Introductory Remarks
• Most methods studied so far have been based on
the assumption of normally distributed data
– Frequently this assumption is not valid
– Sample size may be too small to verify it
• Sometimes the data is measured in an ordinal scale
• Nonparametric or distribution-free statistical
methods
– Make very few assumptions about the form of the
population distribution from which the data are sampled
– Based on ranks so they can be used on ordinal data
• Will concentrate on hypothesis tests but will also
mention confidence interval procedures.
7/26/2004 Unit 14 - Stat 571 - Ramón V. León 2
Inference for a Single Sample
Consider a random sample x1 , x2 ,..., xn from a population
with unknown median µ .
(Recall that for nonnormal (especially skewed)
distributions the median is a better measure of the center
than the mean.)

H 0 : µ = µ0 vs. H1 : µ > µ0
Example: Test whether the median household income of a
population exceeds $50,000 based on a random sample of
household incomes from that population
For simplicity we sometimes present methods for one-sided tests.
Modifications for two-sided tests are straightforward and are given in the
textbook Some examples in these notes are two-sided tests.
7/26/2004 Unit 14 - Stat 571 - Ramón V. León 3
Sign Test for a Single Sample
H 0 : µ = µ0 vs. H1 : µ > µ0
Sign test:
1. Count the number of xi 's that exceed µ0 . Denote this
number by s+ , called the number of plus signs. Let
s− = n − s+ , which is the number of minus signs.
2. Reject H 0 if s+ is large or equivalently if s− is small.
Test idea:
Under the null hypothesis s+ has a binomial distribution,
Bin (n, ½). So this test is simply the test for binomial proportions
7/26/2004 Unit 14 - Stat 571 - Ramón V. León 4
Sign Test Example
A thermostat used in an electric device is to be checked for the
accuracy of its design setting of 200ºF. Ten thermostats were
tested to determine their actual settings, resulting in the
following data:

202.2, 203.4, 200.5, 202.5, 206.3, 198.0, 203.7, 200.8, 201.3, 199.0

H 0 : µ = 200 vs H1 : µ ≠ 200
s+ = 8 = number of data values > 200, so
10 10
  1 
1010 2
  1 
10
P-value = 2∑     = 2∑     = 0.110
i =8  i   2  i =0  i   2 

(The t test based on the mean has P-value = 0.0453. However recall that
the t test assumes a normal population)
7/26/2004 Unit 14 - Stat 571 - Ramón V. León 5
Normal Approximation to Test Statistic
If the sample size is large ( ≥ 20) the common of S+ and S− is
approximated by a normal distribution with
1 n
E ( S + ) = E ( S− ) = np = n   = ,
2 2
 1  1  n
Var ( S+ ) = Var ( S − ) = np(1 − p) = n     =
 2  2  4
Therefore can perform a one-sided z - test with
s − n 2 −1 2
z= +
n4

7/26/2004 Unit 14 - Stat 571 - Ramón V. León 6


P-values for Sign
Test Using JMP

Based on normal approximation


to the binomial ( = z2 )
7/26/2004 Unit 14 - Stat 571 - Ramón V. León 7
Treatment of Ties
• Theory of the test assumes that the distribution of
the data is continuous so in theory ties are
impossible
• In practice they do occur because of rounding
• A simple solution is to ignore the ties and work
only with the untied observation. This does reduce
the effective sample size of the test and hence its
power, but the loss is not significant if there are
only a few ties

7/26/2004 Unit 14 - Stat 571 - Ramón V. León 8


Comfidence Interval for µ
Let x(1) ≤ x(2) ≤ ⋅⋅⋅ ≤ x( n ) be the ordered data values.
Then a (1-α )-level CI for µ is given by
x(b +1) ≤ µ ≤ x( n −b )
where b = bn ,1−α 2 is the lower α 2 critical point
of the Bin ( n,1 2 ) distribution.

Note: Not all confidence levels are possible because


of the discreteness of the Binomial distribution

7/26/2004 Unit 14 - Stat 571 - Ramón V. León 9


Thermostat Setting: Sign Confidence
Interval for the Median
From Table A.1 we see that for n = 10 and p=0.5,
the lower 0.011 critical point of the binomial
distribution is 1 and by symmetry the upper 0.011
critical point is 9.
Setting α 2 = 0.011 which gives 1-α = 1 − 0.022 = 0.978,
we find that
199.0 = x(2) ≤ µ ≤ x(9) = 203.7
is a 97.8% CI for µ .
7/26/2004 Unit 14 - Stat 571 - Ramón V. León 10
Sign Test for Matched Pairs

Drop 3 tied pairs. Then s+ = 20; s- = 3


7/26/2004 Unit 14 - Stat 571 - Ramón V. León 11
Sign Test for Matched Pairs

7/26/2004 Unit 14 - Stat 571 - Ramón V. León 12


Sign Test for Matched Pairs in JMP
Pearson’s p-value is not the same
as the book’s two-sided P-value
because the book uses the
continuity correction in the
normal approximation to the
binomial distribution,
i.e, book uses z = 3.336 (Page
567) rather than z = 3.544745
used by JMP. Note that
(3.544745)2 = 12.5652

book
7/26/2004 Unit 14 - Stat 571 - Ramón V. León 13
Wilcoxon Signed Rank Test
H 0 : µ = µ0 vs. H1 : µ ≠ µ0
More powerful than the sign test, however, it requires the
assumption that the population distribution is symmetric
1. Rank and order the differences in terms of their absolute value
Example 14.1 and 14.4: Thermostat Setting is 200° F

2. Calculate w+ = sum of the ranks of the positive differences


w+ = 6 + 8 + 1 + 7 + 10 + 9 + 2 + 4
3. Reject H0 if w+ is large or small
7/26/2004 Unit 14 - Stat 571 - Ramón V. León 14
Wilcoxon Signed Rank Test in JMP

This test finds a


significant
difference at
α=0.05 while the
sign test did not
at even α=0.1
7/26/2004 Unit 14 - Stat 571 - Ramón V. León 15
Normal Approximation in the Wilcoxon
Signed Rank Test
For large n, the null distribution of W ∼ W+ ∼ W-
can be well-approximated by a normal distribution
with mean and variance given by
n(n + 1) n(n + 1)(2n + 1)
E (W ) = and Var (W ) = .
4 24
For large samples a one-sided (greater than median)
z-test uses the statistic
w+ − n(n + 1) / 4 − 1/ 2
z=
n(n + 1)(2n + 1) 24

7/26/2004 Unit 14 - Stat 571 - Ramón V. León 16


Importance of Symmetric Population
Assumption

Here even though H0 is true the long right hand tail makes the positive
differences tend to be larger in magnitude than the negative differences,
resulting in higher ranks. This inflates w+ and hence the test’s type I error
probability.
7/26/2004 Unit 14 - Stat 571 - Ramón V. León 17
Null
Distribution of
the Wilcoxon
Signed Rank
Statistics

7/26/2004 Unit 14 - Stat 571 - Ramón V. León 18


Null Distribution of the
Wilcoxon Signed Rank Statistics

7/26/2004 Unit 14 - Stat 571 - Ramón V. León 19


Wilcoxon Signed Rank Statistic:
Treatment of Ties
• There are two types of ties
– Some of the data is equal to the median
• Drop these observations
– Some of the differences from the median
may be tied
• Use midrank, that is, the average rank
For example, suppose
d1 = −1, d 2 = +3, d3 = −3, d 4 = +5 With ties Table A.10
Then is only approximate
(2 + 3)
r1 = 1, r2 = r3 = = 2.5, r4 = 4
2
7/26/2004 Unit 14 - Stat 571 - Ramón V. León 20
Wilcoxon Sign Rank Test:
Matched Pair Design Notice that we drop the
three zero differences
Example 14.5: Comparing Two
Methods of Cardiac Output Notice that we average the
tied ranks

Two-Side P-values
Signed test: 0.0008
Signed Rank test: 0.0002
t-test: 0.0000671 (Page 284)

(Notice that these tests require


progressively more stringent
assumptions about the
population of differences)

7/26/2004 Unit 14 - Stat 571 - Ramón V. León 21


JMP Calculation

7/26/2004 Unit 14 - Stat 571 - Ramón V. León 22


Signed Rank Confidence Interval for the Median

7/26/2004 Unit 14 - Stat 571 - Ramón V. León 23


Thermostat Setting: Wilcoxon Signed
Rank Confidence Interval for Median
From Table A.10 we see that for n = 10, the upper 2.4%
critical point is 47 and by symmetry the lower 2.4%
10(10 + 1)
critical point is - 47 = 55 - 47 = 8.
2
Setting α 2 = 0.024 and hence 1-α=1-0.048=0.952
we find that
+200.10 = x9 = x8+1 ≤ µ ≤ x47 = 203.55
is a 95.2% CI for µ

7/26/2004 Unit 14 - Stat 571 - Ramón V. León 24


Inferences for Two Independent Samples
One wants to show that the observations from one population
tend to be larger than those from another population based on
independent random samples

x1 , x2 ,..., xn1 and y1 , y2 ,..., yn2

Examples:
•Treated patients tend to live longer than untreated patients
•An equity fund tends to have a higher yield than a bond fund

7/26/2004 Unit 14 - Stat 571 - Ramón V. León 25


Wilcoxon-Mann-Whitney Test Example:
Time to Failure of Two Capacitor Groups

Reject for extreme values of w1.


7/26/2004 Unit 14 - Stat 571 - Ramón V. León 26
Stochastic Ordering of Populations
X is stochastically larger than Y ( X Y )
if for all real numbers u ,
P ( X > u ) ≥ P(Y > u )
equivalently,
P ( X ≤ u ) = F1 (u ) ≤ F2 (u ) = P(Y ≤ u )
with strict inequality for at least some u.
Denoted by X Y or equivalently by F1 < F2 )

7/26/2004 Unit 14 - Stat 571 - Ramón V. León 27


Stochastic Ordering Especial Case:
Location Difference

θ is called a location parameter


Notice that X 2 ≺ X 1 iff θ 2 < θ1
7/26/2004 Unit 14 - Stat 571 - Ramón V. León 28
Wilcoxon-Mann-Whitney Test
H 0 : F1 = F2 ( X ∼ Y )
Alternatives :
One sided: H1 : F1 < F2 ( X Y)
Two sided: H1 : F1 < F2 or F2 < F1 ( X Y or Y X)

Notice that the alternative is not H1 : F1 ≠ F2


(Kolmogorov-Smirnov Test can handle this alternative)

7/26/2004 Unit 14 - Stat 571 - Ramón V. León 29


Wilcoxon Version of the Test
H 0 : F1 = F2 ( X ∼ Y ) vs. H1 : F1 < F2 ( X Y)
1. Rank all N = n1 + n2 observations,
x1 , x2 ,..., xn1 and y1 , y2 ,..., yn2
in ascending order
2. Sum the ranks of the x's and y's separately.
Denote these sums by w1 and w2
3. Reject H 0 if w1 is large or equivalently w2 is small

7/26/2004 Unit 14 - Stat 571 - Ramón V. León 30


Mann-Whitney Test Version

The advantage of using the Mann-Whitney form of the test is that


the same distribution applies whether we use u1 or u2

P − value = P(U ≥ u1 ) = P(U ≤ u2 )

7/26/2004 Unit 14 - Stat 571 - Ramón V. León 31


Null Distribution of the Wilcoxon-
Mann-Whitney Test Statistic

Under the null hypothesis each of these 10 ordering has an


equal chance of occurring, namely, 1/10
5
  = 10
 2
7/26/2004 Unit 14 - Stat 571 - Ramón V. León 32
Null Distribution of the Wilcoxon-
Mann-Whitney Test Statistic

P ( w1 ≥ 8) = 0.1 + 0.1 = 0.2 (one-sided p-value for w1 = 8)


( H1 : X Y)
7/26/2004 Unit 14 - Stat 571 - Ramón V. León 33
Normal Approximation of Mann-
Whitney Statistic
For large n1 and n2 , the null distribution of U can be
well approximated by a normal distribution with
mean and variance given by
n1n2 n1n2 ( N + 1)
E (U ) = and Var (U ) =
2 12
A large sample one-sided z - test can be based on the statistic
u1 − n1n2 2 − 1 2
z=
n1n2 ( N + 1) ( H1 : X Y)
12

7/26/2004 Unit 14 - Stat 571 - Ramón V. León 34


Treatment of Ties
A tie occurs when some x equal a y.
– A contribution of ½ is counted towards both u1
and u2 for each tied pair
– Equivalent to using the midrank method in
computing the Wilcoxon rank sum statistic

7/26/2004 Unit 14 - Stat 571 - Ramón V. León 35


Wilcoxon-Mann-Whitney Confidence Interval

Example14.8 shows that [d(18) , d(63) ] = [-1.1, 14.7] is a 95.6% CI


for the difference of the two medians of the failure times of capacitors.
This example is in the book errata since Table A.11 is not detailed enough.
7/26/2004 Unit 14 - Stat 571 - Ramón V. León 36
Wilcoxon-Mann-Whitney Test in JMP

With
continuity
correction.
Used in
the book
which gets
a one-
sided p-
value of
0.0502

Without
continuity
correction
7/26/2004 z2=1.6882 Unit 14 - Stat 571 - Ramón V. León 37
Inference for Several Independent Samples:
Kruskal-Wallis Test

Note that this


is a completely
randomized
design

7/26/2004 Unit 14 - Stat 571 - Ramón V. León 38


Kruskal-Wallis Test
H 0 : F1 = F2 = ⋅⋅⋅ = Fa vs. H1 : Fi < Fj for some i ≠ j

Distance from the


Reject H 0 if kw > χ 2
a −1,α average rank
7/26/2004 Unit 14 - Stat 571 - Ramón V. León 39
Chi-Square Approximation
• For large samples the distribution of KW under the
null hypothesis can be approximated by the chi-
square distribution with a-1 degrees of freedom
• So reject H0 if

kw > χ a −1,α

7/26/2004 Unit 14 - Stat 571 - Ramón V. León 40


Kruskal-Wallis Test Example

Reject if
kw is large.
χ 3,.005
2
= 12.837
7/26/2004 Unit 14 - Stat 571 - Ramón V. León 41
Kruskal-Wallis Test in JMP

7/26/2004 Unit 14 - Stat 571 - Ramón V. León 42


7/26/2004 Unit 14 - Stat 571 - Ramón V. León 43
•Case method is different from Unitary method
•Formula method is different from Unitary method

7/26/2004 Unit 14 - Stat 571 - Ramón V. León 44


Pairwise Comparisons:
Is Any Pair of Treatments Different?

• One can use the Tukey Method on the average ranks to make
approximate pairwise comparisons.

• This is one of many approximate techniques where ranks are


substituted for the observations in the normal theory methods.

7/26/2004 Unit 14 - Stat 571 - Ramón V. León 45


7/26/2004 Unit 14 - Stat 571 - Ramón V. León 46
7/26/2004 Unit 14 - Stat 571 - Ramón V. León 47
Tukey’s Test Applied to the Ranks Averaged

Lack of
agreement
with the more
precise method
of Example
14.10. Here
Equation
method also
seems to be
different from
Formula and
Case method

7/26/2004 Unit 14 - Stat 571 - Ramón V. León 48


Example of Friedman’s Test

Ranking is done within blocks

χ 7,.025
2
= 16.012 - P-value =.0040 vs. .0003 for ANOVA table
7/26/2004 Unit 14 - Stat 571 - Ramón V. León 49
Inference for Several Matched Samples
Randomized Block Design:
i a ≥ 2 treatment groups
i b ≥ 2 blocks
i yij = observation on the i-th treatment in the j-th block
i Fij = c.d.f of r.v. Yij corresponding to the observed value yij
For simplicity assume Fij ( y ) = F ( y − θ i − β j )
iθ i is the "treatment effect"
i β j is the "block effect"
i.e., we assume that there is no
treatment by block interaction

7/26/2004 Unit 14 - Stat 571 - Ramón V. León 50


Friedman Test
H 0 : θ1 = θ 2 = ⋅⋅⋅ = θ a vs. H1 : θ i > θ j for some i ≠ j

Distance from the total


Reject if fr > χ 2
a −1,α of the ranks from their
expected value when
there is no agreement
between the blocks
7/26/2004 Unit 14 - Stat 571 - Ramón V. León 51
Pairwise Comparisons

7/26/2004 Unit 14 - Stat 571 - Ramón V. León 52


Rank Correlation Methods
• The Pearson correlation coefficient
– measures only the degree of linear association between
two variables
– Inferences use the assumption of bivariate normality of
the two variables
• We present two correlation coefficients that
– Take into account only the ranks of the observations
– Measure the degree of monotonic (increasing or
decreasing) association between two variables

7/26/2004 Unit 14 - Stat 571 - Ramón V. León 53


Motivating Example
( x, y ) = (1, e1 ), (2, e 2 ), (3, e3 ), (4, e 4 ), (5, e5 )

Note that there is a perfect positive association between between


x and y with y = ex.

•The Pearson correlation correlation coefficient is only 0.886


because the relationship is not linear

•The rank correlation coefficients we present yield a value of 1


for these data

7/26/2004 Unit 14 - Stat 571 - Ramón V. León 54


Spearman’s Rank Correlation Coefficient

Ranges between –1 and +1 with rs = -1 when there is a perfect negative


association and rs = +1 when there is a perfect positive association

7/26/2004 Unit 14 - Stat 571 - Ramón V. León 55


Example 14.12 (Wine Consumption and
Heart Disease Deaths per 100,000

7/26/2004 Unit 14 - Stat 571 - Ramón V. León 56


7/26/2004 Unit 14 - Stat 571 - Ramón V. León 57
Calculation of Spearman’s Rho

7/26/2004 Unit 14 - Stat 571 - Ramón V. León 58


Test for Association Based on Spearman’s Rank
Correlation Coefficient

7/26/2004 Unit 14 - Stat 571 - Ramón V. León 59


Hypothesis Testing Example
H 0 : X = Wine Consumption and
Y = Heart Disease Deaths are independent.
vs.
H1 : X and Y are (negatively or positively) associated

z = rS n − 1 = −0.826 19 − 1 = −3.504
Two-Sided P − value = 0.0004

Evidence of negative association


7/26/2004 Unit 14 - Stat 571 - Ramón V. León 60
JMP Calculations:
Pearson Correlation
350

300

Heart Disease Deaths


250

200

150

100

50
0 2 4 6 8 10
Alcohol from Wine

Plot is fairly linear


Pearson correlation

7/26/2004 Unit 14 - Stat 571 - Ramón V. León 61


JMP Calculations: Spearman Rank Correlation

7/26/2004 Unit 14 - Stat 571 - Ramón V. León 62


Kendall’s Rank Correlation Coefficient:
Key Concept Examples
Concordant pairs:
(1,2), (4,9) (1 - 4)(2 - 9)>0 Kendall’s
(4,2), (3,1) (4 - 3)(2 - 1)>0 idea is to
compare the
Discordant pairs: number of
(1,2), (9,1) (1 - 9)(2 - 1)<0 concordant
(2,4), (3,1) (2 - 3)(4 - 1)<0 pairs to the
number of
Tied pairs: discordant
(1,3), (1,5) (1 – 1)(3 – 5)=0 pairs in
(1,4), (2,4) (1 – 2)(4 – 4)=0 bivariate data
(1,2), (1,2) (1 – 1)(2 – 2)=0
7/26/2004 Unit 14 - Stat 571 - Ramón V. León 63
(X, Y)
(1, 2) Kendall’s Tau
(3, 4) Example
(2, 1)
n  3
Number of pairwise comparisons =   =   = 3 = N
 2   2
Concordant pairs: Discordant pairs: Nc − Nd
τˆ =
(1,2) (3,4) (1,2) (2,1) N
(3,4) (2,1) 2 −1
=
3
Nc = 2 Nd=1 1
=
3
7/26/2004 Unit 14 - Stat 571 - Ramón V. León 64
Kendall’s Rank Correlation Coefficient:
Population Version

7/26/2004 Unit 14 - Stat 571 - Ramón V. León 65


Kendall’s Rank Correlation Coefficient:
Sample Estimate
Let N c = Number of concordant pairs in the data
Let N d = Number of disconcordant pairs in the data
n
Let N =   be the number of pairwise comparisons among
2
the observations ( xi , yi ), i = 1, 2,..., n. Then
Nc − Nd
τˆ = and N c + N d = N if no ties
N
Nc − Nd
τ=
ˆ if ties
( N − Tx )( N − Ty )
where Tx and Ty are corrections for the number of tied pairs.

7/26/2004 Unit 14 - Stat 571 - Ramón V. León 66


Hypothesis of Independence Versus Positive Association

Wine data:
-.696 -4.164

7/26/2004 Unit 14 - Stat 571 - Ramón V. León 67


JMP
Calculations:
Kendall’s
Rank
Correlation
Coefficient

7/26/2004 Unit 14 - Stat 571 - Ramón V. León 68


Kendall’s Coefficient of Concordance
• Measure of association between several matched samples
• Closely related to Friedman’s test statistic
– Consider a candidates (treatments) and b judges (blocks) with each
judge ranking the a candidates
• If there is perfect agreement between the judges, then each
candidate gets the same rank. Assuming the candidates are
labeled in the order of their ranking, the rank sum for the ith
candidate would be ri= ib
• If the judges rank the candidates completely at random
(“perfect disagreement”) then the expected rank of each
candidate would be [1+2+…+a]/a =[a(a+1)/2]/a=(a+1)/2,
and the expected value of all the rank sums would equal to
b(a+1)/2

7/26/2004 Unit 14 - Stat 571 - Ramón V. León 69


Kendall’s Coefficient of Concordance

7/26/2004 Unit 14 - Stat 571 - Ramón V. León 70


Kendall’s Coefficient of Concordance
and Friedman’s Test

7/26/2004 Unit 14 - Stat 571 - Ramón V. León 71


24.667
w= = 0.881
4(8 − 1)

7/26/2004 Unit 14 - Stat 571 - Ramón V. León 72


Do You Need to Know More
“Nonparametric Statistical Methods, Second Edition”
by Myles Hollander and Douglas A. Wolfe. (1999)
Wiley-Interscience

7/26/2004 Unit 14 - Stat 571 - Ramón V. León 73


Resampling Methods
• Conventional methods are based on the sampling
distribution of a statistic computed for the
observed sample. The sampling distribution is
derived by considering all possible samples of size
n from the underlying population.
• Resampling methods generate the sampling
distribution of the statistic by drawing repeated
samples from the observed sample itself. This
eliminates the need to assume a specific functional
form for the population distribution (e.g. normal).

7/26/2004 Unit 14 - Stat 571 - Ramón V. León 74


Challenger Shuttle O-Ring Data

Do we have statistical evidence that cold temperature leads to


more O-ring incidents?
•Notice that assumptions of two sample t test do not hold.
•Original analysis omitted the zeros? Was this justified?
•What do we do?
7/26/2004 Unit 14 - Stat 571 - Ramón V. León 75
Wrong t-test Analysis
Difference of Low
mean to High mean

Notice that the assumptions of the


independent sample t-test do not hold,
i.e., data is not normal for each group.
7/26/2004 Unit 14 - Stat 571 - Ramón V. León 76
Permutation Distribution of t Statistic

Also equal to the two-sided p-value

Equivalent to selecting all simple random samples without replacement of


size 20 from the 24 data points, labeling these High and the rest Low
7/26/2004 Unit 14 - Stat 571 - Ramón V. León 77
Comments
• A randomization test is a permutation test applied
to data from a randomized experiment.
Randomization tests are the gold standard for
establishing causality.
• A permutation test considers all possible simple
random samples without replacement from the set
of observed data values
• The bootstrap method considers a large number of
simple random samples with replacement from
the set of observed data values.

7/26/2004 Unit 14 - Stat 571 - Ramón V. León 78


Calculation of t Statistics from 10,000
Bootstrap Samples
Think that we are placing the 24
Challenger data values in a hat.
And that we are randomly selecting
24 values with replacement from
the hat, labeling the first 20 values
High and the remaining 4 values
Low. We repeat these process
10,000 times. For each of these
10,000 bootstrap samples we
calculate the t-statistic. 35 t-
statistics values were greater than
-2 -1 0 1 2 3 4 5 6
or equal to 3.888 out of 10000 (if sp
= 0, t is defined to be 0). This gives
a bootstrap P-value of 35/10000 =
0.0035
7/26/2004 Unit 14 - Stat 571 - Ramón V. León 79
Bootstrap Distribution of Difference
Between the Means
67 of the 10,000 differences
of the Low mean and the
High mean were greater than
or equal to 1.3. This gives a
bootstrap P-value of
67/10000 = .0067

Conclusion: Cold
weather increases the
-1 0 1 2 chance of O-ring
problems

7/26/2004 Unit 14 - Stat 571 - Ramón V. León 80


Bootstrap Final Remarks
• The JMP files - that we used to generate the
bootstrap samples and to calculate the statistics -
are available at the course web site.
• There are bootstrap procedures for most types of
statistical problems. All are based on resampling
from the data.
• These methods do not assume specific functional
forms for the distribution of the data, e.g. normal
• The accuracy of bootstrap procedures depend on
the sample size and the number of bootstrap
samples generated

7/26/2004 Unit 14 - Stat 571 - Ramón V. León 81


How Were
the
Bootstrap
Samples
Generated?

(see next page)

7/26/2004 Unit 14 - Stat 571 - Ramón V. León 82


7/26/2004 Unit 14 - Stat 571 - Ramón V. León 83
7/26/2004 Unit 14 - Stat 571 - Ramón V. León 84
7/26/2004 Unit 14 - Stat 571 - Ramón V. León 85
7/26/2004 Unit 14 - Stat 571 - Ramón V. León 86
Calculated Columns in JMP Samples File

7/26/2004 Unit 14 - Stat 571 - Ramón V. León 87


7/26/2004 Unit 14 - Stat 571 - Ramón V. León 88
7/26/2004 Unit 14 - Stat 571 - Ramón V. León 89
7/26/2004 Unit 14 - Stat 571 - Ramón V. León 90
7/26/2004 Unit 14 - Stat 571 - Ramón V. León 91
7/26/2004 Unit 14 - Stat 571 - Ramón V. León 92
7/26/2004 Unit 14 - Stat 571 - Ramón V. León 93
7/26/2004 Unit 14 - Stat 571 - Ramón V. León 94
Bootstrap Estimate of the Standard Error of the Mean

Summary: We calculate the standard deviation of the N


bootstrap estimates of the mean
7/26/2004 Unit 14 - Stat 571 - Ramón V. León 95
BSE for Arbitrary Statistic

Example: The bootstrap standard error of the median is calculated by


drawing a large number N, e.g. 10000, of bootstrap samples from the
data. For each bootstrap sample we calculated the sample median.
Then we calculate the standard deviation of the N bootstrap medians.

7/26/2004 Unit 14 - Stat 571 - Ramón V. León 96


Estimated Bootstrap Standard Error for t-
statistics Using JMP

Note N =10,000
7/26/2004 Unit 14 - Stat 571 - Ramón V. León 97
Bootstrap Standard Error Interpretation

• Many bootstrap statistics have an


approximate normal distribution
• Confidence interval interpretation
– 68% of the time the bootstrap estimate (the
average of the bootstrap estimates) will be within
one standard error of true parameter value
– 95% of the time the bootstrap estimate (the
average of the bootstrap estimates) will be within
two standard error of true parameter value

7/26/2004 Unit 14 - Stat 571 - Ramón V. León 98


Bootstrap Confidence Intervals –
Percentile Method: Median Example
1. Draw N (= 10000) bootstrap samples from the data and for
each calculate the (bootstrap) sample median.
2. The 2.5 percentile of the N bootstrap sample medians will
be the LCL for a 95% confidence interval
3. The 97.5 percentile of the N bootstrap sample medians will
be the UCL for a 95% confidence interval

0.025 0.025

LCL UCL
7/26/2004 Unit 14 - Stat 571 - Ramón V. León 99
Do You Need to Know More?

“A Introduction to the Bootstrap”


by Bradley Efrom and Robert J. Tibshirani.
(1993) Chapman & Hall/CRC

7/26/2004 Unit 14 - Stat 571 - Ramón V. León 100