Power: Example Questions and Outline Model Answers

Question 1

Two groups of 20 schoolchildren (20 subjects per group) were tested in a study of
anagram solving. The mean time to solve the anagrams for group 1 (who had not
received special literacy training) was 10.5 seconds (standard deviation= 6.7 seconds).
The corresponding figures for group 2 (who had received the literacy training) were
13.9 and 6.9 seconds.

(a) Explain the elements which appear in the approximate formula for calculating the
statistical power of a two-tailed test, given below. (20% of question marks)

(b) Use the formula to calculate the observed power of this study, for tests at the 5%
significance level. Remember to adjust the formula because 1-tailed tests would
be carried out in this study. Based on the information given below you will be able
to express your answer as a pair of values within which the observed power lies.
Show your working. (35% of question marks)

(c) If there were a real group difference as large as the one observed in the data, then
what is your estimate of how often the null hypothesis would fail to be rejected?
What is the name for this quantity? (10% of question marks)

(d) By using the approximate formula below, estimate the sample size that would be
required in each group in order that the above experiment would have 80% power
to detect a group difference equivalent to that measured in the experiment. Base
your calculations on one-tailed tests at the 5% significance level. The formula is
for a two-tailed test and needs adjusting appropriately. Show your working. (35%
of question marks)

Approximate Power Formula

power = z-1{d*sqrt(n/2) – z(1-α/2)}

where sqrt(x) is the square-root of x

Approximate Sample Size Formula

required sample size per group = 2*{z(power) +z(1-α/2)}2/d2

Data That May Be Needed in The Calculations

α z(1-α/2) x z-1{x}
.050 1.960 -1.95 0.026
.100 1.645 -1.65 0.049
.200 1.282 -1.30 0.097
.300 1.036 -1.00 0.159
.400 .842 -0.85 0.198
.500 .674 -0.65 0.258
.600 .524 -0.50 0.309
.700 .385 -0.25 0.401
.800 .253 -0.05 0.480
.900 .126 0.00 0.500

Note: the z-1{x} function is symmetrical about x=0; so z-1{1.95}=1-z-1{-1.95} etc.

Question 1: MODEL ANSWER

Note that the perfect answer can be pretty short:-

(a) Elements to explain: z and z-1; d; n; α. z( ) is the inverse cumulative distribution

function of the normal distribution -- the function takes a probability and converts
it into a z-score. Thus, a probability of 0.975 is converted to a z-score of +1.96
because the z-score below which 97.5% of the normal distribution lies is +1.96.
The function z-1( ) is the inverse of z and is the cumulative distribution function of
the normal distribution -- the function takes a z-score and converts it to a
probability. Thus, a z-score of +1.96 is converted to a probability of 0.975, i.e.
97.5% of the normal distribution lies below this z-score. (An explanation of z and
z-1 without the exact jargon, but which is accurate, will also get full marks here.) d
is a measure of the between-groups effect size: it is the mean difference between
the groups divided by their pooled standard deviation. n is the number of subjects
in each of the 2 groups (and the group size must be equal for this formula). α is the
Type I error rate, conventionally 5%.

(b) In the example n=20 and α=0.05. To calculate observed power we need the
observed effect size. To calculate the observed value of d: the mean difference
between the groups is 3.4 secs and the groups have s.d. of 6.7 and 6.9. When we
pool the s.d. of equal-sized groups we can take the average s.d. i.e. 6.8. Thus
d=3.4/6.8=0.5. The adjustment to the formula that is required for a one-tailed test
is to replace z(1-α/2) with z(1-α). To see this, consider doing a z-test on a mean
which is supposed to be drawn from a standard normal distribution (mean=0;
s.d.=1). To test (two-tailed, with α=0.05) if the observed value of the mean is not
significantly different from zero then the critical z-values are z(0.975)=+1.96 and
z(0.025)=-1.96. In other words, the value must lie between z(α/2) and z(1-α/2). To
test (one-tailed, with α=0.05) if the observed value is not significantly above zero
the critical z-value is z(0.95), i.e. the value must be below z(1-α).

the term in the {} brackets must be worked out first:-

{0.5*sqrt(10) - z(0.95)}

from the table given if α=0.1 then z(1- α/2)=z(0.95)=1.645

so the {} term works out to be -0.06

thus, from the table, z-1{-0.06} lies between 0.480 and 0.401, but much closer
to 0.480. Thus, the observed power for this experiment is between 40 and
48%.

(c) Between 52% and 60% of the time (but closer to 52%). This is known as the Type
II error rate.
(d) To calculate the required sample size for 80% power by a 1-tailed t-test we must
use an adjusted formula once again. Again, replace z(1-α/2) with z(1-α). [Check: a
1-tailed test should increase power and so reduced required number of subjects.
z(1-α/2) would be replaced with a smaller value z(1-α) in the formula for required
sample size and so this is correct].

the {} term is [z(0.8) + z(0.95)]2

to look up z(0.8) we need to look up z(1-0.4/2) thus we need to use the table for
α=0.4. Thus, the {} term =[0.842 + 1.645]2 = 6.185

and so the required sample size in each group = 2*6.185/0.52= 49.5 = 50.