PURRFECT RESOLUTION

No matter what their area of expertise, statisticians are asked certain questions with such
predictability that it borders on the deterministic. No question is asked more often than:
How many samples do I need?
Most statisticians wish they could answer the sample size question definitively instead of
mumbling about effect sizes and whatnot. It’s just not that simple.
One way to look at how many individual samples (i.e., observations, cases, records, subjects,
survey respondents, organisms, or any other object or entity on which you collect information)
you need for an analysis is in terms of how much resolution you want. Think of the resolving
power of a telescope or a microscope, or the number of pixels in a computer image. The greater
the resolution, the more detail you’ll see.
Consider this picture. You couldn’t make out the image with a resolution of 9 pixels per inch and
maybe not even with a resolution of 18 pixels per inch. At 36 pixels per inch, you can tell it’s an
image of a kitten, even if it’s a bit fuzzy. At 72 pixels per inch, the image is sharp and you can
tell that the kitten is Kerpow. Doubling the resolution again adds little to your perception of the
image; it’s a waste of the additional information. Likewise with statistics, the greater the number
of samples, the more precise your results will be. But beyond a certain point, adding samples
adds little to your understanding. In fact too many samples can have negative consequences. So,
the trick is to collect the fewest samples that will achieve your objective.

Deciding how many samples you’ll need starts with deciding how certain your answer needs to
be given your objective. Now here’s the bad news. There’s no way to know exactly how many
samples you’ll need before you conduct your study. There are, however, formulas for estimating
what an appropriate number of samples might be. In the situations in which the formulas don’t
apply, there are rules-of-thumb or other ways to come up with a number. Unfortunately, it seems
that no matter how many samples you estimate you’ll need, the number is always a lot more than
your client wants to collect. After all, they are the ones who have to pay for collecting and
analyzing the samples. So, any estimate of the number of samples that you tell your client ends
up being a subject of negotiations. But here are a few places to start.
How Many Samples for Describing Data?
Say all you want to do is to collect enough samples to calculate some descriptive statistics.
Maybe you want to characterize some condition, like the average weight of a litter of kittens or
the average age of your favorite professional sports team. How many samples do you need? Well
if your population is small enough, like five kittens or 25 baseball players, you simply use all the
members of the population, a census.
But what if you want to calculate descriptive statistics to characterize a large population? The
number of samples you'll need to describe it will depend on the precision you want, not the
accuracy. The greater the number of samples, the more precise your estimate will be. More
specifically, the precision will be proportional to the variance divided by the square root of the
number of samples. So maximize the number of samples if you can (by a lot, remember,
precision is proportional to the square root of the number of samples), but if you can’t, try to
control the variance.
How Many Samples for Detecting Differences?
Often, the point of calculating statistics is to make an inference from a sample to a population.
You can estimate how many samples you might need to conduct a statistical test of one or more
populations by rearranging the equation for the test you plan to use and solve for the number of
samples. To take this approach, there are usually two other things you need to know—the
difference you want to detect and the population variance.
You should have some idea of the size of the difference you want to detect, called a meaningful
difference. Say you want to compare how long it takes you to commute to work via two different
routes. Differences of a few seconds probably aren’t meaningful but differences of a few minutes
probably are. If you work as a NASCAR driver, go with seconds. The smaller the difference you
want to detect the more samples you’ll need.
Knowing the population variance is the Catch-22 of statistics. You can’t calculate the number of
samples you’ll need without knowing the population variance and you can’t estimate the
population variance without already having samples from the population. Now, there are maybe
a half dozen ways to try to get around this problem but they all require you to know or guess at
some aspect of the population. The approach is often used after a preliminary study (called a
pilot study) is done in part to estimate the population variance.
How Many Samples for Opinion Surveys
If you’re going to survey a small population, like your colleagues at work, send surveys to
everybody and hope you get a representative sample from the people who do respond. If the size
of your population is large compared to the number of surveys you might take, a quick way to
estimate the sample size is:
sample size = 1 / (approximate percent error you want) 2
So if you want a 5% error with 95% confidence, you would need about 400 surveys to be
completed (i.e., 1/0.052). With 1,000 surveys, the error drops to about 3%, but to get to 2% error,
you would have to collect 2,500 surveys. It’s more complicated than this of course. If your
sample will be a sizable proportion of your population or if the opinions aren’t evenly divided,
the short-cut formula will overestimate how many surveys you might need. If you plan to
subdivide the population to look at demographics, you’ll need more surveys to get to your
desired error rates.
How Many Samples for Evaluating Trends?
Say you plan to do a regression analysis to evaluate the relationship between two sets of
measurements. How many samples do you need? There are two ways to answer this question, a
difficult way and an easy way. The difficult way is to calculate it the same way as you would if
you were looking to detect differences. This approach requires a sophisticated understanding of
statistical tests and the populations being tested. It is most often used in experimental situations.
The simpler approach is to base the number of samples on a rule-of-thumb based on the number
of independent variables. The more independent variables (i.e., predictor variables) there are, the
more samples are needed to define their relationship to a dependent variable. The guidelines are
not hard and fast but they boil down to these:
10 samples per predictor variable—the bias may be large but there are often enough
samples to estimate simple linear relationships with adequate precision.
50 samples per predictor variable—the bias is relatively small, linear relationships can
be estimated with good precision, and there are usually enough samples to determine
the form of more complex relationships.
100 samples per predictor variable—the bias is insignificant, linear relationships are
estimated precisely, and complex nonlinear relationships can be estimated adequately.
250+ samples per predictor variable—the bias is insignificant and most complex
relationships can be estimated precisely.
How Many Samples for Forecasting Time Series?
Deciding how many samples to use for analyzing a time series can be a challenge. Here are two
popular rules-of-thumb:
Collect samples at regular intervals from at least three or four consecutive
cycles or units of any pattern in which you might be interested. For example,
if you are interested in seasonal patterns (i.e., a pattern lasting a year) collect
data for at least three or four years.
Collect samples at time units smaller than the duration of the pattern in which
you might be interested. For example, if you are interested in seasonal
patterns, collect data weekly, biweekly, or at least, monthly.
How Many Samples for Identifying Targets?
Sometimes the goal of sampling is to find one or more targets. For example, in World War II,
destroyer captains needed to know how many depth charges to drop to be reasonably certain of
destroying an enemy submarine. Likewise, adventurers looking for sunken ships, like the
Monitor and the Titanic, use statistical sampling to find their targets. In the environmental field,
sampling is often done to look for “hot spots” of contamination in soil. There are two ways this
type of problem is typically handled—judgment sampling and search sampling.
The strategy behind judgment sampling is that an expert picks locations for sampling that he or
she believes are most likely to reveal a target. With this approach, it is assumed that the expert
has some preternatural ability to find the targets. Judgment sampling (a.k.a. judgmental
sampling, biased sampling, haphazard sampling, directed sampling, professional judgment) has
the advantage of involving far fewer samples than search sampling. The disadvantage is that
there is no way to quantify the uncertainty of the result.
Search sampling involves sampling on a regular grid so that it is possible to estimate the
probability of finding randomly located targets. In essence, the probability of finding a target
depends on the size and shape of the target and the size and shape of the cells of the sampling
grid. The downside of this sampling approach is that it usually involves many more samples than
the judgment sampling approach and the results do not always sound very reassuring. For
example, you would need over 10,000 samples taken on a 100-foot square grid in a 1,000,000
square-foot search area to have an 80% probability of finding a circular hotspot 100 feet in
diameter. In search sampling, a large number of samples is the price you pay for being able to
quantify uncertainty. But if you understand the uncertainty, you are one giant step closer to
controlling adverse risks. That’s the resolving power of statistics.
Join the Stats With Cats group on Facebook.

http://statswithcats.wordpress.com/2010/07/17/purrfect-resolution/