American Journal of Epidemiology Vol. 146, No.

12
Copyright © 1997 by The Johns Hopkins University School of Hygiene and Public Health Printed in U.S.A.
All rights reserved

Living Beyond Our "Means": New Methods for Comparing Distributions

Camara Phyllis Jones

This paper introduces the projection methods for describing and testing the differences between pairs of
continuous distributions. These methods include the projection plot, the projection spline, and the iter-1 test.
The projection plot displays the difference between corresponding quantiles against the average of the

Downloaded from https://academic.oup.com/aje/article/146/12/1056/111425 by guest on 12 April 2022

corresponding quantiles. It is analogous to an empirical quantile-quantile plot that has been rotated 45
degrees. The projection spline is a knotted linear spline iteratively fit to the projection plot so that all knots are
associated with significant changes in slope. It summarizes nonrandom deviations from linearity on the
projection plot, allowing classification of the highest level of difference between two distributions as a
difference in shape, in spread, or in location. The iter-1 test compares the first iteration of the projection spline
with the line y = 0, providing a global test of difference between two distributions that is more powerful in
simulations than either the chi-square test of independence or the Kolmogorov-Smirnov test. These methods
will enhance epidemiologic practice by making the comparison of full distributions an accessible tool for
routine data analysis. Am J Epidemiol 1997;146:1056-66.

computer simulation; epidemiologic methods; models, statistical

The data that epidemiologists observe and analyze
are usually sampled from some underlying, unknown
distribution. If that distribution is continuous, it can be
characterized by 1) its location, which is the typical
value, such as the mean, median, or mode; 2) its
spread, which is the dispersion around that typical
value, such as the standard deviation or interquartile
range; and 3) its shape, which describes other aspects
of how the data are grouped, including asymmetry,
peakedness, and the number of data clusters. Common
approaches to the comparison of samples from contin-
uous distributions, including Student's t test and linear
regression methods, represent a comparison of loca-
tions only. The wealth of information available in a
comparison of spreads and shapes most often goes
untapped.
The empirical quantile-quantile plot, introduced by
Wilk and Gnanadesikan (1) and popularized by Tukey
(2) and Chambers et al. (3), provides a succinct graph-
ical comparison of location, spread, and shape for two
samples from continuous distributions. The quantile-
quantile plot has the advantages that no assumptions
are made about the shape of either of the underlying
Received for publication October 7, 1996, and accepted for
publication July 8, 1997.
Abbreviation: LR, likelihood ratio.
Department of Health and Social Behavior and Department of
Epidemiology, Harvard School of Public Health, Boston, MA.
Reprint requests to Dr. Camara Jones, Department of Health and
Social Behavior, Harvard School of Public Health, 677 Huntington
Avenue, Boston, MA 02115.
distributions, that all of the data are used in generating
the plot, and that there is no arbitrary grouping or
smoothing of the data. However, interpretation of the
quantile-quantile plot is based solely on practiced vi-
sual assessment. There is no associated statistical test
to aid in interpreting apparent differences.
This paper introduces the projection methods (4), a
set of new methods for describing and testing the
differences between pairs of continuous distributions.
These methods include 1) the projection plot, a graph-
ical display of the difference between distributions
that retains the advantages of the empirical quantile-
quantile plot from which it is derived; 2) the iter-1 test,
a global test of difference based on the projection plot;
and 3) the projection spline, a summary model fit to
the projection plot that enables a difference between
distributions to be classified as a difference in shape,
in spread, or in location. The projection methods im-
prove on the quantile-quantile plot by including a
statistical test of whether distributions differ and im-
prove on all currently available methods by providing
a statistical summary of how distributions differ. They
will enhance epidemiologic practice by making the
comparison of full distributions an accessible tool for
routine data analysis.
BACKGROUND
The empirical quantile-quantile plot is constructed
by plotting the sample quantiles (order statistics) of

1056

one batch of data against the corresponding sample
quantiles of a second batch. If the two batches contain
the same number of data points, then the smallest
observation from batch X is plotted against the small-
est observation from batch Y, the second smallest
from X against the second smallest from Y, and so
forth until the largest observation from X is plotted
against the largest observation from Y. If the two
batches differ in size, the convention is to plot all of
the ordered data from the smaller batch against inter-
polated values from the larger batch that estimate the
corresponding quantiles (3).
As seen in figure 1, if two distributions are identical,
corresponding quantiles from the distributions will fall
roughly along the line v = x. If two distributions
differ in location only, corresponding quantiles will
fall roughly along a line parallel to the line y = x but
be displaced from it by the magnitude of the location
difference. If two distributions differ in spread but not
in shape, corresponding quantiles will fall roughly
along a straight line the slope of which differs from 1.
x quantiles x quantiles
Location difference
No difference
xquantBes x quantiles
Spread difference Shape difference
FIGURE 1. Interpretation of empirical quantile-quantile plots, con-
structed using simulated data sets for which there was no difference
(top left), a location difference only (top right), a spread difference
only (bottom left), and a difference in skewness only (bottom right)
between the parent distributions.
Am J Epidemiol Vol. 146, No. 12, 1997
New Methods for Comparing Distributions 1057
If two distributions differ in shape, corresponding
quantiles will deviate from a straight line pattern.
Therefore, one approach to test whether two distri-
butions differ would be to fit a line to the empirical
quantile-quantile plot and then test the fitted line for
difference from the line y = x. Although the corre-
lation and unequal variances of sample quantiles vio-
late the assumptions of ordinary least squares regres-
sion, this violation does not present a major stumbling
block to model-fitting, since the asymptotic covari-
ance of sample quantiles has been described (5). The
asymptotic covariance between the ith and jth sample
quantiles (order statistics), for i :£ j , is
Downloaded from https://academic.oup.com/aje/article/146/12/1056/111425 by guest on 12 April 2022
Pi*a - PJ)
where p,; is i/{n + 1), the percentile associated with
the ith order statistic; n is the total sample size; and
densj is the probability density of the underlying dis-
tribution at the ith order statistic. This covariance can
be taken into account with ordinary least squares re-
gression by doing postregression adjustment of the
standard errors (6).
It is the lack of symmetry of least squares regression
methods that is problematic when fitting models to the
empirical quantile-quantile plot. The choice of which
set of quantiles to plot on the x-axis and which set to
plot on the y-axis of the quantile-quantile plot is com-
pletely arbitrary. In fact, reciprocal plots are mirror
images of one another. Yet with least squares regres-
sion methods, the two sets of quantiles are not treated
symmetrically. The slopes from reciprocal models are
not exact reciprocals, since least squares methods min-
imize the vertical distance between the data and the
fitted line. The standard errors from reciprocal models
are likewise not equivalent, since only the covariance
of the quantiles plotted on the y-axis is taken into
account whereas the quantiles plotted on the x-axis are
treated as fixed. Occasionally, reciprocal models fit to
the quantile-quantile plot can lead to conflicting infer-
ences about whether two distributions differ.
PROJECTION PLOT
The problems of asymmetry between "dependent"
and "independent" quantiles are resolved when the
data on the empirical quantile-quantile plot are viewed
from the perspective of the line y = x. This sym-
metry can be seen in the top left panel of figure 2.
With respect to the line y = x, it does not matter
whether group A quantiles are plotted on the x-axis
and group B quantiles on the y-axis, or vice versa. The
points (A,B) and (B,A) are equidistant from the line
y = x, and they project on the line at the same point.
1058 Jones
x quantlles A
Symmetry with respect to line y=x Perpendicular distance from line y=x
(A.B)
/ projection plot.
\ shapes that could be made from the empirical quantile-
/
4±B
x quantiles
2
Projection on line y=x Rotating the plot 45 degrees
FIGURE 2. Derivation of the projection plot from the empirical
quantile-quantile plot, illustrating the symmetry of the points (A,B)
and (B,A) with respect to the line y = x on the quantile-quantile
plot (top left), calculation of the perpendicular distance of the point
(A,B) from the line y = x (top right), calculation of the coordinates
of the perpendicular projection of the point (A,B) on the line y
= x (bottom left), and the view if one rotated the plot 45 degrees
(bottom right).
In the top right panel of figure 2, the calculation of
the perpendicular distance of the point (A,B) from the
line y = x is shown. The vertical distance of the
point (A,B) from the line y = x is the difference
between the corresponding quantiles, B-A. Because
the line y = x makes a 45 degree angle with the
x-axis, the right triangle with hypotenuse of length
B-A has legs of length (B-A)/sqrt(2). The perpendic-
ular distance of the point (A,B) from the line y
= x is therefore (B-A)/sqrt(2).
In the bottom left panel of figure 2, the calculation
of the coordinates of the point at which (A,B) projects
on the line y = x is shown. The small right triangle
with hypotenuse of length (B-A)/sqrt(2) has legs of
length (B-A)/2. The sum of A and (B-A)/2 is
(A+B)/2, or the average of the corresponding quan-
tiles. The projection of the point (A,B) on the line y
= x is therefore at the point ((A+B)/2, (A+B)/2).
The bottom right panel of figure 2 suggests rotating
the quantile-quantile plot 45 degrees so that the line
y = x is the new x-axis. The new coordinates of the
point that is (A,B) on the quantile-quantile plot be-
come ((A+B)/2, (B-A)/sqrt(2)) on the rotated plot.
The proposed projection plot (figure 3) is the rotated
empirical quantile-quantile plot, after the y-coordinate
has been multiplied by the constant sqrt(2) for ease of
interpretation. It is called the "projection plot" because
the data on the quantile-quantile plot are viewed from
the vantage point of the line y = x, as if perpendic-
ularly projected onto that line. On the projection plot,
the difference between corresponding quantiles, B-A,
Downloaded from https://academic.oup.com/aje/article/146/12/1056/111425 by guest on 12 April 2022
is plotted against the average of the corresponding
quantiles, (A+B)/2. In figure 4, the relation between
the quantile-quantile plot and the projection plot is
shown. Note that the line y = x on the quantile-
quantile plot becomes the line difference = 0 on the
Inferences about relative locations, spreads, and
quantile plot can likewise be made from the related
projection plot. If two distributions do not differ, the
data on the projection plot will fall along the line
difference = 0. If two distributions differ in location
only, the data on the projection plot will fall along a
line with slope zero and intercept that corresponds to
the difference in location. If two distributions differ in
spread but not shape, the data on the projection plot
will fall along a line with a nonzero slope. If two
distributions differ in shape, the data will deviate from
a straight line pattern.
The projection plot therefore retains all of the ad-
vantages of the empirical quantile-quantile plot from
which it is derived. Differences in location, spread,
nines
ii
CO
IT
O)
urn
B-A
w
u
I
8 ° 4±B
C
D
line dif=O
2
D
•
O
O
c
D
D
average of corresponding quantiles
FIGURE 3. The projection plot, constructed by plotting the differ-
ence between corresponding quantiles against the average of the
corresponding quantiles.
Am J Epidemiol Vol. 146, No. 12, 1997

x quantiles average of quantlles
Ouantile-quantile plot Projection plot
group A quantiles average of A and B quantlles
Quantile-quantile plot Projection plot
FIGURE 4. Comparison of the quantile-quantile and projection
plots, showing the point (A,B) on the empirical quantile-quantile plot
(top left) and the corresponding point ((A+B)/2, B-A) on the projec-
tion plot (top right). The comparison between two systolic blood
pressure distributions is shown using the quantile-quantile plot (bot-
tom left) and the corresponding projection plot (bottom right).
and shape are readily appreciated from the plot, no
assumptions are made about the shape of either under-
lying distribution, all of the data are used in generating
the plot, and there is no arbitrary grouping or smooth-
ing of the data.
Moreover, the projection plot improves on the em-
pirical quantile-quantile plot through its symmetric
treatment on both the x- and the y-axis of the two
distributions being compared. This feature allows
model-fitting to the data using ordinary least squares
methods.
PROJECTION SPLINE AND ITER-1 TEST
The projection spline is a knotted linear spline iter-
atively fit to the projection plot to provide a flexible,
potentially nonlinear summary of the relation between
two distributions. A knotted linear spline is a piece-
wise polynomial of power 1, that is, a function con-
sisting of line segments connected end to end (7, 8). A
Am J Epidemiol Vol. 146, No. 12, 1997
New Methods for Comparing Distributions 1059
knotted linear spline with two knots (three segments)
can be modeled as
/30 + /3i x + (32 xplus(knoti) + /33 xplus{knot2),
where xplus(k) is evaluated as 0 for x £ k, and as
(x — k)forx > k (7). As is illustrated in figure 5,
the coefficient associated with each knot describes the
change in slope at that knot.
The steps in the iterative fit of a projection spline are
detailed in figure 6 and illustrated in figure 7. Initially,
knots are evenly spaced throughout the range of the
data. The model is fit, and the significance of each
Downloaded from https://academic.oup.com/aje/article/146/12/1056/111425 by guest on 12 April 2022
knot coefficient is evaluated. If any knot is associated
with a nonsignificant change in slope, the knot with
the least significant change in slope is removed from
the model. The reduced model is fit, and the signifi-
cance of each remaining knot coefficient is reevalu-
ated. The iterative removal of knots, one at a time, is
continued until all remaining knots are associated with
significant changes in slope at the specified a level.
The initial iteration of the projection spline, called
the iter-1 spline, provides the basis for a global test of
difference between the two distributions. The multi-
variate Wald test is used to compare the parameter
estimates from the iter-1 spline with a vector of zeroes.
The author refers to this application of the multivariate
Wald test as the iter-1 test.
The final iteration of the projection spline is used to
classify the highest level of difference between the
distributions, with the hierarchy of difference being
shape > spread > location > none. The classification
y = bO + b1 x + b2xplus(knot1) + b3 xplus (knot2)
FIGURE 5. Example of a knotted linear spline with two knots (three
segments), which can be modeled as ft + P\x
+ f}}Xplus(knot2).
1060 Jones

The four initial steps in the fit of a projection spline are:

(1) Decide on alpha, the two-sided type 1 error rate used to evaluate coefficients at each
iteration.
This will be guided by the purpose of the analysis and the sample sizes available.

(21 Choose the initial spacing of the candidate knots.

This determines where and how often the fitted spline is allowed to change slopes.

(3) Choose the minimum sample size allowed in an interval.

This reflects the amount of data considered necessary for estimation of the changes in
slope. If an interval contains fewer data than this minimum, it is merged with the
interval to the right by removing the intervening knot from the model. If the rightmost
interval contains too few data, it is merged with the interval to the left.

(4) Estimate the covariance matrix of the quantile differences.

Downloaded from https://academic.oup.com/aje/article/146/12/1056/111425 by guest on 12 April 2022

Because the two sets of quantiles being compared come from independent samples, the
covariance of the quantile differences is the sum of the quantile covariances. The same
covariance matrix is used at each iteration to adjust the standard errors of the
parameter estimates.

The four iterative steps in the fit of a projection spline are:

(5) Perform ordinary least squares regression to estimate the parameters of the model

y = Ro + B, xplus(O) + B2 xpluslknot,) + S3 xplus(knot2) + . . . + Bn xpluslknot^,).

where y is the difference between corresponding quantiles, and the xplus variables are
functions of the average of the corresponding quantiles. Note that when all x are non-
negative, the variable x can be expressed as the xplus function of a knot at 0.

(6) Adjust the standard errors of the parameter estimates for the covariance of the quantile
differences.
Adjusted standard errors are obtained by pre- and post-multiplying the covariance
matrix estimated in step (4) above by the (X'X)'1 matrix from the regression in step (5).

(7) Calculate the Wald statistic and the associated p value for each slope parameter.
The Wald statistic is the ratio of the parameter estimate to its standard error. Under
the null hypothesis that the parameter is zero, this statistic has a t distribution with
degrees of freedom equal to the sample size minus the number of parameters in the
model.

(8) Eliminate the least significant knot.

If all knot coefficients have/7 values less than alpha, do nothing more. The final model
has been identified. If any knot coefficient has p value greater than or equal to alpha,
identify the knot whose coefficient has the largest p value and eliminate it from the
model. Return to step (5) above to start another iteration.

FIGURE 6. The iterative fit of a projection spline.

is based on three characteristics of the projection
spline: the number of spline segments, the slope if
only one segment, and the intercept if only one seg-
ment with zero slope. The algorithm is summarized in
figure 8 and described below:
1. Evaluate the number of spline segments. If there
is more than one segment, there is evidence of a
shape difference.
2. If there is only one segment, evaluate the slope of
that segment. If the slope differs significantly
from zero, there is evidence of a spread differ-
ence.
3. If there is only one segment with slope equal to
zero, evaluate the intercept of that segment. If the
intercept differs significantly from zero, there is
evidence of a location difference.
4. If there is only one segment with slope equal to
zero and intercept that does not differ signifi-
cantly from zero, there is evidence of no differ-
ence in shape, spread, or location.
PERFORMANCE OF THE PROPOSED METHODS
One thousand samples, each consisting of 200 ob-
servations, are randomly generated from each of 10
known parent distributions as described in table 1. For

Am J Epidemiol Vol. 146, No. 12, 1997

 New Methods for Comparing Distributions 1061

Candidate knots Initial knots Iteration 1

Downloaded from https://academic.oup.com/aje/article/146/12/1056/111425 by guest on 12 April 2022

Iteration 2 Iteration 3 Iteration 4

Iteration 5 Iteration 6 Iteration 7

FIGURE 7. Iterative fit of the projection spline, which in this case is iteratively fit with a = 0.05 to the projection plot comparing two systolic
blood pressure distributions. The candidate knots are spaced every 10 mmHg at multiples of 10. The minimum sample size in an interval is
set at 20, so some knots are removed before the first iteration to coalesce adjacent spline segments. The carat ( A ) marks the knot that is
associated with the least significant change in slope for a given iteration, which is dropped from the model before the next iteration.

Shape? reference distribution are displayed in figure 9. Note

If not shape: Spread? that five of these distributions are non-Gaussian dis-
If neither shape nor spread: Location?
tributions that are generated as mixtures of Gaussian
None?
distributions. Pairwise comparisons between the 1,000
samples from the reference distribution and the 1,000
Shape Spread Location None
samples from each of the other nine distributions are
Segments >1 1 1 1 done to assess 1) the performance of the iter-1 test as
Slope =*0 0 0 a global test of difference, and 2) the performance of
Intercept 40 0
the projection spline as a basis for classifying the
highest level of difference.
FIGURE 8. Classification of the highest level of difference be-
tween two distributions, based on the projection spline. Performance of the iter-1 test
For each simulated comparison with the reference
example, the "reference" distribution is Gaussian, with distribution, the iter-1 test (with candidate knots at
a mean of 120 and a standard deviation of 15. The multiples of 10 and minimum sample size of 20 ob-
relations of each of the other nine distributions to the servations in an interval), the chi-square test of inde-

Am J Epidemiol Vol. 146, No. 12, 1997

1062 Jones

TABLE 1 . Parent distributions for randomly generated data sets

Component Gaussian Differs from
Name dlstribulion(s) "reference"?
of
Standard No.
distribution Mean Location Spread Shape
deviation sampled
Reference 120 15 200

No difference 120 15 200 No No No

Location differs 125 15 200 Yes No No

Spread differs 120 20 200 No Yes No

Skewness differs 117 9 160 No No Yes

132 24.9 40

Downloaded from https://academic.oup.com/aje/article/146/12/1056/111425 by guest on 12 April 2022

Kurtosis differs 120 9 160 No No Yes
120 28.3 40

Modality differs 108 9 100 No No Yes

132 9 100

Location/spread differ 125 20 200 Yes Yes No

All differ, skew 120 15 160 Yes Yes Yes

145 15 40

All differ, symmetric 120 15 100 Yes Yes Yes

145 15 100

pendence (with intervals of width 10 centered at mul-
tiples of 10), and the two-sample Kolmogorov-
Smirnov test are performed. (The parameters of the
iter-1 and chi-square tests were chosen because of an
interest in comparing systolic blood pressure distribu-
tions.) The p values from these tests are dichotomized
as <0.05 or sO.05 to reflect a decision rule that rejects
the null hypothesis of no difference with a nominal
type 1 error rate of 0.05. The results from these sim-
ulations are presented in table 2.
Type 1 error rates. Estimates of the actual type 1
error rates of the three tests are based on the 1,000
comparisons between the reference and "no differ-
ence" distributions. The type 1 error rate of each test is
the proportion of simulations with p < 0.05, in which
the null hypothesis of no difference is erroneously
rejected. The top line of table 2 shows that this occurs
52/1,000 times with the iter-1 test, 51/1,000 times with
the chi-square test, and 43/1,000 times with the
Kolmogorov-Smirnov test. All three tests have ob-
served type 1 error rates near the nominal a level of
0.05.
Power against specific alternatives. Comparisons
of each of the other distributions described in table 1
with the reference distribution provide estimates of the
power of the iter-1, the chi-square, and the Kolmog-
orov-Smirnov tests to detect various types and com-
binations of location, spread, and shape differences.
The power of each test is the proportion of simulations
with p < 0.05, in which the null hypothesis of no
difference is correctly rejected. For example, the sec-
ond line of table 2 shows that the iter-1 test has an
estimated 71 percent power to detect a 5-unit shift in
location relative to the reference distribution. This
compares with the chi-square test with 68 percent
power and the Kolmogorov-Smirnov test with 81 per-
cent power to detect the same difference.
The iter-1 test performs better than the chi-square
and the Kolmogorov-Smirnov tests in most types of
comparison. This superior performance is especially pro-
nounced when there are shape differences between the
two distributions being compared. The Kolmogorov-
Smirnov test is the most powerful of the three tests in
detecting a location shift, but otherwise it performs
poorly. None of the three tests is very powerful in de-
tecting an isolated difference in modality.
Performance of the projection spline
The simulated comparisons described above are
summarized by projection splines fit with candidate
knots at multiples of 10, minimum sample size of 20
observations in an interval, and two-sided a = 0.05.
Each projection spline is classified into one of four
categories according to the number of spline segments

Am J Epidemiol Vol. 146, No. 12, 1997

 New Methods for Comparing Distributions 1063

\
It \\
It \\
ll \\
II \\
l \\
II ll
1 \l
1 \\
II \\
ll \\
l \
J ^

No difference Location differs Spread differs

Downloaded from https://academic.oup.com/aje/article/146/12/1056/111425 by guest on 12 April 2022

Skewness differs Kurtosis differs Modality differs

Location/spread differ All differ skew All differ symmetric

FIGURE 9. Parent distributions for the simulated comparisons. The probability densities for each of the nine distributions named in the
captions (broken lines) are compared with the probability density of the reference distribution (solid line).

(1 vs. >1), the slope (0 vs. #0) if there is a single
segment, and the intercept (0 vs. =£0) if there is a
single segment with slope 0. The highest level of
difference between the two distributions is theorized to
relate to this classification as summarized in figure 8.
The distribution of the four classes of spline in these
simulations is presented in table 3. Each row summa-
rizes 1,000 comparisons of samples from the named
distribution with samples from the reference distribu-
tion. For example, when samples from the no differ-
ence distribution are compared with samples from the
reference distribution, 93/1,000 splines consist of
more than one segment, 23/1,000 splines consist of
one segment with nonzero slope, 43/1,000 splines
consist of one segment with zero slope and nonzero
intercept, and 841/1,000 splines consist of one seg-
ment with zero slope and zero intercept.
Sensitivity. The data in table 3 can be used to
calculate the sensitivity of the various classes of pro-
jection spline. In each row of the table, the bold value
is the number of splines falling into the proper cate-
gory based on the proposed classification of projection
splines. For example, comparison of samples from the
no difference and the reference distributions results in
841/1,000 splines correctly allocated to the "none"
category. In other words, the sensitivity of the one

Am J Epidemiol Vol. 146, No. 12, 1997

1064 Jones

TABLE 2. Power to detect differences in the simulated the hypothesis of no difference.

comparisons, using the iter-1 test, the chi-square test of The results of analogous calculations for each of the
independence, and the two-sample Kolmogorov-Smirnov
(K-S) test
four classes of spline are presented in table 4 and
summarized below:
No. i3f comparisons with p < 0.05
NcUTlQ from 1,000 simulations • One segment, zero slope, zero intercept: Very
of
distribution Iter-1 Chk
K-S strong evidence of no difference in underlying
square distributions against all of the simulated alterna-
No difference 52 51 43
tives except an isolated difference in modality
Location differs 708 683 808
Spread differs 850 751 279 (LR 1.7).
Skewness differs 953 863 418 • One segment, zero slope, nonzero intercept: Very
Kurtosis differs 918 693 310 strong evidence of a location difference against
Modality differs 450 321 83
all of the simulated alternatives.
Location/spread differ 969 935 878

Downloaded from https://academic.oup.com/aje/article/146/12/1056/111425 by guest on 12 April 2022

All differ, skew 884 799 709
• One segment, nonzero slope: Very strong evi-
All differ, symmetric 1,000 1,000 1 ,000 dence of a spread difference against all of the
simulated alternatives except combined differ-
ences in location, spread, and shape (LRs 3.2 and
TABLE 3. Highest level of difference inferred in the 1.4)..
simulated comparisons, based on the projection splinei * • More than one segment: Strong evidence of a
N3JT16
Projection spline classification of the shape difference against all of the simulated al-
nf
rlighest level of difference ternatives except an isolated spread difference
Ul
distribution Shape Spread Location None (LR 1.7) and a combined difference in location
No difference 93 23 43 841 and spread (LR 1.9).
Location differs 107 18 809 66
Spread differs 421 561 1 17
The LRs reported in table 4 can be applied to the
Skewness differs 975 1 1 23 analysis of an actual data set when the alternative
Kurtosis differs 844 1 6 149 hypotheses being considered are analogous in both
Modality differs 507 4 8 481 type and magnitude of difference to the simulated
Location/spread differ 375 613 9 3
All differ, skew 645 183 151 21
comparisons on which the LRs are based.
All differ, symmetric 552 424 24 0
Each row represents the results of 1,000 simulations DISCUSSION
comparing the named distribution with the reference distribution.
Bold values are the number of splines falling into the proper This paper introduces the projection methods for the
category based on the proposed classification of projection splines. pairwise comparison of continuous distributions. The
projection plot, derived from the empirical quantile-
quantile plot, displays the difference between corre-
segment, zero slope, zero intercept projection spline to sponding quantiles against the average of the corre-
detect a true lack of difference between underlying sponding quantiles. It provides a simultaneous
distributions is 84 percent. In these simulations, the graphical comparison of location, spread, and shape
sensitivity of the projection spline ranges from 51
percent in detecting an isolated difference in modality
to 98 percent in detecting an isolated difference in TABLE 4. Likelihood ratios based on simulated
comparisons*
skewness.
Projection spline classification of the
Strength of evidence. Likelihood ratio tests can be Name highest level of difference
done using the data in table 3 to assess the strength of OI
distribution Shape Spread Location None
evidence that a given class of spline represents in support
of one hypothesis over a specified alternative hypothesis. No difference 7.6 25.5 18.8
Location differs 6.6 32.6 12.7
As an example, let us estimate the strength of a spline Spread differs 1.7 809.0 49.5
with one segment, zero slope, and nonzero intercept as Skewness differs 587.0 809.0 36.6
evidence in favor of a location difference over no differ- Kurtosis differs 587.0 134.8 5.6
ence. The likelihood of observing such a spline is 809/ Modality differs 146.8 101.1 1.7
Location/spread differ 1.9 89.9 280.3
1,000 in the "location differs"-reference comparison, and 40.0
All differ, skew 3.2 5.4
43/1,000 in the no difference-reference comparison. This All differ, symmetric 1.4 33.7 Infinite
yields a likelihood ratio (LR) of 0.809/0.043, or 18.8. The * Each likelihood ratio provides a measure of the evidence that
specified spline therefore represents very strong evidence the given class of spline favors the inferred level of difference over
in favor of the hypothesis of a location difference over the named alternative.

Am J Epidemiol Vol. 146, No. 12, 1997


between two distributions. The related projection
spline, an iteratively fitted knotted linear spline, allows
identification of the highest level of difference be-
tween the two distributions being compared. The iter-1
test, based on the first iteration of the projection spline,
provides a global test of difference between the two
distributions.
The projection plot is simple to construct, makes no
assumptions about the shape of either underlying dis-
tribution, and uses all of the available data without
arbitrary grouping or smoothing. Because of the ver-
tical symmetry of the projection plot, ordinary least
squares regression methods can be used to fit consis-
tent statistical models to the data. This improvement
over the quantile-quantile plot enables the develop-
ment of a statistical test for evaluating the significance
of apparent differences.
The iter-1 test provides a global test of difference
between two distributions that is more powerful in sim-
ulations for most types of comparison than either the
chi-square test of independence or the Kolmogorov-
Smirnov test. The difference in performance is especially
pronounced when there are shape differences between
the two distributions being compared. A line fit to the
projection plot could also serve as the basis for a global
test of difference. However, it would be less powerful
than a knotted linear spline for nonlinear alternative
hypotheses, that is, when a shape difference between
distributions is suspected.
The projection spline provides the basis for a de-
tailed description of the nonrandom differences be-
tween two distributions. The algorithm for classifying
the highest level of difference based on the projection
spline is easy to apply, and the performance is gener-
ally quite good. However, a multisegment projection
spline provides only weakly positive evidence for a
shape difference over a spread difference in the sim-
ulations performed. Similarly, a single-segment pro-
jection spline with nonzero slope provides only
weakly positive evidence for a spread difference over
a mixed shape/spread/location difference.
It is noteworthy that although the projection plot is
nonparametric, the projection spline is not. The final
model depends on the number, spacing, and location
of candidate knots, the minimum sample size allowed
in an interval, and the a level used for knot retention.
The effects of variations in each of these parameters
and in total sample size on the performance of the
projection spline merit further study. Development of
a lexicon for interpreting the nature of shape differ-
ences and refinement of a method for quantifying the
degree of shape differences represent other promising
extensions of this work.
Am J Epidemiol Vol. 146, No. 12, 1997
New Methods for Comparing Distributions 1065
Why study distributions?
Information is lost when a full set of data is reduced
to a smaller set of summary statistics, such as a pro-
portion or a mean. Yet before the advent of modern
computing, the best way to understand a large batch of
numbers was to reduce it to a few summary statistics.
Now, powerful computers facilitate the graphical pre-
sentation and statistical manipulation of large amounts
of data, making the study of full distributions quite
feasible.
The simultaneous comparison of location, spread,
and shape is a powerful approach for detecting group
Downloaded from https://academic.oup.com/aje/article/146/12/1056/111425 by guest on 12 April 2022
differences. If two distributions have the same location
but differ in shape or in spread, they will not be
distinguished by Student's t test or by linear regression
methods. Yet differences in shape or in spread can be
of biological significance. The shape of a distribution
contains information about the process that generated
the distribution. A bimodal distribution might suggest
two etiologies, for example, and a truncated distribu-
tion might suggest some biological threshold. Simi-
larly, the spread of a distribution contains information
about tolerable limits of variability and may reflect the
degree of homeostatic control of a biological process.
Indeed, increases in spread with increasing age have
been noted for a number of variables, including sys-
tolic blood pressure (9, 10).
The study of full distributions is also compelling
because distributions characterize populations. Just as
individuals within a population can differ from one
another, whole populations can also differ from each
other (11). The ability to describe differences on a
population level facilitates discourse about population-
level risk factors and population-level interventions.
Potential applications
It is anticipated that the proposed methods will
enjoy wide application in many areas of epidemiologic
inquiry, since they can be used whenever two samples
from continuous distributions are to be compared.
These methods will enable epidemiologists to compare
1) continuous outcomes between exposed and unex-
posed groups in prospective studies, 2) continuous risk
factors between cases and controls in retrospective
studies, 3) continuous baseline characteristics or con-
tinuous outcomes between treatment groups in clinical
trials, and 4) continuous measures between popula-
tions of interest in cross-sectional studies. For exam-
ple, these methods can be used to examine differences
in infant lung function by intrauterine cigarette smoke
exposure, differences in measures of physical func-
tioning by extremes of socioeconomic status, or dif-
ferences in systolic blood pressure by "race."
1066 Jones
By understanding how distributions differ, we may
be led to new hypotheses about why they differ and
may spawn innovative approaches to improve the pub-
lic's health.
ACKNOWLEDGMENTS
This work was supported in part by National Heart, Lung,
and Blood Institute Cardiovascular Training grant T32
HL07024.
The author gratefully acknowledges the valuable input of
Drs. Karen Bandeen-Roche and Curtis Meinert.
REFERENCES
1. Wilk MB, Gnanadesikan R. Probability plotting methods for
the analysis of data. Biometrika 1968;55:1—17.
2. Tukey JW. Exploratory data analysis. Reading, MA: Addison-
Wesley Publishing Company, 1977.
3. Chambers JM, Cleveland WS, Kleiner B, et al. Graphical
methods for data analysis. Pacific Grove, CA: Wadsworth and
Brooks/Cole Publishing Company, 1983.
4. Jones CP. Methods for comparing distributions: development
and application exploring "race"-associated differences in sys-
tolic blood pressure. Doctoral dissertation. Department of
Epidemiology, The Johns Hopkins School of Hygiene and
Public Health, Baltimore, MD, 1994.
5. Serfling RJ. Approximation theorems of mathematical statis-
tics. New York, NY: John Wiley and Sons, 1980.
6. Weisberg S. Applied linear regression. 2nd ed. New York,
NY: John Wiley and Sons, 1985.
7. Smith PL. Splines as a useful and convenient statistical tool.
Am Stat 1979;33:57-62.
8. Wold S. Spline functions in data analysis. Technometrics
9. Drizd T, Dannenberg AL, Engel A. Blood pressure levels in
Downloaded from https://academic.oup.com/aje/article/146/12/1056/111425 by guest on 12 April 2022
persons 18-74 years of age in 1976-80, and trends in blood
pressure from 1960 to 1980 in the United States. Hyattsville,
MD: National Center for Health Statistics, 1986. Vital and
health statistics, series 11, no. 234. (DHHS publication no.
(PHS) 86-1684).
10. Berkman LF. The changing and heterogeneous nature of aging
and longevity: a social and biomedical perspective. Ann Rev
Gerontol Geriatr 1988;8:37-68.
11. Rose G. The strategy of preventive medicine. Oxford,
England: Oxford University Press, 1992.
Am J Epidemiol Vol. 146, No. 12, 1997