P1: GAD

pl-nr2004.cls (03/25/2004 v1.1 LaTeX2e NERV document class) pp1358-nerv-494745 October 8, 2004 17:28

Neuropsychology Review, Vol. 14, No. 3, September 2004 (

C 2004)

A Critique of Miller and Rohling’s Statistical Interpretive

Method for Neuropsychological Test Data

Victor L. Willson1 and Cecil R. Reynolds1,2

A critical review of the 24-step procedure of Miller and Rohling’s (in press) proposed standardization
of clinician’s use of neuropsychological assessment batteries is presented. Each step is examined
for statistical sources of invalidity. It was concluded that parts of the procedure are quite vulnerable
to between-battery variability that cannot be easily estimated or controlled, leading to significant
errors in analysis and classification. A second fatal flaw is the failure to distinguish in the procedures
between standard error measurement and standard error of the estimate in calculations in several
steps. The purpose of the process remains viable, however, and is an important contribution toward
the improvement of clinical diagnosis.
KEY WORDS: neuropsychology; assessment; critique.

The intent of Miller and Rohling (in press) to stan-
dardize the interpretation of the test scores resultant from
a neuropsychological evaluation is laudible. The interplay
between clinician’s expert knowledge and empirical test
information has now had a 100-year history, from the early
writings of Lightner Witmer (Witmer, 1902) and others
(e.g., Pintner, 1923) onward. The concern for idiosyn-
cratic interpretations of data that result in misdiagnosis
leads Miller and Rohling to propose a 24-step procedure
to evaluate test scores and produce consistent, defensible
evidence for diagnosis that is intended to be highly reli-
able across diagnosticians. In particular, it is intended to
be generalizable across test batteries that might be used
in the neuropsychological evaluation. If successful, this
method would indeed be robust. Our concern is in its po-
tential for success and unwarranted levels of confidence
in diagnoses deduced from this model.
Our approach here is to examine each step for its
susceptibility to either error or its probability of error. Be-
cause the probability of error in any statistical procedure
based on successive steps is the product of the probabili-
ties of correct decisions at each step (although it may be
possible for incorrect decisions at one stage to result, nev-
1 Texas
A & M University, College Station, Texas.
2 To whom correspondence should be addressed at Department of Educa-
tional Psychology, MS 4225, Texas A & M University, College Station,
Texas 77843-4225; e-mail: crrh@earthlink.net
ertheless, in correct decisions at some successive stage),
we will attempt to determine at least in general form the
probability of a correct decision based on the cumulative
decisions at the steps proposed. Some steps will have no
incorrect probability associated with them, which will be
noted. Error from step to step is not simply additive but
increases geometrically.
Step 1: Design and Administer a Test Battery. There
is great potential for different diagnoses with different test
batteries. There is no empirical evidence in the research
literature that all test batteries of neuropsychological func-
tioning provide test scores that estimate a set of cognitive
functions identically. That is, for a test score xij


Xij = λij k Fk + eij (1)
where i = test i of construct I
λ = regression (factor) loading on construct k, from
1 to K (including construct I )
j = battery J selected for the patient,
It is improbable that all loadings on a particular kind of
test (say digit span) are identical across test batteries. Note
that we assume that for many tests the loadings are not
zero for constructs other than the one principally intended.
This conclusion affects subsequent steps.
Here there is an additional, nonstatistical point of
disagreement. In choosing to administer a flexible battery,
Miller and Rohling (p. 10) state “test selection should

177
1040-7308/04/0900-0177/0

C 2004 Plenum Publishing Corporation
P1: GAD
pl-nr2004.cls (03/25/2004 v1.1 LaTeX2e NERV document class) pp1358-nerv-494745
178
be based on knowledge of the literature and experience
with the instruments.” A few other guidelines such as
recency of norming are offered; we agree with all of these
rubrics. However, the ultimate argument for using a flexi-
ble versus a standard battery rests upon the argument that
a flexible battery can be chosen to assess best the patient’s
presenting problem. Criticisms of standard batteries cen-
ter around their potential insensitivity to the heterogeneity
of the patient population seen and subsequent lack of di-
agnostic specificity or utility. Miller and Rohling however
recommend picking a battery based upon examiner char-
acteristics and test characteristics without regard for the
presenting problem, and other relevant patient character-
istics. If following the recommendations of Miller and
Rohling, each clinician may have a different battery but it
would be the same battery for all patients seen by a specific
clinician. This seems to defeat the purpose of the flexible
battery choice altogether. Each clinician then would have a
unique but clinician-specific standard battery (that would
of necessity vary as a function of age), which could in fact
lead to more accurate diagnoses if clinicians maintain and
monitor data on their own practices.
Step 2: Estimate Premorbid General Abilities. There
are two problems with this step. First, the construct of gen-
eral ability is not defined. In the absence of a meaningful
definition, it appears that general ability is whatever the
particular combination of test battery and demographic
information available can be factor analyzed to produce.
This vagueness will produce great variation in any con-
struct score developed from it. Equation (1) is invoked
here, except now the equation becomes



gm = Xij + b1 d1 ,
where ability construct gm is the EPGA, estimated premor-
bid general ability estimated from the available test scores
and demographics, and b1 is the regression loading for a
particular demographic variable d1 . This equation clearly
indicates that the ability construct estimate will depend
on the test scores and particular demographics selected,
and in turn on the particular emphases of the test battery
loadings on neuropsychological constructs estimated by
the test battery selected.
Miller and Rohling recommend that multiple mea-
sures be used to produce multiple estimates of premorbid
general ability and that these estimates then be combined.
Each of the estimates mentioned has a different latent
construct with its own distribution. Converting them all to
T scores and using the mean to estimate PMGA is mathe-
matically flawed since the underlying scales of measure-
ment are not the same (e.g., rank in class is on an ordinal
scale and NART scores are interval scales and the two
cannot be averaged in a way that makes sense). Miller
October 8, 2004 17:28
Willson and Reynolds
and Rohling do not specify a means of aggregating these
estimates into a single data point; we are relatively certain
it is difficult to do accurately and that clinicians will vary
greatly in the methods they apply. The choice of method
and the variability in such choices across clinicians will
both add unknown amounts of error to the equation that
cannot at this point be estimated accurately. In using a sin-
gle method, such as class rank, Miller and Rohling ignore
regression effects in prediction of less than perfectly cor-
related variables. This issue also complicates their goal of
consistency across clinicians since many different choices
of procedures and ways of combining them will be chosen
without strict guidance.
Step 3: Convert all Scores to a Common Metric. The
linear transformation of the test scores does not change the
variability in estimation of underlying neuropsychologi-
cal constructs. However, we feel compelled to elaborate
on Miller and Rohling’s discussion of their preference for
normalization of scores. They leave it open to clinicians
to normalize the T -score transformations or to use a linear
transformation. To the extent that clinicians make different
choices, this variance or unreliability in choosing which
transformation to apply (linear or normalization) will add
inestimable error to the diagnostic process and adds fur-
ther to the point above that consistency across clinicians
cannot be achieved following their recommendations.
Miller and Rohling provide some discussion of when
to choose normalization of score distributions and when
not to do so. Our preference is for linear transformations
in the case of well-designed standardization samples and
recommend that only well standardized tests in fact be
used (thus negating the debate between normalized and
(2) linear T scores).
On page 13, the authors note that “when normative
data are highly skewed . . . appropriate transformations
should be conducted to normalize the distribution prior to
transforming the scores into T scores.” It is only appro-
priate to normalize a distribution of scores if the lack of
normality within the score distribution in question is due
to sampling error. The assumption underlying transfor-
mation to a normalized distribution is that the underlying
distribution in question is in fact normal and that sampling
errors have produced the lack of normality in the attained
distribution. If the attained distribution is truly represen-
tative of the actual distribution in question, converting
or normalizing the distribution will introduce additional
error into the process. This is often overlooked by re-
searchers who desire to apply methods that require normal
distributions of scores. The whole concept of normalizing
distributions was developed to correct for sampling error.
It is a widely held misconception that behavioral and cog-
nitive and emotional variables are normally distributed
(e.g., see discussions in Reynolds and Kamphaus, 1992,
P1: GAD
pl-nr2004.cls (03/25/2004 v1.1 LaTeX2e NERV document class) pp1358-nerv-494745
Critique
2002). Many are and others approximate normality. In
these instances, converting score distributions to a normal
curve does not adversely impact the data. However, if the
latent distribution is in fact nonnormal, and many such
psychological variables are not normally distributed, nor-
malization of the distribution is inappropriate and will lead
to invalid inferences or additional error in the inferential
process.
Step 4: Assign Each Test Score to a Cognitive Do-
main. This step has a high degree of uncertainty in that
there is no evidence for the interclinician reliability of
classifications of test scores. Although it might reason-
ably be expected to be moderately high, the error exists
in the clinician selection of the number of domains and in
the classification of a given test xij to that domain. To the
extent clinicians do not all classify tests in the same way
cumulates the errors associated with the estimation of the
construct scores.
Miller and Rohling seem to recognize the problem
of subjectivity of test score classification and attempt to
avoid it by noting that each score is assigned to the cog-
nitive domain for which prior published factor analytic
research has found the highest loadings. This assumes
that all domains of cognitive function are known or at
least that there is agreement on these. There is not. Clas-
sification of test scores by domains within the broader
domains of cognitive versus noncognitive, even in the
face of strong theory and empirical data, is difficult. There
are many discussions of the subjectivity of factor naming
for example in factor analysis (e.g., Cattell, 1978). Many
people argue over the classification of memory versus
attention and even some well-known researchers (e.g.,
Jensen, Kaufman, Horn, and the like) are sharply divided
over whether such a commonly used test as Coding from
the Wechsler scales is a cognitive task that should be
characterized as a memory measure, or a very low level
measure of psychomotor speed, or is better classified as
a measure of attentional skill. There are simply far too
many legitimate disagreements in the field to allow such
a simplification of these classifications. Also, Kaufman
(1978, 1994) has argued cogently that observation of the
patient while taking a test such as Coding or certain other
Wechsler tasks is necessary to determine the appropriate
classification of a task for an individual patient. Thus,
a single task may be assigned to different domains (and
accurately so) for particular patients based upon their ap-
proach to the task. It is interesting to see how Miller and
Rohling have classified Trail Making and certain other
tasks to differentiate them from attention and concentra-
tion and processing speed, and also assuming that process-
ing speed is cognate to psychomotor speed. Many would
debate this. All of this goes to address the complexity of
this issue and the simplicity of the authors’ response.
October 8, 2004 17:28
179
Secondly, Miller and Rohling assume that a defini-
tive factor analysis exists for all batteries considered. The
outcome of a factor analysis as expressed in the number of
factors extracted and the correlation of each variable (test)
with the factor is a function of the variables in the specific
analysis. The number of factors extracted and the relative
loadings for a set of tests may be altered dramatically by
the addition or deletion of several of the tests. No true
meta-analysis that matches factors and relative loadings
on common constructs could be located. Such an analysis
would be necessary to have a strong empirical basis for
assigning tests to domains, but even then, assignments
may need to vary for individual patients (see Kaufman,
1978, 1994, for additional clinical explanations). The un-
reliability of the process at this stage augments the error
component of Miller and Rohling’s procedure. Granted
this would be a major undertaking, but the methods are
available if not the time or the funding (e.g., see Cattell,
1978).
Step 5: Calculate Domain Means, Standard Devia-
tions, and Sample Sizes. Determining the mean and stan-
dard deviation of each domain is not as simple as the
authors present. It is not possible to calculate the standard
deviation of a domain score unless one knows the corre-
lation between each part score and the total score or the
covariance matrix is available. Given that the examiner
selects a wide-ranging number of tests and then combines
them into a domain score, the number of possible standard
deviations to be derived is substantial. One would have to
have a table of intercorrelations of all of the potential
tasks within a domain in order to calculate the standard
deviation correctly. This is crucial because it affects the
score distribution itself. One can find more detailed infor-
mation on this problem in published reviews of the fourth
revision of the Stanford Binet scales, which also fails to
incorporate this problem into their norms table and for
which they have been soundly criticized by the measure-
ment community. Some of the earliest of measurements
texts, such as Guilford’s in psychometric methods from
the 1950s, deal with this issue in some detail. There are
also a variety of formulas in the literature that allow one to
calculate the standard deviation of a composite if one only
has the composite score and the correlation matrix. This
would not be needed if one were norming a composite
and had the complete normative data but then one would
also have the correlation matrix. In Step 6, the test battery
means are discussed but you cannot make sense of these
without having the standard deviations correct because the
underlying metric is in fact unknown and cannot assumed
to be the standard T -score SD of 10.
Step 6: Calculate Test Battery Means. No error is
added, but unless the standard deviations are correct, later
applications of these means may be difficult to interpret.
P1: GAD
pl-nr2004.cls (03/25/2004 v1.1 LaTeX2e NERV document class) pp1358-nerv-494745
180
Step 7: Calculate Probabilities of Heterogeneity. The
probabilities for the computed chi-square statistics are
highly sensitive to the errors from Step 4. The level of
imprecision in classification and domain selection now
becomes important because of the attempt to determine
chi-square probabilities.
Step 8: Determine Categories of Impairment. This
step is affected by cumulative error through Step 4 also.
The resultant errors of category determination depend on
the amount of cumulative error in Step 4 and are aggra-
vated by Step 5 error.
Step 9: Determine the Percentage of Test Scores That
Fall in the Impaired Range. The number of tests that are
classified into the impaired range will depend in part on
the cumulative errors up through Step 5 and in the degree
to which the underlying metrics of the domain scores and
the individual test scores are accurately estimated (i.e., the
degree of deviation of the estimates form the true SDs).
The reliability of classification is an unknown function of
this cumulative error.
Step 10: Calculate Effect Sizes. The computations of
effect size depend on the estimate of the EPGA, which
has been shown to depend on the test batteries and demo-
graphics selected, and the problems of selecting a method
of estimation.
Step 11: Calculate Confidence Intervals. The statisti-
cal thinking here is either flawed or incomplete. Standard
errors of measurement depend on the estimate of reliabil-
ity of the score being considered. The formula presented
by Miller and Rohling is incorrect. They provide an esti-
mate of the standard error of estimate, not of measurement.
The interpretation of these two is quite different, which
affects later steps. Because no evidence for reliabilities for
many of the computed measures is presented by Miller and
Rohling, nor are they likely to be easily estimated without
extensive empirical research and simulation, standard er-
rors of measurement are largely unknown. Standard errors
of estimate will in general be smaller than standard errors
of measurement when
n > 1/(1 − reliability).
Thus, if the reliability of EPGA is .8, then if n > 5, the
standard error of estimate will be smaller than the stan-
dard error of measurement. This will produce artificially
small confidence intervals for test scores and give greater
credence to the accuracy of the results than is warranted.
Step 10 indicates that the confidence intervals will
usually be smaller than is warranted by the reliability of
the data.
Step 12: Determine the Upper Limit Necessary for
Premorbid Performance. The results of Step 11 will
tend to produce estimates of maximum level of pre-
morbid performance that most likely are too low. Be-
October 8, 2004 17:28
Willson and Reynolds
cause the standard error of estimate will be smaller, often
much smaller, than the standard error of measurement
for an individual’s estimated score, the upper (or lower)
limit of the confidence interval will be insufficiently ex-
treme in correctly estimating the upper limit of premor-
bid performance. Then, even with no impairment, there
would be a substantial proportion of scores beyond the
limit that would occur within the confidence interval
of the standard error of measurement. This would re-
sult in overclassification of an individual’s actual perfor-
mance as outside premorbid confidence intervals (loss of
ability).
Step 13: Conduct One Sample t Tests on Each Type
of Mean Generated. The direction here is too vague to be
certain of what is intended. Because t statistics for the dif-
ference from a mean depend on the standard deviation and
sample size, the authors need to make this explicit. The
results of these analyses depend on the Step 2 EGPA esti-
mate, for which no standard deviation has been computed
to this step, or if T scores are used, on the assumption
of adequate estimates of any intermediate statistics. In
principle, however, this is not as troublesome as the con-
fidence interval computations in Step 11. Nevertheless,
there is likely to be considerable cross-battery error in the
t statistics and their probabilities. The lack of specific
direction however will result in inconsistencies across
clinicians defeating the purpose once again of the entire
procedure.
Step 14: Conduct a Between-Subjects Analysis of
Variance with Domain Means. As was noted in Step 4,
domain means will depend on battery selection, so the
ANOVA results will also vary accordingly.
Step 15: Conduct a Power Analysis. The results of
this step depend on the effect size computations, which
depend on the EPGA estimate, with resultant uncertainty
and error.
Step 16: Sort Test Scores in Ascending Order. This
step is highly susceptible to random differences in scores,
even if no between-battery error were present. It is quite
problematic to interpret the ordinal position, which loses
the important distance information.
Step 17: Graphically Display the Summary Statistics.
Of concern is the overinterpretation of mean score vari-
ation by clinicians. Without standard errors of measure-
ment, for example, too much precision may be attached
to means and associated t statistics.
Step 18: Determine the Test Battery’s Validity. Al-
though much of the discussion in this step is valid, the
continued reliance on sample size rather than reliability
in score interpretation is invalid.
Step 19: The procedures discussed here do not in-
crease error, but are expected to be affected by the errors
cumulated in earlier steps.
P1: GAD
pl-nr2004.cls (03/25/2004 v1.1 LaTeX2e NERV document class) pp1358-nerv-494745
Critique
Step 20: Use Test Battery Means to Determine if Im-
pairment Exists. Again, the reliability of the classification
depends on the selection of batteries in Step 1 and the
accuracy of the estimation of the latent score distributions
within domain.
Step 21: Determine Current Strengths and Weak-
nesses. This is problematic because the CIs (confidence
intervals) are overprecise estimates of patient variability.
The general procedure is not necessarily at fault if appro-
priate standard errors of measurement are applied.
Step 22: Examine the Test Scores From Noncognitive
Domains. It is unclear what effects will accrue from this
step beyond adding to the inconsistency across clinicians
due to the lack of guidance in how to conduct and interpret
this analysis.
Step 23: Explore Low Power Comparisons for Type
II Errors. This whole step has misplaced precision and un-
derstanding of the variability in patient test battery scores,
stemming mostly from the confusion between standard
error of estimate and standard error of measurement.
Step 24: Examine the Response Operating Charac-
teristics of Numerically Sorted Scores. This is an over-
application of a highly technical procedure. Although it
may function adequately if all clinicians use the same
battery, there is no attempt to determine false positives
within- or between-batteries, raising another flag for error
in interpretation.
Our conclusion is simple. Parts of the procedure are
susceptible to between-battery variability that has not been
adequately evaluated for reliability or fidelity of analysis
and classification. Other steps founder on a misinterpreted
concept of variability based on standard error of estimate
rather than standard error of measurement. This is a fatal
flaw in our opinion and must be revised before the steps
can even be considered within-battery for patient evalua-
tion. We were not able to achieve our earliest goal of pro-
viding an accurate estimation of the error components of
the model because too many components remain subjec-
tive and the error components are inestimable. Accurate
estimation of the total error of the model will be central to
its application and is necessary for its full evaluation but
as of yet cannot be accomplished.
We fear that clinicians, who will themselves intro-
duce inconsistencies and additional error into the process
October 8, 2004 17:28
181
for reasons described above, will by virtue of the appar-
ent statistical sophistication of the Miller and Rohling
technique, imbue the results with greater confidence than
it is due. The proper level of confidence cannot be de-
fined precisely due to the noted difficulties in estimating
the error components of the model but confidence can-
not be characterized as high at this point given the error
components identified and the many points of subjective
decision making inherent in the model and differences
in how clinicians will apply certain steps where choices
are given. Neither, however, have the results been studied
for validity or reliability and such studies are necessary
before the method is applied clinically. However criti-
cal we seem of the precise methods extolled in Miller
and Rohling, we continue to laud these authors for their
intent despite finding fault with their method of imple-
mentation and trust their work will stimulate more and
improved thinking in this arena by researchers and clin-
icians alike. This alone makes it a worthwhile contribu-
tion. The solutions to these problems are complex and
Miller and Rohling have stimulated what we predict will
be a major line of research in the discipline. Their work
certainly has helped us to understand better the issues
to consider and the complex interplay of these variables
in diagnosis and classification of patient performance on
any set of measures, but particularly those that are not
conormed.
REFERENCES
Cattell, R. B. (1978). The Scientific Use of Factor Analysis in the Be-
havioral and Life Sciences, Plenum, New York.
Kaufman, A. S. (1978). Intelligent Testing With the WISC-R, Wiley
Interscience, New York.
Kaufman, A. S. (1994). Intelligent Testing With the WISC-III, Wiley
Interscience, New York.
Miller. L. S., and Rohling, M. L. (in press). A statistical interpretive
method for neuropsychological tests. Neuropsychol. Rev.
Pintner, R. (1923). Intelligence Testing: Methods and Results,
Henry Holt, New York.
Reynolds, C. R, and Kamphaus, R. W. (1992). Behavior Assessment
System for Children, American Guidance Service, Circle Pines,
MN.
Reynolds, C. R, and Kamphaus, R. W. (2002). Clinician’s Guide
to the Behavior Assessment System for Children, Guilford,
New York.
Witmer, L. (1902). Analytical Psychology, Ginn & Company, Boston.