Evidence based assessment

• There can be no doubt about the critical role that assessmenthas in psychology. For
many psychologists, core elements of the discipline include (a) our heritage of striving
toaccurately measure and evaluate a host of human phenomena and (b) our
development of strategies, tools, and statistics to achieve these goals.

• Historically, assessment was an integral component in the development of the

profession of clinical psychology, and it remains one that is clearly evident in the
contemporary practice of clinical psychology. Indeed, it ishard
toconceiveofassessmentnotoccurring in situations in which psychological services are
provided, research is conducted, or psychological policy decisions are made

 At least among psychologists, that psychological assessmentis important and that

clinical psychologists are well-equippedto conduct assessments that inform
psychological services, the usefulnessofcontemporaryassessmentinstrumentsand
practices has rarely been questioned.

• For example, there is little solid evidence tosupport the usefulness of assessmentfor
improving treatment outcomes, and many commonly used assessmentmethods have
limited or contradictory empirical support

 The neglect of basic question as the usefulnessofcurrent assessment practices in

professional practice also reflects the influence of other factors, including(a) a beliefin
the intrinsicworth of manyestablishedand commonly used assessmentmethods (e.g.,
intelligencetests and projective personality tests) and (b) a lack of clarity regarding the
kinds ofevidence and criteria needed toevaluate the utilityof assessmentmethods and
processes.As a result, systematicevaluationsof the way in which cliniciansintegrateand
use assessment information in case formulation, treatment planning, or treatment
monitoring activitiesare only infrequentlyconducted

Research on psychological measures is not the same as research on the more

complicated task of psychological assessment,a task that involves integrating test results
(includingdata from self-report measures, behavioral observations, performance tasks,
and biologically based indices), life history, information, and collateral data into a
coherent description of the client being assessed. Research onpsychological
instrumentsis anecessary component of, but not a substitutefor, research on
psychological assessment

 concerns about accountability of psychological services

• the need tobase assessmentpractices morefully onscientific evidence
• the need for assessment research thatis directly relevant tothe practice of
 assessment
• the need tomorefully consider assessmentissues in treatment research and
treatment provision

 EBA is an approach to clinical evaluation that uses research and theory to guide the
selection of constructs to be assessed for a specific assessmentpurpose, the methods
and measures to be used in the assessment,andthe manner inwhich the
assessmentprocess unfolds.

• Evidence-based assessment involves the recognition by the psychologist that,

even when datafrom psychometrically strong measures are available, the
assessment process is inherently a decision-making task in which the
psychologist must iteratively formulate and test hypothesesby integratingdata
that may be incomplete or inconsistent.

 As aresult, a truly evidence-basedapproach to assessments evolves an evaluation of

the accuracy and usefulnessof this complex decision-makingtask in light of potential
errors and biases in data synthesis and interpretation, the costs associated with the
assessment process and, ultimately, the impact the assessment had on clinical
outcomes for the person(s)being assessed.

• Defined in this manner, EBA is part of the larger evidence-based practice (EBP)
movement that stresses the use of an amalgamation of systematically collected data,
clinical expertise,and patient preferences by decision makers (including,but not limited
to, clinicians) when considering services options for individual patients

 As aresult, a truly evidence-basedapproach toassessmentinvolves an evaluation of

the accuracy and usefulnessof this complex decision-makingtask in light of potential
errors and biases in data synthesis and interpretation, the costs associated with the
assessment process and, ultimately, the impact the assessment had on clinical
outcomes for the person(s)being assessed.

• Defined in this manner, EBA is part of the larger evidence-based practice (EBP)
movement that stresses the use of an amalgamation of systematically collected data,
clinical expertise,and patient preferences by decision makers (including,but not limited
to, clinicians) when considering services options for individual patients
Scientific Limitationsof SomeCommonly Used Clinical Instruments

• Overthe years, there havebeen numerous surveys ofthe instruments most commonly
used by clinical psychologists and taught in graduate training programs and internships.
• Despite some rather important advances in measurement and test development having
occurred, survey after survey has shown thatfew clinicians engage in assessment
practices that areconsistent with evidence-based guidelines (JensenDoss,2011).
• Moreover,the general patterns of instrument usage and the relative rankings
ofspecific instruments have changed verylittle overatleast 30 years

 These surveys indicate that, among the most commonly used and taught
instrumentsareanumber ofassessmenttoolsthat haveeither little or mixed research
support.
 The Rorschach ink blot testhasbeenthefocus ofa numberof literature reviews.
• The Comprehensive System(CS; Exner, 1993)is now considered by
Rorschachproponents astheprincipalscoringsystemfortheRorschach.
• The Rorschach is a very complex measure to administer,score, and
interpret,andtheCSprovides veryclear directionsonitsadministration and scoring.
• Unfortunately,researchindicatesthatevenif thesedirections arefollowed,
relativelyinnocuous contextualfactorsinRorschachadministration,suchas the layout of
the testingroom and the appearance of the assessor, influence the responses produced
by examinees (Masling, 1992).
 Sabatino, and Southwick (2001) found that both graduate students and psychologists
madenumerous errors inscoring the Rorschach. In fact, these errors wereso extensive
that the overallmean accuracyin scoring the main aspects of the CS was only 65%.
• A longstandingconcern with the CSnorms is that they tend to over pathologize
normal individuals,something that has been found in both child and adult research
samples
 Another example of an instrument that is frequently used by clinical psychologists
despite the lackofevidence forits reliability orvalidity is the Thematic Apperception
Test (TAT; Murray, 1943).
• Although reliable and valid scoring systems for apperceptive tests are employed in
research contexts (e.g., Spangler, 1992), they have not been adopted in clinical practice.
• Very little has changed over several decades of TAT research: There continues
tobeenormous variability in the manner in which the test is administered, scored, and
interpreted, and there is no cumulative evidence that supports the test’s reliability and
validity (Rossini & Moretti, 1997).
 Lally (2001) concluded that the most frequently used approaches to scoring projective
drawings failed to meet legal standards for a scientifically valid technique.
• Commenting on the scoring systems for projective drawings developed within
research contexts, heindicated thatthey fared somewhat better than other
systems.
 • Nevertheless,hesummarized their scientific status in the following manner:
“(a)lthough their validity is weak, their conclusions are limited in scope, andthey
appear tooffer no additional information over other psychological tests, it can at least
be argued that they cross the relatively low hurdle of admissibility

Problems with Instrument Selection and Interpretation

• Evenif clinical psychologists refrain from using instruments that have limited scientific
support, they must still ensure that the instruments they choose to use are appropriate
for the assessmenttask and are interpreted in accordance with the scientific evidence
• As assessments conducted by clinical psychologists often have substantial impact on
the people who are assessed (e.g., psychoeducational assessments that will largely
determine accessto additional educational supports and resources and disability
assessmentsthat will affect the possible awarding of disability pensions),it is critical
that instrumentsare selected and used in a professionallyresponsible manner
 Many clinical psychologistsconduct evaluations that are used to inform the courts in
determining child custody decisions. These evaluations areexpected to provide fair
and balanced information about allparties relevant tochild custody
decisions,andguidelines for conducting these assessments have been available to
psychologists and other health professionals for many years.
 However, there continue tobe many problems with the way in which someclinical
psychologists undertake these assessments.Onesurvey ofpsychologistswho frequently
conducted child custody evaluations found that projective tests were often used to
assess childadjustment (Ackerman & Ackerman, 1997).
• As described in the previous section, these types of tests rarely have appropriate levels
of reliabilityand validity, and areunlikelyto have valid norms. Without evidence
ofstrong psychometric properties,itis all but impossible to make accurate
determinationsabout the psychological adjustment of those tested.
 For reasons such as these, the recent guidelinesof the Association of Family and
Conciliation Courts (AFCC, 2006) stressed the need for assessors to use assessment
methods that are empirically based and warnagainst using instrumentswith
questionablereliability orvalidity
• Horvath, Logan, andWalker (2002) examined child custody evaluation reports
includedin court records and conducted content analyses to determine the extent to
which key aspects of these guidelineswere met in the evaluations. They found evidence
of many gaps in the reports as, for example, evaluators often failed to assess general
parenting abilitiesand the ability of each parent to meet his or her child’s needs—key
elements of any assessmentintended to provide guidance on the issue of child custody
 Even when a psychometrically robust measure is used,challenges can still arise in
accurately interpreting the test data. This fact is abundantly evident in the literature on
the Wechsler intelligence scales, tests that are among the psychometrically strongest
psychological instruments available.A wealth of information exists on the importance
of using country-specific norms for the interpretation of the Wechsler intelligence
scales
• For example, Kamieniecki and LyndStevenson (2002) found that Australian children
obtained slightly higher IQ scoresthan did American children of a similar age. This
 means that, as was typically done in the past, using American norms in the
assessmentof intellectual giftedness or disability among Australian children would
result in some classification errors. Similarly, the use of American norms, rather than
the Canadian norms, for the adult intelligence scale leads to a significant
underestimation of the extent of cognitive impairment expenced by Canadian patients
with substantial neuropsychological problems

Limited evidence for clinical utility of Commonly Used Instruments

• With respect to psychological instruments,clinical utility refers to whether the use of

test data improve upon typical clinical decision making and treatment outcome or, in
other ways, make a difference in terms of the client’s functioningas a result of clinical
services informed by the test data (Hunsley & Bailey, 1999). Although the
definitionsvary, acommonemphasis isplaced ongarnering evidence regarding actual
improvements in both decisionsmade by clinicians and service outcomes
experiencedby patients
• In other words, we have very little evidence that psychological assessmentdatahave
adirect impact onthe improved provision and outcome of clinical services
 A study byLima et al. (2005) illustrates the way in which research can begin to address
the important set of questionsinherent in the concept of clinical utility. In this study,
clients completed the Minnesota MultiphasicPersonality Inventory-2 (MMPI-2) prior to
commencing treatment; half the treating clinicians then received feedback on their
client’sMMPI-2 data, and half did not
 The clients involved in the study presented with a range of diagnoses,
includinganxietydisorders, mooddisorders, substance-relateddisorders, adjustment
disorders, eating disorders, and personality disorders.
• To address questions of utility, the researchers conducted between-group comparisons
on variables related to treatmentoutcome. They found that providing clinicians withthe
MMPI-2resultsas a potentialaid in treatment planning had no discernible impact on
variables such as client improvement ratings or premature termination rates.
• These results illustratethat, even for instruments that have been extensively
researched, such as the MMPI-2,an instrument should not automaticallybe
assumedtopossess utilityfor allthe tasksfor which it is commonly used

Conceptualizing and evaluating evidence-based based Assessment

• The current era ofevidence-basedhealth-care practices both requires and depends upon
the use of scientifically sound assessment methods, instruments, and strategies (Barlow,
2005).
• Without accurate assessmentdata, it is impossibleto determine the psychosocial
functioningofpatients, monitor the treatments they are provided, evaluate the impact of
these services at completion, evaluate the broader program in which the treatments are
provided, or conduct ongoing quality assurance in order to enhance the functioningof
the clinicians working in the program
 The evaluationof, and, ultimately,the identificationof EBTs restsentirely on the
assessmentdata,ignoring the quality of psychological assessment instrumentsand the
mannerin which theyare used placesthepromotion of evidence-
 basedpsychologicalpracticeinjeopardy.Asanexample, fora number of psychological
disorders, research-derived treatment benchmarks can now provide clinicians with
meaningful and attainable targets for their intervention services
• But, to be meaningful, these benchmarks must be based on reliable and valid measures,
and to employ these benchmarks in routine practice, clinical psychologistsmustuse
assessmenttools that havestrong scientific support and areappropriate for the taskat
hand. Moreover, the decision- makingprocesses surrounding
howtodeterminerelevantandappropriate benchmarks, obtain client data, and
comparethese data to the benchmarks must, themselves, be reliable and valid
 Across mental health professions, cliniciansare seeking assessmenttools they can use to
determine a client’s level of pretreatmentfunctioning and to develop, monitor, and
evaluate the services received by the client
• In their continuing efforts to improve the quality of mental health assessments, the
American PsychiatricAssociationpublished the second edition of its Practice Guideline
for the Psychiatric Evaluation of Adults (2006).
• Drawing from both current scientific knowledge and the realities of clinical practice,
this guideline addressed both the contentand process of psychiatric evaluations. It also
built upon the Handbook of Psychiatric Measures (American Psychiatric Association,
2000), a publication that offers information to mental health professionals on the
availabilityof self report measures,clinician checklists,andstructuredinterviews
relevant to the provisionof clinical services.
• For each measure reviewed, there is a brief summary of its intended purpose,
psychometricproperties, likely clinical utility, and the practical issues likely tobe
encountered in its use.
 With respect to children and adolescents,the AmericanAcademy of Child
andAdolescentPsychiatryandAmericanAcademyof Pediatricshavebeen activein
developing practiceguidelinesfor assessmentinstruments(e.g., self-report
instruments)and contexts (e.g., family assessment),as well as for the assessmentof
specificchildhood disorders (e.g., autisticspectrum disorders).
• These guidelines are based on the scientific literature and clinical consensus,and
describe generally acceptedapproaches toassessspecific disorders
• Thus,theneedfor developingEBApracticesis apparentnot only inclinical psychology but
across a variety of health disciplines that,in some areas, may be further along in this
enterprise than professional psychology
Building a Truly Evidence-based psychological Assessment
TheMeaningfulnessofInstrumentMetricsandChangeScores
• Unlike information about weight or height, scores on psychological instrumentsdo not,
of themselves,conveymuch information about a person’s functioning(Blanton & Jaccard,
2006). Knowing that someone obtained a score of 20 tells us nothing unless we know
the range ofpossiblescores, norms for the instrumentand, in the case of clinical
instruments,the cut-score value(s) that indicate whether a score is in the “clinical” or
“normal” range. Because of the diverse range of clients for which instrumentsmay be
used, it is critical that nationally representative norms be available that are sensitiveto
possible gender and age influences
 Although clinical cutoffs are reported for many clinical instruments,either in a test
manual or in the research literature, the manner in which the cut-scores were
derived may be problematic (Meehl & Rosen, 1955). Specifically, using a cut-
scorederived from samples in which clinical cases are overrepresented comparedto
usual base rates will usually result infrequentclassification errors
• Just as a score itself usually has no intrinsic meaning, a change in a person’s
scoresover time is also meaningless without a standardized way of interpreting the
magnitude of the change.
• This is a key concern for clinical psychologists, as monitoring changes in
functioning via self-report and observational instrumentsprovides us with
information about the impact of our treatments(cf. Kazdin, 2006).
• Tothisend,itisnecessaryto determine whethertheobservedchangeinscoresis
(a) more than simply a matterof measurementerror and (b) substantialenough to
indicate that improvement has occurred.
 For any instrumentfor which basic psychometric information is available, itis
possible toderive avalue to determine the minimum changes in scores necessary to
indicate that a client is no longer functioning in the clinical range
IncrementalValidityandEvidence-basedAssessment
• Incremental validity addressesthe questionof whetherdata from an instrument (or
othersourceof data) adds to the prediction of a criterion above what can be predicted
by other sources of data
• Clinical psychologistsoften gather data from multiple measuresand multiple informants,
yetthereislittleevidencein theempiricalliteraturetoaidin determiningwhich measuresto
use,howtocombinethem,andwhattodoif conflictingresultsare obtainedwith
somemeasures.Progressis beingmadein someareas.
• Forexample,Pelham,Fabiano,andMassetti(2005) reportedthat, in diagnosing attention-
deficit/ hyperactivity disorder in youth, structured diagnostic interviews do
notpossessincrementalvalidity overrating scalesand,therefore,are likelyto have little
value in clinical settings.
• More such researchis necessarybefore clinical psychologistswill be able to make
scientifically informeddecisionsabout which instrumentsto usein an assessment(and
which may be redundant and cost ineffective).

ClinicalUtilityandEvidence-basedAssessment
• Evidence about enhancementstodecisionsmadeby clinicians and/or service outcomes
experiencedby patients are central aspects of clinical utility,regardless of whether the
focus ofthe evidence is on diagnostic systems (e.g., First et al., 2004), psychological
instruments (e.g., McFall, 2005) intervention strategies (e.g., American Psychological
Association Presidential Task Force on EvidenceBased Practice, 2006).
• Attentionto clinical utility has grown in recent years but, unfortunately,there
isstilllittle evidence thatbears onthe question of the extent to which EBIs or EBA have
clinical utility.
 The OQ-45 researchprovidesan excellent examplethat could be usedin designing
clinical utility studies for other instruments.
• Specifically, Lambert, Whipple, Hawkings, Vermeersch,Nielsen, and Smart
(2003)conducteda meta-analysis of three utility studies in which more than 2,500 adult
clientscompleted the instrumentafter each therapy session.
• Clinicians in the control condition did not receive information about the client
scores;clinicians in the experimental condition receivedfeedbackabout the scoresthat
was limited to an indication of whetherclients, based on normative data, were making
adequate treatment gains, making less-than-adequatetreatment gains, or
experiencingsofewbenefitsfromtreatmentthattheywereatriskfornegativetreatmentout
comes.
• By theendoftreatment,basedonOQ-45 data,21% ofclientsinthe controlcondition had
experiencedadeteriorationoffunctioningand21% had improvedin their functioning. In
the experimentalcondition, only 13% of clients had deterioratedand 35%
hadimproved.Suchresultsprovidea verycompellingindicationoftheclinical utility of the
instrument
Integration of Data and Clinical Decision-making: From Evidence-based Instruments to
Evidence- based Assessment

• EBA must involve the recognition that psychological assessment is a

complex decision-making task.
• Decisions must be made about which constructs to assess and which instruments
should be used to assess these constructs.
• The dataobtained from instrumentsmust bethen integrated and interpreted, and
the impact of potential errors and biases in this process must be considered.
• However, reviews ofEBAs todate toappear tobe primarily lists of EBIs

 It ispossible thatthismaysimplyreflectthe early stageof developmentof EBAand

that,overtime, hunsley, mashincreasedattentionwill be paid to the pressing question of
how datafrom EBIs can contribute to a truly evidence-based assessment.
• Importantstepsare being taken by researchers in a number of areasto develop
assessmentprotocols thatare evidence-based,straightforwardto use, and provide
feedback that minimizes the reliance on clinician interpretation of the data

• Inthemeantime,though, themostscientificallysound option availablefor many

assessmenttasks may be to ensure that assessmentsare conducted with EBIs in a
manner thatis informed by the research evidence regarding key constructs to assess for
specific patient conditions and assessment purposes.