TRANSLATIONS

From Research to Practice: The Importance of

Appropriate Outcome Selection, Measurement, and
Reporting in Pediatric Mental Health Research
Suneeta Monga, MD, Martin Offringa, MD, PhD, Nancy J. Butcher, PhD, Peter Szatmari, MD

esults of randomized controlled trials (RCTs)
R and, ideally, the synthesis of RCT results into
meta-analyses, drive both clinical and policy de-
cision making about health care interventions and inform
new avenues for research. Selection of the right health
outcomes to measure, analyze, and report on when
designing a trial therefore is key in ensuring that the
research will be useful and will have maximum impact.1-3
As in many areas of medicine, wide variability exists in
pediatric mental health research as to which outcomes
are selected, how they are measured, and how they are re-
ported. Variability in outcome selection and measurement
may stem from various factors, including a tendency to
select what is feasible or historical to measure rather than a
focus on what is important to measure and how best to
measure it. The resulting variability limits comparing and
combining the results from different trials addressing a
similar question in meta-analyses, hindering the translation
of research to practice.1,2 The objective of this article is to
highlight the importance of outcome selection, measure-
ment, and reporting in the translation of research to clinical
practice.
We propose that translation of research may be facili-
tated by having researchers designing clinical trials (or
cohort studies) and knowledge users consider eight key
questions when reviewing either the study protocol or the
final report:
1. Were knowledge users, such as patients, families, and
policy makers, involved in deciding which outcomes to
measure?
2. Are outcomes appropriately defined (eg, outcome
domain, measure, metric, method of aggregation, and
time point)?
3. For clinical trials specifically, are the outcomes clearly
identified as primary and secondary?
4. Is there a rationale for determining the clinical signifi-
cance of change or the minimally important clinical
difference expected in the primary outcome?
5. Are the properties of the outcome measurement instru-
ment (OMI) (eg, reliability, validity, responsiveness,
feasibility) stated or referenced for the studied population
and setting and not merely extrapolated from a typical
(or adult) population?
6. When are outcomes measured, and is there a develop-
mental theory of change that informs the timing of
outcome measurement?
7. Are multi-informant measures used to assess study
outcomes?
8. Are all planned study outcomes fully reported in trial
registries, protocols, and statistical analysis plans, and are
they the same as what is described in the final trial
report?
1. WERE KNOWLEDGE USERS, SUCH AS
PATIENTS, FAMILIES, AND POLICY MAKERS,
INVOLVED IN DECIDING WHICH OUTCOMES
TO MEASURE?
Both the research question and the needs of the knowledge
user, which may include clinicians, payers, and policy
makers, should be considered when choosing the specific
outcome to measure. Through patient engagement and
advocacy, the importance of the patient’s voice is increas-
ingly being heard, which in pediatric mental health research
needs to include youths, and whenever possible children,
and their parents (or primary caregivers).2,3 Different
knowledge users even among patients (eg, youths versus
parents), however, may have different sets of priorities,
which may influence the outcome selection.3 Age and/or
developmental level as well as cultural background may

Journal of the American Academy of Child & Adolescent Psychiatry www.jaacap.org 497
Volume 59 / Number 4 / April 2020
MONGA et al.
influence patient priorities that must be taken into account
when selecting an outcome to measure. With the greater
awareness of patients as important research contributors and
knowledge users, there has been a move to use patient-
centered and idiographic outcomes in addition to compos-
ite measures of clinical judgment or other observational
measures to help ensure that the outcome is truly relevant to
the patient.2-4 As an example of how patient involvement
can be effectively used in making important changes to trial
design, youths, when asked about the choice of primary
outcome in a recent pragmatic RCT of collaborative care
teams, chose “impairment” not “symptom severity,” which
was the intuitive preference of the investigators.5 As patient
involvement in research evolves, the collective voice of
which outcomes are important, developmentally appro-
priate, and culturally sensitive to measure among different
knowledge users will influence the field.
2. ARE OUTCOMES APPROPRIATELY DEFINED
(EG, OUTCOME DOMAIN, MEASURE, METRIC,
METHOD OF AGGREGATION, AND TIME
POINT)?
Once outcomes have been chosen, the question becomes
how they should be defined, measured, and analyzed.1-3
Defining an outcome requires consideration of five ele-
ments: (1) the name or more appropriately the outcome
domain (eg, depression severity); (2) the measure or more
specifically the OMI that will be used (eg, Children’s
Depression Inventory); (3) the metric or more specifically
what outcome change will be measured (eg, change from
baseline); (4) the method of aggregation (eg, continuous
versus categorical); and (5) the time point, specifically the
time point at which the outcome is measured and analyzed
(eg, 4 weeks after treatment start).1
3. FOR CLINICAL TRIALS SPECIFICALLY, ARE
THE OUTCOMES CLEARLY IDENTIFIED AS
PRIMARY AND SECONDARY?
Determining which outcome is the primary outcome is
critical, as the primary outcome is key in obtaining funding,
getting a clinical trial protocol through regulatory and
research ethics reviews, calculating the trial’s sample size,
conducting interim analyses, formulating stopping rules,
and interpreting and comparing results across trials. The
selected primary outcome may also impact patient recruit-
ment, as patients may not want to contribute to trials
focused on outcomes that do not reflect or measure what is
important to them. Multiple primary outcomes are often
reported, which is not consistent with the use of the term
and can lead to type 1 errors if not properly accounted
498 www.jaacap.org
for.1,6 Additionally, it is necessary to consider the extent to
which the primary and other outcomes of the trial are in-
dependent of each other or whether they are correlated.
When outcomes are highly correlated, they may reflect a
common underlying domain, in which case a finding of
significant differences of treatment effects between the two
outcome measures is difficult to interpret.1,6
4. IS THERE A RATIONALE FOR DETERMINING
THE CLINICAL SIGNIFICANCE OF CHANGE OR
THE MINIMALLY IMPORTANT CLINICAL
DIFFERENCE EXPECTED IN THE PRIMARY
OUTCOME?
In addition to considering what outcome change will be
measured (metric), researchers must also have a rationale for
interpreting the clinical relevance of the change, especially
on an individual patient level. Inference about treatment
effectiveness in pediatric mental health research to date has
been based on standardized group change or individual
change (eg, Reliable Change Index),7 as compared between
experimental and control arms. Although statistical criteria
for small, medium, and large effect sizes that are indepen-
dent of specific measurement tools exist, meeting any of
these effect size criteria does not necessarily translate into
clinical relevance or magnitudes in outcome difference that
would lead to adoption of the trial’s “superior” treatment in
clinical practice. A more patient-centered and critical
concept to consider is what is the minimal change the pa-
tient feels is important, or what is the minimally important
clinical difference expected between experimental and con-
trol arms—thus allowing for patient (eg, child, youth, or
parent) input on whether change in an outcome represents a
small, moderate, or large change.8,9 Despite advancement in
other fields of research toward precision medicine, a better
understanding of the biological and genetic underpinnings
of disease processes are needed to fully implement individ-
ualized outcomes in mental health research. However, in
the interim, researchers can do more by involving patients
and using their input to identify what are meaningful and
relevant minimally important clinical differences for them.
5. ARE THE PROPERTIES OF THE OMI (EG,
RELIABILITY, VALIDITY, RESPONSIVENESS,
FEASIBILITY) STATED OR REFERENCED FOR
THE STUDIED POPULATION AND SETTING
AND NOT MERELY EXTRAPOLATED FROM A
TYPICAL (OR ADULT) POPULATION?
How to measure outcomes is equally salient, as there are
many ways in which mental health outcomes can be
measured (eg, use of questionnaires, clinical observation,
Journal of the American Academy of Child & Adolescent Psychiatry
Volume 59 / Number 4 / April 2020

biomarkers). Mental health research relies heavily on the use
of questionnaires and clinical observations, with many
different youth report, parent report, and clinician report
measures available, and to date there is no obvious “gold
standard” for the choice of OMI to use. Measurement tools
vary in terms of their feasibility (eg, time, cost, and inter-
pretability) and their generalizability, which also factors into
the selection process.10 The choice of OMI can impact the
synthesis and comparability of RCT findings, as different
trials use different OMIs because there is no clear consensus
or standardization on which OMI to use for any of the
commonly measured outcomes in mental health research.
Furthermore, determination of which OMI to use will
depend on the age of the child or youth; the domain being
measured; and the resources available, including expertise of
the research team. Although researchers often think about
whether to use “subjective” or “objective” OMIs when
designing trials, the reliability and validity of the OMI in the
population (eg, age and developmental range or clinical
versus nonclinical samples) and the setting (eg, home versus
school versus clinic) are essential considerations, regardless
of whether the OMI is administered by direct observation,
interview, questionnaire, or some biomarker. For example,
many child/youth report questionnaires are not validated
across all age ranges despite being used across different
developmental periods; as well, many child/youth OMI
validation studies to date have been performed in relatively
small sample sizes of selected populations. Additionally, the
method of aggregation planned for analyzing the data yiel-
ded from the OMI should be defined early on, as this can
influence the decision of what OMI should be selected; for
example, the analyses of continuous data as categorical from
an OMI that measures on a continuous scale requires
caution and careful justification of the selection of the cutoff
value used.11
6. WHEN ARE OUTCOMES MEASURED, AND IS
THERE A DEVELOPMENTAL THEORY OF
CHANGE THAT INFORMS THE TIMING OF
OUTCOME MEASUREMENT?
When to measure and analyze an outcome should be based
on a theory of change of the intervention and the sensitivity
of the OMI to detect that change in a given time period.
This constitutes another major aspect of selecting both the
outcome and the OMI.1 Specifically, whether the outcome
is expected to show change over the study’s time frame and
whether the OMI is expected to, or is known to, be sensitive
to change (ie, demonstrates responsiveness) over this time
frame are both important factors to consider. For example,
with depression treatment trials, although change in
Journal of the American Academy of Child & Adolescent Psychiatry
Volume 59 / Number 4 / April 2020
TRANSLATIONS
depression severity may not be expected immediately after
treatment start, change must be measured before what
might be considered natural remission. The concept of
measurement equivalence is also important, as it refers to
the idea that a measurement tool measures the equivalent
underlying outcome over time, which is especially impor-
tant when a study is conducted over a long time frame.12
7. ARE MULTI-INFORMANT MEASURES USED
TO ASSESS STUDY OUTCOMES?
In pediatric mental health trials, choosing the informant is
especially important, as we can ask clinicians, parents,
teachers, or youths themselves to be informants.2-4,13 In
fact, the use of multiple informants in pediatric mental
health may be essential because symptoms may be observed
exclusively within a given context (eg, home versus school),
thus warranting both parent and teacher reports.13 Different
informants may have different sensitivities to change in
outcomes (eg, parent versus youth awareness of internalizing
and externalizing symptoms and symptom change); cultural
differences are another important consideration (eg,
different normative cultural values may influence perception
of change).13 The use of multi-informants and multiple
methods for obtaining outcome information is essential to
avoid inflation of effect sizes.4
8. ARE ALL PLANNED STUDY OUTCOMES
FULLY REPORTED IN TRIAL REGISTRIES,
PROTOCOLS, AND STATISTICAL ANALYSIS
PLANS, AND ARE THEY THE SAME AS WHAT IS
DESCRIBED IN THE FINAL TRIAL REPORT?
A final issue to consider is the extent to which all relevant
outcomes are fully reported in clinical trials.14,15 Trial
protocols reported according to the SPIRIT guideline15
and Statistical Analysis Plans16 are increasingly being
published or made publicly available, making it possible to
compare the planned (prespecified) and the eventually
reported outcomes and their analyses. As demonstrated in
empirical studies, selective reporting of outcomes
threatens trial reproducibility, interpretation, and com-
parison of results across RCTs; threatens the validity of
meta-analyses; and influences the estimates of the effects
of the interventions.1,14 Details on why an outcome was
chosen, when and how it was measured and by whom, and
how the data were analyzed are often not reported, leading
to reduced understanding and incomplete interpretation
of trial results. Whereas consensus now exists on the
immense importance of selecting and measuring key
outcomes in trials, there still is no agreed-on standard for
the reporting of outcomes, although guidance for optimal
www.jaacap.org 499
MONGA et al.
outcome reporting in trial protocols and reports is under
development.17
In conclusion, consideration of the eight key questions
by researchers when designing a pediatric mental health
clinical trial or a cohort study will ultimately lead to
improved outcome selection, measurement, analyses, and
reporting. As the field advances, greater involvement of
knowledge users and decision makers, including patients,
and further standardization around outcome selection,
measurement analyses, and reporting will ensure that new
research is more easily translated into good clinical practice.
Accepted August 21, 2019.
Drs. Monga and Szatmari are with the Cundill Centre for Children and Youth
Depression, Centre for Addiction and Mental Health, Ontario, Canada, The
Hospital for Sick Children, and the University of Toronto, Ontario, Canada. Drs.
Offringa and Butcher are with the Child Health Evaluative Sciences, The
Hospital for Sick Children Research Institute, Toronto, Ontario, Canada. Dr.
REFERENCES
1. Mayo-Wilson E, Fusco N, Li T, Hong H, Canner JK, Dickersin K. Multiple outcomes
and analyses in clinical trials create challenges for interpretation and research synthesis.
J Clin Epidemiol. 2017;86:39-50.
2. Sinha I, Jones L, Smyth RL, Williamson PR. A systematic review of studies that aim to
determine which outcomes to measure in clinical trials in children. PLoS Med. 2008;5:e96.
3. Krause KR, Bear HA, Edbrooke-Childs J, Wolpert M. What outcomes count? A review
of outcomes measured for adolescent depression between 2007 and 2017. J Am Acad
Child Adolesc Psychiatry. 2018;58:61-71.
4. Weisz JR, Kuppens S, Ng MY, et al. What five decades of research tells us about the
effects of youth psychological therapy: a multilevel meta-analysis and implications for
science and practice. Am Psychol. 2017;72:79-117.
5. Henderson JL, Cheung A, Cleverley K, et al. Integrated collaborative care teams to
enhance service delivery to youth with mental health and substance use challenges:
protocol for a pragmatic randomised controlled trial. BMJ Open. 2017;7:e014080.
6. Tyler KM, Normand SL, Horton NJ. The use and abuse of multiple outcomes in
randomized controlled depression trials. Contemp Clin Trials. 2011;32:299-304.
7. Maassen GH. The standard error in the Jacobson and Truax Reliable Change Index: The
classical approach to the assessment of reliable change. J Int Neuropsychol Soc. 2004;10:
888-893.
8. Ebrahim S, Vercammen K, Sivanand A, et al. Minimally important differences in patient
or proxy-reported outcome studies relevant to children: a systematic review. Pediatrics.
2017;139. pii: e20160833.
9. McGlothin AE, Lewis RJ. Minimal clinically important difference: defining what really
matters to patients. JAMA. 2014;312:1342-1343.
500 www.jaacap.org
Offringa is also with the Institute of Health Policy, Management and Evaluation,
University of Toronto, Ontario, Canada.
The authors have reported no funding for this work.
All the authors have contributed significantly to the conception, design,
execution and/or writing of the manuscript, and no honorarium, grant, or other
form of payment was given to anyone to produce the manuscript. All authors
have seen and approved the submission of this version of the manuscript and
take full responsibility for the manuscript.
Disclosure: Dr. Monga has received grant funding support from the Canadian
Institutes of Health Research (CIHR) and has received royalties from Springer
for her book Assessing and Treating Anxiety Disorders in Young Children.
Dr. Szatmari has received grant funding support from the CIHR and has
received royalties from Guilford Press for his book A Mind Apart: Under-
standing Children With Autism and Asperger Syndrome. Drs. Offringa and
Butcher have received grant funding support from CIHR.
Correspondence to Suneeta Monga, MD, Department of Psychiatry, Hospital
for Sick Children, 555 University Avenue, Toronto, ON, Canada, M5G 1XP;
e-mail: suneeta.monga@sickkids.ca
0890-8567/$36.00/ª2019 American Academy of Child and Adolescent
Psychiatry
https://doi.org/10.1016/j.jaac.2019.08.468
10. Boers M, Kirwan JR, Wells G, et al. Developing core outcome measurement sets for
clinical trials: OMERACT filter 2.0. J Clin Epidemiol. 2014;67:745-753.
11. Altman DG, Royston P. The cost of dichotomising continuous variables. BMJ. 2006;
332:1080.
12. Knight GP, Zerr AA. Introduction to the Special Section: Measurement equivalence in
child development research. Child Dev Perspect. 2010;4:1-4.
13. De Los Reyes A, Augenstein TM, Wang M, et al. The validity of the multi-informant
approach to assessing child and adolescent mental health. Psychol Bull. 2015;141:
858-900.
14. Glasziou P, Altman DG, Bossuyt P, et al. Reducing waste from incomplete or unusable
reports of biomedical research. Lancet. 2014;383:267-276.
15. Chan AW, Tetzlaff JM, Altman DG, et al. SPIRIT 2013 Statement: Defining standard
protocol items for clinical trials. Ann Intern Med. 2013;158:200-207.
16. Gamble C, Krishan A, Stocken D, et al. Guidelines for the content of statistical analysis
plans in clinical trials. JAMA. 2017;318:2337-2343.
17. Butcher NJ, Monsour A, Mew EJ, et al. Improving outcome reporting in clinical trial
reports and protocols: study protocol for the instrument for reporting planned endpoints
in clinical trials. Trials. 2019;20:161.
All statements expressed in this column are those of the authors and do not
reflect the opinions of the Journal of the American Academy of Child and
Adolescent Psychiatry. See the Instructions for Authors for information about
the preparation and submission of Translations.
Journal of the American Academy of Child & Adolescent Psychiatry
Volume 59 / Number 4 / April 2020