Professional Documents
Culture Documents
com
Neuromuscular
RESEARCH PAPER
Neuromuscular
Statistical analysis
Table 1 Demographic and clinical characteristics of the patients
We combined CTT and RA approaches to investigate the follow-
(n=485)
ing psychometric properties of ALSFRS-R.
Characteristic
Neuromuscular
were 0.97, 0.98, 0.034 and 0.040, respectively), thus confirming of performance ( patient functional ability minus item difficulty)
the multidimensionality of ALSFRS-R. shown on the x axis. The ‘thresholds’ correspond to the inter-
As for RA, rating scale diagnostics showed that response levels sections (ie, the probabilistic midpoint) between two adjacent
of each item (score 0–4) did not comply with the pre-set criteria response curves. Whether the responses to the items are consist-
for category functioning (average measures, thresholds, etc.). ent with the metric estimate of the underlying construct is indi-
Accordingly, the number of levels was revised, adopting a solu- cated by the ordered set of ‘thresholds’ for each item.
tion able to maximise both statistical performance and clinical Applying this collapsing procedure, the RA showed that all
meaningfulness. For the first 11 items, the 5 original response items included in each of the three subscales (bulbar function:
levels were reduced to 3, always collapsing level ‘0’ with ‘1’, item Nos 1–3; motor function: item Nos 4–9; and respiratory
and level ‘2’ with ‘3’. For item No 12 ‘respiratory insufficiency’, function: item Nos 10–12) fitted the respective constructs to
the best solution was obtained collapsing the three central measure, except for item No 9 ‘climbing stairs’ (infit
levels, thus obtaining the following three response options: Mnsq=1.64; outfit Mnsq=1.53) and item No 12 ‘respiratory
2=no respiratory insufficiency; 1 = use of BiPAP; 0 = invasive insufficiency’ (outfit Mnsq=1.56), which showed an unexpect-
mechanical ventilation. By way of example, a typical graphical edly high variability in the observed data compared with the
presentation of these results is shown in figure 1. In figure 1A, Rasch model prediction.
the graph shows that the probability of using response levels ‘1’ The other main results of RA for each subscale are shown in
and ‘2’ in that item is never modal (ie, higher than that of the figure 2 and table 3; distribution of subject functional ability
other levels). In figure 1B (after combining original level ‘0’ and item difficulty, reliability indices and results regarding the
with ‘1’, and ‘2’ with ‘3’), the probability of selecting each of three PCAs of the standardised residuals (analysing the variance
the three revised response levels (score 0–2) is a clear function explained by the Rasch factor and the first residual factor). The
Neuromuscular
three subscales demonstrated different levels of sample item logits in each subscale, whereas item difficulty span was more
matching (the best was for bulbar function, the worst for limited, ranging from 1.89 logits (bulbar function) to 3.84 logits
respiratory function). Subject ability span was more than 10 (respiratory function). Item separation reliability was high in the
Table 3 Subject functional ability and item difficulty levels, reliability indices and principal component analysis of the standardised residuals for
each of the three ALSFRS-R subscales
Subscale 1 (item 1–3) Subscale 2 (item 4–9) Subscale 3 (item 10–12)
Average subject functional ability levels (range) 0.39 (−5.91 to 5.91) −0.74 (−5.50 to 5.60) 0.90 (−6.17 to 6.08)
Item difficulty levels (range) −1.09 to 0.80 −1.24 to 1.80 −2.03 to 1.81
Person separation reliability 0.78 0.82 0.69
Cronbach’s α 0.88 0.91 0.82
Variance explained by the Rasch factor (%) 55.4 59.2 60.6
Eigenvalue of the first residual factor 1.6 2.4 1.5
ALSFRS-R, Amyotrophic Lateral Sclerosis Functional Rating Scale-revised.
Neuromuscular
three subscales, while the person separation reliability was suffi- Thus our results suggest a rethinking of item ‘climbing stairs’
cient or good in the bulbar and motor subscales, and borderline and a clarification of the conceptual framework and measure-
(0.69) in the respiratory subscale. No PCA of the standardised ment strategy of the whole subscale ‘respiratory function’,
residuals presented residual correlations >0.30, thus confirming including the provision of detailed guidelines for its compil-
the local independence of the items in each subscale. DIF ana- ation. More generally, there is a lack of standardised method-
lysis showed no difference in responses due to gender, age or ology for ALSFRS-R administration25 and no formal interview
disease duration. instructions.
Concern about the metric properties of ALSFRS-R has
DISCUSSION already been expressed.1 Our findings confirm what clinicians
In clinical trials, we measure constructs (ie, ‘latent’ variables, know: the interpretation of a total raw score of ALSFRS-R is
such as functioning), perform statistical tests on the scales’ raw hampered by ambiguities due to its different metric meanings
scores and draw conclusions. The appropriateness of these con- for the different ALS forms. This problem is likely to be compli-
clusions strongly depends on the metric quality of the selected cated by the presence in ALSFRS-R of a typical phenomenon of
measures, and it has a crucial influence on patient care, drug ordinal summed rating scales; the relationship between total raw
efficacy and health policies. scores and linear Rasch transformed measures of global function
Unambiguous interpretation requires that a score represent a is not linear but ogival.26 As patients approach the bottom of
single attribute (dimension). Otherwise, one could not be sure if the scale, each raw score point represents an increasing metric
two individuals with the same score are, in fact, comparable. distance, yet it appears that patients are ‘slowing down’ in their
This problem hampers understanding of clinical trial outcomes, worsening because it becomes increasingly difficult for them to
which in turn has consequences for selecting interventions for lose further raw score points. This finding would imply a
individual patients.22 reduced sensitivity of the raw scores to a change occurring in
Our main finding is that the ALSFRS-R presents a series of high and low functioning ALS subjects,27 28 and on the other
drawbacks that corrupt its metric quality. hand it underlines the complex relationship between progression
The ALSFRS-R items showed good internal consistency accord- of disease and modification of ALSFRS-R raw scores.1 29
ing to CTT, with a Cronbach’s α even higher than in the original Care should be taken in interpreting our data. First, our non-
paper,4 but our dimensionality analysis argues against the validity probability sample might compromise the study’s external valid-
of summing the ALSFRS-R items into a single score. Our data ity. Nevertheless, our sample was a large cross section of
clearly indicate the presence of three different domains (bulbar, patients with a broad spectrum of disease severity, drawn from
motor and respiratory function): in each domain an aspect of func- three different tertiary ALS clinics. Second, we cannot exclude
tional status can be independently assessed but domain scores the fact that some specific (linguistic, cultural or technical) char-
cannot be simply summed to obtain an overall functional status acteristics of the Italian version of the ALSFRS-R could have
measure in ALS. These three functions are clinically meaningful somewhat influenced our results, although our version was
and correctly represent the underlying structure of the domains checked using a thorough procedure of ‘forward/backward
investigated by ALSFRS-R.4 23 24 translation’ followed by pilot testing and expert revision,
RA showed that some rating categories of ALSFRS-R did not without any semantic difficulties being found.
comply with the set criteria for category functioning. This may Our findings suggest that ALSFRS-R fails to satisfy rigorous
be due to rater difficulty in discerning among the five levels of measurement standards and should be, at least in part, revised.
functional ability. As an example, in item No 7 (‘turning in bed Valid inferences on the efficacy of treatment trials require high
and adjusting bed clothes’) the wording of categories ‘0’ (help- quality outcome measures. At present, we believe that
less) and ‘1’(can initiate but not turn or adjust sheets alone) ALSFRS-R should be considered as a profile of mean scores
does not allow a clearly distinct ranking of functional ability from three different domains (bulbar, motor and respiratory
and could introduce error variance rather than metric informa- functions) more than a global total score. Further studies on
tion into the ratings. The same can be said, in the same item, ALSFRS-R using modern psychometric methods are warranted
for the categories ‘2’ (can turn alone or adjust sheets, but with to confirm our findings and refine the metric quality of this
great difficulty) and ‘3’ (somewhat slow and clumsy, but no help scale, through a step by step process.
needed).
Collapsing the scoring options into a 3 level rating for all Acknowledgements We thank the patients and their families for their
items improved the measurement quality of the scale, providing collaboration in this study.
a simpler and more distinct idea of the level of functioning Contributors Study concept and design: FF, GM, AG, PV and AC. Acquisition of
represented by each rating level, without loss of measurement the data: GM, PV and AC. Analysis and interpretation of the data: FF, GM, AG, PV
information.18 These findings show that there is space for a and AC. Drafting of the manuscript: FF, GM, AG, PV and AC. Critical revision of the
refinement of ALSFRS-R by item rewording and/or reduction of manuscript for important intellectual content: FF, GM, PV and AC. Obtained
funding: GM and AC. Administrative, technical and material support: FF, GM, AG,
option number.
PV and AC. Study supervision: FF, GM, PV and AC. AC had full access to all of the
After collapsing the categories, fit statistics showed two data in the study and takes responsibility for the integrity of the data and the
misfits: item Nos 9 and 12. The misfit of item 9 (‘climbing accuracy of the data analysis. All authors have approved the submitted version of
stairs’) is in line with the clinical observation that different the paper.
environmental factors (ie, home architecture) and personal atti- Funding This work was funded in part by Ministero della Salute (Ricerca Sanitaria
tudes can produce high variability in this response, unexpected Finalizzata, RF-MAU-2007-643050) and Centro Nazionale per la Prevenzione e il
by the Rasch model. The high outfit value of item 12 ‘respira- Controllo delle Malattie (grant 31, 2009). The research leading to these results has
received funding from the European Community’s Health Seventh Framework
tory insufficiency’ is due to the presence of subjects without dys- Programme (FP7/2007–2013) (grant agreements Nos 259867 and 278611).
pnoea and orthopnoea (less difficult items) but on permanent
Competing interests FF has received research support from the Italian Ministry of
ventilation (most difficult item); although the finding is clinically Health (Ricerca Finalizzata). GM has received research support from the Italian
understandable, it demonstrates an additional serious bias of the Ministry of Health (Ricerca Finalizzata). AC serves on the editorial advisory board of
scale. Amyotrophic Lateral Sclerosis and has received research support from the Italian
Neuromuscular
Ministry of Health (Ricerca Finalizzata), Regione Piemonte (Ricerca Finalizzata), diagnosis of amyotrophic lateral sclerosis. Amyotroph Lateral Scler Other Motor
University of Torino, Federazione Italiana Giuoco Calcio, Fondazione Vialli e Mauro Neuron Disord 2000;1:293–9.
onlus and European Commission (Health Seventh Framework Programme); he serves 14 Brooks B. Amyotrophic lateral sclerosis clinimetric scales. In: Herndon RM, ed.
on a scientific advisory board for Biogen Idec and Cytokinetics. Handbook of neurologic rating scales, 2nd edn. New York: Demos Medical
Ethics approval The study was approved by the local ethics committees of the Publishing 2006;93–144.
three clinical centres involved. 15 Horn JL. A rationale and test for the number of factors in factor analysis.
Psychometrika 1965;30:179–85.
Provenance and peer review Not commissioned; externally peer reviewed. 16 Jöreskog KG, Sörbom D. LISREL 8 user’s reference guide, 2nd edn. Lincolnwood:
Scientific Software International, 1999.
REFERENCES 17 Hu L, Bentler PM. Cutoff criteria for fit indexes in covariance structure analysis:
1 Orrell RW. Motor neuron disease: systematic reviews of treatment for ALS and SMA. conventional criteria versus new alternatives. Struct Equat Model 1999;6:1–55.
Br Med Bull 2010;93:145–59. 18 Bond TG, Fox CM. Applying the Rasch model: fundamental measurement in the
2 Beghi E, Chiò A, Couratier P, et al. The epidemiology and treatment of ALS: focus human sciences, 2nd edn. Mahwah, New Jersey: Lawrence Erlbaum Associates,
on the heterogeneity of the disease and critical appraisal of therapeutic trials. 2007;219–33.
Amyotroph Lateral Scler 2011;12:1–10. 19 Linacre JM. Optimizing rating scale category effectiveness. J Appl Meas
3 Swash M. We have a problem: why have ALS trials been negative? Amyotroph 2002;3:85–106.
Lateral Scler 2007;8:259. 20 Linacre JM. A user’s guide to WINSTEPS-MINISTEP: Rasch-model computer
4 Chiò A, Canosa A, Gallo S, et al. ALS clinical trials: do enrolled patients accurately programs. Program manual 3.68.2. Chicago, Illinois: WINSTEPS.com. 2009.
represent the ALS population? Neurology 2011;77:1432–7. 21 Franchignoni F, Giordano A, Sartorio F, et al. Suggestions for refinement of
5 The ALS CNTF treatment study (ACTS) phase I-II Study Group. The amyotrophic the Disabilities of the Arm, Shoulder and Hand Outcome Measure (DASH):
lateral sclerosis functional rating scale. Assessment of activities of daily living in A factor analysis and Rasch validation study. Arch Phys Med Rehabil
patients with amyotrophic lateral sclerosis. Arch Neurol 1996;53:141–7. 2010;91:1370–7.
6 Cedarbaum JM, Stambler N, Malta E, et al. The ALSFRS-R: a revised ALS functional 22 Hagell P, Nygren C. The 39 item Parkinson’s disease questionnaire (PDQ-39)
rating scale that incorporates assessments of respiratory function. BDNF ALS Study revisited: implications for evidence based medicine. J Neurol Neurosurg Psychiatry.
Group (Phase III). J Neurol Sci 1999;169:13–21. 2007;78:1191–8.
7 Kaufmann P, Ley G, Thompson JLP, et al. The ALSFRSr predicts survival time in an 23 Ohashi Y, Tashiro K, Itoyama Y, et al. Study of functional rating scale for amyotrophic
ALS clinic population. Neurology 2005;64:38–43. lateral sclerosis: revised ALSFRS (ALSFRS-R). No To Shinkei 2001;53:346–55.
8 Kollewe K, Mauss U, Krampfl K, et al. ALSFRS-R score and its ratio: a useful 24 Campos TS, Rodríguez-Santos F, Esteban J, et al. Spanish adaptation of the revised
predictor for ALS-progression. J Neurol Sci 2008;275:69–73. Amyotrophic Lateral Sclerosis Functional Rating Scale (ALSFRS-R). Amyotroph Lateral
9 Gordon PH, Miller RG, Moore DH. ALSFRS-R. Amyotroph Lateral Scler Other Motor Scler 2010;11:475–7.
Neuron Disord 2004;5(Suppl 1):90–3. 25 Kasarskis EJ, Dempsey-Hall L, Thompson MM, et al. Rating the severity of ALS by
10 Kaufmann P, Levy G, Montes J, et al. Excellent inter-rater, intra-rater, and caregivers over the telephone using the ALSFRS-R. Amyotroph Lateral Scler
telephone-administered reliability of the ALSFRS-R in a multicenter clinical trial. 2005;6:50–4.
Amyotroph Lateral Scler 2007;8:42–6. 26 Grimby G, Tennant A, Tesio L. The use of raw scores from ordinal scales: time to
11 Hobart JC, Cano SJ, Zajicek JP, et al. Rating scales as outcome measures for clinical end malpractice? J Rehabil Med 2012;44:97–8.
trials in neurology: problems, solutions, and recommendations. Lancet Neurol 27 Traynor BJ, Zhang H, Shefner JM, et al. Functional outcome measures as clinical
2007;6:1094–105. trial endpoints in ALS. Neurology 2004;63:1933–5.
12 Conrad KJ, Smith EV Jr. International Conference on objective measurement. 28 Wicks P, Massagli MP, Wolf C, et al. Measuring function in advanced ALS:
Applications of Rasch analysis in health care. Med Care 2004;42(Suppl 1):I1–6. validation of ALSFRS-EX extension items. Eur J Neurol 2009;16:353–9.
13 Brooks BR, Miller RG, Swash M, et al. World Federation of Neurology Research 29 Gordon PH, Cheng B, Salachas F, et al. Progression in ALS is not linear but is
Group on Motor Neuron Diseases. El Escorial revisited: revised criteria for the curvilinear. J Neurol 2010;257:1713–7.
These include:
References This article cites 25 articles, 2 of which can be accessed free at:
http://jnnp.bmj.com/content/84/12/1340.full.html#ref-list-1
Email alerting Receive free email alerts when new articles cite this article. Sign up in
service the box at the top right corner of the online article.
Notes