journal club (4)

Journal club 4: intervention studies

Jennifer Reids series aims to help you access the speech and language therapy literature, assess its credibility and decide how to act on your findings. Each instalment takes the mystery out of critically appraising a different type of journal article. Here, she looks at intervention studies.

READ THIS SERIES IF YOU WANT TO yy BE MORE EVIDENCEBASED IN YOUR PRACTICE yy FEEL MOTIVATED TO READ JOURNAL ARTICLES yy INFLUENCE DEVELOPMENT OF YOUR SERVICE

he intervention study tool we will explore here is suitable for appraising experimental studies which set out to measure outcomes from provision of a therapy (or other kind of intervention) whose purpose is to alleviate either a persons difficulties or their impact on the persons life. We have now covered several appraisal tools, so you may be feeling it is not always obvious which to choose. To help you sort out what a study is really about, and whether it refers to interventions, try to match its aim to one of these questions: 1. What can we observe in people who have a particular risk factor or condition? If the investigators did not provide an intervention themselves but were looking at outcomes in people who may have received one type of intervention or another, you are probably looking at an observational design. In observational studies, the researchers do not provide an intervention or otherwise seek to promote change in individuals. They examine aspects of peoples past and/or present life and seek to find factors that are associated with each other. Sometimes this may involve collecting data about the intervention(s) received. The underlying quest in observational studies is usually to identify factors which might be causally related (and thus may be incorporated into interventions to produce better outcomes in the future). We will come back to the appraisal of observational studies in a future Journal Club. 2. What can people tell us about their experience of living with a particular risk factor or condition? Does the study appear to be less about objective observation of outcomes and facts, and more about eliciting peoples subjective experiences of the impact of their condition or of being the recipient of interventions? You may find that the qualitative study appraisal tool in Journal Club 2 (Winter 2010) is most applicable. 3. Does a given therapy or intervention work? If a quick skim of the abstract and/or method of the article suggests that the author is simply describing an intervention (or range of interventions), it does not qualify as an intervention study. Instead, it may be appraised using the expert opinion tool in Journal Club 1 (Autumn 2010). If the authors are reviewing or combining the results of more than one published study, then the systematic review

tool in Journal Club 3 (Spring 2011) is more likely to be the one to use. Once you have ascertained that the study in question is indeed an intervention study, you then need to ask one further question to drill down to the right tool: 4. Does this study use a group design or a single-subject design? Did the researchers provide the intervention to one group of people and compare the group results with those of people who either received a different intervention or no intervention at all? If the answer is yes, you can use the intervention study tool in this article. If, on the other hand, the study is about one persons treatment results or a series of individuals in which there is no comparison group receiving a different or no treatment, then you are probably dealing with a single-subject design. An appraisal tool for single-subject design studies will follow in a subsequent Journal Club. Single-subject designs most often use multiple baselines to compare treated and untreated targets within a single individual, so look out for that terminology. The gold standard design for an intervention study is the randomised controlled trial (RCT). The framework in this article can also be used for other types of intervention study, such as exploratory, preliminary or pilot-study designs, but you need to use the separate tool for appraising reports of single-case study and case series designs. This is because group designs help minimise the opportunity for bias arising from the Hawthorne effect (people tend to behave or respond differently when they know they are participants in a study) and the placebo effect (people receiving treatment of any kind do better than those receiving no treatment). It is not possible to rule out these effects from research using single-subject designs. Group studies in which the data from individuals is pooled but there is no control (comparison) group can pretty much be avoided as inadequate. Since neither Hawthorne nor placebo effect are controlled, any treatment outcomes could be entirely non-specific. There is little point in demonstrating a change that could have resulted from any old intervention Pam

Enderbys description of early aphasia therapy research as trials of random niceness comes to mind again. One exception may be when research is at the pre-clinical or modelling phase of the development of a complex intervention, when you may be interested in the content of and theoretical justification for untested interventions (MRC, 2000). Having identified your intervention study, ask the following questions to appraise its quality. The issues are the ones that you will find in any of the published frameworks for appraisal of randomised controlled trials, such as Critical Appraisal Skills Programme (CASP) (PHRU, 2006) or the PEDro Scale (Physiotherapy Evidence Database, 1999). And if you want a good laugh (yes, really!) about why these issues are important, I recommend Ben Goldacres Bad Science (2009). Beware, though, as you may never feel the same way again about vitamin supplements, brain gym or homeopathy This appraisal starts the same way as for a systematic review (Journal Club 3, Spring 2011): Question 1: What question was being asked, and is this an important clinical question?

Try formulating the reviewers stated aims into a research question if they have not done so explicitly in the article. Is the question clearly focused in terms of the PICO framework: Population studied Intervention given Comparator/control(s) used Outcomes considered? Is this question important for your clinical practice? If the reviewers question does not quite fit the bill, what question(s) do you wish they had asked instead? (The study may provide a partial answer.)

18

SPEECH & LANGUAGE THERAPY IN PRACTICE SUMMER 2011

journal club (4)

Question 2: Was this a randomised controlled trial (RCT) and, if so, was it appropriately so?
Figure 1 Constraint Induced Language Therapy (CILT) A set of interventions referred to as constraint induced therapy (CIT) has been described that applies principles of use-dependent learning. Constraint Induced Movement Therapy (CIMT) is based on the notion that the potential rehabilitation of the affected limb is detrimentally influenced by the compensatory use of the unaffected limb, through a process of learned nonuse Investigations of CIMT in chronic stroke have suggested substantial increases in the amount and quality of movement which translated to real-world improvements in use of the affected limb The key principles of CIMT include (1) massed practice occurring in an enriched environment, (2) constraint of the unaffected limb during all waking hours using a sling, and (3) forced use of the affected limb Recently, Pulvermller et al. (2001) incorporated these core features of use-dependent learning in the context of language. They studied 17 inpatients with chronic aphasia who had received prior language therapy and were believed to have exhausted their recovery potential. Ten subjects participated in small group therapeutic activity intensively for three hours per day over a two-week period. this preliminary study showed significant improvement for the experimental group in the amount and quality of communication. (Maher et al., 2006, p.844)

Do the authors discuss why this study was carried out, either as a randomised controlled trial or otherwise? Most of the interventions of interest to speech and language therapists are complex interventions. Medical Research Council guidelines define these as ones which, whether therapeutic or preventative, comprise a number of separate elements which seem essential to the proper functioning of the intervention although the active ingredient of the intervention that is effective is difficult to specify. If we were to consider a randomised controlled trial of a drug vs. a placebo as being at the simplest end of the spectrum, then we might see a comparison of a stroke unit to traditional care as being at the most complex end of the spectrum. The greater the difficulty in defining precisely what, exactly, are the active ingredients of an intervention and how they relate to each other, the greater the likelihood that you are dealing with a complex intervention. (MRC, 2000, p.1) Just because the randomised controlled trial design is the gold standard for interventions does not mean researchers can or should be undertaking a randomised controlled trial of a complex intervention before the preliminary work has been done to establish an adequate description of its components: Complex interventions are built up from a number of components, which may act both independently and inter-dependently. The components usually include behaviours, parameters of behaviours (e.g. frequency, timing), and methods of organising and delivering those behaviours (e.g. type(s) of practitioner, setting and location). It is not easy precisely to define the active ingredients of a complex intervention. For example, although research suggests that stroke units work, what, exactly, is a stroke unit? What are the active ingredients that make it work? The physical set-up? The mix of care providers? The skills of the providers? The technologies available? The organisational arrangements? (MRC, 2000, p.2) So you need to ask at what stage of development this intervention is and, therefore, whether a randomised controlled trial at this stage is appropriate. In one of our local journal clubs, we reviewed an article on Constraint Induced Language Therapy (CILT) for people with aphasia (Maher et al., 2006) (figure 1). We thought their question was something like, Does Constraint Induced Language Therapy change the spoken

Figure 2 Enough jargon to sink that little pilot boat... Prospective means looking forward so its trying to answer the question, What will happen if we do x? rather than, What happened to these people in the past? (retrospective). Its use here seems a bit redundant since this is an intervention rather than an observational study. Repeated measures means you use the same tools to measure participants skills, behaviours etc. at pre-set intervals of time before, sometimes during, and after the treatment. This is one of the methods usually associated with a single-case study or case series design. This study used a mixture of standardised measures such as the Western Aphasia Battery and Boston Naming Test, and narrative analysis of a retelling of the Cinderella story, but no measures to detect more functional gains. Pilot study is using the analogy of the pilot boat which leads the way for a larger, less manoeuvrable (but more powerful) ship. Describing it as a pilot study also releases the authors from some of the constraints of the randomised controlled trial, allowing them to get a study into print that would otherwise fall foul of peer reviewers criticisms.

language behaviour of people with chronic, stroke-induced aphasia (with or without dyspraxia) compared to a similar dosage of multi-modal PACE (Promoting Aphasics Communicative Effectiveness)-type therapy? Maybe not the question we would have chosen to ask but still potentially of interest. The design was described by the authors as a prospective, repeated measures pilot study (figure 2). Not a randomised controlled trial then, but probably an appropriate design for early studies of a new therapy. Question 3: Were participants appropriately allocated to intervention and control groups?

Was the method of allocation adequately described and truly random? Each participant should have had an equal chance of being allocated to any of the study groups. It might seem a bit pedantic, putting in all that detail of sealed envelopes and independent personnel, but there is a reason for it. The whole purpose of a group study is to test whether the intervention effect you are interested in occurs when the intervention is provided for a particular

population of people with communication or swallowing difficulties. Randomisation ensures that groups are sufficiently similar to wipe out any differences in results that might occur because of individual differences. Have you ever read the results of a group study and been frustrated at the lack of detail about its impact on individuals? Well, thats exactly the point. The group study should tell you about the strength of the interventions effect irrespective of individual differences, so the study method needs to control for possible interference from inadequate randomisation. There is a large accumulation of evidence that people are influenced by knowledge of study participants and so (even if only subconsciously) may allocate to treatment rather than control groups participants who are, for example, keener, more worthy, more likely to adhere to the treatment schedule Hence the need for distance, independence and no personal contact with the participants. Researchers may use stratification to ensure equal or balanced numbers of certain types of participant such as boys and girls, or people living in three different parts of the country, or people with different levels of severity of a condition. Randomisation then takes place within each stratification group. Were the groups well balanced? Look for tables or other data describing potentially important characteristics of the study groups, and evidence that the authors have checked (with statistical tests if appropriate) that the groups are indeed similar enough to control for

SPEECH & LANGUAGE THERAPY IN PRACTICE sUMMER 2011

19

JOURNAL CLUB (4)

individual differences between participants. Sometimes, despite adequate randomisation, differences between study groups occur. If these are reported you need to ask whether they could confound the study results that is, whether the differences might account for some of the observed outcomes. In Maher et al. (2006), allocation was not random but the procedure and the characteristics of the resulting groups were well enough described. We had concerns about the very small number of participants (9), and that differences between the groups might have introduced some bias (for example, the wider age range in the PACE group; client communication preferences; presence or absence of dyspraxia). Question 4: Were participants, staff and study personnel blind to participants study group? Question 5: Can the results be applied to the local population? investigators do all the work and then are unable to demonstrate convincingly that the intervention had an effect over and above what it was being compared to. They may find that the difference between the outcomes of the different interventions was not statistically significant. It does not necessarily mean there is no effect, but you end up not knowing whether the intervention was ineffective or whether the design of the study was just not powerful enough to capture it. So no further on. This is another reason for doing adequate pre-trial pilot studies, as their results can help provide the basis for the estimates of effect size needed to do a power calculation. We felt the Maher et al. (2006) participants, recruited through advertising and having received unknown amounts of previous therapy, would probably be representative only of our younger, relatively independent and highly motivated clients. Question7: Is the intervention described in sufficient detail?

As in other appraisal frameworks you need to examine the detail to determine whether there are important differences between the population sampled for the research study and the population of your own context. It may also be useful to consider practical aspects, such as could you provide the same intervention in your setting? Evidence-based practice is often modelled as a triangle with its three components being best research evidence + expertise of the practitioner + client preference. As the impact on clinical decision-making of resources and service delivery options arising from the local context cannot be ignored, some authors include these constraints as a fourth component of evidence-based practice. Question 6: Was the selection and follow-up of the participants adequate?

Blinding is about minimising the potential for observer bias, specifically the tendency for assessors to expect more change amongst individuals in treatment than control groups. The existence of this effect, like the placebo, is very well substantiated in the medical and psychological literature, so the need for blinding is an important aspect of good research design. However, there are limitations on where and how much speech and language therapy treatment studies can employ blinding. In a single blind study, the participants do not know which group they are in, while in a double blind study, neither do the investigators. In speech and language therapy research, it is very unlikely that participants and practitioners can be blind to participants group, so it is good practice for other people to undertake the assessments and outcome measures. A bit of common sense can help. Did they employ blinding wherever it was possible to do so? If blinding was missing from some measures, does it really matter for this study? No information was given in Maher et al. (2006) on who administered the standardised measures, but the narrative sample analysis was done blind to both treatment group and whether pre-, post- or 1 month after treatment status. It seemed to us much more important that blinding was applied to these measures, so we were less concerned about the potential for observer bias from possible lack of blinding in the standardised tests.

Were there any differences in data collection that might introduce performance bias? For example, were there any differences between the groups in the time intervals between measures? Did assessments undertaken by participants differ so that one group received a lot more attention from researchers, therapists or other practitioners? Did the study have enough participants to minimise the play of chance? Look for a report of a power calculation if this is a fullscale randomised controlled trial. Power calculations estimate how many participants are needed to be reasonably sure of finding out whether an intervention really does have a specific effect. If you cant handle the stats, at least check that the authors have considered the possibility of having too few participants. The danger of this is that the

Could you replicate the intervention on the basis of the information given? Look back to the earlier quote from the Medical Research Council Guidelines. The authors should have provided a thorough description of the intervention with reference to components such as: intensity (eg. frequency and length of sessions) dosage (eg. how many hours of therapy over what length of time) scheduling (eg. distributed learning or massed practice) content (eg. targets, success criteria, resources) role/responses of therapist or other practitioner (how changes in behaviour are facilitated eg. modelling, contingent responses, scaffolding, recasting) methods used to minimise systematic differences between practitioners delivering the interventions (eg. manualising the intervention, training the practitioners, measuring fidelity). The authors should provide information on the track record of the intervention, if it has one, with a summary of the current evidence for its efficacy (delivery under ideal conditions) or effectiveness (delivery in real world settings). For innovative interventions, ask whether they are founded on sound theoretical and practice-based principles.

Critical appraisal for speech and language therapists (CASLT) Download the intervention study framework document from www.speechmag.com/Members/CASLT for use by yourself or with colleagues in a journal club.
20
SPEECH & LANGUAGE THERAPY IN PRACTICE SUMMER 2011

JOURNAL CLUB (4)

The two core components of Constraint Induced Language Therapy (CILT) are (a) massed practice and (b) forced use of speech. Maher et al. (2006) wanted to see whether (b) still produced better results if the intensity was controlled. They delivered therapy in dyads or triads using a dual card request-and-respond task (collecting pairs) with several levels of difficulty (semantic and syntactic). The PACE group could use any modality while the CILT group had to use speech. Cuing and shaping strategy were used by therapists with the CILT group only. Although their conceptualisation of the interventions felt quite narrow, both the CILT and PACE-type interventions were well enough described for us to be confident we could replicate them (although we did wonder how we would fill 3 hours a day doing the card game activities they described without losing the plot!) Question 8: Were all of the participants who entered the trial accounted for at its conclusion? You need to consider: whether the authors interpretation of numbers was sensible, how the results are expressed (eg. proportion of participants with a given outcome, or as a measurement such as mean or median differences?) how large and how meaningful this size of result is get some more practice in looking at p-values and confidence intervals without that blink and miss reaction! Try to sum up the bottom-line result of the study in one sentence this also helps you communicate the gist of your appraisal to others. For Maher et al. (2006), standardised measures suggested positive post-treatment changes in both groups. Results from the narrative measures were more equivocal, although the authors argued for the existence of a trend towards better outcomes for the Constraint Induced Language Therapy group. Interpretation was also affected by incomplete follow-up data. We concluded that the study was not powerful enough (too few participants) and had too many potential confounding factors (sources of bias) to detect any reliable difference in the effect of the two interventions. Question 10: Were all important outcomes considered so the results can be applied to practice? Attrition in this context is about how many participants were lost from the study before outcomes were measured. Authors should report this clearly and, if necessary, show how they took account of this in their results. You would have cause to worry about the study results if attrition was high and/or unequal across groups. How did the investigators handle the situation of participants wishing to move to a different group? Such moves challenge the integrity of randomisation and may compromise the study results. The solution, however counter-intuitive, is to analyse participants outcomes by the groups to which they were originally allocated. This is called intention-to-treat analysis. It is one of the core standards used internationally in randomised controlled trial quality appraisal schemes, so it is good to know this bit of jargon. One of the participants in Maher et al. (2006) allocated to the PACE group refused to use any modality other than speech. However, his results were retained and analysed within his allocated group. We can give them a point for intention-to-treat analysis even though this was not a randomised controlled trial. Question 9: What are the main results, and how are they presented? How results are expressed can have an important influence on what you perceive as the main result. thought we would need to be very sure about our selection criteria before offering Constraint Induced Language Therapy as an option for a particular client. Question 11: Should policy or practice change as a result of the evidence in this review? Consider whether any benefit reported outweighs any harm and/or cost. If this information is not reported, can you fill it in from elsewhere? You should adopt a broad view of harm here it does not just mean physical harm such as from swallowing difficulties. When people undertake an intervention, this prevents them from doing something else, and it sets up expectations that may be disappointed. How could expectations be managed in your local context? Could you provide equal or fair access to a new intervention? Does your service provide clients with information on intervention alternatives and support them to be involved in decision-making? These are important issues to consider before adopting a new or alternative intervention. Consider cost, too, not just from the perspective of the service. Intensive therapy may be a real challenge for many clients and families because of its costs in time and lost employment. You should conclude with a judgement about whether there is sufficient weight of evidence to justify implementing changes to policy or practice in your service. There was not enough evidence in Maher et al. (2006) alone to make a decision about whether Constraint Induced Language Therapy should be accessible across Fife, but we agreed we needed to look at the broader evidence base. Certainly, as described in the study, it lacked clinical feasibility for us. We need to see results from studies in which intensity and content are varied as well as studies using populations that SLTP more reflect our clients. Jennifer Reid is a consultant speech and language therapist with NHS Fife, email jenniferreid@nhs.net. Cartoons are by Fran, www.francartoons.co.uk.
References Goldacre, B. (2009) Bad Science. London: Harper Perennial. Maher, L.M., Kendall, D., Swearengin, J.A., Rodriguez, A., Leon, S.A., Pingel, K., Holland, A. & Gonzalez Rothi, L.J. (2006) A pilot study of use-dependent learning in the context of Constraint Induced Language Therapy, Journal of the International Neuropsychological Society 12, pp.843-852. Medical Research Council (2000) Guidelines on the development and evaluation of RCTs for complex interventions. Available at: http://www.mrc.ac.uk/ Utilities/Documentrecord/index.htm?d=MRC003372 (Accessed: 5 May 2011). Physiotherapy Evidence Database (1999) PEDro Scale. Available at: http://www.pedro.org.au/english/ downloads/pedro-scale/ (Accessed: 5 May 2011). Public Health Research Unit (2006) Critical Appraisal Skills Programme. Available at: http://www.phru.nhs. uk/Pages/PHD/CASP.htm (Accessed: 5 May 2011).

Could the people included in the study be different from your population in ways that would produce different results? Does your local setting differ much from that of the trial? Could you provide the same treatment in your setting, given local constraints and opportunities? If not, what changes would be needed to your services resources or to their deployment? Consider outcomes from different points of view including that of the client and family, the wider community, professionals and policy makers. These were our comments on Maher et al. (2006): We remain unconvinced about how replicable this conceptualisation of Constraint Induced Language Therapy would be in our settings getting well-matched dyads or triads would be difficult in most settings. How would you do this, for example, with someone in a care home? We thought there might be potential for delivering massed practice intensity via telemedicine or computer, although this would need a different core of therapy tasks. We worried how you would retrieve a negative outcome from using Constraint Induced Language Therapy with a client in order to get back to a total communication approach. We

SPEECH & LANGUAGE THERAPY IN PRACTICE SUMMER 2011

21