10/9/2018 EBSCOhost

Record: 1
Title: The implications of psychotherapy research for clinical practice and
service development: Lessons and limitations.
Authors: Roth, Anthony D.. University Coll London, Sub-Dept of Clinical Health
Psychology, London, England
Parry, Glenys
Source: Journal of Mental Health, Vol 6(4), Aug, 1997. pp. 367-380.
NLM Title Abbreviation: J Ment Health
Page Count: 14
Publisher: United Kingdom : Taylor & Francis
ISSN: 0963-8237 (Print)
1360-0567 (Electronic)
Language: English
Keywords: psychotherapy research designs in relation to clinical practice &
treatment outcomes
Abstract: Describes the difficulties of psychotherapy research designs in relation
to clinical practice. Key findings of a research review are outlined by
highlighting the importance of general factors such as the therapeutic
alliance which transcend diagnosis and therapy type. Using material
from a review of NHS psychotherapy services, research findings are
translated into more clinically effective and cost effective services.
Overall, there is evidence for the efficacy of the psychological therapies,
although evidence for the efficacy of specific therapies in relation to
specific conditions is not strong, and some therapies are better
researched than others. Ways in which the psychotherapy research
agenda could be influenced by service priorities are outlined and policy
implications of research are discussed. (PsycINFO Database Record (c)
2016 APA, all rights reserved)
Document Type: Journal Article
Subjects: *Clinical Practice; *Experimental Design; *Psychotherapeutic
Outcomes; *Psychotherapeutic Processes
PsycINFO Classification: Psychotherapy & Psychotherapeutic Counseling (3310)
Research Methods & Experimental Design (2260)
Population: Human
Format Covered: Print
Publication Type: Journal; Peer Reviewed Journal
Release Date: 19980401
Digital Object Identifier: http://dx.doi.org/10.1080/09638239718699
Accession Number: 1997-43048-004
Database: PsycINFO

http://web.a.ebscohost.com/ehost/delivery?sid=96f8e169-4ca2-46a1-b44e-3430b2662f88%40sessionmgr4008&vid=9&ReturnUrl=http%3a%2f%2fwe… 1/15
10/9/2018 EBSCOhost

THE IMPLICATIONS OF PSYCHOTHERAPY RESEARCH FOR CLINICAL PRACTICE AND

SERVICE DEVELOPMENT: LESSONS AND LIMITATIONS
Abstract
Evidence-based practice is increasingly discussed and practised within the NHS. Psychotherapy research
clearly has a part to play in informing such practice, but the differing agendas of researchers, clinicians and
purchasers need to be recognised if research is to make a constructive contribution. A good starting point is to
acknowledge that research and clinical practice are not isomorphous, because the methodologies
underpinning research inevitably distort outcomes. This makes it risky to generalise uncritically from research
to practice, although it is equally true that good practice should take note of accumulating evidence about
evidence of efficacy. Drawing on recent academic and NHS policy reviews, this article notes that overall there
is good evidence for the efficacy of the psychological therapies, although evidence for the efficacy of specific
therapies in relation to specific conditions is not strong, and some therapies are better researched than others.
There is also evidence that it may be unhelpful to privilege 'brand names' over pantheoretical factors, such as
the therapeutic alliance or therapist skilfulness, both of which appear important to outcomes. The article
discusses the policy implications of research, and suggests some future research directions.

Introduction
The principle that health care planning, commissioning and provision should be guided by evidence of efficacy
is increasingly familiar to all those involved in these processes. Although a relatively new concept in the field of
mental health, a focus on evidence-based practice is evident in a number of policy decisions and reviews in the
UK, the US and elsewhere (e.g. Doll, 1996; Depression Guidelines Panel, 1993; American Psychiatric
Association, 1993). Several forces combine to make this a philosophy of care central to planning. Economic
pressures,within health-care systems where a third party pays for treatment, highlight the need to achieve a
tight balance between expenditure and outcome. Increasingly consumers of health care, although less
preoccupied with the economics of their treatment, expect to be informed about the likely benefits of any
therapy. Psychological therapists, surprisingly, often seem more sanguine about issues of efficacy. In part this
can be attributed to the fact that relatively few practitioners are familiar with studies of efficacy, or feel it is of
little relevance to their work (Morrow-Bradley & Elliot, 1986). Those who are aware of the research literature
complain about the difficulties of measuring outcomes, the lack of generalisability of research to the clinical
situation, and the fact that outcome research, which usually reports normative data, does not help resolve
impasses in the clinical situation, which are essentially idiographic.

The first author recently reviewed psychotherapy outcome research (Roth & Fonagy, 1996) as part of the NHS
Executive strategic review of psychological therapies, led by the second author (Department of Health, 1996).
Both reviews emphasise and encourage the use of research evidence as a basis for planning and providing
psychological therapy services. We are aware that their publication has created concern among psychologists,
psychiatrists and psychotherapists about the inappropriate use of evidence-based directives. Such concern is
legitimate, and does not simply reflect an unwillingness to open practice to scientific scrutiny. For example,
Grahame-Smith (1995) speaks for many when he questions whether one important purpose of research-based
health care is to shackle anarchic, financially irresponsible clinical professionals to the will of health service
managers. Undoubtedly the rhetoric of evidence-based health care can be recruited to serve a number of
different causes. Practice guidelines based on evidential reviews are certainly being used to redirect current
practice in a way which many clinicians would question. For example, in the US, the AHCPR review of primary
care treatments for depression promotes psychopharmacological over psychological treatments, relegating the
latter to secondary care or to those cases who fail to respond to medication (Depression Guideline Panel,
1993). In our view, this guideline is based on a narrow reading of the research evidence, and is hard to defend.
http://web.a.ebscohost.com/ehost/delivery?sid=96f8e169-4ca2-46a1-b44e-3430b2662f88%40sessionmgr4008&vid=9&ReturnUrl=http%3a%2f%2fwe… 2/15
10/9/2018 EBSCOhost

Nonetheless, there will be occasions when research evidence does indeed contradict cherished clinical
impressions or narrow school allegiances. How can psychotherapists, patients and those planning or paying
for services make informed judgements about the quality of the evidence presented in the research literature?

From the outset, a clear distinction needs to be drawn between the interests of the researcher and those of the
clinician. Researchers are concerned to confirm or disconfirm a hypothesis; clinicians focus on improving the
lot of their clients. Researchers devise studies to maximise internal validity - to ensure that the study is
sufficiently well controlled to allow reliable statements about causation to be made. Internal validity can be
reduced by many factors, most obviously by a failure to control for alternative sources of variance which may
better account for the results found. For example, if a study found that one therapy had more efficacy than
another, this would be spurious if patients treated by this method were substantially less disturbed than those
treated by the alternative therapy. Achieving appropriate levels of control often means distorting patterns of
treatment delivery so that they no longer map easily onto standard clinical practice. This is, however, entirely
defensible. Clinicians often question the generalisability - or external validity - of research. However reasonable
this objection, it makes little sense to resolve the tension between research and practice by suggesting that
research should be identical to therapy as carded out in the field. Such an enterprise would fatally undermine
internal validity. It seems more appropriate to accept the inevitable distortions which are produced within
clinical trials, to understand how they are produced, and to use this knowledge to make an informed link
between research and clinical practice. Paradoxically, recognising that research and practice are not
isomorphic is a first step towards the appropriate use of research.

In this paper, we describe the difficulties of psychotherapy research designs in relation to clinical practice,
outline some key findings of the research review, highlighting the importance of general factors such as the
therapeutic alliance which transcend diagnosis and therapy type. Then, using material from the review of NHS
psychotherapy services, we discuss how research findings can be translated into more clinically effective and
cost effective services. Finally we outline ways in which the psychotherapy research agenda could be
influenced by service priorities.

Better by design? Research strategies and their limitations

Over the past 40 years, the randomised controlled trial (RCT) has come to occupy a central place in research.
This research design is often referred to as the 'gold standard' of evidential enquiry, and the systematic
appraisal of results from these trials is now an established feature in many areas of health care delivery (for
example, reviews issued by the Cochrane library rely exclusively on data from RCTs). Random assignment of
patients to treatment or control group powerfully protects internal validity, as any alternative sources of
variance to the one under study, whether known or unknown, should not differ systematically between the
groups.

Despite this major advantage, randomised controlled trials do have their problems, and their supremacy as a
psychotherapy research strategy is not yet established (Aveline et al., 1995). If patients have strong
preferences for one treatment over another, they may refuse to enter the trial. If they do enter the trial and are
assigned to their non-preferred treatment, they are more likely to drop out, and this differential attrition on the
basis of preference introduces a non-random element into the design. For example, where very high rates of
attrition are observed, patients who complete therapy may be atypically receptive and even if these patients do
well, there may be little generalisation from the research setting to the general population. Thus, although
Gunderson et al. (1984) and Alanen et al. (1985) were able to show some benefit for psychodynamic work with
patients with a diagnosis of schizophrenia, this effect was only evident in the small percentage of patients who

http://web.a.ebscohost.com/ehost/delivery?sid=96f8e169-4ca2-46a1-b44e-3430b2662f88%40sessionmgr4008&vid=9&ReturnUrl=http%3a%2f%2fwe… 3/15
10/9/2018 EBSCOhost

continued in therapy. On this basis, it would be inappropriate to recommend such treatment for most people
with schizophrenia.

It is, in practice, very difficult to achieve textbook randomisation. For example, GPs are ingenious in finding
ways to get their patients into the treatment they want. No-treatment control conditions are often far from inert,
as patients inevitably find alternative sources of help. Running a tight design is even more difficult in studies
with uncooperative or chaotic patients, such as some people with personality disorders. Given the costs of
conducting trials, most studies use the smallest sample size with sufficient statistical power to detect clinically
significant differences. Yet where samples are small, and despite randomisation, there often turn out to be
unplanned differences between the groups on important variables such as symptom severity. Investigation of
longer term outcome at follow up is also problematic, since only those who improved in the first place can
relapse and again, a non-random element is introduced. There are methodological steps which can be taken to
address all these difficulties, for example, stratification, partial randomisation, following up the 'intention to treat'
sample, post hoc statistical correction of bias, re-randomisation and so on. However, none of these entirely
resolves the significant difficulties in conducting satisfactory RCTs. These very real difficulties should always be
considered when considering the appropriate research design, or the merits of the evidence yielded by any
individual study.

Quite apart from the considerable difficulties in conducting a well designed RCT, even the best of them have
major limitations with regard to external validity. Three unavoidable threats to external validity are the
homogeneity of patient samples, the standardisation of treatment and randomised allocation.

Patient samples in most RCTs are usually diagnostically homogeneous. The rationale for this is straightforward
- over time, and even without treatment, samples containing patients with mixed diagnoses would show a
range of outcomes as a simple consequence of the natural history of differing mental health conditions.
Diagnostic homogeneity reduces an obvious source of outcome variance. Unfortunately such purity of
diagnosis is less usual in clinical settings, where co-morbidity is common, making it difficult to generalise
results obtained by research trials to routine clinical practice. Although researchers are now examining
outcomes in people presenting with common co-morbidities (such as the frequent co-occurence of depression
and Generalised Anxiety Disorder), it will be some time before the results of such trials are available.

Current research trials go to some lengths to ensure that the treatment being carried out is standardised, both
in terms of its duration, and by using manuals which closely guide the procedures that the therapist carded out.
Usually sessions are taped, so that researchers can ensure that the delivered therapy adheres to the treatment
manual. Where the aim of a trial is to examine the benefits of one therapy over another, it is crucial that at the
very least the therapy was delivered in a pure form. This level of control enables researchers to make
recommendations about which therapeutic techniques appear to have efficacy. However, outside the research
setting, such adherence to a pure form of therapy is rare- therapists are likely to aim for a balance between the
need for a uniform application of therapy while making adaptive choices in order to maintain the integrity both
of the treatment and the therapeutic alliance. Indeed, the clinically skilled psychological therapist is likely to
base his or her treatment choices on a formulation of an individual's problem which may take into account such
factors as the individual's developmental history, characteristic patterns of interpersonal relating, a functional
analysis of behaviour, their cognitive style, coping skills, social support system and so on. There is therefore an
intrinsic tension between the individually tailored approach of the skilled clinician, and the more manualised
and formulaic applications seen in some research trials.

http://web.a.ebscohost.com/ehost/delivery?sid=96f8e169-4ca2-46a1-b44e-3430b2662f88%40sessionmgr4008&vid=9&ReturnUrl=http%3a%2f%2fwe… 4/15
10/9/2018 EBSCOhost

Randomisation removes any choice about which treatment a patient receives; in the clinic such choices are
central to the initial stages of treatment. Although in practice allocation is rarely guided by empirical evidence
(and is prone to error), there is a negotiated best fit between patient preference and aptitude, available
treatments, available therapists and the assessor's judgement of treatment of choice. Such processes are
inaccessible to research using randomised designs, and it is likely that increased attrition from therapy will
result from poorer matching of patients to treatments.

RCTs are therefore an imperfect tool; almost certainly their results are best seen as one part of a research
cycle, most appropriately as a test of the capabilities of a reasonably well-developed therapy. They offer the
best answer to a circumscribed set of questions issues related to outcome rather than to process, and to
efficacy rather than effectiveness[1]. Acknowledging these limitations does not detract from the usefulness of
the RCT as a research strategy, and data from them should be neither overvalued (for example, by excluding
all other forms of evidence when evaluating psychotherapy) nor rejected (for example, by seeing their results
as inevitably trivial and irrelevant to real psychotherapy).

Clinical practitioners often prefer open trials and single-case studies as alternatives to randomised trials,
largely on the basis that both these designs have greater external validity. Both are scientifically flawed,
however. The open trial, characterised by the absence of a contrast treatment, is especially suited to a unit
which wishes to examine the application of one treatment to a cohort of patients. However, unless the effect
size of the treatment is such that its superiority to alternative treatments is unequivocally clear - and few if any
psychological procedures achieve such status - information about the relative efficacy of the treatment under
test is needed. Two areas of study illustrate this requirement clearly. For many years uncontrolled trials
suggested that eclectic in-patient therapy was effective in treating anorexic patients (e.g. Crisp, 1980).
However, a subsequent RCT conducted by Crisp's clinical group (Crisp et al., 1991) suggested, among other
findings, that in-patient treatment was no more effective than therapy conducted on an out-patient basis, with
no clear medical indicators for selection to one or the other form of service delivery. A second RCT (Hobbs et
al., 1996) has questioned the advisability of routine post-trauma counselling. People who had been involved in
severe road traffic accidents either received counselling while in hospital, or no intervention. At one year, those
in receipt of the counselling tended to exhibit higher levels of PTSD. Whatever interpretation is placed on these
findings, it is useful to bear in mind that an open trial of such counselling would have revealed a rate of PTSD
which (because of a lack of contrast intervention) may have been taken as a reduction in psychopathology,
rather than the possible increase implied by the above studies.

An alternative to the use of larger scale designs is the single case study, or the planned use of a series of
single-case investigations. By focusing on the pattern of change in the individual patient, and on change in the
individual rather than in the group, such designs avoid some of the limitations and problems of group designs.
Individual variation is not lost in statistical aggregation, and there is no assumption that individual patients
should represent the whole class of patients from which they are drawn. Single case studies avoid the problem
of the uniformity myth which assumes that samples of patients classified (for example) by diagnosis do not
vary in other important respects. However, there are important restrictions in the use of the single case study.
For example, although there are often reports of patients treated under conditions relatively close to standard
practice, patients are not sampled in a way which would enable conclusions to be extended to wider
populations.

The role of qualitative research should also be acknowledged as part of a broader research strategy. This form
of research is sometimes misunderstood; it is not about eschewing measurement, or letting people talk for
themselves, nor is it an easy option of small scale, non-generalisable research. It does emphasise meaning,

http://web.a.ebscohost.com/ehost/delivery?sid=96f8e169-4ca2-46a1-b44e-3430b2662f88%40sessionmgr4008&vid=9&ReturnUrl=http%3a%2f%2fwe… 5/15
10/9/2018 EBSCOhost

and particularly the negotiation and social construction of meanings. The strengths of the qualitative method for
health care generally are in such areas as patient perceptions, understanding health behaviours, or exploring
taken-for-granted practice in healthcare. It can also enhance and complement quantitative research, for
example in explaining unexpected findings. In psychotherapy research, for example, intensive process analysis
complements outcome research, by examining the meaning of key events in contextual detail (Rice &
Greenberg, 1984; Safran et al., 1990). The contextual approach is a vital antidote to the assumption within
outcome research that psychotherapy is analogous to a drug and can be studied in a similar way. Stiles &
Shapiro (1994) argue that the drug metaphor underlying psychotherapy breaks down at several points,
especially because both therapists and patients actively construct the process of therapy - an example of
which would be the interplay that arises as therapists monitor patients responses to interventions, making
iterative adjustments to what they deliver.

Review of psychotherapy research findings

Roth & Fonagy (1996) collated studies of the efficacy of therapies in relation to mental health conditions
commonly seen in clinical settings, as part of the NHS Strategic review of psychological therapies (Department
of Health, 1996). Overall, they found good evidence for the efficacy of psychological therapies, although
evidence for the differential efficacy of therapies was not strong. This conclusion is tempered by the fact that
some therapies are less well researched than others, and the quality of the research evidence often varies
across orientations. Thus, while there are many studies of behavioural and cognitive behavioural therapies,
evidence for the efficacy of psychodynamic approaches is limited, partly by the paucity of trials, and also
because a number of these studies were conducted by proponents of alternative therapies, with the result that
the psychodynamic therapy studied is offered by inexperienced therapists.

The fact that research is unrepresentative of widely practised therapies raises a number of problems, both
pragmatic and philosophical, since an absence of evidence in support of a therapy poses problems of
interpretation. While some would argue that unvalidated therapies should not be practised, this may be
inappropriate - for some innovative therapies, absence of evidence may simply reflect the inevitable lag in
developing research trials. For other therapies, especially psycho-dynamic, this may reflect a prior research
position. The NHS Review made recommendations on how these difficulties can be addressed in service
provision and commissioning, summarised below. In the meantime, it is important to remember that where
research has not been undertaken, absence of evidence for efficacy is not evidence of a lack of efficacy.

Despite a general conclusion that there is only modest evidence of differential effect between therapies, there
are instances where more certain therapeutic recommendations can be made. For example, while the
evidence that interpersonal therapy, cognitive behavioural therapy and brief psychodynamic therapies can all
be beneficial in the treatment of depression, there is reasonably strong evidence for the superior benefits of
cognitive behavioural therapies in generalised anxiety disorder, for the use of interpersonal therapy in eating
disorders, and of panic control therapies in relation to panic disorder without agoraphobia.

A further caution against assuming that all therapies are of equal value arises where a number of therapeutic
components are offered as a package of care, without recognition that while some elements may be mutative,
others (in relation to outcomes) may be redundant. Thus there is only weak evidence for the additive effect of
cognitive techniques over response prevention in the treatment of compulsive phenomena in obsessive-
compulsive disorder. Similarly, exposure appears to be a central element in the effective treatment of social
phobia, which is sometimes managed using complex packages of care which includes cognitive components.

http://web.a.ebscohost.com/ehost/delivery?sid=96f8e169-4ca2-46a1-b44e-3430b2662f88%40sessionmgr4008&vid=9&ReturnUrl=http%3a%2f%2fwe… 6/15
10/9/2018 EBSCOhost

Outcomes expected from a psychological intervention will vary in relation to the presenting problem, and care
should be taken neither to over- nor understate the likely benefits of treatment. Often outcomes will be modest
but clinically valuable. For example, data from a number of trials (NIMH: Elkin, 1994; Hollon et al., 1992;
Shapiro et al., 1994) suggest that after brief intervention, approximately 50% of patents presenting with acute
depression will reach a recovery criterion. In other presentations a higher rate of recovery may be likely - for
example, Clark et al. (1994) treating patients with panic disorder without agoraphobia had a recovery rate of
90% at one-year follow-up. For other conditions rates of recovery may be low, as appears to be the case in
some of the sexual dysfunctions (Roth & Fonagy, 1996).

Not all patients will benefit from short-term treatment - indeed the natural history of some disorders suggests
that they follow a cyclic path of remission and relapse which therapy may ameliorate, rather than abolish. Thus,
while brief treatments are often effective, they may not be sufficient in longer-term management. For many,
more severely ill patients a maintenance model may be useful, within which acute treatment is followed by
planned but intermittent contact. In the University of Pittsburgh study (Frank et al., 1990) this pattern of care
significantly delayed or even prevented further episodes in depressed individuals with a prior history of
remission and relapse.

Although sometimes seen as incompatible or even alternative forms of treatment, there is good evidence that
medication and psychotherapy can be seen as adjunctive and complementary, although their relative roles
may differ between conditions. Thus in depression there is good evidence for their complementary use, while
there appears to be little additional benefit to medication in panic disorder with agoraphobia. For other
conditions, such as schizophrenia, clinicians would be ill-advised to use psychological interventions in the
absence of medication, despite the known efficacy of family treatments for this group.

Despite the apparent certainty of the above, there is insufficient evidence to allow the prescription of different
therapies on the basis of diagnosis alone. 'Brand names' are only partially predictive, if at all, and as discussed
below, the therapeutic alliance is an important, although complex, factor in outcome. Therapists differ in their
skilfulness; in some trials outcome variance attributable to therapists is greater than that contributed by
therapeutic method. Manualisation of therapies, while assuring some degree of consistency in treatment
delivery, remains controversial (Persons, 1991) and of doubtful utility beyond research trials.

Efficacy and the therapeutic alliance

Traditionally, much research has emphasised the comparative outcomes of different types of psychotherapy,
despite the fact that the method of intervention is only one of a number of influences on outcome (and despite
the fact that eclectic therapy is probably more commonly practised than any 'pure' form). Process research, in
contrast, directs attention not only to the individual characteristics of patients and therapists, but examines the
interplay between therapists and patients, as represented by the therapeutic alliance.

Although having its origins in psycho-dynamic therapy, in a research context the alliance is usually viewed
pantheoretically, comprising three related elements: (i) the degree to which the patient and therapist are
bonded (for example, through an experience and expression of mutual respect); (ii) the extent to which they
agree on the goals of therapy, and (iii) the degree to which they agree on the relevance and the
appropriateness of the tasks being used to reach these goals. The alliance is more than an underpinning to
therapy - although the concept transcends theoretical schools of therapy, it is through the medium of a specific
therapy approach that it is embodied. In a sense, the alliance both carries successful therapy, and in itself
exemplifies that therapy in operation. When considering the practical implications of research into the alliance,

http://web.a.ebscohost.com/ehost/delivery?sid=96f8e169-4ca2-46a1-b44e-3430b2662f88%40sessionmgr4008&vid=9&ReturnUrl=http%3a%2f%2fwe… 7/15
10/9/2018 EBSCOhost

it is worth bearing in mind that this duality makes the alliance easier to conceptualise and measure than to
implement or teach.

Researchers have developed a number of instruments to detect the quality of the alliance, usually by rating
recordings of therapy using scales that distinguish the contribution to the alliance of the patient, therapist or the
two in interaction. Alternatively, therapists and patients are asked to rate their sense of the alliance directly.
Meta-analytic review of an increasingly substantial research literature (Hovarth & Symonds, 1991)
demonstrates the significant contribution of the alliance to outcome, yielding effect-sizes of 0.2 and 0.3 for the
early and late alliance, respectively. Given the difficulty of relating process variables to outcome, this
apparently modest effect-size is impressive, particularly given the bias towards caution used by Hovarth &
Symonds in their analysis.

The significance of the alliance is further emphasised by recently published data from two trials (NIMH trial-
Elkin, 1994; University of Pittsburgh trials, Hollon et al., 1992), both widely known because of their
demonstration of efficacy in relation to therapeutic method. In both studies the alliance was positively
associated with outcome, an effect present across differing therapeutic methods (e.g. Krupnick et al., 1996;
Castonguay et al., 1996). Qualitative examination of therapy sessions showed the negative impact of practising
technique in the absence of the alliance. Thus Castonguay et al. (1996) found that increased use of cognitive
technique was associated with poorer outcomes, although this effect was abolished if the alliance was
controlled for. This finding echoes that of Henry et al. (1993), who found that in increasing their technical
competence, trainee psychodynamic therapists neglected to attend to the alliance, to the detriment of
outcomes.

Quite how the alliance exerts its impact remains unclear, and care should be taken not to reify it by definition it
is a complex admixture of factors. Nonetheless, there has been an increasing research focus on threats to the
alliance, both as a marker for mutative events within therapy, and as an indicator of therapist skilfulness. A
number of workers have suggested that patient challenges to the alliance will reflect aspects of their underlying
pathology, and that these junctures represent an opportunity for therapeutic change - and also for the
temporary or permanent loss of the patient to therapy (e.g. Safran & Muran, 1996). Current work is attempting
to model the management of these events, using task analysis and statistical mapping. Inevitably such work
directs attention to the skilfulness with which therapists manage the alliance.

The competent delivery of therapy is likely to depend on the alertness of the therapist to a range of both
interpersonal and technical factors. However, relating therapist competence to outcome is difficult. A number of
studies have attempted to gauge this by proxy indices (such as the amount of training or the number of years,
experience gained by a therapist). Using such measures, there is at best only modest support for the benefits
of experience or training. In part this may be a methodological problem - differential attrition between the
patients of novice and experienced therapists may lead to the former losing, and the latter treating, more
difficult patients, hence yielding apparently poorer outcomes for the more experienced therapists.

In the absence of longitudinal studies, the impact of experience remains uncertain, but it is clear that
experience is not equivalent to competence. Direct examination of the impact of therapists' abilities is a better
test of the relationship between competence and outcome. However, there is no clear agreement as to what
would constitute competence - there are a variety of reference points which could apply. Examples of
measures used are adherence to a technique, or the degree to which the therapist frames interpretations in the
context of a consensual formulation. Adherence appears to be highly correlated with competence, although
this may reflect some overlap in the operationalisation of these terms. There is some evidence that

http://web.a.ebscohost.com/ehost/delivery?sid=96f8e169-4ca2-46a1-b44e-3430b2662f88%40sessionmgr4008&vid=9&ReturnUrl=http%3a%2f%2fwe… 8/15
10/9/2018 EBSCOhost

competence does relate to outcome - for example, Frank and colleagues showed this in their study of long-
term maintenance therapy for depression (Frank et al., 1991). Patients receiving therapy rated as competently
delivered had a median survival time to relapse of 2 years, contrasted to a median survival time of 4 months in
those receiving poor-quality therapy. Kingdon et al. (1996) found a similar relationship between therapeutic
competence and outcome in cognitive therapy in the UK. However, therapists in Frank et al.'s study were able
to deliver competent therapy with only some of their patients; other patients of the same therapist received
poor quality therapy. This sort of result inevitably draws attention back to the therapeutic alliance. Although it is
clear that patients vary considerably in their capacity to form alliances (Piper et al., 1991; Mallinckrodt et al.,
1995), one index of the competent therapist might be their capacity to draw the more disturbed and distrusting
patient into a working relationship.

Research on the therapeutic alliance and therapist skill has important implications for how psychotherapists
are trained and accredited. Training which uses targeted goals, specific feedback and guided practice in order
to enable therapists to reach a stated criterion of competence is likely to be most effective (Luborsky, 1990).
Beutler et al. (1994) call for more rigorous and specific training methods than current supervision models
provide, suggesting that competency based training using therapy manuals may provide such an alternative.
This method has been adopted to some extent in cognitive and behavioural therapy training (Bootzin & Ruggill,
1988; Dobson & Shaw, 1988; Dobson et al., 1985; Shaw & Dobson, 1988) and although there are examples in
psycho-dynamic therapy (Moss et al., 1991; Strupp et al., 1988), measures of specific psycho-dynamic
competencies are not yet widely available.

NHS Executive strategic policy review of psychotherapy services in England

This review collated evidence and opinions from a wide range of sources, including surveys of health
authorities and NHS Trusts providing mental health and community services, followed by in-depth local
enquiries in two health districts, consultation with professional bodies, voluntary groups and other relevant
public sector agencies, reviews of evidence on psychotherapy outcomes (Roth & Fonagy, 1996), economic
evaluation of psychotherapy (Healy & Knapp, 1995) and psychotherapy needs assessment (Perry, 1995). The
work of the review was guided by an expert advisory group.

It was found that three types of psycho-therapeutic treatments are available for adults and children through
NHS mental health services; A: those integral to wider mental health care programmes, B: eclectic
psychological therapies and counselling, C: formal psycho-therapies. It appeared that eclectic therapies were
most commonly practised. Only half of Health Authorities made any specified provision for psychological
therapies, but what this was varied. Although demand for all forms of psychotherapy outstrips supply, there
was also evidence of poorly targeted, inappropriate interventions and ineffective organisation and delivery of
services, and hence of wasted resources. The review made recommendations for making services more
comprehensive, co-ordinated, user-friendly, safe, clinically effective and cost effective.

The review suggested that those commissioning services should avoid being overprescriptive in purchasing
pre-planned 'packages' of care for specific patient groups. On the other hand, another key recommendation
was that standard practice in delivering psychotherapies could be far more clearly specified through clinical
guidelines than is currently the case. These guidelines should be informed by service evaluation evidence,
formal research findings and clinical consensus and could be used to benchmark best practice and as an aid to
clinical judgement. Thus the clinician is reminded of best evidence from research, where it exists, but retains
the prerogative to make a judgement in the individual case, on the basis of assessment and case formulation,
to do something different. The guideline acts as an aide-memoire to the therapist in formalising normal

http://web.a.ebscohost.com/ehost/delivery?sid=96f8e169-4ca2-46a1-b44e-3430b2662f88%40sessionmgr4008&vid=9&ReturnUrl=http%3a%2f%2fwe… 9/15
10/9/2018 EBSCOhost

expected practice so that deviations from this practice are made thoughtfully and are justified by clinical
evidence.

Although an emphasis on standardised practice might be seen as a threat to innovation, in practice this is
unlikely. There will always be many situations where research evidence- or even clinical consensus - cannot
indicate the best or most effective approach. Solutions to clinical impasses can be suggested by the failure of
systematically applied and monitored standard practice, and hence such practice can be seen as a
springboard to new developments, which themselves become part of the cycle of research activity.

Initially, novel forms of service delivery or experimental clinical techniques are likely to be tested through case
studies or replicated n = 1 evaluation of case series. If the technique or approach appears promising,
randomised controlled trials should be undertaken to establish whether or not the treatment is efficacious,
followed by larger scale field trials carried out to evaluate its clinical effectiveness, as delivered in everyday
practice. This research strategy is dubbed the 'hourglass model' by Salkovskis (1994).

Standard practice which can be described in clinical guidelines also has the great advantage of focusing and
clarifying the audit effort, since the guideline states the right thing to do whereas the audit asks 'has the right
thing been done?' and 'has it been done right?' By targeting audit onto those processes derived from evidence-
based protocols, a clear link can be established between research and audit. In turn, this should reveal gaps in
knowledge, leading to developments in theory and new hypotheses for formal psychotherapy research.

As clinical and service audit examines the reasons for failures to deliver to a set standard of practice, it can
reveal gaps in the skills and competence of psychotherapists which prevent them from delivering a fully
effective service. This highlights the quality of professional training, new education and curriculum needs, and
integrated attention to questions of professional skills updating. It is likely that it is only by following such a
process of service delivery that both pre and post-qualification training in the psychotherapies will become
underpinned by links among theory, research and practice.

The review therefore asserts that the clinical effectiveness of NHS psychotherapy services can be improved by
developing evidence-based practice through the mechanisms of service agreements, outcomes benchmarking,
research-based clinical guidelines, clinical audit, improved professional training and dissemination of research
findings. Clinical audit and evaluation provide the necessary link between research evidence on efficacy and a
guarantee of clinically effective delivery. Commissioning services on research evidence alone does not
guarantee this.

Future research directions

Evidence-based practice in the psychotherapies requires a better research foundation than is currently
available. The NHS Executive review made a number of recommendations for areas where research is
needed, ranging from heuristic studies to randomised controlled trials and extensive field trials of clinical and
cost effectiveness of psychotherapy delivery. To these recommendations can be added other research
priorities, many of which reflect the need to extend and relate research findings to standard service settings.

Bridging the efficacy-effectiveness divide. Research into psychotherapy as delivered is lacking. There have
been few systematic studies of the effectiveness of psychotherapeutic techniques (e.g. Kirk 1983), and
empirical evidence strongly supports the contention that therapeutic impacts are lessened in clinical practice
(Weisz et al., 1995). This suggests that further direct comparisons are required.

http://web.a.ebscohost.com/ehost/delivery?sid=96f8e169-4ca2-46a1-b44e-3430b2662f88%40sessionmgr4008&vid=9&ReturnUrl=http%3a%2f%2fw… 10/15
10/9/2018 EBSCOhost

The external validity of clinical research, and hence its usefulness to the NHS, could be improved in a number
of ways - for example:

by ensuring patients in research trials are comparable to NHS patients in the severity and type of disorder;
by monitoring the fate of intent-to-treat samples, in order to estimate the effects of sample attrition, the
effects of patient preference on outcome and the characteristics of patients leaving treatment prematurely;
by estimating clinical significance of change as well as statistical significance;
by ensuring that change measures are constructed so as to monitor patterns of change across time in
individual patients (yielding data about the stability of change), supplementing the more usual reporting of
results averaged over samples;
by increasing lengths of follow up, not only to detect treatment impacts, but also to determine the degree to
which patients require further interventions

by considering modifications of randomisation procedures (for example, partial randomisation) where strong
patient preferences for one treatment over another will lead to bias from differential attrition;

Monitoring the research base. Currently there is wide variation in the extent, type and availability of empirical
evidence as it relates to specific therapies. This may relate to a lack of an empirical tradition for certain widely
practised therapies, or to the fact that innovative therapies may be at an early stage of the research cycle.
Undoubtedly, however, this picture will change, and there will be a need to maintain updated, systematic
reviews of effectiveness for all therapies. There will also be a need to integrate information about outcomes
which risks falling outside conventional reviewing systems - for example, information about innovative
therapies is particularly likely to be available as 'grey' literature, in the form of unpublished audits or as single
case studies.

Broadening the research base. There is a particularly urgent need for controlled research on clinical
effectiveness of commonly practised therapies - especially psycho-dynamic therapy. Here, however, care
should be taken to include attention to developing valid and reliable technologies to measure psychodynamic
aspects of clinical change, in addition to symptom and behavioural measures.

Refining techniques. Approaches supported by existing research evidence require further research
development - for example, to consider their application to patients with dual diagnoses and long-term care
needs. It is important to know for which patients can these approaches can be delivered effectively in a brief
form, or by generic mental health staff with specific training, in order that these techniques be made more
available to a larger number of patients.

Focusing on challenges to practice. At present, almost all research is focused on one shot interventions for
particular diagnostic groups, although many clinicians seek guidance about patients with complex and
enduring presentations. It would be useful to examine interventions for people with dual diagnoses and co-
morbidity, to explore the most appropriate forms of therapy delivery for people with more chronic presentations,
and to consider the most effective use of psycho-therapeutic intervention in long-term care plans for the more
severely ill patients. Longer term outcomes of therapies could be examined by increasing the lengths of
naturalistic follow up, and by follow-up studies of longer term outcome in previous randomised controlled trials.

Studies of therapy process. In addition to improving the yield of comparative outcome research for NHS
services, further research is required examining those therapy process factors predicting positive outcomes,

http://web.a.ebscohost.com/ehost/delivery?sid=96f8e169-4ca2-46a1-b44e-3430b2662f88%40sessionmgr4008&vid=9&ReturnUrl=http%3a%2f%2fw… 11/15
10/9/2018 EBSCOhost

irrespective of therapy type. Although it is increasingly clear that these pantheoretical factors - and especially
the therapeutic alliance - have a large part to play in determining outcomes, we need a better understanding of
how these components operate.

The measurement of therapist competence is a particular priority, not only in the context of research settings,
but also within standard practice, and especially within training. Clinicians need audit tools to help them
improve the quality of services and those involved in training need to evaluate the performance of those
qualifying and assess the overall effectiveness of the training. There is only a small amount of research on the
training process; in particular longitudinal studies of the impacts of training and supervision on outcomes are
lacking, as are studies which would indicate how to maximise the effectiveness of the supervision process
itself.

Economic evaluation. Health services research is needed to estimate the cost effectiveness of alternative
methods of psychotherapy service delivery. There may be particular benefit in focusing on areas where efficacy
data already suggests that changing care practices may lead to significant reductions in in-patient care, as is
the case in family interventions for people with schizophrenia, or in the psychotherapeutic management of
personality disorder compared with standard psychiatric care.

Acknowledgements
We are grateful to Jennie Popay, Director of the Public Health Research & Resource Centre, University of
Salford, for these points on the role of qualitative research.

1 By convention, efficacy refers to the impact of a therapy in the context of a research trial, whereas its
effectiveness is the result obtained in routine clinical practice.

References
Alanen, Y.O., Rakkolainen, V., Rasimus, R., Laakso, J. & Kaljonen, A. (1985). Psychotherapeutically oriented
treatment of schizophrenia: results of a five year follow-up. Acta Psychiatrica Scandinavia, 319, 31-49

American Psychiatric Association (1993). Practice guideline for major depressive disorder in adults. American
Journal of Psychiatry 150, 1-26.

Aveline, M., Shapiro, D.A., Parry, G. & Freeman, C.P.L. (1995). Building research foundations for
psychotherapy practice. In: Aveline, M. & Shapiro, D.A. (Eds) Research Foundations for Psychotherapy
Practice. Chichester, Wiley.

Beutler, L.E., Machado, P.P.P.& Neufeldt, S.A. (1994). Therapist variables. Chapter 7 in A.E. Bergin & S.L.
Garfield (Eds) Handbook of Psychotherapy and Behavior Change. 4th ed. New York,Wiley, pp. 229-269.

Bootzin, R.R.& Ruggill, J.S. (1988). Training issues in behavior therapy. Journal of Consulting and Clinical
Psychology, 56, 703-9.

Castonguay, L.G., Goldfried, M.R., Wiser, S., Raue, P.J. & Hayes, A.M.(1996). Predicting the effect of cognitive
therapy for depression: A study of unique and common factors. Journal of Consulting and Clinical Psychology,
64, 497-504.

Clark, D.M., Salkovskis, P.M., Hackmann, A., Middleton, H., Anastasiades, P. & Gelder, M. (1994). A
comparison of cognitive therapy, applied relaxation and imipramine in the treatment of panic disorder. British
Journal of Psychiatry, 164, 759769.
http://web.a.ebscohost.com/ehost/delivery?sid=96f8e169-4ca2-46a1-b44e-3430b2662f88%40sessionmgr4008&vid=9&ReturnUrl=http%3a%2f%2fw… 12/15
10/9/2018 EBSCOhost

Crisp, A.H. (1980). Anorexia Nervosa: Let me be. London, Academic Press. Crisp, A.H., Norton, K., Gowers,
S., Halek, C., Bowyer, C., Yeldham, D., Levett, G. & Bhat, A. (1991). A controlled study of the effect of
therapies aimed at adolescent and family psychopathology in anorexia nervosa. British Journal of Psychiatry,
159, 325-333.

Department of Health (1996). A review of strategic policy on NHS psychotherapy services in England. London,
NHS Executive.

Depression Guideline Panel (1993). Clinical practice guideline no 5; Depression in primary care. 2: Treatment
of major depression. Rockville, MD, AHCPR 93-0551 Agency for Health Care Policy & Research.

Dobson, K.S., Shaw, B.F. & Vailis, T.M. (1985). Reliability of a measure of the quality of cognitive therapy.
British Journal of Clinical Psychology, 24, 295-300.

Dobson, K.S. & Shaw, B.F. (1988). The use of treatment manuals in cognitive therapy: Experience and issues.
Journal of Consulting and Clinical Psychology, 56, 673-680.

Elkin, I. (1994). The NIMH treatment of depression collaborative research programme: where we began and
where we are. In: A.E. Bergin & S.L.Garfield (Eds) Handbook of Psychotherapy and Behaviour Change, 4th
edn. New York,Wiley, pp. 114-139.

Frank, E., Kupfer, D.J., Wagner, E.F., McEachrm, A.B. & Comes, C. (1991). Efficacy of interpersonal therapy
as a maintenance treatment of recurrent depression. Archives of General Psychiatry, 48, 1053-1059

Grahame-Smith, D. (1995) Evidence based medicine: Socratic dissent. British Medical Journal, 310, 1126-1127

Gunderson, J.G., Frank, A.F., Katz, H.M., Vannicelli, M.L., Frosch J.P. & Knapp, P.H. (1984). Effects of
psychotherapy in schizophrenia: II Comparative outcome of two forms of treatment. Schizophrenia Bulletin, 10,
564-598.

Healey, A. & Knapp, M. (1995). Economic evaluation of psychotherapy services. Paper commisioned by NHS
Executive. Unpublished.

Henry, W.P., Strupp, H.H., Butler, S.F., Schacht, T.E. & Binder, J.L. (1993). The effects of training in time-limited
psychotherapy: Changes in therapists behaviour. Journal of Consulting Clinical Psychology, 61, 434-440.

Hobbs, M., Mayou, R., Harrison, B. & Worlock, P. (1996) A randomised controlled trial of psychological
debriefing for victims of road traffic accidents. British Medical Journal, 313, 1438-1439.

Hollon, S.D., DuRubeis, R.J., Evans, M.D., Weimer, M.J., Garvey, M.J.,Grove, W.M. & Tuason, V.B. (1992).
Cognitive therapy and pharmacotherapy for depression: Singly or in combination. Archives of General
Psychiatry, 49, 774-781.

Hovarth, A.O. & Symonds, B.D. (1991). Relation between working alliance and outcome in psychotherapy: A
meta-analysis. Journal of Consulting Clinical Psychology, 38, 139-149

Kingdon, D., Tyrer, P., Sievewright, N., Ferguson, B., & Murphy, S. (1996). The Nottingham study of neurotic
disorder: influence of cognitive therapists on outcome. British Journal of Psychiatry, 169, 93-97.

http://web.a.ebscohost.com/ehost/delivery?sid=96f8e169-4ca2-46a1-b44e-3430b2662f88%40sessionmgr4008&vid=9&ReturnUrl=http%3a%2f%2fw… 13/15
10/9/2018 EBSCOhost

Kirk, J.W. (1983). Behavioural treatment of obsessional-compulsive patients in routine clinical practice.
Behaviour Research and Therapy, 21, 57-62.

Krupnick, J.L., Sotski, S.M., Simmens, S., Moyer, J., Elkin, I., Watkins, J. & Pilkonis, P.A. (1996). The role of
the therapeutic alliance in psychotherapy and pharmacotherapy outcome: Findings in the NIMH treatment of
depression collaborative research programme. Journal of Consulting and Clinical Psychology, 64, 532-539.

Luborsky, L. (1990). Theory and technique in dynamic psychotherapy-Curative factors and training therapists
to maximize them. Psychotherapy and Psychosomatics, 53, 50-57.

Mallinckrodt, B, Gantt, D.L. & Coble, H.M. (1995). Attachment patterns in the psychotherapy relationship:
Development of the client attachment to therapy scale. Journal of Counselling Psychology 42, 307-317.

Morrow-Bradley, C. & Elliot, R. (1986). Utilisation of psychotherapy research by practising psychotherapists.

American Psychologist, 41, 188-197.

Moss, S., Margison, F. & Godbert, K. (1991). The maintenance of psychotherapy skill acquisition: a two-year
follow-up. British Journal of Medical Psychology, 64, 233-236.

Perry, I. (1995). Needs assessment for psychological treatment services. Paper commissioned by NHS
Executive. Unpublished.

Persons, J.B., Thase, M.E. & Crits-Christoph, P. (1996). The role of psychotherapy in the treatment of
depession. Archives of General Psychiatry, 53, 283-290.

Piper, W.E., Azim, H.F.A., Joyce, A.S., McCallurn, M., Nixon, G.W.H. & Segal, P.S. (1991). Quality of object
relations vs interpersonal functioning as a predictor of the therapeutic alliance and psychotherapy outcome.
Journal of Nervous and Mental Disease, 179, 432-438

Rice, L.N. & Greenberg, L.S. (1984). Patterns of Change: Intensive Analysis of Psychotherapy Process. New
York, Guilford Press.

Roth, A.D. & Fonagy, P. (1996). What works for whom ? A critical review of psychotherapy research. New York,
Guilford Press.

Safran, J.D., Crocker, P., McMain, S. & Murray, P. (1990). Therapeutic alliance rupture as a therapy event for
empirical investigation. Psychotherapy 27, 154-165.

Safran, J.D. & Muran, J.C. (1996). The resolution of ruptures in the therapeutic alliance. Journal of Consulting
and Clinical Psychology, 64 447-458.

Salkovskis, P.M. (1995). Demonstrating specific effects in cognitive and behavioural therapy. In: M. Aveline &
D.A. Shapiro (Eds) Research Foundations for Psychotherapy Practice. Chichester, John Wiley, pp. 191-228.

Shapiro, D.A., Barkham, M., Rees, A., Hardy, G.E., Reynolds, S. & Startup, M. (1994). Effects of treatment
duration and severity of depression on the effectiveness of cognitive/behavioural and
psychodynamic/interpersonal psychotherapy. Journal of Consulting Clinical Psychology, 62, 522-534.

Shaw, B.F.& Dobson, K.S. (1988). Competency judgements in the training and evaluation of psycho-therapists.
Journal of Consulting and Clinical Psychology, 56, 666-672.
http://web.a.ebscohost.com/ehost/delivery?sid=96f8e169-4ca2-46a1-b44e-3430b2662f88%40sessionmgr4008&vid=9&ReturnUrl=http%3a%2f%2fw… 14/15
10/9/2018 EBSCOhost

Stiles, W.B. & Shapiro, D.A. (1994). Disabuse of the drug metaphor -psychotherapy process outcome
correlations. Journal of Consulting and Clinical Psychology, 62, 942-948.

Strupp, H.H., Butler, S.F. & Rosser, C. (1988). Training in psychodynamic therapy. Journal of Consulting and
Clinical Psychology, 56, 689-695.

Weisz, J.R., Donenberg, G.R., Han, S.S. & Weiss, B. (1995). Bridging the gap between laboratory and clinic in
child and adolescent psychotherapy. Journal of Consulting Clinical Psychology, 63, 688701

~~~~~~~~
By ANTHONY D. ROTH & GLENYS PARRY, University College London & University of Sheffield, UK

Address for Correspondence: Anthony Roth, Sub-Department of Clinical Health Psychology, University College
London, Gower Street, London WCIE 6BT, UK. Tel: 0171 380 7777 Ext. 5925; Fax: 0171 916 1989; e-mail:
a.roth@ucl.ac.uk

This article is copyrighted. All rights reserved.

Source: Journal of Mental Health

http://web.a.ebscohost.com/ehost/delivery?sid=96f8e169-4ca2-46a1-b44e-3430b2662f88%40sessionmgr4008&vid=9&ReturnUrl=http%3a%2f%2fw… 15/15