See

discussions, stats, and author profiles for this publication at: http://www.researchgate.net/publication/10668720

Methodological issues and complementary

therapies: Researching intangibles?
ARTICLE in COMPLEMENTARY THERAPIES IN NURSING AND MIDWIFERY SEPTEMBER 2003
DOI: 10.1016/S1353-6117(03)00042-8 Source: PubMed

CITATIONS

DOWNLOADS

VIEWS

14

43

71

1 AUTHOR:
Bernie Carter
University of Central Lancashire
165 PUBLICATIONS 583 CITATIONS
SEE PROFILE

Available from: Bernie Carter

Retrieved on: 03 August 2015

METHODOLOGICAL ISSUES AND COMPLEMENTARY THERAPIES:

RESEARCHING INTANGIBLES?

Bernie Carter PhD, BSc, RSCN, SRN.

Professor of Childrens Nursing, Associate Head for Research & Development
Department of Nursing, University of Central Lancashire
Preston PR1 2HE, United Kingdom
Tel: 01772 893720/02
Fax: 01772 892998
email: bcarter@uclan.ac.uk

METHODOLOGICAL ISSUES AND COMPLEMENTARY THERAPIES:

RESEARCHING INTANGIBLES?

ABSTRACT
The increasing drive to adopt evidence-based practice within the mainstream health
service creates a sense of urgency for high quality, rigorous research to support CAM. The
RCT is seen as the gold standard for allopathic research. However, the tenets of the RCT
cannot simply be just picked up and applied to CAM research. Critics of the RCT propose
that it fractures and fragments the essence of many complementary therapies. Challenges
including standardisation, blinding, randomisation, practitioner influence, placebos, and
controls are explored and some possible solutions are presented. CAM researchers need to
be creative so that they capture some of the intangibles that currently slip through the
reductionist net of the RCT.

METHODOLOGICAL ISSUES AND COMPLEMENTARY THERAPIES:

RESEARCHING INTANGIBLES?

Complementary medicine should be evaluated as rigorously as conventional

medicine to protect the public from charlatans and unsafe practices, but many
practitioners of complementary medicine are reticent about evaluation of
their practice. Sceptics maintain that this is because of fear that investigations
will find treatments ineffective and threaten livelihoods. In defence, many
practitioners argue that research methods dissect their practice in a
reductionist manner and fail to take into account complementary medicines
holistic nature leading to invalid evaluation (Mason et al., 2002)
INTRODUCTION
In the above statement Mason et al. (2002) succinctly summarise some of the fundamental
issues and debates surrounding research and complementary therapies. These issues relate
to rigour, protection, scepticism, efficacy, effectiveness, safety, vested interests as well as the
more philosophical debates around holism and reductionism. The need to confront some of
these research issues reflects not only an increasing medical pluralism but also the drive to
adopt evidence-based practice within allopathic medicine. Complementary therapies
certainly create a conundrum and a challenge for researchers (Redwood 2002) but these
core research challenges are not solely the domain of complementary therapies. Within this
paper I will explore and untangle some of these issues and consider ways in which
complementary therapy research could, perhaps, progress.
Schools of thought or a case of tunnel vision?
There are probably as many different schools of thought about how best to approach
complementary therapy research as there are people thinking about doing research. In this,
complementary therapy research is little different to many other areas of health related
research where proponents of a particular method or philosophical perspective hold their
approach as being the one true way. These schools of thought could be divided broadly,
and some would say crudely, into two main approaches.
1.

The first supported by those who propose that complementary therapies should be
researched using double-blind randomised control trials as the primary method.
The argument for the use of the gold standard RCT is that complementary
therapies should have to compete and be subject to the same sort of
methodological testing as allopathic treatments. The supporters of this school see
other approaches as being less rigorous and less able to produce robust evidence.
3

2. The second approach broadly takes a stand against the notion of the RCT as the
gold standard and proposes that the controlled, detached objectivity of the RCT
fractures and fragments the whole essence of complementary therapies.
Proponents of this school believe that the RCT is only capable of reducing the
therapy to some of its constituent parts and studying them in isolation.
There are problems with both of these arguments, particularly when they are portrayed as
simplistically as I do in the above points. However, they do sum up opposite ends of the
complementary research debate continuum. To a greater or lesser degree, the same
arguments - pro versus anti RCT - can be seen in many elements of health care research.
They reflect, again to a greater or lesser extent, the old qualitative versus quantitative
debate that raged particularly fiercely in the nursing literature a number of years ago
before people adopted a more rational, considered and pragmatic approach to research.
However, the problems that did, and still do divide researchers, continue to create
challenges as to how to research complementary therapies. The problem with buying into
any single school of thought is that it lends itself to tunnel vision, whereby only your way of
thinking or working is valued and there is little respect given to those who oppose your
way. As with all research, the research problem itself should guide the researcher in their
choice of methods, rather than have the methods constrain the research problem.
Complementary therapies: a victim of their own success?
Complementary therapies have become increasingly popular not only in the UK but also in
Europe, USA and Australia (Walker and Anderson, 1999). The increasing demand and the
increase in access to therapies through mainstream health care is running parallel to the
imperative and drive towards evidence-based practice within the National Health Service.
Currently the sort of evidence base that would convince the sceptics simply does not exist
(Hilsden and Verhoef, 1999). Indeed some would contest whether the sort of evidence that
the sceptic seeks is possible to produce without fracturing the therapy. CAMs are being
called to demonstrate their efficacy, effectiveness and safety to be assured of an equal
place within the NHS. In some ways complementary therapies are a victim of their own
success since they can no longer be seen as a marginal issue for health care delivery. The
statistics reflect the fact that CAM is now increasingly impinging on mainstream
conventional NHS care. As many as 40% of General Practitioner partnerships in England
provide access to CAM for NHS patients (Coates et al., 1998). The move is now towards
integration of CAM and conventional medicine and this integration could be smoothed if
there was some form of consistency in the ways in which evaluation and research were
undertaken. With conventional medicine currently holding the more powerful position the
call is for complementary therapies to be researched within the dominant research method

- the randomised controlled trial. But to what extent is this possible or reasonable? And
why does the RCT currently hold the moral high ground of health research?
THE RCT: GOLD STANDARD OR OLD STANDARD?
The RCT is viewed by many researchers within health care as simply the best research
method for demonstrating substantive findings about whether an intervention has efficacy,
and is effective and safe. The RCT holds a seemingly impregnable position in terms of its
value as a research approach. In terms of its fundamental tenets (objectivity, control,
power, randomisation and blinding) - the RCT, on paper, is hard to beat. There is a shared
understanding of the RCT amongst the community of scientists, researchers, policy and
decision makers and allopathic medical practitioners. The more closely a study adheres to
the above benchmarks of methodological quality, the more rigorous and convincing it is
likely to be. This is not to say, however, that all RCTs are performed well and that their
findings are not open to question. In Tang et als., (1999) review of published RCTs of
traditional Chinese medicine they identified problems with methodological quality.
Leibovici (1999) states that even with the best methodology, it is not easily guarded from
inadvertent introduction of bias and from fraud (p1630). Despite these reservation it is
often asserted that a properly conducted RCT is the epitome of scientific, empirical proof for
health related research. But is it?
The RCT developed from within a very clear paradigm, a very specific way of seeing the
world. Paradigms act as maps and according to Cotgrove (1982) they:
'... constitute guidelines for getting about and for identifying and solving
problems. Above all, paradigms provide the framework of meaning within
which 'facts' and experiences acquire significance and can be interpreted. But
they have a normative as well as a cognitive dimension, indicating not only
what is but what ought to be done' (Cotgrove, 1982, p26)
The scientific paradigm provides a strong framework or frame of reference for research that
responds well to a reductionist approach. It may be the most appropriate way of testing
the efficacy of a single component, such as a single medication. Reductionist approaches
may work as good theoretical descriptions in terms of the constrained experimental reality
of the researcher in the laboratory. But as Visintainer (1986) explains they rarely can be
directly applied within the complex 'other', and maybe more 'real' world of the practitioner
providing therapy and care. How can the scientific paradigm cope with the complexity of a
holistic intervention? Indeed it could be argued that complementary therapies are being
sold short if individual elements of them are subjected to research that strips them of

context and isolates them from their system of healing/medicine. As Nahin and Straus
(2001) state that investigators are:
. faced with either designing a trial of a single intervention that does not
accurately reflect true clinical practice or undertaking a multifaceted
intervention trial that is complicated to design and implement (p162)
SO WHAT ARE THE CHALLENGES OF CT TO RESEARCH METHODOLOGY?
In the past complementary therapy research has been accused of and shown to have
methodological shortcomings (e.g., Linde et al., 2001) that raise serious questions about the
efficacy, effectiveness and safety of the interventions studied.
Undoubtedly the specialness of complementary therapies poses problems and challenges
for researchers in designing good studies but there is more than a hint of double standards
in the way that CTs are considered in comparison to conventional medicine. Chalmers
(1998) quoted by Bower (1998) stated:
Critics of complementary medicine often seem to operate a double standard,
being far more assiduous in their attempts to outlaw unevaluated
complementary therapy practice than unevaluated orthodox practices
The application of double standards may be more than just a little irritating, but it does not
mean that complementary therapy researchers should turn their backs on the need for
good research design. It is in everyones best interests for research to be designed,
performed, analysed and disseminated to the best possible standards (even if there is a
debate as to what those standards should be).
So why does complementary therapy research challenge the gold standard?
According to Ernst (2001) randomised controlled trials of CAMs are:
often more difficult and methodologically challenging that RCT of other types
of interventions. Due to the nature of most CAM modalities and the conditions
they are used for, such RCT often need to be large, of long duration and
require expensive therapists time. In turn this means that CAM research is
expensive and requires high levels of expertise in terms of trial design.(p532)
As stated earlier there are a number of issues that really need to be explored so as to fully
appreciate the concerns and challenges faced when researching CAM (see Figure 1). These
concerns include issues including randomisation, blinding, controls, placebos, shams,
protocols, standardization, outcome measures, inclusion/exclusion criteria, retention and
drop-out rates. In addition there are practitioner and client issues that reflect the very

interactive and engaged types of relationships that characterise CAM and the thorny
problem of determining what exactly a therapy is doing.

Randomisation
In RCT subjects are randomised to the study. This ensures that the two or more groups are
comparable so that any extraneous variables have a similar effect on all/both groups.
However, the holistic and collaborative philosophy that underpins most CAM is predicated
on joint decision-making. Thus, randomisation can be seen to be interfering with the
treatment process and may be avoided in CAM research designs. Feder and Katz (2002)
suggest that blinding and randomisation substantially distorts the context of homeopathic
prescribing potentially weakening its effects. Despite the concerns that some CAM
researchers may have with randomisation some ways around the problem have been
suggested. Brewin and Bradley (1989) propose an approach called partial randomisation
whereby patients are given the choice to indicate a treatment preference. Those that
express a preference are given their chosen treatment and the rest are randomised. This
does provide some leeway for patient preference although it is not altogether without
limitations. Aickin (2002) in a paper titled Beyond Randomisation proposes designadaptive allocations that virtually guarantee better balance than randomization and
that these are particularly appropriate where resources are scarce, such as in CAM studies
and where underpowered studies might lead to premature termination of promising
research paths.

Blinding
Blinding is an important element of the RCT as it helps to isolate the placebo effect. RCTs
are usually double blind where the researcher and the patient are blind to their
treatment. Triple blind studies, where the therapist, researcher and patient are all blinded,
are less common.
It is possible, although sometimes very challenging, for a trial to be single blind (i.e., the
patient is blinded to their treatment). However, it is often even more challenging for the
therapist to be blinded. One way that partially gets round the difficulties of double blinding
within trials is to bring in another person (who is blinded) to assess the outcomes. This
reduces the effect that the non-blinded therapist might have on the outcomes.

Matching controls and matching interventions

RCTs often depend on matching the control group as closely as possible with the
intervention group. This again tries to ensure that the impact of any variables will be the

same within both groups. The challenge with CAM is that it is often problematic finding a
closely matched control group. The less closely matched the less robust the ensuing study.

Standardization, protocols and variability between different therapists

Well designed, robust RCTs operate within strictly agreed protocols and with very
standardised practice relating to all aspects of the intervention. Standardization is a
regulatory ideal within a RCT. Holistic practitioners often feel uncomfortable delivering
standardized care and want to provide best therapeutic practice which may have tensions
with study design. Whilst it may be possible to standardise practice within allopathic
practice it has been much harder to reduce variability between practitioners in CAM
research. Historically, CAM has been noted for its lack of standards in terms of training and
practice and this has an effect on subsequent practice. This is likely to become less
problematic as occupational standards become more universal. In the meantime, if RCT
design is to be used with CAM, treatment manuals and treatment protocols need to be
developed and adhered to. Schnyer and Allen (2002) state that treatment manuals
provide a precise way to train and supervise practitioners, enable evaluation of conformity
and competence, [and] facilitate the training process. The inclusion of more than one
therapist (using standardised practice) in the study will allow effects to emerge across
individuals. Although there is no easy answer to this, practitioners either need to stay within
the research protocol or research studies have to be pragmatic in their design (Vickers,
2000).

Inclusion and exclusion criteria

These criteria are usually tightly and precisely defined within RCTs and they are often
predicated on diagnostic criteria. In CAM research it is more difficult to be sure of
generating a homogenous group of participants with a tightly defined diagnosis since CAM
do not work with the same diagnostic approach. Whilst it is important to maintain
appropriate inclusion and exclusion criteria, Ernst et al., (1997) propose that a possible way
forward is the combination of orthodox and unorthodox criteria.

Sound outcome measures

Reliable outcome criteria have been established for allopathic medicine over a period of
time and many have high levels of credibility and acceptance. However, existing outcome
measures may not be appropriate for CAM and they may miss out important aspects of
care that allopathic interventions do not engage with or produce. Measures that are
sensitive to CAM outcomes as well as having resonance with existing measures need to be
developed as a matter of urgency.

Drop-out issues
Data that are vital to the appreciation of the overall strength of a studys findings are the
drop-out statistics, that is, the number of people who were recruited to the study but who
did not complete either the full course of treatment or research. Some of the literature (ref)
suggests that drop-out data are not consistently reported in CAM studies. This is a reporting
problem that can be relatively easily addressed by good standards within the research
design and in any subsequent publications.

Placebos and shams

Placebos and shams are necessary part of some RCTs. Placebos and shams are used in
studies to try and help determine whether or not the participants belief about the
treatment or intervention they are receiving is effecting them. There are two main types of
placebo. The ideal placebo, as its name suggests, ideally replaces the intervention and it is
therapeutically inert. Ideal placebos are not easy to find and so often a reasonable
placebo is used and whilst it is not completely therapeutically inert it is reasonably inert.
Attaining an ideal or reasonable placebo or sham is problematic and some practitioners
would claim that therapists who are required to use shams actually distort their practice.
The aim is always to use the best placebo/sham possible but even within allopathic trials,
researchers have to accept that it will have limitations.

Symptom/disease focus
Usually in allopathic medicine, the RCT has its focus on the specific effect of an intervention
on symptom(s)/disease. This is another area where the philosophical approach to care
differs as most CAM are far less focussed on disease or diagnostic groups and much more on
an individualised assessment of the client within the knowledge framework of the therapy.

Influence of the practitioner and the practitioner-client relationship

RCTs are characterised by a scientific, objective and detached approach to the outcomes of
the study. This is not to infer that allopathic researchers do not care, simply that they must
remain distant to the findings. Objective detachment is a considerable departure for most
CAM practitioners where a subjective, engaged, helping, holistic approach to healing is
utilised. Any study design that required a practitioner to work against their philosophical
framework would be inappropriate. Clients also bring high expectations with them to their
treatment sessions. The solution is to retain the holistic, reciprocal elements within the
research study but to state this explicitly and ensure that the design accepts this. The
practitioner-client relationship is an element in CAM that can be readily explored using
qualitative methodology.

FROM NOTES OF CAUTION TO WAYS AHEAD?

Instead of simply condemning the RCT out of hand, the previous critique shows that CAM
can be amenable to RCT, albeit in a modified form. This article was born from some of the
thinking that was undertaken in designing a small scale study evaluating Bowen Therapy
on clients with frozen shoulder. The practitioners involved wanted a study that would
provide sound evidence, that would be understood by medical practitioners and yet which
would also capture the essence of Bowen therapy. Within the limited resources and time
frame available a compromise had to be reached that offered some hard (and by that the
general understanding was statistical) evidence as well as some softer data. I guess in some
ways the practitioners wanted the statistics to present to people working in a bio-medical
model but were intrinsically more interested in the qualitative data for themselves.
Hard choices always have to be made in respect to any research design. In many ways it is
problematic lumping all CAM together as if it is homeogenous. CAM is diverse. It
incorporates different techniques, methodologies and approaches (Owen et al., 2001) and
as such each therapy really deserves individual consideration in terms of research design.
RCTs have much to offer CAM but they have limitations (Hilsden and Verhoef, 1999). Other
methodological approaches are needed either to fill the gaps that cannot be addressed by
RCTs or to focus on research questions that could never be addressed by a very structured,
scientific approach. Mixed method approaches can complement each other (Sale et al.,
2002) and contribute to an overall stronger research design for CAM. Observation studies,
for example, can allow pragmatic investigation of areas that are not amenable to RCT
(Nahin & Straus, 2001). Qualitative approaches help researchers understand the meaning,
beliefs and expectations that patients ascribe to CAM interventions (Verhoef 2002). In
order for CAM to compete in the tough world that is allopathic medicine they still have to
be able to play the same games, only play them better. They also need to be creative in
developing new ways of researching to help capture some of those intangibles that slip
through the reductionist data net. CAM research is not static: the quality of the studies
being undertaken is steadily improving and the quantity of applied health research is
increasingly rapidly (Vickers, 2000). Better funding, support and resources are needed to
further enhance the quality of CAM research. The challenge for research methodology is to
fuse both qualitative and quantitative approaches (Mason et al., p832). From that fusion it
will be possible to build some exciting, rigorous research that address therapeutically
important, contextualised and relevant research questions.

10

REFERENCES
Aickin M 2002 Beyond Randomization. The Journal of Alternative and Complementary
Medicine. 6: 765-772
Bower H 1998 Double standards exist in judging traditional and alternative medicine.
British Medical Journal 316: 1694
(http://www.bmj.com/cgi/content/full/316/7146/1694/b)
Coates JR, Jobst KA, Fielding S, Fisher F, Holgate S, Mills S, Peters D, Rees L, Lewith GT,
Marks I, Monkton J, Reilly D, Wadsworth A, Williams M, Daniels R, Ersser S, Fitter M,
Richardson J, Watkins A, Whitelegg M (1998) Integrated healthcare: a way forward
for the next five years? A discussion document from The Prince of Wales's Initiative on
Integrated Medicine. Journal of Alternative & Complementary Medicine 4(2):209-47.
Ernst E 2001 Commentary: Science friction complementary/alternative medicine on the
stony road from opinion to evidence. International Journal of Epidemiology 30:531532
Ernst E, Siev-Ner I, Gamus D 1997 Complementary medicine a critical review. Israel
Journal of Medical Science 33: 808
Feder G, Katz T 2002 Randomised controlled trials for homoeopathy. Who wants to know
the results? BMJ 321: 498-499
Hilsden R J, Verhoef M J 1999 Complementary therapies: evaluating their effectiveness in
cancer. 38(2): 101-108
Leibovici L 1999 Alternative (complementary) medicine: a cuckoo in the nest of empiricist
reed warblers. British Medical Journal 319: 1629-1631
Linde K, Jonas W B, Melchart D, Willich S 2001 The methodological quality of randomised
controlled trials of homeopathy, herbal medicines and acupuncture. International
Journal of Epidemiology 30: 526-531
Nahin R L, Straus S E 2001 Research into complementary and alternative medicine:
problems and potential. British Medical Journal 322: 161-163
Mason S, Tovey P, Long A F 2002 Evaluating complementary medicine: methodological
challenges of randomised controlled trials. BMJ 325: 832-834
M O, Vithoulkas G, Van Haselen R 2003 Clinical Trials of Classical Homeopathy: Reflections
on Appropriate Research Designs The Journal of Alternative & Complementary
Medicine, 9(1): 105-111

11

Redwood D 2002 Methodological challenges in evaluation of complementary and

alternative medicine: issues raised by Sherman et al. and Hawk et al. The Journal of
Alternative and Complementary Medicine. 8(1): 5-6
Owen D K, Lweith G, Stephens C R 2001 Can doctors respond to patients increasing interest
in complementary and alternative medicine? BMJ 322: 154-157
Sale J E M, Lohfeld L H, Brazil K 2002 Revisiting the Quantitative-Qualitative Debate:
Implications for Mixed-Methods Research Quality and Quantity 36(1): 43-53
Schnyer R.N. & Allen J.J.B. (2002)Bridging the Gap in Complementary and Alternative
Medicine Research: Manualization as a Means of Promoting Standardization and
Flexibility of Treatment in Clinical Trials of Acupuncture. The Journal of Alternative &
Complementary Medicine, 8(5): pp. 623-634.
Tang J-L, Zhan S-Y, Ernst E 1999 Review of randomised controlled trials of traditional
Chinese medicine. BMJ 319: 160-161
Verhoef M J, Casebeer A L, Hilsden R J 2002 Assessing Efficacy of Complementary Medicine:
Adding Qualitative Research Methods to the "Gold Standard" The Journal of
Alternative & Complementary Medicine 8(3): 275-281
Vickers A 2000 Complementary medicine BMJ 321: 683-686
Visintainer A 1986 The nature of knowledge and theory in nursing. IMAGE: Journal of
Nursing Scholarship 18: 32-38
Walker L G, Anderson J 1999 Testing complementary and alternative therapies within a
research protocol. European Journal of Cancer 35: 1614-1618

12


Figure
1: Overview of some of the challenges posed by CAM to RCT design

Randomisation:
may distort practice and
create tensions in the
therapeutic
relationship
Randomisation

Shams and Placebos:

attaining the ideal or a
reasonable placebo or sham is
problematic and can distort
practice.

Standardization /
protocols:
practitioners often resistant to
following protocols and
standardising practices

Practitioner-client
relationship:
the subjectivity and
reciprocity of the relationship
can distort RCT.

Blinding:
very difficult to achieve true
double/triple blinding. Single
blinding usually achievable

Symptom/disease focus:
CAM tends to focus less on the
disease and more on
individual assessment/
presentation of the client.

Drop out issues:

retention and drop-out
statistics are not consistently
reported within CAM
literature

Influence of Practitioner:
the subjective engaged
approach of most CAM is at
odds with objective
detachment of RCT.

Matching Controls /
Interventions:
problematic finding a closely
enough matched control or
intervention group.

Strict inclusion/exclusion
criteria:
difficult to generate a
homogenous subject group as
allopathic diagnostic criteria
not usually applied in CAM.

Use of sound outcome

measures:
current outcome measure
may be inappropriate for
CAM or miss important
aspects of care

Variability between
practitioners:
variation between
practitioners due to lack of
occupational standards in
some CAM.

13