Professional Documents
Culture Documents
1093/intqhc/mzr063
Advance Access Publication: 27 September 2011
Abstract
urgency, research into the effectiveness of these programs is (i) The impact of accreditation on the quality and safety
still at an embryonic stage [18, 19] and fewer studies than the of healthcare delivery.
area deserves have embarked on an examination of their per- (ii) The efficiency of accreditation tools and systems for
formance [13, 20 – 22]. providing feedback with reliable information both to
There is a lack of empirical evidence in the literature the accreditation organizations as well as all key
exploring performance-related measures for assessment of stakeholders.
accreditation programs. Existing efforts have largely focused (iii) The impact on the capacity development of systems.
on a single aspect of accreditation’s performance and impact
[cf. 18]. Accordingly, this paper seeks to provide valuable Braithwaite et al. [20] argue that only a multi-method, multi-
insights in direct relation to the assessment of these evalua- disciplinary, multi-level research design is capable of provid-
tory mechanisms, by suggesting a number of generic ing reliable evidence on the performance and impact of
measures for their evaluation. This is an important contri- accreditation. Scrivens’ approaches of ‘experience/perception
bution to the literature and practice for two main reasons. and objective indicator’ could provide a valuable lens to look
First, the provided measures could facilitate the challenging critically into and make sense of the current trends in study-
task of assessing accreditation programs’ performance [14, ing accreditation programs [37, p. 6]. The first approach
23]. Some of the measures could also be used to support requires that the perceptions and experiences of various
646
Evaluation of accreditation † Accreditation
render a valid evaluation of accreditation programs’ perform- the experts from those countries which have comparably
ance in view of those challenges: just as Shaw [14] notifies of settled and successful accreditation systems (e.g. the USA,
the difficulty in evaluating these programs when compared Canada, Australia and France). A ‘snowball sampling’ tech-
with clinical technology. Therefore, owing to the complex nique was used at a later stage in order to identify
nature of the health sector creating the foregoing challenges, additional experts, in a way that at the end of the inter-
the perception approach might be considered safer from this views, respondents were asked to nominate other people
perspective, despite its deficiencies [37, 49]. A fairly wide who would be relevant for the purposes of the research
range of the current literature has utilized this approach as [56, p. 17]. Snowballing was helpful, because the nominator
mentioned earlier [e.g. 50, 51]. The most cautious, yet cred- was used as a form of reference to enhance the credibility
ible, outlook might be to synthesize two approaches. of this research. In addition, the new respondents were sup-
Following from this debate and considering the mixed posed to be capable of answering the questions of the
results in the previous literature upon the performance and research [56]. Overall, 35 experts were contacted, of whom
impact of accreditation programs on HCOs, this paper casts 25 replied and consequently were interviewed by email
further light on the examination of these mechanisms in during 3 months starting from May 2008.
healthcare. It aims to draw upon the ‘perceptions and experi- For the second stage, the senior managerial and clinical
ences’ of both experts and professionals to explore workable members (i.e. hospital managers, matrons, supervisors and
647
Jaafaripooyan et al.
Strengths such as being less costly, quick and more freedom the data), three main groups of measures were achieved and
for respondents, could be further attribute to this technique represented (see Tables 1– 3).
[see 57, p. 93]. However, operationalization of this technique At the end of each stage, results were discussed and
requires access to the internet by all respondents. agreed upon by the authors, in order to minimize the judg-
The experts were asked to answer the open-ended ques- ment bias and maintain the consistency throughout the
tions in the interview [52], drafted based on the review of analysis process. Adopting Thematic Content Analysis, the
the relevant literature [e.g. 5, 6, 37] (see Appendix). They researchers proceeded from categorizing and coding the data
were mainly directed at ascertaining ‘the measures, elements to reflect on how the codes associate with each other,
and dimensions to be utilized for evaluating accreditation leading to a more profound understanding of the issue.
programs in healthcare’. The purpose was to build up
general guidelines for assessing the performance of health-
care accreditation programs. The questions were also fol- Results
lowed by a statement asking for respondents’ other related
comments and leaving room for their suggestions to The findings of the study could be categorized under three
improve the questions. Two follow-up emails were sent as main headings based on the relevance of the emerged
reminders to those respondents who did not respond within
648
Evaluation of accreditation † Accreditation
Measure Explanation
.............................................................................................................................................................................
Uptake of accreditation programs by HCOs Demand for (voluntary) accreditation programs in the health sector,
in comparison with other external evaluation and improvement
initiatives such as ISO and EFQM
Retention rate of HCOs by accreditation The rate of HCOs that have stayed returned or switched to a
programs (voluntary) accreditation program from ISO, EFQM, etc. during a
specific period because of its quality improvement merits
The level of community (i.e. HCOs) awareness of This might be an indicator of (voluntary) accreditation programs’
accreditation programs success and dominance in a society
Reflection of local healthcare priorities in the This is important in the case of national programs, which they give
standards of accreditation programs priority to the evaluation of locally delivered services
Representative governance structure for Representatives from different stakeholders, especially vulnerable
accreditation programs groups (e.g. patients) and third parties, in the programs’ governing
649
Jaafaripooyan et al.
Measure Explanation
.............................................................................................................................................................................
Trained, experienced and healthcare- † The use of specifically tailored training programs for surveyors
(hospital) oriented surveyors † Regular examination of surveyors’ selection and training processes
Appraisal of surveyors’ performance A mechanism for regular appraisal of surveyors’ performance
Surveyor replacement and turnover rate This could reduce the risk of surveys turning into an informal and
routinized process and introduce fresh eyes to the process (low turnover
times might eliminate experience element)
Presence of HCOs’ members (as observers) This could both validate the assessment process and justify the results to
during surveyors’ evaluation the members and create more clarification on HCOs’ activities to
surveyors
Ongoing monitoring of HCOs during the In light of long intervening times (e.g. three years), monitoring HCOs (via
intervening time between two accreditation announced and unannounced visits) during this time could keep HCOs
surveys loyal to the requirements of (mandatory) accreditation programs and safe
650
Evaluation of accreditation † Accreditation
Table 2 Continued
Measure Explanation
.............................................................................................................................................................................
Participation (attraction) of HCOs’ staff † This could increase the credibility of the accreditation programs’
(especially clinical people) in accreditation requirements and facilitate their operationalization in HCOs
processes † This might be undertaken in the self-evaluation stage of HCOs (i.e.
preparation for the accreditation)
A sound and reliable scoring system An evidence-based rather than judgemental system for allocation of the
scores to the related activities (more reliance on evidence for scoring)
The use of a self-evaluation system by Accreditation of HCOs based on newly emerged methods and
accreditation programs to ensure their development of standards for new services afoot in HCOs
continued relevance to current practice of
HCOs
Measure Explanation
.............................................................................................................................................................................
Outputs of accreditation Increased compliance with accreditation standards: the number of HCOs met the
programs (mandatory) accreditation standards vis-à-vis previous accreditation
periods-measured based on a HCOs’ accreditation rank
Immediate tangible actions taken by HCOs before and after an accreditation
program to meet its requirements (e.g. holding meetings, setting up purpose-built
internal committees): This might refer to the degree of importance given to
accreditation programs by HCOs and could be measured through identifying those
efforts of the HCOs which are directed toward these requirements (e.g. during the
preparation phase)
Recommendations offered to HCOs by accreditation programs for improvement:
The percentage of initially failed HCOs which are subsequently successful as a
result of the guidance of (mandatory) accreditation programs
Outcomes of accreditation The substantive outcomes of accreditation programs for HCOs over time
programs (pre/post-accreditation) or in terms of accredited/non-accredited HCOs:
(1) Improved patients’ care
(2) Standardization of care processes in HCOs
(3) Enhanced patient and staff satisfaction
(4) Greater safety for patients and staff
(5) Improved financial performance of HCOs
(6) Improved inter-/intra-HCOs’ communication and information flows
(7) Professional development (learning outcomes of the accreditation programs
for the hospitals’ staff)
(8) Coherent organizational culture (encouraging same norms and language of
communication and shared knowledge in HCOs)
Minimal unintended and HCOs undergo a low rate of adverse effects from/through the process of
dysfunctional effects accreditation (e.g. tunnel vision, stress and tension)
HCOs are not driven towards dysfunctions (e.g. gaming) because of accreditation
programs
Indirect benefits of Financial benefits for HCOs (increased tariffs for services)
accreditation programs for Legitimacy for the HCOs, granted by accreditation programs in the form of
HCOs ranking, which might be construed in two ways (for a HCO):
† As a permission to continue its activity in its current state;
† As an element of reputation
Encouragement of HCOs to imitate their higher ranked similar organizations
651
Jaafaripooyan et al.
Hence, these measures might be considered either as a weak- Flexibility of an accreditation program to adapt to the
ness or strength for these programs. Nevertheless, most of changes in its environment and accommodate the feedback
the measures are stated positively in that their exact opposite of different stakeholders could ensure its sustainability and
could be also assumed as relevant in investigating the dys- relevance. This openness to changes could transfer the pro-
functional effects of an accreditation program. The following grams into a learning organization which always incorporates
section will briefly provide a discussion of the main points feedback in its development process and stays up-to-date.
presented in the tables.
Methodology of accreditation programs
Societal impact of accreditation programs
Table 2 provides related ‘performance measures’ for judging
Table 1 demonstrates the measures which reflect the impact the functionality of accreditation. Surveyors and standards
of accreditation programs at a macro and societal level. are two main elements of any accreditation scheme [66].
Accreditation has undeniably been an influential mechanism Surveyors are envisioned as the ‘eyes, ears and hands’ of any
for protecting society in the sense of safeguarding pubic accrediting organization, without which the accreditation
access to quality and safe healthcare [2]. The accreditation process is argued to be unsustainable [67, p. 1]. Therefore, as
programs play a key role in monitoring the reflection of such, a detailed training program for new surveyors along
652
Evaluation of accreditation † Accreditation
accreditation process [41]. This is noticed and operationalized organizations. Policymakers are increasingly becoming
by various programs. For instance, in the USA, since 1997, worried about dysfunctional and unintended consequences
JCAHO has linked clinical outcomes indicators to accredita- of quality improvement initiatives [77]. Therefore, the merits
tion process through ORYX-initiative integrating outcomes of accreditation programs’ could be also scrutinized for their
and other performance measurement data into the accredita- unintended and dysfunctional effects on HCOs (e.g. the pro-
tion process [72]. ACHS has developed the performance and grams may cause physicians to focus mostly on measured
outcome service to increase the clinical components and care at the expense of unmeasured care).
indicators in its new accreditation program [73, 74]. Accreditation could be seen as a source of economic gain
The existence of a regular review and update system for and legitimacy for HCOs. These organizations are permitted
the entire accreditation process, specifically the standards, to work in some communities on the condition that they
was widely reflected by the respondents. In JCAHO, stan- receive an accreditation award (i.e. legitimacy). Alternatively,
dards are reviewed every year for hospitals and every 2 years they might be allowed to charge higher tariffs or have service
for other HCOs, and Accreditation Canada reviews its stan- contracts with third-party organizations based on the level of
dards every 2 years. The self-evaluation system for accredita- their ranking. In addition, these programs provide a chance
tion programs ensures the relevance of their standards to the for HCOs to identify and emulate those organizations which
newly emerging activities of HCOs. It further helps assimilate have received the highest ranking in accreditation.
653
Jaafaripooyan et al.
diverse participant group and, on the other hand, to the data References
access restraints, as explained earlier. Fourth, despite the fact
that a generic form of the accreditation was conceptualized 1. Hirose M, Imanaka Y, Ishizaki T et al. How can we improve the
for the respondents, an intrinsic weakness in seeking opinions quality of health care in Japan? Learning form JCQHC hospital
from three sources, which may have very different orientations accreditation. Health Policy 2003;66:29–49.
and experience of accreditation, remains. Since the data from 2. Jovanovic B Hospital accreditation as method for assessing
the various groups were not intended to be compared, but quality in healthcare. Oncology 2005;13:156–7.
combined, the effect of contextual differences could be argu-
3. Pomey M, Francois P, Contandriopoulos A et al. Paradoxes of
ably alleviated. Finally, the study is not claimed to be
French accreditation. Qual Safety Health Care 2005;14:51 –5.
all-encompassing in its coverage of measures related to the
performance of accreditation programs and other indicators 4. Shaw CD The external assessment of health services. World
might be explored. Accordingly, it is essential that further and Hosp Health Serv 2004;40:24 –7.
persistent empirical research is performed in an attempt to 5. de Walcque C, Seuntjens B, Vermeyen K et al. Comparative Study
seek confirmatory signs and build additional understanding of of Hospital Accreditation Programs in Europe. Health Services Research
these performance measures. (HSR). Brussels: Belgian Health Care Knowledge Centre
(KCE), 2008.
654
Evaluation of accreditation † Accreditation
19. Nicklin W, Dickson S The Value and Impact of Accreditation in 36. Sunol R, Nicklin W, Bruneau C et al. Promoting research into
Health Care: A Review of the Literature. Ottawa: Accreditation healthcare accreditation/external evaluation: advancing an
Canada, 2009, 2009. ISQua initiative. Int J Qual Health Care 2009;21:27– 8.
20. Braithwaite J, Westbrook J, Pawsey M et al. A prospective, multi- 37. Scrivens E Assessing the value of accreditation systems. Eur J
method, multi-disciplinary, multi-level, collaborative, Public Health 1997;7:4– 8.
social-organisational design for researching health sector accred-
itation. BMC Health Serv Res 2006;6:113. 38. Hurst K The nature and value of small and community
hospital accreditation. Int J Health Care Qual Assur
21. Al Tehewy M, Salem B, Habil I et al. Evaluation of accreditation 1997;10:94–106.
programme in non-governmental organizations’ health units in
Egypt: short-term outcomes. Int J Qual Health Care 2009;21:183. 39. Pongpirul K, Sriratanaban J, Asavaroengchai S et al.
Comparison of health care professionals’ and surveyors’
22. Chuang S, Inder K An effectiveness analysis of healthcare opinions on problems and obstacles in implementing quality
systems using a systems theoretic approach. BMC Health Serv management system in Thailand: a national survey. Int J Qual
Res 2009;9:195. Health Care 2006;18:346 –51.
23. Australian Council on Healthcare Standards. Australian Council 40. Baker S, Dunn D Accreditation: the hallmark of educational
on Healthcare Standards Annual Report 2007–2008. Sydney: quality. Radiol Technol 2006;78:123– 30.
655
Jaafaripooyan et al.
56. Denscombe M The Good Research Guide: For Small-Scale Social 69. Pickering E Evaluating the benefits and limitations of an
Research Projects. Maidenhead, Berkshire, UK: Open University accreditation system. World Hosp Health Serv 1995;31:31 –5.
Press, 2007.
70. Griffith J, Knutzen S, Alexander J Structural versus outcomes
57. Foster G Fishing with the net for research data. Br J Educ measures in hospitals: A comparison of Joint Commission and
Technol 1994;25:91–7. Medicare outcomes scores in hospitals. Qual Manag Health Care
2002;10:29–38.
58. Pope C, Mays N (eds). Qualitative research in health care, 3rd edn.
Oxford: Blackwell Publishing, 2006. 71. Powell A, Rushmer R, Davies H A Systematic Narrative Review of
Quality Improvement Models in Health Care: Social Dimensions of
59. Green J, Thorogood N Qualitative Methods for Health Research.
Health Institute. St. Andrews, Scotland: The Universities of
Thousand Oaks: Sage, 2004.
Dundee and St Andrews, 2009.
60. Ito H, Sugawara H Relationship between accreditation scores
72. JCAHO. Comprehensive Accreditation Manual for Hospitals (CAMH).
and the public disclosure of accreditation reports: a cross sec-
Illinois: Joint Commission Resources, 2005.
tional study. Quality and Safety in Health Care 2005;14:87–92.
73. Luderus E The ACHS evaluation and quality improvement
61. Broadbent J, Laughlin R Organisational and accounting change:
program (EQuIP): helping to give confidence to our future.
theoretical and empirical reflections and thoughts on a future
Aust Emerg Nurs J 1996;1:21 –5.
research agenda. J Account Organisational Change 2005;1:7 –26.
656