You are on page 1of 5

Evaluation and Program Planning 38 (2013) 28–32

Contents lists available at SciVerse ScienceDirect

Evaluation and Program Planning


journal homepage: www.elsevier.com/locate/evalprogplan

Selection of sustainability indicators for health services in challenging


environments: Balancing scientific approach with political engagement
Karl Blanchet a,*, Susan Girois b
a
International Centre for Evidence in Disability, Clinical Research Department, London School of Hygiene and Tropical Medicine, United Kingdom
b
Handicap International Federation, Lyon, France

A R T I C L E I N F O A B S T R A C T

Article history: Sustainability evaluation has become a key component of international health. However, evaluators
Received 5 April 2012 have faced a number of challenges linked to the lack of consensus on the meaning of the concept of
Received in revised form 10 October 2012 ‘‘sustainability’’. This paper aims to describe a methodology, the Sustainability Analysis Process, based
Accepted 24 November 2012
on several conceptual frameworks and tested in five different countries in the physical rehabilitation
Available online 3 December 2012
sector. The methodology consists of five successive steps: (i) overview of the context; (ii) system
boundary; (iii) consensus vision of sustainability, and derivation of stakeholder perspectives; (iv)
Keywords:
selection of sustainability indicators and characterization and analysis of local system sustainability; and
Sustainability
(v) verification and modification. The paper also discusses the place of the evaluator and researcher in
Rehabilitation
Public health the process: the methodology aims to help evaluators objectively measure the level of sustainability of a
Developing country health system with the challenge of dealing with a subjective notion, the concept of sustainability, and a
Health system diversity of actors. The Sustainability Analysis Process also aims to capture the dynamics of systems by
repeating the process on a regular basis. The methodology highlights the need for evaluators build
consensus amongst stakeholders on a common vision of the future of a health system.
ß 2012 Elsevier Ltd. All rights reserved.

1. Introduction as a whole and analyze their dynamics (Bossert, 2012; de Savigny


& Adam, 2009). Capturing the complexity of sustainability of
Sustainability is a key component of health systems planning health systems requires to combine both approaches: (i) using
in developing countries (Leach, Scoones, & Stirling, 2007; UNCF, available evidence; and (ii) understanding the interrelationships
1988; WBW, 1993; WHO, 1987, 1988). Since the mid-1990s, between systems through a more subjective process engaging
governments and development agencies have sought to adopt stakeholders (Chambers, 1994a, 1994b). These challenges are
sustainability indicators as a concrete tool to measure progress magnified in low-income settings where empiric evidence may be
and guide decisions (UNCED, 1992, paragraph 40.6). Early on it lacking, project-focused approaches to development abound, and
became clear that selection and use of indicators for sustainable where sustainability is truly a relative term given the variety of
health systems presented several challenges. In particular, the local socio-political and economic contexts (Atkinson, 2002;
selection and utilization of sustainability indicators is a trade-off Bloom, 2011).
between a reductionist approach and a more systemic view of In this paper we present the Sustainability Analysis Methodol-
sustainability (Bell & Morse, 2008; Sarriot, Ricca, Ryan, Basnet, & ogy (SAP) designed to allow conceptualizing and measuring
Arscott-Mills, 2009). Reductionism describes the wholeness of a sustainability of health systems in low-income countries and
system as the sum of its parts. The objective of reductionism is to fragile states. Specifically, the SAP combines a scientific approach
explain reality using scientific methods and be able to explain with public and professionals’ engagement to define sustainability
reality through measurable and predictable data (Bell & Morse, indicators. We tested the methodology in Cambodia, Liberia, Nepal,
2008, 1999). Reductionism has greatly contributed to scientific Sierra Leone and Somaliland, focusing on the health-related
knowledge but is questioned by the scientific community when rehabilitation sector. The methodology is not specific to the
reductionism is applied to real life situations where interactions rehabilitation sector, however, and could be applied to other sub-
between phenomena and human beings constitute the central sectors of the health system.
subject of the study. System thinking tends to approach systems
2. The Sustainability Analysis Process methodology design

* Corresponding author. The Sustainability Analysis Process (SAP) presented here is an


E-mail address: karl.blanchet@lshtm.ac.uk (K. Blanchet). adaptation of two existing methodologies combining a business

0149-7189/$ – see front matter ß 2012 Elsevier Ltd. All rights reserved.
http://dx.doi.org/10.1016/j.evalprogplan.2012.11.003
K. Blanchet, S. Girois / Evaluation and Program Planning 38 (2013) 28–32 29

sector five-step approach with a conceptual framework for were the following: ‘‘How is the sector organized?’’, ‘‘What are the
sustainability from the development arena. main achievements of the programmes?’’, ‘‘Who are the main
The five-step approach is inspired by the Process Analysis stakeholders and what are their roles?’’. Participants themselves
Method, a business model developed in industry to provide a presented evidence from the sector using reports and data, which
‘‘formalistic and transparent approach’’ to generating sustainabili- provided material for discussion.
ty indicators (Chee Tahir & Darton, 2006). Such an approach allows
for comparability of results between ‘‘operations’’, or systems, 3.2. Step II: system boundaries
while allowing for data relevant to the specific contexts of each
operation (Chee Tahir & Darton, 2006). The Process Analysis The second step aims to define the local system and system
Method endorses a sustainability framework that is ill suited for boundaries, a prerequisite for developing sustainability indica-
the health sector in fragile states and so the Child Survival tors. In this step, a local system for the rehabilitation sector was
Sustainability Assessment (CSSA) framework (Sarriot, Winch, defined as ‘‘a network of people and institutions whose
Ryan, Bowie, et al., 2004) was applied to our methodology. coordinated actions bring about sustainable positive outcomes
The Child Survival Sustainability Assessment (CSSA) uses a in the population [with rehabilitation needs]’’ (Yourkavitch,
conceptual framework for sustainability elaborated by the Ryan, & Sarriot, 2004, p. 22). System boundaries are broadly
Child Survival Technical Group and the Sustained Health governed by spatial and temporal scales (Bell & Morse, 2008,
Outcomes (SHOUT) (Sarriot, Winch, Ryan, Bowie, et al., 1999). Space can be perceived by individuals as anything from a
2004; Sarriot, Winch, Ryan, Edison, et al., 2004). The small village or district to the whole world. It is crucial, then, to
framework was developed for international projects and has define clearly the boundaries of the system under study (Izac &
been successfully applied in a number of developing countries Swift, 1994; Jansen, Stoorvogel, & Schipper, 1995). The temporal
(Sarriot et al., 2009). The sustainability framework uses scale corresponds to the time span considered necessary for the
systems thinking instead of restricting the view to programs system process in full.
or projects. It is based on the assumption that sustainability is
a multidimensional concept that encompasses six different 3.3. Step III: consensus vision of sustainability
components: (i) health outcomes; (ii) health services delivery,
access and quality in terms of inputs, processes and outputs; Step III aims to develop a common vision of sustainability
(iii) organizational capacity; (iv) organizational viability; (v) among participants building on context and system boundaries
community capacity; and (vi) enabling social, economic and as understood by the group. The approach is first to establish a
policy environment. sustainability reference, the level under which the system is
The SAP, thus, combines the Process Analysis Method five-step NOT sustainable (Bossel, 1999). Specifically, participants were
approach with the CSSA conceptual framework for sustainability. divided into small groups to discuss components of a non-
The five steps include: (i) overview of the context; (ii) system sustainable system for every dimension of the CSSA framework.
boundaries; (iii) consensus vision of sustainability; (iv) selection of Following comprehensive discussion and debate on non-
sustainability indicators and analysis of system sustainability; and sustainability, lines of consensus were presented in plenary
(v) verification and modification. Sustainability indicators are for full group discussion.
defined based on context-specific multi-stakeholder consensus Table 1 illustrates responses from the sustainability workshop
(Bell & Morse, 2008; Brown, 1998; Gruen et al., 2008). organized in Somailand in 2011 whereby participants defined
what is for them a non-sustainable rehabilitation system.
3. Implementation of the methodology This consensus of non-sustainability was then translated
into a vision of sustainable rehabilitation system by rephrasing
The Sustainability Analysis Process (SAP) was carried out in the negative sentences into positive statements. For example, a
each of the five countries by two researchers between 2009 and statement such as ‘‘the rehabilitation sector is not sustainable
2011. The participants to the process were identified by the when the delivery time of orthopedic devices is longer than
researchers in collaboration with key informants in the country. standard delivery time’’ was transformed into a statement such
Participants were key stakeholders involved in the public as, ‘‘a rehabilitation sector is sustainable when delivery time of
rehabilitation sector, including delegates from the Ministry of orthopedic devices respects standard delivery time’’.
Health and/or Ministry of Social Affairs, regional health authorities,
selected rehabilitation professional staff (physical therapists and
orthopedic technicians), rehabilitation center managers, repre-
sentatives of disabled people’s organizations and representatives
of international donors and non-governmental organizations Table 1
A vision of non sustainability of rehabilitation services in Somaliland February 2011.
involved locally in the sector. Participants were invited to a 3-
day sustainability workshop. The number of participants per The rehabilitation service delivery is not sustainable when. . .
workshop varied between 25 and 35. Presentations and discus- Service provision
sions were conducted in the local language and dialects. When coverage of rehabilitation services doe not reach every level of the
Translation and sign language interpretation was provided for system – rural areas, urban, districts, villages
the duration of the workshop. The workshop agenda in the five test When the orthopedic technicians do not produce 6–8 devices minimum
per month or the physiotherapists do not consult 10–15 patients per day
sites followed the five-step logic of the SAP and did not vary or
Delivery time is longer than standard delivery time
evolve through the test sites. Specific content and tools for each Technical quality does not respect national quality standards
step are described below. The orthopedic devices cause pain to the user
The rehabilitation center does not have the minimum standard equipment
and infrastructures to deliver services
3.1. Step I: overview of the context
A rehabilitation center does not have a minimum of 1 physiotherapist and
one orthopedic technician
This opening step is designed to foster a common understand- Less there are less than 2 physiotherapist assistants and 2 orthopedic
ing of the organization, stakeholders and processes within the technician assistants in each region of the country
rehabilitation sector. Typical questions asked by the facilitator Source: Karl Blanchet.
30 K. Blanchet, S. Girois / Evaluation and Program Planning 38 (2013) 28–32

Table 2 values is plotted inside the unit circle. The unit circle, therefore,
The sustainability indicators selected in Somaliland in February 2011.
represents the sustainability reference.
Dimensions Sustainability indicator

Dimension 2 % of P&O who produce 6–8 devices per month 3.6. Characterization and analysis of health system sustainability
% of PT who treat 10–15 people per day
% of devices that missed standard delivery times The final task consists of analyzing the six graphs, using both
Existence of national quality standards comparisons between graphs as well as on-depth analysis of each
% of devices that respect quality standards
graph to explore the composite indicators. On the basis of this
% of PWDs who experience pain with their new device
% of rehab centers with adequate level of equipment discussion, workshop participants elaborate an action plan aimed
Number of different suppliers of raw materials to address the main weaknesses and reinforce the system’s
% of regions with at least 2 PT Assistant and 2P&O Assistant strengths.
Number of PT and P&O in Somaliland
Source: Karl Blanchet 3.7. Step V: verification and modification
P&O: prosthesist, orthesist; PT: physiotherapist.

Sustainability indicators are measured during the workshop


3.4. Step IV: selection of sustainability indicators and analysis of even if they are estimation. However, these measures need to be
system sustainability checked using available evidence published in reports, in facilities’
statistics or through surveys. The process is repeated every 12
Step IV aims to generate a set of sustainability indicators that are months in order to capture the dynamics of health systems and
objective, comprehensive and relevant to the local context. Given measure progress made in the various dimensions of sustainabili-
the agreed-upon system boundaries as well as the consensus ty.
vision of sustainability, participants develop indicators that will
allow them to measure the level of achievement in the various 4. Discussion
dimensions of sustainability (Table 2).
The choice of these indicators may reflect a conservative The Sustainability Analysis Process (SAP) presented here allows
outlook, based on limited resources available to track or measure systematic exploration of quantity, quality and cost components to
them. Alternatively, some indicators might mirror international sustainable health systems in low-income countries and fragile
policies and guidelines, which could be considered inappropriate states. Integral to the SAP is a consensus approach among diverse
or unrealistic in the local context. This situation requires stakeholders that ensures local specificity and feasibility within
participants to think both realistically and beyond current each unique context.
resources to come up with a consensus list of sustainability The diversity of definitions of the concept of sustainability is
indicators for the rehabilitation sector. widely recognized (Bell & Morse, 1999). In a systematic literature
Finally, participants are asked to benchmark their actual level of review, Gruen et al. (2008) analyzed 24 studies on sustainability
every indicator based on data available. In several instances no assessment, which had been carried out in low- and middle-
data existed at the time of the workshop. In such cases, key income countries and recognized that sustainability was under-
informants provided estimates to be verified later. Participants stood in different ways by different people resulting in a diversity
also realized that oftentimes data collection mechanisms did not of sustainability assessment systems. In spite of these diverse
exist. As such, the workshop served as an opportunity for definitions, there is general consensus that sustainability indica-
identifying gaps in research and launching processes for monitor- tors should be consistent, sensitive and easy to collect (Casley &
ing and data collection. Kumar, 1987), allowing managers to track distinct progress toward
benchmarks and goals.
3.5. Identification of the sustainability reference and use of radar Through multi-stakeholder consensus the SAP also addresses
diagrams political challenges in at least two dimensions. First, the concept of
sustainability is upheld as normative. Second, the process avoids
After selecting sustainability indicators for the existing system, decisions taken by a limited number of ‘‘experts’’. In other
participants need to define their sustainability reference, or the processes, experts who decide on what to sustain across a range of
level under which their system is considered as unsustainable factors (ecological, economic, and social) are required to make
(Bossel, 1999). The sustainability reference is elaborated for every normative decisions based on technical knowledge, but also on
indicator selected in the six components of the sustainability more or less explicit philosophical perceptions and political
framework (Table 3). intentions. This implies that those participating in the process
In this study, radar diagrams were used to visually illustrate are not only acting in their technical expert capacity, but also as
each indicator (Fig. 1). Each indicator is represented by one arm ‘‘political citizens’’ taking normative decisions on what aspects to
and its value ranked from the center of the star to the end of the uphold. The decision on ‘‘who participates’’ and ‘‘who decides’’ in
axis. The rehabilitation system is not sustainable if any of the indicator development workshops is thus a crucial one in the

Table 3
The reference system and sustainability level in February 2011 for every indicator of the service delivery component of the rehabilitation sector in Somaliland.

Dimensions Sustainability indicator Reference system Sustainability level in January 2011

Dimension 2 % of orthopedic technicians who produce at least 6–8 devices per month 100 70
% of physiotherapists who treat at least 10–15 people per day 100 80
% of devices that missed standard delivery times 100 75
Existence of national quality standards Yes No
% of devices that respect quality standards 100 75
% of PWDs who experience pain with their new device 0 20
% of rehabilitation centers with adequate level of equipment 100 50
Number of different suppliers of raw materials 5 5
K. Blanchet, S. Girois / Evaluation and Program Planning 38 (2013) 28–32 31

Fig. 1. Radar diagram illustrating the level of sustainability of the service delivery component of the rehabilitation sector in Somaliland February 2011. The red line represents
the reference system and in blue the measure in February 2011. Every branch relates to one sustainability indicator. (For interpretation of the references to color in this figure
legend, the reader is referred to the web version of the article.)
Source: Karl Blanchet.

Sustainability Analysis Process, requiring both substantive exper- understandably emphasized coordination and cohesion within the
tise as well as balanced interest representation in a process that is sector, responsibility, accountability and power of professional
essentially ‘‘norm-creating’’. Admittedly, involving a wide range of organizations, as key factors for health system sustainability.
diverse actors of the health system, including users, raises practical Limitations of the methodology are inherent to the process and
problems. First is the imbalance of power existing between context for which it was designed. For example, the consensus
different groups of stakeholders (Greene, 1997; Thurston & Potvin, approach will more often than not lead to selection of indicators
2003); second, some topics can be neglected on the agenda are not well defined. It could also be argued that the process
because people who defend them do not receive enough resembles political negotiation and results in compromise, rather
consideration within the group (Cornelissen, Van De Berg, Koops, than consensus measures. In fragile states there is inherent
& Udo, 2001; Kasemir, Van Asselt, & Durrenberger, 1999; Walt, difficulty to collect data. Health information systems are often
1996). These political aspects, how literature (Cimorelli & Stahl, weak, data collection expensive and time-consuming. In all five
2005). Hence, different ways of bridging the two processes are used countries, workshops were all held among stakeholders at the
in the Sustainability Analysis Process presented here. As such, this national level. In all five settings, the exercise highlighted the
methodology captures the tension between ‘‘knowledge produc- inherent weaknesses in the country’s health information system as
tion’’ and ‘‘norm creation’’ in a particular context with the gaps and a whole. We could expect that fear of failure among leaders and
resources at hand. decision-makers may have limited truthful projections of sustain-
The SAP adeptly highlights the priorities of different country ability to avoid embarrassment at term.
contexts. In our experience differences between countries were In each country where the methodology was tested, discussions
profound and the process allowed for these important specificities at the outset were predictable. Scientists favored expansive
to emerge. For example, Somaliland participants focused on equity datasets with the expectation that data would be published.
and coverage of services with indicators on distribution of Hopes were that decision-makers would use the data for
professionals in the country and per region. This preoccupation rehabilitation sector planning. Politicians were generally wary of
is understandable in Somaliland where the main provision of too much data, citing challenges such as unavailable resources
services is heavily centralized in the capital with poor access for needed for data collection, while underlying issues of accountabil-
those who are in other regions of the country. Cambodia, on the ity and power play were often implied. Scientists formulated
other hand, is in the midst of handing over governance of detailed, specific, time-bound indicators. Policymakers favored
rehabilitation services from international organizations to the indicators in broad and vague terms. The SAP introduced a process
Cambodian Ministry of Social Affairs. Indicators that emerged from of negotiation, which was, in fact, a political negotiation, as no
the Cambodia workshops emphasized Ministerial responsibilities indicator could be published without consensus agreement of all
including budget allocation as key components of sustainability. In stakeholders. As such, this multi-stakeholder consensus approach
Nepal, the rehabilitation sector is composed of a high number of is both the inherent weakness and inherent strength of the
actors (Ref. Blanchet paper). SAP workshop participants in Nepal methodology.
32 K. Blanchet, S. Girois / Evaluation and Program Planning 38 (2013) 28–32

5. Conclusions Chee Tahir, A., & Darton, R. C. (2006). Using indicator sets to monitor the performance of a
sustainable business. Oxford, United Kingdom: The Institution of Engineers.
Cimorelli, A. J., & Stahl, C. H. (2005). Tackling the dilemma of the science-policy
In this paper we present the Sustainability Analysis Process interface in environmental policy analysis. Bulletin of Science Technology Society,
designed to surmount the challenges of conceptualizing and 25, 46–52.
Cornelissen, A. M. G., Van De Berg, J., Koops, W. J., & Udo, H. M. J. (2001). Assessment of
measuring sustainability of health systems in low-income and the contribution of sustainability indicators to sustainable development: A novel
fragile states. Tested in five starkly different contexts, the approach using fuzzy set theory. Agriculture, Ecosystems and Environment, 86,
methodology successfully brings scientific and political stake- 173–185.
de Savigny, D., & Adam, T. (2009). System thinking for health systems strengthening.
holders to consensus with regards to sustainability of a given World Health Organisation, Alliance for Health Policy and Systems Research.
system of services through careful identification of system Greene, J. C. (1997). Evaluation as advocacy. Evaluation Practice, 18, 25–35.
boundaries, local resources and potential developments. The Gruen, R. L., Elliott, J. H., Nolan, M. L., Lawton, P. D., Parkhill, A., McLaren, C. J., & Lavis, J.
N. (2008). Sustainability science: an integrated approach for health-programme
sustainability indicators, which are locally defined, make it
planning. Lancet, 372, 1579–1589.
possible to visualize and track incremental change, even in highly Izac, A. M. N., & Swift, M. J. (1994). On agricultural sustainability and its measurement
volatile environments. Such a tailored approach, comprehensive of in small-scale farming in sub-Saharan Africa. Ecological Economics, 11, 105–125.
local challenges and opportunities, makes sustainability planning Jansen, D. M., Stoorvogel, J. J., & Schipper, R. A. (1995). Using sustainability indicators in
agricultural land-use analysis – An example from Costa-Rica agricultural land-use.
attractive and feasible to both planners and policy makers. Netherlands Journal of Agricultural Science, 43, 61–82.
Kasemir, B., Van Asselt, M. B. A., & Durrenberger, G. (1999). Integrated assessment of
sustainable development: multiple perspectives in interaction. International Jour-
Acknowledgement nal of Environment and Pollution, 11, 407–425.
Leach, M., Scoones, I., & Stirling, A. (2007). Pathways to sustainability: An overview of the
The authors would like to thank the international organization STEPS Centre approach. Brighton, United Kingdom: STEPS Centre.
Sarriot, E., Ricca, J., Ryan, L., Basnet, J., & Arscott-Mills, S. (2009). Measuring sustain-
Handicap International for the funding and support provided to the ability as a programming tool for health sector investments: Report from a pilot
research team. Karl Blanchet, PhD, is a Lecturer at the London sustainability assessment in five Nepalese health districts. International Journal of
School of Hygiene and Tropical Medicine. Susan Girois, MD and Health Planning and Management, 24(4), 326–350.
Sarriot, E., Winch, P. J., Ryan, L. J., Bowie, J., Kouletio, M., Swedberg, E., LeBan, K., Edison,
MscPH, was the Director of the Technical Resources Division at J., Welch, R., & Pacque, M. C. (2004). A methodological approach and framework for
Handicap International. sustainability assessment in NGO-implemented primary health care programs.
International Journal of Health Planning and Management, 19, 23–41.
Sarriot, E., Winch, P. J., Ryan, L. J., Edison, J., Bowie, J., Swedberg, E., & Welch, R. (2004).
References Qualitative research to make practical sense of sustainability in primary health
care projects implemented by non-governmental organizations. International
Atkinson, S. (2002). Political cultures, health systems and health policy. Social Science Journal of Health Planning and Management, 19, 3–22.
and Medicine, 55, 113–124. Thurston, W. E., & Potvin, L. (2003). Evaluability assessment: A tool for incorporating
Bell, S., & Morse, S. (1999). Sustainability indicators. London: Earthscan. evaluation in social change programmes. Evaluation, 9, 453–469.
Bell, S., & Morse, S. (2008). Sustainability indicators measuring the immeasurable. UNCED: Agenda 21. (1992). Programme of action for sustainable development. The
London: Earthscan. United Nations conference on environment and development. New York, USA.
Bloom, G., & Standing, H. (2011). Pluralism and marketisation in the health sector: United Nations Children’s Fund. (1988). The Bamako initiative: Recommendations to the
Meeting health needs in contexts of social change in low and middle-income countries executive board for programme coordination 1989–1993. New York, USA: UNICEF.
(vol. 136, pp. ). ). Brighton, United Kingdom: Institute of Development Studies. Walt, G. (1996). Health policy: an introduction to process and power. Zed Books.
Bossel, H. (1999). Indicators for sustainable development: Theory, method, applications. World Bank. (1993). World development report 1993: Investing in health. New York/
Winnipeg, Canada: International Institute for Sustainable Development. United Kingdom: Oxford University Press.
Bossert, T. J. (2012). Health systems. Health Policy and Planning, 27, 8–10. World Health Organisation. (1987). Operational support for primary health care: the role
Brown, D. R. (1998). Evaluating institutional sustainability in development programmes: of the district level in accelerating HFA/2000 for all Africans. Geneva, Switzerland:
Beyond dollars and cents. Journal of International Development, 10, 55–69. World Health Organisation.
Casley, D., & Kumar, K. (1987). Project monitoring and evaluation in agriculture. World Health Organisation. (1988). The challenge of implementation: District health
Washington DC, USA: World Bank. systems for primary health care. Geneva, Switzerland: World Health Organisation.
Chambers, R. (1994a). Participatory rural appraisal (Pra) – Analysis of experience. Yourkavitch, J., Ryan, L., & Sarriot, E. (2004). Lessons learned from applying the Child
World Developement, 22, 1253–1268. Survival Sustainability Assessment (CSSA) framework to seven maternal and child
Chambers, R. (1994b). The origins and practice of participatory rural appraisal. World health projects. Arlington, USA: Macro International, Child Survival Technical
Developement, 22, 953–969. Support Project.

You might also like