Professional Documents
Culture Documents
A R T I C L E I N F O A B S T R A C T
Article history: Sustainability evaluation has become a key component of international health. However, evaluators
Received 5 April 2012 have faced a number of challenges linked to the lack of consensus on the meaning of the concept of
Received in revised form 10 October 2012 ‘‘sustainability’’. This paper aims to describe a methodology, the Sustainability Analysis Process, based
Accepted 24 November 2012
on several conceptual frameworks and tested in five different countries in the physical rehabilitation
Available online 3 December 2012
sector. The methodology consists of five successive steps: (i) overview of the context; (ii) system
boundary; (iii) consensus vision of sustainability, and derivation of stakeholder perspectives; (iv)
Keywords:
selection of sustainability indicators and characterization and analysis of local system sustainability; and
Sustainability
(v) verification and modification. The paper also discusses the place of the evaluator and researcher in
Rehabilitation
Public health the process: the methodology aims to help evaluators objectively measure the level of sustainability of a
Developing country health system with the challenge of dealing with a subjective notion, the concept of sustainability, and a
Health system diversity of actors. The Sustainability Analysis Process also aims to capture the dynamics of systems by
repeating the process on a regular basis. The methodology highlights the need for evaluators build
consensus amongst stakeholders on a common vision of the future of a health system.
ß 2012 Elsevier Ltd. All rights reserved.
0149-7189/$ – see front matter ß 2012 Elsevier Ltd. All rights reserved.
http://dx.doi.org/10.1016/j.evalprogplan.2012.11.003
K. Blanchet, S. Girois / Evaluation and Program Planning 38 (2013) 28–32 29
sector five-step approach with a conceptual framework for were the following: ‘‘How is the sector organized?’’, ‘‘What are the
sustainability from the development arena. main achievements of the programmes?’’, ‘‘Who are the main
The five-step approach is inspired by the Process Analysis stakeholders and what are their roles?’’. Participants themselves
Method, a business model developed in industry to provide a presented evidence from the sector using reports and data, which
‘‘formalistic and transparent approach’’ to generating sustainabili- provided material for discussion.
ty indicators (Chee Tahir & Darton, 2006). Such an approach allows
for comparability of results between ‘‘operations’’, or systems, 3.2. Step II: system boundaries
while allowing for data relevant to the specific contexts of each
operation (Chee Tahir & Darton, 2006). The Process Analysis The second step aims to define the local system and system
Method endorses a sustainability framework that is ill suited for boundaries, a prerequisite for developing sustainability indica-
the health sector in fragile states and so the Child Survival tors. In this step, a local system for the rehabilitation sector was
Sustainability Assessment (CSSA) framework (Sarriot, Winch, defined as ‘‘a network of people and institutions whose
Ryan, Bowie, et al., 2004) was applied to our methodology. coordinated actions bring about sustainable positive outcomes
The Child Survival Sustainability Assessment (CSSA) uses a in the population [with rehabilitation needs]’’ (Yourkavitch,
conceptual framework for sustainability elaborated by the Ryan, & Sarriot, 2004, p. 22). System boundaries are broadly
Child Survival Technical Group and the Sustained Health governed by spatial and temporal scales (Bell & Morse, 2008,
Outcomes (SHOUT) (Sarriot, Winch, Ryan, Bowie, et al., 1999). Space can be perceived by individuals as anything from a
2004; Sarriot, Winch, Ryan, Edison, et al., 2004). The small village or district to the whole world. It is crucial, then, to
framework was developed for international projects and has define clearly the boundaries of the system under study (Izac &
been successfully applied in a number of developing countries Swift, 1994; Jansen, Stoorvogel, & Schipper, 1995). The temporal
(Sarriot et al., 2009). The sustainability framework uses scale corresponds to the time span considered necessary for the
systems thinking instead of restricting the view to programs system process in full.
or projects. It is based on the assumption that sustainability is
a multidimensional concept that encompasses six different 3.3. Step III: consensus vision of sustainability
components: (i) health outcomes; (ii) health services delivery,
access and quality in terms of inputs, processes and outputs; Step III aims to develop a common vision of sustainability
(iii) organizational capacity; (iv) organizational viability; (v) among participants building on context and system boundaries
community capacity; and (vi) enabling social, economic and as understood by the group. The approach is first to establish a
policy environment. sustainability reference, the level under which the system is
The SAP, thus, combines the Process Analysis Method five-step NOT sustainable (Bossel, 1999). Specifically, participants were
approach with the CSSA conceptual framework for sustainability. divided into small groups to discuss components of a non-
The five steps include: (i) overview of the context; (ii) system sustainable system for every dimension of the CSSA framework.
boundaries; (iii) consensus vision of sustainability; (iv) selection of Following comprehensive discussion and debate on non-
sustainability indicators and analysis of system sustainability; and sustainability, lines of consensus were presented in plenary
(v) verification and modification. Sustainability indicators are for full group discussion.
defined based on context-specific multi-stakeholder consensus Table 1 illustrates responses from the sustainability workshop
(Bell & Morse, 2008; Brown, 1998; Gruen et al., 2008). organized in Somailand in 2011 whereby participants defined
what is for them a non-sustainable rehabilitation system.
3. Implementation of the methodology This consensus of non-sustainability was then translated
into a vision of sustainable rehabilitation system by rephrasing
The Sustainability Analysis Process (SAP) was carried out in the negative sentences into positive statements. For example, a
each of the five countries by two researchers between 2009 and statement such as ‘‘the rehabilitation sector is not sustainable
2011. The participants to the process were identified by the when the delivery time of orthopedic devices is longer than
researchers in collaboration with key informants in the country. standard delivery time’’ was transformed into a statement such
Participants were key stakeholders involved in the public as, ‘‘a rehabilitation sector is sustainable when delivery time of
rehabilitation sector, including delegates from the Ministry of orthopedic devices respects standard delivery time’’.
Health and/or Ministry of Social Affairs, regional health authorities,
selected rehabilitation professional staff (physical therapists and
orthopedic technicians), rehabilitation center managers, repre-
sentatives of disabled people’s organizations and representatives
of international donors and non-governmental organizations Table 1
A vision of non sustainability of rehabilitation services in Somaliland February 2011.
involved locally in the sector. Participants were invited to a 3-
day sustainability workshop. The number of participants per The rehabilitation service delivery is not sustainable when. . .
workshop varied between 25 and 35. Presentations and discus- Service provision
sions were conducted in the local language and dialects. When coverage of rehabilitation services doe not reach every level of the
Translation and sign language interpretation was provided for system – rural areas, urban, districts, villages
the duration of the workshop. The workshop agenda in the five test When the orthopedic technicians do not produce 6–8 devices minimum
per month or the physiotherapists do not consult 10–15 patients per day
sites followed the five-step logic of the SAP and did not vary or
Delivery time is longer than standard delivery time
evolve through the test sites. Specific content and tools for each Technical quality does not respect national quality standards
step are described below. The orthopedic devices cause pain to the user
The rehabilitation center does not have the minimum standard equipment
and infrastructures to deliver services
3.1. Step I: overview of the context
A rehabilitation center does not have a minimum of 1 physiotherapist and
one orthopedic technician
This opening step is designed to foster a common understand- Less there are less than 2 physiotherapist assistants and 2 orthopedic
ing of the organization, stakeholders and processes within the technician assistants in each region of the country
rehabilitation sector. Typical questions asked by the facilitator Source: Karl Blanchet.
30 K. Blanchet, S. Girois / Evaluation and Program Planning 38 (2013) 28–32
Table 2 values is plotted inside the unit circle. The unit circle, therefore,
The sustainability indicators selected in Somaliland in February 2011.
represents the sustainability reference.
Dimensions Sustainability indicator
Dimension 2 % of P&O who produce 6–8 devices per month 3.6. Characterization and analysis of health system sustainability
% of PT who treat 10–15 people per day
% of devices that missed standard delivery times The final task consists of analyzing the six graphs, using both
Existence of national quality standards comparisons between graphs as well as on-depth analysis of each
% of devices that respect quality standards
graph to explore the composite indicators. On the basis of this
% of PWDs who experience pain with their new device
% of rehab centers with adequate level of equipment discussion, workshop participants elaborate an action plan aimed
Number of different suppliers of raw materials to address the main weaknesses and reinforce the system’s
% of regions with at least 2 PT Assistant and 2P&O Assistant strengths.
Number of PT and P&O in Somaliland
Source: Karl Blanchet 3.7. Step V: verification and modification
P&O: prosthesist, orthesist; PT: physiotherapist.
Table 3
The reference system and sustainability level in February 2011 for every indicator of the service delivery component of the rehabilitation sector in Somaliland.
Dimension 2 % of orthopedic technicians who produce at least 6–8 devices per month 100 70
% of physiotherapists who treat at least 10–15 people per day 100 80
% of devices that missed standard delivery times 100 75
Existence of national quality standards Yes No
% of devices that respect quality standards 100 75
% of PWDs who experience pain with their new device 0 20
% of rehabilitation centers with adequate level of equipment 100 50
Number of different suppliers of raw materials 5 5
K. Blanchet, S. Girois / Evaluation and Program Planning 38 (2013) 28–32 31
Fig. 1. Radar diagram illustrating the level of sustainability of the service delivery component of the rehabilitation sector in Somaliland February 2011. The red line represents
the reference system and in blue the measure in February 2011. Every branch relates to one sustainability indicator. (For interpretation of the references to color in this figure
legend, the reader is referred to the web version of the article.)
Source: Karl Blanchet.
Sustainability Analysis Process, requiring both substantive exper- understandably emphasized coordination and cohesion within the
tise as well as balanced interest representation in a process that is sector, responsibility, accountability and power of professional
essentially ‘‘norm-creating’’. Admittedly, involving a wide range of organizations, as key factors for health system sustainability.
diverse actors of the health system, including users, raises practical Limitations of the methodology are inherent to the process and
problems. First is the imbalance of power existing between context for which it was designed. For example, the consensus
different groups of stakeholders (Greene, 1997; Thurston & Potvin, approach will more often than not lead to selection of indicators
2003); second, some topics can be neglected on the agenda are not well defined. It could also be argued that the process
because people who defend them do not receive enough resembles political negotiation and results in compromise, rather
consideration within the group (Cornelissen, Van De Berg, Koops, than consensus measures. In fragile states there is inherent
& Udo, 2001; Kasemir, Van Asselt, & Durrenberger, 1999; Walt, difficulty to collect data. Health information systems are often
1996). These political aspects, how literature (Cimorelli & Stahl, weak, data collection expensive and time-consuming. In all five
2005). Hence, different ways of bridging the two processes are used countries, workshops were all held among stakeholders at the
in the Sustainability Analysis Process presented here. As such, this national level. In all five settings, the exercise highlighted the
methodology captures the tension between ‘‘knowledge produc- inherent weaknesses in the country’s health information system as
tion’’ and ‘‘norm creation’’ in a particular context with the gaps and a whole. We could expect that fear of failure among leaders and
resources at hand. decision-makers may have limited truthful projections of sustain-
The SAP adeptly highlights the priorities of different country ability to avoid embarrassment at term.
contexts. In our experience differences between countries were In each country where the methodology was tested, discussions
profound and the process allowed for these important specificities at the outset were predictable. Scientists favored expansive
to emerge. For example, Somaliland participants focused on equity datasets with the expectation that data would be published.
and coverage of services with indicators on distribution of Hopes were that decision-makers would use the data for
professionals in the country and per region. This preoccupation rehabilitation sector planning. Politicians were generally wary of
is understandable in Somaliland where the main provision of too much data, citing challenges such as unavailable resources
services is heavily centralized in the capital with poor access for needed for data collection, while underlying issues of accountabil-
those who are in other regions of the country. Cambodia, on the ity and power play were often implied. Scientists formulated
other hand, is in the midst of handing over governance of detailed, specific, time-bound indicators. Policymakers favored
rehabilitation services from international organizations to the indicators in broad and vague terms. The SAP introduced a process
Cambodian Ministry of Social Affairs. Indicators that emerged from of negotiation, which was, in fact, a political negotiation, as no
the Cambodia workshops emphasized Ministerial responsibilities indicator could be published without consensus agreement of all
including budget allocation as key components of sustainability. In stakeholders. As such, this multi-stakeholder consensus approach
Nepal, the rehabilitation sector is composed of a high number of is both the inherent weakness and inherent strength of the
actors (Ref. Blanchet paper). SAP workshop participants in Nepal methodology.
32 K. Blanchet, S. Girois / Evaluation and Program Planning 38 (2013) 28–32
5. Conclusions Chee Tahir, A., & Darton, R. C. (2006). Using indicator sets to monitor the performance of a
sustainable business. Oxford, United Kingdom: The Institution of Engineers.
Cimorelli, A. J., & Stahl, C. H. (2005). Tackling the dilemma of the science-policy
In this paper we present the Sustainability Analysis Process interface in environmental policy analysis. Bulletin of Science Technology Society,
designed to surmount the challenges of conceptualizing and 25, 46–52.
Cornelissen, A. M. G., Van De Berg, J., Koops, W. J., & Udo, H. M. J. (2001). Assessment of
measuring sustainability of health systems in low-income and the contribution of sustainability indicators to sustainable development: A novel
fragile states. Tested in five starkly different contexts, the approach using fuzzy set theory. Agriculture, Ecosystems and Environment, 86,
methodology successfully brings scientific and political stake- 173–185.
de Savigny, D., & Adam, T. (2009). System thinking for health systems strengthening.
holders to consensus with regards to sustainability of a given World Health Organisation, Alliance for Health Policy and Systems Research.
system of services through careful identification of system Greene, J. C. (1997). Evaluation as advocacy. Evaluation Practice, 18, 25–35.
boundaries, local resources and potential developments. The Gruen, R. L., Elliott, J. H., Nolan, M. L., Lawton, P. D., Parkhill, A., McLaren, C. J., & Lavis, J.
N. (2008). Sustainability science: an integrated approach for health-programme
sustainability indicators, which are locally defined, make it
planning. Lancet, 372, 1579–1589.
possible to visualize and track incremental change, even in highly Izac, A. M. N., & Swift, M. J. (1994). On agricultural sustainability and its measurement
volatile environments. Such a tailored approach, comprehensive of in small-scale farming in sub-Saharan Africa. Ecological Economics, 11, 105–125.
local challenges and opportunities, makes sustainability planning Jansen, D. M., Stoorvogel, J. J., & Schipper, R. A. (1995). Using sustainability indicators in
agricultural land-use analysis – An example from Costa-Rica agricultural land-use.
attractive and feasible to both planners and policy makers. Netherlands Journal of Agricultural Science, 43, 61–82.
Kasemir, B., Van Asselt, M. B. A., & Durrenberger, G. (1999). Integrated assessment of
sustainable development: multiple perspectives in interaction. International Jour-
Acknowledgement nal of Environment and Pollution, 11, 407–425.
Leach, M., Scoones, I., & Stirling, A. (2007). Pathways to sustainability: An overview of the
The authors would like to thank the international organization STEPS Centre approach. Brighton, United Kingdom: STEPS Centre.
Sarriot, E., Ricca, J., Ryan, L., Basnet, J., & Arscott-Mills, S. (2009). Measuring sustain-
Handicap International for the funding and support provided to the ability as a programming tool for health sector investments: Report from a pilot
research team. Karl Blanchet, PhD, is a Lecturer at the London sustainability assessment in five Nepalese health districts. International Journal of
School of Hygiene and Tropical Medicine. Susan Girois, MD and Health Planning and Management, 24(4), 326–350.
Sarriot, E., Winch, P. J., Ryan, L. J., Bowie, J., Kouletio, M., Swedberg, E., LeBan, K., Edison,
MscPH, was the Director of the Technical Resources Division at J., Welch, R., & Pacque, M. C. (2004). A methodological approach and framework for
Handicap International. sustainability assessment in NGO-implemented primary health care programs.
International Journal of Health Planning and Management, 19, 23–41.
Sarriot, E., Winch, P. J., Ryan, L. J., Edison, J., Bowie, J., Swedberg, E., & Welch, R. (2004).
References Qualitative research to make practical sense of sustainability in primary health
care projects implemented by non-governmental organizations. International
Atkinson, S. (2002). Political cultures, health systems and health policy. Social Science Journal of Health Planning and Management, 19, 3–22.
and Medicine, 55, 113–124. Thurston, W. E., & Potvin, L. (2003). Evaluability assessment: A tool for incorporating
Bell, S., & Morse, S. (1999). Sustainability indicators. London: Earthscan. evaluation in social change programmes. Evaluation, 9, 453–469.
Bell, S., & Morse, S. (2008). Sustainability indicators measuring the immeasurable. UNCED: Agenda 21. (1992). Programme of action for sustainable development. The
London: Earthscan. United Nations conference on environment and development. New York, USA.
Bloom, G., & Standing, H. (2011). Pluralism and marketisation in the health sector: United Nations Children’s Fund. (1988). The Bamako initiative: Recommendations to the
Meeting health needs in contexts of social change in low and middle-income countries executive board for programme coordination 1989–1993. New York, USA: UNICEF.
(vol. 136, pp. ). ). Brighton, United Kingdom: Institute of Development Studies. Walt, G. (1996). Health policy: an introduction to process and power. Zed Books.
Bossel, H. (1999). Indicators for sustainable development: Theory, method, applications. World Bank. (1993). World development report 1993: Investing in health. New York/
Winnipeg, Canada: International Institute for Sustainable Development. United Kingdom: Oxford University Press.
Bossert, T. J. (2012). Health systems. Health Policy and Planning, 27, 8–10. World Health Organisation. (1987). Operational support for primary health care: the role
Brown, D. R. (1998). Evaluating institutional sustainability in development programmes: of the district level in accelerating HFA/2000 for all Africans. Geneva, Switzerland:
Beyond dollars and cents. Journal of International Development, 10, 55–69. World Health Organisation.
Casley, D., & Kumar, K. (1987). Project monitoring and evaluation in agriculture. World Health Organisation. (1988). The challenge of implementation: District health
Washington DC, USA: World Bank. systems for primary health care. Geneva, Switzerland: World Health Organisation.
Chambers, R. (1994a). Participatory rural appraisal (Pra) – Analysis of experience. Yourkavitch, J., Ryan, L., & Sarriot, E. (2004). Lessons learned from applying the Child
World Developement, 22, 1253–1268. Survival Sustainability Assessment (CSSA) framework to seven maternal and child
Chambers, R. (1994b). The origins and practice of participatory rural appraisal. World health projects. Arlington, USA: Macro International, Child Survival Technical
Developement, 22, 953–969. Support Project.