Health Policy and Planning, 2020, 1–11

doi: 10.1093/heapol/czaa116
Review

Downloaded from https://academic.oup.com/heapol/advance-article/doi/10.1093/heapol/czaa116/5999228 by Karolinska Institutet University Library user on 20 December 2020
Economic evaluation and health systems
strengthening: a review of the literature
Susan Cleary *
Health Economics Unit/Division, School of Public Health and Family Medicine, University of Cape Town, Anzio
Road, Observatory, Cape Town 7925, South Africa
*Corresponding author. Health Economics Unit/Division, School of Public Health and Family Medicine, University of Cape
Town, Anzio Road, Observatory, Cape Town 7925, South Africa. E-mail: susan.cleary@uct.ac.za
Accepted on 2 September 2020

Abstract
Health systems strengthening (HSS) is firmly on the global health and development agenda. While
a growing evidence base seeks to understand the effectiveness of HSS, there is limited evidence
regarding cost and cost-effectiveness. Without such evidence, it is hard to argue that HSS repre-
sents value for money and the level of investment needed cannot be quantified. This paper seeks
to review the literature regarding the economic evaluation of HSS from low- and middle-income
country (LMIC) settings, and to contribute towards the development of methods for the economic
evaluation of HSS. A systematic search for literature was conducted in PubMed, Scopus and the
Health Systems Evidence database. MeSH terms related to economic evaluation were combined
with key words related to the concept of HSS. Of the 204 records retrieved, 52 were retained for full
text review and 33 were included. Of these, 67% were published between January 2015 and June
2019. While many HSS interventions have system wide impacts, most studies (71%) investigated
these impacts using a disease-specific lens (e.g. the impact of quality of care improvements on
uptake of facility deliveries). HSS investments were categorized, with the majority being invest-
ments in platform efficiency (e.g. quality of care), followed by simultaneous investment in platform
efficiency and platform capacity (e.g. quality of care and task shifting). This review identified a
growing body of work seeking to undertake and/or conceptualize the economic evaluation of HSS
in low- and middle-income countries. The majority assess HSS interventions using a disease-
specific or programmatic lens, treating HSS in a similar manner to the economic evaluation of
medicines and diagnostics. While this approach misses potential economies of scope from HSS
investments, it allows for a preliminary understanding of relative value for money. Future research
is needed to complement the emerging evidence base.

Keywords: Economic evaluation, costs, cost-effectiveness, health systems, technical efficiency

Introduction determine the affordability of services, establishing or refining

In 2015, United Nations member states signed the Sustainable
Development Goals (SDGs), committing to the achievement of key
health and development targets by 2030 (United Nations, 2015). A
central pillar of these efforts is the movement towards Universal
Health Coverage (UHC), which entails the progressive realization of
equitable access to quality health care, free at the point of use
(Kutzin, 2013). These goals of equity, quality and financial risk pro-
tection are achievable irrespective of country income level, provided
that the chosen health benefits package to be provided by the health
system is affordable (Glassman et al., 2017). Given the need to
in-country processes of Health Technology Assessment to include
economic evaluation is becoming a global priority (Sixty-Seventh
World Health Assembly, 2014). Such processes will further contrib-
ute towards the achievement of the UHC intermediate objectives of
efficiency (through the inclusion of cost-effective services within the
benefits package) and transparency (through an explicit identifica-
tion of the services that are included).
Resource allocation or priority setting for health systems can
include decisions around whether to invest in a new technology (e.g.
medicine, diagnostic or investigation) as well as decisions about

C The Author(s) 2020. Published by Oxford University Press in association with The London School of Hygiene and Tropical Medicine. All rights reserved.
V
For permissions, please e-mail: journals.permissions@oup.com 1
2 Health Policy and Planning, 2020, Vol. 0, No. 0

KEY MESSAGES

Downloaded from https://academic.oup.com/heapol/advance-article/doi/10.1093/heapol/czaa116/5999228 by Karolinska Institutet University Library user on 20 December 2020
• Considerations of value for money are key towards ensuring that services offered within Universal Health Coverage systems achieve
maximum population benefits.
• While a growing evidence base exists regarding the cost and cost-effectiveness of medicines and diagnostics for specific conditions,
less is known about the cost-effectiveness of health systems strengthening (HSS). Similarly, methods for the economic evaluation of
technologies such as medicines and diagnostics are well established, but key gaps exist regarding how to conceptualize and conduct
economic evaluations of HSS.
• This paper offers a review of the literature of economic evaluations of HSS interventions conducted in low- and middle-income coun-
try settings, with a view towards understanding the current scope and methodological practices of the field.
• Of the papers retrieved, the majority assess HSS interventions using a disease-specific or programmatic lens, treating HSS in a similar
manner to the economic evaluation of medicines and diagnostics. While such an approach misses the potential economies of scope
from HSS investments, it allows for a preliminary understanding of relative value for money.

whether to invest in ‘health system strengthening’ (e.g. information
or managerial system). While the former will often have a relatively
narrow scope or disease-specific focus, the latter will often impact
across multiple disease-specific areas (Vassall et al., 2019). Both the
methodology and the evidence base of cost and cost-effectiveness to
date has been more focused on the former, with less attention paid
to the cost-effectiveness of health systems strengthening (HSS). As
argued by Murray et al. (1994, p. 664) ‘cost-effectiveness analysis of
health interventions, which are more often than not disease specific,
tends to neglect the role of the health system in delivering these
interventions. There are no explicit analyses of cost-effectiveness of
improving the physical or human infrastructure of the health system,
which provide for direct comparisons between investing in the deliv-
ery system and purchasing more specific interventions delivered by
the health system’. Indeed, if health systems are weak, it is possible
that investments towards HSS would generate more value than
investments in additional medicines and diagnostics within disease-
specific areas (Mills et al., 2006). Generating evidence on the cost-
effectiveness of HSS and the level of investment needed to achieve
performance outcomes is therefore a priority (Vassall et al., 2019).
This paper seeks to review economic evaluations of HSS inter-
ventions from low- and middle-income country (LMIC) settings to
describe the scope and key methodological aspects of this literature
and thereby to contribute towards future methodological
developments.
Conceptual underpinnings
To gain insight into the methodology of HSS economic evaluations,
it is necessary to first describe ‘standard’ approaches. Following
Drummond et al. (2015) an economic evaluation considers two or
more ways of achieving a given outcome. There are five types of par-
tial or full economic evaluation: cost analysis, cost-consequence ana-
lysis (CCA), cost-effectiveness analysis (CEA), cost-utility analysis
(CUA) and cost–benefit analysis (CBA). In addition, costs can be
estimated using different ‘perspectives’. A patient perspective
includes out of pocket payments (e.g. user fees), costs of accessing
care (e.g. transport costs) and may include other opportunity costs
of time spent receiving care (e.g. waiting times within facilities) that
are incurred by a patient and potentially also by a patient’s carer(s).
A health care provider or health systems perspective includes the
costs borne by the health sector. A societal perspective is broadest
and encompasses both patient and provider perspectives. Table 1
outlines each type of analysis, indicating the types of costs included,
outcomes, approach to decision-making and advantages or disad-
vantages. Cost analysis is a form of partial economic evaluation that
can include costs from across the perspectives. Similarly, CCA and
CBA can include the full spectrum of costs, in addition to outcomes.
In the former, outcomes are listed separately, while in the latter,
these are aggregated and monetized. Both however have disadvan-
tages, with the main disadvantage for CCA being the lack of theoret-
ical basis and lack of decision-making guidance, while the main
disadvantage for CBA is the bias and lack of precision associated
with monetizing outcomes (Helmchen and Lo Sasso, 2010;
Wiseman, 2016). In between these two extremes are the CEA and
CUA. Neither of these accommodates non-health costs nor out-
comes; and both seek to maximize a given outcome within a given
budget constraint and/or equity constraint if relevant (Cookson
et al., 2017). For CEA, given that the outcomes are in natural units,
comparability is normally limited to the specific disease in question
and to evaluations using the same outcome (e.g. CEAs using life
years gained can be compared across a range of HIV-treatment
interventions). The budget constraint in question is then the health
care budget dedicated to that particular treatment programme (e.g.
the HIV-treatment budget) and issues of priority setting are
addressed within the HIV programme (Cleary et al., 2010).
However, in practice, determining the size of the HIV-treatment
budget is itself an allocative decision and CEA is unable to guide this
determination. Because of this, CUA is an increasingly popular form
of economic evaluation. Because health care outcomes are estimated
using multi-attribute approaches such as quality adjusted life years
(QALYs) or disability adjusted life years (DALYs), comparisons can
be made across diseases and interventions, and outcomes can be
maximized across the health care budget constraint in general (again
subject to an equity or other constraint if relevant). This idea of
choosing the set of interventions or programmes that maximize
health within the budget (or other relevant) constraint can be termed
‘allocative efficiency’ (Stinnett and Paltiel, 1996).
Guidelines for economic evaluation recommend that costs
should include any changes in resources that accompany a decision
to implement a particular programme, intervention or course of ac-
tion, as appropriate to the perspective of the costing (Husereau
et al., 2013). For example, within a provider’s perspective, costs
would typically include capital costs (e.g. health facility infrastruc-
ture, furniture, equipment and vehicles) as well as recurrent costs
(e.g. staff, medicines, laboratory investigations, supplies and over-
heads). If relevant, analyses can include ‘above service delivery’ costs
Health Policy and Planning, 2020, Vol. 0, No. 0
Table 1 Types of economic evaluation
CCA
Costs Can include health care and non-health care costs
Outcomes Multiple health outcomes and non-health outcomes
specified in natural units, e.g. cases detected
AND life years gained
Decision-making Not defined. Decision maker needs to define own
rule(s) rules
Advantages A broad scope of costs and outcomes can be
included, including non-health costs and
outcomes
Disadvantages Lack of clarity about usefulness in decision-making
CEA
Costs Only includes health care costs
Outcomes Singular health outcomes specified in natural units,
e.g. cases detected OR life years gained
Decision-making Maximize outcomes within the budget constraint.
rule(s) Can include an equity constraint. Given non-
comparability of outcomes, only addresses tech-
nical efficiency
Advantages Natural outcomes are generally easier to measure
Disadvantages Omits non-health costs and outcomes. Cannot
address allocative efficiency
CUA
Costs Only includes health care costs
Outcomes Multi-attribute health outcome measures such as
QALYs or DALYs
Decision-making Maximize outcomes within the budget constraint.
rule(s) Can include an equity constraint. Given compar-
ability of outcomes, can address allocative
efficiency
Advantages Comparability of diverse programmes is enhanced
through use of multi-attribute health outcome
Disadvantages Omits non-health costs and outcomes
CBA
Costs Can include health care and non-health care costs
Outcomes Health outcomes and non-health outcomes (such as
process utility, e.g. reassurance value associated
with negative test result) combined in monetary
units
Decision-making If benefits (in monetary terms) exceed costs, social
rule(s) welfare is improved through implementing
intervention
Advantages Broad scope of outcomes can be measured in mon-
etary terms
Disadvantages Issues of bias and precision in monetizing outcomes
Adapted from Drummond et al. (2015) and Helmchen and Lo Sasso (2010).
which include procurement and supply chain activities and demand
generation activities (e.g. mass media campaigns) amongst others
(Clift et al., 2016). Within the third edition of Disease Control
Priorities (DCP-3), the authors categorize these capital and recurrent
costs into direct costs at the point of care, including personnel,
drugs, equipment, etc.; costs of facility-level ancillary services,
including rent, building maintenance, etc. and programme costs,
that support services but frequently are incurred separately from
service delivery such as administration, management, logistics, etc.
They use the term ‘service delivery costs’ for the first, and group
the second and third together under the term ‘health system costs’
(Watkins et al., 2017).
The ‘standard’ approach to the inclusion of health system costs
is to use an allocation factor to estimate the share of these costs used
by the intervention under evaluation (Drummond et al., 2015). An
example would be the use of annual clinic visits to allocate annual
3
managerial salaries to calculate the mean cost per visit for manager-
ial staff time. Amongst others, this approach assumes that interven-
tions within a particular setting function independently of each
Downloaded from https://academic.oup.com/heapol/advance-article/doi/10.1093/heapol/czaa116/5999228 by Karolinska Institutet University Library user on 20 December 2020
other (Hauck et al., 2019); however, there are multiple ways in
which interventions are interdependent, with the potential for
variations in economies of scope depending on how services are
organized within and/or across the platform (Hauck et al., 2019).
What this means is that—for two or more interventions—costs, ben-
efits or both costs and benefits are different under joint vs separate
production, and this mutual dependency is argued to support the im-
portance of a systems perspective in evaluation (Hauck et al., 2019).
Following this logic, Hauck et al., have created a typology of three
types of HSS investments. The first category (HSS1) concerns invest-
ments in the technical efficiency of the particular facility or platform
which would impact on the production of all services delivered
within that setting. Examples include health information systems or
laboratory infrastructure (Morton et al., 2016). The second category
(HSS2) concerns investments into the capacity of the facility or plat-
form, including the reorientation of existing resources, enabling a
key resource constraint to be relaxed. A common example is scarce
human resources limiting service expansion, with investments such
as task shifting being a potential solution to this capacity constraint.
The third category (HSS3) concerns investments in new platform
areas such as new mobile services which again potentially alters the
relative cost-effectiveness of existing interventions.
Figure 1 presents these ideas within a conceptual framework,
summarizing ‘standard’ economic evaluation together with HSS1–3.
In the remaining sections, this conceptualization will be used as one
way of categorizing and describing the empirical literature.
Methods
Search strategy
A systematic search for relevant literature was conducted in
PubMed, Scopus and the ‘Health Systems Evidence’ database,
including any publications before July 2019. With the help of a spe-
cialized librarian, the MeSH heading term ‘Costs and Cost Analysis’
was identified as the preferred term to retrieve all concepts related
to economic evaluation. This approach was complemented by
searches for the key words ‘cost(s)’ or ‘costing’ within the heading
and abstract of the paper. Given that all economic evaluations
would include costs, these two strategies were deemed to be ad-
equate. For the concept of health system strengthening, full text
searches were conducted within articles using the key words ‘health
system(s) strengthening’ or ‘health system(s) development’.
One of the key challenges in selecting papers is to find a working
definition for HSS. While a number of definitions can be proposed,
in their seminal work the World Health Organization (2007) catego-
rized health systems into six building blocks and sub-components
including service delivery; health workforce; information systems;
medical products, vaccines and technologies; financing and govern-
ance, as outlined in Table 2. HSS is then defined as ‘improving these
. . . building blocks and managing their interactions in ways that
achieve more equitable and sustained improvements across health
services and health outcomes’ (World Health Organization, 2007, p.
4). However, by this definition, it would be difficult to exclude any
economic evaluation given that they would all evaluate interventions
that have the potential to improve health services and health out-
comes. Therefore, following others (Zeng et al., 2018) the service
delivery building block is restricted to interventions associated with
enhancing demand for care, public private partnerships around
4 Health Policy and Planning, 2020, Vol. 0, No. 0

Downloaded from https://academic.oup.com/heapol/advance-article/doi/10.1093/heapol/czaa116/5999228 by Karolinska Institutet University Library user on 20 December 2020
Figure 1 Conceptual framework of economic evaluation and investments in HSS. Adapted from Hauck et al., (2019)

Table 2 Categories of HSS by health system building blocks

To guide analysis, a data extraction template was created in
Service delivery Microsoft Excel. Data extracted directly from papers included a descrip-
Enhancing demand for care tion of the intervention and the study setting. In addition, based on the
Public–private partnerships around service provision details provided by the author(s), a judgement was made regarding the
Quality and safety of care health systems building block(s) covered within the evaluation (as per
Health workforce Table 2), and the type of HSS investment(s) (as per Figure 1). A distinc-
Supply and distribution
tion was also drawn between disease- or programme-specific HSS (tar-
Management and performance systems
geting system support within specific disease or programme areas) or
Pre-service and in-service training
Medical products, vaccines and technologies
system wide HSS where no explicit disease or programme link is identi-
Public–private partnerships around medical products, vaccines fied, following Warren et al. (2013). Key methodological data extracted
and technologies included type(s) of economic evaluation, costing perspective and out-
Supply chain management come measure(s) if applicable. These data were extracted as per the
Quality and safety of medicines stated method of the author(s) or the article and further coded to enable
Information systems comparison. On type of economic evaluation, studies of the ‘costs of
Health information systems scaling up’ were termed budget impact analysis; while CEA studies that
Management information systems included detailed intermediate outcomes with modelled QALYs or
Financing
DALYs were coded as CEA and CUA.
Payment mechanisms: provider
These data elements are summarized below.
Payment mechanisms: beneficiary
Priority setting or resource allocation General
Governance • Country, countries or region of focus
Oversight, accountability, regulations and incentives (e.g. audits,
• Description of HSS intervention
targets, core standards)
• Health systems building block(s) targeted by the intervention
Strategic planning
Consumer and stakeholder involvement (i.e. community (as per Table 2)
participation) • Type of HSS investment (as per Figure 1)
• System wide or disease-/programme-specific evaluation.
Adapted from Adam et al. (2012) and World Health Organization (2007).

Methods
service provision, and evaluations of quality and safety of care,
• Type(s) of economic evaluation
while economic evaluations of medicines, diagnostics or similar
• Costing perspective(s)
health technologies within a disease-specific area (e.g. GeneXpert
• Outcome measure(s).
for TB diagnosis or antiretrovirals for HIV treatment) are excluded
(see Table 2). In contrast, interventions falling within the medical
products, vaccines and technologies building block are included, Results
given the focus of this building block on public–private partnerships,
The initial search generated 153 hits from PubMed, 51 from
supply chain management and processes associated with securing
Scopus and 60 from the Health Systems Evidence database. One
the quality and safety of medicines.
additional paper was identified as part of a broader DCP volume
In summation, included papers:
that had been retrieved (Levin and Chisholm, 2016) and 2 were
• Conducted a full or partial economic evaluation of an HSS inter- included through prior knowledge of the literature (Lagarde
vention in a LMIC setting (as defined by the World Bank in 2019). et al., 2012; Borghi et al., 2015). After the removal of dupli-
• Were written in the English language. cates, 256 records were screened, of which 204 were excluded
Health Policy and Planning, 2020, Vol. 0, No. 0
Figure 2 PRISMA flow diagram
following title and abstract review. Of the 52 retained for full-
text review, 18 were excluded (systematic review or conceptual
paper ¼ 7; no full text ¼ 1; no economic evaluation ¼ 9; no HSS
intervention ¼ 1). A total of 33 papers were retained and
included in the qualitative synthesis. Details are contained in
Figure 2. A database including all papers is available in supple-
mentary material (Online Table 1).
Of the 33 included papers, the majority (56%) was from African
settings, including Benin (n ¼ 1), Ethiopia (n ¼ 2), Kenya (n ¼ 4),
Malawi (n ¼ 2), Nigeria (n ¼ 1), Rwanda (n ¼ 1) and South Africa
(n ¼ 2). In addition, there were 6 papers from Asia (Cambodia ¼ 1,
China ¼ 3 and Myanmar ¼ 1) and 1 paper from South America
(Chile). Finally, some papers took regional perspectives (sub-
Saharan-Africa and South Asia, n ¼ 1) or focused on income level
(low- and lower middle-income settings, n ¼ 2; low- and middle-
income settings, n ¼ 4). In terms of the scope of the literature
reviewed, of the 33 included studies, 27 (82%) took a programmatic
or disease-specific lens to the evaluation and 6 (18%) were system-
wide. The majority of papers (n ¼ 22) was published in the period
2015–19.
5
Downloaded from https://academic.oup.com/heapol/advance-article/doi/10.1093/heapol/czaa116/5999228 by Karolinska Institutet University Library user on 20 December 2020
Methodological approaches
Figure 3 summarizes the type of economic evaluation, costing per-
spective and outcome measures used across the studies. As is shown,
cost analysis was the most common study type (25% of 33 studies)
followed by CEA (24%) and BIA (15%). The remaining studies
were classified as a combination of different types of economic
evaluation including CEA, CUA and BIA. As mentioned, studies
were classified as CEA and CUA if they provided outcomes in natural
units as well as QALYs or DALYs. In terms of perspective, the major-
ity of studies took a provider’s perspective. Close to half of the
included studies did not state their perspective and were classified as
likely provider (36%) or likely societal (6%) or unclear if it appeared
that different perspectives were being used across different analyses,
without a clear identification of what was being used (n ¼ 1). No stud-
ies took a patient only perspective (although patient costs are included
within the societal perspective). In terms of outcomes, 40% of studies
did not include outcomes (of which 25% are cost analyses and 15%
are BIA) while 33% presented QALYs or DALYs (frequently in add-
ition to intermediate or natural outcome measures). The remaining
27% presented only natural or intermediate outcomes.
6 Health Policy and Planning, 2020, Vol. 0, No. 0
Figure 3 Type of economic evaluation, costing perspective and outcome measure
HSS within system wide evaluations
Of the 33 included papers, 6 studies could be described as system-
wide, although as illustrated below, such a breadth of analysis
makes it hard for the authors to provide sufficient details of what
costs and outcomes are being estimated. Of these, 1 paper estimated
a benefit package for Malawi (Ochalek et al., 2018) and 1 estimated
benefit packages for low- and lower middle-income settings
(Watkins et al., 2017). In Ochalek et al. (2018), the case for inves-
ting in HSS is made through an argument around poor coverage or
low implementation levels for cost-effective disease-specific inter-
ventions. In effect the authors argue that these low implementation
levels indicate the quantum of investment in HSS that would allow
for increased availability or coverage of disease-specific interven-
tions. In other words, they do not specifically evaluate the costs or
cost-effectiveness of the various types of HSS that could be imple-
mented in Malawi but instead provide an indication of the scale of
the investment that could be made while remaining cost-effective.
Similarly, Watkins et al. (2017) examine coverage levels of interven-
tions recommended within their benefits package and argue that this
would be a starting point for countries to think through the key
health systems bottle necks that prevent increased coverage. While
these authors include ‘health systems’ costs as part of the unit costs
of disease-specific interventions, they also do not directly estimate
the resource needs or provide details of HSS.
In contrast, 2 of the system wide papers focused only on assess-
ing costs, with 1 offering a costing of the health strategic plan in
Cambodia (Cantelmo et al., 2018) and 1 estimating the costs of
meeting the SDGs in LMICs (Stenberg et al., 2017). The approach
to costing builds on existing disease specific and HSS strategic goals,
and then assesses the costs needed to achieve these goals. These cost-
ing papers used the OneHealth Tool to generate their estimates,
which includes disease-specific and health systems modules. The
authors argue that their approach includes inputs such as infrastruc-
ture and supply chain (as per standard EE approaches) but in add-
ition includes the costs of strengthening HS performance (audits,
Downloaded from https://academic.oup.com/heapol/advance-article/doi/10.1093/heapol/czaa116/5999228 by Karolinska Institutet University Library user on 20 December 2020
inspections, licencing, etc.) which have been considered in standard
EE as ‘above service delivery’ costs, and frequently excluded (Clift
et al., 2016).
The remaining two system wide papers focused on the health
workforce building block and HSS2 (i.e. relaxing a health system
capacity constraint). One considered upstream and downstream
nurse retention strategies for rural parts of South Africa (Lagarde
et al., 2012); this paper was unique in that it defined a HS perform-
ance outcome measure in the cost-effectiveness findings (rural-nurse
year) as opposed to a health-related outcome. The other performed
a cost analysis of health extension workers (a form of community
health worker, CHW) for Ethiopia (Canavan et al., 2017). None of
these papers sought to unpack the impact of HSS on health out-
comes or on the uptake or utilization of disease-specific services.
HSS within disease-specific or programmatic
evaluations
The majority of papers (27) considered various forms and combina-
tions of HSS investments, across a range of building blocks, using a
disease-specific or programmatic lens. This lens could be relatively
wide, including clusters of conditions or services [e.g. Mental,
Neurological and Substance Use Disorders (MNS); childhood ill-
nesses; surgery; vaccines; Maternal and Child Health services] or
more narrow (e.g. Malaria treatment; facility-based births; diabetes
care; specific contraceptives).
In terms of type of HSS investment included in the disease-
specific category, 10 papers included a focus on HSS1 (i.e. interven-
tions seeking to change the technical efficiency of the health system),
1 focused on HSS2 (i.e. relaxing a health system capacity con-
straint), 3 focused on HSS3 (i.e. developing a new health system
platform) and 9 focused on HSS1&2 (i.e. relaxing a health system
capacity constraint plus implementing measures to ensure that the
new resources are used in a technically efficient manner). The
remaining 4 papers sought to include a broader range of HSS; of
which 2 papers focused on surgical services and 2 focused on MNS.
Health Policy and Planning, 2020, Vol. 0, No. 0
The papers on surgical services were based on a combination of lit-
erature review and cost (Verguet et al., 2015) or cost-effectiveness
(Mock et al., 2015) modelling across all LMICs in the former or all
low- and lower middle-income countries (LLMICs) in the latter. For
Mock et al, which is part of DCP-3, the innovation is a focus on the
cost-effectiveness of the delivery platform, seeking to recommend
that surgeries are conducted at the most cost-effective level of care
where delivery is feasible. In addition, the authors focus on quantify-
ing infrastructure needs, given that first level hospitals with operat-
ing rooms will need to be built to increase access to these surgical
procedures. Similarly, in their cost analysis, Verguet et al. estimate
the number of operating rooms that will need to be constructed to
achieve scale up targets. The two MNS papers included one DCP-3
chapter (Levin and Chisholm, 2016) focused on LMICs, and an add-
itional analysis focused on specific countries within SSA and South
Asia (Chisholm et al., 2017) and specific MNS disorders (psychosis,
depression and epilepsy). The latter paper focused on the develop-
ment and application of a new module within the OneHealth Tool
to cost these conditions. As mentioned, this tool has a health systems
application, although the authors caution that the data and know-
ledge requirements to use the health system components—such as
logistics and governance—are seldom available. Similar to the DCP
chapter on essential surgery, the MNS chapter has a focus on the de-
livery platform and/or delivery agent as being of key concern from
an efficiency and coverage (or equity) perspective (Levin and
Chisholm, 2016).
The 10 papers focusing on HSS1 (i.e. seeking to influence the
technical efficiency of the health system) included one on pay-for-
performance in Tanzania (Borghi et al., 2015); various quality im-
provement initiatives including for IMCI in Kenya (Barasa et al.,
2012) and in Rwanda (Manzi et al., 2018); various voucher schemes
encouraging ANC and facility-based births in Myanmar (Kingkaew
et al., 2016) and Uganda (Alfonso et al., 2015) and encouraging
pregnant women to purchase ITNs in Tanzania (Mulligan et al.,
2008); a ‘detailing’ intervention consisting of external supervision
and training to encourage the uptake of intra-uterine devices in
Kenya (Wesson et al., 2008); capitation as a provider payment
mechanism for diabetes care in South Africa (Volmink et al., 2014);
an IT-based surveillance mechanism to detect disease outbreak in
China (Ding et al., 2015) and a pharmacist intervention focusing on
safe and rational drug use within an intensive care unit in China
(Jiang et al., 2012).
Nine papers focused on interventions that were a combination of
HSS1 and HSS2. Largely, these interventions introduced a new form
of health worker or infrastructure to relax a health system capacity
constraint (HSS2) in addition to various measures to encourage the
efficient utilization of the new resource (technical efficiency—
HSS1). Interventions within this subset tended to be multicompo-
nent, including facility rehabilitation, training, supervision and
performance-based financing in Benin (Paul et al., 2017); a subsi-
dized insurance system including quality of care interventions in
Nigeria (Ruelas et al., 2012); provision of training, medical supplies
and equipment together with home visits by CHWs to antenatal and
postnatal mothers in Uganda (Ekirapa-Kiracho et al., 2017); com-
munity mobilization plus a CHW package of care for mothers and
newborns in Malawi (Greco et al., 2017); a voucher scheme, com-
munity mobilization, training, supportive supervision and provision
of supplies to encourage facility-based births in Uganda (Mayora
et al., 2014); CHW home visits during pregnancy and postpartum
plus provision of equipment in Tanzania (Manzi et al., 2017); train-
ing of shopkeepers including provision of job aids and charts plus
follow-on support by CHWs and community mobilization to enable
7
correct treatment for malaria in Kenya (Goodman, 2006); super-
vised CHWs for diagnosis and treatment of malaria in Myanmar
(Kyaw et al., 2016) and child health days (campaigns) to increase
Downloaded from https://academic.oup.com/heapol/advance-article/doi/10.1093/heapol/czaa116/5999228 by Karolinska Institutet University Library user on 20 December 2020
coverage of Vitamin A and other key child health priorities such as
deworming in Ethiopia (Fiedler and Chuko, 2008).
The final three papers in this sub-group were all focused on
HSS3 and investments in new platform capacities. Of these, two
focused on supply chain management within the medical products,
vaccines and technologies building block (1 on scaling up the vac-
cine supply chain across LMICs (Portnoy et al., 2015) and 1 on inte-
grating nutritional commodities into the existing supply chain in
Kenya (Eby et al., 2019)]. The final paper in this set was focused on
a remote monitoring system for home-based non-invasive positive
pressure ventilation of children with upper airways obstruction in
China (Zhou et al., 2012).
These were all detailed empirical analyses seeking to test out in-
novative approaches to improve quality and demand for services.
These detailed empirical analyses are valuable for providing ideas of
what might work in different contexts and within different disease
areas.
HSS and type of building block
It was possible to classify by type of building block (noting that
many papers included >1 building block), using the categorizations
from Table 2, as illustrated in Figure 3 for 25 of the included papers.
The most common building blocks were service delivery (particular-
ly quality of care and demand for care) and health workforce (with
close to equal numbers for supply, management and training of
health workers). The financing building block was third most com-
mon, with provider payment mechanisms being more frequently
evaluated than beneficiary payment mechanisms. Few papers
focused on medicines (e.g. issues of safety or supply chain) or on in-
formation systems. There were no papers focused on governance.
While some papers focused on community mobilization, this was
expressed in terms of encouraging the demand for services and was
not expressed in terms of community participation to encourage ac-
countability or responsiveness. These interventions were therefore
classified as part of the service delivery building block and not as
governance.
Figure 4cross-tabulates the type of HSS investment against build-
ing block subcomponents. Of the 25 empirical papers considered
here, 40% could be categorized as HSS1, 12% as HSS2, 12% as
HSS3 and 36% were a combination of HSS1 and HSS2. Within the
reviewed literature, HSS1 was more often focused on the service de-
livery and health workforce building blocks. Examples include inter-
ventions to enhance quality of care and voucher schemes to enhance
demand for care within the service delivery building block while
training was a common HSS1 investment under the health work-
force building block. HSS2 is an investment that is designed to relax
a platform capacity constraint; here the most commonly evaluated
intervention was increasing supply of health workers through task
shifting. HSS3 refers to an investment into a new element within the
platform. Here, the few examples within the reviewed literature
included two supply chain evaluations (one for vaccines and one for
nutrition commodities) and one evaluation of a remote monitoring
system for home-based ventilation of children and infants with
upper airway obstruction. Finally, of the 36% of papers that consid-
ered a combination of HSS 1&2 (i.e. technical efficiency and relax-
ing a key capacity constraint), the most common intervention was to
encourage demand for care (e.g. through vouchers and community
8 Health Policy and Planning, 2020, Vol. 0, No. 0
Figure 4 Types of HS building blocks included in economic evaluations
Figure 5 Types of building blocks and types of HSS investments
mobilization) and quality of care while task shifting services to in-
crease the supply of health workers (Figure 5).
Discussion
This review of the literature has identified 33 economic evaluation
of HSS investments from LMIC settings. Most papers (25/33) under-
took economic evaluations of HSS within disease-specific or pro-
grammatic areas, such as Malaria prevention or facility-based
births. While some consider HSS to be horizontal or system wide
Downloaded from https://academic.oup.com/heapol/advance-article/doi/10.1093/heapol/czaa116/5999228 by Karolinska Institutet University Library user on 20 December 2020
(Adam et al., 2012), such a definition would lead to the exclusion of
the majority of the economic evaluation literature. For this reason,
disease-specific or programmatic HSS evaluations were included. A
smaller set of papers (8/33) focused on specifying benefit packages
and included HSS within overall estimates. Only 1 paper was from
South America, while most papers were from African settings.
In terms of methods, the programme- or disease-specific HSS lit-
erature included in this review followed similar approaches as would
be taken within standard economic evaluations of medicines or diag-
nostics. These papers provided detailed empirical data on costs as
Health Policy and Planning, 2020, Vol. 0, No. 0
well as natural outcome measures. In addition, a small number of
these papers demonstrated the feasibility of providing modelled
multi-dimensional units (such as DALYs) to enable the fuller as-
sessment of allocative efficiency. This is an encouraging approach,
enabling a consideration of the relative opportunity cost and value
for money of HSS in comparison to disease-specific investments.
This review did not retrieve any studies evaluating economies of
scope from single vs joint production, or estimating the cost-
effectiveness of HSS interventions across the health system more
broadly, e.g. the implementation of a health information system.
It therefore remains to be seen whether the conceptualizations of
Hauck and colleagues (2019) would be feasible within empirical
research.
In contrast to these detailed empirical papers, eight studies
focused on estimating benefits packages more broadly, whether for
specific settings (e.g. LLMICs) or for specific programmatic areas
(e.g. surgery or MNS across LMICs). These papers are unable to
provide details of the relative value for money of HSS in a granular
or nuanced way, however they are able to provide an indication of
what level of overall investment in HSS might be cost-effective or
might be needed to achieve key strategic goals. In other words, these
papers will not give insight into the cost-effectiveness of implement-
ing a health information system, e.g. they can provide an indication
of how much in total should be invested in HSS.
Standard approaches towards economic evaluation have typical-
ly recommended the inclusion of the costs associated with infra-
structure, facility management and facility information systems,
etc., frequently categorizing these as overhead costs, capital costs
and above service delivery costs (Drummond et al., 2015; Clift
et al., 2016). This review identified a growing practice of naming
these costs ‘health systems’ costs, which is useful for enabling the
economic evaluation literature to speak to a broader health policy
and systems audience. In addition, this approach helps to highlight
that the majority of costs is health systems costs. This is an import-
ant insight for donors that have pursued disease-specific outcomes
without funding the full range of health system inputs, with negative
implications for crowding out the services that are not donor prior-
ities, but which depend on the same health system platforms for
delivery.
This paper has a number of limitations. First, given the focus on
scoping the literature and describing current methods, no attempt
was made to limit the inclusion of studies to those meeting particu-
lar quality standards. Second, this paper has a single author, mean-
ing that only one person carried out the data extraction and
analysis. Part of this analysis included making a judgement about
the type of economic evaluation and type of costing perspective, in
the instances where this was not clearly identified within the papers.
Third, it is relatively difficult to distinguish between disease-specific
interventions that are about HSS vs those that are about medicines
and diagnostics. Realistically, there is a great deal of overlap given
that evaluations of the cost-effectiveness of medicines and diagnos-
tics will also include health systems costs. Fourth, 3 papers were not
found in the electronic search, of which 1 was part of a broader
DCP volume that had been retrieved (Levin and Chisholm, 2016)
and 2 were included through prior knowledge of the literature
(Lagarde et al., 2012; Borghi et al., 2015). The latter two papers did
not use the term ‘health system(s) strengthening OR development’
within the body of the article. It is therefore likely that there are
other economic evaluations of HSS in LMICs that have not been
retrieved for similar reasons. Indeed, while this paper did find a few
articles on performance-based financing, the search did not find all
9
of the papers retrieved by a recent review on this topic (Turcott-
Tremblay et al., 2016). While this shortcoming had potentially been
mitigated through the inclusion of studies that are indexed in the
Downloaded from https://academic.oup.com/heapol/advance-article/doi/10.1093/heapol/czaa116/5999228 by Karolinska Institutet University Library user on 20 December 2020
Health Systems Evidence database, it is likely that studies have been
missed. Thus while this paper does not attempt to provide a system-
atic review, these limitations nevertheless underscore the prelimin-
ary nature of this synthesis.
Conclusion
In conclusion, this review identified a small body of economic evalu-
ations of HSS investments from LMIC settings. The detailed empir-
ical papers included in this review demonstrated the feasibility of
conducting economic evaluations of HSS using standard
approaches. This is complemented by the broader benefits package
papers which provide a mechanism for estimating the overall level
of investment in HSS that could be cost-effective, or that would be
needed to achieve key goals. However, the overall evidence base
remains small, and further research should be encouraged to support
ongoing progress towards UHC.
Funding
Susan Cleary is a member of the Consortium for Resilient and Responsive
Health Systems (RESYST). This article is an output from a project funded by
the UK Aid from the UK Department for International Development (DFID)
for the benefit of developing countries. However, the views expressed and in-
formation contained in this article are not necessarily those of or endorsed by
DFID, which can accept no responsibility for such views or information or for
any reliance placed on them.
Acknowledgements
The author gratefully acknowledges the inputs from two anonymous
reviewers.
Conflict of interest statement. None declared.
Ethical approval. No ethical approval was required for this study.
References
Adam T, Hsu J, de Savigny D et al. 2012. Evaluating health systems strength-
ening interventions in low-income and middle-income countries: are we ask-
ing the right questions? Health Policy and Planning 27: iv9–19.
Alfonso YN, Bishai D, Bua J et al. 2015. Cost-effectiveness analysis of a vou-
cher scheme combined with obstetrical quality improvements: quasi experi-
mental results from Uganda’. Health Policy and Planning 30: 88–99.
Barasa EW, Ayieko P, Cleary S et al. 2012. A multifaceted intervention to im-
prove the quality of care of children in district hospitals in Kenya: a
cost-effectiveness analysis. PLoS Medicine 9: e1001238.
Borghi J, Little R, Binyaruka P et al. 2015. In Tanzania, the many costs of
pay-for-performance leave open to debate whether the strategy is cost-effec-
tive. Health Affairs 34: 406–14.
Canavan ME, Linnander E, Ahmed S et al. 2017. Unit costing of health exten-
sion worker activities in Ethiopia: a model for managers at the district and
health facility level. International Journal of Health Policy and
Management 7: 394–401.
Cantelmo CB, Takeuchi M, Stenberg K et al. 2018. Estimating health plan
costs with the OneHealth tool, Cambodia. Bulletin of the World Health
Organization 96: 462–70.
Chisholm D, Heslin M, Docrat S et al. 2017. Scaling-up services for psychosis,
depression and epilepsy in sub-Saharan Africa and South Asia: development
10
and application of a mental health systems planning tool (OneHealth).
Epidemiology and Psychiatric Sciences 26: 234–44.
Cleary S, Mooney G, McIntyre D. 2010. Equity and efficiency in
HIV-treatment in South Africa: the contribution of mathematical program-
ming to priority setting. Health Economics 19: 1166–80.
Clift J, Arias D, Chaitkin M et al. 2016. Landscape Study of the Cost, Impact,
and Efficiency of above Service Delivery Activities in HIV and Other Global
Health Programs. Results for Development Institute. http://www.r4d.org/
wp-content/uploads/R4D_ServDeliveryLandscaping_Final-0825_web.pdf
Cookson R, Mirelman A, Grif S et al. 2017. Using cost-effectiveness analysis
to address health equity concerns. 20: 206–212. doi:
10.1016/j.jval.2016.11.027
Ding Y, Sauerborn R, Xu B et al. 2015. A cost-effectiveness analysis of three
components of a syndromic surveillance system for the early warning of epi-
demics in rural China Health policies, systems and management in low and
middle-income countries’. BMC Public Health 15: 1–9.
Drummond MF et al. 2015. Methods for the Economic Evaluation of Health
Care Programmes, 4th edn. Oxford: Oxford University Press.
Eby E, Daniel T, Agutu O et al. 2019. Integration of the UNICEF nutrition
supply chain: a cost analysis in Kenya’. Health Policy and Planning 34:
188–96.
Ekirapa-Kiracho E, Barger D, Mayora C et al. 2017. Uganda Newborn Study
(UNEST) trial: community-based maternal and newborn care economic
analysis’. Health Policy and Planning 32: i42–52.
Fiedler JL, Chuko T. 2008. The cost of Child Health Days: a case study of
Ethiopia’s Enhanced Outreach Strategy (EOS). Health Policy and Planning
23: 222–33.
Glassman A, Giedion U, Smith PC. 2017. What’s In, What’s Out? Designing
Benefits for Universal Health Coverage. Washington, DC: Center for
Global Development, 1–398.
Goodman CA. 2006. The cost-effectiveness of improving malaria home man-
agement: shopkeeper training in rural Kenya’. Health Policy and Planning
21: 275–88.
Greco G, Daviaud E, Owen H et al. 2017. Malawi three district evaluation:
community-based maternal and newborn care economic analysis. Health
Policy and Planning 32: i64–74.
Hauck K, Morton A, Chalkidou K et al. 2019. How can we evaluate the
cost-effectiveness of health system strengthening? A typology and illustra-
tions. Social Science & Medicine 220: 141–9.
Helmchen LA, Lo Sasso AT. 2010. Outcomes in economic evaluations. Health
Economics 19: 1300–17.
Husereau D, Drummond M, Petrou S et al. 2013. Consolidated Health
Economic Evaluation Reporting Standards (CHEERS)—explanation and
elaboration: a report of the ISPOR Health Economic Evaluations
Publication Guidelines Task Force. Value in Health 16: 231–50.
Jiang SP et al. 2012. Effectiveness of pharmaceutical care in an intensive care
unit from China’. Saudi Medical Journal 33: 756–62.
Kingkaew P, Werayingyong P, Aye SS et al. 2016. An ex-ante economic evalu-
ation of the Maternal and Child Health Voucher Scheme as a
decision-making tool in Myanmar. Health Policy and Planning 31: 482–92.
Kutzin J. 2013. Health financing for universal coverage and health system per-
formance: concepts and implications for policy’. Bulletin of the World
Health Organization 91: 602–11.
Kyaw SS, Drake T, Thi A et al. 2016. Malaria community health workers in
Myanmar: a cost analysis. Malaria Journal 15: 1–7.
Lagarde M, Blaauw D, Cairns J. 2012. Cost-effectiveness analysis of human
resources policy interventions to address the shortage of nurses in rural
South Africa’. Social Science & Medicine 75: 801–6.
Levin C, Chisholm D, 2016. Cost-effectiveness and affordability of interven-
tions, policies, and platforms for the prevention and treatment of mental,
neurological, and substance use disorders. In: Patel V et al. (eds). Mental,
Neurological, and Substance Use Disorders: Disease Control Priorities, 3rd
edn. Washington, DC, 219–236. doi: 10.1017/CBO9781107415324.004
Manzi A, Mugunga JC, Iyer HS et al. 2018. Economic evaluation of a mentor-
ship and enhanced supervision program to improve quality of integrated
Health Policy and Planning, 2020, Vol. 0, No. 0
management of childhood illness care in rural Rwanda’. PLoS One 13:
e0194187.
Manzi F, Daviaud E, Schellenberg J et al. 2017. Improving Newborn Survival
Downloaded from https://academic.oup.com/heapol/advance-article/doi/10.1093/heapol/czaa116/5999228 by Karolinska Institutet University Library user on 20 December 2020
in Southern Tanzania (INSIST) trial; community-based maternal and new-
born care economic analysis. Health Policy and Planning 32: i33–41.
Mayora C, Ekirapa-Kiracho E, Bishai D et al. 2014. Incremental cost of
increasing access to maternal health care services: perspectives from a de-
mand and supply side intervention in Eastern Uganda. Cost Effectiveness
and Resource Allocation 12: 14–9.
Mills A, Rasheed F, Tollman S. 2006. Strengthening health systems. In:
Jamison D, Breman J, Measham A (eds). Disease Control Priorities in
Developing Countries, 2nd edn. Washington, DC/New York: The
International Bank for Reconstruction and Development/The World
Bank/Oxford University Press.
Mock C, Donkor P, Gawande A et al. 2015. Essential surgery: key messages of
this volume. In: Debas H et al. (eds). Essential Surgery: Key Messages of
This Volume. In Essential Surgery: Disease Control Priorities, 3rd edn.
Washington, DC, 1–19. doi: 10.1596/978-1-4648-0346-8_ch19
Morton A, Thomas R, Smith PC. 2016. Decision rules for allocation of finan-
ces to health systems strengthening. Journal of Health Economics 49:
97–108.
Mulligan JA, Yukich J, Hanson K. 2008. Costs and effects of the Tanzanian
national voucher scheme for insecticide-treated nets. Malaria Journal 7:
1–10.
Murray CJ, Kreuser J, Whang W. 1994. Cost-effectiveness analysis and policy
choices: investing in health systems. Bulletin of the World Health
Organization 72: 663–74.
Ochalek J, Revill P, Manthalu G et al. 2018. Supporting the development of a
health benefits package in Malawi. BMJ Global Health 3: e000607.
Paul E, Lamine Dramé M, Kashala J-P et al. 2017. Performance-based financ-
ing to strengthen the health system in Benin: challenging the mainstream ap-
proach. International Journal of Health Policy and Management 7: 35–47.
Portnoy A, Ozawa S, Grewal S et al. 2015. Costs of vaccine programs across
94 low- and middle-income countries’. Vaccine 33: A99–108.
Ruelas E, Gomez-Dantes O, Leatherman S et al. 2012. Strengthening the qual-
ity agenda in health care in low- and middle-income countries: questions to
consider. International Journal for Quality in Health Care 24: 553–7.
Sixty-Seventh World Health Assembly. 2014. Health Intervention and
Technology Assessment in Support of Universal Health Coverage.
WHA67.23(24 May 2014), 7–9. http://apps.who.int/medicinedocs/docu
ments/s21463en/s21463en.pdf
Stenberg K, Hanssen O, Tan-Torres T et al. 2017. Financing transformative
health systems towards achievement of the health Sustainable Development
Goals: a model for projected resource needs in 67 low-income and
middle-income countries. Lancet Global Health 1–13. doi:
10.1016/S2214-109X(17)30263-2
Stinnett AA, Paltiel AD. 1996. Mathematical programming for the efficient al-
location of health care resources. Journal of Health Economics 15: 641–53.
Turcotte-Tremblay, Anne Marie, Jessica Spagnolo, Manuela De Allegri, and
Valéry Ridde. 2016. Does performance-based financing increase value for
money in low- and middle- income countries? a systematic review. Health
Economics Review 6 (1). 10.1186/s13561-016-0103–9.
United Nations. 2015. Transforming our World: The 2030 Agenda for
Sustainable Development. doi: 10.1201/b20466-7
Vassall A, Bozzani F, Hanson K et al. 2019. Considering health-systems con-
straints in economic evaluation in low- and middle-income settings. Oxford
Research Encyclopedia of Economics and Finance 1–25. Oxford University
Press. doi: 10.1093/acrefore/9780190625979.013.38
Verguet S, Alkire BC, Bickler SW et al. 2015. Timing and cost of scaling up
surgical services in low-income and middle-income countries from 2012 to
2030: a modelling study. The Lancet Global Health 3: S28–37.
Volmink HC, Bertram MY, Jina R et al. 2014. Applying a private sector capi-
tation model to the management of type 2 diabetes in the South African pub-
lic sector: a cost-effectiveness analysis. BMC Health Services Research 14:
1–9.
Health Policy and Planning, 2020, Vol. 0, No. 0
Warren AE, Wyss K, Shakarishvili G et al. 2013. Global health initiative
investments and health systems strengthening: a content analysis of global
fund investments. Globalization and Health 9: 30.
Watkins D, Jamison D, Mills T et al. 2017. Universal Health Coverage and es-
sential packages of care. In: Jamison D et al. (eds). Disease Control
Priorities: Improving Health and Reducing Poverty, 3rd edn. Washington,
DC: The International Bank for Reconstruction and Development/The
World Bank, 43–68.
Wesson J, Olawo A, Bukusi V et al. 2008. Reaching providers is not enough to
increase IUD use: a factorial experiment of “Academic Detailing” in Kenya.
Journal of Biosocial Science 40: 69–82
Wiseman V, Mitton C, Doyle-Waters M et al. 2016. Using economic evidence
to set healthcare priorities in low-income and lower-middle-income
11
countries: a systematic review of methodological frameworks. Health
Economics S37–54. doi: 10.1002/hec
World Health Organization. 2007. Everybody’s Business: Strengthening
Downloaded from https://academic.oup.com/heapol/advance-article/doi/10.1093/heapol/czaa116/5999228 by Karolinska Institutet University Library user on 20 December 2020
Health Systems to Improve Health Outcomes: WHO’s Framework for
Action. Geneva. doi: 10.1371/journal.pone.0013372
Zeng W, Li G, Ahn H et al. 2018. Cost-effectiveness of health systems
strengthening interventions in improving maternal and child health in low-
and middle-income countries: a systematic review. Health Policy and
Planning 33: 283–97.
Zhou J, Liu D-B, Zhong J-W et al. 2012. Feasibility of a remote monitoring
system for home-based non-invasive positive pressure ventilation of children
and infants. International Journal of Pediatric Otorhinolaryngology. 76:
1737–40.