10th European Public Health Conference: Parallel sessions 303
questions posed to all on ‘Easy to Read’ and ‘Easy to
Understand’ language, thus avoiding stigmatization and
methodological issues. The same holds for other
impairments.
10.Q. Workshop: Evaluating policy using natural
experiments and quasi-experimental Methods
Chair: Alastair Leyland, UK
Organised by: EUPHA (EPI)
Contact: alastair.leyland@glasgow.ac.uk
A broad range of policies from the health and non-health
sectors have the potential to impact on population health. But
despite the possibilities for health improvement, and the desire
of governments to improve health, policies are rarely
introduced in such a fashion that facilitates the evaluation of
their impact on health. A randomised design would allow
variations in policy to be compared, or a new policy to be
contrasted with an old one, in a manner that would exclude
contamination from individual or contextual characteristics.
In the absence of a gold standard randomised design,
researchers make the most of natural experiment and quasi-
experimental designs to provide the best possible evidence as
to the impacts of policies on health. Such evidence can be of
great interest to those who have introduced or are considering
the introduction of such policies.
This workshop brings together four papers from experts in the
field covering a broad range of natural experiment and quasi-
experimental methods to evaluate a number of different health
and social policies of relevance to public health and health
systems research.
This workshop is aimed at the increasing number of
researchers and policy-makers who are interested in the use
of non-experimental designs, particularly for the evaluation of
policy. The papers and discussion will consider the necessary
compromise between those methods that provide the best
possible evidence – coming closest to having the properties of a
randomised design – and the need for evidence regarding the
effectiveness of policies.
Key messages:
 Upstream interventions, including policy changes, have the
greatest potential for effective and cost-effective behaviour
change.
 The evaluation of policy requires a range of natural
experiment and quasi-experimental methods.
Natural experiments and observational studies:
where and how should we draw the line?
Peter Craig
P Craig
Glasgow University, Glasgow, UK
Contact: peter.craig@glasgow.ac.uk
Background
It is now widely accepted that natural experimental approaches
widen the range of policies that can be rigorously evaluated,
but there is less consensus about where and how to draw the
line between natural experimental studies and other kinds of
observational studies. ‘As if randomisation’ and the use of
methods that address unobserved confounders have been
proposed as distinguishing features of natural experiments.
This talk will use examples drawn from recent studies of
welfare reform to argue that such criteria risk excluding well-
designed studies that permit strong causal inference.
Lessons
Through adapting systems and their evaluation by taking into
account the needs and views of disabled people we will cater
for all members of society.
Methods
Downloaded from https://academic.oup.com/eurpub/article-abstract/27/suppl_3/ckx187.817/4556778 by guest on 19 April 2020
We used data from the UK Understanding Society Survey to
identify lone mothers who were newly exposed to the
requirement to seek work as a condition of receiving social
security benefits. We compared changes in their health with
changes among mothers who were unexposed to the job-
seeking requirement, and mothers who were continuously
exposed. We fitted difference in differences models to control
for changes in unobserved fixed characteristics of the three
groups.
Results
Mental health scores worsened in mothers newly exposed to
the requirement to seek work, by comparison with mothers
who were continuously exposed or unexposed. There was no
difference in physical health scores. Our study fails the
criterion of as-if randomisation, but plausibly identifies the
impact of the policy on lone parents’ mental health.
Conclusions
Criteria such as ‘as-if randomisation’ are useful starting points
for designing natural experimental studies, but the plausibility
of causal inference depends on the strength of the assumptions
required, the extent to which these can be directly tested, and
whether alternative explanations can be ruled out. These need
to be assessed on a case-by-case basis, rather than by using
general rules to define which methods are best in all cases.
Quasi-experiments in public health and health
systems research
Till Bärnighausen
T Bärnighausen
University of Heidelberg, Heidelberg, Germany
Contact: till.baernighausen@uni-heidelberg.de
Background
Strong causal analysis of health policy impacts is key to good
decision-making, and ultimately good population health
(European Commission 2011, 2015). However, randomized
controlled experiments in the health sector are commonly not
feasible because of ethical, political or financial constraints.
Quasi-experiments provide a good opportunity for causal
analyses in these situations, and they have the added advantage
that they avoid the artificiality that trials introduce into the
study context because of the trial-associated intervention and
measurement.
Methods
We utilize a range of quasi-experimental approaches –
regression discontinuity, instrumental variable, fixed-effects
and difference-in-differences approaches – to illustrate the
potential of quasi-experiments to establish the causal impacts
of health policies.
Results
Quasi-experiments are currently relatively under-utilized in
health research, but they are increasingly adopted for causal
impact evaluation of health practice, programmes and
policy. We illustrate the potential of quasi-experiments
using (1) difference-in-differences analysis to demonstrate
the causal impact of the Family Medicine reform on patient
satisfaction in Turkey; (2) fixed-effects analysis to quantify
304 European Journal of Public Health, Vol. 27, Supplement 3, 2017
the impact of adding health workers to municipalities on
child mortality in Mexico; (3) regression discontinuity
analysis to demonstrate the causal impact of HIV treatment
on mortality, employment and education in South Africa;
(4) instrumental variable approaches to establish the impact
of HIV status knowledge on linkage to HIV treatment in
South Africa.
Conclusions
Quasi-experimental approaches can provide powerful oppor-
tunities to establish causal impacts of health policies. Quasi-
experiments occupy a sweet spot between experiments and
non-experiments, because they can generate results of both
high internal and high external validity. Novel falsification tests
can be used to rule out invalid quasi-experiments.
Evaluating the impact of pictorial health warnings on
Italian smokers: before-after cohort study
Giuseppe La Torre
G La Torre
La Sapienza University, Rome, Italy
Contact: giuseppe.latorre@uniroma1.it
Background
Tobacco consumption causes nearly 6 million deaths world-
wide and costs hundreds of billions of dollars of economic
damage per year. This study investigates the impact of the
introduction of pictorial health warnings on Italian smokers.
Methods
This is a before-after cohort from 9 centres involving a sample
of 370 adult smokers (> 18 years) resident in 47 provinces of
Italy. Smokers included in the study responded, on a voluntary
basis, to a first wave questionnaire before the introduction of
graphic warnings in Italy. Six months after the introduction of
the graphic warnings, the same sample was contacted, by
telephone, for a second wave questionnaire.
Results
370 and 265 questionnaires were collected at the first and
second waves respectively with 28.4% lost to follow-up.
Among those seen at both waves, a significant increase
(p < 0.001) was seen in knowledge of tobacco related diseases
and toxic substances contained in cigarettes, and an increase of
16.1% was recorded in the knowledge of toxic substances
present in cigarettes or tobacco smoke while knowledge of
smoking-related clinical conditions increased by 10.4%. All
four dimensions of the Label Impact Index increased after the
introduction of pictorial health warnings.
42 smokers (15.8%) stopped smoking; 15 before the
introduction of graphic warnings and 27 after. More smokers
stopped smoking after than before the introduction of pictorial
warnings (not significant).
Conclusions
Pictorial health warnings on tobacco packaging have a
greater impact on smokers, compared to text only warnings,
as they attract more attention, cause more discomfort and
fear and promote behavioural changes. Moreover they
increase awareness of the risk for health caused by tobacco
products and of toxic compounds in cigarettes. Finally the
fact that no socio-demographic factors or the level of
nicotine dependence is associated with quitting suggests
that pictorial health warnings may impact the whole smoking
population.
Reductions in housing benefit increases depression
risk in low-income UK households
Aaron Reeves
Downloaded from https://academic.oup.com/eurpub/article-abstract/27/suppl_3/ckx187.817/4556778 by guest on 19 April 2020
A Reeves
LSE, London, UK
Contact: A.Reeves@lse.ac.uk
Background
Housing provides shelter and security but the security of
housing can be undermined when incomes fall. We evaluate
the mental health effects of the UK government’s reduction in
financial support to help low-income persons afford private-
sector housing rent.
Methods
We use a quasi-natural experiment design, using difference-in-
differences models, to compare the prevalence of mental ill
health before and after the April 2011 changes to the Local
Housing Allowances, which was reduced from the median
house price to the 30th percentile, corresponding to a mean
reduction estimated to be £1,220 per year ($2,315).
Data come from the UK Annual Population Survey, a repeated
quarterly cross-sectional survey, between April 2009 and March
2013. We focus our analysis on renters in the private sector
(n = 179,064), disaggregating between an intervention group
receiving housing benefit (n = 36,859) and a control group not
receiving housing benefit (n = 142,205). The main outcome of
interest was self-reported health problems, distinguishing mental
(e.g., depression and anxiety) and physical health (e.g., diabetes)
(1 = self-reported presence of a condition; 0 = otherwise).
Results
After controlling for pre-existing time trends we observed that,
between April 2011 and March 2013, the prevalence of
depressive symptoms in private renters receiving housing
benefit increased by 1.8 percentage points (95% CI: 1.0 to 2.7)
compared with those not receiving housing benefit. This rise in
depressive symptoms was concentrated in urban areas, where
reductions were greatest. Our models estimate approximately
26,000 (95% CI: 14,000 to 38,000) people newly experienced
depressive symptoms in association with the cuts to housing
benefit.
Conclusions
Reducing housing support to low-income persons in the
private rental sector increases the prevalence of depressive
symptoms.