Real-world evidence in the context of health

technology assessment processes – from theory to

action
December 2018 Wija Oortwijn HTAi Policy Forum Scientific
Secretary 2018–2019 8 Background 1.

Introduction
This paper is to inform the discussion at the HTAi Global Policy
Forum (GPF) meeting in Barcelona, January 2019. The main aim of
the HTAi GPF meeting is to discuss – at a strategic level – what
the leadership roles of HTA and all its relevant stakeholders are
in shaping the future availability and potential use of real-
world evidence (RWE) in the context of health technology
assessment (HTA) processes to inform decision-making. In the HTA
glossary1 RWE is defined as “evidence derived from the analysis
of real world data” (RWD).2 RWD is defined as “observational or
administrative data that provides information on the routine
delivery of health care and the health status of the target
population.”3 The topic was selected by HTAi GPF member
representatives and HTAi Board members on the basis that the 21st
century is bringing new sources and methodological ways of
capturing the effects of health technologies in the real world.
However, it is not a new topic; RWE is already used in several
HTA practices4 and regulators are beginning to elevate the
importance of RWE in their deliberations. Much has already been
said and written about the challenges regarding methodologies
used to generate RWE, as well as the relevance and the
reliability of underlying RWD5 , and it is acknowledged that it
is now time for action.6 This paper is therefore not extensively
addressing the methodological and technical issues but rather
focuses on describing the key challenges and opportunities for
using RWE in the context of HTA from a policy perspective.
Already in 2007, the International Society for Pharmacoeconomics
and Outcomes Research (ISPOR) Task Force on RWD stated, “health
decision-makers involved with coverage and payment policies are
increasingly developing policies that seek information on ‘real-
world’ outcomes”.7 Since then RWD and RWE are accelerating at an
unprecedented rate of development, size, and scale. This presents
challenges but also opportunities for stakeholders involved in
the production and use of HTA. The topic has also been
highlighted during previous HTAi Global and Regional Policy
Forums. During the 2014 HTAi GPF, the implications of new
adaptive approaches to licensing, using an “evolving” evidence
base (e.g. via patient registries), were explored.8 At the 2015
HTAi GPF meeting the topic of discussion was improving the
effectiveness and efficiency of evidence production in HTA,
including the opportunities provided by collaborative real-world
evaluation of technologies. It was stated that “HTA needs to…
actively align stakeholder expectations about realistic evidence
expectations…. Collaborations between technology developers and
health systems…. should be encouraged to develop evidence that
will inform decision making. New analytical techniques emerging
for real-world data should be harnessed….for HTA.”9 During the
2016 HTAi GPF, better use of RWD was highlighted as a theme for
changing the HTA paradigm; i.e. an issue where “innovation 1
Available at: htaglossary.net. The 2017 HTAi Asia Policy Forum
highlighted that there were no definitions of RWE and RWD in the
HTA Glossary and has worked to achieve this. The definitions will
soon be included in the HTA glossary. Personal communications
from the Chair of the Editorial Board of the international HTA
Glossary and the Chair of the HTAi Asia Policy Forum 2018-2019 2
Note: RWD are primarily analysed through observational study
designs. This RWE is characterised by the actual use of the
technology in practice and by findings that are generalizable to
the target population for the technology 3 Note: Sources may
include research data, patient-generated data or professional-
generated data. These data may be collected in administrative
datasets, case notes, surveys, product and disease registries,
social media, electronic health records, claims and billing
datasets, or mobile health applications. Source: Makady, A., de
Boer, A., Hillege, H. et al. What Is Real-World Data? A Review of
Definitions Based on Literature and Stakeholder Interviews’ Value
in Health, 2017; 20(7): 858-865 4 Makady, A. Real-world evidence
for health technology assessment for pharmaceuticals:
opportunities and challenges. PhD thesis. Amsterdam: Off Page,
May 2018. Available at:
https://dspace.library.uu.nl/bitstream/handle/1874/364283/Makady.
pdf?sequence=1&isAllowed=y (Accessed 10 October, 2018) 5 U.S.
Food and Drug Administration. Framework for FDA’s Real-World
Evidence Program. Silver Spring: FDA, December 2018. Available
at:
https://www.fda.gov/downloads/ScienceResearch/SpecialTopics/RealW
orldEvidence/UCM627769.pdf (Accessed 8 December, 2018) 6 The
Academy of Medical Sciences. Next steps for using real world
evidence. Summary report of a FORUM follow-up roundtable held on
24 January 2018. Available at: https://acmedsci.ac.uk/file-
download/7021031 (Accessed 2 November, 2018) 7 Garrison, L.,
Neuman, P., Erickson, P., et al. Using Real-World Data for
Coverage and Payment Decisions: The ISPOR Real-World Data Task
Force Report. Value in Health, 2007; 10(5): 326-335 8 Husereau,
D., Henshall, C., Jivraj, J. Adaptive Approaches to Licensing,
Health Technology Assessment, and Introduction of Drugs and
Devices. International Journal of Technology Assessment in Health
Care, 2014; 30(3): 241-248 9 Facey, K., Henshall, C., Sampietro-
Colom, L. et al. Improving the Effectiveness and Efficiency of
Evidence Production for Health Technology Assessment.
International Journal of Technology Assessment in Health Care,
2015; 31(4): 201-206 9 Background in HTA is needed”.10 The 2017
HTAi Asia Policy Forum on “Universal health care in Asia: HTA and
realworld data overcoming barriers” showed that access to RWD is
important for achieving universal health care. However, several
challenges remain, including a disconnection between HTA bodies
and industry regarding their knowledge of, and access to RWD as
well as limited trust between stakeholders regarding the use of
RWD for HTA purposes.11 In this paper and in the 2019 HTAi GPF
meeting the focus lies on important policy issues around the
generation and analysis of RWD, and specifically the use of RWE.
These issues may impact the way HTA is organised and produced, as
well as the relation between traditional, and potentially new,
stakeholders (e.g. those that collect and analyse RWD). There
are, however, different views on how this process towards use of
RWE in HTA should be approached and how to address the key
challenges. In addition, there is a need to ensure that HTA
processes continue to be systematic, unbiased, transparent,
relevant and meaningful for different settings and the
stakeholders involved, including patients, health professionals,
academia, industry, HTA bodies, regulators, policy makers and
payers. In order to inform the 2019 HTAi GPF meeting, this paper
was developed based on scientific and grey literature identified
by the author through an unstructured search in Google Scholar
based on recent key publications, reviewing websites/documents of
relevant networks (e.g. HTAi, International Network of Agencies
for Health Technology Assessment (INAHTA), European Network for
Health Technology Assessment (EUnetHTA), ISPOR), regulatory
agencies (e.g. European Medicines Agency (EMA), United States
(US) Food and Drug Administration (FDA)) and HTA organizations
using RWE,12 as well as input from the HTAi Policy Forum
Committee, Policy Forum members, HTAi Board members and the wider
HTA community.13 The paper starts with introducing the topic, and
describing the key challenges identified by GPF members and HTAi
Board members that need to be addressed in the context of HTA.
Thereafter, relevant information to address these challenges is
provided, as well as a description of the potential value of RWE
in the context of HTA. Finally, the objective of the 2019 HTAi
GPF meeting is provided along with several questions. The
questions are aimed to direct the discussion at the 2019 HTAi GPF
in Barcelona toward the strategic level regarding the key
challenges and opportunities identified and to move forward by
defining key actions. The Annex provides a – non-exhaustive –
overview of existing initiatives and policy-oriented documents
regarding the topic, including links to access the related
document(s). In addition, suggestions for a more in-depth reading
of relevant and brief policy papers (if wished) are given. 10
Husereau, D., Henshall, C., Sampietro-Colom, L. et al. Changing
Health Technology Assessment Paradigms? International Journal of
Technology Assessment in Health Care, 2016; 32(4): 191-199 11
Mundy, L., Trowman, R., Kearney, B. Universal Healthcare in Asia:
HTA and Real-World Data Overcoming Barriers. DIA Newsletter,
January 2017. Available at:
https://globalforum.diaglobal.org/issue/january-2018/universal-
healthcare-in-asia-hta-and-real-world-data-overcomingbarriers/
(Accessed 3 November, 2018) 12 See the Annex for an overview of
publications and websites consulted 13 Through an online
consultation of HTAi members during November 2018, asking to
provide feedback on the draft paper 10 Background 2. Challenges
of using RWE in the context of HTA The increased demand for using
RWE requires new methodologies for capturing RWD along the life
cycle through differing existing and new data sources (e.g.
claims databases, registries, electronic medical records,
wearables, social media platforms, websites from patient
organizations, genomics, biomarkers) and study designs (e.g.
health surveys, pragmatic clinical trials).14 Connecting real-
time data (e.g. via smart applications, wearables)15 with modern
technologies (e.g. big data architectures, blockchain, artificial
intelligence)16 is rapidly evolving17 and is raising high
expectations for the health sector, both in the short and medium
term.18 These developments, especially better use of RWD, will
impact the HTA paradigm as already stated in the Introduction. In
the Annex, we provide an overview of the definitions of RWD and
RWE used by several initiatives around the globe. As described in
the white paper from the Duke-Margolis Center for Health Policy
(2017) the “term RWE is often used when stakeholders are actually
describing the development or use of RWD for a variety of
purposes. However, it is acknowledged that data and evidence are
not the same; RWD is necessary but not sufficient for generating
RWE. There is a clear need to separate these concepts from one
another and to clarify the full range of RWE itself.”19,20 Most
often RWD is defined as: any data used for decision making that
was collected outside of a RCT.21,22 This is also reflected in
the definitions of RWD and RWE provided in the HTA Glossary (see
Introduction section), and these definitions are used throughout
this paper. The abundance of high quality RWD and RWE ultimately
will not only affect how HTA is done, it will also have
substantial implications for those who do HTA, and it will
require new skills or professional profiles. Furthermore, there
is discussion about when and how RWE should be used in HTA and
what consequences that would have for HTA bodies, patients,
health professionals, academia, industry, regulators, policy
makers and payers?23 And for what purpose (i.e. why)? Should RWD
(as part of all available data) be used to get better insight
into the value of different treatment pathways in practice at the
developmental phase, the time of market launch, and/or post-
market launch? As such, how do we better understand the
usefulness and challenges throughout the entire lifecycle, and
how do 14 Goettsch, W., Makady, A. IMI GetReal. WP1: Deliverable
D1.3. Glossary of definitions of common terms. October 2016.
Available at: http://
www.imi-getreal.eu/Portals/1/Documents/01%20deliverables/D1.3%20-
%20Revised%20GetReal%20glossary%20-%20FINAL%20 updated
%20version_25Oct16_webversion.pdf (Accessed 10 October, 2018) 15
Turakhia, M., Desai, M., Rajmane, A. et al. Rationale and design
of a large-scale, app-based study to identify cardiac arrhythmias
using a smartwatch: The Apple Heart Study. American Heart
Journal, 2018; 8 September, in press (corrected proof). Available
at https://doi. org/10.1016/j.ahj.2018.09.002 16 Hernán M.,
Robins, J. Using big data to emulate a target trial when an RCT
is not available. American Journal of Epidemiology,
2016;15;183(8):758-764 17 Turakhia, M., Desai, M., Rajmane, A. et
al. Rationale and design of a large-scale, app-based study to
identify cardiac arrhythmias using a smartwatch: The Apple Heart
Study. American Heart Journal, 2018; 8 September, in press
(corrected proof). Available at https://doi.
org/10.1016/j.ahj.2018.09.002 18 Van den Bergen, K., Oortwijn,
W., ten Have, A. et al. The future of the medical technology
market. Addressing challenges and utilizing opportunities.
Rotterdam: Ecorys, 2018. Available at:
https://www.rijksoverheid.nl/documenten/rapporten/2018/09/21/the-
future-of-themedical-technology-market-engels (Accessed 1
November, 2018) 19 Duke-Margolis Center for Health Policy. White
paper. A framework for regulatory use of real-world evidence,
2017. Available at: https://
healthpolicy.duke.edu/sites/default/files/atoms/files/rwe_white_p
aper_2017.09.06.pdf (Accessed 10 October, 2018) 20 Daniel, G.,
Frank, K., Romine, M et al. Summary of public comment. A
framework for regulatory use of real-world evidence. DukeMargolis
Center for Health Policy, 2018.Available at:
https://healthpolicy.duke.edu/sites/default/files/atoms/files/fin
al_rwe_comment_ synthesis_20180522.pdf (Accessed 10 October,
2018) 21 Miani, C., Robin, E., Horvath, V. et al. Health and
Healthcare: Assessing the real world data policy landscape in
Europe. Cambridge: RAND Europe, 2014. Available at:
https://www.rand.org/content/dam/rand/pubs/research_reports/RR500
/RR544/RAND_RR544.pdf (Accessed 2 November, 2018) 22 Garrison,
L., Neuman, P., Erickson, P., et al. Using Real-World Data for
Coverage and Payment Decisions: The ISPOR Real-World Data Task
Force Report. Value in Health, 2007; 10 (5); 326-335 23 Hebborn,
A. Reflections on the topic during the HTAi GPF scoping meeting
in Vancouver, presentation, June 2018 11 Background we optimise
the use of RWE in the context of HTA (e.g. how to integrate RWE
with other available evidence)? The key challenges related to the
use of RWE in HTA that were identified by the HTAi GPF and HTAi
Board members in the scoping meeting on the topic and through an
online consultation include: • For which HTA questions might RWE
be acceptable as fit for purpose? • When to use RWE across the
lifecycle • Quality of data from real world sources • Data
infrastructure and access to data • Transferability issues These
challenges are also reflected by other stakeholders and
initiatives (presented in the Annex) and further described below
from the HTA perspective, using relevant literature and
documents. For which HTA questions might RWE be acceptable as fit
for purpose? The key questions that are related to this issue
include: For what policy problems will the HTA community use RWE?
How can RWE be used to help inform decision-making at multiple
levels and when to do an update on a prior HTA review? Also, what
is the acceptability of RWE by decision makers and payers (i.e.
accommodating evidence needs)? Most often RWE is used for
multiple purposes, including health technology development such
as providing evidence on the natural history and epidemiology of
a disease, to provide information on treatment pathways and
comparator interventions in clinical practice, regulatory
approval decisions, monitoring pharmacovigilance, and
increasingly for HTA, especially cost-effectiveness analysis24,25
and reassessments26, payer coverage decisions, and outcome-based
contracting.27,28 As stated in the 2017 white paper by the Green
Park Initiative, RWE can be used for answering different
questions, including total costs of care and patient-centered
outcomes research. RWE can also provide useful information to
complement evidence from RCTs or other existing research
findings.29 There is, however, a lack of agreement between
different involved parties regarding what data are needed, at
which point in time, and for what purpose.30 For example, in the
US the lack of consensus 24 Gillespie, J., Erdol, S., Strachan,
L. et al. OP07 Real World Evidence: How can it improve Health
Technology Assessment? Oral Presentations. HTAi 2018 Annual
Meeting. Abstracts Book, p. 18. Available at:
https://htai.org/wp-content/uploads/2018/08/AM18_Abstract-
Book.pdf (Accessed 1 November, 2018) 25 Tarricone, R., Boscolo,
P., Armeni, P. What type of clinical evidence is needed to assess
medical devices? European Respiratory Review, 2016; 25(141): 259-
265 26 Jaksa, A., Pontynen, A., Wang, X. et al. OP05 Use of Real
World Evidence in HTA Decision-Making from 7 Agencies. Oral
Presentations. HTAi 2018 Annual Meeting. Abstracts Book, p. 16-
17. Available at:
https://htai.org/wp-content/uploads/2018/08/AM18_Abstract-
Book.pdf (Accessed 1 November, 2018) 27 Hampson, G., Towse, A.,
Dreitlein, T. et al. Real World Evidence for Coverage Decisions:
Opportunities and Challenges A Report from the 2017 ICER
Membership Policy Summit. ICER, OHE: March 2018. Available at:
https://icer-review.org/wp-content/uploads/2018/03/ICER-
RealWorld-Evidence-White-Paper-03282018.pdf (Accessed 1 November,
2018) 28 Murphy, G., de Léséleuc, L., Kaunelis, D., et al. Use of
real-world evidence in single-drug assessments. Ottawa: CADTH;
2018. (Environmental scan; no. 74). Available at:
https://www.cadth.ca/use-real-world-evidence-single-drug-
technology-assessment-processes-health-technologyassessment-and
(Accessed 7 October, 2018) 29 Green Park Collaborative. RWE
Decoder Framework. A Practical Tool for Assessing Relevance and
Rigor of Real World Evidence. February, 2017. Available at:
http://www.cmtpnet.org/docs/resources/RWE_Decoder_Framework.pdf
(Accessed 9 October, 2018) 30 Annemans, L. The use of real world
data throughout an innovative medicine’s lifecycle. Available at:
https://www.riziv.fgov.be/nl/themas/ kost-terugbetaling/door-
ziekenfonds/geneesmiddel-gezondheidsproduct/terugbetalen/
innovatieve-geneesmiddelen/Paginas/innovativemedicins-
lifecycle.aspx (Accessed 1 November, 2018) 12 Background among
stakeholders about appropriate approaches and methods for using
RWD, and RWE to support evidence from trials has slowed adoption
for regulatory submissions.31 Deloitte (2018) reported that
health care stakeholder receptivity to RWE generated by industry
and lack of an internal understanding of where such analyses can
be applied are key challenges for using RWE. From a survey among
20 leading biopharmaceutical companies they found that 75% of the
respondents felt a lack of receptivity by payers and providers;
70% reported internal stakeholders’ lack of understanding, and
60% lack access to necessary external data.32 Furthermore, a lack
of trust and collaboration between key stakeholders has resulted
in industry being uncertain as to what data is required in the
context of HTA.33 Effective collaboration between industry,
payers and other relevant key stakeholders in the development and
use of RWE for coverage and formulary decisions was discussed at
the Institute for Clinical and Economic Review (ICER) Policy
Summit in December 2017. Based on this discussion, a framework
was developed to guide the optimal development and use of RWE for
coverage decisions. The framework consist of several steps to be
taken when developing and using RWE, the necessary evidence
standards for each step regarding the question that it is
intended to support, and the context in which the decision needs
to be made (see Figure 1). Figure 1. Conceptual framework to
guide optimal development and use of RWE for coverage and
formulary decisions *PICOTS: Patients, Intervention, Comparators,
Outcomes, Time horizon, Setting Source: Based on Pearson,
Dreitlein, Towse, et al., 2018; p. 1534 31 Clinical Trials
Transformation Initiative. Project Real World Evidence. Overview.
Available at:
https://www.ctti-clinicaltrials.org/projects/realworld-evidence
(Accessed 1 November, 2018) 32 Davis, B., Morgan, J., Shah, S.
The future of real-world evidence. Biopharma companies focus on
end-to-end, AI-driven, internally developed solutions, June 28,
2018. Available at:
https://www2.deloitte.com/insights/us/en/industry/life-sciences/2
018-real-world-evidencebenchmarking.html (Accessed 2 November,
2018) 33 HTAi Policy Forum Series Newsletter. Universal health
care in the Asia Region: overcoming the barriers using HTA and
Real World Data. December 2017. Available at:
https://htai.org/wp-content/uploads/2018/05/HTAi_Asia-Policy-
Forum_newsletter_20171208b.pdf (Accessed 14 October, 2018) 34
Pearson, S., Dreitlein, B., Towse, A. et al. Understanding the
context, selecting the standards: A framework to guide the
optimal use and real world evidence for coverage decisions. ICER,
OHE, March 2018. Available at:
https://icer-review.org/material/rwe-white-paper-companion/
(Accessed 1 November, 2018) 13 Background When to use RWE across
the lifecycle As stated above, there is no clear consensus among
stakeholders about when to use RWE. RWE tends to be discussed
within themes that are focused on product development, early
adoption and innovation, especially targeting medicines (e.g. IMI
GetReal project; see the Annex for more initiatives, including
those targeting medical devices).