Health Policy Analysis

- Contents lists available at sciencedirect.com

Journal homepage: www.elsevier.com/locate/jval

A Conceptual Framework for Life-Cycle Health Technology Assessment

Erin Kirwin, MA, Jeff Round, PhD, Ken Bond, MA, Christopher McCabe, PhD

A B S T R A C T

Objectives: Health technology assessment (HTA) uses evidence appraisal and synthesis with economic evaluation to inform
adoption decisions. Standard HTA processes sometimes struggle to (1) support decisions that involve significant uncertainty
and (2) encourage continued generation of and adaptation to new evidence. We propose the life-cycle (LC)-HTA framework,
addressing these challenges by providing additional tools to decision makers and improving outcomes for all stakeholders.
Methods: Under the LC-HTA framework, HTA processes align to LC management. LC-HTA introduces changes in HTA methods
to minimize analytic time while optimizing decision certainty. Where decision uncertainty exists, we recommend risk-based
pricing and research-oriented managed access (ROMA). Contractual procurement agreements define the terms of
reassessment and provide additional decision options to HTA agencies. LC-HTA extends value-of-information methods to
inform ROMA agreements, leveraging routine, administrative data, and registries to reduce uncertainty.
Results: LC-HTA enables the adoption of high-value high-risk innovations while improving health system sustainability
through risk-sharing and reducing uncertainty. Responsiveness to evolving evidence is improved through contractually
embedded decision rules to simplify reassessment. ROMA allows conditional adoption to obtain additional information,
with confidence that the net value of that adoption decision is positive.
Conclusions: The LC-HTA framework improves outcomes for patients, sponsors, and payers. Patients benefit through earlier
access to new technologies. Payers increase the value of the technologies they invest in and gain mechanisms to review
investments. Sponsors benefit through greater certainty in outcomes related to their investment, swifter access to
markets, and greater opportunities to demonstrate value.

Keywords: economic evaluation, health technology assessment, life-cycle, research oriented market access, value of
information.

VALUE HEALTH. 2022; 25(7):1116–1123

Introduction Evidence assessment usually involves the review of an evi-

Public programs face increasing budgetary pressures and
require strong evidence for health technology funding decisions.
These decisions are supported by health technology assessment
(HTA) agencies, making recommendations based on reviews of
clinical evidence and results from cost-effectiveness analysis. HTA
agency recommendations have implications for patients’ access to
treatments, sponsors’ return on investment, public formularies,
and payers’ budgets.
Standard HTA Processes
HTA processes typically begin after or concurrently with reg-
ulatory approval and vary by jurisdiction in terms of governance,
remit (advisory vs decision), links to reimbursement, and au-
thority to monitor and adjust postdecision. Despite interjurisdic-
tional differences, all HTA processes include 2 phases: evidence
assessment and appraisal.1,2
dence dossier submitted by the technology sponsor, which in-
cludes published and unpublished information from clinical trials
and economic evaluations. The sponsor is often the company that
has developed and owns the technology, but patient and clinician
groups may also make submissions. The HTA agency may then
commission or conduct a systematic review of the clinical evi-
dence and a cost-effectiveness analysis, with or without inclusion
of materials beyond the sponsor evidence dossier.
In the appraisal phase, the results of the evidence assessment
are reviewed and discussed by a panel or body tasked with
making recommendations for payer adoption decisions. Issues
such as equity, acceptability, and input from key stakeholders are
sometimes considered. The recommendations typically take the
form of (1) recommend, (2) recommend with conditions/re-
strictions, or (3) do not recommend.3 Conditional recommenda-
tions may specify price reductions to achieve an acceptable value
or budget impact target and clinical conditions that restrict access
to specific patient subgroups.

1098-3015/Copyright ª 2022, International Society for Pharmacoeconomics and Outcomes Research, Inc. Published by Elsevier Inc. This is an open access article
under the CC BY license (http://creativecommons.org/licenses/by/4.0/).

Challenges With Standard HTA Processes
Standard HTA and methods leave several well-documented
challenges and limitations unaddressed, which include the
following4-10:
Health system sustainability: Deadoption and
threshold values
The ever-increasing nature of health technology development
places significant pressures on health system payer budgets.
When decision rules require that technologies demonstrate value
at a cost-effectiveness threshold (CET), the implication is that in
the absence of deadoption elsewhere budgets will continually
expand.11-15 Typical HTA reviews do not include recommendations
on deadoption, meaning that technologies that should be dead-
opted to ensure sustainability may continue to be reimbursed or,
worse, more valuable products are deadopted to pay for the new
technology, harming total population health. Inappropriate
threshold values could cause new technologies to displace more
health than they create.16-21
Evolving evidence
Evidence for new and existing technologies is continually
emerging. Initial decisions may be invalidated by changes in the
evidence base for the technology (eg, new trials) or by changes in
the clinical pathway (eg, new complementary or substitute tech-
nologies), which can affect a technology’s value. There are few
standard processes to reevaluate technologies over the on-market
life-cycle (LC) or to support value-based price renegotiation in
response to these changes.10,22-33 Where conditional funding
recommendations are made, there is the need to reevaluate
technologies for which additional evidence was required at the
recommendation stage, such as managed access agreements.7,34-36
Uncertainty
Evidentiary uncertainty arises from several factors, including
the characteristics of the technology, the care pathway and patient
group, and future events (patent expiry, uptake, and competitor
entry). In health economic modeling, uncertainty is often classi-
fied as methodological, structural, and parameter.37 The cumula-
tive effect of all these sources of uncertainty is to create a
significant risk of incorrect decisions: the adoption of technologies
that will not deliver the promised value and the rejection of
technologies that would.13,38-45 As a direct consequence, payers
accept a substantial risk, which in turn affects the health of the
populations served. This risk is particularly high when regulatory
approval is given to technologies that do not meet conventional
evidentiary standards for effectiveness (such as the 2021 Food and
Drug Administration approval of aducanumab).46 Nevertheless,
there are few mechanisms within standard HTA to mitigate these
risks.
Aim
We aim to address these challenges through our framework for
LC-HTA, designed to improve outcomes for patients, payers, and
sponsors. We describe a sequence of processes and decision rules
that are suitable for implementation within diverse contexts,
rather than aligning the framework to specific jurisdictional
procedures.
The LC-HTA Framework
The LC-HTA framework builds upon standard HTA concepts
and methods and proposes adaptations that will lead to better
HEALTH POLICY ANALYSIS 1117
outcomes for all stakeholders. Our framework is built around on-
market evidence generation and risk-based pricing strate-
gies.25,34,47 There is no current HTA agency guidance that includes
value-of-information (VoI) methods to inform adoption de-
cisions.37,48,49 Existing frameworks with similar aims do not take a
LC perspective50 or are retrospective in nature and do not include
divestment decision rules.51,52 Many of the individual components
of the LC-HTA framework are well established and understood, but
the sequential implementation of these methods and the
embedded decision rules are novel.
A key component of LC-HTA is research-oriented managed
access (ROMA). ROMA provides contractual agreements for tech-
nologies that meet regulatory approval requirements but for
which there is substantial uncertainty regarding effectiveness
and/or cost-effectiveness. These agreements are research oriented
in that the contract defines the exact research protocol that will be
implemented, the duration of the research, and the specific data
sources that will be used, leveraging sources of administrative, or
routinely collected data to generate real-world evidence (RWE).
Evidence generated through ROMA will be used to reassess the
technology and inform adoption decisions, while allowing
managed market access to sponsors and access to promising
technologies to patients.
LC-HTA requires reframing of price targets, given that prices
can be used to compensate for decision uncertainty. In standard
HTA, a theoretical maximum acceptable price for payers is iden-
tified when the incremental cost-effectiveness ratio is equal to the
CET, which can also be expressed as the incremental net monetary
benefit (NMB) being equal to zero. We refer to this price as the
value-based price.
An alternative to the abovementioned model is risk-based
pricing, which relies on a modified VoI output, the independent
expected value of perfect information (iEVPI).47 The iEVPI is
calculated in the same way as standard expected value of perfect
information (EVPI),53,54 but compares only the NMB distribution
of a technology at a given price and an alternative with a constant
value of the expected NMB of the current treatment at value-
based pricing. “Payer risk tolerance” is the maximum per-patient
risk of making wrong decisions that payers are willing to accept
in monetary terms. The risk-based price is the price at which, for
any given value threshold, the iEVPI is equal to payer risk toler-
ance. Finally, the critical price is defined as the lesser of the value-
based price and the risk-based price.
We present the framework LC-HTA in the chronological
sequence of a technology’s LC (Fig. 1). After initial HTA, the process
becomes a cycle where technologies transition between the
reassessment, adoption, and no adoption stages.
The Preassessment Stage
Preassessment meetings and early scientific advice programs
offer opportunities for sponsors to ask questions of regulators and/
or HTA body to better align evidence with requirements. Some
HTA agencies collaborate internationally to provide early feed-
back.55,56 Such programs improve the efficiency of LC-HTA and
should be developed and retained.
The Safety and Efficacy Assessment Stage
In most countries, a regulatory body is responsible for de-
cisions regarding market authorization, with a focus on safety,
efficacy, and manufacturing quality.57,58 The regulatory approval
stage typically involves the review of an evidence dossier sub-
mitted by the technology sponsor. This normally occurs before
HTA review, although there are examples of overlapping pro-
cesses.59 Under LC-HTA, these conventions are retained.
1118 VALUE IN HEALTH
Figure 1. The LC-HTA framework. Dark shaded square boxes represent LC-HTA stages as introduced in the section The LC-HTA
Framework; lighter shaded rounded boxes represent outcomes. White rounded boxes represent statuses within stages and outcomes.
LC-HTA indicates Life-cycle health technology assessment; HTA, health technology assessment; ROMA, research-oriented managed access.
Additionally, regulatory and HTA agencies should share informa-
tion and communicate expected timelines and outcomes of pro-
cesses, to improve the efficiency of review processes. After
regulatory review, agencies make recommendations that fall into
one of 3 general categories:
1. Grant market access as per requested clinical indications
2. Grant market access with conditions/ restrictions
3. Deny market access
The HTA Stage
Technologies receiving full or conditional regulatory approval
proceed to the HTA stage.
Formulating the decision problem
On receipt of acceptable submissions, the HTA agency must
define the decision problem: the indications and populations for
which the technology could be recommended and comparator
technologies that must be included in the review. Ill-defined de-
cision problems will lead to inconsistencies in the HTA process
and could inhibit future reassessments.
HTA agencies must provide guidance on appropriate CET
values. Limited estimates of the supply-side threshold can lead to
the use of inappropriate values, reducing population health ben-
efits. Similarly, varying threshold values by target population can
reduce population and equity-weighted population health.16
Another concern is the overall budget impact for technologies
that are cost-effective at the margins. One solution is to restrict
prices below the point of nonmarginal budget impacts; alternative
approaches could consider varying threshold values as budget
impact increases.21
Evidence review and synthesis
The HTA agency should complete a systematic evidence review
and synthesis, supplementing the submitted evidence dossier.60
JULY 2022
The evidence review should include an analysis of existing care
pathways, to better understand the context in which the technology
could be implemented and to evaluate the generalizability of the
existing evidence base to the decision context. The evidence review
should involve consultation with patient and clinician groups to
document and validate important disease outcomes and clinical
factors that may not be well represented within the existing evi-
dence base.61 Finally, potential sources for RWE related to the
technology and care pathway should be identified to assess the
feasibility of on-market evidence development through ROMA.
De novo model development and VoI analysis
In standard HTA, a health economic model is submitted as a
component of the sponsor evidence dossier, and this model is
reviewed by the HTA agency. In LC-HTA, a de novo model is
developed instead of or in addition to the sponsor model. This de
novo model is used to inform the initial assessment and appraisal,
which data should be collected under ROMA, the design of the on-
market evidence generation, and the reappraisal.
The development of de novo models is recommended to
resolve structural uncertainty (uncertainty because of the rele-
vance of care pathways in varying decision contexts) and to ensure
capability for advanced analytic steps later in the process. Often
sponsor models are developed for global markets and may not
capture the local clinical pathway, with distorting impacts on
technology diffusion, and the design and value of on-market
research. De novo models can better align to the care pathway
identified in the evidence review.
The characterization of parameter uncertainty will be
improved by identifying all potential sources in the evidence re-
view. Methods to improve the quality of initial parameter inputs,
such as meta-analysis, Bayesian updating, expert elicitation, and
bias adjustment exercises, should be implemented as appro-
priate.62-70 Models should be developed to facilitate direct
parameter updating to reflect on-market evidence generation and
from other studies.

The application of VoI methods is central to LC-HTA, by sup-
porting the development of efficient study designs. VoI methods
can be computationally intensive, particularly advanced outputs
that are required by the later stages of LC-HTA. Appropriate se-
lection of model structure and software platforms can greatly
improve efficiency.71,72
Once the LC-HTA model has been developed, the following
outputs are required, using the HTA agency endorsed value
threshold:
1. Expected NMB, with a probabilistic sensitivity analysis to
quantify uncertainty in NMB values45
2. EVPI53,54
3. Value- and risk-based prices
4. The expected value of perfect parameter information (EVPPI)
for key model parameters
These outputs will provide a detailed understanding of the
sources of decision uncertainty, along with pricing inputs required
for the design of potential ROMA protocols.73-77
HTA outcomes under LC-HTA
Four recommendations are available to HTA agencies based on
the outputs of the LC-HTA analyses:
1. Adoption without conditions: the sponsor submitted price is
less than or equal to the critical price.
2. When evidence indicates that adoption of the technology
should only be made for specific indications or patient groups
and/or when the sponsor submitted price is greater than the
critical price and the value-based price is less than the risk-
based price, HTA agencies should recommend adoption with
conditions/restrictions.
3. When the sponsor submitted price is greater than the critical
price and the risk-based price is less than the value-based
price, HTA agencies should recommend ROMA. Under ROMA,
research will be undertaken to reduce uncertainty.
4. When the evidence indicates that the technology will not have
a positive NMB at any price that is acceptable to the sponsor,
HTA agencies should not recommend adoption.
This decision typology builds upon the work of Claxton et al35
adding the role of critical and risk-based pricing. Varying levels of
uncertainty in subsets of the full licensed indication may justify a
mixture of recommendations by clinical indication.
The Adoption Stage
Technology adoption takes 2 forms: regular or through ROMA.
For any technology, these are not necessarily mutually exclusive
and may differ by clinical indication.
Regular adoption
Following HTA agency recommendations, payers make positive
adoption decisions outside of ROMA when the final negotiated
price is lower than the critical price, and any conditions/re-
strictions are agreed upon. These adoption decisions are made
through a contractual agreement describing the inputs, process,
and the conditions that would trigger reassessment including
decision rules. The reassessment conditions in the procurement
contracts should consider26:
Regulatory changes. Includes entry of competitors, patent
expiry, entry of generic forms of the technology on the market, or
changes in regulatory status (ie, regulatory conditions).
HEALTH POLICY ANALYSIS 1119
Health system activities. Includes significant off-label
usage or overutilization, unanticipated budget impacts, or
requests from the technology sponsor to revisit reimbursement
and changes to the care pathways, including alternative
treatment options, which are likely to affect the technology’s
value.
New evidence. Includes new RCTs, real-world evaluations
and other sources of information which could affect the results
of the HTA analysis.
HTA method changes. Includes changes to HTA and/or
economic evaluation guidelines observed by the HTA
agency.37,48,49 Relevant changes could include recommended CETs,
discount rates, or similar.
If reassessment criteria are met, the new evidence should be
reviewed and synthesized. This can then be used to update the
health economic model to support reassessment of the technol-
ogy. The reimbursement status of the product can then be revised,
with the same decision options as the initial assessment.
Research-oriented market access adoption
Protocol development. For technologies that receive a
ROMA recommendation, candidate ROMA protocols should be
developed based on the results of the EVPPI analysis. Under each
protocol, different elements of RWE are collected and used to
update inputs in the health economic model, with the aim of
reducing uncertainty. In principle, ROMA may incorporate only-in-
research or only-with-research designs. It is expected that only-
with-research designs will be the norm, because of a lack of
justification for clinical equipoise and/or patient refusal to be
randomized once a license has been granted.35,41,78-80
Models are developed to simulate the evidence generation
process, characterized by sampling designs that could be imple-
mented through a technology diffusion model (TDM). The TDM
simulates the diffusion of the technology over time, which may
vary by sampling design.39,81,82 The RWE generation model is used
to simulate the additional evidence that would be generated as
the adoption and use of the technology increase. For each sam-
pling design, the outputs of the TDM and RWE generation model
are used together to update parameter information in the health
economic model.
The expected net benefit of sampling (ENBS) is estimated as
the difference between the expected value of sample information
(the difference in EVPI with and without the additional informa-
tion from sampling) and the expected total cost of sampling,
which accounts for the cost of evidence generation and the op-
portunity costs incurred, such as forgone health gains for patients
outside of research. The maximum ENBS is estimated for each
sampling design by selecting the protocol duration that maxi-
mizes the value between the expected value of sample informa-
tion and the expected total cost of sampling. Therefore, for each
sampling design, the optimal duration is identified and the ENBS
is calculated to compare among protocols.72,78
Protocol implementation. If at least one candidate ROMA
protocol has a positive ENBS, it should be considered for imple-
mentation. The protocol with the highest ENBS is the most valu-
able evidence development strategy.
ROMA protocols are implemented after contractual negotiation
with the sponsor. The contracted price must be less than the
value-based price. If the contracted price diverges from the risk-
based price, the ENBS for the candidate protocols should be
recalculated to ensure the highest value protocol is implemented.
1120 VALUE IN HEALTH
The terms of the contract will specify read-out periods for the
evidence development and will define stopping conditions for
ending the ROMA protocol term. At each read-out period, addi-
tional data and model outputs will be analyzed to determine if the
stopping conditions have been met. Once one of the stopping
conditions has been met, a final evaluation can be produced.
The contract is intended to remain in place for the duration of
the ROMA period. The duration of the ROMA period is determined
by updating the ENBS analysis after each read-out period and
continuing if it is . 0.
The outputs from the ROMA protocol will be evaluated and
synthesized with previous evidence. The parameters in the health
economic model will be updated using the synthesized evidence
base, and revised net benefit, VoI, and ENBS estimates will be
produced. The reimbursement status of the product can then be
revised, with the same decision options as the initial assessment.
The De-adoption Stage
Adoption is not recommended under 2 circumstances: when
the evidence does not support an expectation of a positive net
benefit at any price or when the price required for a positive ex-
pected net benefit is not acceptable to the sponsor.
The Reassessment Stage
For technologies under regular approval, reassessment is
initiated should any of the 4 contracted conditions (section Reg-
ular adoption) arise.
When evidence generated through ROMA adoption reports,
reassessment is initiated, and the new evidence is synthesized and
used to update the original HTA analyses and recommendations
(section HTA outcomes under LC-HTA).
For technologies outside of ROMA (regular adoption and de-
adoption), submissions for reevaluation can be made by any
sponsor (manufacturers, clinician, or patient groups). Resubmis-
sion to HTA agencies must indicate the basis for reevaluation, such
as new evidence or revisions to the submitted price.
Discussion
The LC-HTA framework introduced in this article has been
designed to address challenges and limitations of standard HTA
when assessing and appraising high-value high-risk technolo-
gies—sustainability, evolving evidence, and uncertainty.
Sustainability: Deadoption and Threshold Values
LC-HTA seeks to improve health system sustainability in
several ways. The estimation of the risk- and value-based prices
ensures fair prices for health system payers. Risk-based prices will
be lower than value-based prices when there is substantial deci-
sion uncertainty, compensating for additional risk. Payers are
protected against incorrect decisions, whereas sponsors gain
market access more rapidly and the opportunity to demonstrate
their technology’s value, which could inform adoption decisions in
other markets and/or support price increases. By convention, HTA
agencies treat regulatory approval as evidence of safety and effi-
cacy. Within LC-HTA, deadoption decisions based on safety and
effectiveness alone are possible, realized either through a failure
to demonstrate effectiveness or through the (non)acceptability of
the critical price to sponsors.
LC-HTA adoption rules protect health system sustainability by
ensuring adoption is conditional upon technologies not displacing
more value than they produce. This is further supported through
the use of appropriate threshold values.
JULY 2022
Evolving Evidence
Through the decision rules embedded in LC-HTA, the frame-
work is responsive to evolving evidence that can affect the deci-
sion criteria for the adoption and/or deadoption of health
technologies.
LC-HTA also contributes to the development of evidence
through ROMA. Although it generates evidence for a specific
jurisdiction, it adds to the global evidence base and hence im-
proves decisions in other jurisdictions.83
Uncertainty
By expanding the scope of evidence that is included in the
evidence review and synthesis, including RWE and technology
diffusion, the value of technologies is better characterized,
reducing payer decision uncertainty.
The development of a de novo model further reduces uncer-
tainty by ensuring alignment to jurisdiction specific care path-
ways. In addition, by designing models to enable efficient
updating of evolving evidence and VoI outputs, LC-HTA allows
decision makers to respond promptly and predictably to new
evidence, including the launch of similar technologies, all of which
reduces uncertainty for patients, sponsors, and payers.
Using VoI outputs within decision making and the ROMA
process reduces uncertainty efficiently. EVPPI identifies the pa-
rameters that contribute the most to decision uncertainty. The
ROMA component of LC-HTA ensures that RWE prioritizes these
parameters. The use of ENBS to select the best value ROMA pro-
tocol optimizes the value of investments in RWE generation.
An important innovation under LC-HTA is the direct linkage of
uncertainty to risk. When there is a high degree of uncertainty in
the evidence base, there is an increased risk to decision makers of
making incorrect decisions. This is particularly important when
regulatory agencies approve technologies below conventional
evidentiary standards. The LC-HTA framework includes decision
rules that are responsive to risks associated with uncertain
effectiveness evidence at regulatory approval and, through ROMA,
provides mechanisms to share this risk between payers and
sponsors, while reducing delays in access to technologies for pa-
tients compared with standard HTA.41,79
Considerations for Implementation of LC-HTA
Contractual agreements for implementing ROMA could repre-
sent a significant change in the relationship among patients,
payers, and providers. In particular, there are likely legal chal-
lenges related to withdrawing access to treatments that are
deadopted after reassessment. The solutions to such problems will
be jurisdiction specific. Related to this is the need to determine
how technologies approved before the implementation of the LC-
HTA framework should be evaluated, for example, if reassessment
criteria should be applied retroactively or only for products eval-
uated after adoption of an LC-HTA based approach.
The LC-HTA framework includes many opportunities for effi-
ciency gains, including information sharing among sponsors,
regulators, and HTA agencies and the development of de novo
models that can be recycled and repurposed throughout the LC of
a health technology or adapted for additional technologies with
similar care pathways and/or outcomes. Nevertheless, the LC-HTA
framework requires additional evidence review and synthesis, and
resource requirements (particularly trained analysts) are some-
what larger than those required for conventional HTA. The deci-
sion rules and contractual clauses could lead to a larger volume of
reevaluations being required, exacerbating resource challenges.
Although these resources may seem to have high costs compared

with standard HTA, the methods embedded in LC-HTA allow the
quantification of the return on the investment in LC-HTA given
that they can be incorporated as part of the costs of research when
identifying the ENBS. Although the up-front costs related to LC-
HTA de novo model development are substantial, they are small
relative to the expected cost of the investment decision. Such
models are expected to pay substantial dividends over the long-
term, and reassessments will likely require lower resources than
initial assessments.
Implementation of the LC-HTA framework requires efficient
collaboration among HTA agencies, patients, and payers, in
particular to facilitate timely model development. Early sponsor
and patient engagement through parallel regulatory and HTA re-
view can mitigate timeline concerns.
There are opportunities for collaboration among HTA agencies
internationally. Often, technology sponsors submit applications
for review to multiple agencies, closely linked in time and content.
Improved communication and collaboration between agencies
should enable components of evidence review, synthesis, and
possibly health economic models to be shared, easing the burden
on individual HTA agencies. Open science partnerships could
further enhance the efficiency of evidence production.84
Any new process will undoubtedly have unintended impacts.
Deadoption decisions will be politically sensitive and HTA
agencies may be required to include parameters that are incon-
sequential to ENBS estimation, but important to patients and
sponsors in ROMA. Payers could shift evidentiary standards or
threshold values away from the optimum as a delay mechanism,
with negative health impacts. Regulators could justify lower
evidentiary standards through the stronger mechanisms present
in the LC-HTA framework, downloading sensitive approval de-
cisions to HTA agencies. Implementation plans should include
strategies to mitigate undesirable outcomes should they arise.
Successful implementation of the LC-HTA framework will
require excellent science communication and public engagement
to address the challenge of acceptability related to the complex
VoI methods used.
Conclusions
We present an LC-HTA framework designed to address 3
challenges faced by standard HTA: health system sustainability,
evolving evidence, and uncertainty. The use of this framework
could improve outcomes for patients, sponsors, and payers. Pa-
tients benefit by gaining earlier access to technologies that provide
a high level of value. Payers benefit by increasing the value of
adopted technologies and creating mechanisms for reassessment
in response to changes in regulatory conditions, evidence, health
system activities, or methodological guidance. Sponsors benefit
through greater certainty regarding both process and outcomes
related to their adoption and, through ROMA, with swifter access
to markets and greater opportunities to demonstrate value.
Article and Author Information
Accepted for Publication: November 23, 2021
Published Online: February 4, 2022
doi: https://doi.org/10.1016/j.jval.2021.11.1373
Author Affiliations: Institute of Health Economics, Edmonton, AB, Canada
(Kirwin, Round, Bond, McCabe); Health Organisation, Policy, and Eco-
nomics, School of Health Sciences, University of Manchester, Manchester,
England, UK (Kirwin); Faculty of Medicine and Dentistry, University of
Alberta, Edmonton, AB, Canada (Round, McCabe)
HEALTH POLICY ANALYSIS 1121
Correspondence: Erin Kirwin, MA, Institute of Health Economics, #1200,
10405 Jasper Ave, Edmonton, AB T5J 3N4, Canada. Email: ekirwin@ihe.ca
Author Contributions: Concept and design: Kirwin, Round, Bond, McCabe
Drafting of the manuscript: Kirwin, Round, Bond, McCabe
Critical revision of the paper for important intellectual content: Kirwin, Round,
Bond, McCabe
Supervision: Round, McCabe
Conflict of Interest Disclosures: All authors reported receiving grants
from the Government of Alberta during the conduct of the study. Dr
McCabe reported receiving nonfinancial support from Health Canada, the
Government of British Columbia, the Government of Ontario, and the
Canadian Agency for Drugs and Technologies in Health and reported
receiving grants from Merck Canada Inc, Novartis Pharmaceuticals Canada
Inc, AstraZeneca Canada Inc, Takeda Canada Inc, Boehringer Ingelheim
(Canada) Ltd, GlaxoSmithKline Inc, and Hoffman-La Roche Ltd outside of
the submitted work. No other disclosures were reported.
Funding/Support: Authors Kirwin, Round, Bond, and McCabe received
financial contributions from the Government of Alberta, grants #005527,
#005841, #008491, and #013211.
Role of the Funder/Sponsor: The funder had no role in the design and
conduct of the study; collection, management, analysis, and interpretation
of the data; preparation, review, or approval of the manuscript; and de-
cision to submit the manuscript for publication.
REFERENCES
1. Teutsch SM, Berger ML. Evidence synthesis and evidence-based decision
making: related but distinct processes. Med Decis Making. 2005;25(5):487–
489.
2. Oortwijn W, Jansen M, Baltussen R. Use of evidence-informed deliberative
processes by health technology assessment agencies around the globe. Int J
Health Policy Manag. 2020;9(1):27–33.
3. CIRS RD briefing 69: review of HTA outcomes and timelines 2014-2017.
Centre for Innovation in Regulatory Science. https://www.cirsci.org/
publications/cirs-rd-briefing-69-review-of-hta-outcomes-and-timelines-2
014-2017/. Accessed June 16, 2021.
4. O’Rourke B, Werkö SS, Merlin T, Huang LY, Schuller T. The ‘top 10’ challenges
for health technology assessment: INAHTA viewpoint. Int J Technol Assess
Health Care. 2020;36(1):1–4.
5. Daubner-Bendes R, Kovács S, Niewada M, et al. Quo vadis HTA for medical
devices in central and eastern Europe? Recommendations to address
methodological challenges. Front Public Health. 2020;8:612410.
6. Wang T, Lipska I, McAuslane N, Liberti L, Hövels A, Leufkens H. Benchmarking
health technology assessment agencies-methodological challenges and rec-
ommendations [published online September 8, 2020]. Int J Technol Assess
Health Care. https://doi.org/10.1017/S0266462320000598.
7. Nicod E, Annemans L, Bucsics A, Lee A, Upadhyaya S, Facey K. HTA pro-
gramme response to the challenges of dealing with orphan medicinal
products: process evaluation in selected European countries. Health Policy.
2019;123(2):140–151.
8. Stafinski T, Deber R, Rhainds M, et al. The introduction of new non-drug
health technologies into Canadian healthcare institutions: opportunities
and challenges. Healthc Policy. 2019;15(1):95–106.
9. Bryan S, Williams I, McIver S. Seeing the NICE side of cost-effectiveness
analysis: a qualitative investigation of the use of CEA in NICE technology
appraisals. Health Econ. 2007;16(2):179–193.
10. Gutiérrez-Ibarluzea I, Chiumente M, Dauben HP. The life cycle of health
technologies. Challenges and ways forward. Front Pharmacol. 2017;8:14.
11. Birch S, Gafni A. Cost effectiveness/utility analyses. Do current decision rules
lead us to where we want to be? J Health Econ. 1992;11(3):279–296.
12. Birch S, Gafni A. Changing the problem to fit the solution: Johannesson and
Weinstein’s (mis) application of economics to real world problems. J Health
Econ. 1993;12(4):469–476.
13. Sendi P, Al MJ, Gafni A, Birch S. Optimizing a portfolio of health care programs
in the presence of uncertainty and constrained resources. Soc Sci Med.
2003;57(11):2207–2215.
14. Johannesson M, Weinstein MC. On the decision rules of cost-effectiveness
analysis. J Health Econ. 1993;12(4):459–467.
15. Weinstein MC, Stason WB. Foundations of cost-effectiveness analysis for
health and medical practices. N Engl J Med. 1977;296(13):716–721.
16. Paulden M, McCabe C. Modifying NICE’s approach to equity weighting.
Pharmacoeconomics. 2021;39(2):147–160.
17. Vanness DJ, Lomas J, Ahn H. A health opportunity cost threshold for cost-
effectiveness analysis in the United States. Ann Intern Med. 2021;174(1):
25–32.
18. Ochalek J, Lomas J, Claxton K. Estimating health opportunity costs in low-
income and middle-income countries: a novel approach and evidence from
1122 VALUE IN HEALTH
cross-country data [published correction appears in BMJ Glob Health.
20194(3):e000964corr1]. BMJ Glob Health. 2018;3(6):e000964.
19. Claxton K, Martin S, Soares M, et al. Methods for the estimation of the Na-
tional Institute for Health and Care Excellence cost-effectiveness threshold.
Health Technol Assess. 2015;19(14):1–vi.
20. Lomas J, Martin S, Claxton K. Estimating the marginal productivity of the
English National Health Service from 2003 to 2012. Value Health.
2019;22(9):995–1002.
21. Lomas JRS. Incorporating affordability concerns within cost-effectiveness
analysis for health technology assessment. Value Health. 2019;22(8):
898–905.
22. Esmail R, Hanson H, Holroyd-Leduc J, Niven DJ, Clement F. Knowledge
translation and health technology reassessment: identifying synergy. BMC
Health Serv Res. 2018;18(1):674.
23. Makady A, ten Ham R, de Boer A, et al. Policies for use of real-world data in
health technology assessment (HTA): a comparative study of six HTA
agencies. Value Health. 2017;20(4):520–532.
24. Maloney MA, Schwartz L, O’Reilly D, Levine M. Health Technology Agency
insights: informing modification of a qualitative benefit risk framework for
Health Technology Reassessment of prescription medications. Int J Technol
Assess Health Care. 2019;35(5):384–392.
25. Moseley J, Vamvakas S, Berntgen M, et al. Regulatory and health technology
assessment advice on postlicensing and postlaunch evidence generation is a
foundation for lifecycle data collection for medicines. Br J Clin Pharmacol.
2020;86(6):1034–1051.
26. CADTH’s proposed reassessment framework. Canadian Agency for Drugs and
Technologies in Health. https://www.cadth.ca/sites/default/files/cdr/templates/
consultations/CADTH_Consultation_Reassessment_Framework.pdf. Accessed
June 16, 2021.
27. Pereira VC, Barreto JOM, Neves FADR. Health technology reassessment in the
Brazilian public health system: analysis of the current status. PLoS One.
2019;14(7):e0220131.
28. Sevick K, Soril LJ, MacKean G, Noseworthy TW, Clement FM. Unpacking early
experiences with health technology reassessment in a complex healthcare
system. Int J Health Policy Manag. 2020;13(2):156–162.
29. Soril LJ, MacKean G, Noseworthy TW, Leggett LE, Clement FM. Achieving
optimal technology use: a proposed model for health technology reassess-
ment. SAGE Open Med. 2017;5:2050312117704861.
30. Soril LJJ, Niven DJ, Esmail R, Noseworthy TW, Clement FM. Untangling,
unbundling, and moving forward: framing health technology reassessment
in the changing conceptual landscape. Int J Technol Assess Health Care.
2018;34(2):212–217.
31. Soril LJJ, Seixas BV, Mitton C, Bryan S, Clement FM. Moving low value care
lists into action: prioritizing candidate health technologies for reassessment
using administrative data. BMC Health Serv Res. 2018;18(1):640.
32. Succi-Lopez M, Lee SY, Alexander JA. The effects of relative resource config-
uration, organizational legitimacy, and integration on divestiture decisions
among health systems. Health Care Manag Rev. 2003;28(4):307–318.
33. Tadrous M, Ahuja T, Ghosh B, Kropp R. Developing a Canadian real-world
evidence action plan across the drug life cycle. Healthc Policy.
2020;15(4):41–47.
34. McCabe CJ, Stafinski T, Edlin R, Menon D, Banff AEDS. Access with evidence
development schemes: a framework for description and evaluation. Phar-
macoeconomics. 2010;28(2):143–152.
35. Claxton K, Palmer S, Longworth L, et al. A comprehensive algorithm for
approval of health technologies with, without, or only in research: the key
principles for informing coverage decisions. Value Health. 2016;19(6):
885–891.
36. Grimm SE, Strong M, Brennan A, Wailoo AJ. The HTA risk analysis chart:
visualising the need for and potential value of managed entry agreements in
health technology assessment. Pharmacoeconomics. 2017;35(12):1287–1296.
37. Guidelines for the Economic Evaluation of Health Technologies: Canada. 4th
edition. Canadian Agency for Drugs and Technologies in Health. https://www.
cadth.ca/sites/default/files/pdf/guidelines_for_the_economic_evaluation_of_
health_technologies_canada_4th_ed.pdf. Accessed June 16, 2021.
38. Briggs AH, Weinstein MC, Fenwick EA, et al. Model parameter estimation and
uncertainty analysis: a report of the ISPOR-SMDM Modeling Good Research
Practices Task Force Working Group-6. Med Decis Making. 2012;32(5):
722–732.
39. Grimm SE, Dixon S, Stevens JW. Assessing the expected value of research
studies in reducing uncertainty and improving implementation dynamics.
Med Decis Making. 2017;37(5):523–533.
40. McCabe C, Edlin R, Hall P. Navigating time and uncertainty in health technology
appraisal: would a map help? Pharmacoeconomics. 2013;31(9):731–737.
41. Rothery C, Claxton K, Palmer S, Epstein D, Tarricone R, Sculpher M. Charac-
terising uncertainty in the assessment of medical devices and determining
future research needs. Health Econ. 2017;26(suppl 1):109–123.
42. Sculpher M, Claxton K. Establishing the cost-effectiveness of new pharma-
ceuticals under conditions of uncertainty–when is there sufficient evidence?
Value Health. 2005;8(4):433–446.
43. Sendi P, Al MJ. Revisiting the decision rule of cost-effectiveness analysis
under certainty and uncertainty. Soc Sci Med. 2003;57(6):969–974.
44. Stafinski T, McCabe CJ, Menon D. Funding the unfundable: mechanisms for
managing uncertainty in decisions on the introduction of new and innovative
technologies into healthcare systems. Pharmacoeconomics. 2010;28(2):113–142.
JULY 2022
45. Stinnett AA, Mullahy J. Net health benefits: a new framework for the analysis
of uncertainty in cost-effectiveness analysis. Med Decis Making. 1998;18(2
suppl):S68–S80.
46. FDA grants accelerated approval for Alzheimer’s drug. Food and Drug
Administration. https://www.fda.gov/news-events/press-announcements/
fda-grants-accelerated-approval-alzheimers-drug. Accessed June 16, 2021.
47. Kirwin E, Paulden M, McCabe C, Round J, Sutton M, Meacock R. The risk-
based price: incorporating uncertainty and risk attitudes in health technol-
ogy pricing. Preprint. Posted online November 18, 2021. SSRN. https://doi.
org/10.2139/ssrn.3956084.
48. Guide to the methods of technology appraisal 2013. National Institute for Health
and Care Excellence. https://www.nice.org.uk/process/pmg9/resources/guide-
to-the-methods-of-technology-appraisal-2013-pdf-2007975843781. Accessed
June 16, 2021.
49. Sanders GD, Neumann PJ, Basu A, et al. Recommendations for conduct,
methodological practices, and reporting of cost-effectiveness analyses: sec-
ond panel on cost-effectiveness in health and medicine [published correction
appears in JAMA. 2016;316(18):1924]. JAMA. 2016;316(10):1093–1103.
50. Claxton K, Eggington S, Ginnelly L, et al. NICE decision support unit methods
development. In: A Pilot Study of Value of Information Analysis to Support
Research Recommendations for NICE. London, United Kingdom: National
Institute for Health and Care Excellence (NICE); 2005.
51. Chan K, Nam S, Evans B, et al. Developing a framework to incorporate real-
world evidence in cancer drug funding decisions: the Canadian Real-world
Evidence for Value of Cancer Drugs (CanREValue) collaboration. BMJ Open.
2020;10(1):e032884.
52. CanREValue. Developing a framework for incorporating real-world evidence into
drug funding decisions CanREValue Collaboration Policy Working Groups Interim
Report 2019. https://cc-arcc.ca/wp-content/uploads/2019/12/CanREValue-Policy-
WG-Interim-Report-2019_final_17Dec2019.pdf. Accessed June 16, 2021.
53. Ades AE, Lu G, Claxton K. Expected value of sample information calculations
in medical decision modeling. Med Decis Making. 2004;24(2):207–227.
54. Willan AR, Pinto EM. The value of information and optimal clinical trial
design [published correction appears in Stat Med. 2006;25(4):720]. Stat Med.
2005;24(12):1791–1806.
55. NICE collaborates with the Canadian Agency for Drugs and Technology in
Health to offer parallel scientific advice. National Institute for Health and
Care Excellence (NICE). https://www.nice.org.uk/news/article/nice-
collaborates-with-the-canadian-agency-for-drugs-and-technology-in-health-
to-offer-parallel-scientific-advice. Accessed September 9, 2021.
56. Parallel scientific advice. Canadian Agency for Drugs and Technologies in Health.
https://www.cadth.ca/parallel-scientific-advice. Accessed September 9, 2021.
57. Drummond M, Griffin A, Tarricone R. Economic evaluation for devices and
drugs–same or different? Value Health. 2009;12(4):402–404.
58. Sorenson C, Tarricone R, Siebert M, Drummond M. Applying health eco-
nomics for policy decision making: do devices differ from drugs? Europace.
2011;13(suppl 2):ii54–ii58.
59. Notice to Industry: Health Canada and CADTH launch new initiative to
provide early parallel scientific advice. Government of Canada. https://www.
canada.ca/en/health-canada/corporate/transparency/regulatory-transparency-
and-openness/improving-review-drugs-devices/notice-early-panel-scientific-
advice.html. Accessed June 16, 2021.
60. Kristensen FB, Husereau D, Huic M, et al. Identifying the need for good
practices in health technology assessment: summary of the ISPOR HTA
council working group report on good practices in HTA. Value Health.
2019;22(1):13–20.
61. Facey KM, Ploug Hansen H, Single ANV. Patient Involvement in Health Tech-
nology Assessment. Singapore: Springer Singapore Pte. Limited; 2017.
62. Bojke L, Grigore B, Jankovic D, Peters J, Soares M, Stein K. Informing reim-
bursement decisions using cost-effectiveness modelling: a guide to the
process of generating elicited priors to capture model uncertainties. Phar-
macoeconomics. 2017;35(9):867–877.
63. Morris DE, Oakley JE, Crowe JA. A web-based tool for eliciting probability
distributions from experts. Environ Model Softw. 2014;52:1–4.
64. Schnell-Inderst P, Iglesias CP, Arvandi M, et al. A bias-adjusted evidence
synthesis of RCT and observational data: the case of total hip replacement.
Health Econ. 2017;26(suppl 1):46–69.
65. Watt AM, Willis CD, Hodgetts K, Elshaug AG, Hiller JE. Engaging clinicians in
evidence-based disinvestment: role and perceptions of evidence. Int J Technol
Assess Health Care. 2012;28(3):211–219.
66. O’Hagan T. SHELF: the Sheffield Elicitation Framework. http://www.
tonyohagan.co.uk/shelf/. Accessed June 16, 2021.
67. Thompson S, Ekelund U, Jebb S, et al. A proposed method of bias adjustment
for meta-analyses of published observational studies. Int J Epidemiol.
2011;40(3):765–777.
68. Turner RM, Spiegelhalter DJ, Smith GCS, Thompson SG. Bias modelling in
evidence synthesis. J R Stat Soc A. 2009;172(1):21–47.
69. Watson SI, Lilford RJ. Integrating multiple sources of evidence: a Bayesian
perspective. In: Raine R, Fitzpatrick R, Barratt H, Bevan G, Black N, Boaden R,
et al., eds. Challenges, Solutions and Future Directions in the Evaluation of
Service Innovations in Health Care and Public Health. Southampton, United
Kingdom: NIHR Journals Library; 2016.
70. Bun RS, Scheer J, Guillo S, Tubach F, Dechartres A. Meta-analyses frequently
pooled different study types together: a meta-epidemiological study. J Clin
Epidemiol. 2020;118:18–28.

71. Hollman C, Paulden M, Pechlivanoglou P, McCabe C. A comparison of four
software programs for implementing decision analytic cost-effectiveness
models. Pharmacoeconomics. 2017;35(8):817–830.
72. Rothery C, Strong M, Koffijberg HE, et al. Value of information analytical
methods: report 2 of the ISPOR Value of Information Analysis Emerging Good
Practices Task Force. Value Health. 2020;23(3):277–286.
73. Oakley JE, Brennan A, Tappenden P, Chilcott J. Simulation sample sizes for
Monte Carlo partial EVPI calculations. J Health Econ. 2010;29(3):468–477.
74. Sadatsafavi M, Bansback N, Zafari Z, Najafzadeh M, Marra C. Need for speed:
an efficient algorithm for calculation of single-parameter expected value of
partial perfect information. Value Health. 2013;16(2):438–448.
75. Strong M, Oakley JE. An efficient method for computing single-parameter
partial expected value of perfect information. Med Decis Making.
2013;33(6):755–766.
76. Strong M, Oakley JE, Brennan A. Estimating multiparameter partial expected
value of perfect information from a probabilistic sensitivity analysis sample:
a nonparametric regression approach. Med Decis Making. 2014;34(3):
311–326.
77. Brennan A, Kharroubi S, O’Hagan A, Chilcott J. Calculating partial expected
value of perfect information via Monte Carlo sampling algorithms. Med Decis
Making. 2007;27(4):448–470.
HEALTH POLICY ANALYSIS 1123
78. Hall PS, Edlin R, Kharroubi S, Gregory W, McCabe C. Expected net present
value of sample information: from burden to investment. Med Decis Making.
2012;32(3):E11–E21.
79. Edlin R, Hall P, Wallner K, McCabe C. Sharing risk between payer and provider
by leasing health technologies: an affordable and effective reimbursement
strategy for innovative technologies? Value Health. 2014;17(4):438–444.
80. Claxton K, Palmer S, Longworth L, et al. Informing a decision framework for
when NICE should recommend the use of health technologies only in the
context of an appropriately designed programme of evidence development.
Health Technol Assess. 2012;16(46):1–323.
81. Grimm SE, Stevens JW, Dixon S. Estimating future health technology diffu-
sion using expert beliefs calibrated to an established diffusion model. Value
Health. 2018;21(8):944–950.
82. Whyte S, Dixon S, Faria R, et al. Estimating the cost-effectiveness of imple-
mentation: is sufficient evidence available? Value Health. 2016;19(2):
138–144.
83. Eckermann S, Willan AR. Optimal global value of information trials: better
aligning manufacturer and decision maker interests and enabling feasible
risk sharing. Pharmacoeconomics. 2013;31(5):393–401.
84. Gold ER. The fall of the innovation empire and its possible rise through open
science. Res Policy. 2021;50(5):104226.