Perceptions of Development of COVID-19 Diagnostic Tests

Study Personnel:

PI: Jonathan Kimmleman

CO-PI: Stephen B. Broomell, Jesse Papenberg, Brian Ward

Project Manager: Patrick Kane

Introduction

Policy makers and members of the public have a keen interest in the development of medical
countermeasures (MCM) against COVID-19. Diagnostic development can be slow and failure
prone under the best of times. Failure to communicate effectively about the prospects of safe and
effective MCMs can have important policy and public health implications during a pandemic.
Unrealistic expectations about MCM promise can also deter patients from enrolling in properly
controlled randomized trials, thus slowing the development of MCMs. They can also generate
pressures for regulatory agencies to hastily license unproven treatments.

The goal of our project is to elicit expert opinion about timelines for development of safe and
effective COVID-19 MCMs, and compare them with perceptions of a sample of US and
Canadian publics. Our basic method is to approach a small sample of experts for most
optimistic, most likely, and most pessimistic timelines for development of COVID-19 MCMs.
National panels of lay people in the US and Canada will complete the same survey. We will then
report expert projections, and measure deviation between lay and expert projections. Our expert
elicitations can be used to inform public communications and research planning; the comparison
of lay and expert forecasts can be used to guide public communication strategies. All results will
be presented on a publicly accessible portal maintained by the research team.

General Objective

We plan to determine expert beliefs about likely timelines for COVID-19 diagnostic
development and compare these beliefs with public opinion.

Primary Objectives

The primary goal of the study is to assess current expert opinion about the timeline for
reaching diagnostic development milestones, along with expert opinion of the chances of
encountering a potential setback. Expert opinion may be further analyzed once these milestones
have been reached. Here we pre-register several hypotheses to be tested immediately that involve
comparing expert opinion to the current public opinion about these same milestones and
potential setback. We therefore seek to (a) summarize expert opinion, and (b) assess how
different public opinion is from expert opinion and (c) investigate predictors of the deviation
between public and expert opinion.

Method Overview

1. Working with expert collaborators specializing in diagnostic development (see Co-

Investigators), we identified 3 important milestones in the development process for
diagnostic tests. For each milestone we will measure how soon that milestone will occur.
We also identified 2 potential setbacks for the development of diagnostic tests. For each
potential setback we measure the probability of the event occurring.
2. We administer this survey to a small group of international experts in diagnostic test
development.
3. We also created a lay version of the survey. This survey translates the milestones for a
lay audience and is augmented with a few questions probing lay people’s understanding
of the development process, and some basic demographic questions.
4. The lay survey will be distributed to representative samples of the US and Canada
recruited through Qualtrics’s survey panel service. The Canadian sample will have the
option of performing the survey in French or English.

Sampling and Sample Selection

The expert sample was recruited directly through e-mail contact from our expert collaborators.
We developed two mailing lists for the survey: 1) by asking our expert collaborator to make a
list of any experts they felt they were comfortable contacting directly and 2) by collecting the e-
mail addresses of corresponding authors of diagnostic test review articles in top infectious
disease journals. The two lists were then deduplicated. Our expert collaborators then e-mailed
both lists, with the exact content of the e-mail varying slightly based on which list the person
belonged on. After 1 week a follow-up email was sent to those who had not yet taken part in the
survey. We aimed to recruit between 10-20 diagnostic test experts. We may additionally
resurvey those experts who participated in the study every other month to study how their
answers to the questions change over time.

The lay sample will be recruited through Qualtrics’s survey panel service. We have the funding
to recruit approximately 1000 respondents from the US and 1000 respondents from Canada.

Survey Design

For experts, the survey will take the form of 3 “milestone questions” and 2 “potential setback”
questions.
For each milestone question, experts will provide a point prediction of when they think that
milestone will occur along with upper and lower bounds representing the soonest and latest that
milestone could occur.

For each potential setback question experts will be asked about the probability of an event
representing a substantive setback in the development process. They will provide the probability
on a scale of 0-100.

Lay people will receive a survey with the same milestone and potential setback questions. In
addition lay people will answer a single question before the milestone questions asking them
whether we have accurate tests in the three categories representing the milestone questions.
Based on their answer to this question, we may not ask them certain milestone questions if their
answer indicates they believe that milestone has already been met (though in fact none of the
milestones have been met). We will also ask lay people to rate their trust in the scientific
community’s handling of the pandemic using a 5 item scale. Additionally we will collect the
following demographic information from lay people: gender, age, political stance (from right to
left) and political party (from Republican, Democratic, Independent in the US and Conservative,
Liberal, Green, NDP and Bloc Quebecois in Canada).

Analysis – Processing and Descriptive Statistics

In the event that experts provide only a year for milestone questions (no month) we will interpret
the month as being the midpoint of the year for the best guess question, the beginning of the year
for soonest question and the end of the year for the longest question. A response with only a
month no year will be treated as missing data.

Because of the small planned sample size among experts, we plan to report only descriptive
statistics for this group including the demographic breakdown, the average (mean and median)
and standard deviation for the point estimate, soonest and latest entries for each milestone
question and the average (mean and median) for each potential setback question. We will use
these measures of central tendency (whichever is more appropriate given the distribution of the
data, averages for symmetric data without outliers, medians otherwise) as a comparison for the
lay measures of central tendency. Additionally, we will create a graphic displaying the
distribution of expert response for each milestone and failure mode question, like a histogram or
violin plot.

For lay people, we will exclude any response where for each kind of question the respondent
provides the same answer for every item of that type (i.e. providing the same date for every
milestone question and the same probability for every potential setback question).
In the event that lay people provide only a year for milestone questions (no month) we will
interpret the month as being the midpoint of the year for the best guess question, the beginning of
the year for soonest question and the end of the year for the longest question. A response with
only a month no year will be treated as missing data.

For lay people we will also report descriptive statistics (for both the overall sample and within
each national sample) including the demographic breakdown, measures of central tendency
(mean and median) and standard deviation for the point estimate for each milestone question and
the average and standard deviation for each potential setback question. The calculated measures
of central tendency and standard deviations will not include participants who (a) indicate the
milestone has already occurred, (b) indicate a date in the past, or (c) violate the constraint that
soonest <= best <= latest in their response. We will report the proportion of participants who
provide such responses for each milestone. Additionally, we will create a graphic displaying the
distribution of lay response for each milestone and failure mode question, like a histogram or
violin plot.

In addition to the statistics described above, we will derive some variables by combining the lay
and expert data. For each lay judgment (milestone point estimates and potential setback
questions) we will calculate the signed difference of that judgment from the expert average (we
refer to these as deviations going forward). We will report the average and standard deviation of
the deviation variable. Additionally for each lay milestone best estimate we will determine
whether that estimate was between the average expert soonest and latest guesses or not (we refer
to this binary variable as boundedness going forward). We will report the average and standard
deviation of the boundedness variable.

Finally, we will attempt to create a scale measure out of 5 items intended to gauge trust in
scientific quality and rigor in research related to COVID-19. The five items comprising this
scale are (asterisks indicate reverse scoring):

1.The increased urgency associated with COVID-19 research will lead to more errors in the
research and development process.*

2.Expert opinions about COVID-19 are generally trustworthy.

3.The scientific community has made a lot of mistakes when dealing with COVID-19.*

4.COVID-19 research oversight and error monitoring should be relaxed because of the need for
treatments.

5.It is unlikely that COVID-19 research suffers from any major errors.

We plan to use these items as a predictor in the regression analyses described below. If the scale
has a Cronbach's alpha higher than 0.60 then we will treat the items as a scale and calculate the
scale score for each individual. In the event the alpha is too low, we will perform a leave one out
analysis to see if we can construct a reliable scale from a subset of the items. Finally, if no
reliable scale is found we will use just answers to the first item.
Analysis – Hypotheses

1. Experts vs. Lay Forecast Hypotheses

With guidance from our expert Co-PI’s we have selected milestones and potential
setbacks that have not yet occurred as of the launch of our survey. There is a lot of uncertainty
in what the milestone expectations will be like on both the expert and lay sides, especially given
the coverage of diagnostic testing in the media. Thus while we think there may be differences in
the lay and expert milestone judgments we are uncertain about directionality:

H1: The public will provide best guesses that represent systematically different time
frames for the milestones relative to the experts. A substantive minority of the public
may believe the milestones have already occurred.

H2: The public will provide probabilities for the potential setbacks that are systematically
different from the experts.

Along the same lines we expect a fair amount of uncertainty in the expert population as to
when milestones will occur, which we propose to measure by looking at the difference between
the average latest guess and the average soonest guess. We anticipate that lay people will likely
believe that this uncertainty is much higher given that there has been a lot of news coverage of
failures in testing. Thus we expect to find that:

H3: The range between public soonest and latest guesses for the milestone questions will
be systematically different than the experts average range between the soonest and latest guess.

1. Demographics, Perceptions and Forecast Relationships

As an exploratory analysis we plan to explore some individual differences in how closely

lay people’s beliefs mirror those of experts. To do this we will compute a measure of deviation
between each individual participant’s response and the mean expert opinion for each milestone
and potential setback. Individually for each milestone and setback, we will regress these
deviations on answers to the faith in science questions, age, gender, ethnicity, political stance and
nationality (US versus Canada) to investigate predictors of individual differences. We will
include an interaction term between nationality and each of the other predictors in the model.
Specifically, we plan to look for:

H4:A significant association between public-expert deviations and political affiliation.

H5: Political affiliation playing a larger role in predicting judgments in the US and a
smaller role in Canada, as measured by a significant interaction between political stance
and nationality in the regression algorithm.

H6: A significant association between public-expert deviations and education.

Analysis – Inferential Statistics

Our primary analysis is the comparison between the mean lay and expert perceptions. Prior to
running any inferential statistical tests, we will check the distributional assumptions of our data.
We anticipate that the data will not meet the standard parametric assumptions for t-tests. If this
is the case, we will use a non-parametric resampling based t-test instead.

For the milestone questions we will compare the lay point prediction with average expert point
prediction using a one sample t-test (or an equivalent non-parametric test). For the potential
setback questions we will compare the lay probabilities with the average expert probability using
a one sample t-test (or an equivalent non-parametric test). We will also compare the milestone
uncertainties to the difference between the average expert soonest guess and the average expert
latest guess using a one sample t-test (or an equivalent non-parametric test). We will perform
these tests comparing both the US lay sample to the experts and comparing the Canadian lay
sample to the experts. We will also compare the US lay sample to the Canadian lay sample on
all of the same variables using two sample t-tests (or equivalent non-parametric tests). We will
apply a Bonferonni correction to the significance level for all of these tests, resulting in an
approximately 0.05/24=0.002 significance threshold.

As a secondary exploratory analysis, we will test whether there is any relationship between the
deviation variable and the trust in science questions and demographic variables, including age,
gender, political stance and nationality (US versus Canada) using regression analysis. Because
we are interested in differences in the US and Canadian population we will include an interaction
between nationality and all other predictors. We will examine the distribution of the deviations
variable to determine if any transformations are appropriate, or if a generalized linear model
would be more appropriate. In the event that participants who believe the milestones have
already occurred are a large portion of the sample, we will also regress a binary variable
denoting whether a participant’s best guess for the milestone question was in the past or future
on the demographic variables. Finally we may perform a similar regression analysis on the
boundedness variable using logistic regression.