Professional Documents
Culture Documents
Sarvesh Sawant, Aswathi Nair, Shaik Aisha Sultana, Arjun Rajendran & Kapil
Chalil Madathil
To cite this article: Sarvesh Sawant, Aswathi Nair, Shaik Aisha Sultana, Arjun Rajendran & Kapil
Chalil Madathil (2021) Evaluation of the Content Quality of YouTube Videos on Alternate Therapies
for Prevention or Treatment of COVID-19, Journal of Consumer Health on the Internet, 25:2,
133-155, DOI: 10.1080/15398285.2021.1902226
Article views: 24
Introduction
The Internet has become a popular source of healthcare information for
both healthcare professionals and the general public (Cuan-Baltazar et al.
2020; Madathil et al. 2015). In fact, studies have found that the Internet
has become the primary source of health-related information for most
people across the globe (AlGhamdi and Moussa 2012; Griffiths et al. 2012;
Andreassen et al. 2007). Internet users, people with health concerns and
patients use the Internet to find information about a specific disease, obtain
second opinions, find alternative sources of treatment and communicate
with others with similar medical conditions about their experiences and the
treatments they adopted. In addition, people often buy drugs or look for
healthcare treatment plans and providers on the Internet (van Uden-Kraan
et al. 2009; Sadasivam et al. 2013; Scharett et al. 2017; Agnisarman et al.
2018). Internet-based peer support groups is one other form of online
media that enables healthcare consumers to share their concerns and expe-
riences, encompass forums, discussion groups, chat rooms, and listservs.
Most peer support groups are forums, where users can create accounts and
post discussion threads to which other users can reply. Recent studies have
reported that users of such peer support groups share personal experiences,
encourage one another, and exchange advice (Madathil et al. 2013; Ponathil
et al. 2021; Narasimha et al. 2019).
Google sees approximately 6.75 million health-related searches every day
(Eysenbach and K€ ohler 2002), and more than 70% of adult Internet users
in the United States search for healthcare-related information (Fox and
Duggan 2013). Eighty percent of the total users who trust the healthcare
information accessed on the Internet have chronic conditions (Madathil
et al. 2015). However, many online sources contain misleading, false, or
biased information, making it difficult for this number of users to distin-
guish between reliable and misleading information (Young 2011; Madathil
and Greenstein 2018; Madathil, Greenstein, and Koikkara 2014).
Of these Internet sources, YouTube, which is used by millions to find
information on various topics including healthcare, has emerged as the
most popular free video-sharing platform, with 500 h of videos uploaded
per minute. Currently, it has 30 million daily active users and 2 billion
monthly users, with more than 95% of the global Internet population
watching YouTube daily (Mohsin 2020). According to Alexa’s top 500 glo-
bal sites, YouTube is ranked second in terms of daily page views (Alexa -
Top Sites, n.d.). YouTube is extremely effective in disseminating content
without regards for the accuracy or reliability of the information because of
its widespread popularity (Brna et al. 2013; Madathil, Greenstein, and
Koikkara 2014). It is important that correct information is made available
through platforms like YouTube to encourage the viewers to comply with
the preventive guidelines and practice the behavior required to prevent the
spread of diseases (Rubin et al. 2009).
Keelan et al. (2007) analyzed the videos on YouTube focusing on
immunization-related content, discovering multiple incidents of misinfor-
mation. In a study related to the analysis of the content quality of
JOURNAL OF CONSUMER HEALTH ON THE INTERNET 135
Methodology
We analyzed social media trace data related to the COVID-19 pandemic
posted on YouTube one of the most popular social media platforms. In the
sections below, we describe how we collected and analyzed our data.
Search strategy
The search phrases “alternate therapy for the treatment of COVID-19” and
“alternate therapy to prevent COVID-19” were entered into the YouTube
search bar (http://www.youtube.com) through the Google Chrome browser
on April 2, 3, 4, and 11, 2020. Videos were selected by filtering based on
view count and relevance, with 120 videos being selected based on this
search strategy and the snowball technique. Snowball sampling is a tech-
nique in which we analyze data that comes as a recommendation based on
the previous data we analyzed (Wasserman and Faust 1994). In this study,
using the snowball sampling technique, we analyzed specific videos that
were recommended by YouTube recommendation engine as “Up next” vid-
eos based on the previous video we watched. These videos have the similar
traits as the videos we analyzed through our search. To limit filtering to
previous user history and to expand the search results, the “Incognito” set-
tings were selected. Incognito Mode is a privacy feature online on Google
Chrome browser that prevents the storage of a user’s browsing history.
JOURNAL OF CONSUMER HEALTH ON THE INTERNET 137
Data collection
Video features including video title, video URL, clip length, view count, upload
date, likes, dislike, comments, channel name, number of subscribers, video
description (Yes/No) were recorded in an excel spreadsheet. In addition, the
Video Power Index (VPI) was calculated for each video using (1) to determine
the popularity of the videos (Kuru and Yener Erken 2020):
like count
VPI ¼ 100 (1)
dislike count þ like count
The mean daily view count of the videos was calculated using (2) and
(3) (Kuru and Yener Erken 2020):
total view count
Mean daily view count ¼
number of days since the video was uploaded
(2)
number of days since the video was uploaded
¼ Search date upload date (3)
Quality assessment
To address the first research objective, we conducted a quality assessment
of the videos included in this study. All 120 videos and titles were viewed
by two researchers and evaluated using the Journal of the American
Medical Association (JAMA) and the Quality Criteria for Consumer Health
Information (DISCERN) scoring systems (Silberg, Lundberg, and
Musacchio 1997; Charnock et al. 1999). To ensure objectivity, the JAMA
and DISCERN scores were recorded separately by the two observers. These
scores were subsequently averaged to calculate the mean JAMA score using
(4) and mean DISCERN score using (5) (Kuru and Yener Erken 2020):
JAMA score of reviewer 1 þ JAMA score of reviewer 2
Mean JAMA score ¼
2
(4)
DISCERN score of reviewer 1 þ DISCERN score of reviewer 2
Mean DISCERN score ¼
2
(5)
JOURNAL OF CONSUMER HEALTH ON THE INTERNET 139
Table 1. JAMA scoring system (adopted from Kuru and Yener Erken 2020).
Rating
Section Information No Yes
Attribution Sources and references for all content being evaluated 0 1
should be listed, and all relevant copyright
information should be displayed or mentioned.
Authorship Authors and contributors should be mentioned with
their affiliations, and relevant credentials should be
provided clearly.
Disclosure Website “ownership” should be prominently revealed. 0 1
Any advertising, sponsorship, commercial funding
arrangements or support, or potential conflicts of
interest should be disclosed.
Currency There should be clear indication of the dates when 0 1
content was posted and updated
Data analysis
To assess the content related quality of the video uploaded on YouTube,
we conducted a statistical analysis and content analysis to determine the
140 S. SAWANT ET AL.
Table 2. DISCERN scoring system (adopted from Kuru and Yener Erken 2020).
Section Question No Partly Yes
Reliability of the publication Explicit aims 1 2 3 4 5
Aims achieved 1 2 3 4 5
Source of information 1 2 3 4 5
Relevance to patients 1 2 3 4 5
Bias and Balance 1 2 3 4 5
Currency (date) of information 1 2 3 4 5
Reference to areas of uncertainty 1 2 3 4 5
Additional sources of information 1 2 3 4 5
Quality of information on treatment choices Benefits of treatment 1 2 3 4 5
How treatment works 1 2 3 4 5
No treatment options 1 2 3 4 5
Risk of treatment 1 2 3 4 5
Other treatment options 1 2 3 4 5
Quality of life 1 2 3 4 5
Shared decision making 1 2 3 4 5
purpose of uploading these videos, their media type, and their source infor-
mation (category and type), subsequently classifying these videos as either
useful, misleading or neither/nor (Pandey et al. 2010; Murugiah et al. 2011;
Sood et al. 2011; Singh, Singh, and Singh 2012).
Table 3. Coding strategy adopted for content analysis (Kearney et al. 2019).
Theme Code Description
Media Type Blog/Short Film Short film depicting people sharing their experience/
opinions or a blogger (presenter) discussing a topic
Digital Media Videos without a presenter (animations/ slide shows)
News Videos containing clips from news channels, personal
interviews/panel interviews
Personal Video Video shot from personal devices
Purpose of the video Recommendation Video recommending specific type of treatment or
remedy to fight the symptoms of COVID-19 or to
cure the virus
Raise Awareness Videos discrediting myths, promoting information
regarding possible misconceptions, answering
specific question related to the pandemic.
General Information Videos sharing facts about the symptoms, causes,
precautionary measures and immunity strengthening
techniques against flu and cold
Video quality content Misleading Videos promoting information without known scientific
evidence or proof
Useful Videos promoting scientifically sound information with
known scientific evidence for their claims,
suggestions and recommendations
Neither/nor Videos which cannot be directly classified as either
misleading or useful because they contain evidence
but the information was incomplete or insufficient
Content Uploaders Independent Users Content creators who are not linked to any
organization
Government Agency YouTube channels run by government agencies (Center
for Disease Control (CDC)) or the United States
Department of Health and Human Services (HCC))
Health Information YouTube channels or websites specifically providing
Channel/Sites healthcare specific information
News Channel YouTube channels of news agencies
Professional Organization Content creators who are a part of a professional
organization which is not government owned or an
educational institution or a nonprofit organization.
University Channel YouTube channels run by educational institutions
NGO YouTube channels run by nonprofit organizations
Source Type General Channel creates content which covers all domains
including healthcare related videos.
Health Related Channels creating healthcare specific videos only.
Statistical analysis
IBM SPSS 24.0 statistical software was used to analyze the data in the study.
These analyses included descriptive statistics, the Cronbach’s a value, and
Spearman’s correlation coefficient to evaluate the degree of agreement
between the observers. To determine the presence of significant differences
across the mean DISCERN and JAMA scores with the VPI value between the
content creators focusing specifically on healthcare-related videos and the
general content creators, an independent sample t-test was used with a 95%
142 S. SAWANT ET AL.
Results
The 120 most relevant videos selected for this study had a total of
27,515,592 views and a total duration of 10 h and 47 min, with an average
of 5 min and 39 s per video. In this section we will describe the general
characteristics of the videos followed by analyzing the content quality of
YouTube videos disseminating information about alternate therapies for
COVID-19 and finally, we will evaluate the level of correlation between the
quality assessment findings and user ratings.
Figure 2. Number of videos, DISCERN, JAMA and VPI values of Heath care content creators and
general content creators.
Evaluation of the difference among the mean DISCERN, JAMA and VPI
values between healthcare specific content and general content creators
Content creators focusing specifically on health-related topics had a better
DISCERN score overall and thus, better quality content in what they
uploaded about the COVID-19 pandemic. Figure 2 shows the difference
between the mean value of quality parameters of these two groups of con-
tent creators. The results of the independent sample t-test showed that the
mean DISCERN score was statistically higher for the group with only
health-related videos on their channels compared to the ones with videos
on various topics (p < 0.05, mean difference ¼ 8.7). JAMA scores were also
statistically higher for the healthcare-specific channels than for the general
content group (p < 0.05, mean difference ¼ 0.4). There was no statistically
significant difference between the VPI scores for the two groups (p ¼ 0.05).
Videos uploaded by channels supporting only healthcare-specific videos
JOURNAL OF CONSUMER HEALTH ON THE INTERNET 145
Figure 3. Distribution of the quality of the videos according to the DISCERN scoring system.
received more views and likes on average (318,992 views, 5301 likes) than
general content creators (162,999 views 3,169 likes).
garlic to make use of its antiviral properties; maintaining hygiene and social
distancing measures and quarantining. The videos uploaded during the months
of March showed that clinical trials were being conducted on potential medica-
tions such as Remdesivir and hydroxychloroquine/chloroquine, both of which
showed promise in reducing respiratory symptoms. Currently, there is no clin-
ically approved cure for COVID-19. The treatments given for influenza and
other respiratory illnesses referred to as supportive care, are currently being
used to treat patients suffering from COVID-19.
Of the videos examined here, 27.8% (n ¼ 5) contributed to raising aware-
ness by discrediting myths, promoting information regarding possible mis-
conceptions, and answering specific question related to the pandemic,
specifically those related to the use of supplements Vitamin D and Vitamin
C as a treatment option, suggesting that these supplements could be poten-
tially used to strengthen immunity but are not identified as cures for
COVID-19. Additionally, these videos highlighted the possible implications
of the use of antiviral medications without consulting a physician and of
the use of potentially harmful substances that could lead to serious med-
ical issues.
Overall, 40.8% (n ¼ 49) of the videos were classified as misleading, and
37.5% (n ¼ 45) were classified as useful. Videos that couldn’t be classified
as either were termed as neither/nor (21.7%, n ¼ 26), with 34.7% (n ¼ 17)
of the total videos classified as misleading considered potentially harmful
because of possible chemical injury or other health-related issues.
Regarding potentially harmful videos, 52.9% (n ¼ 9) suggested the use or
consumption of potentially harmful substances that could lead to serious
medical issues and 47.1% (n ¼ 8) prescribed medications that were not offi-
cially approved by a government agency for the treatment of COVID-19.
Discussion
The objective of this study was to evaluate the characteristics and content
quality of the YouTube videos providing information regarding the alterna-
tive therapies for preventing or treating COVID-19. Content analysis was
JOURNAL OF CONSUMER HEALTH ON THE INTERNET 147
conducted to identify the themes and characteristics of the videos, and the
content quality of the videos was evaluated using two quality scales, the
DISCERN and JAMA scoring systems. Below we discuss the emerging
themes and theoretical implications of our findings. Subsequently, we pro-
pose interventions that could potentially improve the number and visibility
of high quality videos on YouTube.
disease. Such information threatens the lives of people who could poten-
tially take the unprescribed medications suggested in these videos in an
effort to mitigate their symptoms, possibly leading to an increase in the
mortality rate. Misinformation regarding the spread of the virus can also
cause a sense of panic among people and cause them to take unlawful or
drastic actions against someone else on the pretext of that information.
Moreover, many of the viewers on YouTube are minors (Mueller et al.
2020), important because they may find it difficult to differentiate between
valid and misleading content. As a result, they maybe at a higher risk of
falling prey to misinformation given their age. To address this situation,
children must be taught how to discern between misleading and valid
information. The current situation is further exacerbated because of the
uncertainty about who has the expertise to disseminate information regard-
ing this novel coronavirus (Pennycook et al. 2020). Situations like these
generate a sense of confusion among viewers, making them susceptible to
misleading information.
estimate of the quality of the information and flag what it has found to be
misleading, false or bias content (Nealon and Quirk 2020). Future research
could focus on developing AI-based recommendation engines that could
improve the visibility of high-quality content on YouTube and provide an
easy path for the users to access useful information.
Study limitations
Despite the use of two quality scales (DISCERN and JAMA) to evaluate a
large number of videos on alternative therapies to prevent or treat COVID-
19, our study, like all research, has certain limitations. Although we
conducted a thorough analysis of the video characteristics, we could not
determine and evaluate the potential association of viewer comments with
the video characteristics like view counts, likes, and dislikes. Viewer com-
ments can be an important aspect for gauging viewer engagement and to
understand viewer sentiments toward the content. This study was con-
ducted during the early stage of the pandemic (April 2, 3, 4, and 11, 2020),
and newer developments are evolving every day pertaining to the COVID-
19 pandemic. Therefore, different search results from a different timeframe
are a possibility. We considered only the top 120 videos based on relevance
and view counts because we were evaluating from the viewpoint of
YouTube users, the majority of whom fall into the non-physician or non-
health professional group who tend to view the information that they
encounter first or that which has been watched by the majority of the pre-
vious viewers. Future studies can evaluate these aspects, perhaps using
novel quality assessment techniques.
Conclusion
The study identified that 55% of the videos analyzed were below medical qual-
ity standards and often contained misleading, false, or biased information. A
good proportion of these misleading videos were marked as having potentially
harmful content. However, a poor correlation between the quality assessment
findings and viewer ratings (likes) was observed. Therefore, the viewer rating
cannot be considered as an accurate estimate of the content quality of the vid-
eos on YouTube. The duration of the videos could be a potential reason for the
viewer’s disengagement from the high-quality videos, which tend to be more
detailed and technical and less exaggerated. It is crucial that future studies
evaluate user sentiments to better understand the lack of association between
user ratings and the quality assessment findings and to improve the quality of
medical information on this powerful platform. With developments made daily
regarding various treatment methods and a vaccination for the novel
JOURNAL OF CONSUMER HEALTH ON THE INTERNET 151
References
Agnisarman, S., A. Ponathil, S. Lopes, and K. C. Madathil. 2018. “An Investigation of
Consumer’s Choice of a Healthcare Facility When User-Generated Anecdotal
Information is Integrated into Healthcare Public Reports.” International Journal of
Industrial Ergonomics 66:206–220. doi:10.1016/j.ergon.2018.03.003.
Alexa - Top sites. (n.d.). Retrieved May 23, 2020, from https://www.alexa.com/topsites
AlGhamdi, K. M., and N. A. Moussa. 2012. “Internet Use by the Public to Search for
Health-Related Information.” International Journal of Medical Informatics 81 (6):
363–373. doi:10.1016/j.ijmedinf.2011.12.004.
Andreassen, H. K., M. M. Bujnowska-Fedak, C. E. Chronaki, R. C. Dumitru, I. Pudule, S.
Santana, H. Voss, and R. Wynn. 2007. “European Citizens’ Use of E-Health Services: A
Study of Seven Countries.” BMC Public Health 7 (1): Article number: 53. doi:10.1186/
1471-2458-7-53.
Biggs MA, Gould H, Foster DG. 2013. Understanding why women seek abortions in the
US. BMC Womens Health. 13:29. doi:10.1186/1472-6874-13-29.
Brna, P. M., J. M. Dooley, M. J. Esser, M. S. Perry, and K. E. Gordon. 2013. “Are YouTube
Seizure Videos Misleading? Neurologists Do Not Always Agree.” Epilepsy & Behavior:
E&B 29 (2):305–307.
Centers for Disease Control and Prevention. 2020. Coronavirus Disease 2019 (COVID-19)
– Symptoms [online]. Accessed April 23, 2020. https://www.cdc.gov/coronavirus/2019-
ncov/symptoms-testing/symptoms.html.
Charnock, D., S. Shepperd, G. Needham, and R. Gann. 1999. “DISCERN: An Instrument
for Judging the Quality of Written Consumer Health Information on Treatment
Choices.” Journal of Epidemiology and Community Health 53 (2) :105–111. doi:10.1136/
jech.53.2.105.
Cuan-Baltazar, J. Y., M. J. Mu~ noz-Perez, C. Robledo-Vega, M. F. Perez-Zepeda, and E.
Soto-Vega. 2020. “Misinformation of COVID-19 on the Internet: infodemiology Study.”
JMIR Public Health and Surveillance 6 (2):e18444. doi:10.2196/18444.
Eysenbach, Gunther, and Christian K€ ohler. 2002. “How Do Consumers Search for and
Appraise Health Information on the World Wide Web? Qualitative Study Using Focus
Groups, Usability Tests, and In-Depth Interviews.” BMJ (Clinical Research ed.) 324
(7337):573–577. doi:10.1136/bmj.324.7337.573.
Fat, M. J. L., A. Doja, N. Barrowman, and E. Sell. 2011. “YouTube Videos as a Teaching
Tool and Patient Resource for Infantile Spasms.” Journal of Child Neurology 26 (7):
804–809. doi:10.1177/0883073811402345.
152 S. SAWANT ET AL.
Kapil Chalil Madathil is the Wilfred P. Tiencken endowed Assistant Professor of Civil and
Industrial Engineering at Clemson University. His area of expertise is in applying the
knowledge base of human factors engineering to the design and operation of human-com-
puter systems.