You are on page 1of 7

Received: 18 May 2018

| Revised: 16 October 2018


| Accepted: 2 December 2018

DOI: 10.1111/ajr.12496

ORIGINAL RESEARCH

Efficacy of a remote screening model for oral potentially


malignant disorders using a free messaging application: A
diagnostic test for accuracy study

Kalaiselvi Vinayagamoorthy MDS1 | Shruthi Acharya MDS2 | Mathangi Kumar MDS2 |


Kalyana Chakravarthy Pentapati MDS1 | Shashidhar Acharya MDS1

1
Department of Public Health Dentistry,
Manipal College of Dental Sciences, Abstract
Manipal Academy of Higher Education, Objective: To assess the feasibility of using a remote sensing model as a free
Manipal, India
messaging application tool in the preventive screening of oral potentially malig-
2
Department of Oral Medicine and
nant disorders in a rural area of India.
Radiology, Manipal College of Dental
Sciences, Manipal Academy of Higher Design: An observational cross‐sectional study.
Education, Manipal, India Setting: Primary care setting in Udupi District, Karnataka, South India.
Correspondence Participants: One‐hundred and thirty‐one individuals with a mean (SD) age of
Shashidhar Acharya, Department of Public 37.34 (11.31) years, of whom 64.1% and 35.9% were men and women, respec-
Health Dentistry, Manipal College of
tively.
Dental Sciences, Manipal Academy of
Higher Education, Manipal, India. Interventions: Clinical oral examination followed by photo capture of five areas
Email: sh.acharya@manipal.edu of the patients' mouth.
Main outcome measures: Reliability measures for the use of a photo messaging
service in diagnosing oral potentially malignant disorders, as compared to the
clinical examination.
Results: When lesions were categorised as normal and abnormal, the reliability
(kappa) between the diagnoses, based on photo messaging and clinical oral exam-
ination, was 0.68 and 0.67 for Examiners 1 and 2, respectively. The sensitivity
values for Examiners 1 and 2 were 98.5% and 99.04%, respectively, whereas the
specificity was 72% and 64%, respectively. When the agreement between photo
messaging and clinical oral examination for an exact diagnostic match was
assessed, the reliability (kappa) was 0.59 and 0.55 for Examiners 1 and 2, respec-
tively. The sensitivity values for Examiners 1 and 2 were 98.1% and 98.7%,
respectively, whereas the specificity was 64% and 52% respectively.
Conclusion: There was a substantial agreement between the diagnosis based on
clinical examination and WhatsApp image for both the examiners, when the
lesions were dichotomised as normal and abnormal, but slightly reduced when
assessed for the exact diagnostic match. Screening for oral potentially malignant
disorders using photo messaging can serve as an effective adjunct and a potential
cost‐effective tool in a low‐resource setting.

KEYWORDS
lesions, premalignant, reliability, teledentistry, WhatsApp

Aust. J. Rural Health. 2019;1–7. wileyonlinelibrary.com/journal/ajr © 2019 National Rural Health Alliance Ltd. | 1
2
| VINAYAGAMOORTHY ET AL.

1 | INTRODUCTION
Advances in IT have reduced the distance between What is already known on this subject:
individuals and health care providers with unique • There are numerous applications, which are
prospects for remote access.1 Teledentistry is one such
broadening the scope of teledentistry.
technological innovation in dentistry that can serve to • Many of these apps are proprietary, difficult to
bridge the gap between health care providers and indi-
use, require an excellent Internet connectivity
viduals. Teledentistry is defined as the “diagnosis and
and are out of reach of the rural and remote pop-
treatment of dental patients using electronically aided
ulations.
communication technologies”.2 Also, teledentistry has • There is a lack of diagnostic evidence of health
been reported to be a cost‐effective means of reaching
apps on Apple's and Google's app stores.
the less privileged.3,4 The application of teledentistry is • Some studies have shown the efficiency of free
diverse and its efficiency has been proved in different
messaging apps for remote consultation by
dental settings.
experts in dental settings.
Given the increased incidence rates of oral cancer in • Early diagnosis of oral premalignant lesions is
underprivileged populations, the fragmented health care
the most effective means of reducing the inci-
system and the associated higher mortality and morbidity
dence of malignancies and its associated morbid-
rates in developing countries like India, alternative cost‐
ity and mortality.
effective and readily available methods of early detection
like teledentistry need to be considered.5 Moreover, the
main challenge in the early detection of oral cancer is the What this study adds:
diagnostic pathway that relies on an unhindered exchange • Free messaging apps are a reliable tool for the
of information between primary and secondary health care remote screening of oral premalignant disorders.
providers.6 Teledentistry is a promising alternative to over- • There was a high reliability when the oral mucosa
come geographic distance as a barrier to seek expert con- was dichotomised as normal and abnormal.
sultation for oral potentially malignant disorders (PMDs) • The reliability reduced to moderate levels when
and increase the early detection of oral cancer. In recent the accuracy requirement of the screening
times, the increased usage of smartphones by health care increased to an “exact” lesion diagnosis.
professionals has enhanced the role of teledentistry in • Screening for oral potentially malignant disor-
everyday practice. ders, using photo messaging, can serve as an
Innovations in smartphone imaging technology and effective adjunct and a potential cost-effective
instant messaging services provide better prospects for the tool in a low-income and remote setting.
utilisation of teledentistry as a means for virtual consulta-
tion.7 Smartphones have the basic requirements for a tele-
dentistry system like image capture, storage and transfer
to distant location and view without change in resolution. 2 | METHODS
The ready availability and user‐friendly nature of smart-
phones make it an efficient and cost‐effective alternative A cross‐sectional study was conducted between January
to the conventional telemedicine system.8 Studies have and June 2017, among a convenience sample of partici-
proved the efficacy of photographic methods in the diag- pants of oral screening programs in different areas of
nosis of oral lesions.3,9 Cross‐platform mobile messaging Udupi District, Karnataka. The sample size was estimated
applications that allow exchange of text messages, images, to be 131, considering a power of 90%, kappa of 0.9 and
audio and video messages online have been found to be an expected observed proportion of premalignant lesions of
useful for remote consultation by experts in dental 10%.
settings.10-12 Prior to the start of the study, information sheets and
Reduced access to and utilisation of cancer screening consent forms for clinical examination and photographing
services in rural and remote areas might lead to poor can- the oral cavity were provided to participants who visited
cer outcomes.13-16 However, there is limited evidence on the screening programs. Photographs were focussed only
the use of photo messaging as an adjunct in the screening inside the oral cavity excluding the face, thus preventing
of individuals for oral PMDs in a community setting. direct identification of the participants. Those who had
Therefore, this study was done to assess the feasibility of problems with comprehension, and limited mouth opening
using photo messaging as a tool in the preventive screening restricting accessibility for the mobile phone camera to
for oral PMDs in a rural area of India. click images, were excluded from the study.
VINAYAGAMOORTHY ET AL. | 3

assessment of the images sent through photo messaging


2.1 | Clinical oral examination procedures was conducted to determine the reliability (kappa).
Each participant was given an individualised code number.
Information was collected on demographic details (age,
sex), medical history and oral habits among those who ful-
2.5 | Statistical analysis
filled the inclusion criteria. Following this, a screening of Data were analysed using Statistical Package for Social
the oral cavity of the participants was carried out by a Sciences (SPSS for Windows, version 20; SPSS, Chicago,
trained and calibrated examiner (Examiner 1). All examina- IL, USA). P ≤ 0.05 was considered statistically significant.
tions were carried out under a natural light source using a The reliability was tested at two levels. At the general level,
mouth mirror and explorer. The findings of the clinical oral lesions were categorised as normal and abnormal where oral
examination were recorded in the proforma. lesions, like leukoplakia, tobacco pouch keratosis, postin-
flammatory hyperpigmentation and oral submucous fibrosis,
were coded as abnormal. Clinically, normal mucosa and nor-
2.2 | Image capture mal variants, like benign migratory glossitis, linea alba buc-
In a separate room, a comprehensive set of five photographs calis, depapillation and hyperpigmentation, were coded as
for each participant, covering different areas of the oral cav- normal. At the specific level, the reliability of an exact diag-
ity, was captured by a single calibrated dental student using a nostic match of lesions between the examinations was con-
standardised procedure on a mobile phone camera under nat- sidered. The kappa statistic was calculated to test the the
ural but not direct sunlight. A Samsung Note 2 mobile phone inter‐examiner and intra‐examiner reliability of the clinical
with an 8‐megapixel camera and a display with a resolution oral (gold standard) examination and photographic examina-
of 720 × 1280 pixels with autofocus (Samsung, Seoul, tion (WhatsApp images) for both levels. Accuracy, sensitiv-
South Korea) was used for this purpose. The images were ity, specificity, positive predictive value and negative
obtained in a sequential manner: left buccal mucosa; right predictive value were calculated for photo messaging, as
buccal mucosa; then upper labial mucosa including buccal compared with a clinical oral examination (gold standard).
vestibule; then upper labial mucosa including buccal vesti- Further, the agreement between the two examiners for the
bule and tongue. The photographs were checked immedi- diagnosis based on photo messaging was calculated.
ately for clarity and coverage of the desired area. Following
this, the images of each patient were sequentially numbered
and stored as separate folders in the phone storage. The
2.6 | Ethics approval
images were not modified or enhanced. The study was approved by the Kasturba Hospital Institu-
tional Ethics Committee.
2.3 | Image transfer and evaluation
The photo messaging feature of the WhatsApp messenger 3 | RESULTS
(version 2.17.190; WhatsApp, Mountain View, CA, USA)
was used in the present study. Information, regarding age, A total of 131 individuals (655 images) were screened and
sex, medical and habit history along with five images of the images of five areas of the oral cavity were captured.
the oral cavity of each individual, were sent through the Out of the 655 images, five images were scored as unclear
messenger to both the examiners (1 and 2) after a washout by either of the examiners, when sent through photo mes-
period of 2 months. Both the examiners separately analysed saging (Figure 1).
the images in their mobile phones with the display bright- The mean (SD) age of the screened individuals was 37.34
ness at 50%. Both the examiners used Android (11.31) years. This included 64.1% of men and 35.9% of
smartphones with a display size of 5.50″ and a display res- women. Approximately 13.1% of the screened individuals
olution of 1080 × 1920 pixels. had a history of tobacco smoking habit. Among the 131 indi-
viduals screened, 16 of them were diagnosed with PMDs
during the clinical oral examination, with a prevalence of
2.4 | Training and calibration 12.2%. The PMDs diagnosed were tobacco pouch keratosis
Prior to the main study, a pilot study was conducted among (n = 10), leukoplakia (n = 2), oral submucous fibrosis
patients attending the dental outpatient department of the (n = 3) and postinflammatory hyperpigmentation (n = 1).
university hospital. The patients were examined twice by All of these individuals were men and habitual tobacco users
Examiners 1 and 2, following which the images of five with a mean (SD) age of 41 (13.06) years.
areas of the oral cavity were captured by the calibrated When lesions were categorised as normal and abnormal,
dental student. After a washout period of 1 month, an the reliability (kappa) between the diagnosis based on
4
| VINAYAGAMOORTHY ET AL.

Participants screened
n = 131

Images obtained
n = 655(131*5)

Excluded images
n=5
Diagnosed using
Reasons -
WhatsApp
image scored unclear
n = 650

Examiner 1 Examiner 2

WhatsApp WhatsApp WhatsApp WhatsApp


diagnosis normal diagnosis abnormal diagnosis normal diagnosis abnormal
n = 622 n = 27 n = 627 n = 22

Clinical oral Clinical oral Clinical oral Clinical oral


examination examination examination examination
diagnosis normal diagnosis abnormal diagnosis normal diagnosis abnormal
F I G U R E 1 Flow diagram to report the
n = 624 n = 25 n = 624 n = 25
flow of participants through the study

photo messaging and clinical oral examination was 0.68 (Table 2). Kappa for the agreement between the two exam-
and 0.67 for Examiners 1 and 2, respectively (Figures 2 iners in the diagnosis, based on photo messaging images,
and 3). The sensitivity values for Examiners 1 and 2, were was 0.59 and the accuracy was 97 (Table 3, Figure 4).
98.5% and 99.04%, respectively, whereas the specificity
was 72% and 64%, respectively (Table 1).
When the agreement between photo messaging and clin- 4 | DISCUSSION
ical oral examination for an exact diagnostic match was
assessed, the reliability (kappa) was 0.59 and 0.55 for In our study, there was a substantial agreement between the
Examiners 1 and 2, respectively. The sensitivity values for diagnosis based on clinical examination and photo messag-
Examiners 1 and 2 were 98.1% and 98.7%, respectively, ing image for both the examiners, when the lesions were
whereas the specificity was 64% and 52%, respectively dichotomised as normal and abnormal. However, when the

F I G U R E 3 Clinical oral examination‐tobacco pouch keratosis;


F I G U R E 2 Clinical oral examination‐normal; WhatsApp WhatsApp (Examiner 1), normal; WhatsApp (Examiner 2), tobacco
(Examiner 1), normal; WhatsApp (Examiner 2), normal pouch keratosis
VINAYAGAMOORTHY ET AL. | 5

T A B L E 1 Sensitivity, specificity, reliability, positive and negative predictive values in the diagnosis based on photographic images, as
compared with clinical oral examination (normal/abnormal)

Diagnosis based on Kappa (95% CI) Accuracy (%) Sensitivity (%) Specificity (%) PPV (%) NPV (%)
Photo messaging images
Examiner 1 vs gold standard 0.68 (0.53‐0.83) 97.5 98.56 72.00 98.87 66.67
Examiner 2 vs gold standard 0.67 (0.51‐0.82) 97.6 99.04 64.00 98.57 72.73

CI, confidence interval; NPV, negative predictive value; PPV, positive predictive value.

T A B L E 2 Sensitivity, specificity, reliability, positive and negative predictive values in the definitive diagnosis based on images, as compared
with clinical oral examination (exact diagnostic match)

Diagnosis based on Kappa (95% CI) Accuracy (%) Sensitivity (%) Specificity (%) PPV (%) NPV (%)
Photo messaging images
Examiner 1 vs gold standard 0.59 (0.43‐0.75) 96.77 98.08 64.00 98.56 57.14
Examiner 2 vs gold standard 0.55 (0.37‐0.72) 96.92 98.72 52 98.09 61.90
CI, confidence interval; NPV, negative predictive value; PPV, positive predictive value.

T A B L E 3 Reliability of diagnosis based on images between the two examiners


Diagnosis based on Kappa (95% CI) Accuracy (%) Sensitivity (%) Specificity (%) PPV (%) NPV (%)
Photo messaging images
Examiner 1 vs examiner 2 0.6 (0.43‐0.76) 97.0 98.09 68.18 98.88 55.56

CI, confidence interval, NPV, negative predictive value; PPV, positive predictive value.

messaging‐based image assessment indicated a slight over-


estimation of the lesions, as compared with clinical oral
examination, which could be attributed to the wide array of
diagnoses in oral medicine, the two‐dimensional representa-
tion of the lesions and the factors, affecting the image qual-
ity, like the extent of coverage and lighting.18 This could be
further explained by the subtle nature of early lesions of oral
cancer and precancer, which lack the characteristics demon-
strated in advanced lesions.19
Drawbacks mentioned in previous studies, assessing the
feasibility of teledentistry like image quality, clinical data
and habit history, were minimised in the present study.9,17
Along with images, information on habit history and clini-
cal data for each patient was sent to the examiners through
photo messaging. In our study, images were captured using
F I G U R E 4 Clinical oral examination‐Leukoplakia; WhatsApp
a mobile phone with an 8 megapixel camera. Previous
(Examiner 1), leukoplakia; WhatsApp (Examiner 2), normal
studies have reported 8 megapixels as suitable for an
reliability was assessed for an exact diagnostic match, the image‐based diagnosis.17 Five images were excluded due
kappa values for the examiners decreased to moderate levels to lack of diagnostic details by either of the examiners.
of agreement. Inter‐examiner agreement between the two The images are generally compressed when transferred
examiners in the diagnosis, based on photo messaging using photo messaging, which might have significant effect
images, was moderate. The results of the present study were on the diagnosis.
in agreement with those of earlier studies assessing Overall, the findings demonstrated a good level of
image‐based screening for oral lesions.10,17 Moreover, the agreement between the two methods (clinical and photo-
heterogeneity of oral lesions, importance of palpation and graphic examination) and the two examiners. The results
histological examination in confirmatory diagnosis need to also showed that the examiners were consistent in the way
be considered.10 The sensitivity and specificity of photo‐ they diagnosed premalignant lesions from photographs.
6
| VINAYAGAMOORTHY ET AL.

The sensitivity scores were higher than the specificity PMD. Teledentistry models can be used as an adjunct to
scores across all examiners and assessment methods. The traditional models where they exist and as the next best
higher sensitivity scores were in line with the World Health thing in areas where they are lacking.
Organisation's reference standard of 0.85‐0.90 for sensitiv-
ity values.20
ACKNOWLEDGEMENTS
Recent advancements in telecommunication services and
the provision of high‐speed internet services at low cost, The authors thank the non‐teaching staff who helped carry
covering a majority of the rural areas in countries like out the screening.
India, can be leveraged in the provision of health care ser-
vices to remote areas.21 The fact that the colour gamut of
CONFLICT OF INTEREST
phones varies enormously and that there is considerable
inconsistency between phones, even of the same brand, The authors have no conflict of interest to declare and this
needs to be considered. As such, this is likely to have an study has not in part or whole, been published or is under
impact on the diagnostic yield of any proposed arrange- consideration for publication elsewhere. No external fund-
ment. It has to be conceded that the use of a free messag- ing was obtained for this study.
ing service would be precluded in some countries due to
information governance concerns.
AUTHORS’ CONTRIBUTIONS
Distance and isolation remain the major obstacles for
providing health care in rural and remote areas. Whether Kalaiselvi Vinayagamoorthy, Kalyana Chakravarthy Penta-
the secondary care infrastructure has the capacity to deal pati and Shashidhar Acharya conceptualised and designed
with this diagnostic pathway is still unclear. Furthermore, the study; Kalyana Chakravarthy Pentapati, Shruthi
this capacity might be distributed unevenly and dimin- Acharya and Mathangi Kumar analysed and interpreted the
ished in areas that require it the most. There has been an data; and Kalaiselvi Vinayagamoorthy and Shashidhar
increasing use of telemedicine in cancer care in many Acharya prepared the manuscript. All authors edited and
countries around the world and in countries like Australia reviewed the manuscript.
in particular, to provide cancer care closer to homes.22
Research on these models has demonstrated feasibility,23
ORCID
patient and health professional acceptance24-26 and cost
savings to the health systems.27 The teledentistry model Shashidhar Acharya https://orcid.org/0000-0002-2196-
is very suitable in countries like India with a high preva- 3979
lence of oral PMDs, and paucity of skilled manpower
and existing cancer screening programs. Using free mes-
saging services for screening oral lesions can help REFERENCES
improve the quality of care in remote areas by facilitating
1. Purohit BM, Singh A, Dwivedi A. Utilization of teledentistry as a
better decision‐making through expert consultation. How- tool to screen for dental caries among 12‐year‐old school children
ever, in countries with a lower prevalence, these diagnos- in a rural region of India. J Public Health Dent. 2017;7:174‐180.
tic aids can be used in a “PML present or absent” mode 2. Ignatius E, Makela K, Happonen RP, Perala S. Teledentistry in
to aid the referral pathway from primary to secondary dental specialist education in Finland. J Telemed Telecare.
care. 2006;12:46‐49.
As compared with specialised telemedicine/dentistry 3. Bradley M, Black P, Noble S, Thompson R, Lamey PJ. Applica-
tion of teledentistry in oral medicine in a Community Dental Ser-
apps that require some level of technological sophistica-
vice, N. Ireland. Br Dent J. 2010;209:399‐404.
tion to operate, common freeware is easy to use and 4. Mariño R, Hopcraft M, Tonmukayakul U. Teleconsultation/teledi-
requires much less training. Furthermore, these apps agnosis using teledentistry technology: a pilot feasibility study.
require a much lesser bandwidth than other proprietary Int J Adv Life Sci. 2014;6:291‐299.
apps and consume much less device memory. In cases 5. Coelho KR. Challenges of the oral cancer burden in India. J Can-
where the experts are in short supply or are unable to cer Epidemiol. 2012;2012:17, Article ID 701932.
cover remote areas on a regular basis, this system can be 6. McLeod N, Saeed N, Ali E. Oral cancer: delays in referral and
the next best thing to the physical presence of an expert. diagnosis persist. Br Dent J. 2005;198:681‐684.
7. Estai M, Kanagasingam Y, Xiao D, et al. A proof‐of‐concept
It has to be conceded that no matter how fine the tech-
evaluation of a cloud‐based store‐and‐forward telemedicine app
nique, it is possible to miss a lesion in the teledentistry for screening for oral diseases. J Telemed Telecare. 2016;22:
mode and the patient could later develop oral cancer. But 319‐325.
the same can be said for clinical examination as well 8. Kopycka-Kedzierawski DT, Billings RJ. Teledentistry in inner‐
where a PMD could be missed or misdiagnosed as a non‐ city child‐care centres. J Telemed Telecare. 2006;12:176‐181.
VINAYAGAMOORTHY ET AL. | 7

9. Torres-Pereira C, Possebon RS, Simoes A, Bortoluzzi MC, Leao JC, 20. World Health Organization. Oral Health Surveys: Basic Methods
Giovanini AF. Email for distance diagnosis of oral diseases‐prelimin- World Health Organization, 5th edn. Geneva, Switzerland: World
ary study of teledentistry. J Telemed Telecare. 2008;14:435‐438. Health Organization; 2013.
10. Petruzzi M, Benedittis MD. WhatsApp: a telemedicine platform 21. Mallick R, Perwez E, Noor R, Sachdeva S, Tamrakar AK. Tele-
for facilitating remote oral medicine consultation and improving dentistry: proposal of an implementation model in India. Ann
clinical examinations. Oral Surg Oral Med Oral Pathol Oral Dent Spec. 2016;4:33‐37.
Radiol. 2016;121:248‐254. 22. Hazin R, Qaddoumi I. Teleoncology: current and future applica-
11. Sarode SC, Sarode GS, Anand R, Patil S, Unadkat H. WhatsApp tions for improving cancer care globally. Lancet Oncol.
is an effective tool for obtaining second opinion in oral pathology 2010;11:204‐210.
practice. J Oral Pathol Med. 2017;46:513‐519. 23. Sabesan S, Larkins S, Evans R, et al. Telemedicine for rural can-
12. Zotti F, Dalessandri D, Salgarello S, et al. Usefulness of an app cer care in North Queensland: bringing cancer care home. Aust J
in improving oral hygiene compliance in adolescent orthodontic Rural Health. 2012;20:259‐264.
patients. Angle Orthod. 2016;86:101‐107. 24. Sabesan S, Simcox K, Marr I. Medical oncology clinics
13. Underhill C, Bartel R, Goldstein D, et al. Mapping oncology ser- through videoconferencing: an acceptable telehealth model for
vices in regional and rural Australia. Aust J Rural Health. rural patients and health workers. Intern Med J. 2012;42:780‐
2009;17:321‐329. 785.
14. Garvey G, Cunningham J, Valery PC, et al. Reducing the burden 25. Mair F, Whitten P, May C, Doolittle GC. Patients’ perceptions of
of cancer for Aboriginal and Torres Strait Islander Australians: a telemedicine specialty clinic. J Telemed Telecare. 2000;6:36‐
time for a coordinated, collaborative, priority driven, Indigenous‐ 40.
led research program. Med J Aust. 2011;194:530‐531. 26. Mooi JK, Whop LJ, Valery PC, Sabesan SS. Teleoncology for
15. Roder D. Epidemiology of cancer in Indigenous Australians: indigenous patients: the responses of patients and health workers.
implications for service delivery. Cancer Forum. 2007;31:85‐90. Aust J Rural Health. 2012;20:265‐269.
16. Australian Health Ministers’ Advisory Council. Aboriginal and 27. Thaker DA. Cost savings from a telemedicine model of care in
Torres Strait Islander Health Performance Framework 2014 northern Queensland, Australia. Med J Aust. 2013;199:414‐417.
Report. Canberra, ACT: AHMAC; 2015.
17. Haron N, Zain RB, Nabillah WM, et al. Mobile phone imaging
in low resource settings for early detection of oral cancer and
concordance with clinical oral examination. Telemed J E Health.
How to cite this article: Vinayagamoorthy K,
2017;23:192‐199. Acharya S, Kumar M, Pentapati KC, Acharya S.
18. Estai M, Kanagasingam Y, Huang B, et al. Comparison of a Efficacy of a remote screening model for oral
smartphone‐based photographic method with face‐to‐face caries potentially malignant disorders using a free
assessment: a mobile teledentistry model. Telemed J E Health. messaging application: A diagnostic test for accuracy
2017;23:435‐440. study. Aust. J. Rural Health. 2019;00:1–7.
19. Lingen MW, Kalmar JR, Karrison T, Speight PM. Critical evalu-
https://doi.org/10.1111/ajr.12496
ation of diagnostic aids for the detection of oral cancer. Oral
Oncol. 2008;44:10‐22.

You might also like