[SUPPLEMENT] Using Clinical Natural Language

Processing for Health Outcomes Research: An

Interdisciplinary Workshop with a focus on Mental
Health

Sumithra Velupillai∗,1,2 , Hanna Suominen3 , Maria Liakata4 ,

Angus Roberts1 , Anoop Shah5 , Katherine Morley1,6 ,
David Osborn7,8 , Joseph Hayes7 , Robert Stewart1,9 ,
Johnny Downs1,9 , Wendy Chapman10 and Rina Dutta1,9
1
Institute of Psychiatry, Psychology & Neuroscience,
King’s College London, UK
firstname.lastname@kcl.ac.uk
2
School of Electrical Engineering and Computer Science,
KTH, Stockholm, Sweden
3
College of Engineering and Computer Science,
The Australian National University, Data61/CSIRO, & Universities of
Canberra, Australia and Turku, Finland
Hanna.Suominen@anu.edu.au
4
Department of Computer Science, University of Warwick
m.liakata@warwick.ac.uk
5
Institute of Health Informatics,University College London, UK &
University College London NHS Foundation Trust. London, UK
anoop@doctors.org.uk
6
Melbourne School of Population and Global Health, The University of
Melbourne, Australia
7
Division of Psychiatry, University College London, UK
firstnameinitial.lastname@ucl.ac.uk 8 Camden and Islington NHS
Foundation Trust, London, UK
9
South London and Maudsley NHS Foundation Trust, London, UK
10
Department of Biomedical Informatics, University of Utah, US

∗ Corresponding author

Preprint submitted to Journal of Biomedical Informatics October 14, 2018

 1. Workshop overview: Scientific content and discussions

A one-day workshop with a particular focus on challenging issues related to

evaluation in NLP and mental health research was organized at King’s College
London, UK, on April 28th 2017, where internationally recognized researchers
5 in this area were invited. Previous and ongoing research efforts were outlined,
and moderated discussions were held to identify and define actionable recom-
mendations for future advancements.
The workshop was attended by 35 people, from 5 countries and 11 in-
stitutions. The participants were a mix of clinical practitioners, clinical re-
10 searchers, epidemiologists, computer scientists, informaticians, and computa-
tional linguists.
The workshop was divided into two main sessions. The morning session pro-
vided a) some context to clinical practice reality – notably in mental health care
settings, and b) examples of NLP approaches and evaluation in relation to clin-
15 ical research questions. The afternoon session moved into more specific clinical
perspectives in the mental health domain, in particular through a challenging
clinical outcome use case of automatic identification of suicidal behaviour, where
NLP methods could play an important role because of its complex nature and
the need for high quality data.

20 1.1. Scientific presentations

Dr. Michael Holland (Medical Director at South London and Maudsley

(SLaM) NHS Trust) gave an introduction to SLaM, a health care trust that
provides the widest range of mental health services in the UK, and exemplified
how informatics approaches can be instrumental in improving health care pro-
25 cesses. Prof. Matthew Hotopf (King’s College, London) then introduced the
NIHR Maudsley Biomedical Research Centre (BRC) and provided some con-
text to the importance of addressing and understanding the clinical research
questions specifically in the mental health domain, and the importance of ap-
propriate informatics methodologies for addressing such research problems. The

2
 Figure 1: Translational informatics as depicted in Prof. Matthew Hotopf’s presentation.
EHR: Electronic Health Records. CRIS: The Clinical Record Interactive Search application
used at the Biomedical Research Centre and South London and Maudsley NHS Foundation
Trust for research purposes.

30 NIHR Maudsley BRC is focused on research in mental health, neurology, and

dementia. He also provided insightful thoughts about directions for the future
in translational informatics. There are currently unprecedented opportunities
in transforming health care research through the integration and analysis of
multiple data sources (Figure 1).
35 Adj/Prof Hanna Suominen (The Australian National University and Data61)
described work on developing and evaluating clinical speech recognition for in-
formation extraction and improved information processing in clinical care, pro-
viding an informatics perspective on issues related to assisting clinicians in their
day-to-day activities and evaluation issues related to the many aspects involved
40 in this type of work [1, 2, 3, 4, 5]. Figure 2 shows an example of an NLP
development workflow, including speech recognition. Several case studies were
then presented by Prof. Wendy Chapman (University of Utah), whose research
focuses on user-centered development of NLP approaches, that is, to involve
clinicians more directly in the development of information extraction and access
45 systems for clinical research [6, 7, 8, 9, 10].
Prof. Robert Stewart (King’s College, London) and Dr. Angus Roberts
(University of Sheffield) started the afternoon session by describing approaches

3
 Figure 2: Examples of NLP workflow including speech recognition, from Adj/Prof Hanna
Suominen’s presentation. Adapted from the Introducing the Data61/CSIRO Natural Lan-
guage Processing Team (2015) slides by Adj/Prof Leif Hanlen, Adj/Prof Hanna Suominen,
Dr Gabriela Ferraro, and Dr Lizhen Qu.

that have been taken at the Institute of Psychiatry, Psychology and Neuro-
science (IoPPN), Kings College, London, in collaboration with the Department
50 of Computer Science, University of Sheffield, to develop NLP and information
extraction solutions for mental health research needs. The Clinical Record Inter-
active Search (CRIS) application, developed at the South London and Maudsley
mental health trust (SLaM) and Kings College London, was internationally one
of the first platforms for mental health records analysis, providing researchers
55 access to de-identified data from the full electronic health record within a ro-
bust governance framework [11, 12]. CRIS data have supported over 80 research
papers and a number of funded research projects since the development of the
platform in 20081 . Examples of clinical variables that have been addressed with
NLP methods are shown in Figure 3.
60 The session then continued with a specific, and challenging, clinical use case
of preventing and predicting suicidal behaviour. Dr. Gergö Hadlaczky (Karolin-
ska Institutet, Stockholm) gave a presentation to provide some historical per-

1 http://www.maudsleybrc.nihr.ac.uk/facilities/clinical-record-interactive-search-cris/

cris-publications/

4
 Figure 3: Examples of NLP solutions for mental health documentation in CRIS, from Prof.
Robert Stewart’s presentation.

spectives as to how suicide risk and prediction have been defined and assessed,
and challenged these approaches by highlighting why there are problems with
65 risk assessment tools typically used in clinical practice. Prof. Enrique Baca-
Garcı́a (Fundación Jiménez Dı́az Hospital, Madrid) then described work he and
his colleagues have performed by using text and data mining approaches on
heterogeneous datasets to try to improve suicide risk prediction from different
perspectives, also including patient-generated data.
70 To initiate the afternoon discussion session, Dr. Rina Dutta (King’s College,
London), Dr. Sumithra Velupillai (KTH, Stockholm and King’s College, Lon-
don) and Dr. Johnny Downs (King’s College, London) described a use-case and
study on using NLP to identify adolescents with autism-spectrum disorders at
risk of suicidal behaviour [13]. This use-case specifically highlighted the prob-
75 lem of evaluation requirements from an NLP perspective versus complex clinical

5
 outcomes in the mental health domain.2

1.2. Moderated discussions

The workshop included ample time for focused discussions. The discussions
were moderated and divided into groups of about 6 participants per group, led by
80 a senior clinician and a senior informatician/computer scientist. Furthermore,
the participants were encouraged to join groups where there was either no or
minimal representation from their own institutions, in order to foster novel
discussions and new perspectives.
Broad discussion themes were provided as guidance for each discussion ses-
85 sion, based on the presentations that were given prior to discussions. The dis-
cussion sessions were dynamic, and academically and clinically engaging. The
opportunity to discuss these evaluation and methodology issues with researchers
from the clinical as well as the informatics fields was highly appreciated by all
participants, in particular in relation to scientific presentations that highlighted
90 different aspects of these problems, as identified by a post-workshop question-
naire. Broadly, the focus of each discussion session was to identify current
needs, including barriers and challenges, as well as to define opportunities and
actionable recommendations for the future from both sides in terms of deliver-
ing meaningful and repeatable results, and how to best use NLP methodology
95 in clinical research settings. We outline the main points that were raised with
respect to needs below, and elaborate on these in more detail in subsequent
sections.

• Datasets

– Access to datasets, in particular for evaluation:

100 Although progress has been made in making resources available, there
are still needs in this area, such as anonymized test/learning data,

2A forthcoming publication outlines and challenges suicide risk detection research in more
detail.

6
 annotations, resources for particular domains and specific clinical
research problems.

– Data quality:
105 Even when resources are made available, the quality of the data is
not always of sufficient standard, because of, for example, missing
data, missing documentation, or insufficient annotations.

– Influence of bias:
Information about potential inherent bias (e.g., in sample selection
110 or information bias) in existing as well as new resources need to be
made explicit and discussed in more detail. This issue is particu-
larly important for the NLP community, where bias issues have not
traditionally been addressed.

• Evaluation approaches

115 – Transparency and replicability:

Translation from evaluation approaches in computer science/NLP
to epidemiology to clinical outcomes research is not trivial — more
transparency and details about methodological steps and assump-
tions made in research studies are needed to ensure replicability and
120 generalizability.

– Communication and nomenclature:

Vocabulary and nomenclature also differ by research context — with
the increased efforts in interdisciplinary research of this type, there
is a need to ensure that transparency and clarity is ensured through
125 systematic reporting of methodological steps as well as evaluation
and reporting of results.

This workshop provided a unique opportunity to bring together researchers

from different disciplines to discuss crucial research questions related to evalu-
ation. A majority of the participants particularly appreciated the opportunity
130 to meet researchers from the other discipline, i.e., for informaticians to meet

7
 clinical researchers and vice versa. As a result of the discussions, an outline
of highlighted methodological aspects from a clinical as well as an informatics
perspective is provided in the following sections, to provide further insights and
guidance in current challenges and future opportunities.

135 Acknowledgements

This work is the result of an international workshop held at the Insti-

tute of Psychiatry, Psychology and Neuroscience, King’s College London, on
April 28th 2017, with financial support from the European Science Foundation
(ESF) Research Networking Programme Evaluating Information Access Sys-
140 tems: http://eliasnetwork.eu/. SV is supported by the Swedish Research
Council (2015-00359) and the Marie Sklodowska Curie Actions, Cofund, Project
INCA 600398. RD is funded by a Clinician Scientist Fellowship (research project
e-HOST-IT) from the Health Foundation in partnership with the Academy of
Medical Sciences. JD is supported by a Medical Research Council (MRC) Clin-
145 ical Research Training Fellowship (MR/L017105/1). KM is funded by the Well-
come Trust Seed Award in Science [109823/Z/15/Z]. RS is part-funded by the
National Institute for Health Research (NIHR) Biomedical Research Centre at
South London and Maudsley NHS Foundation Trust and King’s College Lon-
don. We wish to acknowledge contributions by the Data61/CSIRO Natural
150 Language Processing Team in general, and in particular, those by Adj/Prof Leif
Hanlen. David (PJ) Osborn, Joseph (F) Hayes and Anoop (D) Shah are sup-
ported by the National Institute for Health Research University College London
Hospitals Biomedical Research Centre. Prof Osborn is also in part supported
by the NIHR Collaboration for Leadership in Applied Health Research and Care
155 (CLAHRC) North Thames at Barts Health NHS Trust. AR is funded by the
European Unions Horizon 2020 research and innovation programme under grant
agreements No 644753 (KConnect) and No 654021 (OpenMinTeD)

8
 References

[1] L. Dawson, M. Johnson, H. Suominen, J. Basilakis, P. Sanchez, D. Estival,

160 B. Kelly, L. Hanlen, A usability framework for speech recognition technolo-
gies in clinical handover: a pre-implementation study., J Med Syst 38 (6)
(2014) 56.

[2] M. Johnson, P. Sanchez, H. Suominen, J. Basilakis, L. Dawson, B. Kelly,

L. Hanlen, Comparing nursing handover and documentation: forming one
165 set of patient information., Int Nurs Rev 61 (1) (2014) 73–81.

[3] M. Johnson, S. Lapkin, V. Long, P. Sanchez, H. Suominen, J. Basilakis,

L. Dawson, A systematic review of speech recognition technology in health
care., BMC Med Inform Decis Mak 14 (2014) 94.

[4] H. Suominen, M. Johnson, L. Zhou, P. Sanchez, R. Sirel, J. Basilakis,

170 L. Hanlen, D. Estival, L. Dawson, B. Kelly, Capturing patient information
at nursing shift changes: methodological evaluation of speech recognition
and information extraction., J Am Med Inform Assoc 22 (e1) (2015) e48–66.

[5] H. Suominen, L. Zhou, L. Hanlen, G. Ferraro, Benchmarking Clinical

Speech Recognition and Information Extraction: New Data, Methods, and
175 Evaluations, JMIR Med Inform 3 (2) (2015) e19. doi:10.2196/medinform.
4321.

[6] W. Scuba, M. Tharp, D. Mowery, E. Tseytlin, Y. Liu, F. A. Drews, W. W.

Chapman, Knowledge Author: facilitating user-driven, domain content de-
velopment to support clinical information extraction, Journal of Biomedical
180 Semantics 7 (1) (2016) 42. doi:10.1186/s13326-016-0086-9.

[7] K. Doing-Harris, D. L. Mowery, C. Daniels, W. W. Chapman, M. Con-

way, Understanding patient satisfaction with received healthcare services:
A natural language processing approach., AMIA Annu Symp Proc 2016
(2016) 524–533.

9
185 [8] S. Dublin, E. Baldwin, R. L. Walker, L. M. Christensen, P. J. Haug, M. L.
Jackson, J. C. Nelson, J. Ferraro, D. Carrell, W. W. Chapman, Natural
Language Processing to identify pneumonia from radiology reports., Phar-
macoepidemiol Drug Saf 22 (8) (2013) 834–841. doi:10.1002/pds.3418.

[9] D. S. Carrell, S. Halgrim, D.-T. Tran, D. S. M. Buist, J. Chubak, W. W.

190 Chapman, G. Savova, Using natural language processing to improve effi-
ciency of manual chart abstraction in research: the case of breast cancer
recurrence., Am J Epidemiol 179 (6) (2014) 749–758. doi:10.1093/aje/
kwt441.

[10] D. Mowery, D. Franc, S. Ashfaq, T. Zamora, E. Cheng, W. Chapman,

195 B. Chapman, Developing a knowledge base for detecting carotid stenosis
with pyConText, in: AMIA Annu Symp Proc., AMIA, Washington, DC,
USA, 2014.

[11] R. Stewart, M. Soremekun, G. Perera, M. Broadbent, F. Callard, M. Denis,

M. Hotopf, G. Thornicroft, S. Lovestone, The South London and Maudsley
200 NHS Foundation Trust Biomedical Research Centre (SLAM BRC) case
register: development and descriptive data., BMC Psychiatry 9 (2009) 51.

[12] G. Perera, M. Broadbent, F. Callard, C.-K. Chang, J. Downs, R. Dutta,

A. Fernandes, R. D. Hayes, M. Henderson, R. Jackson, A. Jewell, G. Kadra,
R. Little, M. Pritchard, H. Shetty, A. Tulloch, R. Stewart, Cohort profile
205 of the South London and Maudsley NHS Foundation Trust Biomedical
Research Centre (SLaM BRC) Case Register: current status and recent
enhancement of an Electronic Mental Health Record-derived data resource,
BMJ Open 6 (3). doi:10.1136/bmjopen-2015-008721.

[13] J. Downs, S. Velupillai, G. Gkotsis, R. Holden, M. Kikoler, H. Dean, A. Fer-

210 nandes, R. Dutta, Detection of Suicidality in Adolescents with Autism
Spectrum Disorders: Developing a Natural Language Processing Approach
for Use in Electronic Health Records, in: AMIA annual Symposium, AMIA,
2017, p. To appear.

10