International Journal of Medical Informatics 110 (2018) 90–97

Contents lists available at ScienceDirect

International Journal of Medical Informatics

journal homepage: www.elsevier.com/locate/ijmedinf

Clinician user involvement in the real world: Designing an electronic tool to T

improve interprofessional communication and collaboration in a hospital
setting
⁎
Terence Tanga,b, , Morgan E. Limc,d, Elizabeth Mansfieldc,e, Alexander McLachlanf,
Sherman D. Quanc
a
Institute for Better Health and Program of Medicine, Trillium Health Partners, 100 Queensway West, Clinical Administrative Building, 6th floor, Mississauga, Ontario,
L5B 1B8, Canada
b
Department of Medicine, University of Toronto, Toronto, Ontario, Canada
c
Institute for Better Health, Trillium Health Partners, Mississauga, Ontario, Canada
d
Institute for Health Policy, Management, and Evaluation, University of Toronto, Toronto, Ontario, Canada
e
Department of Occupational Science and Occupational Therapy, University of Toronto, Toronto, Ontario, Canada
f
Department of Family Medicine, McMaster University, Hamilton, Ontario, Canada

A R T I C L E I N F O A B S T R A C T

Keywords: Objectives: User involvement is vital to the success of health information technology implementation. However,
Software design involving clinician users effectively and meaningfully in complex healthcare organizations remains challenging.
User involvement The objective of this paper is to share our real-world experience of applying a variety of user involvement
Hospital communication system methods in the design and implementation of a clinical communication and collaboration platform aimed at
facilitating care of complex hospitalized patients by an interprofessional team of clinicians.
Methods: We designed and implemented an electronic clinical communication and collaboration platform in a
large community teaching hospital. The design team consisted of both technical and healthcare professionals.
Agile software development methodology was used to facilitate rapid iterative design and user input. We in-
volved clinician users at all stages of the development lifecycle using a variety of user-centered, user co-design,
and participatory design methods.
Results: Thirty-six software releases were delivered over 24 months. User involvement has resulted in im-
provement in user interface design, identification of software defects, creation of new modules that facilitated
workflow, and identification of necessary changes to the scope of the project early on.
Conclusion: A variety of user involvement methods were complementary and benefited the design and im-
plementation of a complex health IT solution. Combining these methods with agile software development
methodology can turn designs into functioning clinical system to support iterative improvement.

1. Introduction achieving this.

Although health information technology (HIT) has the potential to
improve healthcare through enhancing efficiency and safety [1], this
potential has not yet been fully realized [2]. Challenges to the success of
HIT are largely due to non-technical issues such as poor usability that
impact communication and workflow [3]. Therefore, HIT requires the
design of user friendly tools that are context appropriate [4]. Im-
plementation considerations including the impact on workflow must be
addressed early in the initial planning and design stages if HIT appli-
cations are to be successful [5]. User involvement is one approach for
Although there are multiple definitions of “user involvement” in the
literature [6,7], we use this term in general to describe “any direct
contact with users” [7].
There are three major strategies for involving users in HIT projects
and they lie on a continuum from least to most user control [8,9]. The
first strategy is user-centered design, where the designer actively studies
and understands the users’ perspectives and experiences to ensure the
product is useful and usable to them [8]. Examples include usability
testing and user observation [8]. Because the user is the object of study
rather than driving the design process, it is considered to have the least

⁎
Corresponding author at: Institute for Better Health and Program of Medicine, Trillium Health Partners, 100 Queensway West, Clinical Administrative Building, 6th floor, Mississauga,
Ontario, L5B 1B8, Canada.
E-mail addresses: terence.tang@utoronto.ca, terence.tang@thp.ca (T. Tang).

https://doi.org/10.1016/j.ijmedinf.2017.11.011
Received 12 July 2017; Received in revised form 31 October 2017; Accepted 19 November 2017
1386-5056/ © 2017 The Authors. Published by Elsevier Ireland Ltd. This is an open access article under the CC BY-NC-ND license
(http://creativecommons.org/licenses/BY-NC-ND/4.0/).
T. Tang et al.
user control. The second strategy is user co-design, where the designer
and users design the product together [8]. This gives the user more
control in the design process and examples of such techniques include
design meetings with prototypes and simulation [8]. The third major
strategy is participatory design, where the user is an active participant in
design and has a strong voice in decision-making, driving the innova-
tion while the designer facilitates the creative process [8,10,11]. Ex-
amples of this strategy include design games, CARD, PICTIVE, and the
third-generation participatory design method [8,11,12].
Reported benefits of user involvement in HIT projects include im-
proved system quality due to accurate user requirements gathering,
inclusion of useful features and exclusion of less useful features, higher
level of user acceptance and adoption during implementation, de-
creased training needs due to increased system understanding by users,
and higher level of participation by users in the organization [7,8,13].
Despite the reported benefits of user involvement in HIT develop-
ment, it remains complex and difficult to achieve in practice due to
social, cultural, organizational, and technical factors [14,15]. Other
challenges identified include lack of time for both users and the project
team, difficulty in obtaining representative users as clinicians have busy
schedules, disagreement between users and the inability to reach a
consensus, and the lack of necessary technical skills or knowledge by
users to effectively participate in the design process [7,8]. The chal-
lenges has led to some projects paying only “lip-service” to user in-
volvement, and one qualitative study concluded that physician users
participating in one HIT project were not active participants in decision
making, and were reduced to mere clinical consultants informing about
needs and requirements [15,16].
To increase the success of future HIT projects, research into user
involvement best practices in system development, and the dissemina-
tion of experiences of both successful and failed HIT projects are desired
by the health informatics community [3,8]. With this in mind, we de-
scribe our real-life experience with applying a variety of user involve-
ment methods in the design and development of a clinical commu-
nication and collaboration platform (called Care Connector) for
frontline clinicians.
2. Materials and methods
2.1. Setting
Trillium Health Partners (THP) is a large community teaching
hospital in Mississauga, Ontario, Canada, affiliated with the University
of Toronto. In 2016/2017, it operated 1252 in-patient beds, had 63,334
in-patient admissions, and 270,929 Emergency Department visits.
2.2. Care Connector
Effective communication and teamwork are increasingly recognized
as crucial to providing high quality, efficient, and safe care in health-
care organizations [17–21]. Care Connector is an interprofessional
(when 2 or more professions work together to improve collaboration
and quality of care [22]) clinical communication and collaboration
platform that facilitates coordinated care of complex hospitalized pa-
tients. Its users include physicians, nurses, and the allied health team
(physiotherapists, occupational therapists, dietitians, speech language
pathologists, pharmacists, and social workers). It was designed and
developed at THP to address limitations of existing communication
technology including lack of context [23], workflow interruptions
[24,25], reaching wrong recipients [26], privacy and security concerns
[27], and not being inclusive of all team members [28]. It is a web-
based application distinct from the primary hospital information system
(HIS). However, to integrate seamlessly into clinicians’ workflow, it
does retrieve information from and can write information to the pri-
mary HIS. This paper describes the user involvement methods used
during the design of this platform.
91
International Journal of Medical Informatics 110 (2018) 90–97
Care Connector’s architecture is modular, allowing each module to
be developed independently and address a different yet interconnected
clinical workflow. To date, these include the physician sign-out, pro-
gress note, interprofessional care planner, patient centered messaging,
patient flow planner, and electronic discharge summary modules
(Table 1). These modules are interconnected by the sharing of in-
formation. Examples include displaying a patient’s medical information
contained in the physician sign-out in the interprofessional care planner
so that the entire care team shares this understanding, and the ability
for the discharge summary module to pre-populate fields from the
physician sign-out module to reduce the need for repeated data entry or
the likelihood of missing information.
2.3. Approach to iterative design and development
We built Care Connector using Agile software development meth-
odology where usable software is delivered on a frequent basis so that
the product can be iteratively improved based on feedback from real
use [29,30]. The design team directly engaged clinician users to ensure
design decisions and software requirements fit the true needs of the
users and their clinical workflows. Using this approach, a new version
of software was released once every 2–4 weeks during active develop-
ment. The design team meets on a weekly basis to review feedback from
users, on-going development tasks, and prioritize development activ-
ities including bug fixes and new features. The design team is composed
of 1 project lead, 1 project manager, 2 developers, 1 business analyst, 1
physician lead, 1 nursing lead, and a learner. The physician and nursing
leads are users of the system themselves.
2.4. User involvement methodology
We used a variety of methods at different points to involve users in
the design and development of Care Connector as the design needs
arose. Due to rapid development timeline, we chose methods that were
feasible (balancing effective feedback with resources required). Table 2
summarizes the user involvement methods and the timing of use in the
Care Connector project.
2.4.1. Initial design of modules: user centered design, user co-design, and
participatory design
We used all 3 strategies (user-centered design, user co-design, and
participatory design) during the initial design of modules to ensure the
software truly met the needs of frontline clinicians. We observed users
(a user-centered design method) carrying out their daily work activities
to better understand the potential impact of an electronic tool on
clinical workflow. We shadowed physicians and nurses (weekdays,
weekends, and nights) as they performed their work. We took notes and
recorded the amount of time each spent on communication and doc-
umentation activities.
We also employed user co-design methods extensively during the
requirements gathering and the initial design phase of each module. For
each module, we held design meetings with relevant user groups (e.g.
physicians for the physician sign-out module, and nursing and allied
health staff for the interprofessional care planner). Invitations to these
meetings were open (and non-mandatory) and any interested in-
dividuals were welcome to attend. The initial objective was to confirm
and fully understand the problem to be solved from frontline clinicians’
perspectives. This then led to discussions for creating solutions tailored
for the clinicians and determining how the new solutions would in-
tegrate into their clinical workflow. Based on this knowledge, the de-
sign team built prototypes of the solution in the form of screen mockups
supported by workflow diagrams. The Balsamiq Mockups software [31]
was used to produce these prototypes (see example in Fig. 1). In sub-
sequent meetings, clinician users would provide feedback on these
prototypes and the design was iteratively improved upon. Prior to de-
ployment of a module, frontline clinicians worked collaboratively with
T. Tang et al. International Journal of Medical Informatics 110 (2018) 90–97

Table 1
Care Connector modules.

Module Description Value

Physician Sign-Out
Interprofessional Care
Planner
Patient-centered Messaging
Patient Flow Planner
Electronic Discharge
Summary
Progress Note
ED Intake Sheet
An electronic standardized template (based on IPASSa) to ensure the
timely and comprehensive transfer of patient information for patient
hand-over from one physician to another.
Facilitates nursing hand-over and provides a snapshot of a patient’s
current conditions (medical, social, functional) and plan of care from an
interprofessional perspective, allowing clinical team members to align
their care activities.
Secure method for communicating amongst the interprofessional team of
clinicians in an asynchronous, less disruptive fashion. All messages center
around the patient rather than the provider, so one-to-one care
conversations are recorded, archived, and accessible by the entire care
team.
Supports clinical operations to improve patient flow and discharge
planning. It facilitates communication across the interprofessional care
team to unlock barriers to discharge.
Replaces the process of dictating and transcribing the discharge summary.
The electronic discharge summary would become more interprofessional,
incorporating instructions from nursing and allied health. Since the
discharge summary would be completed prior to discharge, a patient-
friendly version can also be generated based on the Patient Oriented
Discharge Summary (PODS)b project.
Allows physicians and allied health staff to produce daily progress notes
while updating information in the Physician Sign-out Tool or the
Interprofessional Care Planner.
Allows the on-call physician in the Emergency Department a snapshot
view of the census of all in-patient teams and to distribute admitted
patients according to workload.
1) Improves information continuity to facilitate safe, coordinated and
clinically coherent care from one physician to another
2) Improves workflow efficiency
3) Addresses privacy and security issues with alternative handover
methods (e.g., use of personal emails, text messages etc.)
1) Improves shared understanding of the patient’s comprehensive
care needs (medical, social, functional) by the interprofessional
care team. This may lead to:
a.) improved quality and efficiency of care (more timely
arrangement of needed tests/services)
b.) improved teamwork
2) Reduction of unnecessary interruptions for team members, improving
workflow efficiency
3) Addresses privacy and security concerns with alternate methods of
communication e.g. SMS text messages
1) Focuses care team on a goal for discharge (estimated date of
discharge)
2) Prioritizes patients and provides more predictability around
discharges for bed management and planning
1 Is a living document so information can be updated at any time
during the admission, ensuring details are not lost over time.
2 More timely completion of discharge summary
3 Reduction in transcription costs
4 Patient is able to have a patient friendly copy at the time of discharge
1) Improves workflow efficiency by eliminating the need to double
document and reducing the amount of time it takes to document
2) Improves documentation and coding as information is more complete
and legible
1) Improves communication between admitting physicians and in-
patient physicians of newly admitted patients

a
Starmer AJ, Spector ND, Srivastava R, Allen AD, Landrigan CP, Sectish TC, et al. I-pass, a mnemonic to standardize verbal handoffs. Pediatrics. 2012;129(2) :201-4.
b
Hahn-Goldberg S, Okrainec K, Huynh T, Zahr N, Abrams H. Co-creating patient-oriented discharge instructions with patients, caregivers, and healthcare providers. J Hosp Med.
2015;10(12):804-7.

the design team to agree on an implementation plan.
Embracing participatory design principles, we embedded two clin-
ician users (the physician and nursing leads) as an integral part of the
design team. They participated in weekly design team meetings and had
a strong voice in decision making by the team. They were also able to
facilitate a close relationship between the design team and frontline
clinician users, often bringing back on-the-ground feedback and in-
sights into the impact of our modules on workflow both during the
design and implementation phases.
problems [32]. The users were asked to perform 5 representative tasks
with the physician sign-out module while being observed and timed.
They were also asked to think aloud and to provide comments during
the exercise [33]. An observer noted usability issues (e.g. issues with
navigation, unclear labels) during these tasks and recorded comments
and feedback from users. Amount of time required to complete each
task was also recorded. Usability issues with the user interface and
workflow issues were identified so the design team could use this
feedback to improve subsequent releases.

2.4.2. Usability testing: user-centered design
Following the deployment of the first module (the physician sign-
out), we conducted usability testing (a user-centered design method) with
5 physicians recruited using a convenience sampling strategy. A sample
size of 5 users was previously shown to discover 55–99% of usability
2.4.3. Feedback and iterative improvement: user co-design and
participatory design
To gather feedback and generate ideas for improvement, we continued
to hold design meetings (a user co-design method) after each module was
released in clinical use. We continuously received feedback through both

Table 2
User involvement methods during design of Care Connector.

Design Phase User Involvement Method Major Results

Initial design of modules
Usability Testing
Feedback and iterative
improvement
User-centered design: user Significant change to scope and use of Patient Centered Messaging module to initially target non-
observation urgent messages only due to observation of bed-side nurses’ workflow and their communication needs
User co-design: design meeting with Identified additional modules (Progress Note and ED Intake Sheet) that are time-saving features that
prototyping would facilitate user adoption.
To meet the needs of the clinical team, designed an Interprofessional Care Planner (rather than
Nursing Handover tool as initially planned).
Participatory design Facilitated design meetings and initial design of modules.
User-centered design: usability Identified usability issues with user interface, leading to improvement in subsequent versions.
testing
User co-design Identified workflow enhancements.
Participatory design Identified new features/enhancements through actual use of software as it relates to clinical workflow.
Facilitated close relationship with users.

92
T. Tang et al. International Journal of Medical Informatics 110 (2018) 90–97

Fig. 1. Example of low fidelity prototypes used in design meetings.

formal (in design meetings, questionnaires, helpdesk tickets) and informal
(through hallway conversations and user initiated emails to members of
the design team) means. The design team took note of feedback from all
sources during weekly team meetings and acted upon them when appro-
priate. The embedded physician and nursing leads (participatory design)
facilitated this close relationship with users and continued to have a strong
voice in decision making to guide the design team in making improvement
decisions throughout the project. This cycle of deploying then obtaining
feedback then deploying again quickly continued throughout the project,
with changes being released every 2–4 weeks.

93
T. Tang et al. International Journal of Medical Informatics 110 (2018) 90–97

Fig. 2. Different versions of the same screen of Care Connector. Image on the top is the earlier version, while image on the bottom is the current version. A. The Patient List. This is the
view that clinicians use to see a list of own patients. It displays the patient’s demographic information, location, diagnosis, and other administrative information. B. The Physician Sign
Out. This view is where physicians enter information that he/she wants to handover to a covering physician or next physician who would assume care.

3. Results infrequently during observations, with physicians leading in this prac-

Care Connector’s first module (Physician Sign-Out) was released in
January 2015 after engaging with Hospitalist physicians who identified
a need for better physician hand-over to improve clinical workflow,
efficiency, and patient safety. Since then, we have engaged with the
entire interprofessional team involved in the care of complex hospita-
lized patients to design solutions addressing communication and col-
laboration needs. There were 36 software releases while the system was
in real clinical use over the first 2 years (January 2015 to January
2017). Here, we report on how our user involvement experience has
influenced the design of Care Connector. Table 2 summarizes the major
results described in the following subsections.
tice (31 emails and 125 text messages). In addition, observation showed
that a web-based application for communication might not fit bedside
nurses’ workflow as they are busy and rarely sit in front of a computer.
When they did require assistance, the need was often time sensitive,
making urgent paging and phone calls more appropriate. Based on
these results and clinician input, the design team changed the scope of
the patient centered messaging module to initially target only non-ur-
gent messages, while allowing urgent messages to be communicated in
the current form (paging and phone calls). This could potentially reduce
interruptions to clinicians by diverting non-urgent messages away from
paging and phone calls, while providing asynchronous secured com-
munication channels for the entire team.

3.1. Initial design of modules: user observation (user-centered design)
User observation generated significant insights for the design of the
Patient-Centered Messaging module. Trained observers shadowed
nurses and physicians for 282 and 154 hours respectively. Both nurses
and physicians spent significant time on communication (16% for
nurses, 25% for physicians) and documentation (21% for nurses, 33%
for physicians). However, electronic communication happened
3.2. Initial design of modules: design meetings with prototyping (user co-
design)
Design meetings with user groups identified features that would be
required to facilitate clinical workflow. During the development of the
physician sign-out module, physicians identified two modules (ED
Intake Sheet and Progress Note) that were needed to facilitate clinical
workflow. The ED Intake Sheet allows admitted patients to be added to

94
T. Tang et al. International Journal of Medical Informatics 110 (2018) 90–97

Fig. 2. (continued)

the system, and the Progress Note allows documentation and updating physicians, making adoption more difficult.
of the Sign-Out at the same time. Without these modules, the use of the Design meetings with nursing and allied health staff had a sig-
physician sign-out would have increased the workload burden on nificant impact in changing the fundamental design of the tool. We

95
T. Tang et al.
initially set out to build a nursing handover tool. However, during these
meetings with clinicians, it was clear that building an interprofessional
care planner would best support team communication around a pa-
tient’s overall care plan and status. Ultimately, the care planner was
able to support nursing handover as well. The design team worked with
clinicians to define the information to be included in the care planner
and how to pull data from existing systems to avoid duplication of
work. Clinicians also indicated that updating the interprofessional care
planner could increase workload from double documentation, so the
progress note feature was made available to them. The information that
clinicians documented as part of their daily progress note, a required
practice, could now be re-used within the interprofessional care
planner.
3.3. Usability testing (user centered design)
Usability testing has provided the design team with formal feedback
to improve the user interface. With the deployment of Care Connector’s
first module (physician sign-out), 5 physicians each performed 5 re-
presentative tasks. The average time taken to perform each task varied
from 18 to 118 seconds, with considerable variability between users.
Usability testing identified issues including unclear meaning of ab-
breviations and acronyms (e.g. “w.e.” for weekend), unclear location of
items (cannot find “see on weekend” checkbox), and navigation chal-
lenges (scrolling required to find the team menu). Improvements were
made as a result. Fig. 2 shows early versions of the Patient List and
Physician Sign-Out, as compared to the current version.
3.4. Feedback and iterative improvement: participatory design
Our physician and nursing leads helped facilitate continuous in-
volvement of users. This involvement guided many decisions regarding
the direction of the project and on multiple occasions identified features
that would enhance clinical workflow. Examples include adding func-
tionality to the ED intake sheet module to prevent extra admissions to
teaching teams, systematically generating admission emails as safe-
guards to ensure no admissions were missed, and the ability for nursing
staff to print the care planner for multiple patients from one screen.
3.5. System use and reach
The use of Care Connector is voluntary and not mandated by the
design team or hospital leadership. System use statistics showed that
Care Connector had more than 300 unique users spanning the entire
interprofessional team, facilitated physician handover for 13,687 hos-
pitalizations, and generated 27,122 progress notes over 2 years.
4. Discussion
User involvement in HIT can be difficult to achieve in practice, and
many practical implementation details remain areas of active research
[8,15]. We shared our real-life experience of applying user-centered
design, user co-design, and participatory design methods in supporting
the development and implementation of Care Connector with frontline
clinicians. User-centered design was effective in informing the design of
new modules, but was inefficient in obtaining iterative feedback for
enhancements due to resource intensiveness (user observation) and
difficulty of scheduling with busy clinicians (both usability testing and
user observation) to obtain timely feedback necessary for rapid agile
software development cycles. This gap was well addressed by the
complementary methods of participatory design and user co-design.
Using participatory design, our physician and nursing leads were em-
bedded as part of the core design team and played a key role in design
decision making. Their participation and insights allowed us to build
functionality that met the needs of users efficiently and rapidly. To
ensure the varied perspectives of the broader user group is represented
96
International Journal of Medical Informatics 110 (2018) 90–97
beyond the two embedded clinicians, we used user co-design methods
(such as design meetings) to engage a large number of users. Using
participatory design in tandem with user co-design enabled timely
feeedback to support agile software development cycles. This was cri-
tical to the success of the project where adoption was easily achieved
and feedback suggested that it is a highly functional, practical, and user
friendly application. Using a variety of user involvement methods was
complementary and strengthened our design.
Drawing on others’ experiences [7,8,11,12] and reflecting on our
own, we recognize challenges with user involvement. Pollack and col-
leagues described applying participatory design with physicians and
noted that relatively few reports involved clinicians [12]. Pilemalm and
Timpka recognized additional challenges with designing large-scale
systems for a heterogeneous group of users (such as an interprofessional
team) and adapted traditional participatory design principles to pro-
pose the third-generation participatory design method [11]. While
participatory design techniques produce designs that suit the needs and
workflows of users, there is still a gap in turning these designs into
actual implementations [11,12]. Designs remaining too abstract and
lack of design knowledge by users were also challenging [11,12]. In this
current report, we demonstrated that user involvement methods are
complementary and can be effectively applied to a heterogenous group
of clinician users. Using prototypes made early designs less abstract and
allowed users to participate in the process. Finally, combining agile
software development methodology with user involvement can turn
designs and ideas into a functional system and achieve iterative im-
provement in a busy clinical setting.
5. Conclusion
Though there are challenges, we demonstrated that using different
user involvement methods combined with agile software development
methodology is feasible and effective for engaging clinicians in a real-
world healthcare environment for developing complex HIT systems.
Further research into the best practices of achieving user involvement
in the design of complex HIT systems would increase the success of
future HIT projects.
Funding
Funding for the design, development, and implementation of Care
Connector is provided by Trillium Health Partners.
Competing interests
The Care Connector project was funded by, developed at, and
owned by Trillium Health Partners (THP). TT, ML, EM, SQ are staff at
THP. TT, ML, EM, and SQ are also investigators in the mixed methods
evaluation of Care Connector, which has received research funding
from Canadian Medical Protective Association (CMPA), and Physician
Services Incorporated (PSI) Foundation.
Acknowledgements
We would like to acknowledge the clinicians of Trillium Health
Partners for participating in this project, the technical team (Rafael
Cunhar, Jason Percival, Janice Do Rego, Arif Siddiqui, Ellen Roberto,
Andrea Coburn, Emanuel Lucez) for successfully delivering the project,
Dr. Robert Reid for guidance on this manuscript, and Dr. Dante Morra
for conception of the project.
References
[1] R. Hillestad, J. Bigelow, A. Bower, F. Girosi, R. Meili, R. Scoville, et al., Can elec-
tronic medical record systems transform health care? Potential health benefits,
savings, and costs, Health Aff. (Millwood) 24 (5) (2005) 1103–1117.
T. Tang et al.
[2] A.L. Kellermann, S.S. Jones, What it will take to achieve the as-yet-unfulfilled
promises of health information technology, Health Aff. (Millwood) 32 (1) (2013)
63–68.
[3] B. Kaplan, K.D. Harris-Salamone, Health IT success and failure: recommendations
from literature and an AMIA workshop, J. Am. Med. Inf. Assoc. 16 (3) (2009)
291–299.
[4] M.C. Beuscart-Zéphir, P. Elkin, S. Pelayo, R. Beuscart, The human factors en-
gineering approach to biomedical informatics projects: state of the art, results,
benefits and challenges, Yearb. Med. Inf. 10 (2007) 9–27.
[5] K. Cresswell, A. Sheikh, Organizational issues in the implementation and adoption
of health information technology innovations: an interpretative review, Int. J. Med.
Inf. 82 (5) (2013) e73–86.
[6] M. Bano, D. Zowghi, User involvement in software development and system success:
a systematic literature review, Proceedings of the 17th International Conference on
Evaluation and Assessment in Software Engineering: EASE, Apr 14–16, Porto de
Galinhas, Brazil2013. p. 125., 2013.
[7] S. Kujala, User involvement: a review of the benefits and challenges, Behav. Inf.
Technol. 22 (1) (2003) 1–16.
[8] A. Kushniruk, C. Nøhr, Participatory design, user involvement and health IT eva-
luation, Stud. Health Technol. Inform. 222 (2016) 139–151.
[9] M.J. Muller, S. Kuhn, Participatory design, Commun. ACM 36 (6) (1993) 24–28.
[10] J. Clemensen, S.B. Larsen, M. Kyng, M. Kirkevold, Participatory design in health
sciences: using cooperative experimental methods in developing health services and
computer technology, Qual. Health Res. 17 (1) (2007) 122–130.
[11] S. Pilemalm, T. Timpka, Third generation participatory design in health in-
formatics–making user participation applicable to large-scale information system
projects, J. Biomed. Inform. 41 (2) (2008) 327–339.
[12] A.H. Pollack, A. Miller, S.R. Mishra, W. Pratt, PD-atricians: leveraging physicians
and participatory design to develop novel clinical information tools, AMIA Annual
Symposium Proceedings/AMIA Symposium AMIA Symposium. 2016 (2016)
1030–1039.
[13] M.L. Bernstein, T. McCreless, M.J. Cote, Five constants of information technology
adoption in healthcare, Hosp. Top. 85 (1) (2007) 17–25.
[14] K. Cresswell, Z. Morrison, S. Crowe, A. Robertson, A. Sheikh, Anything but engaged:
user involvement in the context of a national electronic health record im-
plementation, Inform. Prim. Care 19 (4) (2011) 191–206.
[15] A.W. Kushniruk, P. Turner, Who's users? Participation and empowerment in socio-
technical approaches to health IT developments, Stud. Health Technol. Inform. 164
(2011) 280–285.
[16] A.M. Høstgaard, P. Bertelsen, C. Nøhr, Methods to identify, study and understand
end-user participation in HIT development, BMC Med. Inform. Decis. Mak. 11
(2011) 57.
[17] G.R. Baker, P.G. Norton, V. Flintoft, R. Blais, A. Brown, J. Cox, et al., The Canadian
97
International Journal of Medical Informatics 110 (2018) 90–97
Adverse Events Study: the incidence of adverse events among hospital patients in
Canada, CMAJ 170 (11) (2004) 1678–1686.
[18] A. Lee, P.D. Mills, J. Neily, R.R. Hemphill, Root cause analysis of serious adverse
events among older patients in the Veterans Health Administration, Joint Comm.
Res. 40 (6) (2014) 253–262.
[19] L. Lemieux-Charles, W.L. McGuire, What do we know about health care team ef-
fectiveness? A review of the literature, Med. Care Res. Rev. 63 (3) (2006) 263–300.
[20] B. Okoniewska, M.J. Santana, H. Groshaus, S. Stajkovic, J. Cowles, D. Chakrovorty,
et al., Barriers to discharge in an acute care medical teaching unit: a qualitative
analysis of health providers' perceptions, J. Multidiscip. Healthc. 8 (2015) 83–89.
[21] R.M. Wilson, B.T. Harrison, R.W. Gibberd, J.D. Hamilton, An analysis of the causes
of adverse events from the Quality in Australian Health Care Study, Med. J. Aust.
170 (9) (1999) 411–415.
[22] S. Reeves, J. Goldman, J. Gilbert, J. Tepper, I. Silver, E. Suter, et al., A scoping
review to improve conceptual clarity of interprofessional interventions, J. Interprof.
Care 25 (3) (2011) 167–174.
[23] B.M. Wong, S. Quan, S. Shadowitz, E. Etchells, Implementationn and evaluation of
an alpha-numeric paging system on a resident inpatient teaching service, J. Hosp.
Med. 4 (8) (2009) E34–E40.
[24] M. Weigl, A. Müller, A. Zupanc, J. Glaser, P. Angerer, Hospital doctors' workflow
interruptions and activities: an observation study, BMJ Qual. Saf. 20 (6) (2011)
491–497.
[25] M. Woloshynowych, R. Davis, R. Brown, C. Vincent, Communication patterns in a
UK emergency department, Ann. Emerg. Med. 50 (4) (2007) 407–413.
[26] B.M. Wong, S. Quan, C.M. Cheung, D. Morra, P.G. Rossos, K. Sivjee, et al.,
Frequency and clinical importance of pages sent to the wrong physician, Arch.
Intern. Med. 169 (11) (2009) 1072–1073.
[27] A.H. Greene, HIPAA compliance for clinician texting, J. AHIMA 83 (4) (2012)
34–36.
[28] R. Wu, V. Lo, D. Morra, E. Appel, T. Arany, B. Curiale, et al., A smartphone-enabled
communication system to improve hospital communication: usage and perceptions
of medical trainees and nurses on general internal medicine wards, J. Hosp. Med. 10
(2) (2015) 83–89.
[29] K. Beck, M. Beedle, A. van Bennekum, Principles, 2001. Available from: http://
www.agilemanifesto.org/principles.html.
[30] K. Beck, M. Beedle, A. van Bennekum, Software Development, 2001. Available
from: http://www.agilemanifesto.org/.
[31] B. Studios, Balsamiq Mockups, 2015.
[32] L. Faulkner, Beyond the five-user assumption: benefits of increased sample sizes in
usability testing, Behav. Res. Methods Instrum. Comput. 35 (3) (2003) 379–383.
[33] A.W. Kushniruk, E.M. Borycki, Integrating low-cost rapid usability testing into agile
system development of healthcare IT: a methodological perspective, Stud. Health
Technol. Inform. 210 (2015) 200–204.