This article appeared in a journal published by Elsevier.

The attached
copy is furnished to the author for internal non-commercial research
and education use, including for instruction at the authors institution
and sharing with colleagues.
Other uses, including reproduction and distribution, or selling or
licensing copies, or posting to personal, institutional or third party
websites are prohibited.
In most cases authors are permitted to post their version of the
article (e.g. in Word or Tex form) to their personal website or
institutional repository. Authors requiring further information
regarding Elsevier’s archiving and manuscript policies are
encouraged to visit:
http://www.elsevier.com/copyright
 Author's personal copy

i n t e r n a t i o n a l j o u r n a l o f m e d i c a l i n f o r m a t i c s 8 0 ( 2 0 1 1 ) 782–792

journal homepage: www.ijmijournal.com

Does the introduction of an electronic nursing

documentation system in a nursing home reduce
time on documentation for the nursing staff?

Esther N. Munyisia, Ping Yu ∗ , David Hailey

School of Information Systems and Technology, The University of Wollongong, NSW 2522, Australia

a r t i c l e i n f o a b s t r a c t

Article history: Purpose: To determine whether the introduction of an electronic nursing documentation
Received 16 January 2011 system in a nursing home reduces the proportion of time nursing staff spend on documen-
Received in revised form tation, and to use this information in evaluating the usefulness of the system in improving
9 August 2011 the work of nursing staff.
Accepted 24 August 2011 Methods: An observational work sampling study was conducted in 2009 and 2010, 2 months
before, and 3, 6 and 12 months after the introduction of an electronic nursing documentation
system. An observer (ENM) used a work classification tool to record documentation activities
Keywords: being performed using paper and with a computer by nursing staff at particular times for
Documentation periods of 5 days.
Efficiency Results: Three hundred and eighty three (383) activities were recorded before implementation
Electronic nursing documentation of the electronic system, 472 activities at 3 months, 502 at 6 months, and 338 at 12 months
Electronic health record after implementation. There was no significant difference between the proportion of time
Long-term care nursing staff spent on documentation 2 months before and 3 months after the implemen-
Nursing home tation of the electronic system. Six months after implementation, the proportion of time on
Nurse documentation increased significantly and after 12 months, settled back to original levels
Observation that were recorded in the paper-based system. Over half of the proportion of time on doc-
Longitudinal study umentation at 6 and 12 months after implementation was spent on paper documentation
tasks.
Conclusion: Introduction of an electronic nursing documentation system did not reduce the
proportion of time nursing staff spent on documentation. This may in part have been a
result of the practice of documenting some information items on paper and others on a
computer. To reduce the use of paper documentation or to achieve a paper-free documenta-
tion environment in this setting, an in-depth understanding of nursing staff’s information
needs, and documentation workflow is necessary.
© 2011 Elsevier Ireland Ltd. All rights reserved.

access to health and aged care services. Information and com-

1. Introduction munication technology (ICT) holds promise for addressing
these challenges [1,2]. The use of modern ICT in healthcare
The healthcare sector in various countries is today faced provides a tremendous opportunity for improved deliv-
with issues surrounding quality, safety, efficiency, cost, and ery of services through increased efficiency, reduced costs,

∗
Corresponding author. Tel.: +61 242215412; fax: +61 242214045.
E-mail address: ping@uow.edu.au (P. Yu).
1386-5056/$ – see front matter © 2011 Elsevier Ireland Ltd. All rights reserved.
doi:10.1016/j.ijmedinf.2011.08.009
 Author's personal copy
i n t e r n a t i o n a l j o u r n a l o f m e d i c a l i n f o r m a t i c s 8 0 ( 2 0 1 1 ) 782–792
equitable health, and better care outcomes [3,4]. Hopes for
such positive outcomes have motivated the development and
implementation of electronic systems into healthcare settings
[5,6].
In recent years in Australia, aged care organizations have
introduced electronic documentation systems into nursing
homes in an effort to reduce documentation load for the nurs-
ing staff [7,8], and allow them more time to care for seniors.
These initiatives are also aimed at improving the quality of
residents’ records in terms of legibility, completeness, and
accuracy. Although there is evidence to suggest that electronic
nursing documentation systems can improve quality of the
residents’ records [9,10], there is a lack of evidence that such
systems can reduce documentation efforts and time for the
nursing staff in aged care facilities.
Several studies evaluating the relationship between the
introduction of an electronic system and documentation
efficiency have been undertaken in hospitals and their
findings have varied. Some studies [11,12] have found a
reduction in time, for example, an observation of nursing
staff using either paper or electronic documentation sys-
tem in the intensive care unit (ICU) of a surgical ward
found a significant reduction in charting time, 6 months after
implementation of the electronic system [11]. In the same
year (2003) and setting, Bosman et al. [12] found a signifi-
cant reduction in documentation time for the nursing staff
after 7 months of using a computerized system to register
patients.
However, some studies [13,14] have found an increase in
documentation time. Saarinen and Aho [13] found nurses took
longer to document care using an electronic system than with
a paper-based system, 2 years after implementation of the
electronic system. In a randomized evaluation of a comput-
erized system in a psychiatric ward, Ammenwerth et al. [14]
found a significant increase in time on report writing and
documentation of tasks, 7 weeks after implementation of the
automated system.
Other studies [15,16] have found no changes in documen-
tation time for the nursing staff. For example, Menke et al.
[15] found no significant difference between manual and elec-
tronic documentation time, after the electronic system had
been in use for 3 months in the ICU of a paediatric ward.
An observation of nursing staff using either manual or elec-
tronic documentation systems in a surgical ward also found
no significant difference in time on admission and routine
documentation of care, 1 year after the introduction of the
electronic system [16].
Thus, it is unknown whether investment in an electronic
documentation system in a nursing home will reduce the pro-
portion of time nursing staff spend on documentation, and
allow them to spend more time on residents’ care [17]. This
information is important in evaluating the usefulness of such
systems in the work of nursing staff. In addition, the infor-
mation is necessary in motivating nursing staff to adopt ICT
solutions, and in encouraging the aged care sector to invest
in ICT innovations. Therefore, the motivation of this study
was to measure the effect of the introduction of an electronic
nursing documentation system in a nursing home, on the
proportion of time spend on documentation by the nursing
staff.
783
2. Study context
2.1. Organizational setting
This work was carried out at an Australian nursing home
between March 2009 and November 2010. The study is part
of a larger investigation of the impact of an electronic docu-
mentation system on nursing staff activities in a residential
aged care facility.
2.2. System details
A commercial Web-based electronic documentation system
was implemented in May 2009. The system is used for
residents’ demographic information, assessments, progress
notes, and residents’ forms and charts. It is also used for
incident and accident reports, care plans, funding of care,
administrative and 24-h shift handover reports. Use of the 24-
h report was introduced at the facility from 6 months after
the implementation of the electronic system. The system is
designed to automatically integrate information entered on
forms, charts and progress notes into nursing care plans, cal-
culation of funding and management reports.
The approach taken by the nursing home was to continue
to use paper documentation for some types of informa-
tion after implementation of the electronic system. These
included information on medication, activities of daily living
(ADL), summary shift handover report, and recreational activ-
ities. Memory aid notation, scheduled tasks, and awareness
information were also captured on paper (Table 1). Conti-
nence information was documented and stored on paper
for 3 months after the introduction of the electronic sys-
tem, after which such information was entered and stored
electronically.
2.3. Training of staff and system setup
Training sessions were held 3 months before the introduction
of the electronic system. Each nursing staff member received a
30 min one-on-one training session. Subsequently, the newly
employed nursing staff learned how to use the system from
their peers with experience.
The system was installed in six desk-top computers. A
username and password was assigned to each nursing staff
member. Data entry into the system was through a combi-
nation of text using a keyboard, and ‘drop and click’ method
using a structured drop-down menu.
2.4. Implementation of the system
The electronic system was implemented in three phases.
Phase I was the introduction of progress notes in May 2009.
Phase 2 followed 1 month later with the introduction of charts
and forms, such as blood glucose chart and restraint chart.
Phase 3 was the introduction of the nursing care plan in July
2009.
 Author's personal copy

784 i n t e r n a t i o n a l j o u r n a l o f m e d i c a l i n f o r m a t i c s 8 0 ( 2 0 1 1 ) 782–792

Table 1 – The types of information documented in computer or on paper after implementation of the electronic system.
Type of information 3 Months after 6 Months after 12 Months after
implementation implementation implementation
Resident demographic details Computer Computer Computer
Assessments Computer Computer Computer
Progress notes Computer Computer Computer
Resident forms and charts (excluding continence chart) Computer Computer Computer
Incident and accident reports Computer Computer Computer
Care plan Computer Computer Computer
Funding of care Computer Computer Computer
Administrative shift handover report Computer Computer Computer
a
24-h shift handover report Computer Computer
Medication Paper Paper Paper
Activities of daily living (ADL) Paper Paper Paper
Summary shift handover report Paper Paper Paper
Recreational activities Paper Paper Paper
Memory aid notation Paper Paper Paper
Scheduled tasks and awareness Paper Paper Paper
Continence Paper Computer Computer
a
The report was not part of the information documented by nursing staff at this period of study.

2.5. Nursing staff and their role in documentation whenever they were involved in direct care of residents, such
duties as feeding.

2.5.1. Registered Nurses and Endorsed Enrolled Nurses

Registered Nurses (RNs) and Endorsed Enrolled Nurses (EENs)
are professional nursing staff in Australia’s healthcare sys-
tem [18,19]. These staff members have differing training and
responsibilities. The RNs need to complete a 3 year baccalau-
reate program in a university, whereas EENs undertake an
18 months program conducted in a vocational training centre
such as College of Technical and Further Education and then
complete additional training on medication management. The
RNs are the team leaders in a work shift, and the EENs work
under RNs’ direction and supervision.
The RNs and the EENs were responsible for medication doc-
umentation, preparing shift handover reports, developing and
updating residents’ care plans, and completing information on
funding of care. However, administration and documentation
of drugs of addiction was fully the responsibility of RNs. With
the introduction of the electronic system, most documenta-
tion was automatically performed as indicated in Section 2.2
(system details).
2.5.2. Personal Carers
Personal Carers (PCs) in Australian nursing homes provide
basic care to the residents, for example, showering. They work
under the direction and supervision of a registered or enrolled
nurse. The PCs were responsible for writing progress notes
and completing charts and forms using the electronic system.
They completed most of this documentation as they provide
most direct care services in a nursing home [20].
2.5.3. Recreational activity officers
Recreational activity officers (RAOs) are responsible of plan-
ning, implementing and evaluating leisure and recreational
programs for individual residents [21]. They work under super-
vision of a residential service manager. The RAOs manually
charted the involvement of each resident in recreational activ-
ities. However, they documented in the electronic system
3. Methods
3.1. Study design
This study was carried out using work sampling with an obser-
vational component. This technique was first developed by
Tippett in 1935 for use in industrial engineering and manage-
ment [22]. It requires a trained observer or a team of observers
to use a pre-defined classification of activities in recording the
specific activity being undertaken at a particular time, based
on pre-defined or randomly selected time intervals.
3.2. Study flow
Following approaches taken by other researchers [23,24], the
current investigation was conducted in four separate peri-
ods. The first period was 2 months before the introduction of
an electronic documentation system. The second, third and
fourth periods were at 3, 6 and 12 months after implementa-
tion of the system, respectively (Fig. 1). These periods of study
represent the different stages of electronic system implemen-
tation [25]: the learning stage (after 3 months), early use (after
6 months) and when the system is fully integrated into routine
practice (after 12 months). Each period of data collection was
conducted in a day shift (6.45 am to 3.15 pm) and lasted 5 days
(Monday, Tuesday, Wednesday, Saturday and Sunday).
3.3. Participants
All nursing staff working in a day shift including RNs, EENs,
PCs and RAOs agreed to participate in the study. On a typical
day shift, staff on the floor comprised one RN, one EEN, 12
PCs and one RAO. Nursing staff working on afternoon or night
shifts were excluded from the investigation as well as staff on
orientation, those on stand-by assisting with activities on the
 Author's personal copy
i n t e r n a t i o n a l j o u r n a l o f m e d i c a l i n f o r m a t i c s 8 0 ( 2 0 1 1 ) 782–792
Data collection 2 Data collection 3
months before the months after the
introduction of the introduction of the
electronic system electronic system
1st period (March 2009) May-July 2009 2nd period (August 2009) 3rd period (November 2009) 4th period (May 2010)
• Intervention
(Introduction of the
electronic nursing
documentation system)
Fig. 1 – The four periods of data collection.
floor, temporary employees from an employment agency and
RNs assigned to administrative tasks.
3.4. Documentation activities
All possible documentation activities for the nursing staff,
including paper-based and computer-based activities were
identified from previously published instruments [12,26,27].
Details of procedures followed in validating these activities,
including assessment of inter-rater reliability, are described
by Munyisia et al. [28]. The resulting activities included taking
records from the storage place, flipping through to identify the
correct page, reviewing resident information/reading notes,
writing progress notes/charts and forms, putting records back
to the filing area, medication documentation, admission docu-
mentation, documentation to transport a resident to hospital,
locating the correct window, inputting a username and pass-
word, typing progress notes/charts and forms, and closing the
electronic system.
3.5. Observer
Observations were made by a Ph.D. candidate (ENM) who
is an experienced observer and with practical knowledge of
residential aged care work. These characteristics helped the
observer to identify and record documentation activities.
3.6. Study procedures
Before the first period of data collection, the observer visited
the facility on two separate days and was introduced by the
residential service manager to the nursing staff in their han-
dover shift meeting. The observer talked to the nursing staff
and reassured them that the study was not meant to seek
faults but to understand changes in the proportion of time
they spent on documentation before and after the introduc-
tion of an electronic documentation system. This meeting
enabled the observer to familiarize with the nursing staff,
and also for them to become comfortable with the observer,
which could potentially minimize the ‘Hawthorne effect’ and
785
Data collection 12
Data collection 6 months after the
months after the introduction of the
introduction of the electronic system
electronic system
Time
lead to an accurate recording of documentation activities for
staff. The observer also explained the purpose and method of
study to the nursing staff and invited them to participate in
the investigation. An information sheet about the study was
provided to each nursing staff member to read, understand
and ask questions, before consent to their participation was
sought.
During data collection, observations were made at an inter-
val of 9 min/h. The observer started making observations from
a fixed location in the nursing home. Following the same
route on each round of observation, all documentation activi-
ties being undertaken by each staff member were recorded.
Brief communication between the observer and a nursing
staff member to clarify a documentation activity being under-
taken was allowed when necessary. The observed activities
for the nursing staff were recorded on a tabular data col-
lection form using a unique code number allocated to each
task. The tabular form contained information about the day
and date of observation, and whether a nursing staff member
under observation was an RN, EEN, RAO or PC. A dash (–) was
used to denote a nursing staff member who was not observed
during a given round of observation. The study methods and
procedures remained the same for the four periods of data
collection.
3.7. Analysis
Data were entered into MS Excel 2003 spreadsheet and
exported to a Statistical Package for Social Sciences (SPSS) 17.0
for analysis. Any items with less than five data entries at a
specific measurement period were excluded from the analy-
sis. Unlike in time motion studies, where the exact time on
an activity is measured, in this study occurrences of docu-
mentation activities were recorded. For ease of reporting this
data, activity occurrences were converted into a percentage,
and the main outcome variable of time reported as ‘percent-
age of time’. This format of reporting work sampling data has
been validated by the previous studies [29,30].
The percentage of time spent on each documentation
activity was calculated using descriptive statistics with 90%
 Author's personal copy
786 i n t e r n a t i o n a l j o u r n a l o f m e d i c a l i n f o r m a t i c s 8 0 ( 2 0 1 1 ) 782–792
confidence intervals (CI). We chose to use this CI because of
our limited resources; a single observer (ENM) made obser-
vations for this study, unlike previous work sampling studies
[12,31] with two or more observers. This situation might have
reduced our sample size. Other researchers in healthcare have
also used this CI in reporting their results [32].
Any significant differences in proportions of time across
the four data points were identified using Pearson’s chi-square
test. Statistical significance was assumed when the p-value
was less than 0.10. If a significant difference was identified,
post hoc comparison of proportions of time between any two
of the four periods of study was conducted using Pearson’s
chi-square test.
We also used Pearson’s chi-square test to analyse the pro-
portion of time spent on paper-based and on computer-based
documentation tasks after the introduction of the electronic
system. The proportions of time spent on paper or on com-
puter documentation tasks across the three data points were
compared. If any significant difference was identified, post
hoc comparison of proportions of time between any two of
the three data points was conducted. The proportions of time
spent on paper-based and computer-based documentation
tasks at the same data point were also compared to determine
any differences amongst them.
4. Results
A total of 1695 documentation activities were recorded. Three
hundred and eighty three (383) activities were recorded before
the introduction of the electronic system, 472 at 3 months, 502
at 6 months, and 338 at 12 months after implementation of the
system. Forty seven (47) nursing staff were observed before the
introduction of the electronic system, 53 after 3 months, and
54, after both 6 and 12 months of using the electronic system
(Table 2). Most of the observed nursing staff were PCs (73.1%).
There was no significant variation amongst the four data col-
lection periods in the number of observed nursing staff in
different job roles.
The recorded numbers of documentation activities for the
RAOs were too few for analysis at all measurement periods.
Thus, they were excluded from the analysis by level of nursing
staff, but are included where the proportion of documentation
for entire nursing staff is reported.
4.1. The proportion of time spent on documentation
activities by nursing staff before and after the
introduction of electronic nursing documentation
Table 3 shows the overall results of the changes in propor-
tion of time on documentation activities before and after the
introduction of the electronic system. In general, there was no
significant variation in the proportion of time spent on docu-
mentation using the paper-based system and the proportion
of time spent 3 months after implementation of the electronic
system. Six months after implementation, the proportion
of time on documentation increased significantly compared
with the proportion recorded when the paper-based system
was used (p = 0.02). The proportion of time on documentation
returned to a level similar to that for the paper-based system
12 months after implementation of the electronic system.
We compared nursing staff’s proportion of time spent on
documentation after the implementation of the electronic
system and the proportion of time spent on documentation
in the paper-based system. At 3 months after implementa-
tion of the electronic system, RNs’ and EENs’ proportions
of time spent on documentation reduced significantly from
those recorded when they were using the paper-based system
(p = 0.08 and p = 0.02, respectively). In contrast, the PCs’ pro-
portion of time spent on documentation in this period greatly
increased (p < 0.01).
At 6 months after the introduction of the electronic sys-
tem, the RNs’ proportion of time on documentation increased
considerably compared with the proportion when the paper-
based system was used (p = 0.04). The EENs’ proportion of time
on documentation rose to a level similar to the proportion in
the paper-based system. Personal Carers’ proportion of time
on documentation remained significantly higher than the pro-
portion recorded when using paper-based methods (p < 0.01).
At 12 months into electronic documentation, RNs’ and
EENs’ proportions of time spent on documentation reduced
significantly compared with the proportions of time at
6 months after implementation. The RNs’ proportion of time
was not significantly different to the proportion recorded in
the period prior to the introduction of the electronic system.
The PCs’ proportion of time on documentation continued to
remain significantly higher than the proportion of time spent
using the paper-based system (p < 0.01); however, this propor-
tion was lower than that at three and at 6 months after the
implementation of the electronic system (Table 3).
4.2. The proportion of time spent by nursing staff on
documentation activities performed using paper and with
computer
Analysis of the proportions of time spent by nursing staff
on documentation activities performed using paper and with
computer is presented in Table 4. Overall, slightly over half of
the proportion of time on documentation at 6 and 12 months
after implementation was spent on paper documentation
activities. A majority of the proportion of time spent on docu-
mentation by both RNs’ and EENs’ involved use of paper-based
records at each measurement period after the introduction of
the electronic system (p < 0.01). The PCs spent a greater pro-
portion of their documentation time working with computer
than with paper across all the measurement periods after the
implementation of electronic documentation (p < 0.01).
5. Discussion
This study examined the effect of introducing an electronic
nursing documentation system in a nursing home on the pro-
portion of time nursing staff spend on documentation. To
our knowledge, this investigation is the first of its kind to
be undertaken in the setting of a nursing home. Moreover,
longitudinal measurement with four data collection points
spanning 14 months is unique amongst studies conducted
with a similar aim in other settings [11–13,15,16,33]. This
 Author's personal copy

i n t e r n a t i o n a l j o u r n a l o f m e d i c a l i n f o r m a t i c s 8 0 ( 2 0 1 1 ) 782–792 787

Table 2 – The number of nursing staff at different measurement periods.a

Type of nursing staff 2 Months before 3 Months after 6 Months after 12 Months after
implementation implementation implementation implementation
n (%) n (%) n (%) n (%)
Number of RNs 5 (10.6) 6 (11.3) 5 (9.3) 7 (13.0)
Number of EENs 5 (10.6) 4 (7.5) 5 (9.3) 3 (5.6)
Number of PCs 34 (72.3) 40 (75.5) 40 (74.0) 38 (70.4)
Number of RAOs 3 (6.4) 3 (5.7) 4 (7.4) 6 (11.1)
Total 47 (100.0) 53 (100.0) 54 (100.0) 54 (100.0)
a
Numbers include double counts of some nursing staff. For example, in the study period 2 months before implementation, only two RNs worked
on a morning shift, one was on duty for 3 days and another for 2 days.

approach has led to findings that enrich our understanding
of the dynamics in nursing staff’s proportion of time on docu-
mentation after the introduction of an electronic system; right
from the learning period to when the system is more stable
and integrated into routine practice [34].
It is widely anticipated that the introduction of electronic
documentation in nursing practice will reduce time spent on
documentation of care by nursing staff, through elimination
of repetitive data entry processes in the paper-based sys-
tem [2]. This expectation was not supported by this study.
The proportion of time spent on documentation 3 months
after implementation of the electronic system was similar to
that when using the paper-based system. Six months after
implementation, a greater proportion of time was spent on
documentation compared with the initial proportion in the
paper-based system. After 12 months of using the electronic
system, the proportion of time on documentation dropped to
a level similar to the proportion when the paper-based system
was used.
One possible reason for the non-reduction in the propor-
tion of time spent on documentation is nursing staff’s practice
of documenting some information items on paper and oth-
ers on a computer. Over half of the proportion of time on
documentation at 6 and 12 months after implementation was
spent on paper documentation tasks. There may be several
reasons for the relatively high proportion of time spent on
paper documentation. First, nursing staff not only charted cer-
tain information on paper (Table 1), but also used paper when
it was more convenient and practical to do so because the
information could not be captured on computer. For instance,
they preferred documenting on paper in real time at the point
of care to aid their memory. Nursing staff would note conti-
nence information on a piece of paper at the time when they
delivered care, then at some point, particularly at the end of a
work-shift, enter the information into a computer. This action
reveals the limitations of some computer systems.
In addition, nursing staff used paper to alert, notify or
prompt their colleagues to take action based on new infor-
mation. They used a diary or ‘communication book’ to pass
essential information to their colleagues working on a dif-
ferent work shift. Such information included the need for
early preparation of a resident to accompany his or her fam-
ily members. Although this information could be recorded in
a resident’s progress notes in the electronic system, it was
charted on paper instead. The electronic system had no alert
signal for new information, thus, paper was seen as a more

Table 3 – The proportion of time nursing staff spent on documentation activities before and after the introduction of the
electronic nursing documentation system.
Nursing staff Percentage of nursing staff time in documentation activity Chi square‡ p-Value§
(90% confidence interval)

2 Months 3 Months after 6 Months after 12 Months

before imple- implementa- implementa- after imple-
mentation tion tion mentation
Nursing staff in 10.7% 11.7% 12.5% (11.6–13.3)b 9.5% (8.7–10.3)c 18.410 <0.001
general (9.8–11.5)† ac (10.9–12.6)ab n = 502 n = 338
n = 383 n = 472
RNs 20% (16.8–23.2)a 15.6% (12.9–18.3)b 26.2% (22.4–30.1)c 16.2% 18.144 <0.001
n = 86 n = 75 n = 92 (13.4–19.0)ab
n = 76
EENs 32.3% (28.1–36.6)a 23.8% (19.5–28.2)b 30.1% (26.1–34.1)a 22.5% (17.5–27.5)b 8.937 0.030
n = 107 n = 62 n = 109 n = 43
PCs 6.5% (5.7–7.3)a 10.3% (9.4–11.2)b 9.2% (8.4–10.1)b 7.8% (7.0–8.7)c 30.990 <0.001
n = 177 n = 329 n = 298 n = 218
n = number of observed recordings.
†
The same superscript letter between measurement periods represents no significant difference in the proportion of time spent on documenta-
tion. Different superscript letters between measurement periods denote a significant difference in the proportion of time on documentation.
‡
Chi-square score across the four measurement periods.
§
Significance difference in proportions of time on documentation across the four measurement periods.
 Author's personal copy

788 i n t e r n a t i o n a l j o u r n a l o f m e d i c a l i n f o r m a t i c s 8 0 ( 2 0 1 1 ) 782–792

Table 4 – The proportion of time nursing staff spent on documentation activities performed using paper and with
computer.
Nursing staff Documentation Percentage of nursing staff time in documentation Chi-square‡ p-Value§
activities activity (90% confidence interval)

3 Months after 6 Months after 12 Months after

implementation implementation implementation
Nursing staff in Performed using 42.6% 56.6% (52.9–60.2)b 50.6% (46.1–55.1)c
19.106 <0.001
general paper (38.8–46.3)† a n = 284 n = 171
n = 201
Performed with 57.4% (53.7–61.2)a 43.4% (39.8–47.1)b 49.4% (44.9–53.9)c
computer n = 271 n = 218 n = 167
RNs Performed using 68.0% 78.3% (71.2–85.3) 75.0% (66.8–83.2)
2.315 0.314
paper (59.1–76.9%) n = 72 n = 57
n = 51
Performed with 32.0% (23.1–40.9) 21.7% (14.7–28.8) 25.0% (16.8–33.2)
computer n = 24 n = 20 n = 19
EENs Performed using 72.6%% 79.8% (73.5–86.1) 67.4% (55.7–79.2)
2.864 0.239
paper (63.3–81.9) n = 87 n = 29
n = 45
Performed with 27.4% (18.1–36.7) 20.2% (13.9–26.5) 32.6% (20.8–44.3)
computer n = 17 n = 22 n = 14
PCs Performed using 31.9% (27.7–36.1)a 41.3% (36.6–46.0)b 38.5%
6.209 0.045
paper n = 105 n = 123 (33.1–44.0)ab
n = 84
Performed with 68.1% (63.9–73.3)a 58.7% (54.0–63.4)b 61.5%
computer n = 224 n = 175 (56.1–66.9)ab
n = 134

n = number of observed recordings.

†
The same superscript letter between measurement periods represents no significant difference in the proportion of time spent on docu-
mentation. Different superscript letters between measurement periods denote a significant difference in the proportion of time spent on
documentation, and no superscript letters amongst the measurement periods denotes no significant difference in the proportion of time on
documentation.
‡
Chi-square score across the three measurement periods. A single chi-square score represents documentation activities performed using paper
and with computer as their analysis datasets were inverse of each other.
§
Significance difference in proportions of time on documentation across the three measurement periods. A single p-value represents docu-
mentation activities performed using paper and with computer as their analysis datasets were inverse of each other.

effective channel for passing this information than computer.
The use of paper in this case provides useful information for
improvement of the electronic system.
There was no difference in the proportion of time spent on
documentation before and 3 months after the introduction of
the electronic system. At this period, nursing staff were still
learning how to use the electronic system in their work and
their speed in data entry and searching the correct module
was slow. This is evident from the relatively high proportion
of time spent on computer-based documentation at this mea-
surement period (57.4%, Table 4). Similar results were reached
in an investigation undertaken 3 months after the introduc-
tion of an electronic documentation system in a hospital’s
paediatric ward [15].
The proportion of time spent on documentation at
6 months after implementation was higher than the propor-
tion recorded in the paper-based system. This finding may be
attributed to increased documentation requirements at this
measurement period. Nursing staff were required to complete
a 24-h shift handover report and continence information using
the electronic system. This task appeared to take longer on
computer than on paper. Our finding is contrary to those of
previous studies [11,12] which reported a reduction in time at
the similar period after electronic system introduction. The
difference may be due to variations in study settings, the mix
of study participants, study design, and documentation activ-
ities under investigation.
The proportion of time spent on documentation at
12 months after implementation was not different from the
proportion when the paper-based system was used. Our find-
ing at this measurement period is similar to results reported
by Hakes and Whittington [16]. However, their finding was
only for the activities of admission and routine documenta-
tion procedures in a surgical ward. A specific explanation for
our current finding may be that after 1 year of using the elec-
tronic system, nursing staff had familiarized with the system,
including documentation of continence information and shift
handover report. Therefore, they could complete their docu-
mentation at a reasonable speed.
Despite the lack of a reduction in time on documentation
after the implementation of the electronic system, care staff
members seem positive toward the system because of its other
benefits in their work. In our recent survey of the percep-
tions of the nursing staff members in the same facility about
quality of information and benefits of electronic nursing docu-
mentation, nursing staff perceived this system had increased
the accessibility, accuracy and legibility of the residents’
records. Repetitions in data entry were also reduced [9]. Such
 Author's personal copy
i n t e r n a t i o n a l j o u r n a l o f m e d i c a l i n f o r m a t i c s 8 0 ( 2 0 1 1 ) 782–792
benefits appear to motivate nursing staff to continue using
this electronic system.
To improve efficiency of documentation, management of
the nursing home is using the feedback from the study to
identify strategies to improve management and usage of the
electronic system. They are exploring the possibility of mov-
ing all paper forms on computer; for instance, computerizing
medication documentation forms through the introduction
of an electronic medication management system. They are
also looking into the possibility of introducing mobile comput-
ing technology at the point of care to support documentation
efforts of the nursing staff.
5.1. The proportion of time spent on documentation
activities by nursing staff in different job-roles before and
after the introduction of electronic documentation
5.1.1. Registered Nurses and Endorsed Enrolled Nurses
An electronic system is not only expected to support nursing
staff in delivery of care, but also to free them from documen-
tation duties and enable them to spend more time on direct
care work [17]. Apparently, there was no conclusive evidence
about a reduction in proportion of time on documentation for
the RNs and the EENs in this study. The RNs’ proportion of time
reduced only in the period after using the electronic system
for 3 months, but not after 6 and 12 months. The EENs’ propor-
tion of time reduced only after 3 and 12 months of using the
electronic system, and not after 6 months.
The reduction in proportion of time on documentation at
3 months after implementation of the electronic system may
be attributed to RNs’ and EENs’ positive attitude and enthusi-
asm to use a newly introduced system in their practice [35].
Such positive characteristics might have enabled RNs and
EENs to quickly learn and apply the electronic system in their
documentation practice. To aid their learning, the electronic
system was user-friendly, with most documentation responsi-
bilities completed automatically (Sections 2.2 and 2.5.1), based
on data already entered into the electronic system by PCs
(Section 2.5.2). This means that RNs and EENs did not have
to create their documentation in the electronic system from
scratch, as they were already recorded in the system. The pos-
itive characteristics of the users and the system might have
contributed to the efficiency in documentation at this period.
The RNs’ and EENs’ proportion of time on documenta-
tion increased at 6 months after implementation compared
with the proportion in the paper-based system. This increase
may partly be attributed to increased documentation require-
ments. Apart from two handover reports completed on
each work shift (summary report and administrative report,
Table 1), an additional handover report covering all shifts in
24 h was introduced at this measurement period (Table 1).
The 24-h report was comprehensive, with detailed description
of the care provided to residents. Although a comprehensive
nursing report is necessary in facilitating continuity of care,
the process involved in completing such a report might have
taken RNs and EENs longer time to learn and adapt in their
daily work [36].
Paper-based workarounds created by RNs and EENs in the
course of their duties may also account for the increase in
the proportion of time on documentation. The RNs and EENs
789
recorded and kept certain information on paper as well as
in the electronic system, i.e., residents’ glucose levels. The
paper-based copy was seen as easier to retrieve and share
with other healthcare staff such as doctors. This copy also
facilitated quick assessment and evaluation of a resident’s
healthcare status because data were organized in a longitudi-
nal format and thus, it was easier for viewing. The electronic
system poorly supported this format of data; data were located
in various sections in the system. Such information could be
used to improve the electronic system, and potentially save
nursing staff time on documentation [37].
The RNs’ and EENs’ proportion of time on documenta-
tion reduced significantly 12 months after using the electronic
system compared with the proportion at 6 months after imple-
mentation. The RNs and EENs appear to have familiarized
with both electronic and paper-based documentation system
at this measurement period. Their skills and knowledge of
applying the complex documentation system including paper
workarounds seem to have increased at this period. Although
EENs managed to achieve a remarkable reduction in propor-
tion of time on documentation at this period compared to
the proportion in the paper-based system, RNs’ proportion of
time on documentation only reduced to a level similar to the
value recorded when the paper-based system was used. Vari-
ations in RNs’ and EENs’ patterns of the proportion of time
on documentation may be attributed to their differences in
documentation practice as indicated in Section 2.5.1.
5.1.2. Personal Carers
The proportion of time PCs spent on documentation at all
measurement periods after the introduction of the electronic
system was significantly higher than the proportion in the
period when the paper-based system was used. This increase
may be a result of the slow typing speed observed in a num-
ber of these staff members. It is possible that they took longer
to input a username, locate the correct module, and type
progress notes, charts and forms, than writing on paper. In
a study exploring barriers to adoption of information technol-
ogy in Australia’s aged care settings, Yu and Comensoli [38]
found limited computer skills to be a problem amongst older
nursing staff. This group of staff members may have missed
the opportunity to learn how to use computers in their nursing
education. Indeed, Lee et al. [39] found old age of nursing staff
to be associated with increased time spent on documentation
when using an electronic system.
The high proportion of time on documentation may also be
a result of the workflow and speed of the electronic system.
The workflow in the electronic system (the process followed to
accomplish a task) [40,41] may have contributed to increased
proportion of time on documentation. The process followed
by a nursing staff member to record a resident’s continence
information provides an illustration. In the electronic system,
a staff member is required to complete three structured drop-
down menus by selecting appropriate continence information
from a list, then type a password before closing a window for
a resident’s continence chart. This process was repeated to
complete another resident’s chart. The paper-based system
required a single input in a resident’s continence chart, and
a flip-over to complete another resident’s chart. The PCs per-
ceived the procedure in the electronic system as more time
 Author's personal copy
790 i n t e r n a t i o n a l j o u r n a l o f m e d i c a l i n f o r m a t i c s 8 0 ( 2 0 1 1 ) 782–792
consuming compared with their previous workflow in the
paper-based system.
In addition, one had to transition through a minimum of
four screens to access and complete the continence chart.
The PCs perceived time taken to navigate from one screen
to another as slow. This situation is not only frustrating but
also increases the amount of time in-front of the screen [5].
In an interview with nursing staff in a nursing home, Yu et al.
[42] found nursing staff dissatisfied with the running speed
of their electronic system. Another study in a hospital set-
ting found similar results [6]. Thus, the speed of an electronic
system is critical for users in healthcare and should never be
undervalued [43].
5.2. Limitations of the study
This study was conducted in a single nursing home. It was
framed by the particular organizational structure, culture, task
allocation, work processes, and the electronic system used.
This limits the generalisability of findings. The study was also
confined to a day shift. The effect of an electronic system on
documentation time for the nursing staff working in the after-
noon and night shifts may be different. Therefore, the change
in proportion of time on documentation following the intro-
duction of the electronic system in the day shift may not be
applicable to the proportion of time in the other two shifts.
Our investigation used a work sampling technique. This
method is useful in evaluating time on activities in health-
care. For example, it allows many observations to be recorded
in a short period, thus increasing the representativeness of
data obtained. However, time obtained is an estimate and not
the exact time on a given activity. There was no measure-
ment of the PCs’ typing speed before and after the introduction
of the electronic documentation system. Such measurement
would provide useful data that increases our certainty about
whether PCs’ increased time on electronic documentation was
associated with slow typing speed.
This investigation used a single observer to record obser-
vations, which may incur personal bias. However, inter-rater
reliability testing achieved a high score [28], suggesting the
observer’s recording is trustworthy.
This study was limited to measuring the proportion of time
on documentation up to 12 months after implementation.
Measurements conducted after 12 months may have provided
additional information that clarifies the recorded pattern in
the proportion of time on documentation. Nevertheless, this
study has provided longitudinal data that has not been found
in the literature to date.
6. Conclusion
Findings of this study show that the introduction of the elec-
tronic documentation system in the nursing home did not
reduce the proportion of time nursing staff spent on docu-
mentation. This could in part be a result of the nursing staff’s
practice of documenting some information items on paper
and others on a computer. An in-depth understanding of nurs-
ing staff‘s information needs and documentation workflow is
important before an attempt is made to redesign or update
an electronic system to reduce use of paper or to achieve a
paper-free documentation environment in a nursing home.
There was no conclusive evidence of reduction in pro-
portion of time on documentation for the RNs’ and EENs’
following the use of the electronic documentation system.
Further research is required to clarify the efficiency of an elec-
tronic system on the amount of time that RNs and EENs spend
charting care. The PCs’ proportion of time on documentation
increased during the electronic documentation period. This
may suggest the need for continuous training on basic com-
puter skills and use of the electronic system, as well as regular
system updates.
Individuals planning to introduce an electronic system in
a nursing home with the aim of reducing documentation
time should consider other factors that may influence the
achievement of this goal. These include speed of the system,
users’ familiarity with the system, and their speed of typing.
Measurement of the PCs’ typing speed is necessary in monitor-
ing their progress in achieving computer skills. Longitudinal
research in other nursing homes is needed to validate findings
of this study.
Ethical considerations
All procedures used in this study were approved by the
Human Research Ethics Committee, University of Wollongong,
Australia, and complied with the National Health and Medi-
cal Research Council National Statement on Ethical Conduct
in Research Involving Humans, 1999.
Authors’ contribution
Esther Naliaka Munyisia contributed toward study design,
development and validation of the work sampling instrument,
acquisition of data, statistical data analysis, data interpreta-
tion and manuscript preparation; Ping Yu was responsible for
study conceptualization and design, validation of the work
sampling instrument, statistical data analysis, data interpre-
tation, revising the paper for important intellectual content
and final approval of the version to be submitted; David Hai-
ley contributed toward data interpretation, revising the paper
for important intellectual content and final approval of the
version to be submitted.
Conflict of interest
The authors have no conflict of interest.
Acknowledgements
This project was funded by Australian Research Council Indus-
try Linkage Grant Scheme, Project No. LP0882430. It is part of
a larger investigation of the introduction of computer-based
documentation to residential aged care. The authors would
like to thank all the nursing staff (RNs, EENs, RAOs and PCs)
at the nursing home for having accepted to participate in the
study. The manager of the nursing home is also acknowledged
for having provided support necessary for completion of the
 Author's personal copy
i n t e r n a t i o n a l j o u r n a l o f m e d i c a l i n f o r m a t i c s 8 0 ( 2 0 1 1 ) 782–792
Summary points
What was already known on the topic?
• Previous studies investigating efficiency of electronic
systems on documentation time in hospital settings
have had inconsistent findings. Some studies have
reported a reduction in time compared to that asso-
ciated with paper-based systems, others an increase,
while others have found no time difference.
• Time efficiency is one of the motivating factors for the
implementation of electronic documentation systems
into nursing homes.
• Nursing homes are increasingly introducing electronic
systems into nursing practice, but there is no empiri-
cal evidence about the efficiency of these systems on
documentation time for the nursing staff.
What this study adds to our knowledge?
• The longitudinal data collected in this study increased
our understanding of the changes in the proportion
of time on documentation at different periods after
electronic system implementation.
• Continued use of paper-based documentation meth-
ods after the introduction of an electronic documen-
tation system in a nursing home presents a challenge
for the achievement of desired outcomes in nursing
practice.
• Other factors that may influence nursing staff’s
proportion of time on documentation after the intro-
duction of an electronic system in a nursing home
include the speed of the system, users’ familiarity with
the system, and their speed of typing.
• To reduce use of paper or to achieve a paper-free
documentation environment in a nursing home, an
in-depth understanding of nursing staff’s information
needs and documentation workflow is important.
study. We are also grateful to Dr. Marijka Batterham for her
statistical support and guidance.
references
[1] E.H. Shortliffe, J.J. Cimino, Medical Informatics: Computer
Applications in Health Care and Biomedicine, 3rd ed.,
Springer-Publishing, New York, 2006.
[2] L.Q. Thede, J.P. Sewell, Informatics and Nursing:
Competencies and Applications, 3rd ed., Lippincott Williams
& Wilkins, New York, 2006.
[3] H.S. Mekhjian, R.R. Kumar, L. Kuehn, T.D. Bentley, P. Teater,
A. Thomas, et al., Immediate benefits realized following
implementation of physician order entry at an academic
medical center, J. Am. Med. Inform. Assoc. 9 (5) (2005)
529–539.
[4] S.J. Wang, B. Middleton, L.A. Prosser, C.G. Bardon, C.D. Spurr,
P.J. Carchidi, et al., A cost-benefit analysis of electronic
791
medical records in primary care, Am. J. Med. 114 (5) (2003)
397–403.
[5] T.T. Lee, C.H. Yeh, L.H. Ho, Application of a computerized
nursing care plan system in one hospital: experiences of ICU
nurses in Taiwan, J. Adv. Nurs. 39 (1) (2002) 61–67.
[6] L.E. Moody, E. Slocumb, B. Berg, D. Jackson, Electronic health
records documentation in nursing: nurses’ perceptions,
attitudes, and preferences, Comput. Inform. Nurs. 22 (6)
(2004) 337–344.
[7] S.Y. Jeong, M. McMillan, Documentation leads to reform:
reality or myth, Geriaction 21 (4) (2003) 22–25.
[8] D.S. Boroughs, Documentation in the long-term care
settings, J. Nurs. Admin. 29 (12) (1999) 46–49.
[9] E.N. Munyisia, P. Yu, D. Hailey, The changes in caregivers’
perceptions about the quality of information and benefits of
nursing documentation associated with the introduction of
an electronic documentation system in a nursing home, Int.
J. Med. Inform. 80 (2011) 116–126.
[10] B. Cherry, M. Carter, D. Owen, C. Lockhart, Factors affecting
electronic health record adoption in long term care, J.
Healthc. Qual. 30 (2) (2008) 37–47.
[11] D.H. Wong, Y. Gallegos, M.B. Weinger, S. Clack, J. Slangle,
C.T. Anderson, Changes in intensive care unit nurse task
activity after installation of a third-generation intensive
care unit information system, Crit. Care Med. 31 (10) (2003)
2488–2494.
[12] R.J. Bosman, E. Rood, H.M. Oudemans-van Straaten, J.I. Van
der Spoel, J.P.J. Wester, D.F. Zandstra, Intensive care
information system reduces documentation time of the
nurses after cardiothoracic surgery, Intensive. Care Med. 29
(2003) 83–90.
[13] K. Saarinen, M. Aho, Does the implementation of a clinical
information system decrease the time intensive care nurses
spend on documentation of care? Acta Anaesthesiol. Scand.
49 (1) (2005) 62–65.
[14] E. Ammenwerth, R. Eichstadter, R. Haux, U. Pohl, S. Rebel, S.
Ziegler, A randomized evaluation of a computer-based
nursing documentation system, Methods Inf. Med. 40 (2)
(2001) 61–68.
[15] J.A. Menke, C.W. Broner, D.Y. Campbell, M.Y. McKissick, J.A.
Edward-Beckett, Computerized clinical documentation
system in the pediatric intensive care unit, BMC. Med.
Inform. Decis. Mak. 1 (3) (2001).
[16] B. Hakes, J. Whittington, Assessing the impact of an
electronic medical record on nurse documentation time,
Comput. Inform. Nurs. 26 (4) (2008) 234–241.
[17] J.R. Buelow, M.H.A.F. Cruijssen, Long-term care nurses speak
out: midwest nurses speak freely about their likes and
dislikes in long-term care, Nurs. Homes 51 (3) (2002) 50–54.
[18] Australian Nursing and Midwifery Council: National
Competency Standards for the Enrolled Nurse, 2002,
Available from: www.anmc.org.au (cited August 08 2011).
[19] Australian Nursing and Midwifery Council: National
Competency Standards for the Registered Nurse, 2006,
Available from: www.anmc.org.au (cited August 08 2011).
[20] K. Holloway, R. McConigley, Descriptive, exploratory study of
the role of nursing assistants in Australian aged care
facilities: the example of pain management, Australas. J.
Ageing 28 (2) (2009) 70–74.
[21] Guidelines to Duty of Care for Diversional Therapists, The
Diversional Therapy Association of Australia National
Council Ltd., 2002, Available from: http://www.
diversionaltherapy.org.au/LinkClick.aspx?fileticket=
5OyBgcvFkoc%3D&tabid=61&mid=408 (cited August 08
2011).
[22] F.G. Abdellah, E. Levine, Work sampling applied to the study
of nursing personnel, Nurs. Res. 3 (1) (1954) 11–16.
[23] C. Mahler, E. Ammenwerth, A. Wagner, A. Tautz, T. Happek,
B. Hoppe, et al., Effects of a computer-based nursing
 Author's personal copy
792 i n t e r n a t i o n a l j o u r n a l o f m e d i c a l i n f o r m a t i c s 8 0 ( 2 0 1 1 ) 782–792
documentation system on the quality of nursing
documentation, J. Med. Syst. 31 (4) (2007) 274–282.
[24] K. Keshavjee, S. Troyan, A.M. Hollbrook, D. VanderMolen,
Measuring the success of electronic medical record
implementation using electronic and survey data, in:
AMIA Annual Symposium Proceedings, Washington, DC,
2001.
[25] J. Talmon, J. Ennings, G. Castaneda, F. Eurlings, D. Hoyer, P.
Nykanen, et al., The VATAM guidelines, Int. J. Med. Inform.
56 (1999) 107–115.
[26] D. Pelletier, C. Duffield, Work sampling: valuable
methodology to define nursing practice patterns, Nurs.
Health Sci. 5 (1) (2003) 31–38.
[27] L.M. Korst, A.C. Eusebio-Angeja, T. Chamorro, C.E. Aydin,
K.D. Gregory, Nursing documentation time during
implementation of an electronic medical record, J. Nurs.
Admin. 33 (1) (2003) 24–30.
[28] E. Munyisia, P. Yu, D. Hailey, Development and testing of a
work measurement tool to assess caregivers’ activities in
residential aged care facilities, in: MEDINFO, ISBN
978-1-60750-588-4, Cape Town, South Africa, 2010.
[29] H. Gabr, The effect of computerized patient record on
nurses’ time spend on documentation in inpatient units,
Int. J. Acad. Res. 2 (2) (2010) 108–120.
[30] G. Gardner, A. Gardner, S. Middleton, P. Della, V. Kain, A.
Doubrovsky, The work of nurse practitioners, J. Adv. Nurs. 66
(10) (2010) 2160–2169.
[31] A. Ampt, J. Westbrook, N. Creswick, N. Mallock, A
comparison of self-reported and observational work
sampling techniques for measuring time in nursing tasks, J.
Health Serv. Res. Policy 12 (1) (2007) 18–24.
[32] S.F. Khuri, J. Daley, W.G. Henderson, The comparative
assessment and improvement of quality of surgical care in
the department of veterans affairs, Arch. Surg. 137 (2002)
20–27.
[33] M.K. Pabs, J.C. Scherubel, A.F. Minnick, The impact of
computerized documentation on nurses’ use of time,
Comput. Nurs. 14 (1) (1996) 25–30.
[34] B.N. Doebbeling, J. Pekny, The role of systems factors in
implementing health information technology, J. Gen. Intern.
Med. 23 (4) (2008) 500–501.
[35] H. Alquraini, A.M. Alhashen, M.A. Shah, R.I. Chowdhury,
Factors influencing nurses’ attitudes towards the use of
computerized health information systems in Kuwaiti
hospitals, J. Adv. Nurs. 57 (4) (2007) 375–381.
[36] E. Ammenwerth, U. Mansmann, C. Iller, R. Eichstadter,
Factors affecting and affected by user acceptance of
computer based nursing documentation: results of a two
year study, J. Am. Med. Inform. Assoc. 10 (1) (2003) 69–84.
[37] J.J. Saleem, A.L. Russ, C.F. Justice, H. Hagg, P.R. Ebright, P.A.
Woodbridge, et al., Exploring the persistence of paper with
the electronic health record, Int. J. Med. Inform. 78 (9) (2009)
618–628.
[38] P. Yu, N. Comensoli, An exploration of the barriers to the
adoption of information technology in Australia aged care
industry, in: Health Informatics Conference Brunswick East,
Vic., Australia, 2004.
[39] T.T. Lee, T.Y. Lee, K.C. Lin, P.C. Chang, Factors affecting the
use of nursing information systems in Taiwan, J. Adv. Nurs.
50 (2) (2005) 170–178.
[40] K.M. Unertl, M.B. Weinger, K.B. Johnson, Applying direct
observation to model workflow and assess adoption, in:
AMIA Annual Symposium Proceedings, Washington, DC,
2006.
[41] M.G. Leu, M. Cheung, T.R. Webster, L. Curry, E.H. Bradley, J.
Fifield, et al., Centers speak up: the clinical context for
health information technology in the ambulatory care
setting, J. Gen. Intern. Med. 23 (4) (2008) 372–378.
[42] P. Yu, D. Hailey, H. Li, Caregivers’ acceptance of electronic
documentation in nursing homes, J. Telemed. Telecare 14 (5)
(2008) 261–265.
[43] D.W. Bates, G.J. Kuperman, S. Wang, T. Gandhi, A. Kittler, L.
Volk, et al., Ten commandments for effective clinical
decision support: making the practice of evidence-based
medicine a reality, J. Am. Med. Inform. Assoc. 10 (6) (2003)
523–530.