Evaluation of a guidelines-based e-health decision
support system for primary health care in South Africa
V. Horner
Affiliation: School of Computing, University of South
Africa, Johannesburg, South Africa
Address: School of Computing, College of Science
Engineering and Technology, Cnr. Christiaan de
Wet and Pioneer Ave. Florida, 1709
Tel: +27116709057
Email: hornevz@unisa.ac.za
ABSTRACT
Problem: This paper reports on the field evaluation of the Basic
Antenatal Care Information System (Bacis) program. Methods:
The field evaluation covered: system accuracy and safety; user
acceptance; improvements to compliance of nurses to the
maternity care guidelines; and resources implications. Results:
On system accuracy, the sensitivity of the Bacis program
matched present norms in maternity care. On usability, the
nurses expressed satisfaction with the Bacis program as a tool
for them. There was a statistically significant improvement in
compliance of nurses to the maternity care guidelines after
implementation of the Bacis program for three out of nine
categories in maternity care. On resource utilization, the Bacis
program is expected to piggyback on existing infrastructure.
Other resources were also assessed such as time, human
resources and policies, and Bacis program implementation was
judged to be feasible. Conclusion: This study has yielded
important data on implementation of e-health decision support
systems at primary health care level in South Africa.
Categories and Subject Descriptors
H.4.4 Information systems applications, types of applications
General Terms
Measurement, performance, design
Keywords
Clinical decision support systems, evaluation, e-health,
guideline adherence, maternal health, primary health care
Permission to make digital or hard copies of all or part of this work for
personal or classroom use is granted without fee provided that copies
are not made or distributed for profit or commercial advantage and that
copies bear this notice and the full citation on the first page. Copyrights
for components of this work owned by others than ACM must be
honored. Abstracting with credit is permitted. To copy otherwise, or
republish, to post on servers or to redistribute to lists, requires prior
specific permission and/or a fee. Request permissions from
Permissions@acm.org.
SAICSIT2014, September 29 - October 01 2014, Centurion, South Africa
Copyright 2014 ACM 978-1-4503-3246-0/14/09…$15.00
http://dx.doi.org/10.1145/2664591.2664599
A. Coleman
Affiliation: School of Computing, University of
South Africa, Johannesburg, South Africa
Address: School of Computing, College of Science
Engineering and Technology, Cnr. Christiaan de
Wet and Pioneer Ave. Florida, 1709
Tel: +27116709108
Email: colema@unisa.ac.za
1. INTRODUCTION
This paper reports on the field piloting of the Basic Antenatal
Care Information System (Bacis) program. The Bacis program
is an e-health decision support system intended to help with
improving compliance of nurses to the maternity care guidelines
and protocols at primary health care level in South Africa.
Maternal health care is the focus of the Bacis program because
there are problems of quality in maternal health care at primary
health care level in South Africa. For example Pattinson [15]
reports that about 20% of all perinatal deaths in South Africa
result from preventable causes, and on the side of the mothers,
the report on the Confidential Enquiries into Maternal Deaths in
South Africa estimates that about 35% of all maternal deaths
arise from preventable factors [5]. Some of the preventable
provider related factors noted by these two references are
delays in referral, non-response to poor uterine fundal growth,
non-response to maternal hypertension, non-response to post-
term pregnancy and failure to screen for important pregnancy
infections. Many of these problems can be traced to non-
adherence to the national guidelines on maternity care and lack
of tools to support clinic managers and staff in their roles at the
clinics [15].
Computerised decision support systems were chosen to address
the problem of compliance because it has been shown by
studies in developed countries that decision support systems can
improve adherence of nurses and doctors to clinical practice
guidelines [8, 20, 13]. However developing and lower to middle
income countries such as South Africa have only recently
started using health information technology systems. For
example South Africa has only recently developed an e-health
strategy [6]. Further the health system in South Africa is that of
a developing country, with a high burden of disease and
inadequate resources. This presents a challenge for
implementation of the country’s e-health strategy. Therefore,
there is interest to see how e-health systems can be
implemented in South Africa, especially at primary health care
level, as well as measurement of their impact on the quality of
care rendered to patients.
142
2. OVERV
VIEW OF BACIS
B PRO
OGRAM poliicies of the Dep partment of Heealth. The Baciis program usees
The Bacis proggram was conceeptualised by th he first author ((VH) patiient specific infformation enterred during an antenatal
a visit inn
and an obstetrician from Meduunsa Medical School
S as a tooll that ordeer to perform: risk classificatation, identificaation of patientts
could be usedd at primary health care clinics c to impprove for referral and d scheduling of maternall health carre
compliance of nurses to the maternity care guidelines andd the inteerventions. To perform
p these taasks the Bacis program
p uses itts
Basic Antenataal Care (BANC C) checklist [77, 14]. The tw wo of rulee base and knowledge base. H However, the final
f decision isi
them subsequeently developedd the system, with w the researrcher takeen by the nursinng sister. The BBacis program is i only intendedd
(VH) playing tthe role of the knowledge eng gineer and softw ware as an
a aid in the decision makingg process, i.e. it i is a decisionn-
developer, andd the obstetrician acting as th he medical dom main supp port system. Figure
F 1 show w a screenshot of the Baciis
expert. In its deevelopment theey also consulteed medical literaature prog gram interface.. The full listinng of the menu us of the Baciis
as well as the national guideelines on maternity care and oother prog gram is in Hornner [9].

Figure1. Screenshot
S of B
Bacis program
m, showing histtory menu.

3. THEOR
RETICAL FRAMEWO
F ORK Impprovements to o compliance: This is argu uably the mosst
Development aand implementtation of clinical decision suppport impportant evaluation for a clinnical practice guideline-basedd
systems has twwo main phasess: (1) systems analysis
a and de sign, deciision support system [2]. Thee question heree is whether thhe
and (2) evaluattion [3]. Evaluaation considers technical aspeccts as deciision support system is able tto improve com mpliance or nott,
well as health systems aspectts. In this paperr the following four and
d by how much h. The results oof this evaluation are the onees
aspects of the B
Bacis program were
w evaluated.. thatt will convince managers and policy makerss on adoption of o
 System acccuracy and safefety; the system.
 Usability and user accepttance testing;
Ressources implications of system
m implementatio on: Introductionn
 Improvem ments to compliaance; and
of a system has immplications onn resources succh as personnell,
 Resourcess implication
ns of syystem finaances and timee. The resourcces needed fo or the system’s
impleementation. impplementation must
m be assesssed so that th hey are knownn.
Systems theory inpputs such as thee following can n be consideredd:
System accuraacy and safetyy: Checking system accuraccy is humman resources, time,
t equipmennt, infrastructurre, finances, andd
sometimes callled system vaalidation [3]. The T aim of syystem poliicies [1].
validation is to ascertain the saafety of the sysstem and to meaasure
its performancee. Measures of performance an nd accuracy succh as
sensitivity andd specificity inn diagnosis (orr classification)) are 4. METHOD
M S
reported underr this headingg [18]. The performance off the
system is oftenn compared too a gold standaard in diagnosiis. A
subpart of sysstem validationn is checking whether
w the syystem 4.1
1 Study settting and stuudy subjects
was built accorrding to its speccification. Afteer its developmment of about a year, the Baccis program waas
field
d piloted for nin
ne months at thhree primary heealth care cliniccs
Usability and uuser acceptancee testing: Usability checks wheether in Tshwane
T Health
h District. The three study clinics are locatedd
the system is aacceptable to thhe user commun nity [11]. Evenn if a in three semi-ru ural townshipss: Winterveldtt, Temba andd
system has paassed system verification and validation it m must Soshanguve. All three townshipss are located at some distance
still undergo usability and user acceptaance testing. U User m the Pretoria central
from c businesss district at a rad
dius of about 455
acceptance testting can be doone following models
m such ass the kilo
ometres from it. The populatioon of Winterveeldt is estimatedd
technology asssessment modell, or other user acceptance moodels at about 80 000, Temba at 50 0000 and Soshangu uve at more thann
that are approppriate for the sysstem under reviiew [4, 17]. 3000 000. The town nships consist oof formal dwelllings as well asa

143
shacks, with many backroom dwellings. They have high
unemployment (40%) and high HIV prevalence (22%) [12].
Their profiles fit those of typical underserved areas in South
Africa, with a high burden of health issues. The Bacis program
is intended for strengthening of the health services at
underserved areas such as these.
The three study clinics have the following average patient loads
and staffing: 13500 patients per month; 37 nurses per clinic;
146 antenatal patients per month; 119 deliveries per month; and,
98 referrals in labour per month.
The Bacis program was used by three study nurses for their
antenatal patients during the study, as well as a data entry clerk
who assisted with patient record capture at one of the three
study clinics. Two of the nurses used it for three months and for
all their patients at the first clinic. One nurse used it for nine
months at the second clinic. And the data capturing clerk used it
for nine months at the third clinic. He entered patient data on
the system after patients had been seen by nurses at the clinic.
In total the Bacis program was used by the study nurses to treat
more than nine hundred (900) patients.
4.2 Data collection and analysis
Quantitative as well as qualitative methods of data collection
and analysis were used in the study. The four aspects of the
Bacis program chosen for evaluation were addressed as follows.
System accuracy and safety: Sensitivity of the Bacis program
was measured. This measure compared the number of patients
the system correctly classified as high risk against those it failed
to correctly classify as high risk. Data for this was taken from
the Bacis program as well as the referral registers at the study
clinics and at the referral hospitals.
Usability and user acceptance testing: To assess user
acceptance, semi-structured interviews were carried out with the
three study nurses. The interviews had the following three
themes: (1) Relevance of the content of the Bacis program. This
theme compared the match between the content of the Bacis
program with actual practice; (2) Usability and cognitive
aspects of the Bacis program, such as the time needed to learn
the Bacis program and difficulties in completing its tasks; And,
(3) usefulness of the reminders and alerts of the Bacis program
as well as the method used for issuing them. These three themes
match the two headings of the technology assessment model.
The themes of the technology assessment model are perceived
usefulness of the system, and perceived ease of use [4, 17]. The
technology acceptance model was used in the study because it is
a common model for user acceptance testing. It has been
validated by many other studies, and it is therefore a good
indicator of user acceptance of a system.
Improvements to compliance: To measure improvements in
compliance, two patient record reviews were carried out on
patients treated by the nurses; one review before introduction of
the system, and the second one after its introduction. The
patient record review criteria consisted of eighteen standard
antenatal care protocol items (18 antenatal care tasks) which
were taken from the national guidelines on maternity care [7,
14]. So there was a comparison of compliance before and after
implementation of the system.
Resources implications of system implementation: Data on
resource implications of the system is drawn from the
experience of development and piloting of the Bacis program as
well as analysis of documentation of the Department of Health
on implementation plans for e-health in South Africa. The
following systems theory inputs were considered: human
resources, time, equipment, infrastructure, finances, and
policies [1].
4.3 Ethics approval
Ethics approval for the study was obtained from the University
of Limpopo Medunsa Research Ethics Committee. Consent was
obtained from the nurses and patients to participate in the study
and consent forms were also signed by them. The study was
performed in compliance with the World Medical Association
Declaration of Helsinki on Ethical Principles for Medical
Research Involving Human Subjects [19].
5. RESULTS
5.1 System accuracy and safety
Firstly, during system development the designer and the study
obstetrician created tests cases to test each of the 49 rules of the
Bacis program to determine whether they yielded a correct
answer or not. Further a meeting, lasting for a full day, was held
with maternity nurses and managers from the eight community
health centres in Tshwane Health District and two district
managers in order to review the Bacis program. During piloting
the nurses were also asked to report errors. Corrections to the
system were therefore made during the study to ensure its
safety. Finally at the end of piloting the Bacis program correctly
identified as high risk 35.5% of patients that were referred, it
failed to correctly identify 17.4% as high risk, and 47.1% were
referred in labour which is out scope for the Bacis program. The
sensitivity of the Bacis program is therefore 67.2%.
5.2 Usability and user acceptance testing
Relevance of the content of the Bacis program: In all the
interviews the nurses generally felt that the clinical content of
the Bacis program matched their understanding of antenatal
care, though they raised a number of points on some of the steps
that it followed. For example, in the beginning of the study, in
the meeting held to review the Bacis program, the nurses asked
why hypertension screening was not part of the examination
menu in the high level menus of the Bacis program and why
proteinuria was not next to glycosuria. The reason for this was
explained by the study obstetrician, and thereafter the nurses
accepted the explanation saying that “…it was a minor
adjustment to the steps they normally followed...”
Learning and cognitive aspects: On the issue of difficulty of
learning to use the Bacis program, the nurses said that “…it was
a simple program which guided one along the process of
antenatal care ...” The younger nurses, who are more computer
literate, said it was simple in their perception, while the older
nurses said that they felt too old for computers. Generally, the
impression gained was that many nurses are computer literate
and that implementation of the Bacis program would be
welcome to them given its benefits.
Usefulness of system reminders and alerts: The nurses reported
that they found its reminders useful on the whole. They were
happy with both the data input alerts as well as the clinical
reminders. They said that “... the program guided one along the
steps of the Basic Antenatal Care (BANC) checklist. What to do
and when to refer ...” They further said that “... if they did not
agree with a reminder, the system offered them an option to
ignore or override it ...” which they said offered them
flexibility.
144
5.3 Improvements to compliance
The Bacis program was piloted for nine months at three clinics.
At the end of the piloting, the level of compliance to the
national guidelines on maternity care was compared to
compliance before implementation of the Bacis program. The
results of improvements in compliance are listed in (Table 1).
The results show a statistically significant improvement in
compliance for three out of nine antenatal patient categories.
These are compliance at booking; compliance for patients
booking after week 20; and compliance for patients younger
than 18 years. These nine categories are all important categories
of antenatal patients. Of the individual protocol items there was
a statistically significant improvement in eight protocol items
out of eighteen overall [10].

Table 1. Results of compliance review.

Item Before After Intervention P-values at

Intervention (%) (%) 95%
confidence

1. Compliance overall 85.1 89.3 0.14

2. Compliance at booking (1st visit) 82.7 90.2 0.016 **

3. Compliance at follow up visits 87.4 86.6 0.87

4. Compliance for patients booking before week 20 89.9 89.7 0.96

5. Compliance for patients booking after week 20 82.8 89.2 0.040 **

6. Compliance for patients younger than 18 years 78.7 92.0 0.0023 **

7. Compliance for patient older than 18 years 85.6 89.0 0.27

8. Compliance for primigravida patients 87.3 91.9 0.18

9. Compliance for multi-gravida patients 83.7 88.1 0.16

** The double star indicates that result is statistically significant.

5.4 Resources implications of system 6. DISCUSSION

implementation
Human resources, time, and policies: Firstly, training of nurses
in utilisation of the Bacis program will be a challenge, as this
will involve many nurses, and the training requires a substantial
commitment in time from the participating nurses. Time is a big
problem at the clinics in South Africa.
Secondly integration of the Bacis program with the clinics'
workflow also needs further attention. Currently patient held
paper records have to be completed, which are used for the
duration of a woman's pregnancy. The patient-held paper
records are important because there is no electronic patient
record in South Africa. In this scenario the Bacis program
becomes an extra item in the duties of the nurses. This may
explain the difficulty in finding a nurse at Temba clinic where a
data capturing clerk was used instead.
The study used quantitative as well as qualitative methods in its
methodology. Four aspects of the Bacis program were
evaluated: system accuracy, user acceptance, improvements to
compliance, and resources utilisation. On the whole for these
four aspects the results were successful.
The Bacis program had a sensitivity of 67% which is a
performance that is within the practice norms set in the national
guidelines on maternity care [7, 14]. The Bacis program is safe
because it uses the process of the national guidelines. And
secondly the Bacis program is meant to be used by a
professional nurse, who makes the final decision. Specificity,
the other common measure of diagnosis and classification was
not measured because the data that was gathered did not allow
for the calculation of specificity. This is therefore a gap in the
study.

On policies, the Department of Health has put in place a number
of important policies for enabling of e-health systems
implementation, such as the e-health strategy [6]. These policies
are welcoming and supportive of implementations of e-health
systems in South Africa.
Equipment, infrastructure, and finances: Implementation of the
Bacis program depends on availability of computers at the
clinics. Currently this is not the case in South Africa. However
the e-health strategy of the Department of Health makes
provisions for computers and software such as Microsoft
Access. When these become available, the Bacis program can
piggyback on them.
Although the number of nurses interviewed in the usability
review were few, they were involved for many months of
piloting of the Bacis program, and can be considered key
informants. Further, introduction of a new intervention such as
the Bacis program within the health system needs to start at a
few sites rather than at many sites. A few sites rather than many
sites during piloting allows for many issues to be ironed out, as
well as to get buy-in from nurses and health sector managers. A
study involving more sites is now possible, after this initial field
piloting.
On improvements to compliance, the Bacis program had a
statistically significant improvement for three out of nine
antenatal care categories. This was considered a success for a

145
first study. It should now be possible to conduct a study with a
larger sample to take up this issue of compliance further. An
important outcome of the study is the formulation of methods,
strategies and instruments for measuring compliance of a
decision support system at primary health care clinics in South
Africa. These methods, instruments and strategies can now be
used in the planned larger study which should utilize a sample
of adequate statistical power. The improvements in compliance
achieved by the Bacis program are small in percentage terms.
This is however not different from other studies on decision
support interventions. For example Balas et al [2] report a range
of 8% to 15% in improvement in compliance with guidelines
due to decision support systems. Even though this improvement
is small, when it is applied to a large population such as that of
maternal patients then the difference made can be a substantial
one. In South Africa the population of maternal patients, is in
the high order hundreds of thousands [16].
On resources utilisation, one of the main requirements for
implementation of the Bacis program is availability of
computers. However these are not yet generally available at
clinics, but according to the e-health strategy documentation
from the Department of Health there are plans to rollout
computers at clinics. At present the Bacis program can be used
as an audit tool by clinic managers who generally have access
to computers, which the general staff do not have. The Bacis
program also has a time utilization factor because it must be
used together with the paper patient records. The extra effort
needed to enter data into the Bacis program during a patient
visit can be offset by the expected benefits of the system. The
nurses and managers at the clinics also agreed with this view,
that the expected benefits should outweigh the effort needed to
enter its data.
An important evaluation that was not carried out in this study is
measurement of improvements to patient outcomes. The
expectation is that if a system (or intervention) improves
compliance to clinical practice guidelines, then patient
outcomes should also improve. This is because guidelines
should normally include recommendations with prior evidence
of benefit [2]. It is however worthwhile to measure patient
outcomes. But this is generally harder to do because it requires
larger samples, as well as meticulous patient follow-up which
are not easy to do. This is therefore another gap in the study.
7. OPPORTUNITIES FOR FUTURE
RESEARCH
There is an opportunity for a study to explore the impact of the
Bacis program on supervision. The current study was focused
on the nurses. The one on supervision will be focused on the
clinic managers. Another study can look at how to integrate the
Bacis program with the e-health strategy of the country and
other existing health ICT infrastructure in the country. These
are two gaps that were identified by the researcher.
8. CONCLUSION
These results on the evaluation of the Bacis program were
considered successful for the four main aspects of an e-health
decision support system. The study has therefore yielded
important data on implementation of e-health decision support
systems at primary health care level in South Africa. These
results should be of value to managers in the health sector and
implementers of e-health systems in South Africa. This first
study has also paved the way for future studies on the Bacis
program, which should expand on the current evaluation and
explore other aspects of the Bacis program that were not
covered in this first evaluation.
9. REFERENCES
[1] Alliance for Health Policy and Systems Research. Briefing
Note 1. Geneva, Alliance for Health Policy and Systems
Research. 2007. Available from: http://www.who.int/alliance-
hpsr/resources/alliancehpsr_briefingnote1.pdf
[2] Balas AE, Weingarten S, Garb, CT, Blumenthal D, Boren, S
& Brown GD. Improving preventative care by prompting
physicians. Archives of Internal Medicine; vol. 160, pp. 301-
308, 2000.
[3] Berners ES. Clinical Decision-support Systems, Spriner-
Verlag, New York; 1999.
[4] Davis F. Perceived usefulness, perceived ease of use, and
user acceptance of information technology. MIS Quarterly:
13(3), pp319- 240, 1989.
[5] Department of Health. Confidential Enquiry into Maternal
Deaths 2007 [report on enquiry into maternal deaths]. Pretoria
(South Africa): Department of Health, South Africa; 2008.
[6] Department of Health. e-Health Strategy, 2012 – 2016
[policy document]. Pretoria(South Africa): Department of
Health, South Africa; 2012.
[7] Department of Health. Guidelines for Maternity Care in
South Africa [national guidelines on maternity care]. third ed.
Pretoria(South Africa): Department of Health, South Africa;
2007.
[8] Grimshaw JM, Thomas RE, MacLennan G, Fraser C,
Ramsay CR, Vale L, et al. Effectiveness and Effeciency of
Guideline Dissemination and Implementation Strategies
[technical report, performance agency]. Health Technology
Assessment; 8(6), 2004.
[9] Horner V. Development and Piloting of an e-Health
Decision Support System for Improving Compliance of Primary
Health Care Staff to the Maternity Care Guidelines and
Protocols [PhD thesis]. Garankua (South Africa): University of
Limpopo, Medunsa Medical Campus; Department of
Community Health; 2013
[10] Horner V, Rautenbach P, Mbananga N, Mashamba T,
Kwinda H. An e-health decision support system for improving
compliance of health workers to the maternity care protocols in
South Africa. Appl Clin Inf; 4(1): 25-36, 2013.
[11] ISO9241-11. Ergonomic requirements for office work with
visual display terminals (VDT) – Part II: Guidance on usability.
International Organisation for Standardisation. 1998.
[12] Jacobs E, Punt C, Bothloko C. A profile of the North West
Province: demographics, poverty, income, inequality and
unemployment [technical report; Internet]. PROVIDE Project;
North West Provincial Government, South Africa; 2009. [cited
2012 June]. Available from:
http://ideas.repec.org/p/provbp/58056.html
[13] Lobach DF, Hammond WE. Computerized decision
support based on a clinical practice guideline improves
compliance with care standards. Am.J.Med; 102: 89-98, 1997.
146

[14] Pattinson RC. Basic Antenatal Care (BANC) Handbook
[training manual]. Pretoria(South Africa): University of
Pretoria, South Africa; 2007.
[15] Pattinson RC. Chapter 7: Maternal Health [report, health
systems research agency]. In: South African Health Review
2003/2004. Durban(South Africa): Health Systems Trust, South
Africa; p. 89-99, 2004.
[16] Statistics South Africa. Recorded live births. [Report from
statutory council]. 2012. Report P0305.
[17] Venkatesh V, Davis F. A theoretical extension of the
technology acceptance model: four longitudinal field studies.
Management Science, 46(2), pp186-204, 2000.
[18] Webb P, Bain C. Essential epidemiology. Cambridge
University Press, Cambridge: 2011.
[19] World Medical Association. WMA Declaration of Helsinki
– Ethical Principles for Medical Research Involving Human
Subjects. 2008; [cited 2010]. Available from:
http://www.wma.net/en/30publications/10policies/b3/
[20] Zielstorff RD. Online practice guidelines: issues, obstacles,
and future prospects. J.Am.Med Inform.Assoc; 5: 227-236,
1998.
147