2013 10th International Conference on Information Technology: New Generations
Assessing the Usability of University Websites in
Saudi Arabia: A Heuristic Evaluation Approach
Mutlaq B. Alotaibi
Vice Dean for Academic Affairs, College of Computer and Information Sciences
Imam University, P.O. Box 93844, Riyadh 11683, Saudi Arabia
motaibi@imamu.edu.sa
Abstract— This paper discusses Heuristic Evaluation (HE) as a
method for evaluating the usability of websites, particularly for
universities. This study has demonstrated the empirical
application of HE, as a means for examining the usability of
university websites in Saudi Arabia. Furthermore, it provides a
comparison of private and government universities, in
accordance with a set of HE components. Based on related
studies, a set of principles and heuristics were developed to
evaluate university websites, including: visual design and
consistency, links and navigation, data entry forms, information
truth and precision, privacy and security, search functionality, as
well as, help, feedback and error tolerance. In fact, twelve (n=12)
university websites were assessed by thirty (n=30) evaluators for
each heuristic, using a four-point scale ranging from zero (0- not
applicable) to three (3- yes). The main results of the study
indicated that Saudi universities exhibit a significant tendency
toward implementing high-quality usability practices. Yet,
websites of Saudi private universities were found to offer 5%
lower usability when compared to government universities.
Keywords- Usability, Heuristic, Evaluation, Saudi Arabia,
University Websites
I. INTRODUCTION
The Information and Communication Technology (ICT)
market in Saudi Arabia is still rapidly growing, due to huge
governmental spending in IT and an increase in ICT awareness.
It is argued that the Saudi ICT market is the largest amongst all
Middle Eastern markets, holding over 50% of the total market
in the region [1]. The Internet has received special attention, as
several factors indicate the potential growth of its usage, such
as long-term ICT national plans, mature ICT infrastructures, an
increase in population and a decrease in access costs [3]. Since
its introduction in 1997, the number of internet users has
exponentially increased, there are over 13 million users which
accounted for a 47% penetration rate in 2011 [2, 3].
The status of Electronic Readiness (E-Readiness) has
indicated a growing potential of Electronic Government (E-
Government) services [4]. Many ministries have initiated E-
Government projects, such as the national student admission
system which was developed by the Ministry of Higher
Education (MOHE). According to the MOHE [5], there are 32
university: 24 government universities (eight developed and 16
emerging) and eight private universities. Although it is argued
that university web-presence is of low impact [6], little was
found about the usability assessment of university websites.
Therefore, this study aims to fill this gap of knowledge by
978-0-7695-4967-5/13 $26.00 © 2013 IEEE
DOI 10.1109/ITNG.2013.26
initially examining the usability of university web-presence in
Saudi Arabia by using the HE approach.
The remainder of this paper will be organized into the
following sections. In section two, a review of the literature
will describe the HE approach, it will also critically analyze
related studies. The third section will describe the research
methods used, including heuristic checklists, sampling and
website selection for evaluation. The results and analysis will
be discussed in the fourth section and finally, the fifth section
will provide conclusions.
II. LITERATURE REVIEW
Heuristic evaluation is used to inspect the usability of user
interface designs [7]. HE is considered to be one of the most
widely used Usability Inspection Methods (UIM), due to
several advantages, such as ease to use, intuitiveness, low cost
and the fact that it does not require any advanced planning [7].
However, HE does not provide an objective judgment of the
interface design, nor does it provide solutions to any of the
identified usability problems [8]. In addition, HE has several
drawbacks, such as the identification of false positive usability
problems, unstructured assessments and the lack of user task
focus [9].
Although usability testing is seen as an alternative UIM, it
can be supplemented by the HE approach. Usability testing
might ignore a great number of usability issues, due to its
experimental nature which may limit the scope of the exposed
functionality of the inspected website [10]. Cost-wise, the HE
is much cheaper than usability testing in terms of time and
resources [11]. Another problem, which might weaken the
results obtained from the two UIM, concerns the evaluators’
background. In the HE approach, the evaluators are experts and
as such might not truly reflect the actual users’ opinion, as the
participants in the usability testing might not be typical users of
the inspected website. Therefore, it is commonly agreed that
the HE approach should be used before the usability testing, in
order to identify usability problems and hence design
experimental tasks for usability testing [10].
Scholarly research has identified the usability status of
websites in the Middle East, particularly in Saudi Arabia. For
example, Noruzi [6] evaluated the impact and intensity of
web-presence in Middle Eastern countries, especially
universities and academic institutions. Noruzi found that Arab
web-presence is generally weak and of low impact, even with
138
regards to university web-presence. In addition, Eidaroos
Probets and Dearnley [12] carried out an investigation, into the
usability of two E-Government web-presences in Saudi Arabia,
using HE. Three web design experts evaluated the websites in
terms of nine principles: consistency, links and navigations,
helping users, features and functions, data entry forms, visual
design, accessibility to visually impaired users, security and
privacy, and the precision of information. This study revealed
that the E-Government web design was immature with regard
to usability elements, but it was advanced in terms of its visual
appearance which was remarkably noticed. In addition, Al-
Khalifa [13] evaluated 14 websites, from different sectors,
using the HE method, to assess the usability of E-Government
websites in Saudi Arabia; the findings indicated that E-
Government web-presence in Saudi Arabia needs considerable
effort in order for it to become usable. In summary, although
the university web-presence is of low impact in the Middle
East, few studies have identified the usability status of websites
in this region. Therefore, this study focuses on the usability
assessment of university websites in Saudi Arabia, in order to
fill this knowledge gap.
III. METHOD
As mentioned previously, although the HE approach is
inexpensive, one of the criticisms of this approach is that the
evaluator’s perspective is a reflection of the expert’s opinion
rather than the actual website users’. Therefore, this study has
included different perspectives of the web-presence evaluators
as it took advantage of the ‘typical’ university websites users
(students both undergraduate and postgraduate) for evaluation
purposes, in addition to the expert opinions. Another advantage
is that the student evaluators had robust informatics and
computing backgrounds, and were therefore knowledgeable of
usability practices and interface designs.
This study relied on a within-subjects design [14] in which
all evaluators were exposed to the 12 inspected university
websites. Prior to the evaluation, a 15 minute lecture was
presented to provide the evaluators with the theoretical
foundations and underlining principles of the HE approach, this
was followed by a short (30 minute) training session to show
how the principles could be put into practice. The order of the
website evaluation was counterbalanced amongst the
evaluators in order to eliminate any possible biases.
A. Heuristics Checklist
Heuristic evaluation aims to identify usability problems
within an inspected website; HE is therefore a method of
website design evaluation [15], it relies on the opinion of
website design experts (one or more) who examine an
inspected website to determine how well it fulfils a set of
predefined guidelines (principles and heuristics). The
guidelines can be derived from theory and practice, or from
specific standards of website usability [16]. For example,
Neilson and Molish (1990) provided a set of principles and
heuristics to indentify usability problems within a user interface
[7]. Another method of usability inspection actually examines
the adherence of a website design to ISO standards [17],
particularly to the ISO 9241-151 [18] which aims to improve
usability practices.
As for the development of evaluation guidelines for
university websites, little was found in the literature to suggest
that special principles or heuristics were being applied in
academic institutions. In particular, a great proportion of the
employed heuristics has been built from Neilson’s heuristics
[8] or ISO standards [18]. For example, in E-Government
contexts, several researchers adapted tailored heuristics to suit
the nature of E-Government websites [2, 13, 15, 19]. This
stream of research was followed for developing the evaluation
guidelines for the university websites, to ensure that the nature
of the academic contexts were considered. The heuristic
checklist consisted of 58 (n=58) questions which represented
the seven principles, as shown in Table 1. The fulfilment of
each heuristic was then rated against the following four-point
scale: (0) not applicable; (1) no (not fulfilled); (2) partially
fulfilled; and, (3) yes (fully fulfilled).
B. Website Selection
There are 34 (n=34) universities in Saudi Arabia, they fall
into one of the following three categories: private, emerging
government and developed government universities. The
procedure planned to select websites for evaluation that were a
true representative sample of all of the universities in Saudi
Arabia. Initially, two evaluators were instructed to examine all
university websites, using the developed heuristics, in order to
identify different quality levels of the university websites. Once
the quality differences were determined, four (n=4) websites
from each category were selected to represent the different
quality levels observed.

TABLE I. FIFTY-EIGHT HEURISTICS REPRESENTING THE SEVEN HE PRINCIPLES

Component Description Number of heuristics

C1 Visual design and consistency Consistent and visually appealing design 9

C2 Links and navigation Appropriate navigation and links to facilitate website browsing 10

C3 Data entry forms Accessible and easy to use web forms 10

C4 Information, truth and precision Precise and trustworthy content 7

C5 Privacy and security Confidential areas are protected, privacy and security policies are presented 6

C6 Search functionality Information is easily found by using effective search features 7

Appropriate help and feedback are provided to users, error prevention mechanisms are
C7 Help, feedback and error tolerance 9
rarely employed

139
 TABLE II. TWELVE UNIVERSITY WEBSITES REPRESENTING THREE CATEGORIES
Sector Category
Government Developed King Fahd University of Petroleum and Minerals (KFUPM)
King Saud University (KSU)
Imam Muhammad Ibn Saud Islamic University
King Faisal University (KFU)
Emerging Jouf University (JU)
Taibah University
Jazan University
Majmaah University
Private Private Prince Sultan University (PSU)
Alfaisal University
Al-Yamamah University
Dar Aluloom University (DAU)
In total, twelve websites (n=12) were considered for
usability inspection, as shown in Table 2.
C. Sampling
The sample consisted of 30 participants from the computing
and informatics field; they therefore had excellent knowledge
and experience of web design. The sample was balanced in
order to eliminate the effects of gender, education and
experience. In fact, equal numbers of male and female
participants were recruited: 15 male (50%) and 15 female
(50%), these participants represented the three groups equally
with five male and five female participants in each of the
following three groups: undergraduate students, post-graduate
students and practitioners.
IV. RESULTS
The instrument was examined for reliability using
Cronbach’s Alpha (), in order to determine the internal
consistency of the scale items using a critical value of 0.7 [32].
The statistical procedures indicated that the seven HE
components were reliable as they had values above the critical
value. The values were as follows:  = 0.823, 0.704, 0.768,
0.749, 0.737, 0.823 and 0.820, respectively for: visual design
and consistency, links and navigation, data entry forms,
information truth and precision, privacy and security, search
functionality, and help, feedback and error tolerance.
In general, the university websites in Saudi Arabia were
found to be relatively acceptable in terms of their usability,
with an average score of 74% for usability, the government
universities had a score of 75% and the private universities
achieved 71%. It is noteworthy that little difference was found
between the emerging and developed government universities
with regard to usability scores across all principles of the HE
approach. Therefore, the analysis of the results, herein, will
compare the private and government universities.
Fig. 1 shows the percentage of usability scores for
government and private universities in Saudi Arabia, in
accordance with the seven principles of the HE approach. In
each principle, the percentage value represents the number of
heuristics met. At a glance, it can be seen that the university
websites in Saudi Arabia did not fully meet any of the seven
principles. However, all of the principles scored higher than
50%, which indicates an acceptable level of usability
implementation. The visual design and consistency was found
to be the best met principle, where 81% and 85% fulfilled the
140
University Reference
[20]
[21]
[22]
[23]
[24]
[25]
[26]
[27]
[28]
[29]
[30]
[31]
heuristics in private and government universities, respectively.
This can be attributed to the fact that most of the evaluated
websites were consistent and pleasant to look at. In contrast,
the data entry forms principle was found to be the least met,
with 56% and 64% fulfilled heuristics in private and
government universities, respectively. This could be indicative
that the web designers in Saudi universities paid less attention
to the web’s form in terms of accessibility and ease of use. For
example, the hints supporting the form filling were generally
lacking and there was uncertainty with regard to distinguishing
the required felids. These two findings support the work of Al-
Khalifa [13].
Furthermore, three other components showed a relatively
similar picture when evaluated: links and navigation,
information truth and precision, and help, feedback and error
tolerance. As for the links and navigation, the number of
fulfilled heuristics was acceptable with 76% and 77% scores
for private and government universities, respectively. It is
believed that this was because most of the university websites
had consistent navigation which provided good feedbacks
whilst also offering a website map. As for the information
truth and precision principle, the private and government
universities scored 77% and 80%, respectively, as most of the
universities presented accurate, updated and precise content.
Information was organized hierarchically, from the general to
the specific, and the organization was clear and logical.
However, most websites failed to provide appropriate third-
party support (e.g. citations, testimonials) to verify the
accuracy of their information. In addition, the help, feedback
and error tolerance component scored 75% and 80% for private
and government universities, respectively. This could be
attributed to the users’ perspectives with regard to previous
experience, expectations and help provided. Most of the
websites provided FAQ, online help and progress indicators.
Furthermore, with the analysis of the HE factors, the
privacy and security component was found to be relatively
similar to the search functionality component. For privacy and
security, the scores for private and government universities
were 69% and 72% which indicate a general tendency towards
implementing good privacy and security practices. This could
be attributed to the presentation of security and privacy policies
and the protection of confidential areas. However, most of the
websites failed to present digital certificates and third-party
seals.
 Private Government
100%
81% 85% 80%
76% 77% 77% 80% 75%
69% 72% 70%
64% 62%
56%
50%

0%
Visual Design Links and Data Entry Information Privacy and Search Help, Feedback
and Navigation Forms Truth and Security Functionality and Error
Consistency Precision Tolerance

Figure 1. Percentage of usability scores for government and private universities in Saudi Arabia according to the seven principles of the HE approach

The search functionality factor was also determined to be of
a good quality, with a score of 62% and 70% for private and
government universities, respectively. Finally, it was noticed
that the inspected websites failed to offer advanced search
features, document relevancy indicators and full-text searches.
In accordance with the seven HE components: visual
design and consistency (C1); links and navigation (C2); data
entry forms (C3); information truth and precision (C4); privacy
and security (C5); search functionality (C6); and, help,
feedback and error tolerance (C7), together provided an overall
usability score. Table 3 shows the usability scores for each
inspected website.
V. CONCLUSIONS
The universities in Saudi Arabia have witnessed an
increasing demand for providing university students with
electronic services, as the number of government and private
universities have increased considerably over the last 10 years.
In this paper, a usability assessment of university websites was
conducted, based on the HE approach. The results, herein,
suggest that the university websites in Saudi Arabia maintain
an acceptable level of usability practices with regard to the
seven HE principles: visual design and consistency, links and
navigation, data entry forms, information truth and precision,
privacy and security, search functionality, and help, feedback
and error tolerance. In particular, the government websites
demonstrated that they were of a slightly higher quality than
the private university websites. In addition, this study revealed
that the emerging government universities have been successful
with regard to their web-presence implementations which
reached a similar usability level to the developed universities.
Although this study will be of use to managers and web
designers at the Saudi universities, the approach used for this
evaluation is also suitable as an initial assessment of website
usability. The potential directions for further research could
include: a comparison of technical and pedagogical usability;
investigation into the HE approach and user testing,
management, and end-user perspectives.

TABLE III. USABILITY SCORES FOR INSPECTED WEBSITES

HE Components Usability
Sector Category University
C1 C2 C3 C4 C5 C6 C7 Score
Government Developed KFUPM 88% 79% 71% 84% 64% 73% 76% 77%
KSU 90% 80% 67% 92% 82% 76% 87% 81%
Imam 81% 70% 59% 72% 68% 75% 79% 72%
KFU 80% 76% 58% 77% 64% 61% 77% 71%
Emerging JU 87% 80% 66% 84% 83% 74% 90% 80%
Taibah 90% 87% 69% 86% 79% 66% 84% 80%
Jazan 79% 71% 64% 72% 68% 69% 78% 71%
MU 84% 71% 59% 69% 70% 63% 66% 69%
Private Private PSU 83% 77% 55% 87% 69% 61% 67% 71%
Alfaisal 87% 81% 60% 82% 67% 38% 78% 71%
YU 74% 71% 55% 70% 67% 71% 75% 69%
DAU 81% 74% 56% 67% 75% 81% 79% 73%

141
 REFERENCES
[1] SAGIA, “Saudi Arabian General Investment Authority,” [Online]
www.sagia.gov.sa, Accessed: 23 August 2012.
[2] A. Garcia, C. Maciel, and F. Pinto, “A quality inspection method to
evaluate E-government sites,” Electronic Government, pp. 198-209,
2005.
[3] CITC, “Communications and Information Technology Commission:
Annual Report,” [Online]
http://www.citc.gov.sa/arabic/MediaCenter/Annualreport/Documents/PR
_REP_007.pdf, Accessed: 23 August 2012, 2011.
[4] F. Al-Sobhi, and V. Weerakkody, “The role of intermediaries in
facilitating E-government diffusion in Saudi Arabia,” in European and
Mediterranean Conference on Information Systems, Abu Dhabi, UAE,
2010.
[5] MOHE, “Academic institutions in Saudi Arabia: A statistical report,”
[Online] http://statistics.mohe.gov.sa/DCFiles/Universities-Guide.pdf,
Accessed: 23 August 2012.
[6] A. Noruzi, “Web presence and impact factors for Middle-Eastern
countries,” Online Magazine, vol. 30, pp. 22-28, 2006.
[7] J. Nielsen and R. Molich, “Heuristic evaluation of user interfaces,” in
the SIGCHI Conference on Human Factors in Computing Systems:
Empowering People Seattle, WA, USA: ACM, pp. 249-256, 1990.
[8] J. Nielsen, “Heuristic evaluation,” Usability Inspection Methods, vol. 24,
p. 413, 1994.
[9] C. Ling and G. Salvendy, “Extension of heuristic evaluation method: A
review and reappraisal,” Ergonomia IJE & HF, vol. 27, pp. 179-197,
2005.
[10] E. T. Hvannberg, E. L. C. Law, and M. K. Lárusdóttir, “Heuristic
evaluation: Comparing ways of finding and reporting usability
problems,” Interacting with Computers, vol. 19, pp. 225-240, 2007.
[11] S. M. Z. Ahmed, “A comparison of usability techniques for evaluating
information retrieval system interfaces,” Performance Measurement and
Metrics, vol. 9, pp. 48-58, 2008.
[12] A. Eidaroos, S. Probets, and J. Dearnley, “Heuristic evaluation for e-
government websites in Saudi Arabia,” in Thethird Saudi International
Conference Surrey, UK, 2009.
[13] H. Al-Khalifa, “Heuristic evaluation of the usability of e-government
websites: A case from Saudi Arabia,” in the 4th International
Conference on Theory and Practice of Electronic Governance Beijing,
China: ACM, pp. 238-242, 2010.
[14] A. P. Field, Discovering statistics using SPSS: SAGE publications Ltd,
2009.
142
[15] A. Eidaroos, S. Probets, and J. Dearnley, “Heuristic evaluation for e-
government websites in Saudi Arabia,” in Thethird Saudi International
Conference, Surrey, UK, 2009.
[16] M. Donker-Kuijer, M. De Jong, and L. Lentz, “Usable guidelines for
usable websites? An analysis of five e-government heuristics,”
Government Information Quarterly, vol. 27, pp. 254-263, 2010.
[17] N. Bevan, “International Standards for usability should be more widely
used,” Journal of Usability Studies, vol. 4, pp. 106-113, 2009.
[18] ISO, “Ergonomics of human-system interaction -- Part 151: Guidance on
World Wide Web user interfaces,” [Online]
www.iso.org/iso/home/store/catalogue_tc/catalogue_detail.htm?csnumb
er=37031, Accessed: 11 March 2012, 2008.
[19] B. Soufi and M. Maguire, “Achieving usability within e-government
websites illustrated by a case study evaluation,” Human Interface and
the Management of Information. Interacting in Information
Environments, pp. 777-784, 2007.
[20] KFUPM, “King Fahd University of Petroleum and Minerals,” [Online]
www.kfupm.edu.sa, Accessed: 1-15 April 2012.
[21] KSU, “King Saud University,” [Online] www.ksu.edu.sa, Accessed: 1-
15 April 2012.
[22] Imam, “Imam Muhammad Ibn Saud Islamic University,” [Online]
www.imamu.edu.sa, Accessed: 1-15 April 2012.
[23] KFU, “King Faisal University,” [Online] www.kfu.edu.sa, Accessed: 1-
15 April 2012.
[24] JU, “Jouf University,” [Online] www.ju.edu.sa, Accessed: 1-15 April
2012.
[25] Taibah, “Taibah University,” [Online] www.ju.edu.sa, Accessed: 1-15
April 2012.
[26] Jazan, “Jazan University,” [Online] www.jazanu.edu.sa, Accessed: 1-15
April 2012.
[27] Majmaah, “Majmaah University,” [Online] www.mu.edu.sa, Accessed:
1-15 April 2012.
[28] PSU, “Prince Sultan University,” [Online] www.psu.edu.sa, Accessed:
1-15 April 2012.
[29] Alfaisal, “Alfaisal University,” [Online] www.alfaisal.edu, Accessed: 1-
15 April 2012.
[30] YU, “Al-Yamamah University,” [Online] www.yu.edu.sa, Accessed: 1-
15 April 2012.
[31] DAU, “Dar Aluloom University,” [Online] www.dau.edu.sa, Accessed:
1-15 April 2012.
[32] J. R. A. Santos, “Cronbach’s Alpha: A tool for assessing the reliability
of scales,” Journal of Extension, vol. 37, pp. 1-5, 1999.