2015 International Seminar on Intelligent Technology and Its Applications
Usability Characteristic Evaluation On
Administration Module of Academic Information
System Using ISO/IEC 9126 Quality Model
Siti Rochimah, Hanifa I. Rahmani, Umi Laili Yuhana
Department of Informatics, Faculty of Information
Technology, Institut Teknologi Sepuluh Nopember
Jl. Raya ITS, Surabaya 60111 Indonesia
siti@its-sby.edu, yuhana@its-sby.edu
Abstract— Administration module in Academic Information
System (AIS) is one of modules in AIS to view academic data
such as students, academic staff, courses, academic activities and
curriculum information. This module is accessed by students,
administration staff, faculty, and chairman of the department.
Software usability as one of software quality aspects is very
important to be considered. The evaluation is needed to
determine the level of usability of software.
This paper discuss the evaluation of usability characteristic
for administration module of ITS’ academic information system.
The purpose of the evaluation is to determine the usability
behavior at AIS administration module using ISO/IEC 9126.
ISO/IEC 9126 usability characteristic has six sub characteristics
i.e understandability, learnability, operability, attractiveness, and
usability compliance. Each sub characteristic has metrics to
determine the results of the evaluation. Values of the parameters
obtained from the metric calculation software attributes are
derived from software documentation. Attributes of software that
is used are feature, interface, message, operation, input, and
regulation. The results of the evaluation on AIS student
administration module show that learnability characteristic
generates a zero value. This shows that the student
administration module AIS has not have the user assistance
feature. Other sub characteristics results have non zero value yet
not perfect. Those overall results show that it is required an
evolution in AIS so that the quality value will be better.
Keywords— usability characteristics, quality model, academic
information system, ISO/IEC 9126.
I. INTRODUCTION
Academic Information System (AIS) is a software
management, which used in a university for academic
activities. AIS ITS is an academic information system owned
by the Institute of Technology Surabaya (ITS). AIS ITS has
two teams of developers. The first team is built the original
AIS ITS. While the second team continues the first team’s
work for maintenance and improve AIS. The second team is
the team from Lembaga Pengembangan Teknologi Sistem
Informasi ITS (LPTSI ITS). Policy differences and features
desired by the university require the developer to make
978-1-4799-7711-6/15/$31.00 © 2015 IEEE
changes to the system. Currently into making changes or
additions to the AIS ITS features, the method used is the
addition and improvement of the software without the use of
standards [1]. One of the modules in AIS is student
administration module. This module aims to help the college
administration activities. Most of the features in the module
are display the data.
Software quality assurance is needed by software
developers because these activities are used to control the
software conformance with requirements. This is also
supported by software that helps business processes from a
variety of fields. According to Galin [2], software quality is
conformance to requirements, fewer errors, lower defect rate,
high reliability levels. If the software is not guaranteed with
good quality, all of activities become unbalanced. Quality
model that often use are McCall, Boehm, FURP, and ISO
9126. ISO/IEC 9126 is the most complete quality model of
all, it has comprehensive characteristics, and easy to use [3].
ISO/IEC 9126 have three measurement metrics, namely
internal, external, and the use of quality metrics. ISO/IEC
9126 are made to improve the quality of products, and provide
evaluation results on a software product that has been
measured [4]. In an internal metric, there are six sub
characteristics, the functionality, efficiency, reliability,
usability, portability, and maintenance.
Therefore, in this study, we present an evaluation on AIS
student administration module to determine measure the
usability characteristic. This evaluation needs usability
evaluation tools based on ISO/IEC 9126.
II. LITERATURE REVIEW
A. ISO/IEC 9126 Quality Model
ISO/IEC 9126 is a model for evaluating the software
quality standards. ISO/IEC 9126 has four parts, such as
ISO/IEC 9126-1, ISO/IEC 9126-2, ISO/IEC 9126-3 and
ISO/IEC 9126-4. ISO/IEC 9126 issued by International
Organization for Standardization and International
Electrotechnical Commission. The first part of ISO/IEC 9126
363
describes the term software quality characteristics and define B. Usability Based on ISO/IEC 9126-3
sub characteristics from those characteristics. While the Based on the ISO/IEC 9126, there is a chapter that
second part describes the external metric used to measure the explains about software measurement using internal metrics,
behavior of computer-based systems. Measurements were
namely the ISO / IEC 9126-3. Usability of the metrics used to
taken when the software operates. The third section describes
make assessment on software in terms of ease of software to
the internal metric used to measure the software. This
measurement can use on the non-executable software. The last understand, learn, operate, level of attractiveness, and
section describes the quality in use metrics [4]. In general, the compliance aspects of usability. The result of these
characteristics and sub characteristic of ISO/IEC 9126 quality measurements is a value or can be used as comparative
models can be described in Table 1 below. measurements of two pieces of software. In addition, the
measurement, the output must issue an average standard error.
Usability has five sub characteristics and weights. Weight of
TABLE I. ISO/IEC 9126 QUALITY MODEL [3]
sub characteristics illustrates the importance of characteristics
Characteristics Sub Characteristics and the effort of evaluators in conducting evaluations on
Functionality Suitability certain sub characteristics. Explanation of sub characteristics
Accuracy existing on usability characteristics are as follows [8].
Interoperability 1. Understandability
Security The metrics in understandability are used to assess
Compliance the level of user understanding of the software used
Reliability Maturity to need. In addition, this metrics are also used to
Fault tolerance assess the user's understanding of the software to
Recoverability
complete a task. Understandability has medium
Compliance
weight. Metrics related to understandability can be
Usability Understandability
Learnability
seen in Table 3.
Operability 2. Operability
Attractiveness The metrics in operability are used to assess and
Compliance control the level of use of the software by the user.
Efficiency Time behaviour Operability has a high weight. Metrics related to
Resource utilization operability can be seen in Table 4.
Compliance 3. Learnability
Maintainability Analyzability The metrics in learnability are used to assess length
Changeability of time users learn to use a partial function of the
Stability software and the effectiveness of the help system and
Testability documentation. Learnability has a low weight.
Compliance Metrics related to learnability can be seen in Table 2.
Portability Adaptability 4. Attractiveness
Installability The metrics in attractiveness are used to assess the
Co-existence
attractiveness of the software that is influenced by
Replaceability
several factors, for example the design and color
Compliance
screen. Attractiveness has medium weight. Metrics
related to attractiveness can be seen in Table 6.
ISO/EC 9126 have six characteristics, namely 5. Usability Compliance
functionality, reliability, usability, efficiency, maintenance, The metrics in usability compliance used to measure
and portability. Each characteristic has its own measurement the software's ability to comply with guidelines for
purposes. Functionality aims to predict the satisfaction level of software usability. The metrics in usability
functionality based on its functional requirements. Reliability compliance keep the software does not violate the
aimed at assessing the level of reliability of software systems. rules of the relevant organization about software
Usability aims to determine the level of ease of use of the usability. Usability compliance has a high weight.
software. Efficiency aims to measure the level of efficiency of Metrics related to usability compliance can be seen in
software behavior while testing and operation. Maintenance
Table 5.
aims to calculate the effort in the maintenance. Portability
aims to determine software's ability if the environment change
[4]. C. Software Quality Assurance
There are some evaluations using ISO/IEC 9126. From the The quality of software becomes important because of the
previous study, some cases have been evaluated by ISO/IEC many areas that have used software, such as transportation,
9126, such as spreadsheet [5], source code [6], and e-learning health, economic exchange, and others. If the software is not
[7]. guaranteed with good quality, then the activity in these areas
becomes unbalanced [2]. Especially for academic area, there is

364
AIS. That information system support business process of TABLE IV. OPERABILITY METRIC [4]
academic. If there is some problem in AIS, it will disturb Metrics Name Parameter A Parameter B Formula
learning activities, such as taking course for students, insert Input validity Number of input Number of Result =
student’s grade, validate the course, and so on. checking items which input items A/B
check for valid which could
TABLE II. LEARNABILITY METRIC [4] data check for
valid data
Metrics Name Parameter A Parameter B Formula User operation Number of Number of Result =
cancellability implemented functions A/B
Completeness of Number of Total of functions which requiring the
user functions number of Result = can be cancelled precancella-
documentation described functions A/B by the user tion capability
and/or help provided User operation Number of Number of Result =
facility Undoability implemented functions. A/B
functions which
TABLE III. UNDERSTANDABILITY METRIC [8] can be undone
by the user
Metric Name Parameter A Parameter B Formula
Customisability Number of Number of Result =
Completeness Number of Total number Result =
functions which functions A/B
of description functions (or of functions A/B
can be requiring the
types of (or types of
customised customization
functions) functions)
during operation capability
described in the
Physical Number of Number of Result =
product
accessibility functions which functions A/B
description
can be
Demonstratio Number of Total number Result =
customised
n capablity functions of functions A/B
Operation Number of Number of Result =
demonstrated and requiring
status functions having functions that A/B
confirmed in demonstration
monitoring status are required
review capability
capability monitoring to have
Evident Number of Total number Result =
capability monitoring
functions functions (or of functions A/B
capability.
types of (or types of
Operational Number of Total number Result =
functions) evident functions)
consistency instances of of operations 1- A/B
to the user
operations with
Function Number of user Number of Result =
inconsistent
understandabi interface user interface A/B
behaviour
lity functions whose functions.
Message Number of Number of Result =
purpose is
Clarity implemented messages A/B
understood by the
messages with implemented
user
clear
explanations. .
According to Pressman [9], software quality assurance is Interface Number of Total number Result =
conformance between functional and requirements, software element clarity interface of interface A/B
quality standards and implicit characteristics that are expected elements which elements
of professional software developers. Based on the IEEE [10], are self-
software quality assurance is a plan and systematic pattern that explanatory.
needed to produce products which is compliance with Operational Number of Total number Result =
requirements, and a set of activities that is made to perform an error functions of functions A/B
evaluation process based on the product being built. The recoverability implemented requiring the
with user error tolerance
purpose of software quality assurance in software
tolerance capability
development is as follows [2].
1. Ensuring confidence level software according to the
requirements. D. Academic Information System (AIS) Student
2. Ensure the level of confidence that the software will be Administration Module
in accordance with the scheduling and development Student administration module is part of academic
funds that have been described when defining information system. AIS has never been tested using standard
requirements. quality on usability characteristics. Student administration
3. Initiation and management activities to improve and module is supporting key business processes of AIS. Features
increase the level of efficiency of software development relating to student administration module have variety of users
and software quality assurance. and most of the function to display the data. Therefore, it is

365
necessary to know the value of usability testing of student documents, design documents, and document review. The
administration module AIS. Features available on the student purpose of collecting this document is to identify the attributes
administration module by its users can be seen in Table 7. needed for the evaluation.
Features on student administration module can be seen as
follow [11]. TABLE VII. FEATURE OF AIS STUDENT ADMINISTRATION MODULE
- View biodata; WHICH IS GROUPED BY USER
- View status of a student;
- View a list of student prerequisite courses; User Features
- View courses schedule; Student - View biodata
- View a list of student trustee; - Change student password
- Print a list of attendees; - View status of a student
- View data of all employees; - View a list of student prerequisite courses
- Change student password; - View courses schedule
- View a list of curriculum; - View a list of curriculum
Lecture - View biodata
- View all courses schedule;
- View all courses schedule
- View schedule of teaching (lecturer); and - View status of a student
- View all subjects. - View a list of student prerequisite courses
- View a list of student trustee
TABLE V. USABILITY COMPLIANCE METRIC [4] - View data of all employees
- View a list of curriculum
Metrics Parameter A Parameter Formula - View schedule of teaching (lecturer)
Name B - View all subjects
Result = Head of - View biodata
Usability Number of correctly Total
compliance implemented items number of A/B department - View all courses schedule
related to usability compliance - View status of a student
- View a list of student prerequisite courses
compliance items
- View a list of curriculum
confirmed in
evaluation - View a list of student trustee
- View data of all employees
- View schedule of teaching (lecturer)
TABLE VI. ATTRACTIVENESS METRIC [4]
- View all subjects
Administra- - View biodata
Metric Name Parameter A Parameter Formula tion staff - Change student password
B - View a list of student prerequisite courses
Attractive - Result = - View status of a student
Questionnaire to
interaction Total of - View all courses schedule
assess the
parameter - View a list of curriculum
attractiveness of
A - Print a list of attendees
the interface to
- View a list of student trustee
users, taking
- View schedule of teaching (lecturer)
account of
- View all subjects
attributes such as
- View data of all employees
colour and
graphical design.
User Interface Number of types Total Result = The next step is to define the evaluation attributes.
appearance of interface number of A/B Attributes in the evaluation is a piece of software to be
customisability elements that can types of
interface
measured. Usability of ISO/IEC 9126 has the following
be customised. attributes, features, interface, message, input, regulations, and
elements.
operations. These attributes will be measured based metrics
are provided. These attributes can be obtained from the
documentation of the software in the first step. Ways to get the
III. USABILITY CHARACTERISTICS EVALUATION OF attribute is make group from all attributes by type. Because the
AIS BASED ON ISO/IEC 9126 evaluation process, these attributes will be assessed behavior
Evaluation in this study is conducted at the student based on ISO/IEC 9126. The output of this step is group of
administration module AIS. Characteristics used in the attributes.
usability evaluation based on the ISO/IEC 9126. Workflow of The next step is to evaluate the usability on student
the evaluation can be seen in Figure 1. administration module AIS. The evaluation process needs
The first step in the evaluation is collecting software usability evaluation tool of the ISO/IEC 9126 [12]. This
documentation, such as software requirements specification evaluation can be performed by evaluators, which is in charge

366
of evaluating the software. Evaluators are required which have
the ability to know the behavior of the software. The results of
this evaluation are needed by requirers and developers [8].
Step by step to perform the evaluation is insert the data
attributes that have been identified in the previous step. Then
answer questions about the behavior associated software has
been provided by usability tool. Questions are presented in
general as in Table 2 through Table 6. The output of this step
is the value of the evaluation. The result of the evaluation is
the assessment of the level of usability of the AIS student
administration module using ISO/IEC 9126.
The next step is to analyze the results of the evaluation.
The analysis was done manually based on the values issued by
usability tool. If the value is perfect, it is not necessary to
change the software. If the resulting value of zero, it is
necessary evolution or changes to the software on the some
parts based on sub characteristics value in order to get
maximum results.
Fig. 1. Evaluation workflow
IV. RESULT
The evaluation was performed on all metrics related. There
are several metrics that are ignored because they do not have
the required attributes. Based on result for each sub
characteristics, AIS student administration module has three
values between 0 and 1, one zero value, and one sub
characteristics ignored. Overall value per sub characteristics
can be seen in Table 10.
In Table 10 there is a weight that indicates the degree of
influence of each sub characteristics. Weights 3 shows that the
degree of influence of a high sub characteristics. While 2 is to
show the level of influence is. Weights 1 indicate a low level
of influence. The weights will be multiplied by the original
value of the sub characteristics produce the final result. The
value of N/A indicates that the metric is not used in the
evaluation.
In Table 10 there is a value of 0 on the learnability. A value
of 0 is generated because the metrics associated with
learnability produce a value of 0 or no corresponding attribute
metric qualification. In addition, there are also sub
characteristics ignored because it has no attributes required by
the measurement.
Sub characteristics which have value between 0 and 1 are
understandability, operability, and attractiveness. Value for
understanding is 1,333 from a perfect score 2 It shows that the
understandability is medium and require effort to improve the
behavior of the attribute to be perfect. Value for operability is
2,056 of perfect 3. This value indicates the level of importance
operability is high and the need for improved behavioral
attributes to fit the qualification of ISO / IEC 9126. The last is
the attractiveness. The result is 0.917 of a perfect score 2 This
suggests that sub characteristics have medium weight. To
approach the applicable standards of the IS0/IEC 9126,
required changes to metrics related. Comparison of the final
results of the evaluation with a perfect score per for each sub
characteristics based on ISO/IEC 9126 can be seen in Table 8.
TABLE VIII. EVALUATION PER SUB CHARACTERISTIC ON
USABILITYAND COMPARE WITH IDEAL RESULT
Sub Chaacteristics Ideal Real Description
Value Result
Understandability 2 1.333 incomplete
Learnability 1 0.0 not
implemented
Operability 3 2.056 incomplete
Atttractiveness 2 0.917 incomplete
Usability 3 N/A no evaluation
compliance
Evaluation results per metric shows that there are five
metrics are ignored, namely demonstration capability, user
operation cancellability, operation status monitoring
capability, operational error recoverability, and usability
compliance. In addition, there are five metrics that are not
implemented. This is caused by the behavior attribute that is
not qualified with metrics. Metrics that are not implemented
are completeness of description, completeness of user
documentation and/or help facility, physical accessibility,
operational consistency, and user interface appearance
customizability. Therefore, the value of the metric is 0.
In addition, there is a metric that has a perfect score or 1.
This is happened because the all attributes has been
implemented in accordance with the requirements given by the
metric. They are evident functions, function understandability,
user operation undoability, message clarity, and interface
element clarity.
Metrics which less valuable is a metric that has a value
between 0 and 1. There are input validity checking,
customizability, and attractive interaction. This is happened
because not all attributes qualify a given metric. The overall
367
score can be seen in Table 9. TABLE X. RESULT EVALUATION BASED ON SUB CHARACTERISTIC
Sub Result Weight FinaL Result
TABLE IX. RESULT EVALUATION Characteristics

Sub- Nama Metrik Result Description Understandability 0.667 2 1.333

characteris Learnability 0.0 1 0.0
-tics
Operability 0.685 3 2.056
Understand- Completeness 0 There are no
Attractiveness 0.458 2 0.917
ability of description description function
Understand- Demonstration N/A No Evaluation Usability N/A 3 N/A
ability capablity compliance
Understand- Evident 1 Evident functions
ability functions are complete
Understand- Function 1 Total of user V. CONCLUSION
ability understandabili interface which
ty clearly for user are Conclusion of this study are as follows.
complete 1. To do usability evaluation on AIS student administration
Learnability Completeness 0 There are no user module needs some data test, called attribute. Kind of
of user documentation or attribute which is used in this evaluation is feature,
documentation help facility interface, message, operation, input, and regulation.
and/or help
facility
2. AIS student administration module has less good result
Operability Input validity 0.9231 Total of input based on usability characteristics. It because there are a
checking validity are lot of metrics that have not been implemented properly in
incomplete the absence of supporting attributes. For further
Operability User operation N/A No evaluation development, AIS student administration module requires
cancellability an evolution on AIS in order to get the maximum result.
Operability User operation 1 Total of functions
Undoability which undoable are
complete
Operability Customisabilit 0.875 Total of functions References
y which customizable
are incomplete [1] Agus Budi Raharjo, "Rekayasa Ulang SIM Akademik ITS Berdasarkan
Operability Physical 0 There are no Karakteristik Pemeliharaan Menggunakan Mode Kualitas ISO/IEC
accessibility functions which 9126,” Final Project Report, ITS, Surabaya, 2013.
physical accessible [2] Daniel Galin, Software Quality Assurance. England: Pearson Education
Operability Operation N/A No Evaluation Limited, 2004.
status [3] Syahrul Fahmy, Nurul Haslinda, Wan Roslina, and Ziti Fariha,
monitoring "Evaluating the Quality of Software in e-Book Using the ISO 9126
Model," International Journal of Control and Automation, vol. 5, no. 2,
capability
pp. 115-122, Juni 2012.
Operability Operational 0 There are no
[4] ISO/IEC JTC1, "Software engineering – Product quality," Japan, 2002.
consistency operation which
consistent [5] J´ acome Cunha, Jo˜ ao Paulo Fernandes, Christophe Peixoto, and Jo˜ ao
Saraiva, "A Quality Model for Spreadsheets," in Eighth International
Operability Message 1 Total of messages Conference on the Quality of Information and Communications
Clarity clarity are complete Technology, 2012, pp. 232-236.
Operability Interface 1 Total of interface [6] Yiannis Kanellopoulos and et.al, "Code Quality Evaluation
element clarity element clarity are Methodology Using the ISO/IEC 9126 Standard," Intl Journal of Soft.
complete Eng. & Applications (IJSEA), vol. 1, no. 3, pp. 17-36, Juli 2010.
Operability Operational N/A No Evaluation [7] Bee Bee Chua and Laurel Evelyn Dyson, "Applying the ISO 9126 model
error to the evaluation of an e-learning system ," in ASCILITE, 2004.
recoverability [8] ISO/IEC JTC1, "Software engineering –Product quality -Part 3: Internal
Attractive- Attractive 0.9167 Total of attractive Metrics," ISO/IEC JTC1, Japan, 2002.
ness interaction user interface is [9] R. S. Pressman, Software Engineering – A Practitioner’s Approach,
incomplete European adaptation by D. Ince. London: McGraw-Hill Intl, 2000.
Attractive- User Interface 0 There are no user [10] IEEE, "IEEE Standard Glossary of Software Engineering Terminology,"
appearance interface appearance The Institute of Electrical and Electronics Engineers, New York, IEEE
ness Software Engineering Standards Collection 1991.
customisability customizable
Usability Usability N/A No Evaluation [11] Badan Teknologi dan Sistem Informasi (BTSI) ITS, "ITS Academic
Information Systems at a Glance," Technical Report, ITS, 2013.
compliance compliance
[12] Hanifa Iradatur Rahmani, Umi Laili Yuhana, and Siti Rochimah,
"Rancang Bangun Kakas Bantu Evaluasi Karakteristik Kebergunaan
Menggunakan ISO/IEC 9126," Final Project Report, ITS, Surabaya,
2014.

368