Measurement of e-learning quality based on

ISO 19796-1 using fuzzy analytical network

process method
Cite as: AIP Conference Proceedings 2014, 020155 (2018); https://doi.org/10.1063/1.5054559
Published Online: 21 September 2018

Eza Rahmanita, Novi Prastiti, and Moh. Adi Purnomo

ARTICLES YOU MAY BE INTERESTED IN

Simulation study of electron beam spot size from thermionic electron gun using SIMION 8.1
software
AIP Conference Proceedings 2014, 020156 (2018); https://doi.org/10.1063/1.5054560

The application of problem based learning to improve students’ self-efficacy

AIP Conference Proceedings 2014, 020024 (2018); https://doi.org/10.1063/1.5054428

An analysis of students’ conceptual understanding: How do students understand some

electricity concepts?
AIP Conference Proceedings 2014, 020154 (2018); https://doi.org/10.1063/1.5054558

AIP Conference Proceedings 2014, 020155 (2018); https://doi.org/10.1063/1.5054559 2014, 020155

© 2018 Author(s).
 Measurement Of E-Learning Quality Based On ISO 19796-1
Using Fuzzy Analytical Network Process Method
Eza Rahmanita1, a) Novi Prastiti1, b) Moh. Adi Purnomo1, c)

1
Department Of Informatics, Faculty of Engineering, Trunojoyo University, Madura Indonesia
a)
Corresponding author: eza.rahmanita@trunojoyo.ac.id
b)
prastitinovi@trunojoyo.ac.id
c)
moh.adipurnomo@gmail.com

Abstract. The development of information and communication technology is now capable of improving the quality of service in various
fields, especially in the field of education, such as the use of e-learning. Currently, there are many types of e-learning that can be used,
with different qualities. The quality of e-learning greatly influences the success rate of e-learning application for a learning process,
education and training. Measurement of e-learning quality becomes very important, therefore required of standard in the measurement of
e-learning quality. ISO 19796-1 is a guideline to build a quality e-learning system. ISO 19796-1 is used for uniformity of quality
assessment to the quality assessment becomes standardized, and a template for the process of creating, implementing and improving the
quality of e-learning. The categories in the ISO 19796-1 process model are used to define e-learning quality assessment criteria. The
weighting criteria are processed using Fuzzy Analytical Network Process (FANP) method until it becomes the final weight. From a
series of quality tests E-Learning of Madura University obtained the results of the quality value 79.7, thus e-learning quality conversion
results show quality results are GOOD. While recommendation improvement suggested by the system based on a test result that is 12
recommendation improving the quality of E-Learning of Madura University.

INTRODUCTION

The rapid development of this technology utilization by many educational institutions is to support the learning process such
as the use of e-learning. E-learning is a mediator that connects information and service resources with users that can be accessed
from anywhere and anytime. Currently many types of e-learning can be used therefore required a standard in the use of e-
learning, both development standards and e-learning quality measurement standards.1
Measuring the quality of e-learning greatly affects the successful rate of e-learning implementation for the learning process,
education and training. Thus, the measurement of e-learning quality is very important. Many e-learning standards provide many
options for educational institutions to determine which standard to use. Besides not easy to determine which standardization to
use also require careful attention and consideration.1
ISO 19796-1 is a guideline to build a quality e-learning system. ISO 19796-1 was published in October 2005 through the
SC36 team and is more specific in Learning, Education and Training, including e-learning. ISO 19796-1 provides RFDQ
(Reference Framework for Description of Quality) to improve the quality of e-learning systems. As a reference standard, ISO
19796-1 shows the description scheme and process model that can be used as a roadmap for building e-learning systems.2 The
Use of ISO 19796-1 is not for certification, but as a tool to provide uniform quality appraisal, for quality assessment to be
standardized, and a template for the process of creating, implementing and improving the process of e-learning
Used ISO 19796-1 standardization in this study is not intended for certification, but rather a tool that provides uniformity of
quality assessment, data format for quality assessment become standardized, and become a template for the process of making,
implementing and improving the quality of e-learning process in an educational institution.
The priority weights on the process can be adjusted according to the institution. The priority weight of the process is then
processed by using Fuzzy Analytical Network Process (FANP) method until it becomes the final weight used to determine the
priority of e-learning recommendation.

International Conference on Science and Applied Science (ICSAS) 2018

AIP Conf. Proc. 2014, 020155-1–020155-9; https://doi.org/10.1063/1.5054559
Published by AIP Publishing. 978-0-7354-1730-4/$30.00

020155-1
 MATERIAL AND METHODS

ISO 19796-1

ISO 19796 consists of several publications. The relationship and relevance of each ISO 19796 publication can be seen in
Figure 1.

FIGURE 1. The relationship between ISO standard 197962

ISO 19796-1 is a guideline to build a quality e-learning system. ISO 19796-1 was published in 2004 through the SC36 team
and is more specific in Learning, Education, and Training (including e-learning). ISO 19796-1 provides RFDQ (Reference
Framework for Description of Quality) to improve the quality of e-learning systems. As a reference standard, ISO 19796-1 shows
the description scheme and process model that can be used as a roadmap to build a comprehensive e-learning system.4 5
There are 7 categories in ISO 19796-1 process model (ISO/IEC, 2005). The seven categories are NA (Need Analysis), FA
(Framework Analysis), CD (Conception/design), DP (Development/production), IM (Implementation), LP (Learning process),
and EO (Evaluation/optimization).

Fuzzy Analytical Network Process (FANP)

Fuzzy ANP method is applied for an extension of the AHP and ANP by combining the fuzzy set theory. In the ANP Fuzzy,
Fuzzy ratio scale used to indicate the relative strength of the factors on which the relevant criteria. The fuzzy decision so that a
matrix can be formed. Kahit of alternatives are also presented in the Figures Fuzzy.6 Based Chang each object of each criterion
and sub-criteria to be considered and extend the analysis to obtain a goal executed. This means it is possible to obtain the analysis
which can extend the value indicated by the notation as follows
, , , , ………. (1)
Set a goal (1,2,3,4,5......m) and (j=1,2,3,4,5....m) are triangular fuzzy (Triangular Fuzzy Number), after identifying initial
assumptions, extend the analysis of Chang can be described with the following stages:

1. Development a pairwise comparison matrix between all the elements/criteria, sub-criteria of the fund each dimension
criteria in a hierarchical system based on an assessment of linguistic variables

FIGURE 2. Pairwise Comparison Matrix

Where
n = number of criteria to be evaluated
Ci = i. Criteria
Aij = interests of i. criteria based j

020155-2
2. Change the linguistic variables in the form of fuzzy numbers. Questionnaire data in the form of linguistic variables
fuzzy numbers are converted to forms. TFN fuzzy numbers to be seen (the scale of the fundamental interests of
Relative ANP) with a different level of importance.
TABLE 1. Triangular fuzzy number (TFN) scale and linguistic variables scale conversion
Values interest in
Linguistic Scale TFN Scale TFN inverse scale
ANP
Equally important 1 (1,1,1) (1,1,1)
A little more important 3 (1,3,5) (1/5,1/3,1)
More important 5 (3,5,7) (1/7,1/5,1/3)
Very important 7 (5,7,9) (1/9,1/7,1/5)
The most important 9 (7,9,9) (1/9,1/9,1/7)

3. Enter the search criteria and the weighting formula contained in steps - steps as follows:6
a) Determining the value of synthetic extend (the) associated with the object to i then represented as follows:
-1
= ∑ =1 ∑ ∑ (2)
To get this ∑ done adding fuzzy operation of m with particular matrix
∑ = ∑ ,∑ ,∑ (3)
-1
To get ∑ ∑ surgery fuzzy value from (j= 1,2,3...m)
∑ ∑ = ∑ ∑ ∑ (4)
At the end of the first step of the determinant of the inverse vector
∑ ∑ −1
= ∑ ,∑ ,∑ (5)
b) Determining the degree of likelihood (degree of possibility) and fuzzy set m2 = (l2, m2, u2) ≥ M1 = (L1, M1, U1)
is defined as V (m2 ≥ M1) = sup y≥x [min (μm1 (x), μm2 (y))] x and y is the value on the axis of each
membership function. Applied to the theory and applications of fuzzy TFN with 3-type of low, medium and upper
(l, m, u) and membership functions have been formed with the following equation :
V (m2 ≥ m1) = hit (m1 ∩ m2) = λm2(d)
1, ≥
V( ≥ )= 0,
≥ (6)
( ) ( ) ℎ
c) Determining the degree of likelihood for Convex fuzzy number is greater than k at Convex fuzzy number for Mi =
(i=1,2..k) can be defined as:
V = (M ≥ M1 , M2........Mk)
= V [(M ≥ M1)] and [M ≥ M2] and (M ≥ Mk)]
= min V (M ≥ Mi ) (7)
It is assumed that d`= min V (Si ≥ Sk)
For k = 1,2,..., n k ≠ i then the weight vector used
W`= (d`(Ai), d`(A2), d` (A3),... ,d`(An))T
Where Ai (i = 1,2,3,.., n) is an element n
d) Through normalization, weighting vector normalization
W` = (d (A1), d (A2),... ... ..., d(An))T
Where we are non fuzzy numbers.
e) Determination of global sub-criteria weights matrix by multiplying matrix interdependence with WT2 (weighted
sub-criteria)
f) Measurement of sub-criteria using linguistic variables by multiplying the weight of global (global weight) with a
value scale (scale value) each sub-criteria.
g) Consistency test is done by looking at the value of l, m and u. Value l ≤ m ≤ u shows fuzzy consistent ratings.
Fuzzy comparison matrix that consists of two dimensions, consistency index are always consistent

020155-3
 RESULT AND DISCUSSION

E-Learning Quality Testing

Quality testing of e-learning aims to measure the quality of e-learning at the University of Madura using applications that
have been made.

Weighting By Expert

The value of each process is determined by the expert, in which the research is conducted by Quality Assurance of Trunojoyo
Madura University according to the level of importance of comparison between processes. The comparative process is a
representation of the quality process of e-learning in the category of conception/design and learning process. Table 2 presents the
process data that will be assigned a weighted value by the expert:
TABLE 2. The process is in weight
Category Name Process Name
Conception / Design Learning Objectives
Concept for Contents
Didactical Concept/Methods
Organizational Concept
Technical Concept
Concept for Media And Interaction
Design
Media Concept
Communication Concept
Concept for Tests And Evaluation
Concept for Maintenance
Learning Process Administration
Activities
Review of Competency Levels

The expert gives an assessment of the process compared in the matrix. The assessment using the Saaty scale and then
transformed into triangular fuzzy number, described in Table 3 below:
TABLE 3. Fuzzy Scale Numbers
Fuzzy
Linguistic Scale TFN scale
Numbers
1 Equally important (1,1,3)
3 A little more important (1,3,5)
5 More important (3,5,7)
7 Very important (5,7,9)
9 The most important (7,9,9)

Weighting step:
1. Creating a pairwise matrix of pairs between processes
2. Assess the process compared to the matrix.

020155-4
 The following weighted results by the expert, are described in Table 4.
TABLE 4. Process Weight

Proses A B C D E F G H I J K L M

A 1 1 1 1 5 1 1 1 3 3 5 3 3
B 1 1 1 1 3 1 1 1 1 3 5 5 3
C 1 1 1 1 5 1 1 1 1 5 5 5 0.33
D 1 1 1 1 3 1 1 1 1 5 5 5 3
E 0.2 0.33 0.2 0.33 1 0.2 0.2 0.2 0.2 0.33 1 0.33 0.33
F 1 1 1 1 5 1 1 1 1 1 5 5 1
G 1 1 1 1 5 1 1 1 1 5 5 5 1
H 1 1 1 1 5 1 1 1 0.33 5 5 5 5
I 0.33 1 1 1 5 1 1 3 1 5 5 5 1
J 0.33 0.33 0.2 0.2 3 1 0.2 0.2 0.2 1 5 0.33 0.33
K 0.2 0.2 0.2 0.2 1 0.2 0.2 0.2 0.2 0.2 1 0.2 0.2
L 0.33 0.2 0.2 0.2 3 0.2 0.2 0.2 0.2 3 5 1 1
M 0.33 0.33 0.33 0.33 0.33 1 1 0.2 1 3 5 1 1

Next for each process, determine all processes in each category and relationships between processes in accordance with the
influence of interdependence between processes, where 1 means exist and 0 means no exist.

The following is the result of the relationship between processes by the expert, described in Table 5.
TABLE 5. Process Addiction Relationship

Proses A B C D E F G H I J K L M

A 1 1 1 1 1 1 1 1 1 1 0 1 1
B 1 1 1 1 1 1 1 1 1 0 0 1 1
C 1 1 1 0 1 1 0 0 0 0 0 1 1
D 1 1 0 1 0 0 1 1 1 0 1 1 0
E 1 1 1 0 1 1 0 0 1 1 0 1 1
F 1 1 1 0 1 1 1 1 1 0 1 1 1
G 1 1 0 1 1 1 1 1 1 0 1 1 0
H 1 1 0 1 0 1 0 1 1 1 0 1 1
I 1 1 0 1 1 1 1 1 1 0 0 1 1
J 1 0 0 0 1 0 1 1 0 1 1 1 0
K 0 0 0 1 0 1 0 0 0 1 1 0 0
L 1 1 1 1 1 1 1 1 1 1 0 1 1
M 1 1 1 0 1 1 1 1 1 0 0 1 1

Alphabets A through M represents each e-learning quality process based on ISO 19796-1.
Information:
A = Learning Objectives H = Communication Concept
B = Concepts of Content I = Concept for Test and Evaluation
C = Didactical Concept J = Concept of Maintenance
D = Organizational Concept K = Administration
E = Technical Concept L = Activities
F = Concept for Media and Interaction Design M = Review of Competency Levels
G = Media Concept

020155-5
 Then performed the calculations with the stages performed in the Fuzzy Analytical Networks Process (FANP) method. To
produce the final weights of each process used to measure the quality of an e-learning. The results from the calculations of
applications made with the Fuzzy Analytical Networks Process (FANP) method are shown in Table 6.
Table 6. Results of the End Weights of Each Process
Process Name End Weight
Learning Objective 0.1303
Concept Of Content 0.1231
Didactical Concept/method 0.0587
Organizational Concept 0.0809
Technical Concept 0.0197
Concept For Media And Interaction Design 0.0986
Media Concept 0.0944
Communication Concept 0.1365
Concept For Test And Evaluation 0.1121
Concept Of Maintenance 0.0148
Administration 0.009
Activities 0.0433
Review Of Competency Levels 0.0786

Assessment by Respondents

The questions contained in this assessment are taken from the instruments made by Cahyani et al.2 Each question represents
the e-learning quality category according to ISO 19796-1.
To assess the quality of e-learning, respondents simply answer the question with Yes or No answer. Each answer Yes will get
a value of 1 and every answer No get a value of 0.
The calculation of the value of each process is done with the following stages:
1. Criteria Value Each Process = Total Value Criteria Each Process
2. Value Each Process = Value Criteria Each Process / Number of Criteria Each Process

Here are the results of quality assessment e-learning The University of Madura (elearning.unira.ac.id). An assessment
conducted by 20 respondents, consisting of lecturers and students of Madura University, is shown in Table 7.
Table 7. Respondents Rating Results
Respondent A B C D E F G H I J K L M
1 1 0.5 0.67 0.8 0.5 0.75 1 0 0.5 0 1 1 0.5
2 1 1 1 1 0.5 0.5 1 1 1 0 0.67 0.5 0
3 1 1 1 1 1 1 1 1 1 1 0.67 1 1
4 1 0.5 1 0.6 0.5 0.75 1 1 1 0 0.67 0.5 0.5
5 1 0.5 1 0.8 1 1 0 1 1 1 1 0.5 1
6 0.5 1 1 1 1 1 1 1 1 1 1 1 0.5
7 0.5 1 1 0.6 1 1 1 1 0.5 1 0.67 1 0.5
8 1 1 0.67 0.8 0.5 0.75 1 1 0.5 1 0.33 1 1
9 1 1 0.67 1 1 1 1 1 1 1 1 1 1
10 1 0 0.67 0.8 1 0.75 0 1 0.5 1 1 0.5 0.5
11 1 0.5 0.67 0.6 0.5 0.5 1 1 1 0 0.33 1 0
12 1 1 1 1 1 1 1 1 1 1 1 1 1
13 1 1 0.33 0.8 0.5 0.75 1 1 0 1 0.67 1 0
14 1 1 1 0.6 1 0.5 1 1 1 0 0.67 1 1
15 1 0.5 0.33 0.8 1 0.75 0 1 0.5 1 0.67 0.5 0.5
16 1 0.5 0.67 1 0.5 0.5 0 1 0.5 1 0.67 0 0.5
17 1 0.5 1 1 0.5 1 0 1 1 1 0.67 1 1
18 1 0.5 0.67 0.4 0.5 0.25 0 1 0.5 0 0.67 0.5 0.5
19 1 1 1 0.8 0.5 1 1 1 1 1 0.33 1 0.5
20 1 0 1 0.8 1 0.75 1 1 1 0 1 1 1

020155-6
 Alphabets A through M represents each e-learning quality process based on ISO 19796-1.
Information:
A = Learning Objectives H = Communication Concept
B = Concepts of Content I = Concept for Test and Evaluation
C = Didactical Concept J = Concept of Maintenance
D = Organizational Concept K = Administration
E = Technical Concept L = Activities
F = Concept for Media and Interaction Design M = Review of Competency Levels
G = Media Concept

E-Learning Quality Testing Results

The results of e-learning quality testing aims to display the value of e-learning quality and recommendation improvement on
E-Learning University of Madura:
The system calculates the total value of the quality in the following manner:
1. Value Process Quality = Average Value of Each Process from Respondent * End Weight Each Process from Expert
2. Total Quality Value = Σ Value of Process Quality

Test results as in Figure 3. In the figure shows that the total value of E-Learning quality of University of Madura obtained is
79.7 of scale 100.

FIGURE 3. Results of E-Learning Assessment of Madura University

The e-learning quality measurement application created in addition to displaying the total value of quality also displays the
value of detail per category so that e-learning managers can analyze the deficiencies contained in e-learning in each category of
e-learning quality based on ISO 19796-1 standard, each category is shown in Table 8.
TABLE 8. Details of Each Category Quality Value
Category Name Process Name Value
Conception / Design Learning Objectives 12.38
Concept for Contents 8.62
Didactical Concept/Methods 4.8
Organizational Concept 6.55
Technical Concept 1.48
Concept for Media And Interaction Design 7.64
Media Concept 6.61
Communication Concept 12.96
Concept for Tests And Evaluation 8.69
Concept for Maintenance 0.96
Learning Process Administration 0.66
Activities 3.46
Review of Competency Levels 4.91
Total Quality Value 79.7

020155-7
 Applications created may also provide recommendations that can be used as an improvement material to improve the quality
of e-learning being tested, the recommended recommendation results are shown in Table 9.
TABLE 9. Results of Quality Recommendations of E-Learning University of Madura
No Recommendations Improvement
1 Increase the number of learning materials per course to fit the SAP curriculum/standards at least 80% of the total
number of meetings
2 Complete the main reference features and supporting references for each learning material
3 Include sample case studies on learning materials
4 Complete the system with a variety of learning strategies, (e.g. discussion/question, problem based learning,
collaborative learning, student review/summary, visual information, examples and analogy, online discussion)
5 Complete the reference resource facility for each learning material
6 Complete the facilities of learning media variations (e.g. ppt, pdf, doc, etc.)
7 Complete the facility of explanation of learning materials to facilitate the understanding of the material
8 Complete the feature to practice the questions of each material to evaluate learning outcomes
9 Evaluate the learning outcomes of each material more than 1x
10 Complete the feature to show lesson scenarios in previous periods
11 Complete the facility of the assessment rules mechanism
12 Complete the facilities of the task collection mechanism

From the results of testing the quality of e-learning above, it can be compiled e-learning quality conversion in accordance
with Table 10 below:
TABLE 10. Quality Conversion
Value Range Interpretation
0 – 20 Very bad
21 – 40 Bad
41 – 60 Enough
61 – 80 Good
81 – 100 Very good

From the result of e-learning quality value data above, it can be concluded that the result of quality value E-Learning of
Madura University with the range of quality values show the number of 79.7 it shows the results obtained is GOOD because it is
in the range 61-80.

SUMMARY

From the results of designing and manufacturing of E-Learning Quality Measurement Application Based on ISO 19796-1
Using Fuzzy Analytical Network Process Method, and from a series of quality tests of E-Learning utilization at The University of
Madura, the result of quality score reaches 79.7. Therefore, the results of e-learning quality have shown that the quality results
are GOOD. Recommendation improvements suggested by the system based on test results generate 12 Recommendations quality
improvement of E-Learning at the University of Madura.

ACKNOWLEDGMENTS

The authors thank Madura University for research permission. Special thanks to Trunojoyo University for facilitate this
research.

REFERENCES

[1] E. S. Sitorus, e-Proceeding of Engineering, 2(1), 1643, (2015).

[2] D. Cahyani, D. O. Siahaan, and Sarwosri, PhD. thesis, Institut Teknologi Sepuluh Nopember, Surabaya, 2010.
[3] R. Suteja dan . A. Harjoko, in Seminar Nasional Aplikasi Teknologi Informasi (Universitas Islam Indonesia,
2008).

020155-8
[4] A. Wicaksono, Application Development Analysis E-Learning Quality Assessment Based on ISO 19796-1,
Universitas Negeri Yogyakarta, Yogyakarta, 2014.
[5] ISO/IEC, “ISO 19796-1 Information technology - Learning, education, and training - Quality management, assurance, and
metrics - Part 1: General approach,” Geneva, 2005.
[6] Y. Kustiyahningsih, E. Rahmanita, and J. Purnama, “Fuzzy Anp Method And Internal Business Perspective
For Performance Measurement In Determining Strategy SMEs,” in Proceeding The 1st IBSC: Towards The
Extended Use Of Basic Science For Enhancing Health, Environment, Energy, And Biotechnology (2016)
(IBSC, 2016). pp. 289- 294.
[7] Y. Aprilinda, Jurnal Manajemen Sistem Informasi dan Teknologi, 34-44, (2011).
[8] Y. Kustiyahningsih, D. Kartika dan F. Adiputra, “E-Learning Based ISO 19796-1 With Simple Additive
Weighting Method (SAW),” in Proceedings The 5th SNST Seminar Nasional Sains dan Teknologi UNWAHAS
(2014) (SNST, 2014), pp. 1-6.
[9] A. F. Abror dan H. Jati, Jurnal Pendidikan Vokasi, 6 (1), 1-14, (2016).
[10] A. H. Suyanto, Journal Komputer, 1-9 (2015).
[11] R. Ritzkal, A. Goeritno dan E. H. P, “Software Quality Measurement of E-Learning System Using Metric
Function Oriented,” in Proceedings The 4th SNATIF Seminar Nasional Teknologi dan Informatika (2017)
(SNATIF, 2017), pp. 769-776.

020155-9