Available online at www.sciencedirect.

com
Available online at www.sciencedirect.com
ScienceDirect
ScienceDirect
Procedia
Available Computer
online Science 00 (2019) 000–000
at www.sciencedirect.com
Procedia Computer Science 00 (2019) 000–000 www.elsevier.com/locate/procedia
www.elsevier.com/locate/procedia
ScienceDirect
Procedia Computer Science 161 (2019) 1225–1232

The Fifth Information Systems International Conference 2019

The Fifth Information Systems International Conference 2019
Evaluating on User Experience and User Interface (UX/UI) of
Evaluating on User Experience and User Interface (UX/UI) of
EnerTrApp a Mobile Web Energy Monitoring System
EnerTrApp a Mobile Web Energy Monitoring System
Kristine Mae P. Escanillan-Galeraa,a,*, Cenie M. Vilela-Malabananbb
Kristine Mae P. Escanillan-Galera *, Cenie M. Vilela-Malabanan
a
Picardal Institute of Science and Technology, Tibanga, Iligan City 9200, Philippines
b
Mindanao
a
State Institute
Picardal University – Iligan and
of Science Insitute of Technoloby,
Technology, Tibanga,
Tibanga, Iligan9200,
Iligan City City Philippines
9200, Philippines
b
Mindanao State University – Iligan Insitute of Technoloby, Tibanga, Iligan City 9200, Philippines

Abstract
Abstract
The EnerTrApp mobile web application prototype was designed and developed where consumers can monitor the energy
The EnerTrApp
consumption mobile
of the homeweb application
appliance usingprototype was designed
their smartphones throughand the
developed where consumers
mobile website can monitor
of the system. the energy
The prototype was
consumption
implemented and of the
the home
mobileappliance using their
web application smartphones
user interface of thethrough
prototypethewas
mobile website
evaluated of the
through system.testing
usability The prototype
to asses forwas
its
implemented and the mobile
effectivity, efficiency and userweb application
satisfaction. userthe
From interface
analysisof of
thetheprototype wasfound
data, it was evaluated through
out that usability
EnerTrApp testingweb
mobile to asses for its
application
effectivity, efficiency
user interface is 100%and user satisfaction.
effective as all tasksFrom
werethe analysis of
completed bytheall data, it was found
participants. It is out that EnerTrApp
efficient mobile
based on the web application
time-base efficiency
user interface
measure is 100%
and overall effective
relative as all tasks
efficiency. were data
Statistical completed
shows bythatalltesters
participants. It iswith
as satisfied efficient based
the user on the
mobile webtime-base
app userefficiency
interface.
measure
Based o and overall consumers
the results relative efficiency.
and willStatistical
likely usedata
andshows that testers
recommend as satisfied
the mobile web with the userasmobile
application a toolweb app reduce
to help user interface.
energy
Based o the results
consumption in theirconsumers
household.and will likely use and recommend the mobile web application as a tool to help reduce energy
consumption in their household.
© 2019 The Authors. Published by Elsevier B.V.
© 2019
© 2019 The
The Authors.
Authors. Published by by Elsevier B.V.
This is an open accessPublished
article under Elsevier B.V.
the CC BY-NC-ND license (http://creativecommons.org/licenses/by-nc-nd/4.0/)
This is an open access article under the CC BY-NC-ND license (http://creativecommons.org/licenses/by-nc-nd/4.0/)
This is an open
Peer-review access article under CC BY-NC-ND licenseThe(http://creativecommons.org/licenses/by-nc-nd/4.0/)
Peer-review under responsibility of the scientific committee ofofThe
under responsibility of the scientific committee Fifth
Fifth Information
Information Systems
Systems International
International Conference
Conference 2019
2019.
Peer-review under responsibility of the scientific committee of The Fifth Information Systems International Conference 2019
Keywords: Home Energy Monitoring; Usability Testing; Mobile Web Application; Mobile Web Usability Testing
Keywords: Home Energy Monitoring; Usability Testing; Mobile Web Application; Mobile Web Usability Testing

1. Introduction
1. Introduction
Electricity remains to be the most widely used fuel in the households with 99.9 percent of the total number of
Electricity
homes remains tofor
using electricity be lighting,
the most89.8
widely used for
percent fuelhousehold
in the households with
recreation, and99.9
74.6percent
percentoffor
thespace
total cooling/air
number of
homes using [1]
conditioning electricity for lighting,
in the Philippines 89.8 percent for household recreation, and 74.6 percent for space cooling/air
alone.
conditioning [1] in the Philippines alone.

* Corresponding author. Tel: +63-908-470-9463.

* E-mail kristinemae.galera@g.msuiit.edu.ph
address:author.
Corresponding Tel: +63-908-470-9463.
E-mail address: kristinemae.galera@g.msuiit.edu.ph
1877-0509 © 2019 The Authors. Published by Elsevier B.V.
This is an open
1877-0509 access
© 2019 Thearticle under
Authors. the CC BY-NC-ND
Published license (http://creativecommons.org/licenses/by-nc-nd/4.0/)
by Elsevier B.V.
Peer-review
This under
is an open responsibility
access of the scientific
article under CC BY-NC-NDcommittee of The
license Fifth Information Systems International Conference 2019
(http://creativecommons.org/licenses/by-nc-nd/4.0/)
Peer-review under responsibility of the scientific committee of The Fifth Information Systems International Conference 2019

1877-0509 © 2019 The Authors. Published by Elsevier B.V.

This is an open access article under the CC BY-NC-ND license (http://creativecommons.org/licenses/by-nc-nd/4.0/)
Peer-review under responsibility of the scientific committee of The Fifth Information Systems International Conference 2019.
10.1016/j.procs.2019.11.236
1226 Kristine Mae P. Escanillan-Galera et al. / Procedia Computer Science 161 (2019) 1225–1232
2 Author name / Procedia Computer Science 00 (2019) 000–000

With the rise of energy consumption, the Philippine Government released an Administrative Order no. 110 to
reduce monthly energy consumption by at least 10%, and with this different information drive campaigns have been
launched. According to Petersen, Shunturov, Janda, Platt, and Weinberger [2], real-time feedback is the key to guiding
consumers in energy conservation and enables them to change their usage behavior and to decrease their demand in
power. It is expected that by supplying detailed and immediate feedback, between 5% and 15% of the electrical
household energy consumption can be saved [3].
EnerTrApp an energy monitoring system that utilizes microcontrollers used in mobile devices through a mobile
web graphical user interface, where users can access real-time data and monitor it using different browsers on different
mobile devices [4]. The mobile web energy monitoring system is a mobile web application that aims to help reduce
residential energy consumption [5].
Based on the data analytics during the implementation of EnerTrApp as a tool to help reduce energy consumption
there was a relevant decrease in energy consumption [5] based on the short load forecasting analysis as predictions on
future energy consumption [6] and prescriptive action list. Evaluating a mobile application in terms of usability,
usability testing is an instrument that is commonly used. A usability test was conducted to evaluate the mobile web
application with regards to the experience of the users while using the mobile web application prototype. The usability
test also evaluates the effectiveness of the system and satisfaction of use, to verify if consumers will likely use the
mobile web application as a tool to help reduce energy consumption in their household.
A quantitative evaluation of attributes such as usability will have some measurement [7]. Usability in terms of a
product is the quality it possesses. A usable product does not break easily and can be used without any difficulty.
There are many ways to measure the quality of a product. Some standards need to be followed to make sure that
products released are of good quality. Having standard usability objectives are defined and measurable with one or
more attributes. The most common usability goals and objectives that are used by different usability models have the
following criteria: usefulness, efficiency, effectiveness, satisfaction and accessibility [8]. Usability testing is a process
to evaluate a product or a system.
Mobile devices have rapidly become popular to consumers the same way as web technology has advanced [9].
Mobile web application is also called a responsive website, as they are designed first for mobile device screens but
can also work as a regular desktop website. Usability testing for mobile web applications is the same as usability
testing for regular websites. It also measures metrics on how user-friendly the mobile website is. In their review of
empirical mobile usability studies, Coursaris & Kim [10] presented a mobile usability testing the three most commonly
used measures: efficiency, effectiveness and satisfaction. The three attributes affect the overall design of the system.
They also affect the interaction of the users with the system itself [11].
The goal of this study is to evaluate the EnerTrApp mobile web application user interface usability and to assess if
it is (1) efficient, (2) effective, and (3) satisfying based on the overall user experience when using the EnerTrApp
mobile web application. The next section presents the review of related literature on usability testing methodologies.
The research design and methodologies used in this chapter are discussed.

2. Methodology

A quantitative process as shown in Fig. 1. The input of the research plan is the developed mobile web application,
while the output is statistical data.

Fig. 1. Evaluation of EnerTrApp research design.

 Kristine Mae P. Escanillan-Galera et al. / Procedia Computer Science 161 (2019) 1225–1232 1227
Author name / Procedia Computer Science 00 (2019) 000–000 3

According to Fig. 1, the process evaluates or measures the effectiveness of the system, the experience of the users
while using the mobile web app and assess if the system is an excellent tool and if the customers will likely use the
EnerTrApp mobile web application as a tool to reduce home energy consumption. Statistical data is gathered and
analyzed, and the outcome is the evaluated mobile web application energy monitoring system user interface.
The study employed a quantitative evaluation and purposive sampling in choosing the participants to evaluate the
EnerTrApp mobile web application user interface usability as shown in Fig. 2. The selection of participants was done
as part of the sampling process for the usability testing. In data collection the sampling frame helps identify the
elements included in the sample [12]. Purposive sampling is a judgmental or subjective sampling and is commonly
used in quantitative research. The main goal of purposive sampling is focusing on specific characteristics on the
population of interest that can answer the research question [13].

Fig. 2. EnerTrApp mobile web user interface.

The selection of the usability metrics for the study is based on the ISO 9241-11:2018 [14] revised from ISO9241
[15]. It recommended that the metrics for usability are effectiveness, efficiency and satisfaction. These metrics were
selected since the attributes of errors from [16] model is associated with effectivity.
A literature study by Kjeldskov & Graham [17] revealed that most (71%) mobile device evaluations were done in
laboratory settings. Usability testing for this study was conducted in a laboratory set-up at a private college institute
computer laboratory. The participants were college students, faculty and staff of the institute who were randomly
approached and profiled using a questionnaire.
The questionnaire ensures that users have mobile phones and know how to use the Internet using their mobile
browser. This study had a total of 35 participants. The participants were either male or female with age group 18-50
years old and were mobile phone users. Participants were gathered and were provided with forms and instructions on
what will happen during the usability testing and any clarification was answered by the researcher before the usability
test began. Participants were given a list of tasks. They performed the specified tasks one at a time as the mobile
application has one user account per energy monitoring device.
The participants used a specific smartphone depending on their mobile device of choice it is either IoS or Android
operating system, in this study iPhone 4S and Samsung J5 android phone was used. The mobile phone needs to be
connected to the Internet. Photos were taken during the usability testing of the EnerTrApp mobile application. The
participants evaluated their satisfaction and user experience of the mobile web application using the System Usability
Scale form by Brooke [18]. SUS by Brooke is a 10-item scale of objective observation of user interaction with the
system; the users are presented with 5-point scales numbered from 1 (anchored with “Strongly disagree”) to 5
(anchored with “Strongly agree”) for each item [19]. It has five (5) negative statements and five (5) positive statements.
The study participants were provided with an individual evaluation sheet which is composed of 10 (ten) tasks, in
1228 Kristine Mae P. Escanillan-Galera et al. / Procedia Computer Science 161 (2019) 1225–1232
4 Author name / Procedia Computer Science 00 (2019) 000–000

which the individual participants control to switch the household appliances on/off using the EnerTrApp mobile web
user interface as shown in Fig. 3 during the testing stage.

Fig. 3. Participant individual evaluation sheet.

Each participant was given 30 minutes to complete the individual evaluation sheet. The effectiveness is measured
based on the number of tasks performed successfully by the participant. Effectiveness can be calculated as shown in
Equation 1.

������ �� ����� ��������� ������������

𝐸𝐸𝐸𝐸𝐸𝐸𝐸𝐸𝐸𝐸𝐸𝐸𝐸𝐸𝐸𝐸𝐸𝐸𝐸𝐸𝐸𝐸𝐸𝐸𝐸𝐸 𝐸 𝐸 × 100% (1)
����� ������ �� �����

Efficiency measures the time spent by the participant in performing the task using the mobile web application. It
can be calculated through time base efficiency as shown in Equation 2 and overall relative efficiency in Equation 3.

� ���
∑�
��� ∑��� ���
𝐸𝐸𝐸𝐸𝐸𝐸𝐸𝐸𝐸𝐸𝐸𝐸𝐸𝐸𝐸𝐸𝐸𝐸𝐸𝐸 𝐸 𝐸 (2)
��

Where:
R: the number of users
N: the number of tasks
nij: result for tasks (i) by the user (j).
If the task is completed, then nij = 1, else nij = 0.
tij: time spent by the user (j) to complete the task (i).
If user did not complete the task, time will be measured
until the moment the user gave up the task
�
∑�
��� ∑��� ��� ���
𝐸𝐸𝐸𝐸𝐸𝐸𝐸𝐸𝐸𝐸𝐸𝐸𝐸𝐸𝐸𝐸𝐸𝐸𝐸𝐸 𝐸 𝐸 ∑� � (3)
��� ∑��� ���

Evaluation of user’s experience and satisfaction of the mobile web application was assessed using the SUS by
Brooke [18]. The system usability scale score ranges from 1-100. To calculate the SUS score, first sum the score
 Kristine Mae P. Escanillan-Galera et al. / Procedia Computer Science 161 (2019) 1225–1232 1229
Author name / Procedia Computer Science 00 (2019) 000–000 5

contributions from each item. Each item's score contribution ranges from 0 to 4. For positive worded items 1,3,5,7
and 9, the score contribution is the scale position minus 1. For negative worded items 2,4,6,8 and 10, the contribution
is 5 minus the scale position. Multiply the sum of the scores by 2.5 to obtain the overall value of SUS [18]. The system
usability scale scoring has a range from 0 to 100 which is in percentile. The following example on the SUS score
calculation is shown in Table 1.

Table 1. Example SUS score calculation.

Strongly Strongly
Question
Disagree Agree
1 2 3 4 5
1. I think that I would like to use this system
 5-1 = 4
frequently.
2. I found the mobile application unnecessary
 5-1 = 4
complex.
3. I think the mobile application was easy to
 4-1 = 3
use.
4. I think that I would need the support of a
technical person to be able to use this mobile  5-2 = 3
application.
5. I found the various functions in this mobile
 5-1 = 4
application were well integrated.
6. I thought there was too much inconsistency in
 5-1 = 4
this mobile application.
7. I would imagine that most people would learn
 5-1 = 4
to use this mobile application very quickly.
8. I found the mobile application very awkward
 5-2 = 3
to use.
9. I felt very confident using the mobile
 4-1 = 3
application.
10. I needed to learn a lot of things before I could
 5-3 = 2
get going with this mobile application.

Bangor, Kortum, and Miller [20] produced a scale for an acceptable SUS score. A total score of SUS that is in the
range of 85 to 100 indicates that the system is highly usable and that your users like the system and will recommend
it to other people. The system’s usability is excellent when the SUS score is between 70 to about 85. With a score of
50 to 70, the acceptability is good; however, the system users experience some issues using it. Finally, if your score
is below 50, then there is a problem with your system that you need to address as users did not find any satisfaction
in using the system. This scale of acceptable SUS scores is used for the purpose of this study.

3. Results and discussion

The evaluation was done in April 2018. There were 35 screened participants, 60% male and 40% female and all
participants were in the same age range of 18-50. They all have mobile phones as per screening and know how to use
the Internet in their mobile phones. Out of the 35 participants only 1 used an iPhone. The total population consists of
5 faculty, 5 staff and 25 students. The result shows that there is a 100% success rate in the tasks done by the participants
indicating that the mobile application works, and it is effective shown in Table 2.
1230 Kristine Mae P. Escanillan-Galera et al. / Procedia Computer Science 161 (2019) 1225–1232
6 Author name / Procedia Computer Science 00 (2019) 000–000

Table 2a. Effectiveness success rate in the tasks performed.

Participant 1 2 3 4 5 6 7 8 9 10
Success Rate 100% 100% 100% 100% 100% 100% 100% 100% 100% 100%
Average Time
3.2 2.4 2.8 2.9 4.2 4.4 2.3 10.9 7.4 5.3
Completion (seconds)

Table 2b. Effectiveness success rate in the tasks performed (continued).

Participant 11 12 13 14 15 16 17 18 19 20
Success Rate 100% 100% 100% 100% 100% 100% 100% 100% 100% 100%
Average Time
4.4 5 4.2 6.4 5 5.4 5.3 5.6 5.2 6.3
Completion (seconds)

Table 2c. Effectiveness success rate in the tasks performed (continued).

Participant 21 22 23 24 25 26 27 28 29 30
Success Rate 100% 100% 100% 100% 100% 100% 100% 100% 100% 100%
Average Time
3.2 2.4 2.8 2.9 4.2 4.4 2.3 4.5 5 4.6
Completion (seconds)

Table 2d. Effectiveness success rate in the tasks performed (continued).

Participant 31 32 33 34 35
Success Rate 100% 100% 100% 100% 100%
Average Time
4.4 5 4.2 3.5 4.7
Completion (seconds)

The time spent by a participant on each task is represented through a bar graph in Fig. 4. It shows that more time
was consumed on tasks 1. In Fig. 5, it shows that participants 8 and 9 took more time in doing Task 1, thus there is
more time consumed in doing Task 1, which is accessing the mobile web application link. One factor which affected
the delay in loading the mobile web application was the slow Internet connection. The 2 participants did Task 1 again,
after the Internet connection was fixed and the time spent on the average was 4 seconds. On average, to do all the 10
tasks it took all 35 participants 4.457 seconds to complete. Using the time-based efficiency, the time goal/second is
3.05 goal/second and the overall relative efficiency of the mobile web application evaluation is 100%.

Average Time (Seconds) per Task

8
Time in Seconds

6
4
2
0
1 2 3 4 5 6 7 8 9 10
Average per Q 5,8 4,57143 3,74286 3,54286 4,8 4,08571 4 4,31429 4,94286 4,77143

Fig. 4. Average time in seconds per tasks.

 Author
Kristine Mae name / Procedia Computer
P. Escanillan-Galera ScienceComputer
et al. / Procedia 00 (2019)Science
000–000161 (2019) 1225–1232 12317

120 Q10
Time in seconds 100 Q9

80 Q8
Q7
60
Q6
40
Q5
20 Q4
0 Q3
T1 T3 T5 T7 T9 T11 T13 T15 T17 T19 T21 T23 T25 T27 T29 T31 T33 T35 Q2
Participant Number Q1

Fig. 5. Total time consumption per second of each participant on all tasks.

User satisfaction was gathered using the SUS form and resulted in a high rating of user satisfaction and experience
was obtained. Fig. 6, shows a graphical representation of the user satisfaction rating of each participant. Participants
1 and 10 show low user satisfaction scores but this does not affect the overall rating. The overall experience of the
participants based on the SUS form is 87.28 which indicates that the participants like the mobile web application and
will recommend it to others.

SUS Evaluation Result

150
SUS Score

100

50

0
1 2 3 4 5 6 7 8 9 1011121314151617181920212223242526272829303132333435
Participant Number

Fig. 6. SUS evaluation score per participant

4. Conclusion and recommendation

In this study the effectiveness, efficiency and user satisfaction and experience for the mobile web application were
evaluated. Results show that the mobile web application is effective in terms of usage from the user’s perspective
since all tasks were completed successfully. The users also found the mobile web application as efficient, even though
it took more time in accessing the mobile app link due to Internet connection issues. Statistical data shows that users
were satisfied and liked the mobile application as a whole and would likely suggest the mobile app to other people.
The type of mobile phone used, and the age of the participants did not significantly affect the results.
The results of the usability testing show that the participants were satisfied in using the web user interface of
EnerTrApp an energy monitoring mobile web application and found it effective and will recommend it in using the
mobile application. Based on the results there is a good chance that consumers will use the mobile application as a
tool to reduce energy consumption in their household.
1232 Kristine Mae P. Escanillan-Galera et al. / Procedia Computer Science 161 (2019) 1225–1232
8 Author name / Procedia Computer Science 00 (2019) 000–000

This research has limitations. The sample included participants who were students, faculty and staff from only one
school in Iligan City. For future studies, participants from other schools in Iligan City may serve as respondents. Also,
factors like educational attainment and marital status may also be considered.

References

[1] Philippine National Statistics Office. (2004) “2004 Household Energy Consumption Survey.” Philippines Statistics Office Website.
Available from: https://psa.gov.ph/sites/default/files/HECS_PUB_2004.pdf. [Accessed May 11, 2018].
[2] Petersen, J. E., V. Shunturov, K. Janda, G. Platt, and K. Weinberger. (2007) “Dormitory Residents Reduce Electricity Consumption When
Exposed to Real-Time Visual Feedback and Incentives.” International Journal of Sustainability 4: 16-33.
[3] Darby, S. (2006) The Effectiveness of Feedback on Energy Consumption, University of Oxford.
[4] Galera, K. E., and O.E. Llantos. (2017) “Mobile Web Energy Monitoring System Using DFRduino Uno”, in 4th Information Systems
International Conference 2017. pp. 706–713.
[5] Galera, K. E., and C.V. Malabanan. (2018) “Implementation of Energy Utilization Through Enertrapp a Real Time Web Energy Monitoring
System”, in 2018 2nd International Conference on Automation, Control and Robots (ICACR 2018), Bangkok, Thailand, pp. 6-10.
[6] Bunn, D. W., and E. Farmer. (1985) “Comparative Models for Electrical Load Forecasting.” International Journal of Forecasting. pp. 241-
242.
[7] Hartson, R., and P. Pyla. (2018) The UX Book: Agile UX Design for a Quality User Experience, 2nd Edition, Morgan Kaufmann.
[8] ISO 9241-11:2018. (2018) “ISO 9241-11:2018.” International Organization for Standardization. Available from:
https://www.iso.org/standard/63500.html. [Accessed 2018, May 11].
[9] Shrestha, S. (2007) “Mobile Web Browsing: Usability Study”, in Mobility '07 Proceedings of the 4th International Conference on Mobile
Technology, Applications, and Systems and the 1st International Symposium on Computer Human Interaction in Mobile Technology,
Singapore. pp. 187-194.
[10] Coursaris, C. K., and D.J. Kim. (2006) “A Qualitative Review of Empirical Mobile Usability Studies.” Americas Conference on Information
Systems 2006. p. 352.
[11] Stary, C., and C. Stephanidis. (2004) User-Centered Interaction Paradigms for Universal Access in the Information Society, Vienna,
Austria, Springer Science & Business Media.
[12] Flinders University. (2018) Flinders University Website. Available from: https://www.flinders.edu.au/. [Accessed May 11, 2018].
[13] Tongco, M. D. (2007) “Purposive Sampling as a Tool for Informant Selection.” Ethnobotany Research & Applications. pp. 147-158.
[14] ISO 9241-11:2018. (2018) “ISO 9241-11:2018.” International Organization for Standardization. Available from:
https://www.iso.org/standard/63500.html. [Accessed 2018, May 11].
[15] ISO9241. (1998) “Ergonomics Requirements for Office with Visual Display Terminals (VDTs)”, in International Organization for
Standardization, Geneva, Switzerland.
[16] Nielsen, J. (2012) “Usability Engineering.” California: Academic Press Limited. Available from:
http://www.adkinsuhs.com/uploads/7/6/9/4/76943899/usability_101_article.pdf. [Accessed 2012, January 4].
[17] Kjeldskov, J., and Graham, C. (2003) “A Review of Mobile HCI Research methods”, in Proceedings of the 5th International Mobile HCI
2003 Conference, Udine, Italy, Springer-Verlag.
[18] Brooke, J. (1996) SUS - A Quick and Dirty Usability Scale, United Kingdom.
[19] Lewis, J. R., and J. Sauro. (2009) “The Factor Structure of The System Usability Scale.” International Conference on Human Centered
Design. pp. 94-103.
[20] Bangor, A., P.T. Kortum, and J.T. Miller. (2008) “An Empirical Evaluation of the System Usability Scale.” International Journal of Human
Computer Interaction. pp. 574-794.