Evaluation


Electronic
Attendance
System

Joseph
Ng
Chow
 
 


joseph.ngchow@utoronto.ca

Victoria
Mui

 
 








victoria.mui@utoronto.ca

Brian
Shim
 
 
 
 brian.shim@utoronto.ca

Veronica
Wong
 





















veve.wong@utoronto.ca


DAVE
DEARMAN

CSC318
–
THE
DESIGN
OF
INTERACTIVE
COMPUTATIONAL
MEDIA

NOVEMBER
26,
2008



Table of Contents
EVALUATION TECHNIQUES  3 
COGNITIVE WALKTHROUGH  3 
USABILITY TESTS  4 
HEURISTIC EVALUATION  6 
PARTICIPANTS 7

RATIONALE  7 
CHOICE OF PARTICIPANTS  7 
CHOICE OF TECHNIQUES/TASKS  7 
CHOICE OF MATERIALS  9 

EVALUATION DATA AND RESULTS  9 

COGNITIVE WALKTHROUGH RESULTS  9 
QUALITATIVE DATA  9 
USABILITY TEST RESULTS  10 
QUALITATIVE DATA  10 
QUANTITATIVE DATA  11 
HEURISTIC EVALUATIONS RESULTS  11 
QUALITATIVE DATA  11 
QUANTITATIVE DATA  12 
ERROR-STEP TEST  12 

DATA ANALYSIS AND DISCUSSION  15 

COGNITIVE WALKTHROUGH  15 
USABILITY TEST  16 
HEURISTIC EVALUATION  19 
ERROR-STEP TEST 22


DESIGN IMPLICATIONS AND IMPROVEMENTS  23 

CRITIQUE OF EVALUATION PLAN  25 

KNOWING WHAT WE KNOW NOW, WHAT WOULD WE HAVE DONE DIFFERENTLY?  25 
WHAT WOULD WE HAVE DONE IF WE HAD MORE RESOURCES?  26 

Page 2
Evaluation Techniques
The objectives of our evaluation are to learn:

• Whether or not the design requirements were met, and

• How well the prototype met the expectations of the design requirements.

To answer these questions, we used usability tests, cognitive walkthroughs, and heuristic
evaluation techniques.

We performed 2 separate sets of tests on our participants; usability tests for the end users, and
cognitive walkthroughs and heuristic evaluations for the HCI experts. To ensure that the results
of one test would not influence the results of the other, our second group of participants either
did a cognitive walkthrough or a heuristic evaluation. Although tests were divided, we felt that
observing the number of errors and testing the result was both important and generic enough
that it needed to be applied to both user groups. This will be discussed later in the Evaluation
of Results section.

Cognitive Walkthrough
Input

Interaction Task: The high-level interaction task was “taking the attendance”. Since this was
what the system was designed to do, it was the primary task that potential users performed.

Action Sequence:
0. Take the attendance
1. Log in with employee number and PIN
1.1. Use the stylus to activate the employee number input field
1.2. Input given employee number
1.2.1 Tap the numbers on the number pad with the stylus
1.3. Tap the numbers on the number pad with the style for PIN
1.4. Hit enter with the stylus
2. Mark each student in the class, present, absent, or late
2.1. Call out the student’s name
2.2. Tap the bubble next to each student’s name that corresponds to their
attendance status
2.3. Repeat 2.1-2.2. for each student on the class list

Page 3
 2.4. Click on the “Next” button to go to the next page of students, if
necessary
3. Submit the attendance.
3.1. Click on the “Submit Attendance” button
4. Correct the attendance
4.1. Click on the “Modify” button next to the student whose attendance needs
to be corrected
4.2. Tap on the bubble that corresponds to their attendance status
4.4. Click on the “Update Attendance” button
5. Logout

Identify the users: As we mentioned above, this test was performed on HCI experts. We
assumed that they are familiar with HCI principles, but have no knowledge of the domain. To
educate them, we gave an overview of our project—a summary that is similar to our project
description from G4, and a description of the existing attendance system.

Prototype: We used our physical prototype in conjunction with our functional screens that we
developed for G3.

Walkthrough

While one of us addressed each step of the task sequence and played the Wizard of Oz, the
remainder of our team made observations as the participant walked through the system.

To debrief, each participant was asked to answer the four questions used to develop a
believability story based on the interface, their knowledge of HCI and their understanding of
the users:

1. Is this a task that you would want to perform?

2. From the interface, can you see how the system can allow you to perform the task?
3. From the interface, do you know if the system would produce the correct result?
4. Does the system provide informative feedback of the system’s response to your
request?

Usability Tests

We designed our usability tests with respect to our usability requirements. Recall our
requirements were:

Page 4
1. Product design and interface must be intuitive so users can use the system with little or no
special training.

To see whether our interface was easy to learn, rather than walking them through our system
as in the cognitive walkthrough, we had the testers explore the physical prototypes for 3
minutes. If the interface was intuitive enough, they would have little-to-no problems
completing the tasks we asked them to perform.

2. Allows users to access features and menus through minimal system interactions; these
include: screen touches, mouse clicks, button presses… etc.

The set of activities that testers were asked to perform were tasks that teachers would
normally carry out with their attendance. We compiled this list from the observations we
conducted in earlier research. For each task, we recorded the time and the number of clicks
that the participants had to make in order to complete it. The tasks were:

Daily Attendance

- Login
- Taking the attendance
- Change student attendance records
- Logout

Emergency Attendance

- Accept global announcement; acknowledge lockdown

- Taking the attendance
- Adopting a missing student
- Contacting the office for EMS

3. Visual feedback of system activities.

To test the visual feedback of system activities, we performed another set of tests. As the
participants performed a task, we would interrupt them with another assignment. Once they
addressed the interruption, they were required to go back the original task. Were they able to
re-orientate themselves? Did they use the “back” button? Did they end up at screen where they
did not intended?

Page 5
The participants were randomly placed into 2 equal sized groups. One group took attendance
with a regular class list with no default status for students; the other group took attendance
with the same list, but with each student’s status defaulted to “Present”. The distraction
involved stopping the participant part way through the class list, and asking them to work on a
Sudoku puzzle. After a few minutes, the participant was urged to continue with the
attendance. We measured how long it took for them to reach the same point in the class list
where they left off before the distraction. The results in both groups were analyzed to
determine the effect of the EAS’s layout.

Heuristic Evaluation

Input
Before the evaluation, using the storyboard we developed in G4, the participants were given an
overview of the project, a description of the existing system, and the high-level intentions of
our proposed solution. We left out the rationale behind the design of our interface so that it
would not influence their judgment. If there was component that they could not rationalize for
themselves, it may be a bug.

Our physical prototype and functional screens were used for the evaluations.

Evaluations

Participants were given the prototype, and a rubric based on five main criterions/heuristics
with which they used to evaluate the interface.

From the Nielsen’s list, the heuristics that we deemed to be the most important are:

1. Match between the system and real world; e.g. was it a natural way to take the
attendance?
2. Visibility of system status; e.g. can the user tell which class the attendance is for?
3. Error prevention; e.g. how difficult it is to forget to mark a student present?
4. Consistency and standards; e.g. changes between screens aren’t jarring
5. Aesthetics and minimalist design; e.g. is anything on the interface unnecessarily?
Unhelpful?

A more detailed rubric can be found in the appendix.

Page 6
Debriefing and information gathering; Severity

To debrief, the experimenter and tester went over the evaluation together to rank the severity
of the bugs that were found. Discussions about possible solutions were also part of the
debriefing.

Participants

In our study, we used randomly selected participants between the ages of 20 and 45. The
research subjects in the cognitive walkthrough comprised of 4 male HCI experts. Similarly,
the heuristic evaluation was applied to 4 other male HCI experts. As well, 4 teachers (3
male, 1 female) were selected to participate in the usability tests; however, all
participants had their number of errors recorded and analyzed.

Rationale
Choice of Participants

It was important to include participants from our end-users because they would be the ones
who would be using the product when completed. Furthermore, having used the existing
system, their insights to our proposed solution in comparison were invaluable.

Due to the difficulty in gathering participants from our end-user group, we supplemented the
study with HCI experts. They lacked the domain background of our end-users, but that was not
critical in the more technical tests. We could have also used their perspectives as users who
were being exposed to the attendance system for the first time

Choice of Techniques/Tasks

Given the limited resources that we had, we chose to use the discounted techniques of a
cognitive walkthrough and a heuristic evaluation. We had neither the time nor the participants
to do some of the more expensive evaluations; however, due to the spoon-fed nature of the
cognitive walkthrough, it was necessary to do a more authentic evaluation to see how the
prototype met some of the more technical requirements of usability.

Page 7
Cognitive Walkthrough: The task of taking the attendance is what the system was designed to
accomplish; it is only natural that we asked our participants to do it. In our choice of
participants and the task, we were able to see whether or not the functional requirements
were met. In particular, whether it kept all the functionality of the old system, and whether it
was an improvement to the old system (e.g. was it less prone to errors? Was it faster to
complete the task?).

Usability Test: As an extension of completing a task with a minimal amount of clicks, we

wanted to test for the percentage of errors they made. We expected the average number of
errors would be less than 5% when taking the attendance as we believed our interface was
fairly intuitive, but not quite ready for production. To determine if this was a correct
assumption, we applied a t-test to the results.

Interruptions were made to test the visual and navigation aids in the system. We learned from
our initial research that distractions while the teacher was in the middle of taking the
attendance were common. Consequently, the teacher may forget who the last student that
they recorded was. In the worst case, the teacher would incorrectly mark a student’s status as
a result.

Similarly, feedback from earlier studies had suggested that there may be advantages if a
default “present” status was set for each student. From this, we designed a test to see
whether or not the layout of the interface would impact the time it took to complete the
attendance. Since the time to complete the attendance is proportional to the number of
students it would not have been accurate to test the total time to complete the attendance.
We chose instead to observe the time it took an individual to recover from distraction and
return to where they were in the class list before interruption. This is the motivation behind
why we wanted to see if the tester experienced any difficulty in recovering from the
interruption. To analyze this, we chose a two sample t-test to apply to the data as it
accurately shows the impact of such a factor.

Heuristics Evaluation: The advantage of this technique is that the participants are equipped
with the tools to objectively identify usability bugs; as the experimenters, we do not have to
interpret their behaviour. We chose to perform the heuristic evaluation on our peers because
they were already familiar with HCI principles and heuristics. We believed it was unrealistic to
give our end users a crash course on material that we were taught over the course of 4 months.

Page 8
Choice of Materials
We developed a paper prototype to demonstrate the functionality of the system. The
advantage of paper-prototyping is its low-fidelity nature which does not discourage the
participants from giving honest feedback. To facilitate that experiment, one of us sat with
each tester as they explored the system to answer any questions, and to “wizard” system
actions. Also, paper and pen are some things that everyone can relate to. In contrast, if we
developed the prototype on the computer, we would give the impression that a lot of time was
spent on it which discourages honest opinion.

We also made use of our storyboard. It was especially helpful in communicating the overview of
our entire system to our HCI peers because it illustrated the system from a third-person
perspective.

Evaluation Data and Results

Participant Information

- 12 testers in total
o 4 teachers; 8 HCI experts
- We had 4 HCI experts perform the Cognitive Walkthrough
- We had 4 teachers perform the Usability Tests
- And we had the remaining 4 HCI experts perform the Heuristic Evaluations
- All participants had their errors noted and counted

Cognitive Walkthrough Results

Qualitative Data

• Participants expressed their approval of the timed log out aspect of the device.
• Users found that the student pictures were quite useful and could help them learn the
names of the students faster.
• Users found that it was clumsy to navigate between pages using the "next" and
"previous" page buttons.
• Keyboard seemed out of place.
• Too few students were displayed on a screen.

Page 9
 • When updating attendance records, some of the users disapproved of the "Modify"
button's placement.

Usability Test Results

Qualitative Data

Daily Attendance Results:

• A user struggled for a minute trying to decide on which hand they should hold the
device with.
• Some participants had a different method of taking attendance. While some preferred
to mark the present students down, others asked if they could only mark the absent
down.
• Participants made use of the back button frequently.
• Users requested that the panel should automatically retrieve the time when marking
down late students.
• Many users also disliked how they could not see a record of what time the attendance
was submitted or updated.
• Too few students seemed to be displayed on a screen since some inquired about class
size.
• Users found that the student thumbnail photos were useful and would help them learn
the names of the students faster.
• When updating attendance records, some users tried to click on the student’s current
status to change it rather than use the “Modify” button.

Emergency Situation Results:

• Teachers were initially confused as to why their emergency class list was missing some
students.
• Users struggled with using the keyboard while still holding the panel. Most users
needed to rest it on a surface. Those same users had difficulty switching between
typing and using the stylus. One such participant actually started pushing buttons on
the keyboard with the stylus.
• Teachers expressed their opinion that many younger students do not know their own
student number, which is required in order to adopt the student in the case of an
emergency.

Page 10
 • Teachers were confused as to where the login screen went during the emergency
situation. They expressed their concern for the possible misuse of the system.

Quantitative Data

Distraction Test

Due to the small sample size of the distraction test, it was more appropriate for our group to
do a nonparametric bootstrap sample with replacement on the data for each group. We took
samples of 500 for each. Using the central limit theorem, we attempted to have the data more
normalized in order to apply a Welch Two Sample t-test with a confidence level of 0.95. We
let H0 : µ1 = µ2 and HA : µ1 ≠ µ2; that is, the true difference in means is 0 versus not equal to 0
respectively. The results can be seen as follows:

Distraction Test: Group 1 (Default “Present”)

Participant Recovery Time (seconds)

1 16
2 13

Distraction Test: Group 2 (No Default)

Participant Recovery Time (seconds)

1 2
2 3

R OUTPUT: Welch Two Sample t-test

data: a and b
t = 170.2069, df = 608.591, p-value < 2.2e-16
Alternative hypothesis: true difference in means is not equal to 0
95 percent confidence interval:
11.90108 12.17892
Sample estimates:
mean of x; mean of y
14.554 2.514

Heuristic Evaluations Results

Qualitative Data

• No title on the login screen and similarly no title on the main attendance taking screen
to identify purpose.

Page 11
 • "Modify" buttons at the summary screen seem out of place.
• Attendance summary looks nothing like the previous attendance page.
• Status modification selector activated by "Modify" button on Attendance Summary page
fails to indicate the student's current status. The student's status is therefore
inconsistent with the attendance taking screen prior to the summary page.
• Keyboard seemed out of place.
• Attendance summary page does not match the bubble sheet format that teachers are
familiar with from the existing Trillium system.
• Too few students displayed on a screen. Teachers are used to having more students
per page.
• Teachers liked that they cannot accidentally submit the attendance until they have
gone through the entire list (i.e., submit button is underneath the last student on the
entire class list).
• Student thumbnails were too small.
• Attendance does not indicate which student was marked last.
• The only indicator distinguishing between the statuses of each student is the coloured
dot.
• The colour-coded bubbles visible when teachers take the attendance appeal to more
senses of the user.
• The static menu bar along the top that remains visible at all times was deemed useful.
• Users liked that the attendance taking bubbles are similar to those of the current
Trillium System.
• Users liked the notification bar as it allows teachers to keep track of what just
happened.

Quantitative Data

See next section.

Error-Step Test

We believe that the percentage of error is less than 5%, that is, H0 : µ0 < 0.05 in both regular
attendance cases and in an emergency. We used t-tests with a confidence level of 0.95.

Regular Attendance

Steps and Error Observations (Regular)

Participant Steps Errors Errors/Step
1 36 4 0.11111111

Page 12
 2 44
3 40
4 37
5 33
6 45
7 35
8 43
9 33
10 44
11 36
12 33
3 0.06818182
4 0.1
1 0.02702703
3 0.09090909
3 0.06666667
2 0.05714286
3 0.06976744
3 0.09090909
3 0.06818182
2 0.05555556
3 0.09090909

R Output: One Sample t-test Regular Attendance Case

data: z
t = 3.7018, df = 11, p-value = 0.001745
Alternative hypothesis: true mean is greater than 0.05
95 percent confidence interval:
0.06271535 Inf
sample estimates:
mean of x
0.0746968

Summary Statistics for Regular Attendance Case

Min. 1st Qu. Median Mean 3rd Qu. Max.
0.02703 0.06429 0.06897 0.07470 0.09091 0.11110

Errors By Task (Normal Attendance)

Task Minimum Presses Average Presses Average # of Mistakes
Login 11 11.16666667 0.166666667
Take Attendance 21 22.25 1.583333333
Change Student Attendance Record 3 3.833333333 1.083333333
Logout 1 1 0

*Note: Some individuals forgot to update the attendance

Page 13
Emergency Attendance

Steps and Error Observations (Emergency)

Participant Steps
1 29
2 33
3 31
4 29
5 28
6 30
7 30
8 29
9 29
10 33
11 28
12 32
Errors Errors/Step
2 0.06896552
3 0.09090909
2 0.06451613
1 0.03448276
1 0.03571429
2 0.06666667
2 0.06666667
2 0.06896552
1 0.03448276
4 0.12121212
1 0.03571429
4 0.125

R Output: One Sample t-test Emergency Case

data: e
t = 1.9512, df = 11, p-value = 0.03848
Alternative hypothesis: true mean is greater than 0.05
95 percent confidence interval:
0.05141497 Inf
sample estimates:
mean of x
0.06777465

Page 14
Summary Statistics for Emergency Case
Min. 1st Qu. Median Mean 3rd Qu. Max.
0.03448 0.03571 0.06667 0.06777 0.07445 0.12500

Errors by Task (Emergency)

Task Minimum Presses Average Presses Average # of Mistakes
Accept Global Announcement 1 1.25 0.083333333
Taking attendance in emergency 16 16.91666667 1.166666667
Acknowledge a student has been adopted 1 1.166666667 0.083333333
Adopt a missing student 7 8.5 0.666666667
Contact main office for EMS 2 2.25 0.083333333

Data Analysis and Discussion

Cognitive Walkthrough
As presented in the previous section, in carrying out our cognitive walkthrough with HCI
experts, many issues regarding our design were brought to our attention. We will be focusing
on a few of these issues in more depth:

• When stepping through the attendance-taking process with our participants, many of
them brought to our attention that it was difficult and unnatural to flip between the
class list through the use of the Previous and Next buttons. The main cause of this
could be the fact that the Previous and Next navigation buttons are positioned with

Page 15
 respect to the content of the page. In other words, they do not have a consistent
designated position on each screen, which is not very user-friendly.

• When asking participants to perform tasks that strictly required the use of the external
keyboard, participants often hesitated and were unsure as to what they were to do
with the stylus they had been using throughout the course of the evaluation. This led
to our reconsideration of whether or not the external keyboard is necessary, and if an
on-screen keyboard would be more suitable.

• In terms of our attendance-taking interface, all 4 participants commented on the fact

that there is too much whitespace in between the names of each student on the class
list. They felt that this space could be used more appropriately by including a few
more students on each page of the class list.

• Many participants appreciated how EAS supports a timed expiry session in additional to
the manual logout option. They mentioned that this is a good security feature that will
be useful in school environments where potential users may often have to handle a
number of issues at once. One participant also mentioned that having their interactive
session expire and automatically log them out is a feature that they can find in many
electronic systems they are familiar with. For example, they encounter these features
regularly when they use school or library computers.

• Two of our participants commented on how we have designed our interface to allow
users to modify student attendance records. They mentioned that having the Modify
button beside the current student status could potentially lead to confusion. In taking
this feedback into consideration, we starting rethinking our attendance summary
interface, and whether or not we could possibly remove the Modify button if it seems
unnatural to the users.

Usability Test
We also conducted usability tests with our target users, that is, elementary and secondary
school teachers. As in the analysis for the cognitive walkthroughs performed, we will be
analyzing a few aspects of the collected feedback more thoroughly:

Page 16
Daily Attendance Taking Usage:

• As mentioned in the Evaluation Technique section, we provided our evaluation

participants with 3 minutes so they could explore the EAP used for evaluation. During
these 3 minutes, it was evident to our observers that the participant experienced some
difficultly in deciding how they should hold the device. Specifically, the participant
asked our team members whether or not the EAP accommodates for left-handed users.

• When we asked our participants to take the attendance as they would normally do so,
many participants completed the task assuming that unmarked students had the
default status of being present. This is because the existing Trillium Attendance
System treats unmarked students as marked present. However, we also had a few
participants that commented on the fact that they liked how we let the teachers mark
the students present as well because it is more intuitive for them to mark present as
they call out the names of each student. This required us to look into our interface a
little more thoroughly, and decide if there was something we could do in order to
satisfy both usability preferences.

• A participant suggested on how we could make our operations return more informative
results. For example, teachers may often want to know how much later a student
arrived on a particular day. So they suggested that we timestamp when a teacher
updates a student’s status from absent to late, and have this time marked beside the
student’s name. Along the same lines, they also suggested that we include the time at
which the attendance record was submitted or updated on the notifications bar. This
way, teachers and school staff will have more descriptive information in dealing with
attendance related tasks and procedures.

• Because our participants were teachers, they were able to point out that there were
too few students on a single attendance-taking screen from a different perspective
than the HCI experts that participated in the cognitive walkthrough. Our participants,
as part of this usability test, felt that there should be more students on each screen of
the class list because they are more accustomed to being able to view more students’
names at once.

• All 4 participants responded positively to EAS’s thumbnail pictures of each student on

the attendance-taking screen. In particular, they indicated that it would help them

Page 17
 learn the names of the students at the beginning of the academic year, as well as
facilitate substitute teachers’ ability to control the class.

Emergency Situation Usage:

• We also asked our usability test participants to accomplish high-level tasks during
simulated emergency situations. When we asked teachers to take the attendance of
the students as they would normally during the case of an emergency, we saw how
they hesitated because they were confused as to why the emergency class list was
missing some students. Although we had a brief description along the top of the screen
explaining that this class list is only comprised of those students that were present that
day, it still lead to confusion. This means that we will have to present this message in
a more eye-catching and understandable way.

• In designing the EAP, one of our central design requirements was to have the panel
portable so teachers can use it in different ways and carry it around with them during
emergency situations. However, it was evident to our observers that our evaluation
participants preferred to place it on a flat surface rather than to hold onto it like a
clipboard. Are there certain physical characteristics of the EAP that are making our
users feel as through they need to put it on a flat surface? For example, is the EAP too
big? Is it too heavy? Or is there a way we can design it so that users know that they
can hold onto it like a clipboard?

• Also in the case of an emergency, when users discovered that they had to use the
external keyboard in order to interact with the system, we saw that their interaction
became more difficult and clumsy. There were many instances where the participants
were holding onto the panel and trying to figure out how they were to use the
keyboard. Specifically, they were trying to find flat surfaces where they could use
both the panel and the keyboard simultaneously. In one interesting instance, the user
began using the physical tactile external keyboard with the stylus that he was using
throughout the entire course of the evaluation process.

• In the case of a lockdown, a student must remember their own student number in order
for a teacher to adopt them into their class. However, as all 4 participants pointed
out, from their experience, younger students are not likely to know the student
numbers. This means that we will be required to come up with a way of identifying
students in which young children can still inform the teachers as to who they are.

Page 18
 • Most teachers were worried about the way we lifted all security features in cases of
emergencies. Users will no longer be required to log into the EAPs just to use them.
However, there will be global announcements instructing school staff as to what
procedures need to be carried out during emergencies. Hence teachers have identified
their concern regarding the possibility of an EAP being misused by an intruder.

Analysis of Distraction Test Results

With a p value of less than 2.2e-16 we have strong evidence to reject H0. Thus we can see that
the layout of the EAS does affect the usability.

Heuristic Evaluation
As mentioned in the evaluation criteria, participants were asked to assign a severity rating to
each concern or issue that had with the prototype. In analyzing these results, we categorized
the most common issues according to the five Neilson’s heuristic values that we have been
focusing on. We also assigned each issue an overall severity rating by taking the median of the
collection of participant ratings.

Heuristic Severity Comment/Bug

Visibility of System Status 1 • No title can be found on the login screen to
indicate what the application is.
1 • No title can be found on the main
attendance-taking screen to identify the
purpose of the current screen.
Aesthetic and Minimalistic 2 • The Modify button provided on the
Design attendance summary screen is not very
natural to users.
4 • Keyboard seems out of place.
2 • The student thumbnail photos are small.
Consistency 3 • The attendance summary screen does not
look like the previous attendance-taking
screen.
3 • Status modification selector activated by
the Modify button on the attendance
summary screen fails to indicate the
student’s current status. The student’s

Page 19
 status on the summary screen is therefore
inconsistent with the same student’s status
on the attendance-taking screen.
Match Between System and 3 • Attendance summary does not match the
Real World bubble sheet format that users are familiar
with from their existing Trillium System.
4 • There are too few students displayed on
each page of the class list for attendance
purposes. Teachers are used to having
more students visible per page.
Error Prevention 2 • The only indication to distinguish between
the statuses of each student is a coloured
dot.

These heuristic results can be analyzed as follows:

Visibility of System Status:

• Although the notification bar on the side of the EAP provides users with a log of all the
events that have happened since their login, participants have indicated that they
were not aware of the system state at all times. Specifically, on the screen that users
are required to take the attendance on, there is no title indicating the purpose of the
screen. A similar visibility problem is the fact that there is no title on the login screen
specifying that this device is an EAP. This may be problematic for substitute teachers
who may not be familiar with EAS.

Aesthetic and Minimalistic Design:

• In terms of the aesthetics of EAS, participants have indicated that they liked how the
status bubbles of the attendance sheets are colour-coded with green bubbles for
present students, red for absent, and yellow of late students. They found this
especially useful because it appeals to more cognitive senses. Although they really
liked the idea of having thumbnail sized photos of each student beside their names,
many participants indicated that they would like these thumbnails larger.

• As for having a minimalistic design, almost all participants found it cumbersome to

have an external keyboard during emergency situations. A few participants also
pointed out that having a Modify button on the summary page is not very natural.

Page 20
Consistency:
• One of our central design requirements is to develop an interface that is similar to the
Trillium Attendance System so users will find it easier to transition from the existing
system to EAS. However, we did not successfully maintain the consistency between the
attendance-taking screen and the attendance summary screen. We have designed the
attendance-taking screen to resemble the current Trillium system bubble sheets, but
the summary of student attendance records have been organized into a text-based
format (i.e., present will appear beside the name of a present student, as opposed to
having the present bubble beside the student’s name).

• In order to update the status of a student, users are required to press the Modify
button on the attendance summary screen in order to activate the status modification
selector. This selector has 3 empty bubbles: 1 for present, 1 for absent, and 1 for late.
However, most participants pointed out that this is not consistent with the attendance-
taking screen prior to the summary page since it does not indicate the current status
the student is marked as.

• On the other hand, all participants indicated that having the menu bar along the top of
the EAP remains constantly visible was very useful.

Match Between System and Real World:

• As mentioned earlier, participants found the attendance summary screen inconsistent
with the attendance-taking screen. This is because the attendance summary screen is
not in the bubble sheet format that users are familiar with from the existing Trillium
system.

• With the existing Trillium system, teachers can view more students on one page than
on a single EAP screen. Therefore, this is a mismatch between the real world
application and the system, since teachers are used to having more students visible at
once.

• On the other hand, many participants have indicated that they appreciate that in
general, our attendance-taking screen somewhat resembles the Trillium bubble sheets
with which they are all familiar.

Page 21
Error Prevention:
• A single participant responded that they noticed that the only indicator that
distinguishes between the statuses of each student is the coloured dot in the
attendance column corresponding to their status. They mentioned that it might be
more informative if we could make these attendance statuses more eye-catching.

• Many participants noticed that how we positioned the Submit attendance button below
the last student on the class list. This prevents the teacher from submitting the
attendance record before they have entirely gone through the class list.

Error-Step Test

Regular Attendance:
With a confidence level of 0.95 and a p-value = 0.001745 < 0.05 there was strong evidence
against the null hypothesis; that is, our claim was incorrect when we believed the average
percent of errors would be less than 5% when taking regular attendance.

When errors were broken down by task, it was apparent that many participants had difficulty
just doing the attendance. As well, modifications to the system also seem unclear.

Emergency
With a confidence level of 0.95 and a p-value = 0.03848 < 0.05 there is there is strong evidence
against the null hypothesis; that is, our claim was incorrect when we believed the average
percent of errors would be less than 5% when taking attendance in an emergency. Errors in
modifying attendance defeat the purpose of our initial goal of providing an efficient and quick
way of taking the attendance.

When errors were broken down by task, it was apparent that many participants again had
difficulty completing the attendance. We saw that even adopting a student was difficult for
them. This could lead to a very serious problem during such an emergency.

From these results, it is clear that errors were much more apparent than we had thought. It
seems that our system is not as intuitive as expected as the error value was much greater than
our expected 5%. As a result, we need to focus more on the basic usability issues that we did
not address correctly. Although the number of steps were above the minimum number
required, it is clear that errors were the major cause of this. In many cases errors and

Page 22
additional steps were not in a 1:1 ratio. If many people are making errors on the main usability
requirements, then it is clear that we need to rethink our interface. These will be discussed in
the next sections.

Design Implications and

Improvements
After examining the implications, there are a few points in the design that need to be modified
that can address these problems.

Implications Improvements

The device interface should be flexible enough
so that users can use it regardless of hand
preference. This would be simple to
implement as all interaction is done within the
touch display.
• Add option that detects how the device is
oriented. The display will be flipped
depending on with which hand you hold it
with.

The keyboard should to be easy to use and not
cumbersome.
• The physical keyboard should be left out
of the final implementation. The
interface can be changed to have on
screen keyboard as method of input using
the stylus.

• Silent messaging system should allow for

handwritten text.

The display should to make good use of the on
screen space. There must be a balance
between readability and efficiency of use.
• Decrease font size and increase the
amount of students displayed onscreen at
a time.
• Add a magnification option, similar to the
way “Mac” computers implement it.
• Add line separators in combination with
the above methods to promote
readability.

Page 23

The device should allow the user to distinguish
whether a particular student's attendance has
been marked for that session yet.
When adopting a student, the system should
be robust enough to distinguish a student's
identity without depending on only 1 source of
information (i.e. student number.)
The device should to cater to user preferences
when taking attendance. (i.e. Some prefer to
just mark absent students, while some may
prefer to mark present students)
The device should to continue to ensure data
integrity and prevent potential misuse
especially in an emergency.
• Remove the “next” and “previous” page
buttons and add smooth scrolling for
efficiency.
• Add a color coding scheme to display the
currently selected student (I.e. the entire
line would change colour)
• Add a coloured background to the already
completed students' status.
• In the adoption procedure, add multiple
ways to input student data. Rather than
be restricted to just a student number,
the user will also have the option of
inputting a student name, address or any
other identifying information. For
example, if multiple students exist with
that same name, a list of all the students
with that matching name will appear with
corresponding pictures to distinguish
them.
• When the user profiles are created, there
should be an option to set status defaults.
Students could be marked as
absent/present by default.
• Require users to login even in emergency
procedures.
Page 24
Critique of Evaluation Plan
The goal of this project was to provide teachers with a fast, efficient and safe way of taking
their attendance on both a daily basis, and in the case of an emergency. Now that we have
executed our evaluation plan, we must reflect upon both it and how it might have influenced
our results. Given what we know now, there are many things that we could have done
differently. Similarly, there are many things that we were aware of, but could not do due to a
lack of resources.

Knowing what we know now, what would we have

done differently?
In the evaluation process, one of the biggest drawbacks was the use of a functional paper-
prototype. While we still believe that presenting a high-fidelity prototype might affect the
honesty of our participants, paper-prototyping is not without its own shortcomings. In playing
the wizard to facilitate the functionality, we had to intrude on the participant’s personal
space. Our involvement also skews some of our results like the measure of time and the
impressions of system actions. For example, we measured the time it took for users to
complete tasks; while it gives us a means of comparing the speed of tasks relative to each
other, we cannot use the numbers as the absolute time it takes to complete each task because
it takes time for us to react as wizards. We used stickers instead of markers to simulate radio
buttons because it’s gives the impression the EAS requires bubbling in a student’s status like
the existing system. Instead, the EAS was designed to replace the bubbling process with a
tapping gesture. This way, it eliminates the errors of inaccurate attendance records from the
system misreading an incorrectly filled bubble. With a self sustaining prototype like a digital
simulation, we minimize our presence in the evaluation process to create a more realistic
usage environment.

Cognitive Walkthrough
Even though the cognitive walkthrough is supposed to be done with HCI experts, we feel that it
could have been done with our end-users. In fact, we might have had more beneficial results if
we performed it with the teachers. In doing so, we would not need to develop a believability
story to speculate the value of the task to the user; the users would be evaluating it for
themselves. Teachers are also in a better position to compare the functionality of the existing
system with our proposed solution because they are users of the current system.

Page 25
Usability Tests
In retrospect, the Sudoku distraction test may have been inappropriate as our participants
found it slightly awkward. Although teachers experience distractions, this was an unrealistic
example.

Heuristic Evaluations
The heuristic evaluations were our smoothest set of tests. We suspect it was because both the
evaluators and the experimenters were familiar with the nature of heuristic evaluations and
HCI principles.

What would we have done if we had more

resources?
While we were able to collect a set of useful feedback from the users, we could have acquired
even more valuable feedback if it were not for certain restricting factors such as time, cost,
and participant co-operation.

First and foremost, a greater number of evaluation participants would yield a more
representative sample population. We would then be able to perform tests on participants
across all age, sexes and technological skill levels. We would be able to determine things like a
default font-size for old and young users, and an ideal weight for both sexes.

A larger sample population would have also meant that we would not have had to do
bootstrapping in our analysis which would greatly increase the power of our statistical tests.

With more resources, we could have afforded to do the more expensive usability techniques.
For example, we would have liked to do a focus group composed of end-users, HCI experts and
developers. The advantage of this technique is that we could discuss the different aspects of
the prototype face-to-face. By bouncing ideas off each other, and voicing our understanding of
the problem, we reduce the chance of misinterpretations any results. Any solutions would be a
collaborative effort with input from the end-user. Of course, we were unable to do this
because we could not schedule a time when it would be convenient for all the concerned
parties.

The HCI participants that evaluated our system were our peers due to the lack of connections
to professional HCI experts. The problem with this is that most of them were aware of our
project. For the same reason that we cannot objectively evaluate our own system because we

Page 26
are biased, our peer’s connection to us may influence the feedback that they give. For
example, because of our affiliation, some of them saw our device even before the evaluation.
Consequently, their proficiency of our system may be from previous exposure rather than
current exploration. On, another note, taking one HCI course does not make any of us experts
in the field.

Given more time and participants, we would have liked to design more tests based on the
feedback of an initial set of tests. This way, we could explore further some particular aspect of
design. To illustrate, we were certain that a keyboard was valuable addition to the device, but
feedback from our usability tests suggests otherwise. Keeping in mind that we only performed
the test on 4 users, we are uncertain of the representative power of their evaluations. Ideally,
we would have designed an experiment that tests the value of the keyboard based on what we
have learned.

Page 27