Lect ure Topi c 12 Usabi lity Testing - part 1

Dr. SENG LOKE
References: E-Commerce Fundamentals and Applications, by Henry Chan et al., John Wiley and Sons Pty Ltd, 2001

Lecture 12

USABILITY

• Usability can be defined as:
“the extend to which end-users can successfully accomplish required goals in a specified environment subject to various comfort, effectiveness and satisfaction criteria” (Thimbleby, SIGCHI94)

• Usability is becoming an integral part of the software development lifecycle. • Software developers aim at improving their product from the end-users viewpoint which involves identifying problems with the interface, and after some careful analysis of the problems, the interface is redesigned with the hindering aspects removed. • The interface design is the main factor affecting usability, since it is the only part of the system with which the user interacts with.

Lecture 12

USABILITY TESTING

• Usability testing usually entails building prototypes of the system and testing it on users. • Usability testing is a broad context where the interface is put under a test, with users critically analysing the system, evaluating the design layout and flow of interactions by interacting with the implemented version of the system. • The main reasons for adopting this approach are that it can be used :
– – – – to measure user performance and satisfaction, to identify and assess usability defects, to provide ideas as to how a system might be improved, to educate design teams.

Lecture 12 USABILITY TESTING

• Many usability testing strategies have been developed. Most of these testing can be categorised into two main categories:
– – Empirical Testing Inspection Testing

Lecture 12 USABILITY TESTING Phases of product usability testing
Phase Investigation Design and build Purposes Set •usability objectives Discover opportunities • Resolve conflicts • Answer questions • Debug • Measure against • objectives Get •data for sales/marketing Input • to next release Competitive analysis • Objects Existing solutions • Early • concepts Simulations • Prototypes • Partial • products Training • Documentation • Finished product • Competitive product •

Post release

Lecture 12

Empirical Testing

• Empirical testing involves the user testing the product under review in a testing laboratory. • Users are asked to use the system to complete a work related task that is likely to target problem areas. • The users’ task results are recorded, often through a video camera. • The users’ concerns, which are expressed through the use of a questionnaire, are also documented. • The results are analysed by the development team in an attempt to refine their product. • The primary focus of empirical testing if the product is still in a prototype stage is to use the results of the testing for refining the final version of the product.

Lecture 12

Empirical Testing

• If the product to be tested is in the final stage of development, this approach is useful for determining whether one selects a given product. • There are three types of Empirical testing:
–Fixed Laboratory Usability Testing –Portable Usability Testing – Remote Usability Testing

Lecture 12

Empirical Testing : Fixed Laboratory Testing

• Fixed laboratory testing involves the usability testing of a product in a fixed laboratory. This method is often referred to as in-house testing, since the users have to come to the developer to complete the usability test. • The user interact with the product by completing a work-related task which is created by the development team. • To assist the development team in analysing the results of the usability test, the tests are recorded using a video camera installed in the laboratory.

Lecture 12

Empirical Testing : Fixed Laboratory Testing

• ADVANTAGE:
– All necessary equipment is already in place. This will make it easier for the development team to conduct the usability test and record all essential data.

• LIMITATION:
– Can be very expensive, a fully equipped fixed laboratory costs between $50,000 to $100,000 and even more to operate and maintain. A big proportion of the amount goes towards recruiting and training of staff. – The user has to come to the developer to perform the test. (Explain what consequences this may cause ?)

Lecture 12

• IMPROVED Fixed Laboratory

Empirical Testing : Fixed Laboratory Testing

– The recent version of fixed laboratory is designed to address the problems of the earlier version of the laboratory. – The main emphasis is to design a comfortable, natural office scenario for the users and to eliminate the artificial element that is often associated with usability testing in a fixed laboratory. – This improved version fixed laboratory has a separate observation room, which is linked to the main laboratory through a one-way glass. This enables the observer to view the tester without the tester conscientious of an observer peering over their shoulder.

Empirical Testing : Fixed Laboratory Testing • IMPROVED Fixed Laboratory (ctd.) – This new layout enables testers to proceed without interruption as they would while performing the task in their own work environment. – Very often, the layout of the observation room is as similar as possible to the users real working environment. – However, this improved laboratory still does not address the problem of high costs. – Note that this fixed laboratory testing is normally used for building a new sophisticated software system or a new operating system whose usability will not be affected by internet connection, download time etc. (eg. Microsoft windows operating system releases).

Lecture 12

Lecture 12

• Portable usability testing was introduced to address the problem of high cost found in a fixed laboratory setting. • Portable usability testing takes the testing laboratory to the users. • The product is tested while the users interact with it in their own work environment.

Empirical Testing: Portable Usability Testing

Lecture 12

• Portable Testing is best suited when the customer base is widely distributed. • The developers will test a wide range of users from each of the customer sites. • The recording of the task results of the user is best accomplished with the use of a video camera. However, an over the shoulder video camera can place undue stress on the users, which could jeopardise the field test results. • The recording equipment must be setup in such a manner that it does not interfere with the usability testing process.

Empirical Testing: Portable Usability Testing

Lecture 12

• The SETUP:

Empirical Testing: Portable Usability Testing

• The computer output is directed to two monitors. • The video camera should focus on the second monitor. • The test results can then be recorded without the video equipment getting in the user’s way.

• Another important element is to encourage the users to
think out loud while they undertake the tasks. These verbal comments can prove to be valuable for analysis.

Lecture 12

Empirical Testing: Remote Usability Testing

• Remote usability testing is a usability testing where neither the developers or the users have to travel. • Users are observed and tested through the aid of computer networks and modem connections in a time efficient and cost effective manner. • Can you think of any limitations of this type of usability testing?

Lecture 12

Empirical Testing: Summary

• In general, the main advantage of empirical testing is that it is able to directly monitor the performance reactions of the user to the particular interface and one does not have to rely on a usability expert trying to second guess what a user’s reaction might be. • The disadvantage is that it is only useful for testing an implemented product, and therefore the ability to correct the problem immediately is greatly reduced.

Lecture 12 Empirical Testing: Test Procedure

• The test layout in an empirical testing consists of:
– the test subject (the user) – the computer with the software being tested – the test monitor (the tester)

• The user must perform a set of tasks and answer a set of questionnaire prepared earlier by the tester. The tasks must be designed such that:
– they be work related to stimulate the manner in which the
software would actually be used in the work environment – they cover most of the significant functions of the package so that all significant and frequently used features of the interface are tested – the time of the test session clearly reflects work periods and is not too short (too artificial) or too long (may cause tiredness).

Lecture 12

Inspection Testing

• Usability Inspection testing is often referred to as “discount usability engineering methods”. • Inspection testing is performed by usability inspectors who will inspect the design layout of an interface and comment based on the result of their test and experiences. • Results from inspection methods influence the design layout of an interface since it captures potential problems at an early stage in the development life cycle. • Most commonly used Inspection testing procedures are:
– Cognitive Walkthrough – Heuristic Evaluation

Lecture 12

Inspection Testing: Cognitive Walkthrough

• People who are evaluating the interface (called the inspectors) are asked to answer a set of derived questions for each step of the user task scenario. These questions are based on the relationships between user’s goals and the actions available for achieving these. • This inspection testing can be done during the design stage especially when the dynamic aspects of the web site/application is determined. • The user task scenario can be drawn from the Web Navigation Analysis Model (lecture 10) by defining a scenario for each branch of the hierarchical structure in the Site Navigation Net.

Lecture 12 Inspection Testing: Cognitive Walkthrough
SNN2

• For example, we can perform a cognitive walkthrough test on the ‘Warehouse Booking’ activity as shown in the left most branch of the SNN.

W

B1

GT1

GD 1

Prc: sesion active Poc:

Prc: sesion active Poc:

Prc: sesion active Poc:

Enter Booking Details
Di: Do: Booking details WB2

Search Lot MT3
Di: Do: GT2

Search Lot
Di: Do: GD 2
GD 1.2

Booking Details Confirm
Di: Booking details Do: WB 3

GT1.2

Prc: sesion active Poc:

Prc: sesion active Poc:

Prc: sesion active Poc:

Lot Result List
Di: Warehouse ID Do: Lot.no. GT3

Lot Result List
Di: Warehouse ID Do: Lot no. GD 3 Prc: sesion active Poc:

Prc: sesion active Poc:

Prc: sesion active Poc:

Booking Receipt
Di: Do:

Enter Transfer Details MC3 Di: Lot no.
Do: GT4

Enter Delivery Details MT3 Di: Lot no.
Do: GD 4 Prc: sesion active Poc:

Prc: sesion active Poc:

Transfer Details Confirm
Di: Do: GT5

Delivery Details Confirm
Di: Do: GD 5 Prc: sesion active Poc:

Prc: sesion active Poc:

Transfer Receipt
Di: Do:

Delivery Receipt
Di: Do:

Lecture 12 Inspection Testing: Cognitive Walkthrough

Cognitive Walkthrough Sheet example Task description: Warehouse Booking WB1(-MT3) WB2-WB3 Tester name: ……….. Problems identified: 1. Failure to identify a goal in the booking detail form, there is a ‘+’ sign which indicates a facility to add a new transporter if it does not exist in the list. User might fail to identify this goal. 2. Failure to identify the correct flow after clicking ‘+’ sign and transferred to the ‘Add New Transporter’ form (MT3), the user might fail to identify the correct action to come back to the enter booking form.

Lecture 12 Inspection Testing: Cognitive Walkthrough

Cognitive Walkthrough Sheet example (ctd.) 3. Correct action does not match goal in the booking confirmation form, the user might want to edit the booking details (goal=edit). The correct action here is to click on the ‘Cancel’ button, which does not match to the user’s goal.

Lecture 12

Inspection Testing: Cognitive Walkthrough

• Cognitive walkthrough requires an in depth, step-by-step evaluation of the interface. • However, it often requires a substantial investment in training.

• See also:
http://www.infodesign.com.au/usabilityresources/evaluation/conductingwalkthroughs.asp

Lecture 12

Inspection Testing: Heuristic Evaluation

• In heuristic evaluation, the interface is analysed based on a set of known heuristic rules. • In general, those rules can be summarised into the following nine usability heuristics:
1. 2. 3. 4. 5. 6. 7. 8. 9. Visibility of system status Match between system and real world User control and freedom look & feel or Consistency and standards functionality? Error prevention subjective & Recognition rather than recall objective? Flexibility and efficiency of use Aesthetic and minimal design Helping users recognise, diagnose and recover from errors

Lecture 12

Inspection Testing: Heuristic Evaluation

• Heuristic evaluation can be improved by a group of inspectors working independently on the same system interface. The inspectors do their initial evaluations individually and then collaborate afterwards. • BENEFITS of Heuristic evaluation:
– intuitive and easy to motivate people to do it – does not require advanced planning –can be used early in the development process

Lecture 12

Inspection Testing: Heuristic Evaluation

Example of Heuristic issues: • Visibility of system status
– Does the system let one know where one is (site title, page title, window title)? – Does the system acknowledge that an action requested has been carried out?

• Consistency and standard with respect to the following:
– Message display methods (prompts, warning, helps) – Colour use (entry form, menu/submenu, foreground/background) – Keys definition – Data entry method – Navigation, dialogue, window display method – Search facility

Lecture 12

Inspection Testing: Heuristic Evaluation

Example of Heuristic issues (ctd.): • Match between system and real world
– Menu hierarchy that is consistent with the real world – Terminology used is the same as in real life of that domain

• Flexibility and efficiency of use:
– Browser compatibility – Time taken for task completion – Unproductive period – Download time

Lecture 12

Inspection Testing: Heuristic Evaluation

Example of Heuristic issues (ctd.): • Helping users recognise, diagnose and recover from errors:
– Does the system explain why an action cannot be performed? – Does the system give prompts on how to proceed? – If an action is not allowed, does thet system give the reason?

Lecture 12

Inspection Testing: Heuristic Evaluation

• Heuristic evaluation is one of the most widely used method at assessing the usability of an interface at the design stage. • Heuristic evaluation has been described as the usability assessment method which identifies the most serious usability problems with the least amount of effort. • However, it can only be used to find usability problems. It cannot assess the performance of the user while they interact with the system like empirical user testing can. • Another problem is that heuristic evaluation is biased towards more experienced inspectors, since they seem to predict more problems than people without usability expertise.

Further reading on Heuristic Evaluation
• Case studies:
– http://www.bls.gov/ore/htm_papers/st960160.htm
– http://www.infodesign.com.au/usabilityresources/evaluation/webevaluation.asp

• Background:
– http://www.useit.com/papers/heuristic/

Next Lecture ...

Next Lecture: Usability Testing - part 2