Testplan 1

Heather Kfoury Usability Test Plan #1
ENG 6330 Usability Research Methods

Spring 2013 Page 1 of 10

Job Search and Application Usability Study
Chevron Phillips Website
Purpose and Goal

The purpose of this test is to evaluate the efficiency, effectiveness, and satisfaction of the Career section of the
Chevron Phillips website, www.cpchem.com, amongst two user demographics.

The company has recently received complaints that the websites job navigation and application process is difficult
to use, resulting in a decreased number of qualified applicants for open positions. In anticipation of the coming
hiring boom Chevron Phillips will experience during the next five years, it is necessary to evaluate the Career
section of the companys website now and ensure it is accessible to the diverse population of potential applicants the
company intends to attract. The company primarily attracts two types of applicants: experienced professionals and
recent graduates. How easily and successfully do both types of users search and apply for jobs on the Chevron
Phillips website?

The goal of this study is to understand whether both of the companys major types of users are able to use the Career
section of the website equally well.

Target Audience

Chevron Phillips has identified two primary types of users for the Career section of the companys website:
experienced professionals and recent graduates.

An experienced professional is defined as a person who has been in the workforce five (5) or more years and has
acquired skills and knowledge as the result of active participation in a particular field of work. During the next
hiring boom, Chevron Phillips will primarily hire experienced professionals in the following job categories:
engineers (mechanical and chemical), maintenance (instrumentation, electrical), operations (process technology),
supervisors, and senior managers.

A recent graduate is defined as a person who has been in the workforce less than five (5) years and has acquired
some skills and knowledge that allows them to work alongside experienced professionals, but not in an independent
capacity. A recent graduate typically has a two- or four-year degree and some experience as an intern. During the
next hiring boom, Chevron Phillips will primarily hire recent graduates in the following job categories: engineers
(mechanical and chemical), maintenance (instrumentation, electrical), and operations (process technology).

Testing Method

The test is exploratory using multiple user groups. The two user groups are novice and experienced
professionals who would use a website to apply for a job. A total of six (6) users are needed; three from each user
group. The test objective is to see if there are any differences in ability in using the Career section of the website
between the two user groups. Each user group will be given three (3) tasks to complete. In an effort to avoid transfer
of learning effects, the tasks will be randomized for each user. Example:


Novice Tasks
User 1 1, 2, 3
User 2 2, 1, 3
User 3 3, 2, 1

Experienced Tasks
User 1 1, 3, 2
User 2 2, 3, 1
User 3 3, 1, 2

The moderator, the observers, and the Morae software will collect data on three criteria: 1) Efficiency The
quickness with which the users goal can be accomplished accurately and completely during a pre-determined
measure of time. 2) Effectiveness The extent to which the website behaves in the way the user expects it to.
3) Satisfaction the users perceptions, feelings, opinions of the product.

Session Timing
Each test should take 20 minutes to complete. By testing six (6) users, the entire usability study should be completed
in two (2) hours. The pre-test introductions and interview will take five (5) minutes. The tasks will take ten (10)
minutes. The post-test debriefing will take five (5) minutes.

Pre-test Requirements
Review and sign the nondisclosure agreement and recording permission.
Fill out the background questionnaire.

Pre-test Introductions (5 minutes)
Moderator is the first to greet the user.
Introduce observers in the room.
Briefly explain the test and why the user is there.
Explain to the user what everyones role is the moderator, the observers, the user.
Explain the recording systems.
Explain the thinking aloud technique.
Explain the protocol for the remainder of the study.

Tasks (10 minutes)
Users will be given the first task and asked to say done when they feel they have completed it.
Users will be given the second task and asked to say done when they feel they have completed it.
Users will be given the third task and asked to say done when they feel they have completed it.

Post-test Debriefing (5 minutes)
Use the post-test questionnaire to collect qualitative data on the users experience with the website.
Ask the observers if they have any questions for the user.
Thank the user for their time.
Excuse the user from the room.


Task List

Task 1: Locate open positions
The purpose of this task is to see if a user can locate the websites job search engine quickly and accurately with
relative satisfaction. The user will begin at the Chevron Phillips website homepage, www.cpchem.com. The
moderator will give the user the following task:

SCENARIO:
You are an engineer living in Houston, Texas. You are interested in working as an engineer at
Chevron Phillips Chemical. You are willing to relocate anywhere in the United States.

TASK:
Find all open engineering positions in the United States with Chevron Phillips Chemical.
Once you have successfully located all the positions, please say done.

Task 2: Register as a new user
The purpose of this task is to simulate the application process of the website, and see if the user is able to complete
the task quickly and accurately with relative satisfaction. The user will begin at the Career section of the website,
http://www.cpchem.com/en-us/careers/Pages/Careers.aspx. The moderator will give the user the following task:

SCENARIO:
You are a Mechanical Engineer living in Houston, Texas. You are interested in working at
Chevron Phillips Chemical. You are willing to relocate anywhere in the United States.

TASK:
Find an open position for a Mechanical Engineer in the United States with Chevron
Phillips Chemical.
Start the application process. If you have never applied to this company before, you will
need to register as a new user.
Once you reach the job application, please say done.

Task 3: Select and view multiple jobs in the job cart
The purpose of this task is to see if a user can utilize the job cart function quickly and accurately with relative
satisfaction. The user will begin at the Career section of the website, http://www.cpchem.com/en-
us/careers/Pages/Careers.aspx. The moderator will give the user the following task:

SCENARIO:
You are a college graduate with two engineering degrees living in Houston, Texas. You are
interested in working at Chevron Phillips Chemical as either a Mechanical Engineer or an
Electrical Engineer. You are willing to relocate anywhere in the United States.

TASK:
Find all open engineering positions in the United States with Chevron Phillips Chemical.
Add any Mechanical Engineer and Electrical Engineer positions to your job cart.
View your job cart.
Once you view your job cart, please say done.


Test Environment

Users for this study will be tested in the University of Houston-Downtown MSPWTC usability lab on the 10
th
floor
of the One Main Street, South building. The room is equipped with three desks, three chairs, three computers, a
conference table and chairs, a refreshments table, and a book shelf. The lighting and general environment closely
mimics a University of Houston-Downtown classroom. A one-way window is affixed to the room, allowing
observers to anonymously monitor the usability test.

The users will be tested one at a time at the computer facing the one-way window. This computer has the Morae
software that will be used to monitor and measure the users experience during the usability test. The moderator will
be seated behind and slightly to the right of the user. Three observers will be used during the test. One observer will
be in the room during the test, seated to the left of the user near the conference table. The other two observers will be
out of the room during the test, standing directly behind the one-way window.

User
Observer
Moderator

Test Moderator

Moderator is the first to greet the user.
Introduce observers in the room.
Briefly explain the test and why the user is there.
Explain to the user what everyones role is the moderator, the observers, the user.
Explain the recording systems.
Explain the thinking aloud technique.
Explain the protocol for the remainder of the study.

Please see the following moderator script:

Pre-Test Introductions:

Hello. My name is (your name). This is (observers names). Thank you for coming. Would you care for any
refreshments? Please help yourself.

Thank you for agreeing to participate in this website evaluation. Today we are asking you to serve as an evaluator of
a website and to complete a set of scenarios. Our goal is to see how easy or difficult you find the site to use. We will
record your reactions and opinions; so, there may be times when Ill ask you to explain why you said or did
something.

My job is to record your reactions and comments of the website youll view. In the next room, we have a few people
who also will observe your interaction with the site. During this session no one is able to offer any suggestions or
hints.

I will ask you to search for information on this site to learn if it works well for you. Well do this by giving you
scenarios and tasks to complete on the site. You also will be asked a series of questions about your experience at the
end of this session.

Your job today is very important. What you do today will help us find any issues with the website and ultimately
makes it a better product for everyone. I want you to use the website just like you would if you were home alone and
none of us were here. I do ask that when looking for information, you do so as quickly and as accurately as you can.

Here are some things that you should know about your participation:
This is not a test of you; youre testing the site. So dont worry about making mistakes.
There is no right or wrong answer. We really just want to know if we designed the site well for you.
If you ever feel that you are lost or cannot complete a scenario with the information that you have been
given, please let me know. Ill ask you what you might do in a real-world setting and then either put you on
the right track or move you on to the next scenario.
We will be video recording this session for further study if needed. Your name will not be associated or
reported with data or findings from this evaluation.

It is very important that you speak during the test. I want you to say everything you are thinking while using the
website. It will feel a little weird at first, but if you stop talking I will remind you to please explain what you are
thinking. Do you have any questions before we begin?


Possible scenarios during the test

The user has a question about the task.
The moderator should clarify the task as best they can without explaining how to complete it.

The user has a question about completing the task.
Moderator: Im sorry. I cannot help you complete the task. Please try to complete the task the best you can.

The user says they are done when they have not completed the task.
Moderator: Im sorry, but you have not completed the task. Please review the task again and try to complete it as
best you can.

The user completes the task, but does not say done.
Moderator: Have you completed the task?

The user asks if they can start over.
The moderator should start the test again from the beginning of the task at hand.

The user is frustrated and wants to stop.
Moderator: Yes, you can stop. This test is voluntary. Thank you again for your time.

The user is not using the talk aloud method.
Moderator: Please say aloud what you are thinking right now.

Post-test Debrief

The usability test is now over. You did a great job. I now have a few questions for you and then I will go around the
room and ask the observers if they have any questions for you. Please be honest about your experience today. Your
opinion is important to us for improving this website.

1. What are your overall impressions of the Career section of this website?
2. If you had to give the Career section a grade, from A to F, where A was exemplary and F was failing, what
grade would you give it, and why?
3. Name three words or characteristics that describe the Career section of the website.
4. What do you like best about the Career section of the website?
5. What do you like least about the Career section of the website?
6. If you could make one significant change to the Career section of the website, what change would you
make?
7. Are there materials you would like to see added to the Career section of the website? Which ones?
8. Would you recommend this website to a colleague? To a friend?
9. Do you have any other questions or comments about the website or your experiences with it?

Thank you. We are finished. Do you have any questions for us? You are now excused. Thank you for your time and
help.


Data Collection and Evaluation

Task 1: Locate open positions
The user should be able to locate the Career section of the website and then the job search engine. Based on the
tasks instructions, the user should search for engineering jobs in the United States. If done successfully, the
Experienced Professional user will see seven (7) jobs listed. The Recent Graduate will see six (6) jobs listed.
The following criteria will be measured:

2 Points 1 Point 0 Points

Efficiency

The user is able to see
engineering positions
currently posted on the
website for their job
experience level.

The user takes two (2)
minutes or less to do this.

The user is able to see
engineering positions
currently on the website,
but not necessarily for
their job experience level.

The user takes between
two (2) minutes and five
(5) minutes to complete
the task.

The user is unable to find
any engineering positions
on the website.

The user takes more than
five (5) minutes to
complete the task.

Effectiveness

The user is able to locate
the job search engine with
no more than three (3)
uses of the back button.

The user is able to
properly use the job search
engine with no more than
three (3) uses of the back
button.

The user is able to locate
the job search engine with
three (3) to five (5) uses of
the back button.

The user is able to
engine with three (3) to
five (5) uses of the back
button.

The user is unable to
locate the job search
engine.

engine.

button to accomplish the
task.

Satisfaction

The user feels the website
is easy to use when
searching and applying for
jobs, and generally has a
positive opinion about the
websites job search and
application functionality.

is somewhat easy to use
when searching and
applying for jobs, and has
both positive and negative
opinions about the

is not easy to use when
searching and applying for
jobs, and generally has a
negative opinion about the


Task 2: Register as a new user
The user should be able to repeat Task 1 and locate a Mechanical Engineer position on the companys website.
Based on the tasks instructions, the user should click on the Apply button and register as a new user. If
registration is completed successfully, the user with see a Privacy Agreement followed by the start of the
application. The following criteria will be measured:


Efficiency

The user is able to register
as a new user and reach
the jobs application.


as a new user and reach
the jobs application.

the task.

register as a new user and
reach the jobs application.

five (5) minutes to
complete the task.

Effectiveness

as a new user with no
more than three (3) error
messages.

as a new user with no
more than three (3) uses of
the back button.

as a new user with three
(3) to five (5) error
messages.

as a new user with three
(3) to five (5) uses of the
back button.

register as a new user.

The user receives more
than five (5) error
messages.

task.

Satisfaction

is easy to use when
registering as a new user,
and generally has a
positive opinion about the
websites new user
registration functionality.

when registering as a new
user, and has both positive
and negative opinions
about the websites new
user registration
functionality.

registering as a new user,
and generally has a
negative opinion about the
websites new user


Task 3: Select and view multiple jobs in the job cart
The user should be able to repeat Task 1 and Task 2 (if not logged on). The user should be able to add at least two
(2) jobs to the job cart. The user should be able to locate the job cart button and view the jobs they selected. The
following criteria will be measured:


Efficiency

The user is able to select
two (2) or more jobs and
place them in the job cart.

The user is able to view
the job cart with only the
selected jobs.



selected jobs.

the task.

The user is unable to select
multiple jobs and place
them in the job cart.

The user is unable to view
the job cart

five (5) minutes to
complete the task.

Effectiveness

place them in the job cart
with no more than three
(3) error messages.

with no more than three
(3) uses of the back button.

selected jobs with no more
than three (3) error
messages.

selected jobs with no more
than three (3) uses of the
back button.

with three (3) to five (5)
error messages.

with three (3) to five (5)
uses of the back button.

selected jobs with three (3)
to five (5) error messages.

selected jobs with three (3)
to five (5) uses of the back
button.

The user is unable to select

The user is unable to view
selected jobs.

The user receives more
than five (5) error
messages.

task.

Satisfaction

is easy to use when
utilizing the job cart, and
generally has a positive
opinion about the
websites new user

when utilizing the job cart,
and has both positive and
negative opinions about
the websites new user

utilizing the job cart, and
generally has a negative
opinion about the
websites new user


Final Report
The usability test report will be provided at the conclusion of the usability test. It will consist of an evaluation of the
usability measurements against the test goals, subjective evaluations collected by the moderator and observers, and
specific usability problems and recommendations for future improvement. The report is anticipated to be delivered
to Dr. Mike Duncan during class on Monday, Nov. 12.

Testplan 1

Uploaded by

Document Information

Original Title

Copyright

Available Formats

Share this document

Share or Embed Document

Sharing Options

Did you find this document useful?

Is this content inappropriate?

Copyright:

Available Formats

Testplan 1

Uploaded by

Copyright:

Available Formats

Heather Kfoury Usability Test Plan #1

ENG 6330 Usability Research Methods

You might also like