Crumbl Cookie Test

Marial Krenzer

September 22nd, 2023

U.S. Department of Health & Human Services - 200 Independence Avenue, S.W. - Washington, D.C. 20201 1
 Introduction
Crumblcookies.com serves as the main platf orm f or the popular dessert shop.
Crumbl Cookies is best known f or their delicious and everchanging gourmet
cookies. The site is used most f requently f or order placements and updates on
cookie f lavors.

The Crumbl Cookie usability test was conducted to determine the extent an
interf ace f acilitates a user’s ability to complete routine tasks. The test was
conducted by observing one individual’s navigation through the website. Data
was collected throughout the test as the participant was encouraged to think
aloud and talk through each step. The test administrator jotted down
noteworthy mentions and asked questions to better gauge the participant’s
overall impression of the website.

Executive Summary
The Crumbl Cookie usability test took place on September 22nd , 2023, in a
f ace-to f ace setting. The purpose of the test is to identif y pain points and
potential navigation dif f iculties of the brand’s website. The test involved one
participant and one test administrator. The test lasted f or twenty-f ive minutes
and was broken up into f ive segments: the instructions, the questions, the
home page tour, the tasks, and the conclusion.

In general, the participant f ound the site to be organized in a clean and

comprehensible way. The test identif ied only a couple of minor problems
including:
• The lack of a location search f eature. The participant demonstrated
f rustration while scrolling through the 913 locations listed.
• The inability to check out as a guest.
• The need to conf irm your phone number to sign up.

This document contains the participant’s f eedback, satisf action ratings, task
completion rates, ease or dif f iculty of completion ratings, errors, and
recommendations f or improvements.

Methodology
Sessions
The test administrator recruited a participant who had very little prior
experience with the website. The session lasted f or approximately twenty -
f ive minutes. The participant was asked to complete three tasks during the
test, and to provide f eedback through answering both scaled and open-ended
questions post-test.

The test administrator asked the participant scaled questions based on ease
of use and aspects of the tasks:

U.S. Department of Health & Human Services - 200 Independence Avenue, S.W. - Washington, D.C. 20201 2
 On a 5-Point Likert Scale ranging f rom strongly disagree to strongly
agree,
• The website was designed in a user-f riendly manner.
o Answer: Agree
• It was easy to f ind cookie f lavors f rom the home page.
o Answer: Strongly Agree
• It was easy to f ind a Crumbl location near you.
o Answer: Disagree
• The website was easy to navigate through.
o Answer: Agree
The test administrator then asked questions regarding the satisf action
• The inf ormation was provided in an organized manner
o Answer: Strongly Agree
• I am satisf ied with the length of time I spent on the website.
o Answer: Disagree
• I would recommend using Crumbl Cookie’s website to my f riends and
f amily
o Answer: Agree
Finally, the test administrator asked open-ended questions:
• In thinking about the entire site, what is one thing you liked the most?
o Answer: The video at the top. I watched it like 5 times.
• In thinking about the website, what is one thing you liked the least?
o That I had to sign up to place an order.
• How did the website compare to your expectations?
o The design was a lot better than I was expecting. The overall
f eel of the site made me want to become a loyal customer.

Participant
This test was conducted within twenty-f ive minutes on a single participant
who was mildly f amiliar with the brand. The participant’s demographics are as
f ollows:

Age: 27
Gender: Male
Occupation: Service Manager
Hours Online: 30 hrs.
Browsing Behavior: They have brief ly visited the website, but only to see the
f lavors of the week.
Favorite Websites: Google News, Delish.com, YouTube.com

Evaluation Tasks/Scenarios
The task scenarios were created by the test administrator. The tasks included:

1. You need to purchase a Crumbl Cookie party box for store

pickup. Go through the process of selecting and purchasing.
Please stop right before you enter your payment
information.

U.S. Department of Health & Human Services - 200 Independence Avenue, S.W. - Washington, D.C. 20201 3
 2. You need to place a catering order for October 3 rd, 2023. Go
through the process of selecting your location and an order
of cookies. Please stop right before you enter your payment
information.

Results
Task Completion Success Rate
The participant successf ully completed task one without any noteworthy
issues. However, the participant was unable to complete task two as the
catering page would not load. He simply stared at a loading cookie f or several
minutes. It was concluded the issue was unrelated to any network
connectivity problems and was simply an error on the site’s end.

Task Completion Rates

Participant Task 1 Task 2

1 √ -

Completion
100% 0%
Rates

Task Ratings
Af ter the completion of each task, the participant rated the ease or dif f icult of
completing the task f or three f actors:
Task 1:
• It was easy to f ind my way to this inf ormation f rom the homepage.
▪ Answer: Agree
• As I was searching f or this inf ormation, I was able to keep track of
where I was on the website.
▪ Answer: Agree
• I was able to accurately predict which section of the website
contained this inf ormation.
▪ Answer: Agree
Task 2:
• It was easy to f ind my way to this inf ormation f rom the homepage.
▪ Answer: Agree
• As I was searching f or this inf ormation, I was able to keep track of
where I was on the website.
▪ Answer: Agree
• I was able to accurately predict which section of the website
contained this inf ormation.
▪ Answer: Agree

The 5-point rating scale ranged f rom 1 (Strongly disagree) to 5 (Strongly

agree). Agree ratings are the agree and strongly agree ratings combined with
a mean agreement ratings of > 4.0 considered as the user agrees that the
inf ormation was easy to f ind, that they could keep track of their location and
predict the section to f ind the inf ormation.

U.S. Department of Health & Human Services - 200 Independence Avenue, S.W. - Washington, D.C. 20201 4
 Ease in Finding Information
For both tasks one and two, the participant agreed it was easy to find the
order page from the home page.
Keeping Track of Location in Site
The participant agreed it was easy to keep track of location in the site. The
layout is very easy to understand. He mentioned it would be difficult to lose
his place in the site and “get lost”.

Predicting Information Section

The participant agreed it was easy to predict where to find information on
the website. Again, he emphasized the page was organized in a very user-
friendly and clear manner.

Test 1 – Mean Task Ratings & Percent Agree

Ease – Location in Predict
Task Overall
Finding Info Site Section
1 – Party Box Order 5 5 5 5.0

2 – Catering Order 5 4 3 4.0

*Percent Agree (%) = Agree & Strongly Agree Responses combined

Time on Task
The participant attempted two tasks which only took about 10 minutes each.
The total testing period was estimated to be around twenty-five minutes due
to additional segments mentioned above. The participant took five and a half
minutes to complete task one, and spent five minutes attempting task two,
but ultimately failed to do so for reasons out of his control.

Time on Task
P1 Avg.
TOT*
Task
330 330.0
1
Task
300 300.0
2

Overall Metrics
Overall Ratings
Af ter task session completion, participants rated the site f or eight overall
measures. These measures include:
• Ease of use
• Frequency of use

U.S. Department of Health & Human Services - 200 Independence Avenue, S.W. - Washington, D.C. 20201 5
 • Dif f iculty of keeping track of where they were in the site
• How quickly most people would learn to use the site
• Getting inf ormation quickly
• Homepage’s content f acilities exploration
• Relevancy of site content
• Site organization

The participant agreed that the website was indeed easy to use. They also agreed
that they would use the website f requently, and that others would learn quickly how
to navigate through the site. Even though the participant believed inf ormation could
not be quickly f ound, he still though highly of the website and desired to explore it
f urther and return to the site.

Post-Task Overall Questionnaire

Strongly Strongly
Disagree Neutral Agree
Disagree Agree

Thought Website
1
was easy to use
Would use website
1
frequently
Found it difficult to
keep track of
1
where they were in
website
Thought most
people would learn
1
to use website
quickly
Can get
1
information quickly
Homepage’s
content makes me
1
want to explore
site
Site’s content
would keep me 1
coming back
Website is well
1
organized
*Percent Agree (%) = Agree & Strongly Agree Responses combined

4.6.2 Likes, Dislikes, Participant Recommendations

Upon completion of the tasks, the participant provided f eedback f or what they
liked most and least about the website, and recommendations f or improving
the website.

Liked Most
The participant noted they liked the overall f eel of the website the most. It
was consistent throughout and lef t him excited to return to the site. The site
was successf ul in making someone unf amiliar with the brand want to
purchase f rom them.

Liked Least
The participant noted that they did not appreciate how dif f icult it was to f ind a
location near them to place an order. He would have appreciated a sort of

U.S. Department of Health & Human Services - 200 Independence Avenue, S.W. - Washington, D.C. 20201 6
 dropdown menu on each state rather than having to scroll through all 913
Crumbl Cookies locations.

Recommendations for Improvement

The participant recommended that the website include a dropdown menu to
save user’s time while searching f or their closest location. An additional
recommendation he mentioned would be to allow f or guests to checkout
without the commitment of signing up f or a Crumbl Cookie account. He
mentioned he f ound it f rustrating to have to sign up and conf irm your phone
number just to place an order especially when nearly all other sites allow you
to checkout as a guest.

Recommendations
The recommendations section provides recommended changes and
justif ications driven by the participant success rate, behaviors, and
comments. Each recommendation includes a severity rating. The f ollowing
recommendations will improve the overall ease of use and address the areas
where participants experienced problems or f ound the interf ace/inf ormation
architecture unclear.

Find Organizational or Individual Funding Information (Task 2)

Change Justification Severity

• Add a more user-friendly
search feature to find a
location nearby
The participant had difficulty finding a location High
closest to him as they were all listed by state and
took several scrolls down the page to find one.
His overall satisfaction with the website would be
much improved with a simple dropdown menu
over each state instead of listing all locations
upfront.

Participant commented on not being able to

• Inability to see the catering
page.
High
complete the second task because the catering
page would not load. He attempted several times
and even refreshed the page and closed it out to
start a new attempt.

Conclusion
Implementing the recommendations and continuing to work with users (i.e.,
real lay persons) will ensure a continued user-centered website.
The participant f ound the website to be very clearly and comprehensively
organized, inviting, and user-f riendly. He decided he would be willing to use
the site again to order Crumbl Cookies and would even recommend it to his
closest f riends and f amily. He noted there were only a f ew minor issues with
it; none being enough to deter him f rom returning to the site or purchasing
f rom Crumbl Cookies in the f uture. Upon implementation of the above
recommendations, he would be excited to return and make a purchase.

U.S. Department of Health & Human Services - 200 Independence Avenue, S.W. - Washington, D.C. 20201 7