EBOOK

Remote Moderated 101

How to replicate the lab from anywhere in the world

UserZoom, The UX Insights Company More info at userzoom.com

 Introduction 3

Why Moderated? 6

Moderated: In-person vs. Remote 8

The Ten Phases of Remote Moderated 11

#1 Create Your Research Roadmap 12
#2 Set Expectations 14
#3 Recruit and Prepare Participants 15
#4 Run a Pilot or Three 18
#5 Prep Your Observers 19
#6 Right Before The Session 21
#7 Inviting Your Guest In 22
#8 The Session 24
#9 Debriefing After the Session 25
#10 Plan For the Future 26

Remote Moderated Checklist 28

Frequently Asked Questions 30

Kuldeep Kelkar Jamie Miller

SVP, Global UX Research Services, UserZoom Sr. UX Researcher, previously at UserZoom
Introduction
Between both of us authors, we have been in the research,
design and user-experience fields for over 30 years. We have
collectively conducted hundreds of studies with a wide range
of different research methods - everything from qualitative
to quantitative testing, and loads of both moderated and
unmoderated studies.

As researchers, as designers, as people who are working on

user experiences, we all understand the value of user-centered
design. We know that we need to understand and analyze
customer behaviors as well as user attitudes and perceptions
and that there are a myriad of ways to get there.

Which is why we find it odd that out of the vast range of

research techniques and methods within a researcher’s
toolkit, it sometimes feels like in-person moderated
is the only tool available (or the only approach that
stakeholders want you to use). That’s like using a hammer
for everything from hanging posters to cutting down a
tree. In-person moderated is a powerful tool
that can yield great insights but it is far from a
panacea for all your user research needs.

Case in point – you’re probably familiar with the

grid on the following page, as previously published
by Christian Rohrer (a member of the UserZoom
advisory board) on the Nielsen Norman website.

3
 A LANDSCAPE OF USER RESEARCH METHODS

BEHAVIORAL / Eyetracking Clickstream Analysis

A/B Testing
Usability Benchmarking (in lab)

UsabilityW Lab Studies Moderated Remote Usability Studies

Unmoderated Remote Panel Studies Unmoderated UX Studies

Ethnographic Field Studies True Intent Studies

Concept Testing
Diary/Camera Studies
Participation Customer Feedback
Focus Groups Desirability Studies Intercept Surveys
ATTITUDINAL Interviews Card Sorting Email Surveys

QUALITATIVE (DIRECT) QUANTITATIVE (INDIRECT)

KEY FOR CONTEXT OF PRODUCT USE DURING DATA COLLECTION

Natural use of product De-contextualized / not using product
©2014
Christian Rohrer Scripted (often lab-based) use of product Combination / hybrid

Each dimension provides a way to distinguish among studies in terms of the

questions they answer and the purposes they are most suited for.

QUESTIONS ANSWERED BY RESEARCH

METHODS ACROSS THE LANDSCAPE

BEHAVIORAL
WHAT PEOPLE DO

WHY & HOW TO FIX HOW MANY & HOW MUCH

WHAT PEOPLE SAY

ATTITUDINAL

QUALITATIVE (DIRECT) ©2014 Christian Rohrer QUANTITATIVE (INDIRECT)

4
Many of you reading this have probably seen that article or
referred to that grid at some point in your careers or made it
into a poster to hang in your cubicle. It’s a classic!

Christian Rohrer’s grid is extremely helpful because it covers

behavior and attitude on the Y-axis and quality versus quantity
on the X-axis. Those of us who have conducted research
for any length of time understand that sometimes there is
a difference between what people say and what they do.
Christian accounts for this, and plots several research methods
such as A/B testing, eye tracking, usability studies, true intent
studies and more.

Why are we bringing this up in an ebook about remote

moderated, you ask? Simply because it helps to set the context
that there are the right tools for the right research need (and
because you eventually might need to be able to talk an over
zealous stakeholder or gung-ho executive off the ledge when
it comes to conducting remote moderated studies.) To expand
a toolkit it helps to crack it open and take a peek at what’s
already in there and what else is available.

And while we could talk your ear off about all of the methods
on the grid (seriously, we’ve been kicked out of dinner parties)
for the purpose of this ebook we will only be focusing on
remote moderated.

5
Why
Moderated?
Part One
why moderated?

Why, in general, should you go with moderated as your methodology?

The core reason for getting into a moderated session is so that you can
be in a live setting with a participant.

This allows you to have a conversation with your users as you’re observing
what they are doing and dig deep into usability issues in order to better
understand their behavior. Plus, you can modify your script on the fly
to probe further if there are confusing areas and ask ad-hoc follow-up
questions.

It’s important to not introduce bias to the participant (and there are best
practices around not asking leading questions) but given that this is a live
interaction, there are advantages of adapting to the needs of the moderator,
stakeholders/observers, and the participant on the fly. Namely, not needing to
run another test just to ask follow up questions.

7
Moderated:
In-person vs.
Remote
moderated: in-person vs remote

Within the moderated world there are two options: In-person and remote.

The clear advantages of in-person are that you can read their body language,
your participants are within your local area, and you typically control the
hardware – laptop, desktop machine, mobile phone, maybe an iPad; whatever
device you are testing on is your device and so you control the hardware.

This is a huge advantage, because technical challenges can become an issue

when you’re conducting remote moderated sessions. In fact, several tips and
tricks that we’ll talk about later are related to some of the hardware or
networking challenges that might arise.

That being said, there are several instances where remote moderated truly
shines.

9
moderated: in-person vs remote

One of the most common challenges we hear from researchers (from around
the five hundred companies we work with) is that they find it harder and
harder to find the right participants within their geographic area. Geographic
distribution is important – your target market might not be in the exact same
location where you are and that’s where remote moderated comes in handy.
We also hear reasons like travel budget being limited or constrained.

There is also merit in observing a participant using their own technology

rather than the hardware and the technology that we give them. It might
create some stumbling blocks, but we will walk you through several tips and
tricks around potential technical issues that might happen.

Also, depending on whether you request participants have a webcam, you can
still read their body language and pick up on their non-verbal communication.
Paired with the aforementioned, you’re getting quite a lot of research value
squeezed into one methodology.

It’s not just us here at UserZoom who feel this way. Kate Meyer and Kara
Pernice from the Nielsen Norman Group wrote an article in March 2018 titled,
“Remote Moderated Usability Tests: How and Why to Do Them.” In it they
described why remote moderated is beneficial:

“Remote unmoderated testing has the benefit or being fast,

inexpensive, and easy. It can get some great insights, and
should be part of every UX researcher’s toolbox. However,
remote moderated testing can give you significantly
more useful, interesting, detailed findings than you’ll
get from remote unmoderated tests. It takes a bit more
coordination, but the small amount of extra effort is well
worth the beneficial impact this methodology will have on
your research.”

10
The Ten
Phases of
Remote
Moderated
the ten phases of remote moderated

On that note we will now walk everyone through what we are calling the “Ten
Phases of Remote Moderated.” We came up with the analogy of baking a cake
because we’re hungry and who doesn’t like cake? (We know there are a few of
you who will inevitably be pie people, but we forgive you.)

Remote moderated studies are not like an “out of the box” cake, though.
They’re a more complicated cake, like a tiered wedding cake. It might sound a
little bit more daunting than it really is at first, but essentially anybody can do
this with the right planning.

To help, however, we have included a checklist further down in the ebook that
will help you be successful at remote moderated studies.

cipe
CakeRe
ite sugar
2 cups wh
ite sugar
2 cups wh owder
n n s w e e te n ed cocoa p
ov
e 3/4 cup u
eps

owder
tt he a s p o o n s baking p
St

r te
hea tte 1 1/2
re- he ba pan
P
ix
t t he 1 ecnup milk
M se ov
r ea n the 2 eggs
i

#1 Create Your Research

G
ke
Ca

Roadmap

The first step is one we think you all already know: put a plan together. Think
of this as writing out your recipe or finding one that works for you and your
soon to be delicious cake. We recommend planning out the following four
things in advance.

Participants
What type of participants do you want? How many participants do you
want? Do you have multiple profiles you need to take into account?

12
the ten phases of remote moderated

Prototype/Stimulus
What prototype are you using? Is it hosted? That’s one of the biggest things
you want to take into consideration: hosting your prototype so that your
participants can get to it. And if you can’t host it, how do you show it to your
participants? You can either have your participants control your computer
or they can control their own.

Schedule
When will you be conducting these studies? Are you doing quick iterations?
Or is this a deep dive? What are you doing in your study? Make sure you set
a schedule that takes all of this into account. Have an overall timeline and
take into consideration how many sessions you’re running and whether
your stakeholders can actually join in.

Sessions
Finally, how long will your moderated sessions last? Typically, remote
moderated is the same as in-lab moderated – anywhere from three to 60
minutes. Participants tend to hit a wall with fatigue around the 60 minute
mark, however, so try not to pass it.

That being said, we have personally run sessions that have gone a bit longer
than 60 minutes with remote moderated. Since remote moderated allows
participants to be in their own environment (read: are more comfortable), it
seems that you can push sessions a little bit longer than normal as long as
you set expectations upfront.

13
the ten phases of remote moderated

#2 Set Expectations

This is one of the big ones. Setting participant expectations upfront is like
preheating the oven. It’s basically the difference between a really successful
session and a failed one.

What you want to do is make sure it is crystal-clear that this is a remote

session and there are some technical things that your participants will have
to get set up on their side.

So what we recommend in this case – and have done when we had to recruit
and also while working with recruiters – is to either help the recruiter come
up with that first contact email or, if they come up with it, make sure to
review it.

Just make sure everything is included in that first contact email so that
the stage is set and your participants are ready to join and don’t have any
misconception about what actually is going to happen for the session.

14
the ten phases of remote moderated

#3 Recruit and Prepare

Participants

Recruiting is, if you’ll pardon the pun, the bread and butter (butter and sugar?)
of your study. This is where you’re gathering all the ingredients you’ll need to
make a cake.

Of course there are all the usual suspects when it comes to recruiting that we
won’t get into here, such as figuring out who you want to test with, what
your sample size should be, etc. However for remote moderated, there are
a few items to consider beyond your normal considerations.

1. The first first thing you’ll want to do is add technology questions to

your screener. This could be anything under the sun depending on what
you need to run your study successfully. For example, are there certain
browser requirements that you need? Do they need to join with Chrome or
Firefox? There are prototypes that only run in one or the other, so if that’s
the case make sure to put that upfront and make sure your participants
are willing to download one or the other. Or you can just screen them out
entirely if that’s too much of a hassle.

15
the ten phases of remote moderated

2. Be sure to over recruit, just as you would for in-lab, to make sure that
you have enough participants in the end. We typically over recruit by at
least one or two participants and what you can do is schedule a day just for
your make-ups. Consider telling a couple participants, “You’re our backup
and will get a phone call if we need you; what time slot do you want on
your make-up days?” That or just over recruit. If you’re shooting toward ten,
recruit twelve, and if you get twelve, yay! It depends on your budget and
your timeline.

3. Remember to put reminders and tech instructions in the email.

Your goal is to make sure your participants understand the technical side of
what you need from them in advance. Ask them to install Zoom or WebEx,
for example, or make sure they’re running X on Chrome. Reminders are key
as well, and thankfully there are some tools out there you can use to make
this easy. Calendly and YouCanBook.me are great, and they have built in
reminders that are all automated so you don’t have to even think about that.

4. Make sure you have a dial in. This is one of the key things that has
saved my bacon a couple of times. Imagine that you’re sitting there in your
WebEx or your Zoom or whatever meeting, waiting for the participant to
join and they’re not joining. You’re trying to figure out what’s going on only
to learn they’re having issues joining the audio through their computer. So
having a dial in number as a backup is a good idea.

5. Be considerate of time zones. I know this is pretty obvious, but there

are a lot of mix-ups when it comes to time zones, especially if you’re in a
different time zone than your participant. So in your reminder and booking
confirmation emails, always put the time slot in the participant’s time zone.
We don’t recommend putting multiple time zones in there, just put the
participant’s time zone in there to avoid confusion.

6. Ensure there is enough time between the sessions to reset. You

might want to talk to your stakeholders in between sessions or you might
have a talker. That way even if you go over your time you’ll have buffer.
We recommend 30 minutes to help you reset and get ready for the next
session. Depending on how intense the sessions are, you may actually want
to make that buffer closer to an hour.

16
the ten phases of remote moderated

7. This is something we know a lot of people do, which is send NDAs and
consent forms. If you’re going to do this we recommend doing this before
the session starts because it takes one more thing out of the equation that
you have to worry about. At the beginning of the session you can double-
check to make sure that they did sign those forms and they’re good with
the recordings. As a joke, I say “Don’t worry we’re not going to put you
on YouTube” because I sometimes get questions about where the video
will end up – so be prepared to reassure participants that it will only be
used internally.

8. Finally, have a way to easily reschedule or cancel. Those tools we

mentioned before, like YouCanBook.me, have that all built in. It is pretty
slick to be honest. You can go into the original email that was sent to the
participant and they can click a button and reschedule or cancel from
there. If you’re doing it manually make sure it is going to a human, i.e.
yourself or your recruiter, and not to an info@company.com email that you
may forget about.

Just to stress the point on preparation: We have watched and conducted

hundreds of these sessions and even then we cannot always anticipate all the
problems that might arise. Anything that can go wrong usually does, so you
just have to prepare, prepare, prepare beforehand for as many situations as
possible to try and mitigate them.

17
the ten phases of remote moderated

#4 Run a Pilot or Three

At this point you’re mixing the batter and tasting it, which means you need to
pilot your sessions at least once. Kick the tires of your project and test your
test. We actually recommend at least two pilots because it’s the last chance to
tinker with the recipe.

If possible we recommend two internal pilots and then, if you have the
time and the budget, running at least one external pilot to test with
someone who is not close to the project. It should be the same for your
internal pilots as well; make sure it is with someone who is not close to your
product or your project to really test your setup.

What you’re doing in this pilot is testing everything fully: you’re setting up your
meeting software, you’re recording, you’re making sure your prototypes are
working well and your participants can bring up the prototypes.

You’ll want to test the flow of your script and the time needed to run the
study. Take into account that in the beginning of the test there needs to be
set up time and you’ll need to get the participant into the study. Also make
sure there is some time leftover to add in potential questions from your
observers and your stakeholders.

18
the ten phases of remote moderated

The idea behind running two pilots is because the first one can be really
clunky and you might forget about certain things once you get into it. This is
why we make thorough notes on our scripts to say, “When we transition over
to this part in the prototype make sure to turn on recording.” Pretty sure
we’ve all been there and almost forgotten! This is why having very detailed
notes on your script helps set the stage.

#5 Prep Your Observers

Getting your observers prepped is you greasing the pan. Obviously you want
them to join, but you want them to join and not interrupt the session. This is
why it’s important to talk to your observers beforehand to make sure they
understand the rules of the session.

They’re going to want to send you questions during the sessions. Some
people opt to do it through chat in the meeting software, but if you’re trying
to observe the participant and their screen and there’s a lot of action going
on, that may not be the best approach. Another option is to do it through an
IM application that’s separate from your meeting tool. You’re still balancing
a number of windows, however, and might miss the questions or not get to
them in time.

The most basic option, and therefore arguably the easiest, is to have your
cell phone next to you and have your observers actually text you. Set your
phone to vibrate and as soon as a question comes in you can quickly grab it
and see if it fits in the session or if you have to make a note and maybe do the

19
the ten phases of remote moderated

question at the end. It might be a follow-up question, for example.

If not they can join remotely, which is another great factor about doing this
kind of research. There’s a lot of chat that goes on while you’re actually
running the session and your stakeholders can have a great discussion.

If not they can join remotely, which is another great factor about doing
remote. You can have your remote teams join in if the times all line up. For
example, in a previous role I had designers who were in Taiwan so I made
sure that they could join a couple of sessions and it was very beneficial for
them. Being able to remotely observe and then ask questions afterwards and
debrief over a phone call streamlined our efficiency as a team.

Lastly, when you are sending out calendar invites to your stakeholders and
your observers be sure to send them separately from your participants.
This allows you to give them the rundown on how to join and how to send
questions, as well as the ground rules for the session.

Plus you can remind them to add it on their calendar and ensure they have
the invite so there is no excuse not to join. Hopefully this allows you to get
everybody in the session.

20
the ten phases of remote moderated

#6 Right Before The Session

Now you’re ready to pour the batter into your pan. Right before the session
starts take some time to get things ready on your side.

Usually ten minutes before the session begins we make sure that the meeting
software is ready to go and that it’s up and running. We also make sure to
close everything else so there’s no other distractions. Last thing you want is
something popping up while you’re in the middle of a session!

You also want to ensure that your observers are hidden, on mute, and their
cameras are off if they’re joining.

Here’s something to keep an eye out for when starting your session –
depending on what meeting software you’re using, it is possible that when
one of your observers joins, their camera turns on automatically or they aren’t
muted. You have to be lightning quick and turn that off.

Make sure everything is hidden so you can be 100% focused on the

participant.

21
the ten phases of remote moderated

#7 Inviting Your Guest In

Getting the participant in the session is like putting your cake in the oven.
This is the crucial part. Try and get your participant in the study as soon as
possible.

It’s worth mentioning again that no matter how much you prepare it is
possible something will go wrong. Thanks, Murphy’s Law. Take a deep breath
and it’ll be okay!

This is why we recommended that you have a solid five to ten minutes at
the beginning of your session just to get that participant in the study.
There’s a couple of ways you can do this.

If you have a recruiter who is able to do a basic tech check beforehand,

great. They can get on the phone with the participant and ask, “Is your Zoom
installed? Is the WebEx installed? Ok, great. What browser are you using?”
They can do a very basic test setup and go over everything regarding setup
that you sent over in the previous emails.

One of the most common challenges you’ll face is the line, “I’m on a different
computer.” Just walk them through how to reinstall the meeting software and
you’ll be good to go.

Anecdote time – once I was about to run a moderated session and right as
we started to do the tech check there was a power outage in Chicago where

22
the ten phases of remote moderated

the participant was located. One moment he was in the study and the next he
was gone.

This is why it is crucial to have your participant contact information on

hand. All I did was pick up the phone and call him and he said, “I’m really
sorry the power is out, I can’t really do the session.” So, in this case, I was
able to just reschedule. Which leads us to the next point: don’t be afraid to
cancel or reschedule.

If there’s a tech issue you just can’t figure out, rather than spend a lot of
fruitless time trying to solve it, just cancel the session or reschedule it. If you
cancel still pay the participant, obviously, because they’ve put time into it but if
it is something like a power outage try to reschedule if you have an open time
slot that works for them.

Finally, once you get the person in the session, have them bring whatever it is
you’re testing up on their screen. If, by chance, it’s something where you can’t
have it hosted and it is has to run locally on your machine or if there’s some
other firewall issues, have them control your computer. It is not as slick as
experience as if it was on theirs, but it’s still a great way to get feedback

23
the ten phases of remote moderated

#8 The Session

The next one is running the session, which is a lot like patiently checking in on
the cake as it’s in the oven. Your goal at this point is not to change the cake,
just to be observant. This is very similar to running an in-lab session, if you’ve
done that before.

To establish rapport at the beginning we recommend you have your

camera turned on. Yes, you! This helps because the participant sees an
actual person when they join, and not a mysterious council of black screens
with disembodied voices. Which is very James Bond villainesque, no?

Up to that point it’s been all emails and maybe some phone calls, so this time
make sure they’re actually seeing a human face.

If you’ve required participants to have a webcam of their own, or are lucky

enough that they simply assume without having been asked, you can read
their facial expressions, body language, and all the other non-verbal forms
of communications. If not, it is a little bit trickier but still possible to get
great feedback.

Typically when we’re in a session with no webcam, we focus on the tone of

their voice. Specifically, you want to listen for hesitations. Once you’ve heard
them hesitate it’s a balance of waiting until they’re done speaking and then
being able to interject or probe as necessary.

24
the ten phases of remote moderated

It takes a bit of practice but once you do it a few times you get the hang of it
and it becomes not that big of a deal.

Finally, during the session you have to be able to perform some

multitasking. Not only are you observing their screen, you’re taking notes,
you’re potentially receiving and asking questions, maybe even doing some
tech troubleshooting. This too gets easier the more you practice it.

#9 Debriefing After the Session

Let’s pull the cake out and critique it. At the end of the session it’s a great
idea to do a debrief with your observers.

If they’re remote, try and schedule a time to meet and discuss after the
session concludes. It might not be able to be right after, depending on
whether or not you have another session scheduled, but it should be close
enough to the session so that details aren’t left out or forgotten. If you do
meet after but have another session scheduled, be sure to leave enough time
to set up the next session and set a follow up meeting to finish debriefing.

If you have them in the next room, or if they’re all gathered in a conference
room, it’s great to go in there and have a discussion about what the
participant liked, what they didn’t they like, where they had problems, what
stood out to your observers, etcetera.

25
the ten phases of remote moderated

We’ve had observers actually start whiteboarding while they were observing
and start a discussion right then and there as the session went on. They had
two columns: all the stuff that’s really going great, and what’s not going great.
We’ve even had designers just get up and start sketching solutions.

The point is that being an observer can really get the creative juices flowing,
and having a debrief about the session can lend itself to the creative
energy. That’s why it’s recommended to do it after every session, if possible.

Again, sometimes it is not entirely possible to debrief after every session, but
it’s really great just to cap off the sessions with a discussion.

cipe
CakeRe
hite sugar
1 1/2 cup w hite sugar
2 cups w
extract
s vanilla
2 teaspootenen d cocoa powder ocoa powder
1 cup unsw unsweetened
ee c
o ven 3 /4 cup
eps

e g powder
t th er a s p o ons bakin
St

a 1 1/ 2 te
he tt
re- he ba pan
P t cup milk
ix e the o1ven
M s
ea the 2 eggs
Gr e in

#10 Plan For the Future

k
Ca

Lastly: adjusting your recipe. What we like to do – and you’ll probably get
this throughout the session if observers are interacting and sending you
questions anyways – is to gather any feedback about your script.

Look at the sessions, look over the questions from observers, and use your
researcher instincts to answer things like: Do you need to add questions?
Do you need to add a task? Do you need to delete a task? Think about any
adjustments can you make to fine-tune the script.

26
the ten phases of remote moderated

Experienced researchers tend to do this on the fly but if you haven’t run many
moderated sessions on your own this can be an invaluable step. Regardless
of experience you can still ask your stakeholders and observers for any other
feedback you would like to get, as well as if there’s anything that you can do to
optimize the session. Then you rinse and repeat.

For example, maybe you heard from observers that five people on a specific
task was enough, and now they want you to add a different task. You can do
this, it just depends on how you set up your sessions. We’ve had designers
actually make changes to the prototype before the next session.

This has to be highly coordinated though, so take caution, but in certain

cases some things might just need a small adjustment: like maybe they need
to set a link to a button, or maybe there’s a typo in the prototype that all
participants were pointing out which distracted from the session. You can
even do rapid iterations either in-between sessions or after a few sessions.
Just keep in mind that as you make changes to your prototype you’re also
changing your script.

Try and optimize everything so everybody is happy and you’re getting all the
feedback that you need.

27
Remote
Moderated
Checklist
remote moderated checklist

As promised, we’ve distilled down the tips and tricks here into a checklist that
you can print out and mark off as you go through. It’s a nice visual reminder
on the current and next steps you need to take for a successful remote
moderated study.

Click on the image below to open a printable PDF version.

Checklist | Remote Moderated Testing

PLAN
__ Remote meeting account and a way to record sessions
__ Determine scope / length of study (typically 30 to 60 minutes)
__ Create and communicate an overall timeline for the study
__ Determine how many participants
__ Decide on type of stimuli (Prototype, beta site / staging site, live site) – ask yourself: can participants access the stimuli on their
computer vs. do you have to have them control yours?
__ Prepare study script
 
RECRUIT

__ Write detailed screener

__ Set clear participant expectations
__ Include all technical requirements as screener questions
__ If necessary, include an informed consent / recording statement / NDA
__ Determine study schedule. Ensure you include appropriate time of day time slots (ex: if you on the west coast, make sure there are
appropriate slots for east coast participants.)
__ Leave at least 30 minutes in between sessions for reset and debrief
__ Have a plan to deal with no shows (Over recruit by 1-2 participants. Add a “make-up day” to your schedule)
 
RUN PILOTS

__ Run at least 1 internal pilot (2 is recommended) with someone who is not close to the project
(Test the tech set up fully. Test the flow of the full script)
__ If time/budget allow, run 1 external pilot (this helps with technical set up)
 
PREPARE OBSERVERS

__ Send out meeting invites to the observers (separately from participants)

__ Ensure observers cannot interrupt the session, but have a way to send questions
__ If time allows, field questions from observers at the end of the session
 
RIGHT BEFORE A SESSION

__ Give yourself 5-10 min to get the technology working for you (Make sure the participant is fully set up technically before starting the
session. Be ready to cancel the session if you are unable to get the tech set up to work)
__ Have the participant contact information handy (phone, email, etc.)
__ Contact the participant via phone or email if they do not join within the first 10 minutes
__ Make sure cameras are on, working, and pointing in the correct direction
__ Ensure observers are hidden/on mute/ cameras are off
__ Ask the participant to hide anything they don't want you to see and ask them to share their screen
__ Have the participant pull up the stimuli
 
RUN THE STUDY

__ Introduce yourself and give the participants some context for the study
__ Turn on your camera first before asking the participant if they are comfortable turning on their camera
(If not, we recommend, proceed as planned)
__ Remind the participant to think out loud during the session
 
DEBRIEF WITH OBSERVERS

__ Talk about their observations

__ Gather feedback about any changes needed to the script

www.userzoom.com

29
Frequently
Asked
Questions
frequently asked questions

We receive hundreds of questions each year regarding best practices and tips
for conducting research with any number of methodologies. Here are some
of the ones we often receive about remote moderated testing.

Q: What meeting tools would you recommend for remote

moderated testing?

Our recommended solution is Zoom, because that’s our preferred tool and
is the one that we actually built into our product solution. But the question
really ends up being what tool do you already have access to? I’ve worked
in several different corporate environments – in some places it was WebEx,
in others it was GoToMeeting, at other places it was BlueJeans or some
other tool.

So the primary recommendation would be: if you already have access to a

tool, see if you can use that. You just need to make sure that you figure out all
the settings to choreograph correctly. They all pretty much work the same, it
is just tweaking the settings to make sure they work for that session.

Q: What tips do you have for testing on mobile devices?

Testing on mobile devices remotely is a little bit more challenging than

desktop. The range of challenges that can arise vary on different mobile
devices and different versions.

So, during that screening process, you might want to pare it down to iOS only
or Android only. In your pilots, go with a specific platform versus just keeping
it open to all mobile.

31
frequently asked questions

Here at UserZoom we have heard about how some folks use mirroring
technology. As an example, with iOS and Mac, asking participants to mirror
their mobile device with their desktop and then record from the desktop.
That’s also why we prefer Zoom, because with Zoom and the UserZoom
app on the participant’s mobile device we can mirror it on the desktop. For
Android, there are more hacky solutions.

All in all, it’s just going to be more difficult, which means more preparation
and lots of experimenting but is by no means impossible.

Q: How do you send the participants stimuli, especially if

confidentiality is a concern; for example, you don’t want to
email them a copy?

Nine out of ten times, or even ten out of ten times, if it’s that secrative its
usually a hosted prototype and isn’t anything that’s emailed to the participant.
Even if it is just a PDF. You can run that from your machine and let the
participant access your machine and then run the study that way.

When it comes to prototypes in a UserZoom study, the prototype is

associated with a task so the participant never truly gets the URL. If it is not
task-based, and if it is a starting URL, we’ve used either Zoom chat or WebEx
chat to send that link and then just remove the prototype after the session
or after all sessions are done. So there are several ways of essentially be
in control.

This is where it is up to you and your company to discern the level of

confidentiality because there are definitely projects that need to remain a
secret. As a general warning, there is no way to really prevent people from
taking screenshots. So it just depends – is doing an unbranded prototype
good enough? You have to test the waters with your team and say “is it okay
if we un-brand it?” and then do a blind study so participants won’t know from
which company it’s from.

32
frequently asked questions

Also, you need to make sure that NDA is in place and are enforcing it. But it’s
really up to you to figure out if it is super confidential and highly sensitive –
maybe it’s a brand new product that is pre-released – then maybe not run a
remote study.

Q: In regards to adjusting a script on the fly to test the

robustness of your results – what are your rules of thumb
about what not to change between sessions?

This is just a general moderated question: how much script adjustment do

you allow for between sessions. The answer really depends on what was the
purpose of the study.

We’ve done a lot of benchmarking studies, for example. We do not adjust

the script in a benchmarking study because the intent of that study, either
moderated or unmoderated and either in-person or remote, is to get to Key
Performance Indicators. In those benchmarks we have defined and piloted
and adjusted the script before collecting a lot of data.

If the intent is rapid iteration and to refine designs then that’s okay. The tricky
part is where we observe a problem – does it have an obvious solution? Or
is this an obvious problem with no solution? You don’t want to make massive
changes when trying to figure out what is happening.

If you know it’s an obvious issue due to repeated participants running into it
and you’re fairly certain of it’s fix, then by all means confirm your hypothesis.
But if you’re not yet sure how many participants are running into it or if you’re
not quite sure what the solution is, we wouldn’t recommend going in and
tweaking like crazy until you have more data.

33
frequently asked questions

Q: From my experience it is very hard to pay attention and

fully engage with the participant while trying to take thorough
notes. Do you typically try to be your own note-taker or do you
have a partner or an observer assigned to notetaking or do
you watch the videos later or you just don’t even take notes?

It really just depends on if you have someone available to take notes for you
and if that’s what works best for you. Personally, I feel like I’m more engaged
if I’m taking notes. They may not be super thorough, but I’m definitely writing
down key insights and/or highlighting timestamps as the session is going
along. Then I actually end up going back through the videos to catch those
key points and write down whatever that key insight is. “Participants struggled
with drop down” or whatever it was that happened.

This might not work with you, so it’s all about finding what will work for you.
But, yes, I totally know what you mean as far as the multitasking of observing
and taking notes and troubleshooting. It’s not easy!

Q: Sometimes, even if we make this crystal-clear before

the session, participants still join the session with a noisy
background, i.e. Starbucks. How do you handle this situation?

Fair question – I have little doubt that this has happened to many of us. You
try to do as much as you can to set the right expectations, but things slip by
or the participant forgets or has no choice. The only recommendations are to
work with a good recruiter you have a good relationship with and to work very
hard on setting expectations. Give examples of how quiet you need it - “more
library, less coffee shop.”

When we are doing this for our customers, we work with the panel team to
set the right expectations. Most of the time participants are not doing this on
purpose, they just don’t realise how loud the background is. In some cases, if
it becomes extremely noisy and you really can’t hear, feel free to reschedule.

34
frequently asked questions

Q: How do you address pushback from stakeholders if they

are not a fan of remote moderated studies?

I normally focus on what goals you’re trying to accomplish. One thing that
really sells it with stakeholders is the variety of participants you can reach
within a geographical area and the fact that you can get international
participants.

We hear that sometimes they are upset that after spending money on a
usability lab you’re not using it, so remember that you can always use the lab
for this. Just run the remote study from your lab. You are still using the lab,
and you’ll still have your observers in the next room. The only thing that’s
different is that your participant just isn’t there. But as long as you focus on
the goals and on what you’re trying to accomplish, you can really sell this to
your stakeholders.

Also, keep in mind that it doesn’t have to be 100% in-lab or 100% remote. You
can do 5 in person and 10 remote, for example. Figure out what’s going to work
best for you and don’t get bogged down by only using one available option.

Q: Do you compensate participants differently for in-person

and moderated?

Yes we do. The in-person is usually a higher compensation because if it’s an

hour-long session they’re probably leaving from work/home an hour earlier,
coming in, and then going back. We are essentially taking up almost three
hours of their time. Logistics involves some support, after all.

With remote they’re in their home, so it is usually lower. They can essentially
be in their pajamas and do a study with you, as opposed to asking them to
take time out of their day to drive in and make sure they’re there ten minutes
beforehand. There’s a lot more commitment than if they are doing the
study remotely.

35
 Want to learn
more about
UserZoom?
Get in touch

UserZoom, The UX Insights Company More info at userzoom.com