THE NEW RESEARCH PLAYBOOK

Product, design and research leaders identify

3 shifts that will unlock the value of research

Stripe Partners
Executives are rethinking the role research plays in
product development and strategy. Stripe Partners
interviewed senior designers, product managers and
researchers to define a new research playbook for
decision makers in the midst of reorgs. The playbook
outlines three fundamental shifts that executives
must facilitate to ensure researchers are set-up to
create meaningful impact in a fast evolving context.

Stripe Partners – The new research playbook 2

It’s the end of an era. The economic uncertainty
has combined with maturing business models and
accelerating product development cycles to trigger
a reassessment of the value research creates in
technology companies. As a result, researchers
have been badly impacted by the layoffs of the last
12 months.
Executives are reshaping product research
with the hope it will provide a greater return on
investment. But do decision makers know how to
maximise the value of research for their company?
We interviewed seven high profile insiders—
spanning product, design and research—to
understand what’s gone wrong, what good looks
like and what’s next. Inspired by their insights, we
have identified three key shifts that executives
must instigate if they care about making research,
and researchers, more effective.

Stripe Partners – The new research playbook 3

 TODAY
Diminishing value
KNOWLEDGE MANAGEMENT
Siloed
SKILLS AND METHODS
Specialised
PROBLEM SPACES
Hypothesis driven
1 The knowledge management The UXR role is relatively young. Re-
shift: from siloed expertise to searchers have worked hard to estab-
organisational knowledge
“As a designer, it makes it very real
sitting and listening to someone say it,
and seeing their context. It just makes
my work so much easier, right?”
Design Director, Spotify
Successful start-ups don’t hire re-
searchers immediately. Founders are
often focused on a narrow problem
that they have experienced them-
selves. They understand their target
market intuitively. But as organisa-
tions mature and offerings diversify,
product teams become distant from
end users and / or become biassed
towards the narrow band of users that
reflect themselves. To solve for this,
executives hire specialists who “own”
the user perspective within an organi-
sation: the user experience
researcher (UXR).
Stripe Partners – The new research playbook
TOMORROW
Expanding value
Embedded
Integrated
Ambiguous
lish their discipline in a short period
of time. Effort has been made to build
and maintain boundaries, carving out
a distinctive role within and across
product teams. This involves advocat-
ing for the skills and credentials re-
quired to conduct legitimate research.
Understanding the user has shifted
from a pervasive form of knowledge
that imbues all decisions, to a narrow,
specialist domain, embodied in a spe-
cific type of person. Charles Eames
advised “Never Delegate Understand-
ing”, but that’s what executives, de-
signers, engineers and product man-
agers are tacitly expected to do.
In reality, stakeholders are unwilling
to delegate understanding. “I talk to
a lot of different users informally”,
Chris Butler (Google) told us, “I try to
understand what are the problems
4
people are having and contextualise
that in some way. But it’s not “official”
research.”
The dissemination of “official”
research is an important step
in establishing standards
of quality and integrity, but
it can also destabilise other
legitimate forms of knowing.
It invites a battle between the
researcher—the certified “voice of
the user”—and stakeholders who
intuitively combine personal hunches
about the user with knowledge from
their own expert domains (be that
business, design, or engineering-
orientated).
This dynamic is exacerbated by the
fact that the qualitative nature of user
research doesn’t naturally integrate
with the mental models of stakehold-
ers who pride themselves on being
hypothesis-led and (big) data driven.
As a result they instinctively reject
insights that don’t reflect their own
experiences and observations.
Research is set-up to fail
Product research is also devalued
because it is judged on criteria
that set it up for failure. Some
organisations are, as a Google Design
Manager put it, “very measurement
oriented. As a researcher, if you want
to get promoted, you have to point to
something pretty real that happened
with the product that can be tied back
to your research.”
Stripe Partners – The new research playbook
First, this grounds the value of re-
search solely in terms of shipped code
(an output), rather than the end user
experience or business impact (the
outcome). Second, this output orien-
tation pulls researchers towards short
term product iteration where impact is
incontrovertible, but user research is
increasingly less valuable than experi-
mentation (see theme 3, below).
And when a study has no short-term
product implications, impact is often
judged through a citation system, by
accumulating references in the work
of other researchers. This creates the
opposite problem, where research is
consumed and discussed largely by
researchers rather than product de-
cision makers, amplifying the impres-
sion of a siloed caste of specialists.
From archiving knowledge to
meshing knowledge
This drive towards short-term prod-
uct impact, on the one hand, and the
production of well-cited artefacts on
the other, misses an important third
dimension: knowledge transfer to
non-research stakeholders. Research
is truly successful, according to our
interviewees, when it has entered into
everyday language and become tacit
knowledge that informs daily deci-
sions, be they strategic or tactical.
Tacit knowledge is not created
through slide decks, however well-de-
signed. Our interviewees agreed that
impact is invariably driven by the level
of leadership engagement. Senior Ex-
ecutives, Designers, Product Manag-
ers and Engineers should be incentiv-
ised to actively participate: immersing
5
themselves in the research process so
they embody the insights first-hand
(especially in the context of remote
work where teams may never even
have met each other face to face, let
alone their users). A seasoned Product
Manager, Hugo Alves (Synthetic Us-
ers) reflected on the best practice he
observed in one position: “Every Mon-
day myself or our engineer would run
a user interview…and then at the end
we would debrief and we would get
feedback from our user researcher on
how we could improve.”
It is only through participation
that teams will develop deep
and intuitive understanding
of their users, so research-
ers must therefore be direct-
ly evaluated on the quality of
other people’s knowledge,
rather than their own.
Of course, curating the knowledge of
others is not just about encouraging
senior stakeholders to participate.
Researchers should be incentivised to
embrace and evangelise what is al-
ready known. “Running another study
becomes the coin of the realm” says
Melissa Cefkin (ex-Waymo), “But who
says we don’t already have answers?
You might just need somebody push-
ing at it and pulling together things in
such a way that helps you.”
Decision makers, then, should design
a research function that surfaces,
validates and shapes existing forms of
knowledge that exist across the or-
ganisation, as well as producing and
proselytising their own, original work.
Stripe Partners – The new research playbook
The mental model needs to be “mesh”
rather than “archive”. To do this there
needs to be an openness to different
forms of knowledge, and the diversity
of methods that create it. This brings
us to our second shift that executives
need to enable in their organisation.
2 The skillset shift: from
narrow user research to
integrated data
“How does qualitative data help to tell
the story of behavioural data? How do
we start to link up patterns of stories
with the data science world in some
way? I just don’t know how it should
be done yet in a way that still keeps
the right fidelity of information.”
Chris Butler, Lead Product
Manager, Google
One downstream consequence of or-
ganisational specialisation is method-
ological specificity. When executives
define the role of the researcher nar-
rowly as “the voice of the user”, certain
ways of understanding the world are
privileged, and by extension, attach-
ments to certain approaches.
User research is dominated by people
with backgrounds in anthropology,
sociology and the human sciences.
Their training makes them instinctive-
ly wary of deracinated commercial
and behavioural data. Rightly, they
value the importance of context, and
the rich, multifaceted insights that,
they believe, only qualitative methods
can solicit. Methodologically they are
minded towards inductive approach-
es, in which patterns and insights
emerge bottom up.
6
Data Science as a discipline has
grown in parallel with user research,
and is populated with practitioners
with different sympathies: they value
data that is unambiguous, quantifiable
and scalable. Their approach is built
on the confidence that large datasets
can provide.
The pre-eminence of Data Science
On paper, these ways of knowing the
world are complementary. In practice,
they can be antagonistic—particular-
ly when data scientists and user re-
searchers work in isolation from one
another, often on the same problem.
“In a big tech environment”, says one
Google Design Manager, “they’re re-
ally very different teams with totally
different reporting lines. I don’t see
the kind of deep collaboration that
I thought would happen and I think
should happen.”
As more and more digital traces be-
come legible, data science strength-
ens its position relative to user
research. “As data collection and
engineering has evolved”, observes
Heli Rantavuo (ex-Spotify), “there is
increasing pressure for all decisions
to be driven by data science.” By com-
parison, once progressive qualitative
approaches such as “Design Thinking”
feel staid and indeterminate, a relic of
a past era. Data Scientists sense the
expanding influence of their dynamic
discipline, and, in truth, have little in-
centive to engage with user research.
The fact that qualitative data is wrong-
ly judged on quantitative terms is not
new, and is partly an issue of educa-
tion. But this false competition also
Stripe Partners – The new research playbook
points to a misunderstanding of the
kinds of questions each approach is
suited to address.
Translating between complex
and complicated
Drawing on the Cynefin
framework, Chris Butler
(Google) delineates between
“complicated” problems
that can be addressed by
experts working with data
that is already legible, and
“complex” problems—opaque
human systems which are
non-legible to quantitative
and data science methods.
For him, qualitative research
thrives when it explores
complex problems, and
translates them into simply
complicated ones that can in
turn be interrogated through
data science.
This integrated, virtuous loop between
complex and complicated is some-
thing that user researchers need to fo-
cus on building and advocating. Deci-
sion makers must design for intimate,
open-minded collaboration between
user research and data science col-
leagues, founded on new interdiscipli-
nary processes.
It also means keeping researchers fo-
cused on work that provides genuine
business value, which brings us to the
final shift businesses need to make.
7
3 The problem space shift: from A new wave of tools are being
testing hypotheses to enthusiastically adopted to
addressing ambiguity
“Everything’s just getting faster and
faster and faster, right? The turna-
round from coming up with an idea to
get it in people’s hands is much faster
than it used to be. And I think that has
all sorts of implications in terms of
what research you do.”
Design Director, Spotify
Current organisational incentivises
focus researchers on “middle-range”
questions. As Judd Antin (ex-Airbnb)
puts it, these questions are “a dead-
ly combination of interesting to re-
searchers and marginally useful for
actual product and design work.”
Middle-range research
addresses questions that
product teams already
think they know the answer
to, essentially vetting
and iterating existing
hypotheses. Middle-range
research slows down the
product development
process without adding
sufficient value. It is a major
reason research is often
dismissed as either obvious
(if it confirms assumptions),
insubstantial (if it challenges
them) or is ignored entirely
(if it takes too long).
Stripe Partners – The new research playbook
circumvent traditional user research
and speed up product iteration. We
heard that cutting edge A / B testing
capabilities are cheap, personalised
and rapid, enabling product teams
to jump ahead with their hypotheses
and collect “real” usage data rather
than wait for a user research process
to deliver. And where hypotheses
are less clear cut, generative AI
tools promise to provide teams with
sufficient stimulus to swiftly graduate
to the experimentation phase. User
research is squeezed in the process,
constrained by what it is invited to do
and then ostracised for its inability to
deliver in a timely fashion.
Research’s unique value proposition
In this context researchers need to
ask themselves what is the unique
and defensible value they will contin-
ue to provide. The answer from our
interviewees was simple: focus on the
intersection between novel, inspir-
ing and useful. Chris Butler (Google)
explains, “Good research is always
surprising. The biggest value is iden-
tifying the things that we didn’t know.
Because otherwise they’re like, I just
spent all this money and you just told
me exactly what I know already.”
Hugo Alves (Synthetic Users) agrees
that this is the superpower of good
researchers “I think humans will al-
ways have a role, because we are de-
lightfully surprising. We can say things
and do things and explain motives in a
way that I don’t think will be ever fully
grasped by AI.”
8
To deliver genuine insight
it’s important to delegate
questions with strong
hypotheses to automated,
standardised techniques,
enabling researchers to
dedicate their time to more
ambiguous problem spaces.
A Design Director at Spotify
calls this sweet spot “the 20%
where the magic happens”,
another interviewee talked
about the sustaining value of
“handcrafted insights.”
Focusing on ambiguity
Locating ambiguous problems has
implications for who researchers work
with. In Kent Beck’s formulation, prod-
ucts are always at one of three stages
of the “S-Curve”: explore, expand
or exploit.
If a product is in expand and exploit
mode then it’s hard for a researcher
to compete with fast iteration. But if
they’re in explore mode, then gener-
ative research makes a huge differ-
ence. This means decision makers
need to give researchers autonomy to
move on from teams when they are no
longer adding sufficient value.
But it’s not just through upstream,
strategic work that researchers can
shine, argues Judd Antin. Ambiguity
can also be located in more “tactical”
usability work: “there’s a thing that an
A / B test can never do, which is tell
you why the result occurred.
Stripe Partners – The new research playbook
There is no data science that can tell
you the why and how user behaviour
led to this outcome. It’s not sexy, it’s
not what a lot of researchers want to
work on, but I think it earns us busi-
ness credibility.” In this sense, the key
heuristic is not whether a team is in
explore, expand or exploit, but how
ambiguous are the problems they’re
trying to solve for.
For some it will be hard to find this
ambiguity in the current context.
Which may mean proactively looking
for a new company. We heard from
some of our informants that there is
little point in advocating for meaning-
ful research because of the nature of
its market or maturity.
As Melissa Cefkin (ex-Waymo) ex-
plains, “I’ve reconciled. There is not
always going to be a greater thirst for
more research. But nonetheless, I will
bring that perspective and have im-
pact when and where I can. I think
we can do that very well where it
is desired.”
9
What’s next: From truthsaying to changemaking

“Researchers need to say, hey, I have a hypothesis.

What would it look like if we did this? Actually
bringing things to life more directly, even without
necessarily having the craft skills of actually doing
tactical design work.”
Design Manager, Google
Executives hire researchers to find and report
the truth. But unfortunately the facts don’t speak
for themselves. Creating meaningful change in
a modern product company requires a different
body position. The senior product, research,
design professionals we spoke to taught us that
organisations need to rethink the role of research in
three ways.
First, the research job description needs to be re-
written. From the passive specialist disseminating
personal expertise to an active curator of other
people’s knowledge. This means incentivising
researchers to establish relationships across silos,
and understand and embrace what stakeholders
already know about users (through their own expert
practices). Once these stakeholders feel seen,
they will be more open to having their assumptions
challenged, especially when the research process
involves their full participation.
Second, combine user research with data
science and market research. Recommendations
based on small-samples alone don’t drive significant

Stripe Partners – The new research playbook 10

decisions. It is only by expanding the methodological
arsenal, and encouraging close integration with
data science and market research that decision
makers will pay attention to user research. This
means commissioning interdisciplinary studies
and processes that produce more compelling,
well-rounded data. User research, in this context,
becomes the human thread that ties other data-
points together in a meaningful way to compel
action. User researchers will become pivotal in
the process.
And third, researchers need to be focused
on ambiguous problem spaces where their work
really moves the dial. Asking them to compete with
accelerating experimentation cycles in companies
that already have clear product-market fit is a
recipe for redundancy. It is by focusing on “complex”
questions where there are few existing hypotheses,
and translating them into “complicated” questions
that are legible, that user researchers will really add
value. This means training them to automate or triage
hypotheses-strong requests, and guiding them to
work for teams and companies in which expansive
thinking is a necessity not a nice-to-have.
Researchers can’t be expected to achieve
these three shifts themselves. These are structural,
organisational changes that are mandated by
executives. The current flux offers an opportunity
to do so.

Stripe Partners – The new research playbook 11

Our interlocutors:
Judd Antin, Design and Research Leader, ex-Airbnb, ex Meta
Chris Butler, Lead Product Manager, Google, ex Meta
Design Manager, Google, ex Meta
Design Director, Spotify
Heli Rantavuo, Research Leader, ex-Spotify
Hugo Alves, Product Manager and Co-Founder, Synthetic Users
Melissa Cefkin, Research Leader, ex-Waymo

Stripe Partners will continue to host a series of discussions

on the Future of Research in the coming months. To join the
conversation email tom.hoy@stripepartners.com - we’d love
to hear your perspective.

Stripe Partners – The new research playbook 12