Pulse Survey

CLOSING THE
DATA GAP IN PRODUCT
DEVELOPMENT

Sponsored by
SPONSOR PERSPECTIVE
From Code to Boardroom: How Engineering Practices and Data-Driven
Insights Can Influence Business Goals
In today’s digital world, instant gratification is everything,
which means a poor online user experience can be the
difference between winning and losing. To keep up with
heightened product expectations, businesses must focus on
creating customer-centric applications. Fast. But how does an
organization ensure that it does not acquiesce to the pressure
to move perpetually faster at the expense of delivering a
quality product that delights the customer? What we need is
a cultural revolution in the engineering world—one that keeps
the customer at the center of everything we do.
The Unexpected Customer Advocate
Engineers and developers used to work deep within the
organization in an often-isolated capacity. Now, they’re more
front and center, with more customer data and insights than
many others in the organization. As a result, this traditionally
execution-focused group has catapulted into taking part
in decisions that affect the entire business. After all,
experimentation with new application features and rollouts
means they’re working with real-time customer behavior
data on a daily basis. Who better to ask whether customers
are loving your new application than the folks tasked with
building it?
When it comes to delivering the next killer application
feature, it can’t be just about speed, and we’ve learned that
the hard way. Unfortunately, the reality is that the majority of
applications delivered today have little to no business value.
But, thanks to product experimentation, development teams
now have the ability to test, adjust, and roll back features
based on real-time customer data; businesses now have
direct visibility into the success of their investment through
the eyes of its engineers.
Creating a Culture of Continuous Improvement
As development-cycle times shrink, more modern and
iterative approaches are imperative. Today, we can measure
the impact of change on business and/or engineering metrics
in real time. Lean software development—a translation
of lean manufacturing principles and practices to the
software development domain—has allowed IT teams to
move faster than ever. In an effort to keep up, teams began
to tackle challenges one piece at a time. By working in
very small increments, the feedback loop became faster
and more efficient. By bringing in more people to make
incremental decisions, teams found that they could keep
the process moving forward without getting stuck in a
bottleneck of approvals. Traditional processes left nearly all
feature approvals with leadership, causing a slowdown in
development cycles.
As business leaders, we must learn from this iterative
approach—one that assumes both that ongoing evaluation
of success is part of the process and that the entire team has
a hand in contributing to progress. By facilitating a culture
of continuous improvements, we have an opportunity to
change business models to decentralize decision-making,
encourage experimentation, and foster leaders throughout
the organization.
Not every idea cooked up in the boardroom works in
practice, and that’s OK. It’s more important to be flexible and
constructively critical of your own ideas than to be tied to a
product road map that’s not aligned with ongoing customer
feedback and usage data. Experiment early and often,
understand what works, and don’t be afraid to scrap things
that appear to be providing little benefit, or worse, could
adversely impact your customer’s experience.
At today’s pace of business, managers can’t keep tabs on
every last detail of the business, so trust and empower your
team to make good decisions. And, thanks to high-quality
and real-time data, it’s easier than ever to trust that those
decisions are right for the business. Imagine the innovations
that could be around the corner if we all focus a little more
on continuous learning and always keep the customer at the
center of our business goals.
— ADIL AIJAZ, COFOUNDER AND CEO, SPLIT
CLOSING THE
DATA GAP IN PRODUCT
DEVELOPMENT

Only a small minority of executives—17%—are very confident that HIGHLIGHTS

their organizations are getting the greatest possible return on their
product development investments. A global Harvard Business Review
Analytic Services study also found that nearly 30% have little or no 52%
confidence at all. OF COMPANIES SOMEWHAT OR
STRONGLY AGREE THAT PEOPLE IN
THEIR ORGANIZATIONS HAVE THE
TOOLS THEY NEED TO MAKE DECISIONS.
The challenge isn’t so much how new products and features are being developed.
Nearly 50% of companies are using agile development approaches, and more plan
to do so in the next three years. More than half report that their organization’s
engineering, product development, marketing, sales, and service functions 43%
REPORT THAT DATA IN THEIR
collaborate on developing new products and features. And nearly 70% say that
COMPANIES IS WELL MANAGED
employees are comfortable bringing up new ideas. AND PEOPLE ARE CONFIDENT IN ITS
ACCURACY.
So why are confidence levels so low? Lack of sufficient data. The vast majority of
executives believe too many product decisions are based primarily on intuition and
experience. Business leaders crave more testing with customers and more detailed
information about how products perform in the market. Respondents also say they 30%
HAVE SUFFICIENT ADVANCED
need more timely and accurate data about product development investments.
ANALYTICS SKILLS TO DESIGN AND
Decision-making processes and corporate structures aren’t helping matters. CONDUCT COMPLEX ONLINE PRODUCT
EXPERIMENTS.
Most organizations still have hierarchical decision-making processes in rigidly
structured organizations. Senior executives make most of the product decisions.
By the time decisions go up and down the chain and issues are discussed, the
product development process can slow to a crawl. At a time of rapidly changing
customer demands and market conditions, barely 30% of respondents say that their
companies can bring products to market faster than their competitors can.
However, the survey also suggests that dramatic changes are on the near horizon.
A significant number of companies are planning to revamp their organizational
structures and empower greater numbers of teams and employees with product
development and other decisions. Many organizations are also planning to adopt
agile product development approaches and increase product testing.
But one gap still needs to be filled: Business leaders say that their organizations
need better data and analytics capabilities than they currently have. Without
them, empowerment will lead to chaos and poor decisions. Strict top-down
decision-making processes will crawl at a snail’s pace and possibly collapse under
their own weight.

Pulse Survey | Closing the Data Gap in Product Development Harvard Business Review Analytic Services 1
Companies should take greater advantage of the ballooning number
of user-friendly applications and algorithms that can help everyone
touching the product development process gain insights and market
intelligence needed for decisions.
The Data Gap
By and large, executives say they
aren’t getting enough information
about the business impact of products.
Nearly 80% of executives say they
would have greater confidence in
their organizations’ return on product
development if more decisions were
based on data instead of intuition. FIGURE 1
The data gap in many organizations
stems from insufficient analytics skills
and technology. Most organizations
barely have the essentials covered.
For example, only 52% of companies
somewhat or strongly agree that
people in their organizations have the
tools they need to make decisions.
Only 43% report that data in their
companies is well managed and people
are confident in its accuracy. An even
smaller number—30%—have sufficient
advanced analytics skills to design
and conduct complex online product
10 20 30 40 50 60 70 80 90 100
experiments.
FIGURE 1
EXECUTIVES WANT MORE PRODUCT AND MARKET DATA
Percentage of respondents who say each of the following would increase their confidence
somewhat or to a greater extent that their organization is getting the best possible return
on its product investments
More detailed analysis of product performance in the market
90%
More testing with and feedback from customers during the development process
90%
Fewer decisions based mostly on intuition and/or experience
80%
More timely and accurate tracking of product development expenses
78%
SOURCE: HARVARD BUSINESS REVIEW ANALYTIC SERVICES SURVEY, AUGUST 2018
2 Harvard Business Review Analytic Services
Adi Alon, a partner and managing
director at Accenture Strategy, stresses
that organizations need to build three
foundational blocks to achieve the
analytics capabilities required to
assess the business value of and drive
business decisions about products and
to test new ideas. In the past few years,
the market has developed a wide
variety of tools that make it possible
for companies to up the analytics game
without armies of PhDs.
According to Alon, organizations first
have to cover the basics. For product
development, that means integrating
customer, product, and market
intelligence data to enable analytics-
driven insights. For example, if the
company isn’t a digital native or an
e-commerce business, it needs to
liberate customer data from legacy
systems in different channels to
create a single view of its clientele.
The process should be driven by
meticulous and clearly articulated
data management processes to ensure
confidence in the accuracy of the data.
Since only slightly more than half of
companies say their employees have
access to the analytics tools, Alon
stresses that companies should take
greater advantage of the ballooning
number of user-friendly applications
and algorithms that can help everyone
touching the product development
process gain the insights and market
intelligence needed for decisions. But
he also cautions that widespread use of
advanced analytics, including artificial
intelligence, represents a significant
cultural change: Decisions that were
once viewed as the exclusive realm of
creativity and judgment can now be
informed by or even acted upon based
on algorithms and digital tools. Leaders
Pulse Survey | Closing the Data Gap in Product Development
 FIGURE 2
and managers will be working with
machines, and sometimes technology HOW FREQUENTLY SENIOR MANAGEMENT REVIEWS THE
will make decisions. That represents a
major cultural change that needs to be
BUSINESS IMPACT OF PRODUCTS
Percentage of respondents who indicate each of the following frequencies for business
managed skillfully. impact analysis of new products
Online product experimentation
is becoming the next major digital • LESS THAN ONCE A YEAR
• ANNUALLY
• SEMIANNUALLY
• MONTHLY
• WEEKLY
• ONGOING BASIS

capability. Companies such as

35%
Microsoft, Amazon, and Facebook
conduct as many as 10,000 30 28%
experiments a year, according to 25%
25 25% 24% 25%
Stefan Thomke, a professor at Harvard
21%
Business School and an expert in 20
online experimentation. But very 15 14%
few organizations have the needed
10% 10%
skills to conduct disciplined online 10 8%
7%
experiments. 5
5%

These experiments can capture 0

highly accurate data about markets
and product ideas. When done on a
large scale, online experimentation
also provides intelligence about
market trends and when they start
to change—a challenge for many
organizations.
As an example, Thomke points to
Microsoft’s Bing search engine. By
experimenting with large-scale
enhancement ideas to increase
revenue, Bing’s revenue per search
has jumped by 10% to 25% per year.
These improvements, combined with
hundreds of other ideas focused on
customer satisfaction, have boosted
the company’s share of U.S. searches
on personal computers to 23% in 2017
from 9% in 2009.
Alon also points out that advanced
analytics can help organizations
become more risk tolerant by
monitoring and quantifying the risks
of various product development forays.
“Product development processes
often narrow in on the safest ideas,” he
says. “But if the risk data is accurate,
companies will be more confident
that risk is under control and can
then develop a new product portfolio
that balances different levels of risk
according to the company’s
risk appetite.”
The amount and accuracy of data
aren’t the only issues. Infrequent
data analysis is another impediment.
Fewer than 30% of companies analyze
NEW PRODUCTS NEW FEATURES OR ENHANCEMENTS
TO EXISTING PRODUCTS
SOURCE: HARVARD BUSINESS REVIEW ANALYTIC SERVICES SURVEY, AUGUST 2018
the business impact of products on
an ongoing basis. Thus, a significant WHEN DONE ON A
number of senior executives are in the
dark about product performance and LARGE SCALE, ONLINE
the return on new product and feature
investment for much of the year. FIGURE 2 EXPERIMENTATION
Product Development
ALSO PROVIDES
Advances—Without Enough Data INTELLIGENCE ABOUT
At the same time that businesses
suffer from a relative dearth of MARKET TRENDS AND
information about new product
business performance and investment,
WHEN THEY START TO
they are geared to make strides in the
use of nimble and customer-centered
CHANGE.
approaches to product development in
the next three years.
Approaches such as agile and lean
startup are on the rise. According to
the survey, nearly half of companies
currently use agile development
approaches. But that number is
expected to climb to 65% in three years.
Even more significant, only about
25% of organizations are currently
embracing lean startup methodologies.
That number is expected to nearly
double in the next three years. FIGURE 3
More traditional methodologies such as
waterfalls and stage gates are holding

Pulse Survey | Closing the Data Gap in Product Development Harvard Business Review Analytic Services 3
 10 20 30 40 50 60 70 80 90 100

FIGURE 3 more nimble if stages and gates are

HOW WILL PRODUCT BE DEVELOPED IN THE NEAR FUTURE? fundamentally rethought. “The idea
of using stage-gate processes is to
Percentage of respondents who indicated that each of the following is currently in use
and which will be used in three years have linear process management
checkpoints along the way that
• CURRENTLY
• IN THREE YEARS invite input from stakeholders and
give management some control over
Stage-gate or funnel process decisions and how resources are
46% deployed,” he says. “However, the
50%
problem with a linear flow is that it’s
Agile product development processes not a flexible and responsive process
46% that is slow in discovering problems
65% (often too late).”
DevOps methodologies (engineers and product developers collaborating throughout the entire development process)
To make stage gates and waterfalls
42%
58% more flexible and responsive,
companies are making the process
Agile product management processes
more iterative while preserving the
38%
59% checkpoints. It all boils down to the
work that’s done at each phase and
Continuously testing new product ideas with a customer panel
what is required at the checkpoint.
27%
“Instead of having separate and long
55%
phases such as requirement definition,
Lean startup methodologies design, build, test, and scale, all these
26% phases are collapsed into one single,
48%
fast phase/iteration,” he says. “At the
end, this is an experiment or test. This
SOURCE: HARVARD BUSINESS REVIEW ANALYTIC SERVICES SURVEY, AUGUST 2018 sequence is then repeated with each
iteration covering different aspects of a
product with increasing accuracy and
test coverage.”

THE PROPENSITY OF ORGANIZATIONS TO MAKE SURE Alon points out that for certain
products, waterfalls or a linear
THEIR PRODUCT DEVELOPMENT STAFF IS FULLY approach is adequate and will likely

UTILIZED CAN LEAD TO NEW PRODUCT PROJECTS start to combine with more agile
techniques. If a product development

LINED UP IN QUEUES. project focuses on feature extensions

their own and are expected to continue
doing so in the next three years: Nearly
50% of companies rely on waterfall
or stage-gate processes and plan to
continue using them in the future.
Waterfall processes have been subject
to a great deal of criticism in the past
few years. As insiders vetting the
development process at each stage, the
leaders who manage these processes
are subject to the organization’s
pressures to maintain the status quo.
As a result, the waterfall process can be
very slow and often veers toward only
incremental improvements.
However, Thomke points out linear
processes can be much quicker and
for a product in an established market
with not much uncertainty, the heavy
emphasis on advanced planning
inherent in waterfalls can make for a
smooth process. But if there is a great
deal of uncertainty and change may be
needed down the road, agile is a much
better approach.
Irrespective of what methodologies
a company uses, Thomke points
out that there are certain product
development principles that should
be followed to make sure the process
is efficient and transparent enough
to regularly assess the expenses and
use of resources. For example, the
propensity of organizations to make
sure their product development staff is
fully utilized can lead to “new product
projects” lined up in queues and
project delays. Surprisingly, lowering

4 Harvard Business Review Analytic Services Pulse Survey | Closing the Data Gap in Product Development
 10 20 30 40 50 60 70 80 90 100

FIGURE 4
utilization can actually speed up
product development. HOW WILL COMPANIES BE STRUCTURED IN THE FUTURE?
Percentage of respondents indicating their organization’s current structure and what
they expect it to be in three years
Radical Change Coming to
Corporate Structure and
Decision Making • CURRENTLY
• IN THREE YEARS

The majority of organizations currently Separate business and functional units with little or modest amounts of collaboration between them
have a traditional structure with 57%
separate functional and business units 19%
with little collaboration between them.
In three years, however, companies are Mostly team-based structure with a great deal of cross-functional collaboration
planning to make a dramatic shift and 40%
adopt a mostly team-based structure 71%
with significant cross-functional work.
FIGURE 4 SOURCE: HARVARD BUSINESS REVIEW ANALYTIC SERVICES SURVEY, AUGUST 2018
At the same time that corporate
structures are being revamped,
decision-making processes are also
being turned on their heads. Currently, FIGURE 5
decisions in most companies are top
down. In the near future, however, THE FUTURE OF CORPORATE DECISION MAKING
Percentage of respondents indicating their organization’s current decision-making structure
businesses will opt for more open
and what they believe it will be in three years
structures where employees and teams
are empowered to make decisions,
including those about products. FIGURE 5 • CURRENTLY
• IN THREE YEARS

Top-down management hierarchy and decision-making processes

The pressures to make such dramatic
64%
changes are powerful. As customer
23%
demands constantly change and
technology alters what is possible,
A more open and flexible structure where employees and teams are empowered to make many decisions
empowerment and collaboration are
34%
no longer simply progressive ideas 70%
with a few visionary devotees. Flatter
and faster organizations are becoming
necessary as business complexity SOURCE: HARVARD BUSINESS REVIEW ANALYTIC SERVICES SURVEY, AUGUST 2018
grows around the world. The possibility
of surviving global competition with
slow top-down decision making seems and challenges. Companies need to
increasingly less viable. choose carefully.
As organizations turn to more open For example, self-managed
structures and empowered decision organizations push decision-making
making, companies must master responsibility closer to where
analytics and get the right tools into the the work is done, which many
right hands. If people don’t have access find reduces red tape, eliminates
to data whose meaning is agreed upon, cumbersome sign-off processes, and
chaos and indecision will run rampant solves for the fact that sometimes
with errors happening everywhere. in a traditional organization—with
hierarchical reporting relationships
Organizations need to be disciplined
and complicated titles—it’s difficult
in making these transitions, says
to even figure out who should be
Ethan Bernstein, the Edward W.
making the decision in the first place.
Conard Associate Professor of
Self-management also can allow
Business Administration at Harvard
organizations, like Zappos, to be even
Business School, who specializes in
more responsive to customers and
organizational behavior and design.
the market, since the person who
Many attributes of self-managed
interacts with the customer can also
organizations have their advantages
make decisions.

Pulse Survey | Closing the Data Gap in Product Development Harvard Business Review Analytic Services 5
SELF-MANAGED ORGANIZATIONS
PUSH DECISION-MAKING
RESPONSIBILITY CLOSER TO
WHERE THE WORK IS DONE, WHICH
MANY FIND REDUCES RED TAPE.

6 Harvard Business Review Analytic Services Pulse Survey | Closing the Data Gap in Product Development
Companies are making advances in the methodologies they use
for product development and are planning radical changes to the
structure and decision-making processes in their enterprises.

On the other hand, self-management

is not a panacea. Sometimes,
Conclusion
The corporate innovation picture may
employees are reluctant to take on
not be as bleak as is often suggested.
the responsibilities involved in self-
According to the study, companies are
management, or they find the new
making advances in the methodologies
“self-managed” processes required
they use for product development
to coordinate with others even more
and are planning radical changes to
cumbersome and time-consuming than
the structure and decision-making
the top-down, red-tape processes they
processes in their enterprises. These
replaced. Bernstein and several of his
moves will certainly help companies
colleagues point out that organizations
bring products to market faster than
can be too responsive at the front
their competitors can. But there is still
line. Shifting an entire organization’s
a missing link—data and analytics.
strategy often requires some level of
To get all the product development
hierarchical push—what we think of as
pistons firing, business leaders need
leadership or vision. Companies have
to fill the data gap, especially if they
failed, not just succeeded, by being too
want to empower more people to make
responsive to their current customers.
product 10decisions
20 and 30
beat competitors
40 50 60 70 80 90 100
While self-managed organizations
to the punch instead of hobbling
attempt to align the activities of all
behind them.
their members by using technology
to make it all transparent and open
to everyone, it is not yet clear that FIGURE 6
openness and so-called flatness can
completely substitute for traditional
ACCELERATING PRODUCT DEVELOPMENT
Percentage of respondents indicating that their organizations are doing each of the following
strategy setting.
Bernstein therefore stresses that the Charging cross-functional teams to solve new product challenges
next generation of open, self-managing 49%
cultures will need a new breed of
leaders. “They will need the vision Providing information on market and customer and competitor challenges in a timely way
34%
to see where the hierarchy needs
to be replaced,” he says. “But these Social collaboration platforms and tools
leaders will also need to defend where 31%
hierarchy needs to remain to ensure the
organization’s goals are met.” Streamlined decision-making processes
31%
One thing seems certain, though.
Working in silos will become a thing of Offers non-financial incentives such as awards
the past. The use of cross-functional 20%
teams is the top means that companies
are using to accelerate product Offers financial incentives such as bonuses
development. It ranks significantly 18%
higher than financial and nonfinancial Other
rewards rank. FIGURE 6 3%

SOURCE: HARVARD BUSINESS REVIEW ANALYTIC SERVICES SURVEY, AUGUST 2018

Pulse Survey | Closing the Data Gap in Product Development Harvard Business Review Analytic Services 7
METHODOLOGY AND PARTICIPANT PROFILE
A total of 169 respondents drawn from the HBR audience of readers (magazine/
newsletter readers, customers, HBR.org users) completed the survey.

SIZE OF ORGANIZATION
28% 35% 11% 27%
10,000 OR MORE 1,000–9,999 500–999 499 OR FEWER
EMPLOYEES EMPLOYEES EMPLOYEES EMPLOYEES

SENIORITY
24% 40% 24% 13%
EXECUTIVE MANAGEMENT/ SENIOR MIDDLE OTHER GRADES
BOARD MEMBERS MANAGEMENT MANAGERS

KEY INDUSTRY SECTORS

17% 10% 9% 8%
MANUFACTURING FINANCIAL SERVICES HEALTH CARE OR LESS OTHER
SECTORS

JOB FUNCTION
19% 14% 13% 10% 7%
GENERAL/EXECUTIVE R&D/INNOVATION/ SALES/BUSINESS MARKETING/PR/ OR LESS OTHER
MANAGEMENT PRODUCT DEVELOPMENT DEVELOPMENT/ COMMUNICATIONS FUNCTIONS
CUSTOMER SERVICE

REGIONS
39% 24% 21% 7% 7% 1%
NORTH AMERICA EUROPE ASIA/PACIFIC MIDDLE EAST/ LATIN AMERICA REST OF WORLD
AFRICA

Figures may not add up to 100% due to rounding.

8 Harvard Business Review Analytic Services Pulse Survey | Closing the Data Gap in Product Development
 hbr.org/hbr-analytic-services

CONTACT US
hbranalyticservices@hbr.org Copyright © 2018 Harvard Business School Publishing.
MC210561018