You are on page 1of 16

9 -8 2 2 -0 3 5

OCTOBER 5, 2021

CHRISTINA WALLACE

Sparking Growth at Consumer Reports


It was nearing the end of the day and Ted Bongiovanni and Heather Myers were wrapping up a
marathon Zoom call in late April 2020. As the Director of New Ventures and Products at Consumer
Reports (CR), Bongiovanni was one of the leaders who had been tasked with discovering new business
opportunities for the 84-year-old nonprofit that historically had focused on providing independent
testing and research on consumer goods. Unlike many nonprofits that are substantially funded by
charitable giving, Consumer Reports earned the vast majority of its revenue from a paid membership
model (primarily recurring digital and print subscriptions for its reviews). But with a customer base
that averaged 55 years old, the organization was keenly aware of the need to diversify its audience and
business model. Bongiovanni’s mandate was to bring in new revenue from new customers without
straying from the nonprofit’s mission.

With a bare bones team (just 1.5 FTEs between him and his deputy, Rosaline Bernstein and a few
borrowed staff from the organization’s other divisions) Bongiovanni relied heavily on outside partners,
including Myers and her digital marketing and market research firm Spark No. 9. Their partnership to
identify potential digital products and services for younger audiences had kicked off just as the
COVID-19 pandemic was spreading worldwide. After narrowing a field of potential ideas from nine
to three, the team used robust digital marketing tests to vet consumer interest before investing in
product development. The three finalists were apps all focused on the home: Aboderly was a way for
homeowners to get expert answers to their questions; Upkept was pitched as an all-in-one home
management tool; and Greenedwell offered homeowners the opportunity to become “sustainability
ninjas,” with a quantifiable sustainability score and tips to make changes to improve it.

Now two months later, they were analyzing the data from the project and preparing their
recommendations for the New Ventures Steering Committee in the morning. After spending just over
$3,000 on digital tests, the results from the first round of testing were mixed. Upkept had met their goal
of under $1.00 cost-per-click (CPC) and netted 140 customer email signups on the landing page.
Aboderly garnered exactly zero email signups and missed the $1.00 CPC mark in seven out of eight of
their ad-audience pairs. Greenedwell ads landed above the $1.00 CPC for every ad-audience pair,
though they did generate four email signups.

Bongiovanni and Bernstein were diligently preparing their deck for the Steering Committee on
which of these opportunities they thought were worth investing in further. But how confident would
CR be from this initial round of testing? Did they learn enough to recommend pursuing Upkept? And
what should they do with Greenedwell? What could they confidently conclude from their digital

Senior Lecturer Christina Wallace prepared this case. It was reviewed and approved before publication by a company designate. Funding for the
development of this case was provided by Harvard Business School and not by the company. Certain details have been disguised. HBS cases are
developed solely as the basis for class discussion. Cases are not intended to serve as endorsements, sources of primary data, or illustrations of
effective or ineffective management.

Copyright © 2021 President and Fellows of Harvard College. To order copies or request permission to reproduce materials, call 1-800-545-7685,
write Harvard Business School Publishing, Boston, MA 02163, or go to www.hbsp.harvard.edu. This publication may not be digitized, photocopied,
or otherwise reproduced, posted, or transmitted, without the permission of Harvard Business School.

This document is authorized for use only by Hangyu Fan (hangyu.f@columbia.edu). Copying or posting is an infringement of copyright. Please contact customerservice@harvardbusiness.org
or 800-988-0886 for additional copies.
822-035 Sparking Growth at Consumer Reports

experiments, and what other factors might they need to consider beyond the data? Finally, should
Bongiovanni lobby for more dedicated resources to execute on his team’s mandate for growth, or
should he keep things lean and continue to rely on a broad bench of external partners? The stakes were
high and Bongiovanni had a lot to consider.

History of Consumer Reports


Consumer Reports was founded in 1936 as Consumers Union, a nonprofit dedicated to helping
consumers assess the safety and performance of products. A patchwork of poorly enforced regulations
offered few safeguards to Americans at a time of rapid technological advancement and little
governance over the veracity of advertising claims. 1 So, a group of scientists decided to build an
organization that would offer independent testing and research on consumer goods, helping
consumers compare products and make more informed choices. As the nonprofit grew, its influence
grew as well, and CR soon leveraged its position as a trusted advocate to sway regulators and
manufacturers to prioritize consumers’ best interests.

Unlike many nonprofits that rely on philanthropic contributions to fund their charitable and
advocacy activities, CR built a social enterprise model that relied almost exclusively on revenue from
paid memberships. In the fiscal year ending in May 2020, nearly 90% of CR’s $245M revenue came from
approximately six million members who paid $30 per year for access to digital and print reviews,
ratings, and tools that cover a wide range of products including cars, major appliances, consumer
electronics, personal health and fitness gear, baby and child-safety gear, home and garden supplies,
and personal finance and insurance products (see Exhibit 1 for 2019 and 2020 Income Statements).
These independent reviews were conducted by more than 140 scientists, researchers, and technicians
across 63 labs and CR’s 327-acre auto test center. Since its founding, the goal had been to put
manufacturers’ claims to the test and ensure consumer products were safe, reliable, and deliver on the
advertised promises. CR had a strict editorial independence policy, which it enforced by accepting no
outside advertising and paying for all products it tests.

In addition to its product testing, Consumer Reports engaged in consumer advocacy. Over its more
than eight decades in operation, the nonprofit won significant protections for consumers, including
lobbying for legislation that made seat belts standard in all cars in the 60s, being one of the first to raise
awareness of the dangers of cigarettes, increasing the accessibility of safety and side effect data for birth
control, and supporting the creation of the Consumer Financial Protection Bureau. 2

From Print to Digital to New Ventures


In the late 1990s print media began its migration to the internet. While early experiments in digital
content were largely free, like the New York Times initially offering their articles online without a
paywall in 1996 3, CR chose to replicate its paid print subscription model with a digital subscription.
“We used to say, ‘We have more paid subscribers than The New York Times, The Wall Street Journal,
and the Financial Times combined,’ because we were the only ones charging at the beginning,” laughed
Bongiovanni. “Of course, twenty-some years later some of those newspapers with good digital models
have more than caught up.”

CR’s CEO Marta L. Tellado, who took the helm of the organization in 2014, recognized these
challenges and quickly sought to accelerate CR’s evolution from being an information provider to a
powerful marketplace shaper. The idea: launch products and services that provide solutions for
consumers, disrupt marketplace problems, and drive revenue to create financial sustainability. This

This document is authorized for use only by Hangyu Fan (hangyu.f@columbia.edu). Copying or posting is an infringement of copyright. Please contact customerservice@harvardbusiness.org
or 800-988-0886 for additional copies.
Sparking Growth at Consumer Reports 822-035

clear mandate for innovation from the top, in conjunction with an organizational culture of
inventiveness, provided strong tailwinds.

But even with the head start on a digital revenue stream, and its transformation to a membership
model that included benefits and tools beyond its flagship ratings and reviews, CR struggled to attract
younger consumers. “If I had a dollar for every single person who said, ‘Oh yeah, Consumer Reports,
my grandfather wouldn’t buy a thing without consulting it,’ I’d be a rich woman,” quipped Leonora
Wiener, SVP and Chief Operating Officer.

When it comes to setting the strategy for the future of Consumer Reports, the metaphor
I use a lot is low beams and high beams. You have to toggle between the two. On low
beams, looking at the near future, the organization is strong. Our revenue is strong. But
turn on the high beams and we have an aging customer base and we have to grow our
digital business faster. The hardest part is, how much investment do you put to growing
the legacy business versus resourcing the new?

Eric Wayne, SVP and Chief Financial Officer of Consumer Reports, agreed. “There’s no question
that there’s urgency to find new audiences. And we think we’re going to need to serve those audiences
with new products and services. A couple of years ago, we decided to establish a separate team that
would be fully dedicated to launching new products and services for new audiences.”

New talent would work with CR’s digital marketing, product and ratings teams as part of the
transformation. That’s where Ted Bongiovanni came in. He joined Consumer Reports in October 2015
as the newly created Director of Membership Platform. Brought in by Jason Fox, the organization’s
Chief Digital Officer, Bongiovanni was tasked with leading CR’s efforts to transform its transactional
subscription business into a membership program and updating the core value proposition. His eclectic
resume was unsurprising for someone who was interested in innovation and new ventures.
Bongiovanni held a bachelor’s degree in political science from Susquehanna University, a Master of
Public Administration degree in management and finance from Columbia University’s School of
International and Public Affairs, and a Master of Arts degree in digital media design for learning from
New York University. He also completed a certificate in disruptive strategy from Harvard Business
School Online. His three decades of work experience ranged from teaching English in Lithuania as a
Peace Corps volunteer, to developing a community publishing platform at the New York Times in the
late 1990s, working in education administration with major universities and edtech nonprofits, and
creating new community outreach programs at NYU’s Abu Dhabi campus.

In Summer 2019 Bongiovanni moved to the New Ventures team, where his time was split 50/50
with the Digital Product team. Because of the membership transformation, Bongiovanni was able to
lean on colleagues with deep expertise. “New Ventures was half of me and 10% of 10 people, which
we all know, 10% of 10 people does not equal one person, even when they’re amazing,” he pointed out.
The talent and dedication of those CR teammates made solid progress but leadership wanted to move
more quickly. In early 2020, he was able to hire Rosaline Bernstein as a full-time Associate Director of
New Ventures, adding expertise in product management, marketing, and design to the group and
expanding their headcount to 1.5 FTEs. Still, like many startups, they needed to keep the team scrappy
and collaborate with outside partners and vendors on an ad hoc basis rather than bring on additional
employees.

Wayne saw this team structure as an advantage, rather than a limitation. “We couldn’t move as fast
if we tried to do everything internally. Too often, organizations fail by moving too slowly. We wanted
the New Ventures team to be nimble, and we want them to work with the right external partners to
help us bring more products to the market quicker.”

This document is authorized for use only by Hangyu Fan (hangyu.f@columbia.edu). Copying or posting is an infringement of copyright. Please contact customerservice@harvardbusiness.org
or 800-988-0886 for additional copies.
822-035 Sparking Growth at Consumer Reports

Smoke Tests
As Bongiovanni and Bernstein plotted opportunities for the New Ventures group to pursue, they
quickly realized they would need a digital marketing partner to help them test and validate potential
concepts before they invested too much time or energy. “We had experience with a previous project
called ‘Checkd‘ in running smoke tests online: setting up a landing page with a sign-up form, running
a search engine marketing campaign to test viability,” explained Bongiovanni. “But that’s like doing a
couch-to-5K training program. It works, you can finish the race and be proud of your 10-minute-mile
pace, but it’s not the same as training with a serious coach. We knew we needed a partner to get the
rigor and speed and volume we were looking for in our next round of new venture testing.”

“Smoke tests” are a concept that originated in the computer hardware industry, where a new device
passed its initial test if it didn’t catch fire and smoke the first time it was turned on. In the world of
software development, the term referred to early rounds of testing on new versions of programs that
indicated they were stable enough for further, in-depth testing. Marketers and entrepreneurs extended
the idea to vet the viability of a new business at the very earliest stages, even before a “minimum viable
product” (or MVP) was developed.

The most rudimentary version of a smoke page involves building a simple one-page website (or
“landing page”) with little more than a description of the value proposition, a mocked-up image or
two, and an email sign-up form for interested customers to receive more information when the product
launches. The rate that users convert from website visitors to email subscribers is one simple metric to
quantify early demand for the product, which in turn can validate (or invalidate) your hypothesis and
determine whether you have a feasible business case (or not). Additional testing can be done to refine
the value proposition, target audience, brand identity, price point and more, all before a single line of
code or physical prototype is developed.

Bongiovanni and Bernstein knew they needed an external partner to help them winnow down nine
potential businesses to three, and then validate (or invalidate) those three concepts before they could
pitch the leadership of Consumer Reports for meaningful investment. So, Bongiovanni reached out to
a colleague in CR brand licensing for leads and was referred to Heather Myers, founder and CEO of
Spark No. 9, a boutique market-testing agency.

History of Spark No. 9


Heather Myers earned a bachelor’s degree in English from Dartmouth and an MBA from Harvard
Business School before establishing a career in strategy and new business development. After a short
stint in management consulting, she spent a decade in media, first with Universal Music Group, then
with Scholastic Corporation. By 2009 she was ready to set out on her own and founded Spark No. 9 to
help design and launch new businesses with rigorous, in-market digital testing. She explained the
problem Spark No. 9 wanted to solve:

When you are launching something new, there’s no data. You have lots of hypotheses
and it’s fairly expensive to prove them when the default is to just go build the product
and launch it. What we do is we apply scientific principles to pre-launch digital testing,
which means we can learn a lot in a relatively short period of time to help guide our
clients’ strategy and decision-making. By creating this series of hypotheses that we test
with ad campaigns before they build anything substantial, we’re able to validate and de-
risk their product development process.

This document is authorized for use only by Hangyu Fan (hangyu.f@columbia.edu). Copying or posting is an infringement of copyright. Please contact customerservice@harvardbusiness.org
or 800-988-0886 for additional copies.
Sparking Growth at Consumer Reports 822-035

The New York City-based firm was comprised of a dozen employees across three teams: creative
(designers and copywriters), analytics (data scientists), and strategy (account managers and business
strategists). (See Exhibit 2 for more background on key team members.) Together they developed the
Spark No. 9 methodology, a standard template that could be tailored to each client. The process roughly
mirrored the customer acquisition funnel.

The first wave of testing focused at the top of the funnel, where they would validate and refine the
product concept and test how that product comes to life through creative. The goal of the first wave is
to quickly identify product-market fit, as measured through metrics like landing-page views and the
cost-per-click across several discrete audiences. If the first wave generates traffic that is robust, yet
inexpensive, the team uses the highest engagement rates to tell them where they may be able to develop
product-market fit.

In the second wave of testing, the focus is on refining how to present the concept to the audience
that has shown interest. If the first wave identified which audience responded best to the overall value
proposition, now the goal would be to tighten the bond between concept and audience. This often
means testing different messaging or imagery to drive engagement and increase conversions from
visiting the site to sharing an email address.

By the third wave of testing, the team is confident in their ability to drive people to the landing
page, know that they’re genuinely interested, and optimize acquisition metrics like cost-per-email-
sign-up. At this stage, the focus is on testing pricing. By sending traffic to several different landing
pages with different priced offerings, and tracking conversion rates for email sign-ups, they can
establish a rough draft of the price elasticity curve for the (still yet to be developed) product.

Staying Close to Home


With a mandate to focus on attracting new customers to the Consumer Reports brand, CR
leadership tasked the New Ventures team with identifying potential businesses to help consumers be
more financially secure. But after some initial research with customers, the team recognized the
“money” space was already crowded and there wasn’t a clear white space for CR to step into. Next,
they dug into the most expensive pain points for consumers and landed on two specific areas:
homeownership and car purchasing and maintenance. The CR New Ventures team had already started
exploring the homeownership ecosystem with Checkd, a product to help prospective home buyers find
an ASHI-certified a inspector.

“Most home buyers go with their realtor’s recommendation for a home inspector. The catch is that
one in three people regret that hire because the home inspector misses a critical issue,” reported
Bongiovanni. CR did its due diligence on the home inspection space and identified the American
Society of Home Inspectors, which pioneered and professionalized home inspections as a partner and
source of trustworthy home inspectors. Initial smoke tests on Checkd were encouraging and the team
was excited by the potential. The team also gleaned from user interviews that home maintenance was
a deep concern for first time home buyers. This data kept the CR team bullish on the home space for
further discovery and testing, particularly now that they had a partner in Spark No. 9.

Bongiovanni and Myers scoped out a five-phase workplan for new business concepts focused on
the home. (See Exhibit 3 for the detailed workplan.) The opportunity: create a suite of homeowner tools

a The highest level of certification offered by the American Society of Home Inspectors, the primary professional association in
the industry.

This document is authorized for use only by Hangyu Fan (hangyu.f@columbia.edu). Copying or posting is an infringement of copyright. Please contact customerservice@harvardbusiness.org
or 800-988-0886 for additional copies.
822-035 Sparking Growth at Consumer Reports

and services that would attract new customers to the Consumer Reports brand. The challenge: identify
and validate the product concepts that were most likely to both attract and convert users with a
sustainable and profitable product. The testing would follow Spark No. 9’s three-wave methodology:

Wave 1: Brand Position Appeal


• Which concepts generate the most interest?
• Which brand positions for each concept resonate the most? With which audiences?
• Which audiences are worth additional exploration?
Metric: Cost per Click
Platform: Instagram
Ad Type: Story

Wave 2: Audience Effectiveness


• How can we optimize brand position, creative, and message via ad formats for each ad-
audience pair to capture email addresses on the landing page?
Metric: Cost per Email Sign-up
Platform: Instagram
Ad Type: Story, Static Feed

Wave 3: Optimization
• How can we validate plan elements—additional creative, audience segments, other factors—
to support a scalable customer acquisition campaign?
Metric: Cost per Email Sign-up
Platform: Instagram
Ad Type: Story, Static Feed

The project was scheduled to kick off in mid-March 2020. Then the COVID-19 pandemic shut down
New York City and, quickly thereafter, much of the rest of the United States. Suddenly “home” had
become the central player in most Americans’ lives, serving as the setting for work, school, exercise,
socializing, self-care, hobbies, family time, and three meals a day (plus snacks) for the foreseeable
future. Pivoting the kickoff workshop to Zoom, Myers, Bongiovanni, and Bernstein were joined by
designers and data scientists from Spark No. 9 and many of the “10%” New Ventures team members
from Consumer Reports.

The first step was to winnow nine potential ideas down to three for further testing. This process
began with a workshop where the joint team discussed unmet needs for the nine concepts CR brought
to the table, brainstormed consumer needs related to major home milestones (first rental, first purchase,
etc.), and identified values, behaviors, and trends related to the potential audiences. Then Spark
brought to life (visually) the concepts up for consideration (see Exhibit 4) and a framework for
winnowing the ideas.

They weighed internal factors, such as relevance with the CR brand, the opportunity to leverage CR
data and other assets, and the ability of the CR team to execute. They also considered external factors
that would affect the business value and market impact of the idea, such as the number of potential

This document is authorized for use only by Hangyu Fan (hangyu.f@columbia.edu). Copying or posting is an infringement of copyright. Please contact customerservice@harvardbusiness.org
or 800-988-0886 for additional copies.
Sparking Growth at Consumer Reports 822-035

customers each idea would reach and the fit with millennial trends and behaviors. After considering
the concepts along these criteria (see Exhibit 5), the assembled group used “dot voting” b to note their
preferences.

The three winning finalists c were:

1. Aboderly

Home ownership? You got this. Home with confidence. Ask questions, get expert answers. Start
your home ownership journey on the right foot.

2. Upkept

Manage your home like a pro. Your all-in-one home management tool.

3. Greenedwell

Smart decisions for home, wallet, and planet. Ready to become a sustainability ninja? Find your
sustainability score, get tips on how to improve it, and save $$ along the way.

Source: Consumer Reports company documents.

Defining an Audience: Who and Where


The first major hurdle to designing an effective smoke test is defining the intended audience(s). “A
key principle of audience design is ‘difference,’” said Spark No. 9 head of analytics Mike Alwill. “If
you are testing audiences that are 50% or 60% similar, it’s hard to tell from the raw data if the audience
was a meaningful driver for the response rates.” After more than a decade at Google working in
marketing, product, and data analytics, he left the tech giant to join the smaller, more nimble market

b “Dot voting” is a common tool in design thinking and user experience (UX) workshops to winnow down ideas to a select few
finalists. Participants are given a set number of votes and they individually place dots (often stickers or pen marks) to note their
votes. Once everyone is finished, the ideas with the greatest number of dots are selected.
c The selected ideas were renamed to avoid branding and trademark issues.

This document is authorized for use only by Hangyu Fan (hangyu.f@columbia.edu). Copying or posting is an infringement of copyright. Please contact customerservice@harvardbusiness.org
or 800-988-0886 for additional copies.
822-035 Sparking Growth at Consumer Reports

testing agency. “To actually learn something from your tests, you need your drivers to be isolated. So,
your audiences must be defined in a way that makes them discrete.”
Some marketers think of “audiences” in the same way as “buyer personas.” They are ways to break
down potential customers into smaller segments based on similar interests, behaviors, or motivations.
But Alwill bristles at using “personas” when it comes to articulating the edges of his target audiences
for digital tests.
When I start thinking with personas, I start to index on anecdote quite a bit, which I find
pigeonholes my thinking. It’s very easy to say, ‘I’ve got a friend who’s a doctor who just
bought a home and he might like this, but he also is such a homebody and…’ Right there,
I’ve already made all these assumptions because I have a persona in my head as opposed to
trying to break down the actual underlying characteristics of who that audience might be.
With home ownership, you might start by breaking down new homeowners versus
experienced ones. That’s a tenure thing. You might also look at the age of the homeowner,
young versus old or by generational markers. That’s a demographic choice. They could
actually be much more complicated, but the point is you begin to identify audiences in
different pieces of a market across two or more axes, and then you have to ask, what other
dimensions are interesting we haven’t considered yet?
The second hurdle for designing smoke tests is deciding which digital platform(s) to leverage. While
all digital ads could direct audiences to the same landing page, which consumers you have access to
and your ability to segment them varies across platforms. As of 2021, “Facebook has the most robust
audience insights tools given the vast amount of data it collects on its users,” reported Alwill. (Their
heavy investment in such tools is unsurprising given they made 97.9% of their revenue from
advertising in 2020. 4) Audiences can be defined by age, gender, education, income, geography,
relationship status, religion, political views, interests, behaviors, purchasing history, and more. These
audiences can be created from scratch or can be derived from existing customer lists, and can be
leveraged for ads on Facebook, Instagram, WhatsApp, and Messenger.
Facebook also pioneered the concept of a “lookalike” audience, using its vast data on its social
network members to create a group of users that share similar interests and behaviors with a given set
of customers. This helps marketers find highly-qualified customers who would otherwise be difficult
to identify and reach, lowering the cost of customer acquisition in the process. The algorithm Facebook
uses to create lookalike audiences is not public, and they do not share details on which characteristics
of a customer set they are targeting when creating a lookalike audience. Startups and companies with
smaller customer bases may find it challenging to create a lookalike audience due to the limited size of
their customer data set. And regulators have raised privacy and discrimination concerns over the types
of data used to draw connections.
Beyond Facebook, platforms like Twitter, LinkedIn, and TikTok also offer digital advertising tools.
“Twitter’s audience tool emphasizes basic user demographics, interests, hashtags, and the accounts
they follow and interact with, while LinkedIn primarily focuses on professional data like industry,
seniority, geography, tenure, and companies and accounts you follow,” noted Alwill.
In addition to considering how well a platform can segment and surface your potential audience,
digital marketers must also consider whether the platform is a natural fit for the customer acquisition
funnel and buyer journey. “Just because you can get in front of your perfect customer doesn’t mean
they are in the right mindset to engage with your ad or take the action you hope they would take,”
cautioned Alwill. “Ultimately, you have to choose the right platform for the right problem.”

This document is authorized for use only by Hangyu Fan (hangyu.f@columbia.edu). Copying or posting is an infringement of copyright. Please contact customerservice@harvardbusiness.org
or 800-988-0886 for additional copies.
Sparking Growth at Consumer Reports 822-035

Wave 1
For all three concepts, the Spark team chose to run their tests on Instagram. In the first wave of
testing, 700,000 Instagram users encountered ads for each concept in their Story feed. Each concept was
given two brand positions and each brand position was given two creative treatments to ensure a given
treatment was not the sole driver of ad performance. Aboderly and Greenedwell creative treatments
were tested against four audiences each, while Upkept was tested against three audiences. The result
was 44 ad-audience pairs for Wave 1. (See Exhibit 6 for a visual representation of the ad-audience pairs.
See Exhibits 7 and 8 for examples of Wave 1 audience definitions and creative treatments.) An ad-
audience pair was deemed a success if it resulted in a cost-per-click (CPC) d of under $1.00.
Aboderly’s two brand positions were “Instant Expert” and “Confidence” with four audiences: folks
who were in the process of buying homes, buyers who also were seeking guidance via online platforms,
folks who were already homeowners, and homeowners who were also seeking guidance via online
platforms. “Confidence” was the winning brand position for Aboderly’s first wave of testing, but not
by much, and only one ad-audience pair out of eight came in at a CPC under $1.00. Higher performing
ads for Aboderly were visually simple and featured faces, but no landing page visitors signed up to
learn more. The team surmised that the winning audience—existing homeowners—were put off by the
copy emphasizing new homeownership and didn’t connect with the value proposition, which was
targeted more toward prospective buyers.
Upkept’s two brand positions were “All in One” and “Control” with three audiences: existing
homeowners, homeowners who liked to be hands-on with maintenance, and homeowners who liked
tracking data. “Control” was the winning brand position for Upkept, with both creative treatments
performing at under $1.00 CPC across all three audiences. The “manage your home like a pro” creative
averaged around $0.80 across the three audiences, while the “spending more time at home?” creative
came in between $0.40-0.60 across all three audiences. More than 140 people submitted their email
address on the Upkept landing page to learn more about the product when it was available.
Greenedwell’s two brand positions were “Responsible” and “Fun Progress” with four audiences:
existing homeowners, homeowners who value efficiency, homeowners who care about the climate,
and homeowners who care about recycling. While higher-performing ads emphasized gaming or
visualized home improvement in action, neither brand position was a clear success for Greenedwell
with every ad-audience pair coming in at more than $1.00 CPC. Moreover, only four visitors to the
landing page submitted their email address to learn more.
Upkept was a clear winner in Wave 1. But what should CR do with Aboderly and Greenedwell?
The CPCs were above the threshold with no winning treatment or audience. And Greenedwell netted
just four email sign-ups while Aboderly garnered zero. The team wasn’t ready to give up on either just
yet. But the bar had been raised: now they were seeking CPCs under $0.50 to match the best-performing
positioning for Upkept. Could they make the case to the Steering Committee to continue funding these
concepts that missed the mark? Or should they go all-in on Upkept this early in the process?

d “Cost-per-click” is defined as the total spend on the ad divided by the total number of landing page conversions via the ad.
This is different from cost-per-email-signup, another common metric in smoke tests.

This document is authorized for use only by Hangyu Fan (hangyu.f@columbia.edu). Copying or posting is an infringement of copyright. Please contact customerservice@harvardbusiness.org
or 800-988-0886 for additional copies.
822-035 Sparking Growth at Consumer Reports

Exhibit 1 2020 and 2019 P&L

Source: Consumer Reports audited financial statements for the year ending May 31, 2020.
https://www.consumerreports.org/cro/about-us/policies-and-financials/financial-reports/index.htm, accessed
October 2021.

10

This document is authorized for use only by Hangyu Fan (hangyu.f@columbia.edu). Copying or posting is an infringement of copyright. Please contact customerservice@harvardbusiness.org
or 800-988-0886 for additional copies.
Sparking Growth at Consumer Reports 822-035

Exhibit 2 Spark No 9 key team bios

Michael Alwill is the Director of Data at Spark No. 9, where he guides data & analytics, audience
strategy, and test management. Previously he worked within sales operations and revenue operations
at Google, and prior to that studied pure mathematics at the University of Rochester.

Amanda Jaquin is the Director of Creative Strategy & Ops at Spark No. 9 where she leads a team of
creatives in developing data-driven strategies for stopping thumbs, engaging eyeballs, and generating
clicks. She studied graphic design and communications at Rochester Institute of Technology and is
fueled by curiosity, collaboration, and aha moments.

Heather Myers is the founder and president of Spark No. 9, a firm that develops, tests, and validates
growth strategy. Previously, Heather led strategy and M&A for Scholastic, Universal Music Group,
and Universal Studios. She has a BA in English from Dartmouth and an MBA from Harvard Business
School.
Source: Spark No. 9 company documents.

Exhibit 3 Project workplan

Source: Consumer Reports company documents.

11

This document is authorized for use only by Hangyu Fan (hangyu.f@columbia.edu). Copying or posting is an infringement of copyright. Please contact customerservice@harvardbusiness.org
or 800-988-0886 for additional copies.
822-035 Sparking Growth at Consumer Reports

Exhibit 4 The initial concepts considered and dot voting results from the March 2020 workshop

Source: Consumer Reports company documents.

Exhibit 5 Criteria for Steering Committee winning concepts

Source: Consumer Reports company documents.

12

This document is authorized for use only by Hangyu Fan (hangyu.f@columbia.edu). Copying or posting is an infringement of copyright. Please contact customerservice@harvardbusiness.org
or 800-988-0886 for additional copies.
Sparking Growth at Consumer Reports 822-035

Exhibit 6 Testing structure for selected concepts

Source: Consumer Reports company documents.

Exhibit 7A Audience strategy

Source: Consumer Reports company documents.

13

This document is authorized for use only by Hangyu Fan (hangyu.f@columbia.edu). Copying or posting is an infringement of copyright. Please contact customerservice@harvardbusiness.org
or 800-988-0886 for additional copies.
822-035 Sparking Growth at Consumer Reports

Exhibit 7B Example niche audience definitions for Aboderly

Source: Consumer Reports company documents.

Exhibit 8 Example creative treatment and audiences

Aboderly

14

This document is authorized for use only by Hangyu Fan (hangyu.f@columbia.edu). Copying or posting is an infringement of copyright. Please contact customerservice@harvardbusiness.org
or 800-988-0886 for additional copies.
Sparking Growth at Consumer Reports 822-035

Upkept

Greenedwell

Source: Consumer Reports company documents.

15

This document is authorized for use only by Hangyu Fan (hangyu.f@columbia.edu). Copying or posting is an infringement of copyright. Please contact customerservice@harvardbusiness.org
or 800-988-0886 for additional copies.
822-035 Sparking Growth at Consumer Reports

Endnotes

1 Consumer Reports 2021-2023 strategic plan.

2 Silber, Norman Isaac. Test and protest – the influence of Consumers Union. New York: Holmes & Meier. 1983

3 Lichterman, Joseph. "20 years ago today, NYTimes. com debuted" on-line" on the web." Nieman Journalism Lab (January 26,
2016). https://www.niemanlab.org/2016/01/20-years-ago-today-nytimes-com-debuted-on-line-on-the-web/, accessed
November 18, 2021.
4 Facebook Annual Report 2020, page 66 as cited in “Facebook's advertising revenue worldwide from 2009 to 2020” Statista
https://www.statista.com/statistics/271258/facebooks-advertising-revenue-worldwide/, accessed November 2021.

16

This document is authorized for use only by Hangyu Fan (hangyu.f@columbia.edu). Copying or posting is an infringement of copyright. Please contact customerservice@harvardbusiness.org
or 800-988-0886 for additional copies.

You might also like