The Signal and the Noise

Lessons for marketers, insight professionals, and

users of big data from Nate Silvers recent book

Ray Poynter
January 2014

Vision Critical 2014 www.visioncritical.com/university

The Signal and the Noise: Lessons for

Marketers and Insight Professionals
White Paper

Introduction
Nate Silvers recent bestseller, The Signal and the Noise, highlighted a number of important and
disruptive implications for anybody trying to understand markets, customers, and brands,
especially for anybody who is looking to make predictions based on data. This white paper is
designed to extract the key messages for marketers and insight professionals.

From Gallup to Silver

In 1936, with his accurate prediction of the US
Presidential election, George Gallup provided
the model of how to predict events from
consumer and citizen data, for the next 75
years. Gallup showed that by using
representative samples of the target group,
and by applying statistics based on the Normal
or Gaussian distribution, he could create
useful forecasts of voter and customer
behavior.
However, over the last few years the role
model of how to estimate what voters and
customers will do has shifted to Nate Silver.
Silver is the author of the bestselling book
The Signal and the Noise", founder of the
fivethirtyeight.com website, and unlikely pinup
boy for the use of Big Data, aggregators, and
innovative probability models to predict
outcomes.

Lessons from Nate Silver

Nate Silvers book was not specifically written
for marketers and insight professionals; it is
more a general business book, containing an
unstructured journey through a variety of
topics that Silver appears to find interesting.
However, it does contain several key
messages for marketers and market
researchers; and it is these key messages that
this paper seeks to highlight.

George Gallup and the 1936 US

Presidential Election
In 1936 the Literary Digest conducted a poll
to predict the result of the election contest
between Democrat Franklin Roosevelt and
Republican Alf Landon. They had done this
for the last few elections and had predicted
the correct result each time.
The Literary Digest mailed over 10 million
people, they had over 2 million replies.
Because of their past success and large
sample size they printed their prediction in
their magazine. They predicted Alf Landon
would win, but Roosevelt won all but two
states.
Why was the Literary Digest so wrong? Their
sample skewed towards people with a
telephone, people who had registered a car,
and subscribers to the magazine, in the
economically depressed 1936.
At the same election, George Gallup used a
carefully selected sample of a few thousand
people to correctly predict the election result.
In doing so, he set the pattern for the next 75
years for polling and market research, i.e.
using representative samples and statistics
based on the normal distribution.

Vision Critical 2014 www.visioncritical.com/university

The Signal and the Noise: Lessons for

Marketers and Insight Professionals
White Paper

The Key Messages

The key messages investigated in this whitepaper, from Nate Silvers book include:

1. Big Data has less potential to help businesses than most people seem to be claiming.
2. Many things cant be forecasted or predicted accurately.
3. Humans and machines working together tend to beat just humans and just machines.
4. We need to move our statistics on from Gauss towards Bayes.
Big Datas Feet of Clay
Nobody, including Nate Silver, is saying that Big Data has no uses. Indeed, Silvers election
forecasts are an application of multiple data sources, as are his predictions of baseball teams
successes and failures. In his book, Silver talks about the great strides that have been made in
weather forecasting because of one type of Big Data. But, Silver sounds a massive note of
caution commenting our predictions may be more prone to failure in the era of Big Data.
Silvers concerns about Big Data stem from two interrelated aspects: noise (defined as unhelpful
and possibly misleading information) and scale.
The first concern is that in the era of Big Data, noise is growing faster than the signal. If there is
an exponential increase in the amount of available information, there is likewise an exponential
increase in the number of hypotheses to investigate. With the noise growing faster than the
signal, messages will become harder to find, not easier.
The second concern is that sheer scale of Big Data will make people think that the old rules, the
rules of ordinary data, no longer apply. Silver criticizes Chris Anderson (editor of Wired
magazine and author of The Long Tail) who wrote in 2008 that Big Data would obviate the
need for theory and even the scientific method. Silver points out that when there are an almost
infinite number of possible connections, ones based on prior knowledge, theory, and
experiments where variables can be controlled, are the keys to success.
Silver also highlights the problem that with more data we are tempted to create more complex
models, but this is often beyond us. As a case in point, Silver looks at climate change. He
shows that simple models of global warming have been very accurate. However, attempts to
predict the impact of climate change on specific regions, countries, and particularly cities, have
been much less successful which in turn has provided ammunition to those who try to deny
climate change.

Vision Critical 2014 www.visioncritical.com/university

The Signal and the Noise: Lessons for

Marketers and Insight Professionals
White Paper

Not everything can be predicted

This should not be a shocking concept; Nassim Taleb covered the same territory in his book
The Black Swan. However, it seems to offend many Big Data fans to acknowledge that many
aspects of the world cannot be accurately and reliably forecast. Silver highlights four key
components in lack of predictability:

1. Chaotic systems
2. Missing data
3. Extrapolation
4. Feedback loops
Chaotic Systems
Chaotic systems are ones where a tiny change in the input data can result in a massive change
in the outcome event. The best known example of this is the weather, with the well-known
metaphor that the fluttering of the wings of a butterfly in the Amazon can cause a hurricane in
another part of the world. Silver shows how the quality (i.e. accuracy) of weather forecasts have
improved radically over the last twenty five years, but he also highlights two tests that are highly
relevant to anybody seeking to make forecasts about markets and consumers, which he terms
No Change and Climate.
One way of predicting the weather tomorrow, or next week, is to say it will be the same as
today. This is the No Change prediction. Any good prediction scheme should be able to beat
No Change. The Climate prediction for tomorrow, or next week, or next month, is the average of
the same day for the last few years. What Silver showed was that despite the improvements in
weather forecasting, its chaotic nature meant that forecasts only beat No Change and Climate
for about a week, further out than that No Change and Climate win.
In market research, a forecast of no change in a market, or of average performance for that type
of product (with a given distribution, ad spend, promotion budget etc) is the test that market
predicting needs to beat. Similarly, market research needs to assess the time frame over which
its predictions can beat its equivalents of the No Change and Climate tests.

Vision Critical 2014 www.visioncritical.com/university

The Signal and the Noise: Lessons for

Marketers and Insight Professionals
White Paper

Missing Data
One of the issues about the scale of Big Data is that it can blind people to what is not being
measured. For example, a project might collect a respondents location through every moment
of the day, their online connections, their purchases, and their exposure to advertising. Surely
that is enough to estimate their behavior? Not if their behavior is dependent on things such as
their childhood experiences, their genes, conversations overheard, behavior seen etc.
Silver highlights this issue in the context of earthquakes. Despite a massive effort by scientists,
data scientists, and researchers we have almost no ability to forecast an earthquake. This is
probably because an earthquake is a chaotic event and we dont have enough information
about what is happening under the ground and around the world, i.e. the failure is caused by
missing data and compounded by chaos theory.
The limited explanatory power of many market research models raises the question about how
much the lack of consistent accuracy arises from missing information. As the earthquake
illustration shows, missing information, especially in a complex system, can have massive
consequences.

Extrapolation
Extrapolation is when data is collected for one range or area and then the results are forecast
for some other range or area. For example, collecting the data for the UK and forecasting sales
for Europe, or collecting data on Monday and Tuesday and using that to forecast behavior over
a seven day week.
Silver highlights this problem by showing how his models are good at forecasting how major
league baseball players will perform next season, and how bad they are at forecasting how a
minor league player will perform in the majors next year. The major league performance is
within the box, but predicting what happens to a minor league player is extrapolation, i.e. out of
the box.
This within-the-box versus extrapolation problem also happens within marketing and market
research. Models do a pretty good job of predicting the performance of a line extension, but a
less good job when faced with truly new products, such as the Apple iPhone or the Apple
Newton. (The Apple Newton was a personal digital assistant with handwriting recognition
launched by Apple in 1993 to rave reviews, it bombed and was discontinued in 1998.)

Feedback Loops
In social systems a feedback loop is when the cause and the outcome become correlated with
each other, removing a clear cause and effect, and removing the ability of researchers to find
enduring laws to govern the market. Silver points out that most economic models developed

Vision Critical 2014 www.visioncritical.com/university

The Signal and the Noise: Lessons for

Marketers and Insight Professionals
White Paper

during one cycle of the economy fail when applied to the next cycle. The market factors in the
knowledge of the rules discovered, so the response to the levers of the economy changes.
In marketing and market research this means that when new laws are discovered they start to
cease to be true. As marketers start to employ a new method, competitors and customers start
to adapt to it. When the large retailers first introduced sales it was to sell surplus stock and to
boost expenditure during periods when spending was low. However, patterns adapted, people
started deferring their expenditure waiting for the sales, stores started stocking some items for
the sales, then the sales were moved forward into the lull created by deferred expenditure. Now
customers and retailers are in a cat and mouse game, as soon as retailers think they have
found a successful pattern, competitors and customers change their behavior and create new
patterns.
This issue was addressed by Einstein when he commented No problem can be solved from the
same level of consciousness that created it.

People plus machines tend to win

While Silver highlights many cases where models and computers do a really good job of
predicting the future, for example short-term weather forecasts, he also highlights that when
people are combined with models the result can be even better. Silver highlights this in the
context of talent scouts in baseball. Those scouts that use his forecasting tool and their own
knowledge and skills outperform his tool and outperform scouts who do not use his tool.
Silver also spends quite a bit of time looking at the chess tournament between Kasparov and
IBMs Deep Blue computer program.
The big story is that this was the first time a machine beat a world champion, but an interesting
footnote is that the software team worked on the program between matches to add what they
had learned to what the machine was able to determine. Kasparaov was beaten by a
combination of machine and people.
In marketing, this is the situation confronting Big Data, people who are combining market
research with their Big Data appear to be doing better than those who are just relying on Big
Data. At the 2013 MRS Conference in the UK, Lucien Bowater, BskyBs Director of Strategy and
Insight, talked about how he uses market research to show where to dig, and the Big Data
scientists to then do the digging.

Vision Critical 2014 www.visioncritical.com/university

The Signal and the Noise: Lessons for

Marketers and Insight Professionals
White Paper

From Gauss to Bayes

The most technical point in Silvers book is also one of the most important, and underpins why
we need to move from the era of George Gallup to a new age. In his book, The Black Swan,
Nassim Taleb highlights the problems caused by applying the normal distribution to situations
where it does not work in Talebs case improbable events. Silver is equally harsh on those
who stick to the normal distribution (using the Bayesian term of abuse, frequentist). The
example below illustrates the difference, in the real world of the marketer, between the
traditional and Bayesian approaches.
Using Bayes to assess a new agency:
Assume that you are a client and you need
to test new products before launching them.
You conduct a procurement process to
appoint a new research agency and on the
first project for you they test a new product
and assert that it has an 80% chance of
success and a 20% chance of failure.

What is Bayesian Thinking?

At one level Bayes is simply a statistical
approach, developed in the 18th Century by
Thomas Bayes. It assesses the probability of
one event, given that some other event has
happened.

The product fails!

Was the new agency unlucky or are they not
as good as you had hoped?
Frequentist assessment?
The agency said there was a 20% chance of
failure, so there is a 20% chance they were
unlucky and 80% chance they are not good
at predicting.
However, one weakness of this approach is
that it assumes that you, the knowledgeable
client, played no role in changing the
chances of the agency being good. Bayes
takes into account an assessment of the
quality of your decision what it calls the
prior probabilities.

However, there is a growing school of thought

that the Bayesian approach is both a philosophy
and a description of how humans tend to live
their lives (The Economist 2006).
Bayes, when applied to everyday lives is the
theory that underpins both trial and error and
pattern recognition. When we meet a new
situation we apply what we already know, and if
we are wise we then modify our initial thoughts
based on what happens next.
In frequentist statistics the default assumption is
that events are independent, in Bayes what we
have learned in the past should shape our
expectations of the future.

Vision Critical 2014 www.visioncritical.com/university

The Signal and the Noise: Lessons for

Marketers and Insight Professionals
White Paper

Bayes Approach?
With a Bayesian approach we need to assess the
prior probabilities. You did not select the agency at
random, so we want to factor that into the
assessment. Lets assume that you feel that the
chance that you appointed a good agency, given all
the checks and processes you went through, was
75%.
With that prior probability, and the assumption that
a bad agency is simply spinning a coin (50% of the
time they are right, 50% of the time they are wrong)
then Bayes would suggest that there is a 55%
chance the agency was simply unlucky (the
arithmetic is shown in the side box).
Frequentist statistics say the odds are 20% that the
agency was unlucky, so the agency should probably
be fired. However, Bayes* would suggest that the
odds they were unlucky was 55% - so they should
probably be given a second chance.
*Note, the Bayes estimate includes two subjective
numbers, the chance that you made a mistake (we
set this at 25%), and the chance that a poor agency
gets the result right (we set this at 50%). Change
one or both of these numbers and the result
changes.
In a world of frequentist statistics there is no place
for prior probabilities based on our knowledge of the
world but there is also no scope for blame if the
assessment is wrong. In Bayes we can take prior
knowledge into account, but we have to take
responsibility for creating the best prior probabilities
we can.
Bayes is an iterative approach. Before this test we
thought there was a 75% chance the agency was a
good agency, we now think there is a 55% chance
they are a good agency. This number will be used
when we next assess their project.

Working out the Bayesian Odds

Facts: your agency forecast success
with an 80% probability and it failed.
This outcome could have happened in
one of two ways:

A. You appointed a good agency and

they were unlucky.

B. You appointed a bad agency and

they got it wrong.
If we assume that this pattern was
repeated 1000 times then we dont need
to focus on fractions and decimal
places.
In 1000 trials, how many times should A)
occur? We have assigned a prior
probability of 75% that you appointed a
good agency, and they had said that
there was 20% chance the product
would fail. So in 1000 trials this should
happen 75% * 20% * 1000 = 150 times.
What about B)? We have said that there
is a 50% chance that a bad agency get
their forecasts right, so the chance of
this happening are 25% * 50% * 1000 =
125.
So, there are 275 ways that a forecast of
success could be followed by product
failure (150 + 125). The chance the
agency was unlucky is 150 / 275 = 55%.
The chance they were a bad agency is
45%.

Vision Critical 2014 www.visioncritical.com/university

The Signal and the Noise: Lessons for

Marketers and Insight Professionals
White Paper

Closing thoughts
It is not certain that Nate Silvers prescriptions and recommendations for the process of
forecasting human behavior and intentions will prove to be as useful or as long lasting as
George Gallups were in 1936.
However, it is clear that the age of representative samples, long timelines, and statistics based
on the normal distribution is passing. New approaches such as insight communities, social
media research, Big Data, mobile market research and Bayesian analytics are the tools of the
day, demanding new mindsets.
Marketers, market researchers, and insight professionals need to find ways of using the new
tools and approaches to ensure that they improve their ability to forecast future outcomes, to
understand what can and cannot be forecast and how to make this information useful to
decision makers.
The arrival of Big Data is not a reason for businesses to sit back and assume that this will solve
all their needs. In many ways Big Data creates as many problems as it solves and the solving of
these new problems is likely to require the involvement of the human mind, the development of
hypotheses, and the application of market research.
One major challenge for marketers, market researchers, and insight professionals is to
become knowledgeable users of Big Data and Bayesian thinkers. Companies who do not
gain an overview of what Big Data is, and what it can and cant do, are likely to be seen
as easy targets by aggressive sales teams looking to create momentum and to sell, what
can be in some cases, the modern day equivalent of snake oil.

Vision Critical 2014 www.visioncritical.com/university

The Signal and the Noise: Lessons for

Marketers and Insight Professionals
White Paper

References
The Economist. (2006). Bayes Rules. 5 June, 2006. Available from:
http://www.economist.com/node/5354696. [October 2013].
Taleb, Nassim. (2007). The Black Swan. Random House, New York.
Silver, Nate. (2012). The Signal and the Noise. Penguin, New York.

About the Author

Ray Poynter
Director, Vision Critical University
Ray Poynter is the author of The Handbook of Online and Social Media
Research, joint-editor of ESOMARs Answers to Contemporary Market
Research Questions, co-author of University of Georgias Principles of
Marketing Research courses new module on Mobile Marketing
Research, founder of NewMR.org, and a Director of Vision Critical
University.

About Vision Critical

Vision Critical is the world's leading provider of insight
communities, currently supporting over 650 brands worldwide.
We build software and provide comprehensive services that
empower organizations to engage groups of customers and
stakeholders on a continuous basis for the express purpose of
extracting and managing insight to drive better, faster
decisions.
Learn more about the fastest-growing solution in market
intelligence, meet our global partners and view our client
stories at www.visioncritical.com. Follow us on Twitter at
https://twitter.com/visioncritical.

10

Contact Us
info@visioncritical.com
North America: 1 877 669 8895
UK: +44 (0) 20 7633 2900
Australia: +61 (2) 9256 2000
Hong Kong: +852 3489 7009

Vision Critical 2014 www.visioncritical.com/university