Zeynep - Tufekci - That Catcalling Video and Why "Research Methods" - Nov - 3 - 2014

That Catcalling Video and Why “Research Methods” is such an Exciting Topic (Really!
) — The Message — Medium 11/4/14 7:38 PM
Hollaback and Why Everyone Needs Better Research

Methods
And Why All Data Needs Theory
A few days ago, a video produced by a third party “viral video creative
agency” for the anti-harrasment NGO, “Hollaback,” went viral. The two-
minute clip showed a conventionally attractive, white, actress walking the
streets of New York for “10 hours”, and being repeatedly catcalled. But you
couldn't help noticing that all men catcalling to the white actress were people
of color — actually almost all black men. At the end of the video a text said that
“100+” instances of “verbal street harassment” had occurred during these 10
hours, from “people of all backgrounds”
https://medium.com/message/that-catcalling-video-and-why-research-methods-is-such-an-exciting-topic-really-32223ac9c9e8 Page 1 of 10
That Catcalling Video and Why “Research Methods” is such an Exciting Topic (Really!) — The Message — Medium 11/4/14 7:38 PM
So, what gives?
In effect, this was a research project, and it had an implicit research question
(“Do conventionally attractive white women get verbally harassed in New
York?”) and produced an answer: the video. However, in doing so without any
reflection on its own method, it amply demonstrates the crucial substantive
and political importance of research methods.
I’ve taught “introduction to research methods” to undergraduate students for

many years, and they would sometimes ask me why they should care about all
this “method stuff”, besides having a required class for a sociology major out of
the way. I would always tell them, without understanding research methods,
you cannot understand how to judge what you see.
The Hollaback video shows us exactly why.
The Hollaback video also shows why “data” without theory can be so mislead-
ing—and how the same data can fit multiple theories. Since all data collection
involves some form of data selection (even the biggest dataset has selection go-
ing into what gets included, from what source), and since data selection is al-
ways a research method, there is always a need for understanding methods.
First, let’s list all the hypotheses compatible with the “data”, this video:
Hypothesis 1- Men of color are disproportionately more likely to catcall, espe-

cially to a white, conventionally attractive female.
Hypothesis 2- All men are equally likely to catcall but the makers of the video
were biased, consciously or unconsciously, against black men (and edited out
men of other races on purpose.)
2.a Consciously: they are racists and are playing to the “white women endan-
gered by black men” trope — which has a long and ugly history, hence the con-
cern raised by many over the past week.
2.b Unconsciously: There is a methodological twist to the research which cre-

ates this outcome.
2.c. Both 2.a. and 2.b are true.
Hypothesis 3- It’s a spurious correlation: there is some other reason that

caused these two events to go together
The important methodological point is that the video, without further

reflection, can support all three wildly incompatible propositions. In other
words, if you just look at the video, you can believe any three, and you will
likely choose whichever fits your existing conclusions and prejudices..
Let’s start with 3, the easiest to dismiss.
A spurious correlation occurs when a third, unrelated variable, causes a change

in other variables, which then seem like they are causally connected even
though they are not. And since the human brain is a narrative writing machine,
seeing A and B together makes people write stories that tie A and B. A silly, but
correct, example is the correlation between ice cream and murder: during
months when more people eat ice cream, there are more murders. This is not
because popsicles are good murder devices. This spurious correlation caused
by a confounding variable: the season. There is more crime in summer, and
there is more ice cream in summer.
http://xkcd.com/552/
While it’s so obvious in summer and murder example, many major errors have
been made historically because researchers missed confounding variables. A
famous one is the Hormone Replacement Therapy which researchers thought
decreased cancer risk when in fact, it increased it. The confounding variable
was that healthier women were more likely to get the therapy in the first place.
This mistake has certainly caused many deaths.
The most obvious confounding variable here would have been if Hollaback
had their actress walk the streets of Harlem, or other Black neighborhoods, and
skipped over white ones. But they claim they walked all of New York, and did
not just pick minority neighborhoods.
Thankfully, someone did the leg work, and mapped out where the shots were
—something any researcher producing a two minute video from 10 hours
should have done, and a great example of why “anectodes” rather than system-
atic research can be so misleading.
Take a look at their pie-chart:
As the authors correctly state, here’s the question this distribution raises:
The filmmakers claim to have shot this video while walking the streets of
Manhattan for 10 hours, but over half of the shots in the video are actually
taken from just one street, namely 125th St. in Harlem. It makes one wonder
whether the filmmakers intentionally chose to concentrate their filming on a
couple of neighborhoods, or if, out of many locations, these are the only places
where harassment occurred.
Let’s now take hypothesis 1. This will obviously appeal to many people, as it
plays to long-rooted prejudices, and people may well believe that this is their
experience. Most people, especially white people, are more likely to make note
of a person’s race if they are not white. In other words, a man who is a black
man is more likely to be described as a “black man” than a white man is likely
to be described as a “white man” instead of … just a man. This differential
salience of information is why research studies try to carefully systemize their
observations rather than rely on people’s impressions which bias towards
whatever is personally salient—whatever you care about, you will notice more
of.
It’s clear many viewers jumped to the “hypothesis 1” conclusion: that the video
was a representative sample of catcalling. For example, prominent author Joyce
Carol Oates seems to believe this. She’s certainly not the only one.
https://twitter.com/JoyceCarolOates/status/527813215770935296
https://twitter.com/JoyceCarolOates/status/527813848792051712
This is a great example to why personal casual observation and anecdotes,

which is so important to our sense of identity, cannot and should not be trusted
blindly. Systematic study aimed at removing our biases is crucial to under-
standing the world.
Removing the means of implicit biases can be eye-opening. For example, after
decades of lack of women in major orchestras, some big symphonies started
doing their interviews blind — musicians played their instruments behind a
curtain. Lo and behold, women, previously greatly lagging in professional em-
ployment in symphonies, started performing much better.
Hence, given great methodological distortion in selecting examples, the video

will remain compatible with some people’s pre-existing idea that catcalls are
racially disproportional, but will not actually provide a methodological sound
way of evaluating it.
To further evaluate hypothesis 2, let’s turn to what the producer of the video
said. According to a Slate article, he explained what happened like this:
“We got a fair amount of white guys, but for whatever reason, a lot of what
they said was in passing, or off camera,” or was ruined by a siren or other
noise. The final product, he writes, “is not a perfect representation of every-
thing that happened. “
And that right there, is indefensible, methodologically or substantively. The

only neutral explanation is that there is a lot of construction, ambulances and
sirens going on in more white parts of New York, and somehow they just can-
not catch a catcalling white guy. That sounds implausible to me, but if—a big if
—that were the case, an ethical researcher would redo their study because the
data is no longer valid because of the confounding variable—noise at non-
minority neighborhoods. At a minimum, some effort would need to make sure
the presentation of the findings were not tainted by this methodological
obstacle.
There is also a whole discussion about the impact of the choice of a convention-
ally attractive white, young, actress on the findings (even if they were present-
ed perfectly which they are not) and other choices made. In fact, every element
of research design requires hard thinking—and that’s what good social science
does, and what bad anecdotal, jumbled, biased viewpoints miss.
Over at Twitter, David Chen makes another important point: This video was
designed to “go viral” and a video that started with a few white guys doing the
catcalling might not have.
https://twitter.com/davechensky/status/529276016812777472
That is a great point, and a great discussion about the context of “going viral”;
however, the point here is that this is methodologically unsound regardless of
intent or conscious (or unconscious bias).
Research methods, a topic that is seemingly so dry, are the heart and soul of
knowledge. Most data supports more than one theory. This does NOT mean all
data supports all theories: rather, multiple explanations can fit one set of
findings. Choosing the right underlying theory, an iterative process that always
builds upon itself, requires thinking hard on how data selection impacts
findings, and how presentation of findings lends itself to multiple theories, and
how theories fit with existing worldviews, and how better research design can
help us distinguish between competing explanation.
A good research project consciously grapples with these.
A bad one? It flunks, as would this one.
Email me when The Message publishes stories
Follow
Responses
View 2 more responses

Zeynep - Tufekci - That Catcalling Video and Why "Research Methods" - Nov - 3 - 2014

Uploaded by

Document Information

Original Description:

Original Title

Copyright

Available Formats

Share this document

Share or Embed Document

Sharing Options

Did you find this document useful?

Is this content inappropriate?

Copyright:

Available Formats

Zeynep - Tufekci - That Catcalling Video and Why "Research Methods" - Nov - 3 - 2014

Uploaded by

Copyright:

Available Formats

That Catcalling Video and Why “Research Methods” is such an Exciting Topic (Really!

) — The Message — Medium 11/4/14 7:38 PM

Hollaback and Why Everyone Needs Better Research

So, what gives?

I’ve taught “introduction to research methods” to undergraduate students for

you cannot understand how to judge what you see.

The Hollaback video shows us exactly why.

Hypothesis 1- Men of color are disproportionately more likely to catcall, espe-

2.b Unconsciously: There is a methodological twist to the research which cre-

2.c. Both 2.a. and 2.b are true.

Hypothesis 3- It’s a spurious correlation: there is some other reason that

The important methodological point is that the video, without further

Let’s start with 3, the easiest to dismiss.

A spurious correlation occurs when a third, unrelated variable, causes a change

Take a look at their pie-chart:

This is a great example to why personal casual observation and anecdotes,

Hence, given great methodological distortion in selecting examples, the video

And that right there, is indefensible, methodologically or substantively. The

A good research project consciously grapples with these.

A bad one? It flunks, as would this one.

Email me when The Message publishes stories

View 2 more responses

You might also like