8/18/2020

Science
Evaluating Claims to
Knowledge

The Story of
Clever Hans

1
 8/18/2020

The individual asking the question

must know the correct answer

Hans’ Two AND Hans must have an

unobstructed view of the
Conditions individual

If testing was performed when

either of these conditions were not
met, Hans could not solve the
problems

• Hans had mastered the art of reading

nonverbal clues (Which is still really
cool)

SO, What’s • What were the cues?

the deal • Oskar Pfungst used scientific

with Hans? reasoning/methods to demonstrate that
Hans was not solving calculations
• BUT, equally as cool, he showed that
Hans could read nonverbal behavior!!

2
 8/18/2020

 A way to know things

 A method or process to come to know
things
What is  A way of thinking using powerful tools to
Science? help us better understand our reality

 HOW does this relate to Clever Hans?

• A large number of questions were used

to eliminate the effects of chance

The • Different questioners were used in case

Hans was picking up signals from von

science •
Osten
The questioners sometimes knew the
behind the answers to the questions they were
asking, other times they did not
Hans • The questioners would stand at different
distances from Hans during different
effect trials
• Some trials would be run with Hans
blinkered

3
 8/18/2020

What is useful about scientific

reasoning?
 It continually refines/improves our understanding of
reality
 If one theory is disproved…
 It is usually replaced by another theory which is equally
as fascinating
 Science gives us the power to enhance our knowledge
and paves the way for innovation!

Before we continue…

 Theory:
 a well-established principle that has been
developed to explain some aspect of the natural
world
 Hypothesis:
 a specific, testable prediction about what you
expect to happen in your study

4
 8/18/2020

Falsifiability
i

Six Tests of
Logic
Comprehensiveness

Science
Honesty
e
Replicability
Sufficiency

 1998: claimed to find a positive

relationship between use of MMR vaccine
Andrew and autism rates.
 Vaccination rates plummeted.
Wakefield:  Investigators looked into his scientific
MMR practice
 Found:
Vaccine  Funded by lawyers.

and Autism  Unreliable lab techniques

 Wakefield has resisted all criticism and
continues to stand by his findings.

JUNK SCIENCE!!!

10

5
 8/18/2020

Falsifiability

11

Falsifiability

 A hypothesis must be
able to be disproved
 There must be a way
to demonstrate its
falsity
 Things that cannot be
falisfied are devoid of
meaning…

12

6
 8/18/2020

1. Falsifiability
 Definition: an idea framed in a specific, measurable
way so that it can be tested and potentially found to be
incorrect. “Risky test.”

 Reminder: Class demonstration of memory.

 True science sets up hypotheses that are falsifiable:

 Support hypothesis
 Reject hypothesis

 95% confidence
 FALSIFIABILITY GIVES THE FREEDOM TO MAKE A MISTAKE,
TO LEARN, TO GROW!!

13

Falsifiability: A
Weird Example

 The Little Green Men

Problem
 Be skeptical if: it’s
there as long as you
don’t look too
carefully – then it
disappears!

14

7
 8/18/2020

Logic

15

 Based on two criteria:

 The premises on which the argument is
Logic based must all be true
 The proposed conclusion must validly
follow from the premises

16

8
 8/18/2020

1. Crop circles are extremely

complex and numerous.

Logic of 2. Human beings are

incapable of such
Crop complexity on so grand a
scale
Circles 3. Therefore crop circles are
made by extraterrestrials

17

18

9
 8/18/2020

2. LOGIC

 Is the reasoning
behind a claim
logical?

 Does the actual

practice make logical
sense?

19

Logic

Deductive reasoning: a
tool of formal logic in
which a conclusion
necessarily follows
from a set of
premises.
 From a general rule
to a specific example

20

10
 8/18/2020

Logic

Inductive reasoning: a
tool of formal logic in
which a conclusion
probably follows from
a set of premises.
 From a specific
example to a general
proposition

21

• The Marshmallow Study

• What did they want to measure?
• https://www.youtube.com/watch?v=QX_oy9614
Examples HQ

of Logical • What were some potential problems with

measuring it?

Scientific Logic:
Need to measure delayed gratification ONLY.
Thinking
•

• Problem: other environmental factors may

intrude on experience of delayed gratification
• Conclusion: need to conduct in a sparse
laboratory environment.

22

11
 8/18/2020

 The experiment
Additional  If I introduce a treatment (X)

Logic In  And if I compare it to a control group on

outcome Y

Scientific  And I randomize participants to cancel

out other possible contributors

Method  THEN if I find group differences, I can say

that Treatment X caused outcome Y.

23

Illogic: The Case of SOME

Alternative Remedies
• Illogical argument for ancient remedies:
– Treatment X was developed to treat Condition Y 5000 years
ago.
– These ancient people had some understanding of the
natural world that we no longer have.
– Therefore, Treatment X is still the best treatment to use
today.

• An example from actual practice:

– A rhinoceros horn is shaped like a phallus
– Therefore eating ground up rhinoceros horn can cure
male impotence.
[Illogical “like cures like” fallacy]

24

12
 8/18/2020

Wakefield (1998)
• Reported a “correlation” between
children receiving MMR vaccine and
onset of symptoms of autism and PDD.
Example of • In press announcements, declared causal
relationship between MMR and autism.
Illogic in
Junk Problems:

Science 1. Correlation ≠ Causation

2. MMR is given at about the age that
autism becomes more obvious
3. Children were selective group with
gastrointestinal problems (n=12)

25

Comprehensiveness

26

13
 8/18/2020

• A claim to knowledge must account for

ALL data relevant to the topic, not just
bits and pieces
• Examples:
Comprehensiveness – New York City blackout
– Alternative cancer treatments
• IMPORTANT: whenever a claim cannot
explain all the relevant evidence, it fails
the comprehensive test

27

Comprehensiveness, Defined

 Taking in to account all relevant variables

 Taking into account all possible interpretation of results –
OR- Using a study design that cancels out all
interpretations other than what you are studying.

 Correlations reconsidered…

 Bias can lead to consideration of only one potential

interpretation.
 The case of gun violence…

28

14
 8/18/2020

Comprehensiveness In
Research
 Want to make sure to account for everything
 How do we do that?

 The Experiment is the answer!

 A comparison group with no intervention provides a
controlled situation where non-intervention factors are
accounted for.

 However, not all questions can be answered with an

experiment.
 Ex. class attendance prediction

29

 Question: What factors affects student

class attendance in college?
 Hypothesis: Students will be more likely
to attend class more in their major area
than they will a required class in another
area.
 Format: Questionnaire asking about
Hypothetical student’s major, enrolled classes, and
estimated number of missed absences
Study from each class that semester
 Findings: Students attended classes in
their major area significantly more than
they did classes in other areas.
 Conclusion: Student’s interest in and
commitment to a topic area affects
their attendance.

30

15
 8/18/2020

Students And
Class Attendance
• What do you think affects class
attendance?

• Variables we’d need to consider:

– Amount of sleep
– Interest in the class topic
– Workload in other classes
– Health
– Time of day
– Etc….

31

 Wakefield and MMR

problems:
 Small, retrospective study
of specific types of
children.
 No accounting for other
possible causes of autism
Comprehensiveness: at all!
The Question of  Other factors to consider:
What Causes  Genetics
Autism?  Prenatal issues
 Other environmental
factors

 Should consider other

explanations for his findings
 Recall bias
 Gastric problems

32

16
 8/18/2020

33

 Claims to knowledge must

be examined with honesty
and without any form of
Honesty deception
 Children in day care study
 Old requirements for PhD

34

17
 8/18/2020

Honesty in Reporting
Results
 Report all results
 Do not report in a biased manner

 Avoid:
 Fabrication
 Falsification
 Plagiarism

 Usually difficult or impossible to identify from

the outside!
 So be cautious when identifying it in your
research!

35

The Cheerios Example

 Controlled trials, 1998, 2010

 Cheerios versus cornflakes
 Claim: Cheerios can lower your cholesterol.

FACTS:
 Required 2-3 servings a day!
 300-450 calories

 Fact: LDL cholesterol only dropped 7-9 points

 No change in triglycerides
 Fact: Any oat cereal will do.

36

18
 8/18/2020

 Wakefield claim: 8 out of

12 parents reported
symptoms within days of
MMR vaccine

Wakefield –  Investigator findings:

symptoms were expressed
Autism and BEFORE vaccine.

MMR  Wakefield claim: pathology

Vaccine specimens from GI tract
were abnormal
 Independent pathologist
evidence: no sign of GI
pathology

37

Honesty? Misrepresentation?
You Decide.
• Keep an open mind and put on your critical thinking caps

• The importance of falsifiability in assessing honesty in an

extraordinary claim… and repeated demonstrations
(“sufficiency”)
• Psychics/Mediums example:
– https://www.youtube.com/watch?v=5P_0s1TYpJU

38

19
 8/18/2020

Replicability

39

Results of studies need to be

consistent across time and place

Most results are replicated

Replicability multiple times and in different
labs before they are published

Flukes are more likely to happen

during one experiment, but not
as likely to happen across
multiple experiments

40

20
 8/18/2020

• The ability to recreate a finding or

phenomenon (not the actual act of doing
so… that comes later.)

• Scientific research studies: designed to

be replicable.
Replicability – Methods Section
• Participants
• Procedures
• Materials
• Statistical methods

41

Sufficiency

42

21
 8/18/2020

• Just as it sounds:
– A claim to knowledge must
be backed by enough
evidence to enforce the
belief
• Things to remember
Sufficiency 1. Burden of proof is on the
claimant
2. Extraordinary claims
require extraordinary
evidence
3. Evidence based on
authority is inadequate

43

 “Absence of evidence does

not constitute evidence of
absence.”
 Meaning, just because you
Sufficiency cannot find evidence
against someone’s claim,
(cont.) does not mean they are
right
 Example:
 UFO sightings

44

22
 8/18/2020

• Has there been an

attempt to evaluate the
claim repeatedly and
with rigorous
methodology

Sufficiency • “Enough” = quality and

quantity!

• Enough ≠ multiple
authority figures

45

Theories in Psychology with

MORE than Sufficient Evidence

 Cognitive Behavior Therapy (CBT) for depression:

 Over 80 clinical trials since 1977
 Over 15 meta-analyses
 Findings across most studies:
 Mostly found to be effective
 However, this is irrelevant to finding sufficiency!
 All studies could reject hypothesis and we could still
determine that the approach was scientific

46

23
 8/18/2020

 Attempts to reproduce Wakefield’s

claims resulted in:
 12 studies (observational) up through
The Case of 2003

MMR and  None showed any association between

MMR vaccination and autism rates
Autism:  No sufficient evidence on the claim.

Replicability
and  His conclusions have been challenged
and his resistance has grown
Sufficiency
 His results were unreplicable due to
flaws and misreporting.

47

Two KEY About the tests of

science
POINTS…

48

24
 8/18/2020

1. Burden of Proof is On the

Claimant
 If you want to prove that something exists, you need to
provide good scientific evidence. We’re not going to just
believe you.
 EMPIRICAL EVIDENCE
 SCIENTIFIC METHOD

 It is not the role of the scientific community to disprove

an idea
 However, this happens a lot. (e.g., MMR and autism)

 Analogy: “Innocent until proven guilty.”

49

2. Extraordinary claims
require extraordinary
evidence!!!

Why?
• Usually contradict
claims supported by
stronger evidence.

• Must support new

claim while explain
why old claims
appeared correct.

50

25
 8/18/2020

Falsifiability

Logic

Comprehensiveness
FiLCHeRS
Honesty

Replicability

Sufficiency

51

End of Signs of
Science

52

26
 8/18/2020

The Scientific
Process
 Science is an approach to
understanding the world

 It is NOT a topic area!

 Alternative medicine and ESP can

be part of the scientific process

53

It is Not Scientific if You….

 Rely on people’s personal reports to reach conclusions

 Attribute causes to the supernatural realm (without
precision measurement of the supernatural)
 Use scientific language improperly and without data
 Try to justify your claim with meaningless language
 Avoid attempts to examine your claim using the
scientific method

54

27
 8/18/2020

The Question: Is it
Pseudoscience or is it Science?

 Not quite the right question

 More precisely: is the approach scientific or
pseudoscientific?
 Even more precisely: Do the practitioners or
claimants generally use the scientific method to
evaluate their premises, or do they tend to avoid
this approach and use unfounded claims and
language tricks?

55

Is the approach

Examples: pseudoscientific,
scientific, or a
combination of both?

56

28
 8/18/2020

• You find 20 websites promoting an

intervention, using testimonials and
appeals to authority. The language
sounds fancy and refers to obscure
Example bodily functions, but there are no
research studies mentioned. You search
#1 but do not find any research in the
scientific literature.
Pseudoscience, science, or a combination of
both?

57

• You find 20 websites promoting an

intervention, using testimonials and
appeals to authority. One website
mentions a research study that
Example supported the claim, but does not
provide a citation. You search the
#2 scientific literature, and find one
observational research study.

Pseudoscience, science, or a combination?

58

29
 8/18/2020

 You find 20 websites promoting a

supplement for weight loss, with lots of
personal success stories and an
endorsement from a physician. 5 of the
websites also mention some clinical
trials published in academic journals.
Example You look up those articles and find that
they are peer reviewed and are RCTs.
#3 None of them strongly support the use of
the product in the way it is advertised
on the websites. Most are laboratory
studies not done in humans.

 Science, pseudoscience, or both?

59

 You find some websites for therapists

who practice a particular therapeutic
intervention. The websites explain how
the therapy is supposed to work and
mention that it is supported by research.
They do not mention any specific
Example research studies. In the scientific
literature, you find several RCTs and
#4 observational studies, some of which
support the use of this therapeutic
approach and some of which indicate it
was no better than a wait list control
group.
 Science, pseudoscience or both?

60

30
 8/18/2020

 Scientific approach versus “Does it work?”

 Scientific approach: empiricism + logic

 Hypothesis supported
 Hypothesis rejected
Important
Does it work: Was the hypothesis
Distinction! 
supported? Should we investigate this idea
further? Should we incorporate this idea
into our regular practice?
 Many ideas evaluated scientifically found
NOT TO WORK!
 It’s still science IF evaluated
systematically with FiLCHeRS!

61

“What works is
different for
everybody fallacy”
Pseudoscience
Fallacies
“Probability is
irrelevant to the
unique individual
fallacy”

62

31
 8/18/2020

Pseudoscientists claim
that different treatments
work for different people
“What (DUH!)

works if
different They claim scientists use
large groups to test
treatments and ignore the
for effects on individuals

everybody
fallacy” THIS IS NOT TRUE,
scientific research
actually studies
individuals

63

Pseudoscientists: because
scientists study large groups of
patients, results have no bearing
on the individual.
“Probability
is irrelevant
to the Probabilities are irrelevant
unique
individual
fallacy” What if someone was to ask,
“The patient before me is a
unique individual, how can you
apply probabilities to a single
case?”

64

32
 8/18/2020

Four forms of Voodoo Science

(Park, 2000)
• Pseudoscience
– Has the appearance of science, but lacks its rigorous
methodology and skeptical reasoning
• Pathological science
– Deceiving oneself
• Fraudulent science
– Deceiving other
• Junk science
– Theories based on what might be possible rather than
what has been tested and supported

65

33