Professional Documents
Culture Documents
Critical Thinking Articles W6xywo
Critical Thinking Articles W6xywo
The late George Carlin worked weakness, public health problems, crime,
“critical thinking” into one of his comedic and avoidable poverty? Perhaps that
monologue rants on the perils of trusting our realization, along with its obvious
lives and fortunes to the decision-making of advantages for high level strategic decision
people who were gullible, uninformed, and making, is what lead the Chairman of the
unreflective. Had he lived to experience the Joint Chiefs of Staff to comment on critical
economic collapse of 2008 and 2009, he thinking in his commencement address to a
would have surely added more to his graduating class of military officers.
caustic but accurate assessments regarding
how failing to anticipate the consequences
of one’s decisions often leads to disastrous
results not only for the decision maker, but
for many other people as well.
To support the expenses of making this essay available free for non-commercial uses, the publisher has inserted information about
its critical thinking testing instruments. These tools assess the critical thinking skills and habits of mind described in this essay. To
build critical thinking skills and habits of mind use Dr. Facione’s newest book THINK_Critically, Pearson Education 2011.
their own futures and become contributing commonly used concept contains? Take
members of society, rather than burdens on care, though, we would not want to make
society. Becoming educated and practicing the definition so broad that all movie
good judgment does not absolutely violence would be automatically “offensive.”
guarantee a life of happiness, virtue, or And check to be sure your way of defining
economic success, but it surely offers a “offensive violence” fits with how the rest of
better chance at those things. And it is the people who know and use English
clearly better than enduring the would understand the term. Otherwise they
consequences of making bad decisions and will not be able to understand what you
better than burdening friends, family, and all mean when you use that expression.
the rest of us with the unwanted and
avoidable consequences of those poor Did you come up with a definition
choices. that works? How do you know?
Facione, PA, “Critical Thinking: What It is and Why it Counts” 2011 update Page 2
scenes in movies that were offensively when a person or a group of people decides
violent, and did you contrast them with other important matters without pausing first to
scenes that were either not violent or not think things through.
offensively violent? If you did, good. That is
one (but not the only) way to approach the
problem. Technically it is called finding
paradigm cases. Happily, like many things
in life, you do not have to know its name to
do it well.
Facione, PA, “Critical Thinking: What It is and Why it Counts” 2011 update Page 3
Now, consider the example of the Good. What can we learn about
team of people trying to solve a problem. critical thinking from such a case? Maybe
The team members, unlike the courtroom’s more than we can learn from just looking at
adversarial situation, try to collaborate. The the easy cases. For when a case is on the
members of an effective team do not borderline, it forces us to make important
compete against each other. They work in distinctions. It confronts us and demands a
concert, like colleagues, for the common decision: In or Out! And not just that, but
goal. Unless they solve the problem, none why? So, our friend who is fair-minded
of them has won. When they find the way about some things, but close-minded about
to solve the problem, they all have won. So, others, what to do? Let’s take the parts we
from analyzing just two examples we can approve of because they seem to us to
generalize something very important: critical contribute to acting rationally and logically
thinking is thinking that has a purpose and include those in the concept of critical
(proving a point, interpreting what thinking, and let’s take the parts that work
something means, solving a problem), but against reason, that close the mind to the
critical thinking can be a collaborative, possibility of new and relevant information,
noncompetitive endeavor. And, by the way, that blindly deny even the possibility that the
even lawyers collaborate. They can work other side might have merit, and call those
together on a common defense or a joint poor, counterproductive, or uncritical
prosecution, and they can also cooperate thinking.
with each other to get at the truth so that
justice is done.
Facione, PA, “Critical Thinking: What It is and Why it Counts” 2011 update Page 4
habitually seem to exhibit which the weak includes the sub-skills of categorization,
critical thinkers seem not to possess? decoding significance, and clarifying
meaning. Can you think of examples of
interpretation? How about recognizing a
problem and describing it without bias?
How about reading a person’s intentions in
the expression on her face; distinguishing a
main idea from subordinate ideas in a text;
constructing a tentative categorization or
way of organizing something you are
studying; paraphrasing someone’s ideas in
your own words; or, clarifying what a sign,
chart or graph means? What about
identifying an author’s purpose, theme, or
point of view? How about what you did
above when you clarified what “offensive
violence” meant?
Facione, PA, “Critical Thinking: What It is and Why it Counts” 2011 update Page 5
purpose of the passage? What about statements, principles, evidence,
graphically organizing this essay, in your judgments, beliefs, opinions, concepts,
own way, knowing that its purpose is to give descriptions, questions, or other forms of
a preliminary idea about what critical representation.” As sub-skills of inference
thinking means? the experts list querying evidence,
conjecturing alternatives, and drawing
The experts define evaluation as conclusions. Can you think of some
meaning “to assess the credibility of examples of inference? You might suggest
statements or other representations which things like seeing the implications of the
are accounts or descriptions of a person’s position someone is advocating, or drawing
perception, experience, situation, judgment, out or constructing meaning from the
belief, or opinion; and to assess the logical elements in a reading. You may suggest
strength of the actual or intended inferential that predicting what will happen next based
relationships among statements, what is known about the forces at work in a
descriptions, questions or other forms of given situation, or formulating a synthesis of
representation.” Your examples? How related ideas into a coherent perspective.
about judging an author’s or speaker’s How about this: after judging that it would
credibility, comparing the strengths and be useful to you to resolve a given
weaknesses of alternative interpretations, uncertainty, developing a workable plan to
determining the credibility of a source of gather that information? Or, when faced
information, judging if two statements with a problem, developing a set of options
contradict each other, or judging if the for addressing it. What about, conducting a
evidence at hand supports the conclusion controlled experiment scientifically and
being drawn? Among the examples the applying the proper statistical methods to
experts propose are these: “recognizing the attempt to confirm or disconfirm an
factors which make a person a credible empirical hypothesis?
witness regarding a given event or a
credible authority with regard to a given Beyond being able to interpret,
topic,” “judging if an argument’s conclusion analyze, evaluate and infer, strong critical
follows either with certainty or with a high thinkers can do two more things. They can
level of confidence from its premises,” explain what they think and how they
“judging the logical strength of arguments arrived at that judgment. And, they can
based on hypothetical situations,” “judging if apply their powers of critical thinking to
a given argument is relevant or applicable themselves and improve on their previous
or has implications for the situation at hand.” opinions. These two skills are called
“explanation” and “self-regulation.”
Do the people you regard as strong
critical thinkers have the three cognitive The experts define explanation as
skills described so far? Are they good at being able to present in a cogent and
interpretation, analysis, and evaluation? coherent way the results of one’s reasoning.
What about the next three? And your This means to be able to give someone a
examples of weak critical thinkers, are they full look at the big picture: both “to state and
lacking in these cognitive skills? All, or just to justify that reasoning in terms of the
some? evidential, conceptual, methodological,
criteriological, and contextual considerations
To the experts inference means “to upon which one’s results were based; and
identify and secure elements needed to to present one’s reasoning in the form of
draw reasonable conclusions; to form cogent arguments.” The sub-skills under
conjectures and hypotheses; to consider explanation are describing methods and
relevant information and to educe the results, justifying procedures, proposing and
consequences flowing from data, defending with good reasons one’s causal
Facione, PA, “Critical Thinking: What It is and Why it Counts” 2011 update Page 6
and conceptual explanations of events or The experts define self-regulation
points of view, and presenting full and well- to mean “self-consciously to monitor one’s
reasoned, arguments in the context of cognitive activities, the elements used in
seeking the best understandings possible. those activities, and the results educed,
Your examples first, please... Here are particularly by applying skills in analysis,
some more: to construct a chart which and evaluation to one’s own inferential
organizes one’s findings, to write down for judgments with a view toward questioning,
future reference your current thinking on confirming, validating, or correcting either
some important and complex matter, to cite one’s reasoning or one’s results.” The two
the standards and contextual factors used sub-skills here are self-examination and
to judge the quality of an interpretation of a self-correction. Examples? Easy — to
text, to state research results and describe examine your views on a controversial issue
the methods and criteria used to achieve with sensitivity to the possible influences of
those results, to appeal to established your personal biases or self-interest, to
criteria as a way of showing the check yourself when listening to a speaker
reasonableness of a given judgment, to in order to be sure you are understanding
design a graphic display which accurately what the person is really saying without
represents the subordinate and super- introducing your own ideas, to monitor how
ordinate relationship among concepts or well you seem to be understanding or
ideas, to cite the evidence that led you to comprehending what you are reading or
accept or reject an author’s position on an experiencing, to remind yourself to separate
issue, to list the factors that were your personal opinions and assumptions
considered in assigning a final course from those of the author of a passage or
grade. text, to double check yourself by
recalculating the figures, to vary your
Maybe the most remarkable reading speed and method mindful of the
cognitive skill of all, however, is this next type of material and your purpose for
one. This one is remarkable because it reading, to reconsider your interpretation or
allows strong critical thinkers to improve judgment in view of further analysis of the
their own thinking. In a sense this is critical facts of the case, to revise your answers in
thinking applied to itself. Because of that view of the errors you discovered in your
some people want to call this “meta- work, to change your conclusion in view of
cognition,” meaning it raises thinking to the realization that you had misjudged the
another level. But “another level” really importance of certain factors when coming
does not fully capture it, because at that to your earlier decision. 4
next level up what self-regulation does is
look back at all the dimensions of critical
thinking and double check itself. Self-
regulation is like a recursive function in
mathematical terms, which means it can
apply to everything, including itself. You
can monitor and correct an interpretation 4
The California Critical Thinking Skills Test, and the
you offered. You can examine and correct Test of Everyday Reasoning, the Health Science
an inference you have drawn. You can Reasoning Test, the Military and Defense Reasoning
review and reformulate one of your own Profile, The Business Critical Thinking Skills Test,
explanations. You can even examine and and the Legal Studies Reasoning Profile along with
correct your ability to examine and correct other testing instruments authored by Dr. Facione and
yourself! How? It is as simple as stepping his research team for people in K-12, college, and
graduate / professional work target the core critical
back and saying to yourself, “How am I thinking skills identified here. These instruments are
doing? Have I missed anything important? published in English and several authorized
Let me double check before I go further.” translations exclusively by Insight Assessment.
Facione, PA, “Critical Thinking: What It is and Why it Counts” 2011 update Page 7
The Delphi Research Method published under the title Critical Thinking: A
Statement of Expert Consensus for
The panel of experts we keep Purposes of Educational Assessment and
referring to included forty-six men and Instruction. (The California Academic
women from throughout the United States Press, Millbrae, CA, 1990). You may
and Canada. They represented many download the executive summary of that
different scholarly disciplines in the report free. Visit
humanities, sciences, social sciences, and
education. They participated in a research http://www.insightassessment.com
project that lasted two years and was
conducted on behalf of the American You might be wondering how such a
Philosophical Association. Their work was large group of people could collaborate on
Facione, PA, “Critical Thinking: What It is and Why it Counts” 2011 update Page 8
this project over that long a period of time Wait a minute! These are all well-
and at those distances and still come to known experts, so what do you do if people
consensus. Good question. Remember disagree? And what about the possible
we’re talking the days before e-mail. influence of a big name person? Good
points. First, the central investigator takes
Not only did the group have to rely precautions to remove names so that the
on snail mail during their two-year panelists are not told who said what. They
collaboration; they also relied on a method know who is on the panel, of course. But
of interaction, known as the Delphi Method, that is as far as it goes. After that each
which was developed precisely to enable experts’ argument has to stand on its own
experts to think effectively about something merits. Second, an expert is only as good
over large spans of distance and time. In as the arguments she or he gives. So, the
the Delphi Method a central investigator central investigator summarizes the
organizes the group and feeds them an arguments and lets the panelists decide if
initial question. [In this case it had to do they accept them or not. When consensus
with how college level critical thinking appears to be at hand, the central
should be defined so that people teaching at investigator proposes this and asks if
that level would know which skills and people agree. If not, then points of
dispositions to cultivate in their students.] disagreement among the experts are
The central investigator receives all registered. We want to share with you one
responses, summarizes them, and transmits important example of each of these. First
them back to all the panelists for reactions, we will describe the expert consensus view
replies, and additional questions. of the dispositions which are absolutely vital
to strong critical thinking. Then we will note
a point of separation among the experts.
Facione, PA, “Critical Thinking: What It is and Why it Counts” 2011 update Page 9
The Disposition Toward Critical them grow weak with lack of practice. But
Thinking dancers get tired. And they surrender to the
stiffness of age or the fear of injury. In the
What kind of a person would be apt case of critical thinking skills, we might
to use their critical thinking skills? The argue that not using them once you have
experts poetically describe such a person them is hard to imagine. It’s hard to
as having “a critical spirit.” Having a critical imagine a person deciding not to think.
spirit does not mean that the person is
always negative and hypercritical of Considered as a form of thoughtful
everyone and everything. judgment or reflective decision-making, in a
very real sense critical thinking is
pervasive. There is hardly a time or a
The experts use the metaphorical place where it would not seem to be of
phrase critical spirit in a positive sense. By potential value. As long as people have
it they mean “a probing inquisitiveness, a purposes in mind and wish to judge how to
keenness of mind, a zealous dedication accomplish them, as long as people wonder
to reason, and a hunger or eagerness for what is true and what is not, what to believe
reliable information.” and what to reject, strong critical thinking is
going to be necessary.
Almost sounds like Supreme Court
Justice Sandra Day O’Connor or Sherlock And yet weird things happen, so it is
Holmes The kind of person being described probably true that some people might let
here is the kind that always wants to ask their thinking skills grow dull. It is easier to
“Why?” or “How?” or “What happens if?”. imagine times when people are just too
The one key difference, however, is that in tired, too lax, or too frightened. But imagine
fiction Sherlock always solves the mystery, it you can, Young Skywalker, so there has
while in the real world there is no guarantee.
Critical thinking is about how you approach
problems, questions, issues. It is the best
way we know of to get to the truth. But!
There still are no guarantees — no answers
in the back of the book of real life. Does
this characterization, that strong critical
thinkers possess a “critical spirit, a probing
inquisitiveness, a keenness of mind...” fit
with your examples of people you would call
strong critical thinkers?
to be more to critical thinking than just the
But, you might say, I know people list of cognitive skills. Human beings are
who have skills but do not use them. We more than thinking machines. And this
cannot call someone a strong critical thinker brings us back to those all-important
just because she or he has these cognitive attitudes which the experts called
skills, however important they might be, “dispositions.”
because what if they just do not bother to
apply them? The experts were persuaded that
critical thinking is a pervasive and
One response is to say that it is hard purposeful human phenomenon. The ideal
to imagine an accomplished dancer who critical thinker can be characterized not
never dances. After working to develop merely by her or his cognitive skills but also
those skills it seems such a shame to let by how she or he approaches life and living
in general. This is a bold claim. Critical
Facione, PA, “Critical Thinking: What It is and Why it Counts” 2011 update Page 10
thinking goes way beyond the classroom. prudent people would want to ask to
In fact, many of the experts fear that some manage their investments!
of the things people experience in school
are actually harmful to the development and The experts went beyond
cultivation of strong critical thinking. Critical approaches to life and living in general to
thinking came before schooling was ever emphasize that strong critical thinkers can
invented, it lies at the very roots of also be described in terms of how they
civilization. It is a corner stone in the approach specific issues, questions, or
journey human kind is taking from beastly problems. The experts said you would find
savagery to global sensitivity. Consider these sorts of characteristics:
what life would be like without the things on
this list and we think you will understand. * clarity in stating the question or concern,
The approaches to life and living which * orderliness in working with complexity,
characterize critical thinking include: * diligence in seeking relevant information,
* reasonableness in selecting and applying criteria,
* inquisitiveness with regard to a wide range of * care in focusing attention on the concern at hand,
issues, * persistence though difficulties are encountered,
* concern to become and remain well-informed, * precision to the degree permitted by the subject and
* alertness to opportunities to use critical thinking, the circumstances.
* trust in the processes of reasoned inquiry,
* self-confidence in one’s own abilities to reason, So, how would a weak critical thinker
* open-mindedness regarding divergent world views, approach specific problems or issues?
* flexibility in considering alternatives and opinions Obviously, by being muddle-headed about
* understanding of the opinions of other people, what he or she is doing, disorganized and
* fair-mindedness in appraising reasoning, overly simplistic, spotty about getting the
* honesty in facing one’s own biases, prejudices, facts, apt to apply unreasonable criteria,
stereotypes, or egocentric tendencies, easily distracted, ready to give up at the
* prudence in suspending, making or altering least hint of difficulty, intent on a solution
judgments, that is more detailed than is possible, or
* willingness to reconsider and revise views where being satisfied with an overly generalized
honest reflection suggests that change is and uselessly vague response. Remind you
warranted. of anyone you know?
Facione, PA, “Critical Thinking: What It is and Why it Counts” 2011 update Page 11
place of “is a strong critical thinker” or “has
A person disposed to be averse or strong critical thinking skills.” This is not
hostile toward using critical thinking only a helpful conversational shortcut, it
would probably disagree with the suggests that to many people “critical
statements above but be likely to agree with thinker” has a laudatory sense. The word
these: can be used to praise someone at the same
time that it identifies the person, as in “Look
“I prefer jobs where the supervisor says at that play. That’s what I call a defender!”
exactly what to do and exactly how to do it."
“No matter how complex the problem, you
can bet there will be a simple solution.” “If we were compelled to make a
"I don't waste time looking things up." choice between these personal
“I hate when teachers discuss problems attributes and knowledge about
instead of just giving the answers.”
“If my belief is truly sincere, evidence to the the principles of logical
contrary is irrelevant." reasoning together with some
“Selling an idea is like selling cars, you say degree of technical skill in
whatever works."
manipulating special logical
processes, we should decide for
We used the expression “strong the former.”
critical thinker” to contrast with the
expression “weak critical thinker.” But you John Dewey, How We Think, 1909. Republished as
will find people who drop the adjective How We Think: A Restatement of the Relation of
Reflective Thinking to the Educational Process. D. C.
“strong” (or “good”) and just say that Heath Publishing. Lexington, MA. 1933.
someone is a “critical thinker” or not. It is
like saying that a soccer (European
“football”) player is a “defender” or “not a We said the experts did not come to
defender”, instead of saying the player’s full agreement on something. That thing
skills at playing defense are strong or weak. has to do with the concept of a “strong
People use the word “defender” in place of critical thinker.” This time the emphasis is
the phrase “is good at playing defense.” on the word “good” because of a crucial
Similarly, people use “critical thinker” in
Facione, PA, “Critical Thinking: What It is and Why it Counts” 2011 update Page 12
ambiguity it contains. A person can be would prefer to think that critical thinking, by
good at critical thinking, meaning that the its very nature, is inconsistent with the kinds
person can have the appropriate of unethical and deliberately
dispositions and be adept at the cognitive counterproductive examples given. They
processes, while still not being a good (in find it hard to imagine a person who was
the moral sense) critical thinker. For good at critical thinking not also being good
example, a person can be adept at in the broader personal and social sense.
developing arguments and then, unethically, In other words, if a person were “really” a
use this skill to mislead and exploit a gullible “strong critical thinker” in the procedural
person, perpetrate a fraud, or deliberately sense and if the person had all the
confuse and confound, and frustrate a appropriate dispositions, then the person
project. simply would not do those kinds of
exploitive and aggravating things.
The experts were faced with an
interesting problem. Some, a minority,
This self-rating form also appears in Chapter 3 of Think Critically, Pearson Education, 2011. For a fuller and more
robust measure of critical thinking dispositions see the California Critical Thinking Disposition Inventory (CCTDI) by
Facione and Facione, published in 1992, by Insight Assessment.
Facione, PA, “Critical Thinking: What It is and Why it Counts” 2011 update Page 13
The large majority, however, hold “Thinking” in Popular Culture
the opposite judgment. They are firm in the
view that strong critical thinking has nothing We have said so many good things
to do with any given set of cultural beliefs, about critical thinking that you might have
religious tenants, ethical values, social the impression that “critical thinking” and
mores, political orientations, or orthodoxies “good thinking” mean the same thing. But
of any kind. Rather, the commitment one that is not what the experts said. They see
makes as a strong critical thinker is to critical thinking as making up part of what
always seek the truth with objectivity, we mean by good thinking, but not as being
integrity, and fair-mindedness. The majority the only kind of good thinking. For example,
of experts maintain that critical thinking they would have included creative thinking
conceived of as we have described it above, as part of good thinking.
is, regrettably, not inconsistent with abusing
one’s knowledge, skills, or power. There Creative or innovative thinking is the
have been people with superior thinking kind of thinking that leads to new insights,
skills and strong habits of mind who, novel approaches, fresh perspectives,
unfortunately, have used their talents for whole new ways of understanding and
ruthless, horrific, and immoral purposes. conceiving of things. The products of
Would that it were not so! Would that creative thought include some obvious
experience, knowledge, mental horsepower, things like music, poetry, dance, dramatic
and ethical virtue were all one and the literature, inventions, and technical
same. But from the time of Socrates, if not innovations. But there are some not so
thousands of years before that, humans obvious examples as well, such as ways of
have known that many of us have one or putting a question that expand the horizons
more of these without having the full set. of possible solutions, or ways of conceiving
of relationships which challenge
Any tool, any approach to situations, presuppositions and lead one to see the
can go either way, ethically speaking, world in imaginative and different ways.
depending on the character, integrity, and
principles of the persons who possess The experts working on the concept
them. So, in the final analysis the majority of critical thinking wisely left open the entire
of experts maintained that we cannot say a question of what the other forms good
person is not thinking critically simply thinking might take. Creative thinking is
because we disapprove ethically of what the only one example. There is a kind of
person is doing. The majority concluded purposive, kinetic thinking that instantly
that, “what ‘critical thinking’ means, why it is coordinates movement and intention as, for
of value, and the ethics of its use are best example, when an athlete dribbles a soccer
regarded as three distinct concerns.” ball down the field during a match. There is
a kind of meditative thinking which may
Perhaps this realization forms part of lead to a sense of inner peace or to
the basis for why people these days are profound insights about human existence.
demanding a broader range of learning In contrast, there is a kind of hyper-alert,
outcomes from our schools and colleges. instinctive thinking needed by soldiers in
“Knowledge and skills,” the staples of the battle. In the context of popular culture one
educational philosophy of the mid-twentieth finds people proposing all kinds of thinking
century, are not sufficient. We must look to or this-kind of intelligence or that-kind of
a broader set of outcomes including habits intelligence. Some times it is hard to sort out
of mind and dispositions, such as civic the science from the pseudo-science – the
engagement, concern for the common kernel of enduring truth from the latest
good, and social responsibility. cocktail party banter.
Facione, PA, “Critical Thinking: What It is and Why it Counts” 2011 update Page 14
“Thinking” in Cognitive Science precisely because drivers are able to see
and react to dangerous situations so
Theories emerging from more scientific quickly. Many good decisions which feel
studies of human thinking and decision- intuitive are really the fruit of expertise.
making in recent years propose that thinking Decisions good drivers make in those
is more integrated and less dualistic than moments of crisis, just like the decisions
the notions in popular culture suggest. We which practiced athletes make in the flow of
should be cautious about proposals a game or the decisions that a gifted
suggesting oversimplified ways of teacher makes as she or he interacts with
understanding how humans think. We students, are borne of expertise, training,
should avoid harsh, rigid dichotomies such and practice.
as “reason vs. emotion,” “intuitive vs. linear,”
“creativity vs. criticality,” “right brained vs. At the same time that we are
left brained,” “as on Mars vs. as on Venus.” immersed in the world around us and in our
daily lives, constantly making decisions
There is often a kernel of wisdom in unreflectively, we may also be thinking quite
popular beliefs, and perhaps that gem this reflectively about something. Perhaps we’re
time is the realization that some times we worried about a decision which we have to
decide things very quickly almost as make about an important project at work, or
spontaneous, intuitive, reactions to the about a personal relationship, or about a
situation at hand. Many accidents on the legal matter, whatever. We gather
freeways of this nation are avoided information, consider our options, explore
Facione, PA, “Critical Thinking: What It is and Why it Counts” 2011 update Page 15
possibilities, formulate some thoughts about review and revise our work in the light of
what we propose to do and why this choice relevant guidelines or standards or rules of
is the right one. In other words, we make a procedure. While System 2 decisions are
purposeful, reflective judgment about what also influenced by the correct or incorrect
to believe or what to do – precisely the kind application of heuristic maneuvers, this is
of judgment which is the focus of critical the system which relies on well articulated
thinking. reasons and more fully developed evidence.
It is reasoning based on what we have
Recent integrative models of human learned through careful analysis, evaluation,
decision-making propose that the thinking explanation, and self-correction. This is the
processes of our species is not best system which values intellectual honesty,
described as a conflictive duality as in analytically anticipating what happens next,
“intuitive vs. reflective” but rather an maturity of judgment, fair-mindedness,
integrative functioning of two mutually elimination of biases, and truth-seeking.
supportive systems “intuitive and reflective.” This is the system which we rely on to think
These two systems of thinking are present carefully trough complex, novel, high-
in all of us and can act in parallel to process stakes, and highly integrative problems.5
cognitively the matters over which we are
deciding. Educators urge us to improve our
critical thinking skills and to reinforce our
One system is more intuitive, disposition to use those skills because that
reactive, quick and holistic. So as not to is perhaps the best way to develop and
confuse things with the notions of thinking in refine our System 2 reasoning.
popular culture, cognitive scientists often
name this system, “System 1.” The other System 1 and System 2 are both
(yes, you can guess its name) is more believed to be vital decision-making tools
deliberative, reflective, computational and when stakes are high and when uncertainty
rule governed. You are right, it is called is an issue. Each of these two cognitive
“System 2.” systems are believed to be capable of
functioning to monitor and potentially
In System 1 thinking, one relies override the other. This is one of the ways
heavily on a number of heuristics (cognitive our species reduces the chance of making
maneuvers), key situational characteristics, foolish, sub-optimal or even dangerous
readily associated ideas, and vivid errors in judgment. Human thinking is far
memories to arrive quickly and confidently from perfect. Even a good thinker makes
at a judgment. System 1 thinking is both System 1 and 2 errors. At times we
particularly helpful in familiar situations misinterpret things, or we get our facts
when time is short and immediate action is wrong, and we make mistakes as a result.
required.
5
Chapters 9 and 10 of Think Critically, Pearson
While System 1 is functioning,
Education, 2011, locate critical thinking within this
another powerful system is also at work, integrative model of thinking. The cognitive heuristics,
that is, unless we shut it down by abusing which will be described next, and the human capacity
alcohol or drugs, or with fear or indifference. to derive sustained confidence decisions (right or
Called “System 2,” this is our more wrong),-- known as “dominance structuring,” – are
presented there too. There are lots of useful exercises
reflective thinking system. It is useful for and examples in that book. You may also wish to
making judgments when you find yourself in consult the references listed at the end of this essay.
unfamiliar situations and have more time to The material presented in this section is derived from
figure things out. It allows us to process these books and related publications by many of
these same authors and others working to
abstract concepts, to deliberate, to plan
scientifically explain how humans actually make
ahead, to consider options carefully, to decisions.
Facione, PA, “Critical Thinking: What It is and Why it Counts” 2011 update Page 16
But often our errors are directly related to negative reaction to some idea, proposal,
the influences and misapplications of person, object, whatever. Sometimes called
cognitive heuristics. Because we share the a “gut reaction” this affective response sets
propensity to use these heuristics as we up an initial orientation in us, positive or
make decisions, let’s examine how some of negative, toward the object. It takes a lot of
them influence us. System 2 reasoning to overcome a powerful
affective response to an idea, but it can be
done. And at times it should be, because
there is no guarantee that your gut reaction
is always right.
Facione, PA, “Critical Thinking: What It is and Why it Counts” 2011 update Page 17
happened to that person is therefore more the time each of the decisions along the
likely to happen to us. The similarity way was “good enough for the time being.”
heuristic functions much like an analogical
argument or metaphorical model. The We are by nature a species that is
similarity we focus on might be fundamental averse to risk and loss. Often we make
and relevant, which would make the decisions on the basis of what we are too
inference more warranted. For example, the worried about losing, rather than on the
boss fired your coworker for missing sales basis of what we might gain. This works out
targets and you draw the reasonable to be a rather serviceable approach in many
conclusion that if you miss your sales circumstances. People do not want to lose
targets you’ll be fired too. Or the similarity control, they do not want to lose their
that comes to mind might be superficial or freedom, they do not want to lose their lives,
not connected with the outcome, which their families, their jobs, their possessions.
would make the inference unwarranted. For High stakes gambling is best left to those
example you see a TV commercial showing who can afford to lose the money. Las
trim-figured young people enjoying fattening Vegas didn’t build all those multi-million
fast foods and infer that because you’re dollar casino hotels because vacationers
young too you can indulge your cravings for are winning all the time! And so, in real life,
fast foods without gaining a lot of excess we take precautions. We avoid
unsightly poundage. unnecessary risks. The odds may not be
stacked against us, but the consequences
Heuristics and biases often of losing at times are so great that we would
appearing to be somewhat more associated prefer to forego the possibilities of gain in
with System 2 thinking include: satisficing, order not to lose what we have. And yet, on
risk/loss aversion, anchoring with occasion this can be a most unfortunate
adjustment, and the illusion of control. decision too. History has shown time and
time again that businesses which avoid
Satisficing occurs as we consider risks often are unable to compete
our alternatives. When we come to one successfully with those willing to move more
which is good enough to fulfill our objectives boldly into new markets or into new product
we often regard ourselves as having lines.
completed our deliberations. We have
satisficed. And why not? The choice is, Any heuristic is only a maneuver,
after all, good enough. It may not be perhaps a shortcut or impulse to think or act
perfect, it may not be optimal, it may not in one way rather than another, but certainly
even be the best among the options not a failsafe rule. It may work out well
available. But it is good enough. Time to much of the time to rely on the heuristic, but
decide and move forward. it will not work out for the best all of the
time.
The running mate of satisficing is
temporizing. Temporizing is deciding that For example, people with something
the option which we have come to is “good to lose tend toward conservative choices
enough for now.” We often move through politically as well as economically. Nothing
life satisficing and temporizing. At times we wrong with that necessarily. Just an
look back on our situations and wonder why observation about the influence of Loss
it is that we have settled for far less than we Aversion heuristic on actual decision
might have. If we had only studied harder, making. We are more apt to endure the
worked out a little more, taken better care of status quo, even as it slowly deteriorates,
ourselves and our relationships, perhaps we than we are to call for “radical” change.
would not be living as we are now. But, at Regrettably, however, when the call for
change comes, it often requires a far
Facione, PA, “Critical Thinking: What It is and Why it Counts” 2011 update Page 18
greater upheaval to make the necessary future events may be quite negligible. At
transformations, or, on occasion, the times we do have some measure of control.
situation has deteriorated beyond the point For example we may exercise, not smoke,
of no return. In those situations we find and watch our diet in order to be more fit
ourselves wondering why we waited so long and healthy. We are careful not to drink if
before doing something. we are planning to drive so that we reduce
the risks of being involved in a traffic
The heuristic known as Anchoring accident. But at times we simply are
with Adjustment is operative when we find mistaken about our ability to actually
ourselves making evaluative judgments. exercise full control over a situation. Sadly
The natural thing for us to do is to locate or we might become ill even if we do work hard
anchor our evaluation at some point along to take good care of ourselves. Or we may
whatever scale we are using. For example, be involved in an accident even if we are
a professor says that the student’s paper is sober. Our business may fail even if we
a C+. Then, as other information comes our work very hard to make it a success. We
way, we may adjust that judgment. The may not do as well on an exam as we might
professor, for example, may decide that the hope even if we study hard.
paper is as good as some others that were
given a B-, and so adjust the grade upward. Related to the Illusion of Control
The interesting thing about this heuristic, is heuristic is the tendency to misconstrue our
that we do not normally start over with a personal influence or responsibility for past
fresh evaluation. We have dropped anchor events. This is called Hindsight Bias. We
and we may drag it upward or downward a may over-estimate the influence our actions
bit, but we do not pull it off the bottom of the have had on events when things go right, or
sea to relocate our evaluation. First we may underestimate our responsibility or
impressions, as the saying goes, cannot be culpability when things go wrong. We have
undone. The good thing about this heuristic all heard people bragging about how they
is that it permits us to move on. We have did this and how they did that and, as a
done the evaluation; there are other papers result, such and such wonderful things
to grade, other projects to do, other things in happened. We made these great plans and
life that need attention. We could not long look how well our business did financially.
endure if we had to constantly re-evaluate Which may be true when the economy is
every thing anew. The unfortunate thing strong; but not when the economy is failing.
about this heuristic is that we sometimes It is not clear how much of that success
drop anchor in the wrong place; we have a came from the planning and how much
hard time giving people a second chance at came from the general business
making a good first impression. environment. Or, we have all been in the
room when it was time to own up for some
The heuristic known as Illusion of thing that went wrong and thought to
Control is evident in many situations. Many ourselves, hey, I may have had some part in
of us over-estimate our abilities to control this, but it was not entirely my fault. “It
what will happen. We make plans for how wasn’t my fault the children were late for
we are going to do this or that, say this or school, hey I was dressed and ready to go
that, manipulate the situation this way or at the regular time.” As if seeing that the
that way, share or not share this information family was running late I had no
or that possibility, all the time thinking that responsibility to take some initiative and
some how our petty plans will enable us to help out.
control what happens. We act as if others
are dancing on the ends of the strings that
we are pulling, when in actuality the
influences our words or actions have on
Facione, PA, “Critical Thinking: What It is and Why it Counts” 2011 update Page 19
“Insanity is doing the same not what happens.6 When seeking to
explain how people decide on an option with
thing over and over again while such conviction that they stick to their
expecting a different outcome.” decision over time and with such confidence
that they act on that decision, the concept
that what we do is build a Dominance
Albert Einstein Structure has been put forth. In a nutshell
this theory suggests that when we settle on
Research on our shared heuristic a particular option which is good enough we
patterns of decision-making does not aim to tend to elevate its merits and diminish its
evaluate these patterns as necessarily good flaws relative to the other options. We raise
or bad patterns of thinking. I fear that my it up in our minds until it becomes for us the
wording of them above may not have been dominant option. In this way, as our
as entirely neutral and descriptive as decision takes shape, we gain confidence in
perhaps it should have been. In truth, our choice and we feel justified in
reliance on heuristics can be an efficient dismissing the other options, even though
ways of deciding things, given how very the objective distance between any of them
complicated our lives are. We cannot and our dominant option may not be very
devote maximal cognitive resources to great at all. But we become invested in our
every single decision we make. dominant option to the extent that we are
Those of us who study these able to put the other possibilities aside and
heuristic thinking phenomena are simply act on the basis of our choice. In fact, it
trying to document how we humans do comes to dominate the other options in our
think. There are many useful purposes for minds so much that we are able to sustain
doing this. For example, if we find that our decision to act over a period of time,
people repeatedly make a given kind of rather than going back to re-evaluate or
mistake when thinking about a commonly reconsider constantly. Understanding the
experienced problem, then we might find natural phenomenon of dominance
ways to intervene and to help ourselves not structuring can help us appreciate why it
repeat that error over and over again. can be so difficult for us to get others to
change their minds, or why it seems that our
This research on the actual patterns reasons for our decisions are so much
of thinking used by individuals and by better than any of the objections which
groups might prove particularly valuable to others might make to our decisions. This is
those who seek interventions which could not to say that we are right or wrong.
improve how we make our own heath care Rather, this is only to observe that human
decisions, how we make business beings are capable of unconsciously
decisions, how we lead teams of people to building up defenses around their choices
work more effectively in collaborative which can result in the warranted or
settings, and the like. unwarranted confidence to act on the basis
of those choices.
Popular culture offers one other
myth about decision-making which is worth
questioning. And that is the belief that when
we make reflective decisions we carefully 6
Henry Montgomery, “From cognition to action: The
weigh each of our options, giving due
search for dominance in decision making.” In Process
consideration to all of them in turn, before and Structure in Human Decision-Making,
deciding which we will adopt. Although Montgomery H, Svenson O (Eds). John Wiley & Sons:
perhaps it should be, research on human Chichester, UK, 1989. For a more accessible
decision-making shows that this simply is description along with reflective exercises on how to
avoid becoming “locked in” to a poor decision
prematurely, see chapter 10 of Think Critically.
Facione, PA, “Critical Thinking: What It is and Why it Counts” 2011 update Page 20
Realizing the power of dominance you have greater success in your work?
structuring, one can only be more Would you get better grades?
committed to the importance of education
and critical thinking. We should do all that Actually the answer to the grades
we can to inform ourselves fully and to question, scientifically speaking, is very
reflect carefully on our choices before we possibly, Yes! A study of over 1100 college
make them, because we are, after all, students shows that scores on a college
human and we are as likely as the next level critical thinking skills test significantly
person to believe that we are right and they correlated with college GPA.7 It has also
are wrong once the dominance structure been shown that critical thinking skills can
begins to be erected. Breaking through that be learned, which suggests that as one
to fix bad decisions, which is possible, can learns them one’s GPA might well improve.
be much harder than getting things right in In further support of this hypothesis is the
the first place. significant correlation between critical
thinking and reading comprehension.
There are more heuristics than only Improvements in the one are paralleled by
those mentioned above. There is more to improvements in the other. Now if you can
learn about dominance structuring as it read better and think better, might you not
occurs in groups as well as in individuals, do better in your classes, learn more, and
and how to mitigate the problems which get better grades. It is, to say the least,
may arise by prematurely settling on a very plausible.
“good enough” option, or about how to craft
educational programs or interventions which Learning, Critical Thinking, and Our
help people be more effective in their Nation’s Future
System 1 and System 2 thinking. There is
much to learn about human thinking and “The future now belongs to societies
how to optimize it in individuals of different that organize themselves for learning...
ages; how to optimize the thinking of groups nations that want high incomes and full
employment must develop policies that
of peers and groups where organizational
emphasize the acquisition of knowledge
hierarchies influence interpersonal and skills by everyone, not just a select
dynamics. And, happily, there is a lot we few.”
know today about human thinking and
decision-making that we did not know a few Ray Marshall & Marc Tucker, Thinking For A Living:
years ago. Education And The Wealth of Nations, Basic Books. New
York. 1992.
Facione, PA, “Critical Thinking: What It is and Why it Counts” 2011 update Page 21
level technical and professional programs challenge, question, and dissent. In fact,
have a half-life of about four years, which this is exactly what the professors want.
means that the technical content is They want their students to excel on their
expanding so fast and changing so much own, to go beyond what is currently known,
that in about four years after graduation to make their own contributions to
your professional training will be in serious knowledge and to society. [Being a
need of renewal. So, if the only thing a professor is a curious job — the more
college is good for is to get the entry level effective you are as a teacher, less your
training and the credential needed for some students require your aid in learning.]
job, then college would be a time-limited
value. Liberal education is about learning
to learn, which means learning to think for
yourself on your own and in collaboration
with others.
Facione, PA, “Critical Thinking: What It is and Why it Counts” 2011 update Page 22
appraisal of all that is good and all that is scale economic disaster would become
bad in the human condition. As the mind extremely likely. So, given a society that
awakens and matures, and the proper does not value and cultivate critical thinking,
nurturing and educational nourishment is we might reasonably expect that in time the
provided, these others central parts of a judicial system and the economic system
liberal education develop as well. Critical would collapse. And, in such a society, one
thinking plays an essential role in achieving that does not liberate its citizens by teaching
these purposes. them to think critically for themselves, it
would be madness to advocate democratic
Any thing else? What about going forms of government.
beyond the individual to the community?
Facione, PA, “Critical Thinking: What It is and Why it Counts” 2011 update Page 23
Imagine that because of war, or AIDS, or Consider the “cultural revolutions”
famine, or religious conviction, parents undertaken by totalitarian rulers. Notice how
could not or would not teach their children in virtually every case absolutist and
how to think critically. Imagine the social dictatorial despots seek ever more severe
and political strife, the falling apart of limitations on free expression. They label
fundamental systems of public safety and “liberal” intellectuals “dangers to society”
public health, the loss of any scientific and expel “radical” professors from teaching
understanding of disease control or posts because they might “corrupt the
agricultural productivity, the emergence of youth.” Some use the power of their
paramilitary gangs, strong men, and petty governmental or religious authority to crush
warlords seeking to protect themselves and not only their opposition but the moderates
their own by acquiring control over what as well -- all in the name of maintaining the
food and resources they can and destroying purity of their movement. They intimidate
those who stand in their path. journalists and those media outlets which
dare to comment “negatively” on their
Look at what has happened around political and cultural goals or their heavy
the world in places devastated by economic handed methods.
embargoes, one-sided warfare, or the
HIV/AIDS epidemic. Or, consider the The historical evidence is there for
problem of global climate change, and how us to see what happens when schools are
important it is for all of us to cooperate with closed or converted from places of
efforts to curtail our uses of fossil fuels in education to places for indoctrination. We
order to reduce emissions of harmful know what happens when children are no
greenhouse gases. longer being taught truth-seeking, the skills
of good reasoning, or the lessons of human
history and basic science: Cultures
disintegrate; communities collapse; the
machinery of civilization fails; massive
numbers of people die; and sooner or later
social and political chaos ensues.
Facione, PA, “Critical Thinking: What It is and Why it Counts” 2011 update Page 24
violently extreme religious zealots. intellectuals, or regulations aimed at
Education which includes a good measure suppressing research and frustrating the
of critical thinking skills and dispositions like fair-minded, evidence-based, and unfettered
truth-seeking and open-mindedness, is a pursuit of knowledge, can happen wherever
problem for terrorists and extremists of and whenever people are not vigilant
every stripe because terrorists and defenders of open, objective, and
extremists want to control of what people independent inquiry.
think. They are ideologists of the worst kind.
Their methods include indoctrination, Does this mean that society should place a
intimidation, and the strictest authoritarian very high value on critical thinking?
orthodoxy. In the “black-and-white” world of
“us vs. them” a good education would mean Absolutely!
that the people might begin to think for
themselves. And that is something these Does this mean society has the right to
extremists do not want. force someone to learn to think critically?
History shows that assaults on Maybe. But, really, should we have to?
learning, whether by book burning, exile of
Facione, PA, “Critical Thinking: What It is and Why it Counts” 2011 update Page 25
EXPERT CONSENSUS STATEMENT REGARDING CRITICAL
THINKING AND THE IDEAL CRITICAL THINKER
American Philosophical Association, Critical Thinking: A Statement of Expert Consensus for Purposes of Educational Assessment
and Instruction. "The Delphi Report," Committee on Pre-College Philosophy. (ERIC Doc. No. ED 315 423). 1990
Brookfield, Stephen D.: Developing Critical Thinkers: Challenging Adults to Explore Alternative Ways of Thinking and Acting. Josey-
Bass Publishers. San-Francisco, CA. 1987.
Browne, M. Neil, and Keeley, Stuart M.: Asking the Right Questions. Prentice-Hall Publishers. Englewood Cliffs, NJ. 2003.
Costa, Arthur L., & Lowery, l Lawrence F.: Techniques for Teaching Thinking. Critical Thinking Press and Software. Pacific Grove,
CA. 1989.
Facione, Noreen C, and Facione Peter A.: Critical Thinking and Clinical Judgment in the Health Sciences - An International
Teaching Anthology. The California Academic Press, Millbrae CA. 2008.
Facione, Noreen C. and Facione, Peter A. : Critical Thinking Assessment and Nursing Education Programs: An Aggregate Data
Analysis. The California Academic Press. Millbrae, CA 1997.
Facione, Noreen. C., and Facione, Peter A., Analyzing Explanations for Seemingly Irrational Choices, International Journal of
Applied Philosophy, Vol. 15 No. 2 (2001) 267-86.
Facione, Peter A and Noreen C,: Thinking and Reasoning in Human Decision Making. The California Academic Press. Millbrae CA,
2007
Facione, Peter A, Think Critically, Pearson Education: Englewood Cliffs, NJ, 2011.
Facione, P.A., Facione, N.C., Talking Critical Thinking, Change: The Magazine of Higher Education, March-April, 2007.
Facione, P.A., Facione N. C., and Giancarlo, C: The Disposition Toward Critical Thinking: Its Character, Measurement, and
Relationship to Critical Thinking Skills, Journal of Informal Logic, Vol. 20 No. 1 (2000) 61-84.
Facione, PA, “Critical Thinking: What It is and Why it Counts” 2011 update Page 26
Gilovich, Thomas; Griffin, Dale; and Kahneman, Daniel: Heuristics and Biases: The Psychology of Intuitive Judgment. Cambridge
University Press. 2002.
Goldstein, William, and Hogarth, Robin M. (Eds.): Research on Judgment and Decision Making. Cambridge University Press. 1997.
Esterle, John, and Clurman, Dan: Conversations with Critical Thinkers. The Whitman Institute. San Francisco, CA. 1993.
Janis, I.L. and Mann, L: Decision-Making. The Free Press, New York. 1977.
Kahneman, Daniel; Slovic, Paul; and Tversky, Amos: Judgment Under Uncertainty: Heuristics and Biases. Cambridge University
Press. 1982.
Kahneman Daniel: Knetsch, J.L.; and Thaler, R.H.: The endowment effect, loss aversion, and status quo bias. Journal of Economic
Perspectives. 1991, 5;193-206.
King, Patricia M. & Kitchener, Karen Strohm: Developing Reflective Judgment. Josey-Bass Publishers. San Francisco, CA. 1994
Kurfiss, Joanne G., Critical Thinking: Theory, Research, Practice and Possibilities, ASHE-ERIC Higher Education Report # 2,
Washington DC, 1988.
Marshall, Ray, and Tucker, Marc, Thinking for a Living: Education and the Wealth of Nations, Basic Books. New York, NY. 1992.
Resnick, L. W., Education and Learning to Think, National Academy Press, 1987.
Rubenfeld, M. Gaie, & Scheffer, Barbara K., Critical Thinking in Nursing: An Interactive Approach. J. B. Lippincott Company.
Philadelphia PA, 1995.
Siegel, Harvey: Educating Reason: Rationality, CT and Education. Routledge Publishing. New York. 1989.
Sternberg, Robert J.: Critical Thinking: Its Nature, Measurement, and Improvement. National Institute of Education, Washington DC,
1986.
Wade, Carole, and Tavris, Carol: Critical & Creative Thinking: The Case of Love and War. Harper Collins College Publisher. New
York. NY 1993.
GOVERNMENT REPORTS
U.S. Department of Education, Office of Educational Research and Improvement, National Center for Educational
Statistics (NCES) Documents National Assessment of College Student Learning: Getting Started, A Summary of Beginning
Activities. NCES 93-116.
National Assessment of College Student Learning: Identification of the Skills to Be Taught, Learned, and Assessed, A
Report on the Proceedings of the Second Design Workshop, November 1992. NCES 94-286.
National Assessment of College Student Learning: Identifying College Graduates' Essential Skills in Writing, Speech and
Listening, and Critical Thinking. NCES 95-001.
Dr. Peter A. Facione and his co-investigators have been engaged in research and teaching about reasoning,
decision-making, and effective individual and group thinking processes since 1967. Over the years they developed
instruments to measure the core skills and habits of mind of effective thinking, these instruments are now in use in
many different languages throughout the world. Since 1992 Dr. Facione has presented hundreds of workshops about
effective teaching for thinking and about leadership, decision-making, leadership development, planning and
budgeting, and learning outcomes assessment at national and international professional association meetings and on
college and university throughout the nation. Dr. Facione, is a principal of the research and consulting firm,
Measured Reasons, and a Senior Researcher with Insight Assessment. He earned his Ph.D. at Michigan State in
1971, and in subsequent years chaired the Department of Philosophy at Bowling Green State University, served as
Dean of the School of Human Development and Community Service at California State University Fullerton, Dean of
the College of Arts and Sciences at Santa Clara University, and Provost of Loyola University Chicago. In 1999-2000
Dr. Facione was Chair of the American Conference of Academic Deans. He has been on many boards and panels,
including the California Commission on Teacher Credentialing and the ACE Presidents’ Task Force on Education. He
Facione, PA, “Critical Thinking: What It is and Why it Counts” 2011 update Page 27
has contributed articles to The Chronicle of Higher Education, Change - The Magazine of Higher Education, and
Liberal Education. With Dr. Noreen Facione he co-authored Thinking and Reasoning in Human Decision Making
(2007) and Critical Thinking and Clinical Judgment in the Health Sciences (2008). From 1988 through 1990 Dr.
Facione was the principal investigator for the American Philosophical Association research project which culminated
in the Delphi Report – An Expert Consensus Conceptualization of Critical Thinking. The executive summary of that
report is available free of charge from Insight Assessment. His email is pfacione@measuredreasons.com Visit his
website www.measuredreasons.com and the Insight Assessment website, www.insightassessment.com
Facione, PA, “Critical Thinking: What It is and Why it Counts” 2011 update Page 28
Developing critical thinking Page 1 of 3
It means not taking what you hear or read at face value, but using your critical faculties to weigh up the
evidence, and considering the implications and conclusions of what the writer is saying.
Imagine two situations. On the first, you are on a country walk and you come across a notice which tells
you not to attempt to climb a fence because of risk of electrocution. Would you pause to consider before
obeying this instruction? On the other hand, suppose you were to receive a letter from a local farmer
announcing that he proposed to put up an electric fence to protect a certain field. In this case, would you
not be more likely to think about his reasons for doing so and what the implications would be for you and
your family? In the first case, you are thinking reactively and in the second, you are thinking critically.
An allied skill is the ability to analyse – that is, to read or listen for the following points:
The key to critical thinking is to develop an impersonal approach which looks at arguments and facts and
which lays aside personal views and feelings. This is because academic discourse is based according to
key principles which are described as follows by Northedge (2005):
Critical and analytical thinking should be applied at all points in academic study - to selecting information,
reading, writing, speaking and listening. Of these, learning to read and evaluate information critically is
perhaps the most important skill, which if acquired can then be applied to other areas.
The first stage in reading critically is to exercise care in the information you use - how trustworthy is it? For
printed material, consider:
• For books, who is the publisher? Is it a reputable academic publisher? Is the book part of a series
(in which case it will also have another layer of ‘quality control’, from the Series Editor).
• For journal articles, does the article appear in an academic journal? (Your tutor should be able to
tell you what the leading journals are in your field.)
• For both, who is the author and does he or she come from a respectable academic organization?
• How recent it the publication date, and are you using the latest edition of a textbook?
http://0-www.emeraldinsight.com.wam.city.ac.uk/learning/study_skills/skills/critical_... 04/11/2010
Developing critical thinking Page 2 of 3
Particular care needs to be exercised when using information from the Internet. This will be the topic of
another article on this site, but you need to consider relevance and in particular:
• What is its source? Is it from a commercial or academic organization, and if the latter, is it from well
-known one? (For example, when I looked up ‘critical thinking’ I got a lot of commercial sites who
were trying to sell particular services such as software or consultancy.)
• Is it written in an academic style, with references,
substantiated claims etc.?
• There are many journals which are published on the Internet. Not all of these are subject to the
process of peer review, which involves the content being checked by people of standing.
• When was it posted/updated?
Reading critcally
• Are the reasons sufficient, and are they valid to the argument, in other words do they support it, or
would it be possible to draw other conclusions from them?
• Does the author develop the argument in a logical and coherent fashion, i.e. premise/point A/point
B/conclusion, avoiding confusing breaks in the logical flow?
• Is the author’s logic always valid, or does he/she draw arguments from false premises, or are there
flaws in the reasoning assuming a causal connection where none is justifiable or generalizing from
too few examples?
• Is the author’s style objective, or does he/she use emotive language, designed to get the reader’s
sympathy, for example, words or phrases such as cruel, inhuman, Golden Age?
• What is it – statistics, surveys, case studies, findings from experiment are all examples of evidence
that may be presented.
• Is it valid? Validity may be affected by external criteria such as the source (for example an article
from an academic journal is likely to be more reliable than one from a newspaper) or by the
particular bias of the party concerned (for example if a women’s hospital is resisting closure, look
carefully at evidence of other women’s services in the area). You should also examine the intrinsic
qualities of the evidence, for example how recent are case studies? How robust are experiments?
How large and representative is the survey? Is evidence anecdotal (for example, stories of one
person being cured from a particular treatment are less impressive than clinical trials)?
4. What are the conclusions, and are they supported by the evidence? It may be possible to present what
appears to be flawless research, which may yet not justify the conclusions. A good example here is the
ongoing debate on child care, and whether mothers are better off at home looking after their children
themselves. In the 1950s, John Bowlby presented good arguments why mothers should stay at home,
which was subsequently reputed by later researchers, whilst the stay at home argument is now making a
return. The studies themselves may not present valid evidence and need to be seen against other trends,
such as the need to ensure full male employment after the war, the rise of feminism, and women’s desire
for choice over whether or not they work.
5. What are the alternatives? Look at the author’s work from different perspectives - how does the view
presented differ from others? Does the author have a particular agenda, revealed (as in the case of a
particular view of research, see above) or hidden (for example, particular reasons, political or other, for
arguing a case)? Does the evidence really lead to the conclusions offered or might there be other
explanations (see the example in 4 above).
Air traffic in the Southeast of the country is becoming increasingly great, there are
three airports and the plan is to expand the airport at Lutwick to ease congestion at
the other airports and help with the expected tripling of demand by 2030 (1). Pollution
from aircraft is one of the biggest problems of our times. (2) A recent survey of local
residents by the Lutwick Times showed that 60 per cent of local residents would
oppose the plan. (3) Lutwick was severely bombed in the war (4), and has suffered
enough (5) without the further incursion of gridlocked motorways which would result
from the enlarged airport. (6) Forecasts show that many more houses will be affected
by noise pollution than with other airports, that the projected increase in jobs is
http://0-www.emeraldinsight.com.wam.city.ac.uk/learning/study_skills/skills/critical_... 04/11/2010
Developing critical thinking Page 3 of 3
dubious, and that flooding may result from hard surface run-off (7). We must therefore
oppose the plans and press for an environmental impact tax on aviation (8).
1. The first sentence is a descriptive statement: unclear what the author’s premise is.
2. The second sentence is a non sequitur - there is no explicit link with the first statement.
3. A survey is a piece of evidence, but how reliable is the source, 60 per cent of what number and
when was the survey carried out?
4. That Lutwick was bombed during the war is a non sequitur, and is not essential to the case.
5. This is emotive language.
6. This assumes a causal connection between the enlarged airport and congested motorways, but
there may be other reasons why motorways are congested.
7. These statements constitute evidence, but they are not substantiated, referenced or quantified.
What forecasts, how many more houses over what area, where will the flooding be and why will it
result from hard surface run-off?
8. The conclusion is clearly stated, but its first part (that plans must be opposed) clearly shows the
bias of the writer, and the second part (the environmental impact tax) does not necessarily follow
from the evidence, which is specific to a particular location. In order to arrive at that conclusion it is
necessary to report evidence on a more general link between flights and the environment, to argue
for the practicality of the tax looking at opposing arguments such as would it price air travel out of
some people’s reach and would this be a bad thing?
Writing Critically
Much has been written elsewhere on this site about the writing process, so we will only make brief
reference here. Planning is the key: if you organize your ideas carefully in your plan, you will be clearer
what you have to write.
You need to employ the same critical judgement to your own writing as you do to that of other people,
although it can be more difficult to assess your own work! In particular:
• Check your line of reasoning is clear – start out by stating what you propose to do, organize your
information in a logical pattern, and reach a clear and substantiated conclusion.
• Ensure that the evidence you use is valid according to the criteria set out above, under Reading
critically.
• Be aware of the difference between descriptive writing, which tells a story, using statements,
explanation and lists etc., and analytical writing, which presents an argument, giving reasons,
weighing up information, and drawing conclusions.
Much of learning is carried out by dialogue, and by tossing ideas around, but you can’t expect yourself or
others always to substantiate with the same degree of rigour as in writing! However, listen out in yourself
and other people for inconsistencies and contradictions; if you are in a seminar, notice how ideas are
‘developed’ through dialogue, how your ideas fit in or contradict with those of others, etc. Be as prepared
to ask questions as you are to listen, for example if someone offers a point of view about a particular
author or text, don’t be afraid to challenge them to substantiate their claim. You will also need to put
forward a reasoned argument, which will help develop your thinking skills, particularly as verbal debate
proceeds at a more rapid pace than writing or reading, which are mostly solitary activities. Remember, too,
that you have extra ‘data’ in the form of body language - does the latter fit in with what they are saying, or
are you noticing contradictory signals, for example, a raised eyebrow?
Reference
• Northedge, A. (2005), The Good Study Guide, Open University Press, Buckingham, UK
http://0-www.emeraldinsight.com.wam.city.ac.uk/learning/study_skills/skills/critical_... 04/11/2010
http://logical-critical-thinking.com/critical-thinking/10-ways-or-techniques-to-help-improve-critical-thinking/
Corresponding author
Correspondence to: William J. Sutherland
Comments (19 comments)
Tal Galili • 2013-12-08 06:39 PM
While the article offer nice simplifications of many important topics in understanding scientific
claims, the description of significance (e.g: p-value), is severely lacking. To quote Wikipedia's
section "misunderstandings of p-value" on this: 1) The p-value is not the probability that the null
hypothesis is true, nor is it the probability that the alternative hypothesis is false – it is not
connected to either of these. In fact, frequentist statistics does not, and cannot, attach
probabilities to hypotheses. Comparison of Bayesian and classical approaches shows that a p-
value can be very close to zero while the posterior probability of the null is very close to unity (if
there is no alternative hypothesis with a large enough a priori probability and which would
explain the results more easily). This is Lindley's paradox. But there are also a priori probability
distributions where the posterior probability and the p-value have similar or equal values. 2) The
p-value is not the probability that a finding is "merely a fluke." As calculating the p-value is based
on the assumption that every finding is a fluke (that is, the product of chance alone), it cannot be
used to gauge the probability of a finding being true. The p-value is the chance of obtaining the
findings we got (or more extreme) if the null hypothesis is true. https://en.wikipedia.org/wiki/P-
value#Misunderstandings
Patrick Elliott • 2013-12-06 06:15 AM
I would add in one more: #21: Understand what placebo actually does mean. Placebo isn't "does
nothing". Rather, it can be very effective for pain, and even has been shown to have other
physiological effects, which can even include improvements in immune responses (though, this
and other effects are not as great as for pain). The reason being that its literally the "mind over
matter" effect. The default assumption is, among those that don't believe in "mind over matter"
is that all this stuff kind of runs on automatic, and the brain has no control over it. Ironically, the
"true believers" in mind over matter seem to think that the mind (not the brain, duh...) has some
sort of mystical, super-power to effect everything, if you just spend enough time meditating. The
reality is a bit more.. complicated, and vastly less mystical. Pain needs to react on a "huge" scale
to mental state. For a wounded animal, the ability to, unconsciously, fade out "existing" pain
from their wound, such as a badly injured leg, for the duration of a fight, or escape, can be life or
death. Even the ability to do things that are familiar, and thus comforting, and thus reduce the
pain, might be critical to handling the stress of such injuries. In short, there is already a
mechanism there, which can **selectively** knock out pain, and humans have way more
complex, not just conscious perceptions, but layers of sub-conscious responses. Its hardly a
surprise that, some where in there, is a way for a placebo to selectively have a major effect on
pain, even over long periods of time (like for acupuncture, a practice based on the very silly idea
that the body contains nine rivers of power in it, with tributaries, just like ancient China, where it
was invented). Obviously, state of mind "might" be useful to enhance, or suppress, intentionally
or otherwise, immunity, digestion, and a whole lot of other semi-automatic things. Things that,
when upset by certain conditions, or even conditioned responses (I feel like running away every
time someone gets sick at work, and I am the one "assigned" to clean the floors that day, for
example), which may have perfectly reasonable causes (if everyone else is sick, it might be
something you ate, so maybe you need to be sick too?, for example), but which can be
unlearned, or otherwise modified by experience. So, rather than placebo being "does nothing", a
more accurate description is, "Takes advantage of preexisting triggers and experience, to cause
**limited** changes in what the body is doing." If you think about that for a moment, you can
see why its huge headache for medicine research, and a major problem for the credibility of
people selling things that are *not* tested as carefully. In both cases, trust in the person giving it
to you, trust that it will work, belief in the method being employed, and other factors, will all
"trigger" this placebo effect. Someone taking "big pharma" drugs might feel sicker taking them,
but better taking something given to them by a guru, but its the lack of trust in the former, and
the unreasonable trust in the later, as well as the products being given, which is having the
effect, not the actual drugs themselves. Its a major pain for drug makers, to actually get provable
results, which are not **purely** due to everyone in the study believing in the medicine, and the
doctor. For the guru/herbalist, etc.... its less of a problem, since the people going to them are
probably completely certain it will help them, and trust them, even when, from the stand point
of actually treating the disease, it might be totally worthless. And, that can kill people, who
would have otherwise *been* cured, by proven medications. Note: this isn't a denial of there
being problems with Big Pharma, or with medicine as its practiced. The former is about money
(and so it big-CAM, which is quickly creeping up, closer and closer to being worth as many
billions), while doctors are all too often paid to offer new drugs, instead of proven ones, paid by
number of procedures, instead of based on outcomes, and almost everything we rely on that
*does* undeniably work, is no longer even made by Big Pharma, but by smaller, less exacting,
more prone to errors, shortfalls, or price gouging, because those, often life saving drugs, are, "no
longer cost effective to produce." The same reason why some promising drugs, for rare,
sometimes even fatal, but possibly treatable, manageable, or even curable, diseases, never make
it to market at all. Its undeniable we have problems with the industry, but.. its also undeniable
that the only reason that doctors "generally" are not allowed to push placebos, but insurance
companies are willing to pay for unlicensed, not recognized by the AMA, "specialists" for them
instead, is because insurance companies can't have their licenses pulled for lying to their patients
about the chances of something curing them. Instead, they just decide which one will cost less to
give you, something that isn't proven, but a lot of people like, but it vastly cheaper, or something
that might not help you anyway, but makes the doctor a lot of money, involves lots of ,
sometimes, unnecessary and inconclusive, lab work and tests, and then, might not help you, if
you are ill enough, any more than the placebo would have. Of course the insurance companies
are willing to pay for that as an option...
Mike Taylor • 2013-12-03 01:33 PM
See also the excellent article in response by Chris Tyler (director of the UK's Parliamentary Office
of Science and Technology) on Top 20 things scientists need to know about policy-making. These
two articles make a very helpful complementary pair.
Edward Powell • 2013-11-30 07:19 PM
To these twenty, I would add two more: 21) The predictive ability of computer simulations is
extremely limited, and should only be relied on when substantial and detailed agreement of the
simulation results with experimental data is achieved, *and* when a detailed and thorough
understanding of the elements of uncertainty in the models and simulations is achieved. A true
scientist tries his best to falsify, disprove, or even "break" his models and simulations in any
possible way, before relying on the results. John von Neuman said, "With four parameters I can
fit an elephant, and with five I can make him wiggle his trunk." Any simulation with *any* free
parameters is suspect, and detailed, conclusive, and scientifically justified causal relationships
must be well understood for each and every free parameter before a simulation can even begin
to be considered reliable. 22) No scientist shall, under any circumstances, refer to a simulation
run as an "experiment". Simulations are not experiments, only data taken from actual reality-
based experiments can be called real data.
Oliver H. • 2013-12-07 06:22 PM
What is "reality-based"? The vast majority of basic research is not "reality-based", but based on
lab experiments that do not present reality but are actively controlled to keep certain factors fix
that would vary in "reality", so as to isolate an effect. Tissue culture is nothing BUT a simulation
as to what happens in actual tissue.
Chris Atkins • 2013-11-28 08:30 PM
This is an excellent article and a subject close to my heart. It could (and should) be used with
secondary (high) school students, particularly (but not exclusively) of science. For that purpose
each 'tip' would benefit from a short (one page) article which describes the issue in more detail
with one or two examples to bring it to life.
Lee De Cola • 2013-11-27 05:48 PM
this is a great list, but to be widely read it needs to be shortened...i've tried and come up with 3
groupings: psychology (of scientists), experiments, and statistics - could we get along with 3 or 4
tips in each group?
James Scanlan • 2013-11-27 05:29 PM
The is a useful list in many respects. But it overlooks what may be the most serious problem in
the interpretation of data on differences between outcome rates. Virtually all efforts to interpret
data ongroup differences in outcome rates, including data on rates of treated and control
subjects in clinical trials, are undermined by failure to recognize patterns by which standard
measures of differences between outcome rates tend to be systematically affected by the
prevalence of an outcome. The most notable of these patterns is that by which the rarer an
outcome the greater tends to be the relative difference in experiencing it and the smaller tends
to be the relative difference in avoiding it. By way of example pertinent to item 20 on the list,
lowering test cutoffs (or generally improving test performance) tends to reduce relative
differences between rates at which two groups pass a test while increasing relative differences
rates at which they fail the test. Among other examples, reducing poverty tends to increase
relative differences in poverty rates while reducing relative differences in rates of avoiding
poverty; improving health tends to increase relative differences in mortality and other adverse
health outcomes while reducing relative differences in survival and other favorable health
outcomes; improving healthcare tends to reduce relative differences in receipt of appropriate
care while increasing relative differences in rates of failing to receive appropriate care; reducing
adverse lending outcomes or school suspension rates tends to increase relative difference in
experiencing those outcomes while reducing relative differences in avoiding those outcomes. It
is not possible to interpret data on group differences or advise policy makers on such issues
without knowing these things. Yet very few people interpreting data on differences in outcome
rates know that it is even possible for the two relative differences to change in opposite
directions, much less that they tend to do so systematically. A number of references explaining
these and related patterns and their implications in varied contexts are listed below.
Reference 1 provides a fairly succinct explanation of the pattern of relative differences described
above and does so in the course of explaining that, as a result of the failure to understand the
pattern, the US government encourages lenders and schools to take actions that make it more
likely that the government will sue them for discrimination. Reference 8 explains some of the
clinical implications of the failure to understand the pattern, explaining as well that the rate ratio
mentioned as a measure of effect in item 20 of the list is not merely a flawed measure of effect,
but an illogical one. References 9 and 10 are discussions of the above-described pattern of
relative differences by other persons. The latter reference observes that governments that
ignore the pattern “run the risk of guaranteeing failure, largely for conceptual and
methodological reasons rather than social welfare reasons.” The observation was focused on the
meeting of health inequalities reduction goals cast in relative terms. But, as reflected in
references 1 through 8, failure to understand the pattern, and relative patterns by which
measures tend to be affected by the prevalence of an outcome, undermines society’s
understanding of a great many things.
1. “Misunderstanding of Statistics Leads to Misguided Law Enforcement Policies” (Amstat News,
Dec. 2012):
http://magazine.amstat.org/blog/2012/12/01/misguided-law-enforcement/
2. “Measuring Health and Healthcare Disparities,” Federal Committee on Statistical Methodology
2013 Research Conference, Nov. 5-7, 2013.
http://jpscanlan.com/images/2013_Fed_Comm_on_Stat_Meth_paper.pdf
http://jpscanlan.com/images/2013_FCSM_Presentation.ppt
3. “The Mismeasure of Discrimination,” Faculty Workshop at the University of Kansas School of
Law, Sept. 20, 2013:
http://jpscanlan.com/images/Univ_Kansas_School_of_Law_Faculty_Workshop_Paper.pdf
http://jpscanlan.com/images/University_of_Kansas_School_of_Law_Workshop.pdf
4. “The Mismeasure of Group Differences in the Law and the Social and Medical Sciences,”
Applied Statistics Workshop at the Institute for Quantitative Social Science at Harvard University,
Oct. 17, 2012:
http://jpscanlan.com/images/Harvard_Applied_Statistic_Workshop.ppt
5. “Can We Actually Measure Health Disparities?” (Chance, Spring 2006):
http://www.jpscanlan.com/images/Can_We_Actually_Measure_Health_Disparities.pdf
6. “The Misinterpretation of Health Inequalities in the United Kingdom,” British Society for
Populations Studies Conference 2006, Sept. 18-20, 2006:
http://www.jpscanlan.com/images/BSPS_2006_Complete_Paper.pdf
7. “Race and Mortality” (Society, Jan./Feb. 2000):
http://www.jpscanlan.com/images/Race_and_Mortality.pdf
8. Subgroup Effects subpage of the Scanlan’s Rule page of jpscanlan.com
http://www.jpscanlan.com/scanlansrule/subgroupeffects.html
9. Lambert PJ, Subramanian S. Disparities in Socio-Economic outcomes: Some positive
propositions and their normative implications. Society for the Study of Economic Inequality
Working Paper Series, ECINEQ WP 2012 – 281:
http://www.ecineq.org/milano/WP/ECINEQ2012-281.pdf
10. Bauld L, Day P, Judge K. Off target: A critical review of setting goals for reducing health
inequalities in the United Kingdom. Int J Health Serv 2008;38(3):439-454:
http://baywood.metapress.com/app/home/contribution.asp?referrer=parent&backto=issue,4,1
1;journal,7,157;linkingpublicationresults,1:300313,1
Benjamin Allen • 2013-11-26 11:04 PM
An excellent article - well done! Translation from science into policy is made even more difficult
when scientists themselves produce and promote unreliable science. Sutherland et al's 20 tips
are well worth the read. For an ecological example that epitomises many of the 20 points they
raise, see 'http://www.sciencedirect.com/science/article/pii/S0006320712005022'.
Douglas Duncan • 2013-11-26 06:38 PM
Almost all those politicians have been to university, but these aspects of science are rarely
discussed with non-science majors there. They need to be! I have been doing so for about 6
years, and have published the effects it has on students. Remember, these people have probably
never been to a scientific meeting; they have little idea how scientists interact, and they sure
don't learn that in a classroom, because most of us behave entirely differently there than we do
with colleagues. The way scientists interact, how they arrive at consensus, what causes them to
doubt (even the basic distinction between peer reviewed and non-peer reviewed publication)
are things they have never encountered (according to our interviews of non-science majors).
Anyone is welcome to use my curriculum, which is here (along with reference to the published
results): http://casa.colorado.edu/~dduncan/pseudoscience/
Bart Penders • 2013-11-26 09:44 AM
The public credibility of science and the trust non-scientists place in scientific claims and the
institution of science is not primarily derived from the methodological quality of scientific
inquiry. The list above is worthwhile to determine that quality (or any threats to it), but not to
help establish the public credibility (or trust). Let me quote Harvard historian of science Steven
Shapin who wrote, eloquently: "When King Lear decided to take early retirement, he announced
his intention to divide up the kingdom among his three daughters, each to get a share
proportioned to the genuine love she bore him. Each is asked to testify to her love. For Goneril
and Regan that presents no problem, and both use the oily art of rhetoric to good effect.
Cordelia, however, trusts to the authenticity of her love and says nothing more than the simple
truth. For Lear this will not do. Truth is her dower but credibility has she none. Cordelia, we
should understand, was a modernist methodologist. The credibility and the validity of a
proposition ought to be one and the same. Truth shines by its own light. And those claims that
need lubrication by the oily art thereby give warning that they are not true". (Shapin 1995).
Cordelia acts from the conviction that contained in her claims is the truth and that it can be
accessed by those who hear it. The credibility of her claim is evident to herself - but only to
herself. For a scientist to convince non-scientists, persuading them to see the truth as (s)he does,
requires - next to that truth - a construction of credibility. Scientists have very strict and shared
rules to establish the credibility of their claims, these include notions of peer review, citations,
transparant methodologies and much more. They are part of the well-known academic
credibilising strategy in which the work scientists do to make their data, arguments, theories or
claims become stable and uncontested truths (i.e. to achieve absolute credibility). "The same
work enables them to conduct further research by strengthening their reputation and attracting
new funding. The process of gathering credibility is never-ending and cyclical: it drives the
credibility cycle. Important in this process are moments of conversion, or translation. Latour and
Woolgar show how time, money and effort are translated into data; data is translated into
arguments, which are subsequently written down in publications. Peer scientists will read or cite
them, resulting in recognition for their claim or argument. This recognition can be mobilised to
support new funding requests. The new funding, translated into new personnel, projects or
machineries, will produce new data, continuing the cycle. Every translation that takes place,
contributes to the credibility of a given scientific claim, which extends to the researchers,
laboratory or institute making the claim. If the ‘scientific wheel of credibility’ continues to turn, it
will uphold or heighten the credibility of claims and claimants. Academic conventions, such as
scientific publications, citations and peer review are part of this credibility cycle. The credibility
cycle as a strategy to construct credibility is powerful only within the scientific community:
outside the university walls, peer review and citation cultures are relatively unknown and cannot
accrue similar credibility." (Penders 2013). Non-scientists (as well as scientists, when it comes to
non-scientific claims) use other criteria, standards and strategies to evaluate the credibility of
claims. What matters most, in real life, are the shape or form (visually, or rhetorically) of a
message, the medium or location of a claim (e.g. quality newspaper vs. casual water-cooler
chat), the degree in which a claim specifically addresses a problem (or question) the audience
identifies with, the source of the claim (a trusted colleague or friend vs. an unknown online
forum) and the relationship between the claim and the audience. The route a scientific claim
takes before it reaches any given policy maker, may matter more than its content. Who whispers
it into her ear may matter more than its content and the dominant political climate may matter
more than the content of the claim. The authors wisely acknowledge that they "are not so naive
as to believe that improved policy decisions will automatically follow". The reason for that is that
any scientific claim wildly varies in its credibility over time, location and much more. That
credibility is not derived from its content, but it is the reason why a claim is takes seriously, acted
upon, insert into or excluded from policy.
Gavin Cawley • 2013-11-25 11:35 AM
For policy related issues, the tip I would add would be: Always interpret a scientific finding in
the way that provides the least support for ones current position. This helps to guard against
the sort of confirmation biases etc. to which we are all susceptible. If the finding genuinely
supports your position, it will still do so under its least favourable interpretation, and adopting
this interpretation will demonstrate your self-skepticism.
Anand Ramanathan • 2013-11-21 07:11 PM
The article is good about warning policy makers about new scientific claims. However, it is a little
worrisome that the same logic can be applied to any science to reject established results such as
the warming over the last few decades or the effect of smoking. I would add one more point.
Science typically does not change overnight and results that have been tested and confirmed
over years (or decades) are unlikely to be wrong. Any new result that seeks to overthrow some
bit of well-established science should be backed by a much higher standard of proof, such as
independent replications, or carefully controlled studies with large sample sizes.
Gavin Cawley • 2013-11-25 11:42 AM
In the case of the warming of the last few decades, I would say this is addressed by the points
"Separate no effect from non-significance" and "Data can be dredged or cherry picked" as such
arguments are normally based on the lack of a statistically significant warming trend since some
cherry picked start (e.g. 1998 El-Nino) and ignore the statistical power of the test. A test for the
existence of a change in the underlying rate of warming would likely also give a non-significant
result.
Christopher Tong • 2013-11-21 06:37 AM
A few other tips might include an awareness of Simpson's Paradox, the Curse of Dimensionality,
and the ease with which data models can overfit one data set and fail to generalize to others.
The distinction between exploratory and confirmatory findings should also be emphasized.
Exploratory research seeks to generate new hypotheses, and confirmatory research aims to
evaluate pre-specified hypotheses. Sometimes a single study will have both confirmatory and
exploratory findings, as follows. The confirmatory claims are pre-specified before the data was
collected, and their consistency with the subsequent data are then evaluated. Exploratory
findings are tentative findings of other, unanticipated patterns in the same data. The
epistemological status of both kinds of findings must be kept distinct. Finally, data analysis
methods must be chosen for fitness for purpose, and not by pattern matching, as is too often the
case.
Brad Louis • 2013-11-21 03:40 AM
Great list. Can I suggest a few more? We don't know everything (Science is never settled. Any
scientist can be wrong.) - that's why we keep investing in scientific research. We will know more
in another hundred years, by which time a lot of what we think we know now will be proven to
be nonsense. Proper experiments produce the most robust scientific evidence - good
experimental design can eliminate many of the problems mentioned in earlier tips.
Observational studies and computer models are not a patch on proper experiments. Science
funding supports a process not an outcome - well done science doesn't pretend to know the
outcome of research, diversity of research perspectives is a good thing.
Ian Campbell • 2013-11-20 11:00 PM
This article provides a valuable summary of issues that need to be appreciated in interpreting
science. However, in the environmental decision making arena, the science is never sole factor to
be considered - there is always a value judgement as well. Science tells us the consequences of
greenhouse gas emissions, or the likelihood that logging a particular area of forest will cause the
extinction of a particular species, but then we have to make a value judgement. Do I value the
profits I make from my power station more than the contribution it makes to global warming?
Do I value the timber products I can get by logging this forest more than the possum that will
become extinct? As value judgements these decisions should be informed by the science, but are
made by politicians who are expected to reflect the values of their society. Politicians are always
averse to making value judgement calls that will be unpopular with a substantial proportion of
their community, or an influential group within the community, so they often try to deflect that
responsibility. Two common deflection techniques are appeals to (often dodgy) short term
economics (it would be too expensive, cost too many jobs), and raising doubts about the science
either by citing poor science, or by suggesting that there is a lack of scientific consensus on the
issue. It is difficult to tell whether a true lack of scientific understanding, as addressed in the
article, or a misunderstanding of convenience to deflect criticism is a greater problem.
Fred Sachs • 2013-11-21 05:16 PM
The article does emphasize some important issues but at the core is how one might evaluate the
contributions of all these factors. The probability of success in a project cannot be calculated and
a legislator would have to assign a weighting factor to each of the 20 criteria, let alone the
societal consequences, to decide on the level of support and it can't be done. Real answers like
this depend on numbers and we don't have the information to generate the numbers. I suspect
that we are going to have to put up with intuition on the part of the policy makers. I have a few
specific issues. One is that 5% significance is an entirely subjective choice by the researcher and
has no significance if the distribution is not Gaussian. That should be the first test. Even at 5%
there is a 1 in 20 chance that the hypothesis is wrong. Does this data represent that time?
Especially in clinical data one should know the probability of getting better compared to the
probability of getting worse. You should not do a one sided test to ask if subjects got better and
ignore the times they got worse. The outcome should also be weighted by the payoff; a number
not easily evaluated. Did the positive patients feel better while the negative ones died? Finally,
for the cause and effect issue I like to bring up something seen a billion times a day: the roosters
crowed and then the sun came up; clearly the roosters caused sunrise.