UNCLASSIFIED

Methodologies for Sorting

Through the Chaff

Presentation to: DHS/IAIP

27 January 2005

Pherson Associates, LLC • Email: furson@aol.com

UNCLASSIFIED
 UNCLASSIFIED

Five Approaches

Recognizing the Good Stuff:

• Using What If? Analysis and Outside-In Thinking
to generate generic Indicators or Signposts
• Empirically-derived Checklists
Eliminating the Bad Stuff:
• Deception Detection
• Analysis of Competing Hypotheses
UNCLASSIFIED
 UNCLASSIFIED

Recognizing
the Good Stuff

UNCLASSIFIED
 UNCLASSIFIED

What If? Analysis

Definition: Taking as a given that an event has occurred

and then explaining how it came about.

Example:
• Three years ago, terrorists just tried to crash a plane
into the Eiffel tower. What if we had asked ourselves
then: “Would they do something similar in the United
States? How would they pull this off?”

UNCLASSIFIED
 UNCLASSIFIED

What If? Analysis

Value Added:
• Focuses attention on all the things that must fall
into place for a low probability--but high impact--
event to actually occur.
• Alerts you to potentially useful reporting that you
might have ignored or would have regarded as
noise.
UNCLASSIFIED
 UNCLASSIFIED

What If? Analysis

The Method:
Assume that what might be the case, is the case.
• Develop a chain of argumentation based on both evidence
and logic explaining how this outcome actually could have
come about. This is called “thinking backwards.”
• Generate a list of signposts or “observables” that would
indicate that this outcome is coming about.
• Monitor the traffic for any evidence that relates to the
signposts or indicators.

UNCLASSIFIED
 UNCLASSIFIED

Outside-In Thinking
Definition: A technique for identifying the full range of
forces, factors, and trends that would indirectly shape
an issue.

Examples:
• In brainstorming how al-Qaeda elements are
communicating with each other, are there any
technological trends or new technologies that we need
to consider (eg., use of “unsent” email messages, MP3,
or IPods)?

UNCLASSIFIED
 UNCLASSIFIED

Inside-Out versus
Outside-In Thinking
H ow W e W ork:
O u t s id e - I n T h in k in g

T h e W o r ld

T he Industry

O ur
O r g a n iz a t io n
H ig h
I n f lu e n c e
• • MM aar rk keet t s si zi zee, ,
g gr ro owwt thh, , aan ndd K ey F actors:
v vo ol al at ti li il ti ty y S o m e I n f lu e n c e
• • CCu us st to omm eer rs s •• SSo oc ci ai al l
• • CCo omm p peet ti ti to or rs s Key F orces: •• TTeec chhn no ol ol og gi ci ca al l
• • SSu up pp pl il ei er rs s L it t le o r N o I n f lu e n c e •• EEc co on no ommi ci c
• • OO wwn neer rs s •• EEn nv vi ri ro on nmm e en nt taal l
• • CCo omm mmu un ni ti ti ei es s •• P Po ol il ti ti ci ca al l
• • P Paar rt tn neer rs s

UNCLASSIFIED
 UNCLASSIFIED

Competing Approaches
Question: How do we assess a terrorist threat?
Inside-Out Approach:
• Monitor reporting for tipoffs/lead information.
• Extrapolate patterns from reporting trends.
Outside-In Approach:
• Identify relevant global trends.
• Assess how they might affect when, where, and how
a terrorist might launch an attack.
UNCLASSIFIED
 UNCLASSIFIED

Outside-In Thinking
The Method:
• Generate a generic description of the problem at hand.
• List all the factors (social, technological, economic) that could
have an impact (the subject usually has little influence over
these factors but can exploit them).
• Next list the factors over which the subject can exert some
influence (choice of partners, methods of communication,
capability to acquire feedback, etc.).
• Assess how each of these factors could have an impact.
• Look for data that suggests they actually have an impact.

UNCLASSIFIED
 UNCLASSIFIED

Empirically-derived Checklists
The Method:
• Establish categories of data (walk-ins, detainee reports, émigré
reporting, human sources, etc.)
• Review the reporting within each category and establish criteria for
what turned out to be useful or not.
• Develop a rough scale. For example, reporting that turned out to
be useful usually met these criteria; bad reporting often fell into
these boxes, etc.
• Use these lists to rate the utility of incoming reporting.
• Rate the new reporting based on these lists and revise/refine the
lists over time.

UNCLASSIFIED
 UNCLASSIFIED

Eliminating

the Bad Stuff

UNCLASSIFIED
 UNCLASSIFIED

Detecting Deception

Look for deception when:

• Accepting new information would require you to change
your mind, alter a key assumption, or divert significant
resources (protect all apartment buildings or shopping centers).
• Your analysis hinges on a single or key piece of data.
• The terrorists have a great deal to gain, or lose, if you
take a specific action (discount a key source).
• You know they have an effective feedback channel.
(they are likely to learn of your reaction in the press).
UNCLASSIFIED
 UNCLASSIFIED

Tactical Indicators of Deception

• Is the source reliable?

• Does the source have access?
• Is the source vulnerable to control or
manipulation by the terrorists?
• Have the terrorists tried to deceive us in this way
in the past?
UNCLASSIFIED
 UNCLASSIFIED

Tactical Indicators of Deception

• How accurate is the source’s reporting?

– Examine the whole chain of evidence, including translations!
• Does the critical evidence check out?
– The subsource can be more critical than the source.
• Does evidence from one source (HUMINT) conflict with
another source (OSINT)?
• Do other sources of information provide corroborating
evidence?
UNCLASSIFIED
 UNCLASSIFIED

How to Avoid Deception

• Be suspicious if forced to rely on sources who have

not been seen or directly interviewed.
• Try not to rely exclusively on non-material evidence
(verbal intelligence).
• Check all instances in which a source’s reports that initially
appeared correct later turned out to be wrong-- and yet the
source always seemed to offer a good explanation for the
discrepancy.
• Heed the opinions of those closest to the reporting.
• Know the enemy’s limitations as well as his capabilities.
UNCLASSIFIED
 UNCLASSIFIED

Analysis of Competing Hypotheses

Definition:
The identification of a complete set of alternative
hypotheses, the systematic evaluation of data
that is consistent and inconsistent with each
hypothesis, and the rejection of hypotheses
that contain too much inconsistent data.
UNCLASSIFIED
 UNCLASSIFIED

The Value of ACH

ACH helps you overcome three fundamental

analytic traps:
• Selective perception (or coming to closure too quickly) that
usually results from focusing on a single hypothesis.
• A failure to generate—at the outset—a complete set of
alternative hypotheses.
• Focusing on the evidence that tends to confirm rather
than to disconfirm the hypothesis.

UNCLASSIFIED
 UNCLASSIFIED

Analysis of Competing Hypotheses

Advantages:
• Ensures that all the information and argumentation
is evaluated.
• Helps avoid premature closure.
• Highlights the evidence that is most “discriminating”
in making the case.
• Removes the relatively unimportant data from
the equation.

UNCLASSIFIED
 UNCLASSIFIED

ACH: The Eight Step Process

1) Identify the possible hypotheses to be

considered. (use brainstorming techniques)
2) List significant evidence and arguments
for and against each hypothesis.
(include the absence of evidence)
3) Prepare a matrix to analyze the “diagnosticity”

of the evidence.
UNCLASSIFIED
 UNCLASSIFIED

ACH: The Eight Step Process

4) Delete evidence and arguments that have
no diagnostic value. (that support all hypotheses)
5) Assess the relative likelihood of each
hypothesis. (try to refute each hypothesis rather
than confirm it)
6) Determine how sensitive the conclusion is
to just a few critical pieces of evidence.
(would the judgment still stand if the evidence
were wrong?)
UNCLASSIFIED
 UNCLASSIFIED

ACH: The Eight Step Process

7) Report conclusions; establish the

relative likelihood of all hypotheses.
8) Identify milestones for further observation.
(to validate that the most likely hypothesis is
correct or to show that events are taking a
different direction than anticipated)
UNCLASSIFIED