High Quality
Open the downloaded document, and select print from the file menu (PDF reader required).
Evaluating Intelligence
kwheatonPage 22/8/2009
Evaluating Intelligence
Evaluating intelligence is tricky.Really tricky.Sherman Kent, one of the foremost early thinkers regarding the analytic process in theUS national security intelligence communitywrote in 1976, “Few things are asked theestimator more often than "How good is your batting average?" No question could bemore legitimate--and none could be harder to answer.” So difficult was the question thatKent reports not only the failure of a three year effort in the 50’s to establish the validityof various National Intelligence Estimates but also the immense relief among theanalysts in the Office of National Estimates (forerunner of the National IntelligenceCouncil
) when the CIA“let the enterprise peter out.”Unfortunately for intelligence professionals, the decisionmakers that intelligencesupports have no such difficulty evaluating the intelligence they receive. They routinelyand publicly find intelligence to be “wrong” or lacking in some significant respect.Abbot Smith, writing for
Studies In Intelligence
in 1969, cataloged many of these errorsin
.The list of failures at the timeincluded the development of the Soviet H-bomb, the Soviet invasions of Hungary andCzechoslovakia, the Cuban Missile Crisis and the Missile Gap. TheTet Offensive,the
collapse of the Soviet Union and theWeapons of Mass Destruction fiasco in Iraqwould
soon be added to the list of widely recognized (at least by decisionmakers) “intelligencefailures”.
1
This article originated as a series of posts on my blog, Sources and Methods(www.sourcesandmethods.blogspot.com
). This form of “experimental scholarship” -- or using themedium of the internet and the vehicle of the blog as a way to put my research online—provides for moreor less real-time peer review. Earlier examples of this genre include:A Wiki Is Like A Room...,The
Revolution Begins On Page 5, What Is Intelligence?and What Do Words Of Estimative Probability
Mean?. Given its origin and the fact that it is stored electronically in the ISA archive, I will retain thehyperlinks as a form of citation.In addition, astute readers will note that some of what I write here I have previously discussed in other places, most notably in an article written with my long-time collaborator,Diane Chido, for Competitive
Intelligence Magazineand in a chapter of our book on Structured Analysis Of Competing Hypotheses(written with Diane, Katrina Altman, Rick Seward and Jim Kelly). Diane and the others clearly deservefull credit for their contribution to this current iteration of my thinking on this topic.
kwheatonPage 32/8/2009 Nor was the US the only intelligence community to suffer such indignities. The Sovietshad their Operation RYAN, the Israelis their Yom Kippur War and the British their
Falklands Island. In each case, after the fact, senior government officials, the press andordinary citizens alike pinned the black rose of failure on their respective intelligencecommunities.To be honest, in some cases, the intelligence organization in question deserved thecriticism but, in many cases, it did not -- or at least not the full measure of fault itreceived. However, whether the blame was earned or not, in the aftermath of each of these cases, commissions were duly summoned, investigations into the causes of thefailure examined, recommendations made and changes, to one degree or another,ratified regarding the way intelligence was to be done in the future.While much of the record is still out of the public eye, I suspect it is safe to say thatintelligence successes rarely received such lavish attention.Why do intelligence professionals find intelligence so difficult, indeed impossible, toevaluate while decisionmakers do so routinely? Is there a practical model for thinkingabout the problem of evaluating intelligence? What are the logical consequences for both intelligence professionals and decisionmakers that derive from this model? Finally,is there a way to test the model using real world data?I intend to attempt to answer all of these questions but first I need to tell you a story…
A Tale Of Two Weathermen
I want to tell you a story about two weathermen; one good, competent and diligent andone bad, stupid and lazy. Why weathermen? Well, in the first place, they are
not
intelligence analysts, so I will not have to concern myself with all the meaninglessdistinctions that might arise if I use a real example. In the second place, they are enoughlike intelligence analysts that the lessons derived from this thought experiment – sorry, Imean “story” – will remain meaningful in the intelligence domain.Imagine first the good weatherman and imagine that he only knows one rule: If it issunny outside today, then it is likely to be sunny tomorrow (I have no idea why he onlyknows one rule. Maybe he just got hired.
Maybe he hasn’t finished weatherman schoolyet.
Whatever the reason, this is the only rule he knows). While the weatherman onlyknows this one rule, it is a good rule and has consistently been shown to be correct.His boss comes along and asks him what the weather is going to be like tomorrow. Thegood weatherman remembers his rule, looks outside and sees sun. He tells the boss, “Itis likely to be sunny tomorrow.”
Add a Comment