This action might not be possible to undo. Are you sure you want to continue?
Kristan J. Wheaton Associate Professor Department Of Intelligence Studies Mercyhurst College 501 E. 38th St Erie, PA 16546 814 824 2023 firstname.lastname@example.org
Teaching Strategic Intelligence Through Games
Strategic intelligence is considered by intelligence professionals to be the highest form of the analytic art. There is a tremendous demand for this type of intelligence product and a lack of trained professionals capable of producing it. Developing effective teaching methods for this challenging subject, therefore, is an area of ongoing concern for the business, law enforcement and national security intelligence communities. Previous research suggests that a game-based approach to teaching can be successful but no report so far has examined game-based learning in intelligence analysis. I hypothesized that a game-based approach to teaching strategic intelligence analysis would increase learning and improve performance while also increasing student satisfaction with the course. This paper reports the initial results and lessons learned from teaching three full courses (2 undergraduate and one graduate) in strategic intelligence using games as a teaching tool. The paper will begin by examining the unique challenges in teaching strategy, strategic decisionmaking and the types of intelligence that supports those efforts. This will be followed by a short discussion concerning games-based learning generally before examining in detail the specific approaches used in these three courses. This paper will also examine both the learning outcomes and student satisfaction with the courses. Finally, this paper will discuss appropriate course modifications for undergraduate and graduate students when teaching advanced subjects with games-based on the evidence from this study. What is a strategy and what are strategic decisions? "As war is a game through its objective nature, so also is it through its subjective. -- Carl von Clausewitz, On War, Chapter 1.1 While there are many definitions of strategy and strategic decisions, for the purposes of this paper, a strategy is an idea or set of ideas about how to accomplish a goal and strategic decisions are ones which typically put at risk a substantial portion of an entity's disposable resources.
1 A note on the references in this paper: Wherever possible, I have hyperlinked directly to the sources of my information. My intent is to make it easy for readers to rapidly access and evaluate the sources I have used in the preparation of this paper. Where an adequate online source is not available, I have used standard footnotes.
Defining strategy broadly is important. Far too often, strategy is also associated with terms such as "long-term" or "large" and strategic thinking is something accomplished only at corporate headquarters or by generals and kings. Defining strategic decisions in the context of the resources risked by the entity (person or organization) making the decision puts the role of strategy into perspective. Under this definition, it is possible for the exact same decision to be strategic in one case and tactical (or even trivial) in another context. For example, imagine an individual who owns a successful dry cleaning store. Deciding to open up another branch of the store in a different part of town is clearly a strategic decision for this owner. This owner will likely spend many of his disposable resources (time, money, personnel) getting the new branch set up and operating efficiently. The same decision, to open another branch in the same town by the owner of a chain of 7000 dry cleaning stores across the US does not have the same strategic quality as in the first case. In fact, such a decision, in such a large, national organization, might not even be made at the owner’s level. It is entirely possible that such a decision would be pushed down to regional or even sub-regional levels. More importantly, defining strategy in terms of the resources at risk broadens the scope of what arguably constitutes strategic intelligence as well. Under this definition, strategy is not confined to large, powerful organizations. Small businesses, police units and even students can have strategies and, in turn, require strategic intelligence to support their decision-making processes. What is intelligence and what is the role of intelligence in the formulation of strategy? "Well is the Game called great! I was four days a scullion at Quetta, waiting on the wife of the man whose book I stole. And that was part of the Great Game! From the South—God knows how far—came up the Mahratta, playing the Great Game in fear of his life. Now I shall go far and far into the North playing the Great Game. Truly, it runs like a shuttle throughout all Hind. -- Rudyard Kipling, Kim, Chapter 12 For many people, “intelligence” is an even more misunderstood word than “strategy”. Conjuring up images of James Bond or, at least George Smiley, intelligence, for many, is exclusively about secrets and spying. This has been patently untrue for some time, however. As early as 1949, Sherman Kent, the father of intelligence analysis at the Central Intelligence Agency, claimed that as much as 80% of information needed in intelligence work came from open, non-secret, sources.2 In a world that moves 21 exabytes of information via the internet each month, the role of both secrets and spying in intelligence, while still important, is clearly further diminished.
2 Kent, Sherman. Strategic Intelligence for American World Policy. Princeton, NJ: Princeton University Press, 1949.
In recent years, in fact, intelligence has moved from a narrow government function to a broad reaching discipline. Intelligence-led policing is a highly regarded public safety strategy while competitive intelligence has, for many years, been a driver for some successful businesses. Likewise, commercial intelligence agencies, such as IJet and Stratfor, provide intelligence analysis services to private clients. Even non-governmental organizations, such as the Southern Poverty Law Center, have established intelligence organizations to cover issues of interest to them. In addition, intelligence studies programs, like the one at Mercyhurst have sprung up all over the US and abroad. The International Association For Intelligence Education now boasts some 20 colleges and universities among it members. There are several common themes running through these activities that help define intelligence3: First, intelligence is about those organizations and activities that are outside your control but are relevant to your entity's success or failure. In short, intelligence is externally focused. Ever since Moses sent scouts "through the Negev and on into the hill country" of Canaan to see "what the land is like and whether the people who live there are strong or weak, few or many” intelligence has been about "the other guy"; the enemy, the criminal or the competitor. Second, intelligence uses information from all sources and, more than that, most of this information is unstructured. Very few disciplines truly deal with information from all sources. Intelligence analysts, however routinely have to integrate economic, political, military, cultural and other types of data into a forecast for decisionmakers within an organization. In addition, much of the information used by intelligence analysts is incomplete or unverified and may even be the result of a deliberate attempt to deceive. This messy, dirty, unstructured data requires a unique set of analytic methodologies. Traditional statistical methods often don't work, for example, with the kinds of anecdotal data normal in intelligence work. Third, intelligence is designed to reduce the level of uncertainty for a decisionmaker. It is part of the decisionmaking process. If intelligence is about the question, "What is going on out there and how is it likely to change?" then the other half of the decisionmaking question (the so-called operational half) is "What can we/should we/will we do about it?" Finally, intelligence is a process. It is something that happens, that is both iterative and reflexive; not something that just is. While emphasis is typically placed on the output of this process, the intelligence product, the quality and utility of this product (as with any product) is a direct function of the process used to create it. More importantly, since intelligence is a process, it can be improved upon through careful and intelligent change, improving, in turn, the quality and utility of the final intelligence product. Intelligence products come in many flavors but one of the most useful distinctions is between descriptive and estimative products. Descriptive products outline the relevant facts, figure, personalities and issues surrounding a topic of interest. Estimative products,
3 For a more exhaustive discussion of these themes and how they lead to this definition of intelligence, see Wheaton and Beerbower, Towards A New Definition Of Intelligence, Stanford Law and Policy Review, Vol. 17, Issue 2 (2006), p. 319-330.
on the other hand, attempt to forecast what will likely happen because of the intersection of those facts, figures, personalities, etc. Of the two, decisionmakers typically value quality estimative intelligence over even the very best descriptive products. In a world dominated by a 24/7 news cycle and supported by the vast resources of the internet, providing mere facts and figures is rarely enough to justify the expense of a dedicated intelligence unit. In short, strategic intelligence should be the foundation for all strategic planning. Without some sense of how the external world will likely change to support or hinder attempts to achieve a person’s or organization’s strategic goals, any allocation of resources in support of those goals will likely be sub-optimal. Why games? "...Life is a kind of Chess, in which we have often points to gain, and competitors or adversaries to contend with, and in which there is a vast variety of good and ill events... By playing at Chess then, we may learn: 1st, Foresight... 2nd, Circumspection (and) 3rd, Caution..." -- Benjamin Franklin, The Morals Of Chess Mercyhurst College has the oldest and largest full time, residential Intelligence Studies program in the world. With 350 student on campus in Erie, PA and 10 full time faculty representing all three major sub-disciplines of intelligence (business, law enforcement and national security), Mercyhurst produces qualified entry-level intelligence analysts with both undergraduate and graduate degrees. Critical to the success of this program (now in its 18th year) is its focus on application. Like an engineering or architecture (or, for that matter, games design) program, the goal of the curriculum is to produce graduates who understand both theory and practice and are ready, at an entry-level, to apply this knowledge to real world problems. Strategic Intelligence, as taught at Mercyhurst, is the capstone course of this program at both the graduate and undergraduate levels.4 The primary purpose of the course is to integrate and apply the knowledge and skills gained in earlier courses while adding the specific additional knowledge and skills necessary to prepare a complete strategic intelligence product -- to up their game, as it were, as analysts. The centerpiece of this course is a strategic intelligence project for a real-world decisionmaker. Previous clients have included a number of US national security agencies and organizations such as the Defense Intelligence Agency, the National Security
4 At the undergraduate level, the course is referred to simply as Strategic Intelligence and at the graduate level the course is called Managing Strategic Intelligence Analysis. Both classes are project based but the topics chosen for the graduate class are conceptually more difficult. In addition, there is more emphasis in the graduate class on managing small groups and on other managerial level tasks such as budgeting and personnel selection. Given that graduate students come to our Masters in Applied Intelligence program from all disciplines, many of the core concepts are the same in both classes with the primary difference arising in the expectations regarding performance.
Agency, the National Intelligence Council; law enforcement organizations such as the Federal Bureau of Investigation and the Drug Enforcement Agency; Fortune 500 businesses such as Target Corporation and local firms, such as Dispatch Printing; and international organizations including the European Parliament, the Iraqi government, Dutch counter-terror organizations and a Spanish bank. The intelligence requirements (a fancy name for the questions) posed by these organizations are as diverse as the organizations themselves. A sampling of the kinds of questions typically asked of the student-analysts in the class includes: – What are the most important and most likely impacts on, and threats to, US national interests (including but not limited to political, military, economic and social interests) resulting from infectious and chronic human disease originating outside the US over the next 10-15 years? What are the likely current best practices or combination of best practices utilized by suburban public high schools with respect to curriculum, buildings and green initiatives? What are the likely causes for objection and consent to the ratification of the Fissile Material Cutoff Treaty (FMCT) by "states of interest" (i.e. states of concern and de facto states) over the next 5 years? What are the current and future direct threats to (a Fortune 500 company’s) existing global supply chain over the next 12-24 months? What is the current severity and effectiveness of the insurgency in the North and South Caucasus regions (in regards to the quantitative and geographic growth and spread of violence) and how is it likely to change between now and the 2008 Russian Presidential election? Who are the likely innovative users and what are the likely innovative uses of wiki technology over the next two years?
Students rarely have any previously acquired expertise in these subjects at the beginning of the course. Instead, they have acquired during their time at Mercyhurst a set of skills and techniques that allow them to rapidly orient themselves within the domain of the question and to begin to generate meaningful analytic estimates in response to these questions fairly quickly. This is a good thing as Mercyhurst operates on a 10-week term system and the finished intelligence product is due before the end of the term.5 Students on a team working on a strategic project are expected to efficiently organize themselves to accomplish all of the analytic and administrative tasks associated with their project. The professor in this class provides mentorship, methodological guidance,
5 For an example of such a product, see www.nie.wikispaces.com . This product responded to the first of the six questions listed above and was requested by the National Intelligence Council. You can see their review of the product at http://www.dni.gov/nic/research_globaldisease.html
moral support as well as tangible resources to help the team complete the project. The professor does not, however, provide “answers”. In short, while the professor may help the students draw the map, the students pick their own path. The successful completion of a large-scale project such as this is designed to give these students confidence in their skills and abilities as they approach graduation – a second major purpose of the class. In addition to the project, which takes up approximately 80% of the total time of the course (including time spent both in and out of class), the remaining coursework focuses on three overlapping themes: – – – Strategic theory The current practice of strategic intelligence Review of previously learned concepts that are particularly appropriate in a strategic environment.
For the first six years of the course, I taught this remaining 20% using a standard mix of lecture, discussion and classroom exercise. Despite various tweaks, it became obvious to me that the overwhelming emphasis of the class on the current project – an emphasis that I both encouraged and approved of – severely weakened the impact of the remaining 20% of the course. Unfortunately, it was in this last 20% where the course materials designed to accomplish the third major goal of the course -- preparing these students for the kinds of strategic intelligence challenges they are likely to face throughout their careers, as well as information important to the success of the previous two goals -- largely lay. Inspired by the speakers at the Game Education Summit at Carnegie Mellon University in June, 2009 (and, in particular, by Prof. Ian Schrieber’s lecture on Innovative Teaching Through Game Design), I decided to integrate games into the syllabus such that they were the fundamental pedagogical approach for this remaining 20%. I knew from my own experience that games could be an effective way to teach strategy and strategic intelligence. I realized that much of my own understanding of these concepts had originated with a variety of wargames and other type games I had either played or designed over the years.6 Likewise, these games encouraged me to delve deeper into the literature regarding strategy and strategic intelligence. It was precisely this type of virtuous circle that I hoped to set up in my own class. I also knew that there is an increasing body of literature about the effectiveness of gamesbased learning strategies in the classroom. Studies have been conducted, the results published and briefed and respected individuals outside the gaming industry have endorsed games-based learning. Games-based learning forms a critical component to the US Department of Education’s national education technology plan. Thus, despite
6 I began playing wargames when I was 12, wrote occasionally for the Dragon Magazine and designed simple computer games and simulations while in college and unsuccessfully ran a small games design company in the 1980’s. I developed an expert system designed to simulate the activities of an attorney in giving advice regarding landlord-tenant disputes while in law school. I also designed several simulations while in the army (including the first dedicated tactical air defense simulation) and developed a simulation to understand transitions from authoritarian rule as part of my masters work in Russian and East European Area Studies while at Florida State University.
detractors, the hypothesis that a games-based approach to strategic intelligence would be effective does not seem entirely out of line. Most comments regarding the efficacy of games-based learning initiatives center on the fact that they are fun in one of the 14 different ways that researchers define that term. The fun translates to increased attention to the subject and increased attention, in turn, facilitates learning. My goals were more ambitious. I wanted to try to address all three of the major objectives of this course. First, I wanted them to improve as analysts, specifically by improving their ability to see patterns and connections buried deeply in unstructured data sets that were confusing, incomplete, of unknown reliability and possibly deceptive. These conditions are far more common in intelligence analysis than not. I wanted my students to expand their ways of thinking about intelligence problems -- to develop a flexibility of mind that would help them no matter what kind of problem the world threw at them. I wanted them to become better at discovering solutions on their own rather than merely getting good at recognizing the “right” solution when it was handed to them. I expected to see this most clearly, albeit qualitatively, in classroom discussions and, in the one class where implemented, the students' weekly writing assignments. Second, I also wanted the students to gain confidence in their skills as analysts. I believed that a learning environment dominated by games would foster a more creative, exploratory atmosphere and that this would translate into better final products. Third, I wanted the students to remember not just the experience but also some more general lessons learned that would apply the next time they encountered a strategic intelligence project. This would be the most difficult goal to measure but I believed that I would be able to get at least anecdotal feedback from former students several months after the classes had ended. Finally, I wanted to measure student satisfaction with the course. For this, I would use the results from the Student Instructional Report-II (SIR-II). There is a good bit of discussion about the value and accuracy of the SIR and other student evaluations of teaching effectiveness among college and university faculty.7 It is particularly difficult, I think, to use these scores in a project-based course, where the survey is administered in week eight but the major learning event of the course takes place in week ten, the last week, when the students present the results of their analysis to their decisionmaker (It is sort of like asking a high school biology student what they think of biology before they get to dissect the frog). That said, I had SIRs data from most of the previous classes and, whatever effect the timing of the SIR had on each class's results, the results could still be compared effectively from class to class. My expectation was that this unique, nontraditional approach, if integrated correctly with the core material of the course coupled
7 Conversations with several psychologists who study games and game-based learning during the 2010 Game Education Summit (where this paper was presented) highlighted another important issue in using the SIRs: These tests were not designed to evaluate alternative forms of pedagogy. In fact, the only options on the SIR are "Lecture", "Discussion" or "Combination". It may well be that this tool is wholly inappropriate for evaluating game-based courses.
with the fun of being able to play games would increase student satisfaction with the course. How, specifically, were games used in class? “Indeed, experiments have shown that the more mental work readers have to do to infer a cause from a set of facts, the more memorable the causal inference will be.” – The Trouble With Intuition, Daniel J. Simons and Christopher F. Chabris While I wanted games to be a central tool within the context of the class, I established some fairly rigorous pre-conditions for myself before deciding which games and how exactly they would be used in class. First, I did not want the games to detract from the project. The experience gained from working on a real world project for a real world decisionmaker trumped, in my mind, any value games might bring to learning the essential lessons of strategic intelligence. In addition, these projects have significant tangible value to the students beyond the knowledge they gain from them. In many cases over the years, these projects have led to jobs (or at least job offers) either directly or indirectly. I was worried that the games might detract in two ways. First, the games were to add value, if at all, to only that 20% of the class that was indirectly involved with the project. I did not want to create an environment where a student felt they had to choose between playing a game and getting work done on the project. Second, I did not want the course to become about the game or games. I know from experience that games can be genuinely compelling. There are many good strategic simulations (Diplomacy, The Total War series) around which a strategic course could easily be built. In these cases, I saw the games not just competing for time with the project but actually overwhelming the project completely. Beyond the potential for detraction from the project, the games I did select needed to resonate in a meaningful way with the core concepts of the course. I either needed a single game through which I could articulate all of the core concepts of the course more or less in the order they needed to be presented (unlikely) or find a number of different games which could help me accomplish the same goals. Of course, if I were to use a number of games, the time to learn the new rules of each game would become a factor as well. If I were to use a number of games, then I also believed I needed to include a variety of game types and genres. I know from experience that not all game types appeal to all game players. In fact, I believe that one of the major hurdles to overcome with regard to game-based learning will be the re-packaging of core concepts in any given subject into a variety of game genres such that at least one approach will work for every student. Finally, I was very conscious of the cost. Textbooks are expensive and I believe that, in many cases, they are unreasonably so. I could not see adding to that burden.
In the end, I settled on using casual online or downloadable games or games, such as World Of Warcraft, that came with free trials (for a complete list of the games and the core concepts, see the course syllabus attached at Annex A). The exception to this general rule was the addition of one "old-school", paper-pencil wargame, Defiant Russia. Students were required to play the game before each class. In addition, students were required to come to some defensible conclusion about how the game related to the topic of that particular class and to be prepared to discuss it when they came to class. I indicated to the students that the relationships between the games and the topics were rarely obvious and that in some cases there were many possible defensible conclusions. In many cases, I informed them, there might appear, on the surface, to be no real connection between the game and the topic. I wanted them to have to think hard about the possible connections, to evaluate them and to come to a conclusion that they were prepared to defend. Classroom time was devoted in part to examining what the students saw compared with what I saw as the essential connections. A careful matching of games and topics yielded a fairly high overlap between what the students saw and what I hoped they would see. In addition, I had the genuine pleasure of having students come up with unique and deep interpretations far beyond my expectations (I will discuss this in greater detail in the subsequent posts). For example, in the class where we discussed strategic intelligence requirements (i.e. the questions that intelligence professionals are asked at the strategic level), students had to play World of Warcraft (WOW) or some other quest-based game. While there are many defensible answers to the question regarding the connection between WOW and intelligence requirements, I was able to leverage the student's experience with a wellformed quest (typical of the high-end MMORPGs) and contrast it with the consequences of poorly formed intelligence requirements. This, in turn, gave the students another way of looking at their upcoming meeting with their decisionmaker where they would be receiving the intelligence requirement relevant to their particular project. As a result of my experience with the first two classes, I required the third class in which I used this approach to write down their conclusions and post them to Mercyhurst’s Blackboard course management software prior to class. This writing assignment was modest (100-150 words) but it allowed me to better prepare for the class itself. While students were required to play the game and to come to a conclusion, I also made a wide variety of supplemental readings available. These provided "hints' to the connections between the game and the topic that I saw. I believed that, in some cases (particularly where the material was more in the way of a review), students would be able to come to reasonable conclusions without any additional reading. I also believed that students that were not overly familiar with the topic under consideration would be more engaged in the reading if they were working to answer a question, even one as difficult as the one I posed. So, how did it all work out? KJW--Teaching Strategic Page 10
While mostly anecdotal, the available evidence suggests that students significantly increased their ability to see patterns and connections buried deeply in unstructured data sets, my first goal. This was particularly obvious in the graduate class where I required students to jot down their conclusions prior to class. An example of the growth I witnessed from week one to week ten would likely be helpful at this point. The same student wrote both examples below and I consider this example to be representative of the whole: Week 2 response – “This game (The Space Game: Missions) was predominantly about budgets and space allocation…. Strategy and forethought goes into where you place lasers, missile launchers and solar stations, so that you don’t run out of minerals to power those machines and so repair stations are available for those that are rundown. It’s clear that resource and space allocation are key elements for a player to win this game, just as it is for the Intelligence Community and analysts to win a war.” Week 8 response: “I think if Chess dropped acid it’d become the Thinking Game. When the computer player was contemplating its next move colorful lines and curves covered the board… To me, Chess was always a one-on-one game; a conflict, if you will, between black versus white… Samuel Huntington states up front that he believes that conflict will no longer be about Princes and Emperors expanding empires or influencing ideologies, but rather about conflicts among different cultures: “The great divisions among humankind and the dominating source of conflict will be cultural.” Civilizations and cultures are not black and white, however; they’re not defined by nation-state borders. There are colors and nuances in culture requiring a change in mindset and in strategy to approach these new problems.” While difficult to assess quantitatively, literature from the critical thinking community helps assess the degree of change here.8 In particular, the Guide to Rating Critical Thinking developed by Washington State University identifies seven broad categories for assessing if and to what degree critical thinking is taking place: – – – – – Identification of the question or problem, willingness to articulate a personal position or argument willingness to consider other positions or perspectives identification and assessment of key assumptions identification and assessment of supporting data
8 There is a widespread belief among intelligence professionals that teaching critical thinking methods will improve intelligence analysis (See David Moore’s comprehensive examination of this thesis in his book Critical Thinking And Intelligence Analysis). A minority of authors are less willing to jump on this particular bandwagon (See Behavioral and Psychosocial Considerations in Intelligence Analysis: A Preliminary Review of Literature on Critical Thinking Skills by the Air Force Research Laboratory, Human Effectiveness Division) citing a lack of empirical evidence pointing to a correlation between critical thinking skills and improved analysis.
considers the influence of context on the problem identifies and assesses conclusions, implications and consequences
While such a list may not be perfect, there is certainly nothing on it that is inconsistent with good intelligence practice. Likewise, when reading the representative example above with these criteria in mind, the increase in nuance, the willingness to challenge an acknowledged authority, the nimble leaps from one concept to another all become even more obvious. The majority of the students in the class showed this kind of growth over the course of the term both in the quality of the classroom discussions and in their written reports. In addition to seeing an improvement in students’ ability to detect deep patterns in complex and disparate data sets, I also wanted that increased ability to translate into better quality intelligence products for the decisionmakers who were sponsoring projects in the class. Here the task was somewhat easier. I have solicited and received good feedback from each of the decisionmakers involved in the 78 strategic intelligence projects my students have worked on over the last 7 years. This feedback, leavened with a sense of the cognitive complexity of the requirement, yields a rough but useful assessment of how “good” each final project turned out to be. Mapping this overall assessment onto a 5 point scale (where a 3 indicates average “A” work9, a 2 and 1 indicates below and well below A work respectively, a 4 indicates A+ or young professional work and a 5 indicates professional quality work), permits a comparison of the average quality of the work across various years.
Average Quality Score For Student Generated Strategic Intelligence Products 5 4.5 4 3.5 3 2.5 2 1.5 1 0.5 0 2002
4.57 3.8 4.09 3.67 3.64 3.92 3.73
9 “A” is average for the Mercyhurst Intelligence Studies seniors and second year graduate students permitted to take Strategic Intelligence. In order to be employable in this highly competitive field, the program requires students to maintain a cumulative 3.0 GPA simply to stay in the program. In addition, the major is widely considered to be “challenging” and those who do not see themselves in the career of intelligence analysis, upon reflection, often change majors. As a result, GPAs of the seniors and second year graduate students are often quite high. The graduating class of 2010, for example, averaged a 3.66 GPA.
The chart above summarizes the results for each year. While the subjectivity inherent in some of the evaluations possibly influenced some of the individual scores, the size of the data pool suggests that some of these variations will be eliminated or at least smoothed out through averaging. There are, to be sure, a number of possible reasons to explain the surge in quality evidenced by the most recent year group. The students could be naturally better analysts, the quality of instruction leading up to the strategic course could have dramatically improved, the projects could have been simpler or the results could be a statistical artifact. None of these reasons, in my mind, however, hold true. While additional statistical analysis has yet to be completed, the hypothesis that games-based learning improves the quality of an intelligence product appears to have some validity and is, at least, worthy of further exploration. My third goal for a games-based approach was to better lock in those ideas that would likely be relevant to future strategic intelligence projects attempted by the students, most likely after graduation. To get some sense if the games-based approach was successful in this regard, I sent each of the students in the three classes a letter requesting their general input regarding the class along with any suggestions for change or improvement. I sent these letters approximately five months after the undergraduate classes had finished and approximately 2.5 months after the end of the graduate class. Seventeen of the 75 students (23%) who took one of the three courses responded to the email and a number of students stopped by to speak to me in person. In the end, over 40% of those who took the class responded to my request for feedback in one way or another. This evidence, while still anecdotal, was consistent – games helped the students remember the concepts better. Comments such as, “Looking back, I can remember a lot of the concepts simply because the games remind me of them” or “I am of the opinion that the only reason that the [lessons] stood out was because they were different from any other class most students have taken” were often mixed in with suggestions on how to improve the course. The verbal feedback was even more encouraging, with reports of discussions and even arguments centered on the games and their “meaning” weeks and months after the course was completed. The evidentiary record, in summary, is clearly incomplete but encouraging. Games– based learning appears to have increased intelligence students’ capacity for sensemaking, to have improved the results of their intelligence analysis and to allow the lessons learned to persist and even encourage new exploration of strategic topics months after the course has ended. \What did the students think about it? The SIRs actually measure a number of variables and identifying those that might be most closely associated with the underlying pedagogy of a course are difficult to identify. Instead, I chose to look at just one of the SIR-generated ratings, the Overall Evaluation of the course. I believe that this is the single best indicator of effectiveness. A large change here (in either the positive or the negative direction) would be a clear indication of success or failure. KJW--Teaching Strategic Page 13
Furthermore, my assumption at the beginning of the course was that there would be a large change in one direction or the other. I assumed that students would either love this approach or hate it and that this would be reflected in the SIR results. The chart below, which contains the weighted average of the Overall Evaluation score (1-5 with 5 being best) for all classes taught in a particular year, indicates that I was wrong:
SIR Results for Strategic Intelligence 2003-2009
5 4.4 4 3 2 1 0 2002 4.15 4.71
Clearly, while students did not love it, they did not hate it either. The drop in score from recent years could simply be attributed to the fact that the course changed from a fairly well-oiled series of lectures and exercises to something that had the inevitable squeaks and bumps of a new approach. Feedback from the student surveys given after the course was over, while extremely helpful in providing suggestions for improving the class, gave no real insight into the causes of this modest but obvious drop in student satisfaction. Comparing this chart with the previous one concerning the quality of the final product yields an even more interesting picture:
SIRs vs. Results
5 4.5 4 3.5 Score (1-5) 3 2.5 2 1.5 1 0.5 0 2002 2003 2004 2005 2006 2007 2008 2009 2010 Results SIR
This chart seems to be saying that the more a student thinks they are getting out of class (as represented in their Overall Evaluation of the course) the better their final strategic intelligence project is likely to be. This holds true, it seems, as long as strategic intelligence is taught through more or less traditional methods of lecture, discussion and classroom exercises. Once the underlying structure of the course is centered on games, however, the students are less satisfied but actually perform better where it matters most – on real-live projects for real-world decisionmakers. Taken at face value (and ignoring, for the moment the possibility that this is all a statistical anomaly), a possible explanation is that the students don’t realize what they are getting “for free” from the games-based approach. Other researchers have noted that information that had to be actively taught, assessed, re-taught and re-assessed in other models is passively (and painlessly) acquired in a games-based environment. I noted this effect myself in my thesis research into modeling and simulating transitions from authoritarian rule. My goal, in that study, was to develop a predictive model; not to teach students about the target country. One of my ancillary results, however, was that students routinely claimed that they learned more about the target country in three hours of playing the game than in a semester’s worth of study. This “knowledge for free” aspect of the games-based model was nowhere more obvious than in the fairly detailed understanding of the geography of western Russia acquired by the students in all three classes while playing the boardgame, Defiant Russia. While this information was available in the form of the game map, learning the geography was not explicitly part of the instructions. Students rapidly understood, however, that they had to understand the terrain in order to maximize their results within the game. Furthermore, an understanding of the geography of western Russia was critical to the formulation of strategic options.
This raises a broader question regarding games based learning: If students don't know they are learning, how can they evaluate the learning process? While I have not had time to dig deeply into the literature regarding implicit learning, I intend to. Giving students a tangible sense of what they are learning in a game based environment may be one of the biggest challenges to overcome with the approach, at least in higher education. What else did you learn? Many students have provided excellent feedback for improving the course. The single most requested ‘tweak’ is, surprisingly, to include more games like Defiant Russia. The old-school boardgame with its dice, hex maps and counters seemed to encourage a thoughtful, collaborative (at least within the two teams playing) learning experience. In addition, the idea of replaying history was clearly appealing to many of the students. Only one of the students had played anything similar prior to this class and it was unclear if any would voluntarily play something like Defiant Russia again but the overwhelmingly positive response to the game in the feedback suggests that there is still a place for these types of games in educational environments. The main problem with a game like Defiant Russia and using it in an educational setting is the amount of time it takes to play. For two experienced players, the game can move very quickly. However, when playing it as I did, with two teams of inexperienced players, the first turn can last the better part of an hour. The popularity of this experience demands, however, that I take an additional look at how I might be able to carve out time for another game like it. Several other comments surfaced routinely. First, there was a fairly common request to cut back on the number of games or to cut back on the games as the end of the course approached. This request seemed to be driven by two separate reasons. The first was that the lessons learned lost some of their potency, as students had to rapidly drop one game only to pick up and analyze another. The second was that, for people who did not routinely play games, learning the rules to new games – even casual games -- every couple of days was difficult. On the one hand, “more time on fewer subjects” is classic pedagogical advice; on the other, “practice makes perfect” is also sound. One of my goals was to encourage the students to not only be better but also quicker thinkers; to identify the patterns in complex, confusing issues rapidly and flexibly. The incessant drumbeat of games over the course of the term seemed to accomplish this. Another goal, however, was to lock in knowledge important to the practice of strategic intelligence. This kind of learning requires reflection and reflection takes time. Clearly, the right answer lies in properly balancing these competing goals. How to do that in the context of a specific syllabus is the real question and one that I will spend the next several months pondering. Another suggestion that seemed to make sense was to do a better job of explaining how games-based learning worked. I provided students with some explanation and resources early on in the course but decided not to spend much time discussing this unique pedagogical approach. Given the feedback and the results of this study, it probably KJW--Teaching Strategic Page 16
makes some sense to discuss this approach more fully with the students. In fact, it is my intent to give them a copy of this paper when classes begin in the fall. Finally, there is one recommendation that I am considering with some hesitation: Make the connections between the games and the topics covered in the course “more clear”. My instincts say that this would be a mistake; that the purpose of the course is to challenge students deeply, to make them travel unlit paths in darkened forests, to attempt to climb insurmountable mountains. I would rather have them try and fail for, the way I have constructed the course, there is no penalty in failing, only in not trying. Clearly, here, too, the question is one of balance. At some point, the connection between the game and the topic can be so obtuse as to be impossible to find except through dumb luck. Likewise, simple connections do little to foster the sense of exploration and discovery I think it critical to this approach. Beyond these more or less common themes, I have received a wide variety of other suggestions (including some game recommendations) that I intend to examine in detail before the next time I teach the class. Regardless of what changes, additions or deletions I make, the conclusion seems inescapable: Games-based learning, while not a perfect pedagogical approach, has merit worth exploring when teaching strategic intelligence.
Annex 1: Syllabus STRATEGIC INTELLIGENCE COURSE SYLLABUS (3 Credits) – Winter 09 Mercyhurst College Erie, Pa INSTRUCTOR: Kristan J. Wheaton E-mail: email@example.com Office Phone: 814 824 2023 Office Hours: TBD TEXTS: Clavell, James (ed.), Sun Tzu’s The Art of War (Delacorte Press, New York, 1983) Machiavelli, Nikolo, The Prince (Bantam Classics, New York, 1984) Howard, Michael, Clausewitz (Oxford University Press, Oxford, 1983) Huntington, Samuel, The Clash of Civilizations Tetlock, Phillip, Expert Political Judgment Kidder, Rushworth; How Good People Make Tough Choices : Resolving the Dilemmas of Ethical Living (Fireside, 1996) Additional readings are also available on Blackboard Also available online Required COURSE DESCRIPTION: Strategic estimative intelligence is perhaps the single most difficult skill an analyst must master. Done correctly, it helps decisionmakers allocate resources today in such a way that it positively impacts results for years to come. Done incorrectly, decisionmakers can squander time, money and even lives before a mistake is caught and corrected. This course is designed to teach the analyst strategic theory and the practice of strategic intelligence. In addition, the analyst will apply these principles to “real-life” questions posed by senior decisionmakers involved in national government, law enforcement, business and academia. COURSE OBJECTIVES: The Analyst will: 1. Learn and apply the fundamentals of the strategic estimative process. 2. Learn the fundamentals of strategy. KJW--Teaching Strategic Page 18
3. Actively participate as the member of a group in a large-scale estimative project. 4. Prepare and brief an intelligence estimate on a current issue to a senior decisionmaker involved with that issue 5. Gain a significant level of understanding of small group dynamics and the fundamentals of small group leadership COURSE REQUIREMENTS: The course contains a number of interlocked themes including, but not limited to, strategic theory, the practice of strategic intelligence and the application of those principles to a “real-life” problem. Analysts will participate throughout the course as the member of a group in a large-scale estimative project. Students will be expected to meet with the decisionmakers who have defined the real-life problem at the time and convenience of those decisionmakers. Students will attend all face-to-face or videoconference meetings in appropriate attire. Much of the work of the course will be done via the web or through the Blackboard system. Students are expected to check Blackboard and their Mercyhurst e-mail service regularly. Analysts will play or review all assigned games prior to the class for which they are assigned. Students should come to a conclusion about how the game they are playing relates to the topic of the class before entering class. Note: – The relationship is rarely obvious, though some may see it immediately. – The additional readings provided each week contain hints pointing to the connection between the game and the topic. These readings are optional. GRADING SYSTEM: Attendance/Class Participation = 20% Terms of Reference = 10% Final Product = 35% Final Brief = 35% ADMINISTRATIVE ISSUES: Class attendance is required. Every student is expected to attend every class. Students are responsible for all readings in textbooks and any handouts, including those readings that may be placed on reserve in the library. Students are expected to have read the materials assigned prior to class. Neither cheating nor plagiarism will be tolerated. A= 93-100% B+=90-92% B=83-89% C+=80-82% C=70-79% D=60-69% F=<59%
OUTLINE: Week 1: Class 1 – Course Outline, Intro to the MIE, Elements of Strategy, the Role of Intelligence • Teams Assigned • Game: Desktop Tower Defense Class 2 – Developing Strategic Background Information, Assessing Strategic Requirements • Game: World Of Warcraft (or other quest-based game) Class 3 – Requirements Meeting with DM NLT 2359 11 DEC 2009 Week 2: Class 4 –Theory Review: The Intelligence Process; Terms of Reference, Budgeting (TOR) • Game: The Space Game: Missions • Budget Exercise Assigned Class 5 – Group Conceptual Modeling, Managing Analytic Teams • Game: Budget Hero Class 6 – Group Meetings with Instructor (Actual times TBD) • Draft TOR due 24 hours before Group Meeting • Budget Exercise due NLT 14 DEC 2359 Week 3: Class 7 – Method Review: Case Studies • Game: Train (Not playable) – WWI Medic ? Class 8 – Strategy Overview: The Art of War • Game: Go Class 9 – Group Meetings with Instructor • Group Conceptual Model due Class 10 – Method Review: ACH And Source Reliability • Game: This Is The Only Level • Get TOR approved before 2359 21 DEC 2009 Class 11 – Strategy Overview: The Prince • Game: Samarost Class 12 – Method Review: Multi-criteria Intelligence Matrices (MCIM) • Game: Clue Class 13 – Group Meetings with Instructor Class 14 – Theory Review: Communicating With Decisionmakers • Game: Auditorium Class 15 – Strategy Overview: Clausewitz • Game: The Game Of Life Class 16 -- Group Meetings with Instructor Class 17 – Strategy Overview: Clash of Civilizations Page 20
• Game: Dice Wars, Thinking Machine 4 Class 18 – Crisis Intelligence: A Strategic Perspective No Group Meetings this week due to MLK Birthday Week 8: Week 9: No Class – Battle Week! Class 19 – Method Review: Decision Trees And Bayes • Game: Bow Street Runners Class 20 – Theory Review: Evaluating Intelligence • Game: Global Conflicts – Child Soldier Class 21 -- Group Meetings with Instructor Draft Key Findings Due 24 hours before Group Meeting Class 22 -- Expert Political Judgment Class 23 – How Good People Make Tough Choices Class 24 -- Group Meetings with Instructor Draft Final MIE due 24 hours before Group Meeting Draft Final Briefings due at Group Meeting
Week 11/12: Final MIE due on date of presentation Presentation and Critique of MIE NLT 19 FEB 1700