EXPLICIT CONCEPTUAL MODELS: SYNTHESIZING DIVERGENT AND CONVERGENT THINKING

SHANNON L. FERRUCCI

A Thesis Submitted to the Faculty of Mercyhurst College In Partial Fulfillment of the Requirements for The Degree of MASTER OF SCIENCE IN APPLIED INTELLIGENCE

DEPARTMENT OF INTELLIGENCE STUDIES MERCYHURST COLLEGE ERIE, PENNSYLVANIA MAY 2009
   

   

DEPARTMENT OF INTELLIGENCE STUDIES MERCYHURST COLLEGE ERIE, PENNSYLVANIA

EXPLICIT CONCEPTUAL MODELS: SYNTHESIZING DIVERGENT AND CONVERGENT THINKING A Thesis Submitted to the Faculty of Mercyhurst College In Partial Fulfillment of the Requirements for The Degree of MASTER OF SCIENCE IN APPLIED INTELLIGENCE

Submitted By: SHANNON L. FERRUCCI

Certificate of Approval:

___________________________________ Kristan J. Wheaton Assistant Professor Department of Intelligence Studies

___________________________________ William J. Welch Instructor Department of Intelligence Studies

___________________________________ Phillip J. Belfiore Vice President Office of Academic Affairs

May 2009
   

   

Copyright © 2009 by Shannon L. Ferrucci All rights reserved.

iii   

   

ACKNOWLEDGMENTS

I would like to thank Kristan J. Wheaton, my thesis advisor and primary reader, for his continued guidance and encouragement throughout the course of this work. I would also like to thank Professor Hemangini Deshmukh for her patience and assistance with the statistical analysis of this work, it was greatly appreciated.

iv   

   

ABSTRACT OF THE THESIS

Explicit Conceptual Models: Synthesizing Divergent And Convergent Thinking By Shannon L. Ferrucci Master of Science in Applied Intelligence Mercyhurst College, 2009 Professor Kristan J. Wheaton, Chair

Explicit conceptual modeling (ECM) within intelligence analysis is a topic on which very little specific research has thus far been done. However, when considering the complexity and depth of most intelligence requirements it becomes evident that consideration of this topic is both crucial and long overdue. This thesis examines what little literature exists on conceptual modeling within intelligence analysis, in addition to discussing relevant studies from other fields that help to shed light on the need for, and value of, incorporating this technique into intelligence analysis. After examining the

relevant literature, an experiment was conducted to test the hypothesis that intelligence analysts who engage in ECM will generate better analytic products, as evaluated by thoroughness of process and accuracy of product, than analysts who do not. However, despite a wealth of literature strongly suggesting that ECM will improve analysis the results of this study’s experiment did not support that notion. The author ends by drawing conclusions from the experimental data highlighting the notion that ECM requires a combination of robust divergent and convergent thinking techniques to be successful. v 
 

   

TABLE OF CONTENTS Page COPYRIGHT PAGE………………………………………………………………... ACKNOWLEDGEMENTS…………………………………………………………. ABSTRACT……………………………………………………………………….... TABLE OF CONTENTS…………………………………………………………… LIST OF FIGURES…………………………………………………………………. CHAPTER 1 INTRODUCTION…………………………………………………… Conceptual Models………………………………………………….. Explicit Modeling and Intelligence Analysis……………………….. 2 LITERATURE REVIEW………………………………………….... Constructivist Roots…………………………………………………. Mental Models and Intelligence……………………………………... Memory Limitations………………………………………………… Group Intellect………………………………………………………. Combating Groupthink……………………………………………… Related Mapping Disciplines………………………………………... Mind Maps…………………………………………………………... Concept Maps……………………………………………………….. Technology Aids…………………………………………………….. Learning Styles……………………………………………………… Hypotheses…………………………………………………………... 3 METHODOLOGY…………………………………………………... Research Design……………………………………………………... Subjects……………………………………………………………… Preliminaries………………………………………………………… Control Group: Day 1……………………………………………….. Experimental Group: Day 1…………………………………………. Bubbl.us……………………………………………………………... Control Group: Day 2……………………………………………….. Experimental Group: Day 2…………………………………………. vi 
 

iii iv v vi ix

1 2 3 6 6 7 8 9 10 11 13 14 15 17 17 18 18 18 22 22 24 26 27 28

   

Data Analysis Procedures…………………………………………… 4 RESULTS…....……………………………………………………… Significance Testing…………………………………………………. Pre- and Post-Questionnaire Results………………………………… Process – Conceptual Model Findings………………………………. Process – Logic/Quality of Supporting Evidence Findings…………. Product – Forecasting Findings……………………………………... Quality Of Supporting Evidence Vs. Forecasting Accuracy………... Product – Source Reliability and Analytic Confidence Findings…… 5 CONCLUSIONS……………………………………………………. Excessive Possibilities Confuse……………………………………... Importance of Convergence…………………………………………. Final Thoughts………………………………………………………. Future Research……………………………………………………... BIBLIOGRAPHY…………………………………………………………………... APPENDICES………………………………………………………………………. Appendix 1: Experiment Sign-Up Form…………………………….. Appendix 2: IRB Research Proposal………………………………... Appendix 3: Control Group Consent Form…………………………. Appendix 4: Experimental Group Consent Form…………………… Appendix 5: Research Question……………………………………... Appendix 6: Important Supporting Information…………………….. Appendix 7: Experiment Answer Sheet……………………………... Appendix 8: Control Group Expectation Sheet……………………... Appendix 9: Experimental Group Expectation Sheet……………….. Appendix 10: Pre-Experiment Questionnaire……………………….. Appendix 11: Contact Information………………………………….. Appendix 12: Conceptual Modeling Lecture………………………... vii 
 

29 30 30 31 36 40 41 42 46 48 49 51 52 54 55 57 58 59 66 67 68 69 70 71 72 73 74 75

   

Appendix 13: Bubbl.us Instruction Sheet For Experimental Group… Appendix 14: Bubbl.us Instruction Sheet For Control Group………. Appendix 15: Structured Conceptual Modeling Exercise…………... Appendix 16: Control Group Post-Experiment Questionnaire……… Appendix 17: Experimental Group Post-Experiment Questionnaire... Appendix 18: Control Group Debriefing Sheet……………………... Appendix 19: Experimental Group Debriefing Sheet……………….. Appendix 20: Significance Testing Results………………………….

77 79 80 81 83 86 87 88

 

viii 
 

   

LIST OF FIGURES

Page Figure 2.1 Figure 2.2 Figure 2.3 Figure 3.1 Figure 3.2 Figure 3.3 Figure 3.4 Figure 3.5 Figure 4.1 Figure 4.2 Figure 4.3 Figure 4.4 Figure 4.5 Figure 4.6 Figure 4.7 Figure 4.8 Figure 4.9 Figure 4.10 Figure 4.11 Figure 4.12 Figure 4.13 Example Conceptual Model Example Mind Map Example Concept Map Subject Education Level Subject Education Level By Group Original Control Group Vs. Actual Control Group Original Experimental Group Vs. Actual Experimental Group Bubbl.us Screenshot Pre- Vs. Post Experiment: Time Dedicated To Experiment Control Vs. Experimental: Learning Style Bubbl.us Screenshot With Concept/Connection Labels Experimental Group: Average Concepts And Connections Control Vs. Experimental: Average Concepts And Connections Control Group Conceptual Model Example Experimental Group Conceptual Model Example Control Vs. Experimental: Accuracy Of Forecasting Forecasting Accuracy: Top Vs. Bottom Half Process Rankings Control Forecasting Accuracy By Process Ranking Experimental Forecasting Accuracy By Process Ranking Control Vs. Experimental: Source Reliability Control Vs. Experimental: Analytic Confidence ix 
 

12 14 15 21 21 23 25 27 32 34 37 38 38 39 40 42 44 44 45 46 47


 

CHAPTER I: INTRODUCTION

In an introduction to The Jefferson Bible: The Life and Morals of Jesus of Nazareth, Forrest Church, Minister of Public Theology at the Unitarian Church of All Souls in New York City, tells the reader of an historical offer made by Thomas Jefferson to Congress.1 Jefferson’s offer consisted of selling his personal library to replace the volumes in the Library of Congress burned by the British during the War of 1812. While some might find the most interesting aspect of Jefferson’s proposal to be the reaction it elicited from members of Congress who were insulted by the specific makeup of the collection, Church places importance on a different aspect of the story altogether. According to Church, “Jefferson’s scheme of classification as formulated in the catalog that he submitted to Congress” was more telling than anything else.2 Furthering a method established by Francis Bacon in 1605, Jefferson categorized his books by “the process of mind employed on them.”3 Therefore, books having to do with philosophy were classified under reason, history books could be found under the label of memory and books focused on fine art could be located under a section entitled imagination.4 However, Jefferson did not stop there. Under each of the overarching categories mentioned above was a variety of intricate subdivisions that further organized Jefferson’s collection. Based on Church’s retrospective examination of the library, it
                                                            
***This research partially funded by the Mercyhurst College Academic Enrichment Fund*** Forrest Church, Introduction to The Jefferson Bible: The Life and Morals of Jesus of Nazareth, by Thomas Jefferson (Boston: Beacon Press, 1989), 1. 2 Church, The Jefferson Bible, 2. 3 Ibid. 4 Ibid. 
1

   


 

appears that the true value of Jefferson’s system of categorization lies in its ability to provide a glimpse into the inner thoughts and beliefs of Jefferson himself. Due to the detail with which the library was constructed, Church was able to surmise Jefferson’s viewpoint on a variety of issues, particularly religion, based on the placement and relationship amongst books within Jefferson’s hierarchical structure.

Conceptual Models What Bacon in the early 1600s, and Jefferson in the early 1800s, were essentially doing through their systems of classification was attempting to make their individual mental models of the world around them explicit. Surprisingly enough, not only do the likes of Bacon and Jefferson develop such mental models, but each and every one of us carries out this same exercise on a variety of different levels numerous times per day. For example, we construct mental models of the route we take on the way to the grocery store and of our routine for getting ready in the morning. This implicit modeling is extremely interesting in the context of intelligence analysis, when considering that we also build models when faced with questions. Whether the issue is a simple one, such as what to do on our day off, or as complex as an intelligence requirement set forth by a decision maker, the human mind automatically attempts to model the question and arrive at possible preliminary answers. Oftentimes in doing this, we are able to recognize not only what we currently know about a given situation, but also what we think we need to know in order to arrive at a comprehensive answer. According to an article from the journal of Information Research by Kalervo Jarvelin, Academy Professor in the Department of Information Studies at the University
   


 

of Tampere in Finland, and T.D. Wilson, Professor Emeritus at the University of Sheffield in the United Kingdom: All research has an underlying model of the phenomena it investigates, be it tacitly assumed or explicit. Such models called conceptual frameworks, or conceptual models… may and should map reality, guide research and systematize knowledge. A conceptual model provides a working strategy, a scheme containing general, major concepts and their interrelations. It orients research towards specific sets of research questions.5 Obviously, the more complex the question, the more intricate the subsequent model tends to be. This is especially true of requirements posed to intelligence analysts, which often entail the understanding of multifaceted relationships between people, states, organizations, industries, etc. Therefore, the odds of any analyst being able to develop a complete model of an intelligence requirement on their first try are very slim. More often than not, analysts are able to fill in pieces of their model with information they already know, but are forced to fill in the rest with topics they recognize they need to understand more about.

Explicit Modeling and Intelligence Analysis The complexity of intelligence requirements leads to the core purpose of this study: determining the value of making these conceptual models explicit within the analytic process. In considering the scope of most intelligence requirements, it becomes obvious that the vast majority of related conceptual models will become too complex to be held in an individual’s memory, and hence would benefit from being made explicit. This is especially true when considering that conceptual models are not static, but actually quite amorphous, constantly evolving and adapting to new information and
                                                            
5

Kalervo Jarvelin and T.D. Wilson, “On Conceptual Models for Information Seeking and Retrieval Research,” Information Research 9, no. 1 (2003), http://informationr.net/ir/9-1/paper163.html (accessed January 15, 2009).

   


 

improving knowledge on a specific topic. Consequently, at present, an exploration of explicit conceptual modeling’s (ECM) place within the field of intelligence is both crucial and long overdue. Within the intelligence community, this topic is primarily of interest to intelligence professionals holding managerial positions, intelligence educators, and individual intelligence analysts (both students and practitioners). For these groups the incorporation of ECM into the analytic process would likely be beneficial on a variety of fronts. First, explicit modeling may increase efficiency in the analyst’s collection Second, by

process, in addition to aiding in the identification of knowledge gaps.

organizing ideas and information in a simplistic, straightforward and graphic way, managers at the head of small analytic teams might more easily grasp what needs to be done, the best method for doing it, and the most efficient way to originally task project analysts. In addition, after initial areas of responsibility are assigned to each analyst it is likely that managers might find it easier to supervise analysts, due to the organizational foundation provided by the model. ECM may also be useful in helping analysts to assess their level of analytic confidence in the estimate produced. In addition, analysts may share, compare and discuss models amongst themselves and with other professionals. Finally, ECM would likely be useful for after the fact review and in providing a solid starting point for any related questions posed to an analyst in the future. However, while these are only some of the potential benefits stemming from the incorporation of ECM into the analytic process, this thesis will show that obtaining the abovementioned results is not easy.

   


 

Furthermore, this study will call into question conventional wisdom regarding what makes a good explicit conceptual model. Taken as a whole, this thesis will argue that despite the relative dearth of studies focused specifically on conceptual modeling within the field of intelligence, literature and examples from other fields will shed light on the need for, and value of, incorporating this technique into intelligence analysis. As one study on conceptual modeling within the field of intelligence has stated, “Conceptual models both fix the mesh of the nets that the analyst drags through the material in order to explain a particular action or decision and direct him to cast his net in select ponds, at certain depths, in order to catch the fish he is after.”6

                                                            
Graham T. Allison. “Conceptual Models and the Cuban Missile Crisis,” in The Sociology of Organizations: Classic, Contemporary and Critical Readings, ed. Michael Jeremy Handel, (Thousand Oaks: SAGE, 2003), 185. 
6

   


 

CHAPTER II: LITERATURE REVIEW

The following is a review of literature relevant to conceptual modeling and intelligence analysis. The section begins with a discussion of constructivism’s bearing on conceptual modeling and an illustration of the importance of a given analyst’s mental model to the analysis produced. Next is a discussion of the relationships between

intelligence requirements, human memory limitations, group intelligence and groupthink and the support they provide for the need to make our mental models explicit. This is followed by a segment on the various methods for making these models explicit, in particular mind maps, concept maps and explicit conceptual models. The benefit of using technological aids to assist in the creation of explicit conceptual models is also mentioned, as is the notion that the utility of ECM may be affected by varying individual learning styles. Finally, this section concludes with the author’s original hypotheses for this study.

Constructivist Roots The array of concepts and relationships between them, illustrated through conceptual models, is closely related to constructivist notions of knowledge formation. In particular, the work of the famed Swiss psychologist Jean Piaget is relevant, as his viewpoint holds that individuals continually construct cognitive models to make sense of the world around them by organizing and connecting their ideas, observations and experiences.7 Additionally, according to Piaget these cognitive models are always

                                                            
7

John W. Santrock, Adolescence, 8th ed.(New York: McGraw-Hill, 2001), 102.

   


 

evolving to include new information that aids individuals in furthering their understanding of the world around them.8 Piaget called the constructs for assembling these models schema or “a concept or framework that exists in the individuals’ mind to organize and interpret information.”9 While a discussion of the pros and cons of

constructivism theory is outside the reach of this paper, using this theory to aid in thinking about the development of models within our minds is actually quite useful.

Mental Models and Intelligence When taking constructivist theory and applying it to the field of intelligence analysis it becomes clear that each analyst’s cognitive, or mental model, is uniquely shaped by the context and purpose of the requirement posed to them, along with the summation of that individual’s prior experiences, schooling, cultural values, professional position and organizational standards.10 As stated in “Intelligence Analysis: Once

Again,” by Charles A. Mangio, of Shim Enterprise, Inc., and Bonnie J. Wilkinson of the Air Force Research Laboratory: Given the importance of the mental model in influencing and shaping the analysis (i.e., from problem exploration and formulation, to purpose refinement, through data acquisition and evaluation, and ultimately determining meaning and making judgments), it is not surprising how it influences the discussion of intelligence analysis.11 However, despite the importance of a well-defined and thorough mental model to an analyst’s subsequent analysis, it is rarely touched upon in intelligence literature.

                                                            
8 9

Ibid. Ibid.  10 Charles A. Mangio and Bonnie J. Wilkinson, “Intelligence Analysis: Once Again” (paper presented at the annual international meeting of the International Studies Association, San Francisco, California, 26 March, 2008): 8. 11 Ibid.

   


 

Memory Limitations Despite the lack of attention given to mental models in the intelligence community, the formulation of these models can significantly impact the process of intelligence analysis. However, eventually it becomes obvious that as the amount of concepts and relationships included in the analyst’s mental model continues to grow, it becomes difficult to store all of that information accurately in working memory. In “The Magical Number Seven, Plus or Minus Two: Some Limits on our Capacity for Processing Information,” George A. Miller, Professor Emeritus of Psychology at Princeton University, argues that we only have the ability to hold 7 things (plus or minus 2) in our mind at a given time without making mistakes in differentiation.12 Having said that, there are methods individuals can use to help them surpass these known limits, as well as there are a variety of exceptions to the rule in the first place. In Psychology of Intelligence Analysis, Richards Heuer, prior staff officer and contractor of the CIA for almost 45 years, discusses one such method for aiding analysts in exceeding memory constraints. Essentially, what Heuer describes is none other than making an individual’s mental model explicit: “The recommended technique for coping with this limitation of working memory is called externalizing the problem—getting it out of one’s head and down on paper in some simplified form that shows the main elements of the problem and how they relate to each other…Breaking down a problem into its component parts and then preparing a simple ‘model’ that shows how the parts relate to the whole. When working on a small part of the problem, the model keeps one from losing sight of the whole.”13 Regardless of the specific number of concepts that our working memory can handle, the idea that there is an upper limit is quite evident, as well as is the fact that most
                                                            
12

George A. Miller, “The Magical Number Seven, Plus or Minus Two: Some Limits on our Capacity for Processing Information,” Psychological Review 63 (1956): 81-97.  13  Richards J. Heuer, Psychology of Intelligence Analysis (Center for the Study of Intelligence, 1999), 27. 

   


 

intelligence requirements will easily exceed that maximum. Limits on working memory, although one of the major concerns considered in the argument for making mental models explicit, are only one of the factors behind the need for engaging in the process of ECM. Another reason strengthening the argument for explicit modeling stems from the inherent complexity in intelligence requirements, which oftentimes leads analysts to work in groups in order to tackle compound issues and topics.

Group Intellect People often criticize the collective judgment of groups as unreliable, viewing individual conclusions as being much more sensible and sound. In fact, groups are often viewed as bringing out the worst in individuals, resulting in illogical and foolish behavior. However, in The Wisdom of Crowds, James Surowiecki, a staff writer at The New Yorker, actually defends group decisions, noting that the four conditions distinguishing wise crowds are: Diversity of opinion (each person should have some private information, even if it is just an eccentric interpretation of the known facts), independence (people’s opinions are not determined by the opinions of those around them), decentralization (people are able to specialize and draw on local knowledge), and aggregation (some mechanism exists for turning private judgments into a collective decision. If a group satisfies those conditions, its judgment is likely to be accurate.14 According to Surowiecki, the collective intelligence of groups often far surpasses the individual intelligences of the people making up that group.15 However, group work does entail its own unique set of problems, one of which is groupthink.

                                                            
14 15

James Surowiecki, The Wisdom of Crowds (New York: Anchor Books, 2004), 10. Surowiecki, Wisdom of Crowds, XIII.

   

10 
 

Combating Groupthink Most classrooms and professional environments are made up of a mix of individuals ranging from those inclined to chime in to discussions and offer opinions ad nauseam to those who shudder at the thought of speaking up. While there can of course be a wide variety of reasons certain individuals are hesitant to actively participate in classroom discussion or workplace meetings, a common fear is that their responses will somehow be inadequate, causing them to embarrass themselves in front of others. In McKeachie’s Teaching Tips, Wilbert J. McKeachie suggests that, “Asking students to take a couple of minutes to write out their initial answers to a question can help. If a student has already written an answer, the step to speaking is much less than answering when asked to respond immediately.”16 Essentially, the notion is that even the most timid will contribute when simply asked to read off what they have already written down. While McKeachie, Professor Emeritus of Psychology at the University of Michigan, speaks solely of students, this same concept applies to professionals. By asking all individuals to jot down answers to a proposed question, then focusing on each person in turn and having them voice those ideas out loud, equal involvement is fostered. No one is allowed to passively soak up the information being offered by others, while at the same time a select few individuals are prevented from dominating the discussion. This method of systematically focusing on each group member’s opinion also helps to combat instances of groupthink. In Groupthink: Psychological Studies of Policy Decisions and Fiascoes, Irving L. Janis defines groupthink as, “A mode of thinking that people engage in when they are deeply involved in a cohesive in-group, when the
                                                            
16

Wilbert J. McKeachie, McKeachie’s Teaching Tips (Boston: Houghton Mifflin Company, 2002), 42.

   

11 
 

member’s strivings for unanimity override their motivation to realistically appraise alternative courses of action.”17 According to Janis, Professor of Psychology at Yale University prior to his death in 1990, there are three main categories of groupthink: group overestimations of its power and morality, closed-mindedness and pressures toward group uniformity.18 In situations where groupthink prevails, teams of individuals often have trouble successfully completing the requirements placed upon them, subsequently failing to meet their goals. By having all group members write down their thoughts and then repeat those thoughts aloud, the phenomenon of individuals keeping quiet so as not to voice unpopular or contrasting views is limited. Furthermore, this method also limits having only a select few outspoken individuals’ perspectives heard and considered. Therefore, the process of ECM is likely beneficial not only in surpassing memory limitations but also in combating groupthink, one of the most common problems plaguing group work.

Related Mapping Disciplines While ECM is the method used for explicitly visualizing information and knowledge in this study, a variety of related mapping disciplines with similar functions do exist. Two in particular that warrant a brief discussion are mind maps and concept maps, both of which are highly analogous to conceptual modeling. For the purposes of this study however, conceptual models were found to be the most functional and userfriendly method for experiment participants to learn and understand in a short time period. Based on the fact that the methods of mind and concept mapping are both
                                                            
17

Irving L. Janis, Groupthink: Psychological Studies of Policy Decisions and Fiascoes (Boston: Houghton Mifflin Company, 1982), 9. 18 Janis, Groupthink, 174-175.

   

12 
 

essentially “coined” exercises, their creation involves following a set of predetermined criteria (see below mind map and concept mapping sections for further detail). ECM on the other hand, simply focuses on the visualization of concepts and their relationships without the added emphasis on the specific construction of the model, allowing individuals the maximum freedom possible to organize the model in whatever way was most helpful to them. As a result, the method and design for conceptual model construction used within this thesis has been operationalized from the relevant literature (see the methodology section for further detail regarding development of the models). Please refer to Figure 2.1 below, taken from Bubbl.us, for an illustration of a conceptual model created by experiment participants in this study. Figure 2.1

Of course, taking a somewhat abstract concept and translating that into a concrete and measurable product, is not without its problems. First, this author’s interpretation of the physical creation of the conceptual models may differ from the interpretation of others. Additionally, whereas this author treats the notion of conceptual modeling as
   

13 
 

distinct and unique, others may disagree, believing it to be simply a subset of a related mapping discipline.

Mind Maps The popular exercise of mind mapping that many individuals are familiar with today got its start in the 1960s with its originator, Tony Buzan. In “Mind Maps as Classroom Exercises,” John Budd, professor in the Industrial Relations Center at the University of Minnesota’s Carlson School of Management, provides a thorough description of the accepted format for creating such maps: As with a traditional outline, a mind map is based on organizing information via hierarchies and categories. But in a mind map, the hierarchies and associations flow out from a central image in a freeflowing, yet organized and coherent, manner. Major topics or categories associated with the central topic are captured by branches flowing from the central image. Each branch is labeled with a key word or image. Lesser items within each category stem from the relevant branches.19 Additionally, a strong emphasis is placed on the incorporation of colors and images into the creation of a mind map.20 Therefore, the essential function of a mind map is very similar to that of the explicit conceptual model, but the process for developing one is more formalized. Please see Figure 2.2 below, taken from the TechKNOW Tools

website, for an illustration of a typical mind map.

                                                            
19

John W. Budd, “Mind Maps as Classroom Exercises,” Journal of Economic Education (Winter 2004): 36. 20 Budd, “Mind Maps as Classroom Exercises.” 

   

14 
 

Figure 2.2

Concept Maps According to Alberto Canas, Associate Director of the Institute for Human and Machine Cognition, and Joseph Novak, known for his development of concept mapping in the 1970s: Concept maps are graphical tools for organizing and representing knowledge. They include concepts, usually enclosed in circles or boxes of some type, and relationships between concepts indicated by a connecting line linking two concepts. Words on the line, referred to as linking words or linking phrases, specify the relationship between the two concepts.21 The detailing of the relationship between concepts by naming links accordingly is very important to the notion of concept mapping and helps to distinguish this form of mapping
                                                            
Joseph Novak and Alberto Canas, The Theory Underlying Concept Maps and How to Construct and Use Them, Technical Report IHMC Cmap Tools 2006-01 Rev 01-2008, Florida Institute for Human and Machine Cognition, 2008. http://cmap.ihmc.us/Publications/ResearchPapers/TheoryUnderlyingConceptMaps.pdf (accessed August 8, 2008).
21

   

15 
 

from mind mapping and conceptual modeling. Another difference between concept maps and mind maps is that the latter are organized around one central concept, whereas the former tend to be organized around several. Like conceptual models, concept maps are intended to evolve over time as an individual’s understanding of a topic increases. However, once again, while concept mapping serves a very similar purpose to that of conceptual modeling, its construction is much more structured in nature. Please see Figure 2.3 below, taken from the cited work of Novak and Canas, for an illustration of a typical concept map. Figure 2.3

Technology Aids At this point, after a discussion of the abovementioned techniques for information visualization, one may be left to wonder if making such models explicit is hard to do. In truth, the answer to this question is no, in large part due to technological advances that do
   

16 
 

away with much of the burden of creation for these models. In fact, the relatively recent emergence of technological aids designed to augment the creation of conceptual models, concept maps and mind maps has brought about an increased interest in these techniques. Compared to the traditional pencil and paper construction, software programs make the editing and formatting of such models considerably easier. This evolution in

functionality encourages users to revise and expand their maps as their knowledge base regarding a specific topic grows and changes.22 Christina De Simone, in “Applications of Concept Mapping,” states that “by externalizing information as they create concept maps, students are better able to detect and correct gaps and inconsistencies in their knowledge.”23 However, the development and evolution of concept maps as a response to detailed questions and requirements can sometimes prove difficult, lengthy and disorganized when created by hand. De Simone, Assistant Professor of Education at the University of Ottawa, further states that many students in her classes “find electronic concept mapping …very useful, as it minimizes the cumbersome and time-consuming activity of erasing, revising, and beginning anew. It allows them greater freedom to adjust their conceptual thinking and mapped representations.”24 Technologically based conceptual modeling aids, of which a wide variety exist, are likely the most useful and advantageous tools for intelligence students and analysts to utilize in the construction of complex and fluid conceptual models.

                                                            
Josianne Basque and Beatrice Pudelko. “Using a Concept Mapping Software as a Knowledge Construction Tool in a Graduate Online Course,” in Proceedings of ED-MEDIA 2003, World Conference on Educational Multimedia, Hypermedia &Telecommunications, Honolulu, June 23-28, 2003, ed. D. Lassner and C. McNaught (Norfolk: Association for the Advancement of Computing in Education, 2003), 2268-2274. 23 Christina De Simone, “Applications of Concept Mapping,” Journal of College Teaching 55, no. 1 (2007): 34. 24 De Simone, “Applications of Concept Mapping,” 35.
22

   

17 
 

Learning Styles However, even with the increased efficiency in creating conceptual models brought about through technological aids, it is important to note that some individuals may be better suited towards this type of exercise than others. Research conducted by Josianne Basque and Beatrice Pudelko, from the LICEF Research Center in Canada, showed that some graduate students claiming to be auditory learners found little to no utility in constructing a visual representation of knowledge in the form of a concept map.25 However, other students identifying themselves as visual learners claimed to better understand a topic when concepts were structured in such a visual way.26

Hypotheses Based on review of the above literature the following hypotheses were formed: First, intelligence analysts who engage in ECM will generate better analytic products, as evaluated by thoroughness of process and accuracy of product, than analysts who do not. Second, that the individual to group method employed for creation of the conceptual models in this study will affect, either positively or negatively, the models’ ability to aid in intelligence analysis.  

                                                            
25 26

Basque & Pudelko, Using a Concept Mapping Software, 2268-2274. Ibid.

   

18 
 

CHAPTER III: METHODOLOGY

In order to test the stated hypotheses I conducted an experiment that examined the value of ECM as it applies to the quality of the analysis produced. The experiment was designed to determine if intelligence analysts who engaged in ECM would generate better analytic products, as evaluated by thoroughness of process and accuracy of product, in comparison to analysts who did not. The following methodology section will provide the details of this experiment.

Research Design In conducting my experiment, I divided the subjects into an experimental and a control group. Efforts were made to ensure that conditions in both groups were identical, with the exception of the addition of ECM in the experimental group. The experimental group was instructed to use a structured ECM approach, facilitated by the use of the open-source program Bubbl.us (a free, internet-based conceptual modeling program), to aid in their analysis. The control group on the other hand was not instructed to use any particular method in conducting their analysis, as this group served as a baseline from which to measure the experimental group against.

Subjects In order to better relate the findings of my experiment to the United States Intelligence Community, I chose to draw from the undergraduate and graduate student population at the Mercyhurst College Institute for Intelligence Studies (MCIIS). The aim

   

19 
 

of this program is to produce graduates qualified to enter the government or private sector as entry-level intelligence analysts and it was therefore considered to be an appropriate pool from which to draw a sample. Mercyhurst College houses the oldest institution in the world dedicated specifically to the study of intelligence analysis. The program offers coursework in the fields of national security, law enforcement and competitive business analysis. Students in both the undergraduate and graduate programs are subjected to a rigorous academic curriculum during their time at Mercyhurst. They are expected to meet certain foreign language proficiency and internship requirements and are often faced with accelerated project deadlines for real-world decision makers in the field of intelligence. MCIIS considers their students to be experts in the exploitation of open source information for analytic purposes. For these reasons, MCIIS turns out capable and well-trained entrylevel analysts with a wide variety of analytic skills and abilities. To obtain participants for my experiment I contacted professors within the Intelligence Studies program at Mercyhurst College via email to ask if they would be willing to let me briefly speak to their classes in order to recruit students. I then followed up in-person with all professors who responded to my emails positively to setup a schedule for class visits that was convenient for them. After establishing this schedule I made appearances in a variety of undergraduate and graduate intelligence classes where I gave a broad overview of the experiment and passed out individual signup sheets to interested students (see Appendix 1). These sheets asked for the student’s name, email address, class year and available time slots for experiment participation. Students were given a total of twelve time slots to choose from,

   

20 
 

with six slots on the first day of the experiment and six slots on the second day. The only requirement placed on students was that they choose at least two time slots, one on each day of the experiment. Additionally, of these two time slots students were asked to select one for the duration of ninety minutes and the other for the duration of thirty minutes. Students could either turn these sheets back into me on the spot or they could drop them off at their convenience to my worksite located within the Intelligence Studies building. After receiving all signup sheets students were divided into subgroups based on class year. Individuals within each subgroup were then randomly divided between the control and the experimental group in an attempt to control for the educational level of participants. For example, after the subgroup of freshman who signed up for the

experiment was established, individuals within the group were randomly assigned to either the control or experimental group. Students were then notified of their designated time slots for participation via an email, which included information on the location of the experiment. The email also stated that participation would be rewarded with extra credit from select professors within the Intelligence Studies Department, in addition to free refreshments and pizza on the second day of the experiment. Finally, my contact data was included in the email so all participants would easily be able to access me if they had any questions or concerns in the days leading up to the experiment. Intelligence Studies students of all grade levels were welcome to participate as any information needed to complete the experiment would be provided during the sessions. Testing took place approximately three-quarters of the way through the fall term, unfortunately falling at a time when students were especially busy trying to meet the

   

21 
 

demands of their coursew t work. A tot of 47 stu tal udents (25 c control and 2 experime 22 ental) ac ctually parti icipated in, a finished this exper and d, riment. For a breakdow of participants wn by educationa level and g y al group, pleas see Figure 3.1 & 3.2 below. se es Figure 3.1

Subject Educat tion Level
15
14 10 7 6

Number Of Subjects

10 5

9

1

0 Freshman Sophomore Junior Senior 1st Year  Graduate G 2nd Year  2 Graduate G

Year In S School Figure 3.2

Subject Ed ducation L Level By Gr roup
10 10
Number Of Subjects

8 5 4 2 4 4

5

3 1 0

3 3

Con. C Exp. E

0

Year In Scho ool

   

22 
 

Preliminaries Prior to conducting the experiment, I had to submit a detailed research proposal to the Mercyhurst College Institutional Review Board (IRB) for approval (see Appendix 2). It is Mercyhurst College policy that any student conducting research involving the use of human subjects be granted permission by the IRB. In order to receive a green light from the IRB students must provide a description of the proposed research, its purpose, and an explanation of any potential dangers (physical or psychological) that could befall individuals as a result of their participation in the experiment. However, not only did I need to secure the consent of the IRB I also needed the consent of each individual experiment participant. Therefore, on the first day of the experiment (for both control and experimental sessions) all participants were given a formal consent form upon arrival (see Appendix 3 for control group consent form and Appendix 4 for experimental group consent form). This form outlined what would be expected of them as participants, along with the fact that there were no foreseen dangers or risks associated with involvement in the experiment. The form also asked for basic contact information such as name, class year, and telephone number.

Control Group: Day 1 Control group participants were asked to attend two experiment sessions. The first session was slotted for thirty minutes and the next session, scheduled for one week later, was slotted for 90 minutes. Three control group sessions were run on both days of the experiment, for a total of six sessions, in order to make it as convenient as possible for subjects to schedule participation into their busy agendas. Out of those who

originally signed up for the experiment and were assigned to the control group (37
   

23 
 

in ndividuals), a total of 2 actually a 25 attended and completed the experim d ment. Please see Figure 3.3 below for a gra aphic representation of t breakdow by class year. this wn Figure 3.3 ontrol Group p Vs. Actual Control Gro oup By Class Year Original Co 12 2 12
Number Of Subjects

10 8 6 4 2 0 7 5

9

10

5 2

4 1 1

3 3

Original  Con. Actual  Con.

Ye ear In School

Both groups were given the same quest e tion to analy yze, regardin October 2008 ng pr residential elections in Z e Zambia (see Appendix 5 The con e 5). ntrol group w simply a was asked to forecast the winner of the elections and to prov a list of the main pi o s vide f ieces of evid dence th aided the in their a hat em analysis. Bo groups w oth were provided with inform d mation rega arding so ource reliabi ility and ana alytic confid dence as they were asked to supply m y d measures of both f in their final product (see Appendix 6). Also, both groups w n e were given a semi-struc ctured an nswer sheet with space for their n t e name, a pre-written fore ecast with b built-in word of ds es stimative pro obability and presidentia candidates to choose f d al s from and spa for a bul ace lleted discussion (se Appendix 7). The bo ee x ottom of the answer shee also asked them to ide et d entify

   

24 
 

their source reliability and analytic confidence on a scale from low to high and to provide the names of their professors offering extra credit for participation in the experiment. Lastly, participants were given a sheet of expectations for the second and final session of the experiment one week later (see Appendix 8) and were asked to fill out a short pre-experiment questionnaire (see Appendix 10). They were also once again

provided with my contact information in case they encountered problems or had questions while working on their analysis during the course of the week (see Appendix 11).

Experimental Group: Day 1 Experimental group participants were also asked to attend two sessions. The first session was slotted for ninety minutes and the next session, scheduled for one week later, was slotted for thirty minutes. Three experimental group sessions were run on both days of the experiment, for a total of six sessions, once again to make scheduling more convenient for participants. Out of those who originally signed up for the experiment and were assigned to the experimental group (37 individuals), a total of 22 actually attended and completed the experiment. Please see Figure 3.4 below for a graphic representation of this breakdown by class year.

   

25 
 

Figure 3.4 Original Exper rimental Gro oup Vs. Actu ual Experime ental Group By Class  Year 15 10 10 5 0 7 4 8 6 4 3 0 0 3 3
Original Exp. Actual Ex xp. 

Number Of Subjects

11 1

Ye ear In School

The experimental group was given the same taskin as the co e l s ng ontrol group (see p Appendix 5). This group was also p A p provided with the same i h information regarding so ource re eliability and analytic co d onfidence as the control group (see Appendix 6 along wit the s l 6), th sa answer sheet (see A ame Appendix 7). Although t group w given the same tasking as . this was e th control group in regards to fore he g ecasting the winner of t elections they were also the s, e re equired to us ECM to a se assist them in completing this endeav n g vor. There efore, I bega by giving a lecture, approximate ten minu an g ely utes in lengt to th, fa amiliarize ex xperimental group participants with what conce h eptual mode are, how they els w ca be used and the prop an a posed value of making them explic in the fiel of intellig cit ld gence (s Appendi 12). Fol see ix llowing the lecture, I ha all partic ad cipants sign into a com mputer whereby I le them th w ed hrough a ste ep-by-step t tutorial of the program Bubbl.us (see m Appendix 13) Once eve A ). eryone was c comfortable w how th program w with he worked, I beg a gan st tructured EC exercise. CM .
   

26 
 

Participants began the exercise by making individual lists of concepts they felt would be important to answering the question asked of them. After individual lists were completed, participants were asked to read their lists aloud one at a time. As concepts were read off, they were written on a whiteboard at the front of the room, creating one combined group list. For every time a concept was repeated a check mark was placed next to it in order to highlight the most commonly thought of concepts. Also, concepts that the group immediately recognized as useful but that were mentioned only once or twice were made note of as well. Due to limitations on time, participants were not asked to engage in a convergent thinking exercise as a group, whereby they would critically evaluate the master list before producing their own conceptual models. Instead,

participants were simply asked to use Bubbl.us to construct a conceptual model based on their individual list and thoughts as well as that of the collaborative group list. Finally, participants were given a chance to briefly look at the way others around them had assembled their models and were then asked to electronically share what they had created with me through the collaboration function within Bubbl.us (see Appendix 15). Following this task participants were given a sheet of expectations for the second and final session of the experiment one week later (see Appendix 9) and were asked to fill out a pre-experiment questionnaire (see Appendix 10). Lastly, they were provided with a sheet of my contact information (see Appendix 11).

Bubbl.us Bubbl.us is a free, internet-based, conceptual modeling program. It was chosen for use in this experiment due to its extremely simple user-interface. Not only did the program encompass all the relevant functions necessary to complete the conceptual
   

27 
 

modeling seg m gment of my experimen it could easily be ta y nt, aught and le earned within the n minimal amo m ount of time I had durin sessions. Basic func ng ctions includ the creatio of de on bu ubbles and lines to illus l strate concep and their relationshi internetpts r ips, -based sharin of ng work with oth Bubbl.us users, and exporting fi w her s inished prod ducts as phot or embed tos dding th hem into a web page. P w Please see Fi igure 3.5 bel low for a sa ample produc taken from the ct m Bubbl.us web B bsite, illustra ating a variet of the program’s featu ty ures Figure 3.5

Con ntrol Group: Day 2 : On the second da of the exp ay periment, the control gro was expected to arri at e oup ive heir ted with ompleted ans swer sheets r ready to turn in. All rese n earch th designat session w their co an analysis was to be do prior to arriving at t session. Since the c nd w one this control group had p not received any training on Bubbl.us in the first session, I b a t began their s second sessio by on as sking them to login to a computer a follow a t and along with m as I taugh them the basic me ht fu unctions of the program However, they still r t m. received no lectures deta ailing conce eptual

   

28 
 

modeling background information and were not given any specifics regarding how to actually construct their models in Bubbl.us (see Appendix 14). After this group became familiar with the program they were asked to illustrate the concepts and relationships they found to be important over the course of the week in answering the question posed to them. This was done in order to draw a comparison between the quality of conceptual models made after background information was provided and those made with little to no prior instruction. Additionally, it compared the quality of conceptual models made pre-collection and updated throughout the analytic process with those made post-analysis. In bringing the session to a close participants were asked to fill out a post-experiment questionnaire (see Appendix 16) and were given a debriefing sheet thanking them for their time and further explaining the purpose of the experiment (see Appendix 18).

Experimental Group: Day 2 Once again, the experimental group was expected to arrive on the second day of the experiment with their research and analysis completed and a finished answer sheet ready to be handed in. Additionally, they were expected to have electronically updated the conceptual models made during the first session of the experiment throughout the course of the week to reflect their expanding knowledge base in regards to the question at hand. Therefore, after handing in their answer sheets this group was asked to fill out a follow-up questionnaire (see Appendix 17) and was then provided with the same debriefing sheet as the control group (see Appendix 19).

   

29 
 

Data Analysis Procedures Since the intent of this experiment was to test the value of ECM as it applies to analysis, control and experimental group results were compared in terms of quality and accuracy of process and product. To evaluate and compare the processes of the two groups, three MCIIS second year graduate students independently ranked the discussion section of each participant’s answer sheet from best to worst. Students were used in lieu of professors who tend to have unique grading styles, as the students all received the same training regarding what makes a sound analysis and were thus thought to be on more equal footing. All identifying information including the student’s name, class year and group were removed prior to evaluation. Additionally, the students doing the

evaluating did not know the outcome of the elections at the time of ranking in order to keep measures of process and product independent of one another. The product measure was derived through a simple tally of whether or not the participant predicted the question correctly. Finally, the actual conceptual models created were compared in terms of complexity, based on how many concepts and connections between concepts each encompassed. All measurements and rankings were compiled into a Microsoft Excel Spreadsheet along with subject education level breakdowns and information from the pre- and post-experiment questionnaires. Statistical analyses were then undertaken to determine whether or not the experiment results were statistically significant. The

control group was generally expected to fall lower in the graduate student process rankings than the experimental group and was also expected to be less accurate overall in terms of forecasting the outcome of the election.

   

30 
 

CHAPTER IV: RESULTS

The results of the ECM experiment generated a variety of interesting and surprising results. The next section will first provide a brief explanation of the statistical significance testing conducted throughout this thesis and will then detail the results derived from the analysis of pre- and post-experiment questionnaires. Next, findings from the experiment itself will be discussed as a function of process and product, with experimental and control group findings initially reported on individually, followed by a comparison of the groups to one another.

Significance Testing All significance tests related to this thesis were conducted at the 0.10 significance level (see Appendix 20). The reason behind setting what some may consider to be a rather lax level of significance is the fact that the research conducted in this thesis regarding ECM and intelligence analysis is exploratory in nature. According to G. David Garson, Professor of Public Administration at North Carolina State University, in Guide to Writing Empirical Papers, Theses, and Dissertations, “It is inappropriate to set a stringent significance level in exploratory research (a .10 level is acceptable in exploratory research).”27 Although debate remains lively amongst researchers regarding proper significance levels based on situation, this author felt that a 0.10 level was most appropriate when dealing with this particular set of research and data.

                                                            
27

G. David Garson, Guide to Writing Empirical Papers, Theses, and Dissertations (New York: CRC Press, 2002), 199.

   

31 
 

Pre- and Post-Questionnaire Results Prior to the experiment participants were asked whether or not they thought they would be able to dedicate a sufficient amount of time to completing the experiment over the course of the next week. In response, 64.3% of control group participants thought that they would have ample time, 32.1% were not sure and 3.6% did not expect to be able to dedicate a sufficient amount of time to the experiment. On the other hand, 58.3% of experimental group participants expected to have enough time and 41.7% were unsure. When asked post-experiment whether or not they had actually been able to devote a sufficient amount of time to the experiment over the course of the past week, 68% of control group participants claimed they had, 16% were unsure and 16% said that they had not. In the experimental group 60% claimed to have had enough time to dedicate to the experiment, 20% were unsure and 20% had not. While the percentage of participants in both the control and experimental group claiming to have been able to dedicate a sufficient amount of time to completion of the experiment increased slightly from pre- to post-experiment, the percentage of those who claimed that they did not have a sufficient amount of time to dedicate to the experiment increased from pre- to post-experiment more substantially. Post-experiment 20% of the experimental group claimed that they had not had enough time (up from 0% preexperiment) and 16% of control group participants claimed the same (up from 3.6% preexperiment). As previously stated in the methodology section of this thesis the timing of the experiment fell during an extremely busy time in the participants’ trimester, serving as a limitation to this study as students had to balance the experiment with their class

   

32 
 

work and oth responsib w her bilities. Plea see Figur 4.1 below for a graph display o this ase re w hic of data. Figure 4.1

Pre‐ Vs. Post‐Experimen nt: Able To o Dedicate A  ufficient Am mount Of T Time To Ex xperiment? ? Su
20 15 10 5 0 Con ntrol Exper rimental Co ontrol Experimental 32 2.1% 3.6% 64.3% % 58.3% % 41.7% 4 16% 16% 0% 68% 0% 60 20% 20% Yes Maybe No

Pre‐Experimen nt

Post‐Experim ment

e-experimen all partic nt cipants wer asked t identify how re to Additionally, pre nterested the were in the study, w ey with 1 being not intere g ested and 5 being extre emely in in nterested. The average r T response for both groups was a 3.5, illustrating that both co r ontrol an experime nd ental groups were on a s average equ ually interest in the e ted experiment at its on nset. The di ifference bet tween contro and exper ol rimental group response to this que es estion was w not found to be statis d stically signi ificant at the 0.10 level ( (p-value = 0. .851). When asked the same quest n tion post-ex xperiment, t average response o the the of co ontrol group increased to 3.8, while the avera response of the exp p age e perimental g group re emained the same at 3.5 This show a slight a 5. ws average incre ease in intere on the pa of est art th control gr he roup from p to post-experiment. However, once again the differen in prence

   

33 
 

control and experimental group responses was not found to be statistically significant at the 0.10 level (p-value = 0.267). Another question asked of participants prior to their involvement in the experiment was how useful they feel structured approaches to the analytic process are, with 1 being not useful and 5 being extremely useful. On average, control group

participants responded with a 3.8 and experimental group participants responded with a 4.2. Although a significant difference at the 0.10 level was not found between control and experimental group responses to this question, the results did approach significance (p-value = 0.127). Faced with the same question post-experiment the average control group response increased to a 4.1, while the experimental group response maintained steady at 4.2. However, the difference in control and experimental group post-experiment responses was not found to be significant at the 0.10 level (p-value = 0.617). Results for this question show that the experimental group found structured approaches to the analytic process to be more useful than did the control group, both pre- and post-experiment. The control group’s feelings regarding the utility of structured approaches to the analytic process grew throughout the course of the experiment, whereas the experimental group’s did not. When experiment participants were asked to identify the learning style they most closely associated with, over half of both control and experimental group participants identified themselves as visual learners. Since ECM is a visual learning aid, it is likely that the exercise was generally more beneficial to those claiming to be visual learners than to those who chose an alternative learning style. Please see Figure 4.2 below for the

   

34 
 

fu range of control and experimenta group resp ull al ponses to the question re e egarding lea arning st tyles. Figure 4.2 2

Control Vs. Exper rimental G Group: Learning Style e
Unsu ure Kinesthetic &  Tactile Visu ual Audito ory 0
4.2% % 6.9 9% 12.5% 1 3.4% % 29.2% 31% 54.2% 58.6 6%

Experi imental  Contro ol

10

20 0

ment both co ontrol and e experimental groups wer asked to g l re gauge Also, post-experim th level of understandin of concep heir ng ptual modeli prior to t study on a scale of 1 to 5, ing the n with w 1 being extremely low and 5 being extrem g mely high. The averag control g ge group re esponse to this question was a 3.6, whereas th average e t n , he experimental group resp ponse was w a 3.2. While the d W difference be etween contr and expe rol erimental re esponses wa not as fo ound to be significant at the 0.10 le s t evel, the resu did appr ults roach signifi icance (p-va alue = 0.187). experiment b both groups were also asked to ide entify their u understandin of ng Post-e co onceptual modeling foll m lowing the e experiment, using the s same scale. Post-experi iment th average response for both the c he r r control and experimenta group wa a 4.0 and was al as d th herefore not statistically significant at the 0.10 level (p-valu = 1.0). R y ue Results from this m qu uestion show that post w t-experiment both contr and expe t rol erimental gr roup participants
   

35 
 

claimed to have the same understanding of conceptual modeling, an increase for both groups from their pre-experiment knowledge on the topic. However, the experimental group claimed to have less of an understanding of conceptual modeling than the control group at the onset of the experiment, signaling that on average the experiment raised their knowledge of conceptual modeling more than it did the control group’s. In relation to the above question, post-experiment all participants were asked how often ECM had been a part of their personal analytic process prior to the experiment on a scale of 1 to 5, with 1 being never and 5 being every time they produced an intelligence estimate. On average the control group responded with a 2.8 and the experimental group responded with a 2.75. The difference between control and experimental group responses was not found to be statistically significant at the 0.10 level (p-value = 0.872). However, post-experiment both groups were also asked how often they plan to incorporate ECM into their personal analytic process in the future, using the same scale. In response to this question, the control group average was a 3.5 and the experimental group average was a 3.6. Once again, the difference between experimental and control group responses was not found to be statistically significant at the 0.10 level (p-value = 0.617). Results from the above question highlight that although the control group

claimed to employ ECM in their analytic process prior to the experiment on average slightly more than the experimental group, the experimental group claimed that they will employ ECM in their analytic process on average more than the control group in the future. Post-experiment the experimental group was asked a series of four questions regarding their specific responsibilities within the experiment. First, the experimental

   

36 
 

group was asked to rate whether or not they found that ECM aided them in developing a more thorough and nuanced intelligence analysis in this experiment. In response, 33% of experimental group participants claimed that ECM definitely aided them in their analysis, and 66.7% claimed that it helped them somewhat, with no participants responding that it did not help at all. The experimental group was also asked post-experiment to rate how useful they found the conceptual modeling training provided at the beginning of the experiment to be, with 1 being not at all helpful and 5 being extremely helpful. On average,

experimental group participants responded to this question with a 3.9. Additionally, the experimental group was asked how effective they found the conceptual modeling method used in this experiment, inclusive of both individual work and group collaboration, to be. The average response to this question was a 3.7. Finally, the experimental group was asked how useful they found the technology aid, Bubbl.us, to be in creating and updating their conceptual models, with 1 being not useful and 5 being extremely useful. Overall, experimental group participants found Bubbl.us to be quite valuable, averaging a response of 4.1.

Process - Conceptual Model Findings The 2007 Intelligence Community Directive Number 203 on analytic standards confirms that, “To the extent possible, analysis should incorporate insights from the application of structured analytic technique(s) appropriate to the topic being analyzed.”28 As such, the conceptual models resulting from the experiment were analyzed in terms of complexity by simply tallying the number of concepts and connections between concepts
                                                            
United States Government, Intelligence Community Directive Number 203, June 21, 2007, http://www.fas.org/irp/dni/icd/icd-203.pdf (accessed January 26, 2009).
28

   

37 
 

fo ound in each model (please see Figu 4.3 below to see the distinction b h ure w between con ncepts an connectio nd ons). Contr group co rol onceptual m models avera aged 12.6 co oncepts and 12.9 co onnections, per model. p Figure 4 4.3

Concept 

Concept 

Connection

Connection 

Pre-an nalysis expe erimental gr roup concep ptual models averaged 25 concepts and s s 28.9 connect tions, per m model. Pos st-analysis e experimenta group conceptual m al models

av veraged 30.9 concepts a 31 conn 9 and nections, per model. Re r esults of sig gnificance te esting fo the numb of concep in pre- a post-exp or ber pts and perimental g group concep ptual models was s fo ound to be significant at the 0.10 level (p-va 0 alue = 0.056), however the number of r co onnections was not foun to be sig w nd gnificant (p-v value = 0.23 31). As illu ustrated belo in ow Figure 4.4, bo the average number of concepts and the ave oth s erage numbe of connec er ctions between conc cepts increa ased between pre-analys and post n sis t-analysis co onceptual m models within the exp w perimental g group.

   

38 
 

Figure 4.4 4

Exp perimatal G Group: Pre Vs. Post‐Analysis  e‐ Averag ge Concept ts & Conne ections Per Conceptu ual  Mode el
40 4 30 3 20 2 10 1 0 Con ncepts Connecti ions 25 2 30.9 31

28.9

Pre‐An nalysis Post‐A Analysis

When looking a experime n at ental group conceptual models as a whole (not l g pre- and pos st-analysis) the models averaged 27 concepts and 7.9 s distinguishing between p 29.9 connecti ions, per mo odel. As illu ustrated belo in Figure 4.5, when comparing these ow e xperimental group aver rages with control group averages t differenc in compl p the ce lexity ex between the two groups’ models beco t omes quite o obvious.

Figure 4.5

Con ntrol Vs. Ex xperimenta al Group: A Average  Concep pts & Conn nections Pe er Concept tual Mode el
30 20 10 0

27.9

29.9 Contr rol

12 2.6

12.9

Exper rimental

Con ncepts

Connect tions
 

 

39 
 

The control group’s models were much simpler, consisting on average of less than half the amount of concepts and connections between concepts present in experimental group models. Results of significance testing for the number of concepts in control versus experimental group conceptual models was found to be significant at the 0.10 level (p-value = 0.000), furthermore the number of connections was also found to be significant (p-value = 0.000). Please see Figure 4.6 below for an illustration of a typical conceptual model made by a control group participant and Figure 4.7 for an illustration of a typical conceptual model made by an experimental group participant. Figure 4.6

   

40 
 

Figure 4.7

Process – Logic/Quality of Supporting Evidence Findings Intelligence Community Directive Number 203 highlights the need for logical argumentation within analytic products, stating that, “Analytic presentation should facilitate clear understanding of the information and reasoning underlying analytic judgments.”29 As a result, three Intelligence Studies graduate students independently ranked the analytic products of all 47 participants from best to worst, based on the quality and logic of evidence supporting the analyst’s estimate. Based on the rankings assigned by each graduate student, the overall average ranking of control group participants was a 22.4 and the overall average ranking of experimental group participants was a 25.8. Therefore, control group participants scored approximately 3 points higher than did experimental group participants in terms of the reasoning used to substantiate their
                                                            
29

Ibid.

   

41 
 

estimates, implying that participants not using ECM to aid in their analysis were able to formulate slightly better supporting arguments than participants who did in fact use ECM. Correlation scores amongst the three graduate student rankers at the source of this finding were relatively high across the boards (0.76, 0.72 and 0.63), illustrating consistency in experiment participant scoring.30 As Jacob Cohen, an influential

statistician and Professor Emeritus at New York University before his death in 1998, discusses in Statistical Power Analysis for the Behavioral Sciences, according to convention, correlations above a 0.5 are traditionally considered to be large within the social sciences.31 This lends support to the Mercyhurst method for the evaluation of analytic products, as the ranking consistency of the three graduate student raters was high.

Product – Forecasting Findings According to Intelligence Community Directive Number 203, analytic products should “make accurate judgments and assessments.”32 Therefore, not only must the

quality of the analyst’s process be accounted for, but the correctness of estimates must be measured as well. In terms of forecasting the correct outcome of the October 2008 Zambian presidential elections, 68% of control group participants predicted accurately, whereas only 40.9% of experimental group participants did. Therefore, individuals who
                                                            
 The common measures of inter rater-reliability, known as Cohen’s and Fleiss’ Kappa, were not used when conducting tests of correlation in this study. This is due to the fact that both measures are designed for use in situations where the data is categorical (ex. yes vs. no) and were therefore felt by the author to be inappropriate measures for the type of data present (ordinal numbers). 31 Jacob Cohen, Statistical Power Analysis for the Behavioral Sciences (Philadelphia: Lawrence Erlbaum Associates, 1988). 32 United States Government, Intelligence Community Directive Number 203.
30

   

42 
 

did not use ECM to aid i their anal E in lysis were ab to identi the actua outcome o the ble ify al of re esearch ques stion posed to them much more oft than indi ten ividuals who did incorp o porate ECM into th E heir analytic process. c Forecasting result dif g fferences we found t be ere to

st tatistically significant at the 0.10 le t evel (p-value = 0.065). Please see F e Figure 4.8 b below fo a graphic representatio of this da or on ata. Figure 4.8

Control V Vs. Experim mental Group:       Accur racy Of For recasting 
20 15 10 5 0

68 8% 59.1% % 32% 40.9% Contr rol Experimental 

Inaccurate 

Acc curate 

Qu uality Of Sup pporting Evi idence Vs. F Forecasting A Accuracy Fin ndings After looking at both the qu uality of th evidence supporting the particip he pants’ an nalysis and the participa t ants’ forecasting accurac separately the opport cy y, tunity to com mpare th two findi he ings present itself. T ted Therefore, t followin suppleme the ng entary conclu usion re egarding gra aduate student process rankings a and forecast ting accurac although not cy, h directly relate to the hyp ed potheses of t experime was thou this ent, ught to be in nteresting en nough to warrant me o entioning at this time. Out of curiosity, graduate student process rankings, essentiall measuring the o ly g qu ualitative str rength of the individual’ assessmen were com e ’s nt, mpared again whether o not nst or

   

43 
 

the individual correctly forecasted the outcome of the election. To make this comparison the process rankings were simply split in half and a tally of the amount of individuals in the top half and bottom half who forecasted the elections correctly was conducted. This measurement was carried out three times: first for the group as a whole, second for just the control group and lastly for just the experimental group. The expectation following this comparison was that individuals in the top half of the graduate student process rankings would forecast the winner of the elections correctly considerably more often than individuals falling in the bottom half of the rankings. However, this was not found to be the case. In fact, results of the tally showed little difference in forecasting accuracy between those who were ranked better qualitatively than those who were not. When looking at the group as a whole, 14 individuals in the top half of the graduate student process rankings forecasted the outcome of the elections correctly and 9 individuals forecasted incorrectly, compared with an even split in the bottom half of 12 individuals forecasting correctly and 12 forecasting incorrectly. When looking at solely the control group, 9 individuals in the top half of the graduate student process rankings forecasted the outcome of the elections correctly and 4 individuals did not, compared to 8 individuals who forecasted correctly in the bottom half and 4 who did not. Finally, when looking at the experimental group on its own, there was an even split of 5 individuals in the top half of the graduate student process rankings who forecasted correctly and 5 who did not, compared to 4 individuals in the bottom half who forecasted correctly with 8 who did not. Please see Figures 4.9, 4.10 and 4.11 below for graphic representations of this data.

   

44 
 

Figure 4.9

Foreca asting Accu uracy: Top p Vs. Bottom Half Of  Graduate Student Pro G ocess Rank kings
14 15 9 10 5 0 Cor rrect Inc correct Correct C Incorrect I 12 12

Top Half

Bottom Half

Figure 4.1 10

Contr rol Group F Forecasting Accuracy y: Top Vs.  Bott tom Half O Of Graduate Student Process  gs Ranking
10 8 6 4 2 0 9 8 4 4

Corr rect

Inco orrect

Correct

Incorrect

Top Half

Bottom Ha alf

   

45 
 

Figure 4.1 11

Experim mental Gro oup Foreca asting Acc curacy: Top p  Vs. Bo ottom Half Of Gradua f ate Studen nt Process  Ranking gs
8 8 6 4 2 0 Corre ect Inco orrect Correct C Incorrect I 5 5 4

Top Half

Bottom Ha alf

Althou ugh a stric methodol ct logy was n applied in reaching this parti not g icular co onclusion, making it dif m fficult to ascertain the ex xtent to whic any extran ch neous variab or ble variables has impacted it, the point in its most basic form rem , n mains the sam me. efore, this co onclusion sug ggests that in ndividuals w are bette writers, or who who er r There ar able to craft more convincing arguments, are not ne re c ecessarily an nymore like to ely fo orecast corre ectly than in ndividuals w are lack who king in thos skills. T se This notion is an offshoot of th general a he argument m made in Phili Tetlock’s Expert Political Judgm ip s ment, which basica states th the way in which w reason or think abou things is more w ally hat we r ut im mportant tha our backg an grounds and accomplishm ments or eve our belief systems.33 How en f we w think, then appears to be more im o mportant tha what we t an think when i comes to b it being pr roficient for recasters.

                                                            
33

Philip E. Tetlo Expert Po ock, olitical Judgme (Princeton: Princeton University Press, 2 ent 2005). 

   

46 
 

Product – So ource Reliab bility and An nalytic Confi fidence Findi ings Intelli igence Com mmunity Dir rective Num mber 203 st tates that a analytic products hould “prop perly describ quality an reliability of underly be nd y ying sources and “pro s” operly sh
4 ca aveat and ex xpress uncer rtainties or c confidence in analytic ju n udgments.”34 As a result, all

ex xperiment participants w p were asked to assess b both their so ource reliabi ility and ana alytic co onfidence on a scale f o from low to high. o Co ontrol group findings r p regarding so ource

re eliability illu ustrate that 4 of partic 4% cipants claim low sou med urce reliabili 80% cla ity, aimed medium and 16% claim high. F m med Findings for the experim mental grou show 4.5% of up participants claimed low reliability, 68.2% clai c w imed medium and 27.3% claimed high. m Although pe A ercentages reveal that approxima ately 11% more experimental g group participants claimed to ha high sou c ave urce reliabili than did control grou participan it ity up nts, s t this nce two particip pants. As a r result is necessary to note that t differen is a function of just t so ource reliabi ility finding were not found to be significant at the 0.10 level (p-val = gs e lue 0.451). Pleas see Figure 4.12 below for a graph representa se e w hic ation of this data. Figure 4.12 e

Contro ol Vs. Experimental G Group:       Source Reliability                    
80 0% 20 15 10 5 0 Lo ow
                                                            
34

68.2% Co ontrol 27.3% 4% 4.5% % Me edium 16% Ex xperimental

High

Ibid.

   

47 
 

In te erms of anal lytic confide ence, 12% of control gro participants claimed low oup d an nalytic conf fidence, 80% claimed m % medium and 8% claimed high. On the other h n hand, 22 2.7% of ex xperimental group part ticipants cla aimed low confidence, 63.6% cla aimed medium and 13.6% claim high. A m med Although pe ercentages re eveal that ap pproximately 6% y more experim m mental group participant claimed to have high analytic con p ts o nfidence tha did an co ontrol group participants, it is neces p ssary to note that this di e ifference is a function of just on participa ne ant. As a r result, findin for ana ngs alytic confid dence were not found t be to si ignificant at the 0.10 lev (p-value = 0.745). Pl vel lease see Fig gure 4.13 below for a gra aphic re epresentation of this data n a. Figure 4.13 3

Control Vs. Experime C ental Grou up:       lytic Confid dence Anal
20 15 10 5 0 Low Medium High H 12% 22 2.7% 8% 8 13.6% 80% 63.6% Contro ol Experim mental

   

48 
 

CHAPTER V: CONCLUSIONS

As previously stated, the purpose of this study was to determine the value of ECM within the analytic process. This was accomplished by requiring the experimental group to incorporate the use of conceptual models, created through a structured individual to group approach, into their analytic process. The control group on the other hand was simply asked to analyze the question posed to them, using no particular method. While the control group correctly predicted the outcome of the elections 68% of the time, the experimental group forecasted the outcome correctly only 40.9% of the time. This result was found to be statistically significant at the 0.10 level (p-value = 0.065). Additionally, although the difference was marginal, a larger percentage of

experimental group participants claimed to have high source reliability and analytic confidence than did control group participants. Therefore, the experimental group

members did considerably poorer in terms of correctly forecasting the result of the elections, but felt they had more reliable sources and were more confident in their assessment. However, neither source reliability (p-value = 0.451) nor analytic confidence (p-value = 0.745) results were found to be statistically significant at the 0.10 level. Even so, in terms of product measures only, these results paint a bleak picture of the role of ECM within the analytic process. Turning to process measurements, results of graduate student rankings placed control group participants, on average, roughly 3 points higher than experimental group participants in terms of the quality and logic of the evidence used to support their analysis. Correlation scores amongst the three graduate student rankers were relatively
   

49 
 

high across the boards (0.76, 0.72 and 0.63), illustrating consistency in experiment participant scoring. Furthermore, the experimental group’s conceptual models were

appreciably more complex than the control groups in terms of the amount of concepts and relationships between concepts. This result was found to be statistically significant at the 0.10 level (p-value = 0.000). Therefore, although the experimental group’s conceptual models appear to be more complex and thorough than the control group’s models, they scored lower on average in regards to the reasoning used to substantiate their estimates. Once again, these results appear largely to invalidate any suggested value of ECM within intelligence analysis. Since the literature strongly suggests that ECM will improve

analysis, what could account for these counter-intuitive results?

Excessive Possibilities Confuse Sheena S. Iyengar and Mark R. Lepper in, “When Choice is Demotivating: Can One Desire Too Much of a Good Thing?,” state that, “It is a common supposition in modern society that the more choices the better—that the human ability to manage, and the human desire for, choice is infinite.”35 Traditionally, research has tended to support the concept that having some choice produces better outcomes than having no choice. However, a growing body of literature concludes that when the amount of choices available becomes too large, people have a very hard time managing that complexity. As a result, Iyengar, professor in the Management Department of the Columbia Business School, and Lepper, professor of psychology at Stanford University, conducted a field experiment at an upscale grocery store whereby they observed the outcome of consumers visiting one of two tasting booths. One tasting booth displayed only 6 jams,
                                                            
35

Sheena S. Iyengar and Mark R. Lepper, “When Choice is Demotivating: Can One Desire Too Much of a Good Thing?,” Journal of Personality and Social Psychology 79, no. 6 (2000): 995.

   

50 
 

while the other displayed a variety of 24 different flavored jams. Iyengar and Lepper’s findings showed that initially, shoppers who encountered the booth with 24 flavors were more attracted to the display (stopping 60% of the time) than shoppers who encountered the booth with only 6 (stopping only 40% of the time).36 Additionally, even though one booth displayed only 6 flavors, whereas the other displayed 24, there were no significant differences in the amount of jams sampled by visitors to each of the different booths.37 Finally, almost 30% of consumers who stopped at the 6 flavor booth bought a jar of jam, while only 3% of consumers who stopped at the booth with 24 flavors did.38 This suggests that although individuals originally found the booth with the plethora of flavors to be more attractive it hampered their ability and motivation to make a choice when it came time to purchase the product. A similar point is made in Expert Political Judgment, when Tetlock discusses a series of scenario exercises tested on a group of experts comprised of individuals he refers to as hedgehogs (those who know one big thing) and foxes (those who know many little things).39 Essentially participants were provided with an exhaustive variety of possible future scenarios in regards to a particular country and were asked to forecast the scenario that was most likely. This presentation of scenarios did not substantially affect the predictions of hedgehogs who were quite easily able to reject scenarios that they believed would not actually happen.40 However, the foxes, being more open-minded, found it very difficult not to consider even the strange or implausible scenarios.41
                                                            
36 37

Iyenger and Lepper, “When Choice is Demotivating,” 997. Ibid. 38 Ibid.  39 Tetlock, Expert Political Judgment, 190. 40 Ibid. 41 Ibid.

   

51 
 

Therefore, for this group in particular, the danger of attributing limited resources to the contemplation of a plethora of possibilities did little more than send them on a wild goose chase. This illustrates that “foxes become more susceptible than hedgehogs to a serious bias: the tendency to assign so much likelihood to so many possibilities that they become entangled in self-contradictions.”42

Importance of Convergence Conventional wisdom has long been a proponent of the process of divergent thought, focusing on the need for thinking outside the box, maintaining an open mind and encouraging an ever-increasing flow of ideas. In fact, until recently, convergent thinking has perpetually received a bad rap. However, research has finally begun to unearth the benefits of a combined approach including both divergent and convergent thinking. “In Praise of Convergent Thinking” by Arthur Cropley, states that, “Convergent thinking is oriented toward deriving the single best (or correct) answer to a clearly defined question…Divergent thinking, by contrast, involves producing multiple or alternative answers from available information.”43 Cropley, visiting professor of psychology at the University of Latvia for the past eleven years, argues that divergent thinking is essential to the creation of novel ideas, but that convergent thinking is then vital to the exploration of those ideas.44 Truly utilitarian creative thought, says Cropley, can only be achieved through the generation of ideas through divergence, followed by the criticism and evaluation of those ideas through convergence.45

                                                            
42 43

Ibid. Arthur Cropley, “In Praise of Convergent Thinking,” Creativity Research Journal 18, no.3 (2006), 391. 44 Cropley, “In Praise,” 398. 45 Ibid.

   

52 
 

According to Michael Handel, as quoted by Stephen Marrin, “While the absence of competition and variety in intelligence is a recipe for failure, its institution does not guarantee success.”46 Handel, joint founding editor of the journal Intelligence and

National Security, further notes that while divergent thinking exercises lead to an increased number of opinions for consideration, they are not able to aid in ascertaining the best alternative.47 Richard Betts, director of the Institute of War and Peace Studies, and the director of the International Security Policy Program at Columbia University, makes a similar point in “Analysis, War, and Decision: Why Intelligence Failures Are Inevitable.” Betts states that, “To the extent that multiple advocacy works, and succeeds in maximizing the number of views promulgated and in supporting the argumentative resources of all contending analysts, it may simply highlight the ambiguity rather than resolve it.”48 Essentially, both Handel and Betts acknowledge that while divergent

thinking methods may indeed be useful and necessary within intelligence analysis, they are not without their limitations. In specific, the generation of numerous ideas alone does not automatically result in better answers, highlighting the need for a combination of both divergent and convergent approaches to intelligence analysis.

Final Thoughts The jam experiment and scenario exercises discussed above tie directly into the findings of this experiment, showing that although experimental group conceptual models were significantly larger than control group conceptual models, the control group did
                                                            
Michael I. Handel, “Intelligence and the Problem of Strategic Surprise,” The Journal of Strategic Studies (September 1984), 268. In Stephen Marrin, “Preventing Intelligence Failures by Learning from the Past,” International Journal of Intelligence and CounterIntelligence 17, no. 4 (2004), 665. 47 Ibid.  48 Richard K. Betts, “Analysis, War, and Decision: Why Intelligence Failures Are Inevitable,” World Politics 31 (1978): 76.
46

   

53 
 

significantly better than the experimental group in forecasting the correct outcome of the research question posed. As a result of the individual to group conceptual modeling method employed in this study, the generation of a multitude of ideas seemed to do little more than confuse and overwhelm experimental group participants. Faced with such a large number of concepts, experimental group participants appeared to struggle to make sense of the relevant relationships and to identify adequately the information most important to answering the question. At this juncture, the discussion of convergent and divergent thinking discussed in detail above, becomes relevant. Experimental group participants, having been involved in a structured individual to group divergent thinking exercise, were left with many more options to consider than control group participants who simply set off to research the question on their own. Traditionally, this would be considered an excellent outcome as the divergent thinking exercise appeared to work, leaving the experimental group with a much wider array of concepts to consider. However, this experiment found that while this is true, it is not enough. Divergent thinking on its own appears to be a handicap, without some form of convergent thinking to counterbalance it. Therefore, it is likely that experimental group participants, once faced with the plethora of ideas generated by the group, would have greatly benefitted from a structured convergent thinking exercise before creation and development of their conceptual models. The goal of this exercise being to critically evaluate the ideas proposed, possibly eliminating those concepts that were clearly off base and prioritizing what was left into a meaningful arrangement. One way this convergent aspect is often accomplished in

groups is based on the amount of individual members within that group. For example,

   

54 
 

often times a group with four team members will find a way to organize and break down information into four separate sections, thereby allowing them to assign one section to each team member. While this is certainly not the ideal way to engage in convergent thinking, it is likely better than not incorporating it at all. However, at this point the best method for engaging in convergent thinking within the process of ECM has yet to be determined.

Future Research Future research should explore the importance of incorporating both divergent and convergent thinking into the method for the construction of conceptual models. While divergence was sufficiently accounted for in this study, due to time limitations convergence was not. Therefore, it would be interesting to see the effect of taking the individual to group conceptual modeling method employed in this study one-step further. As Surowiecki, McKeachie and Janis tell us, starting the process at the individual level and then moving into group collaboration is very valuable. However, at this juncture, once all possibilities the group can think of have been accounted for, it is necessary for the group to narrow the scope of the conceptual model into only the most relevant concepts and relationships and then organize them accordingly. Therefore, additional research is needed to establish the best method for carrying out this task.

   

55 
 

BIBLIOGRAPHY

Allison, Graham T. “Conceptual Models and the Cuban Missile Crisis.” In The Sociology of Organizations: Classic, Contemporary and Critical Readings, ed. Michael Jeremy Handel. Thousand Oaks: SAGE, 2003. Basque, Josianne and Beatrice Pudelko. “Using a Concept Mapping Software as a Knowledge Construction Tool in a Graduate Online Course,” in Proceedings of ED-MEDIA 2003, World Conference on Educational Multimedia, Hypermedia &Telecommunications, Honolulu, June 23-28, 2003, ed. D. Lassner and C. McNaught (Norfolk: Association for the Advancement of Computing in Education, 2003). Betts, Richard K. “Analysis, War, and Decision.” World Politics 31 (1978).
Budd, John W. “Mind Maps as Classroom Exercises.” Journal of Economic Education (Winter 2004).  

Church, Forrest. Introduction to The Jefferson Bible: The Life and Morals of Jesus of Nazareth, by Thomas Jefferson, 1-31. Boston: Beacon Press, 1989. Cohen, Jacob. Statistical Power Analysis for the Behavioral Sciences (Philadelphia: Lawrence Erlbaum Associates, 1988). Cropley, Arthur. “In Praise of Convergent Thinking.” Creativity Research Journal 18, no. 3 (2006). De Simone, Christina. “Applications of Concept Mapping.” Journal of College Teaching 55, no. 1 (2007). Garson, G. David. Guide to Writing Empirical Papers, Theses, and Dissertations (New York: CRC Press, 2002). Heuer, Richards J. Psychology of Intelligence Analysis (Center for the Study of Intelligence,
1999).

Iyengar, Sheena S. and Mark R. Lepper. “When Choice is Demotivating: Can One Desire Too Much of a Good Thing?.” Journal of Personality and Social Psychology 79, no. 6 (2000). Janis, Irving L. Groupthink: Psychological Studies of Policy Decisions and Fiascoes. Boston: Houghton Mifflin Company, 1982.

   

56 
 

Jarvelin, Kalervo and T.D. Wilson. “On Conceptual Models for Information Seeking and Retrieval Research.” Information Research 9, no. 1 (October 2003), http://informationr.net/ir/9-1/paper163.html (accessed January 15, 2009). Mangio, Charles A. and Bonnie J. Wilkinson. “Intelligence Analysis: Once Again.” Paper presented at the annual international meeting of the International Studies Association, San Francisco, California 26 March, 2008. Marrin, Stephen. “Preventing Intelligence Failures by Learning from the Past.” International Journal of Intelligence and CounterIntelligence 17, no. 4 (2004). McKeachie, Wilbert J. McKeachie’s Teaching Tips. Boston: Houghton Mifflin Company, 2002. Miller, George A. “The Magical Number Seven, Plus or Minus Two: Some Limits on our Capacity for Processing Information.” Psychology Review 63 (1956). Novak, Joseph and Alberto Canas. The Theory Underlying Concept Maps and How to Construct and Use Them, Technical Report IHMC Cmap Tools 2006-01 Rev 012008, Florida Institute for Human and Machine Cognition, 2008. http://cmap.ihmc.us/Publications/ResearchPapers/TheoryUnderlyingConceptMap s.pdf (accessed August 8, 2008). Santrock, John W. Adolescence, 8th ed. New York: McGraw-Hill, 2001. Surowiecki, James. The Wisdom of Crowds. New York: Anchor Books, 2004. Tetlock, Philip E. Expert Political Judgment. Princeton: Princeton University Press, 2005. United States Government. Intelligence Community Directive Number 203 (June 21, 2007), http://www.fas.org/irp/dni/icd/icd-203.pdf (accessed January 26, 2009).

   

57 
 

APPENDICES

   

58   

Appendix 1: Experiment Sign-Up Form

Full Name: _______________________________________________________ Email Address: ____________________________________________________ Class Year: ________________________________________________________

Please sign up for at least 1 time slot in column A & 1 time slot in Column B 20 October 2008 Time Slot A 27 October 2008 Time Slot B

11:00 – 12:30 2:00 – 3:30 5:00 – 6:30

1:00 – 1:30 4:00 – 4:30 7:00 – 7:30

Please sign up for at least 1 time slot in column A & 1 time slot in Column B 20 October 2008 Time Slot A 27 October 2008 Time Slot B

1:00 – 1:30 4:00 – 4:30 7:00 – 7:30

11:00 – 12:30 2:00 – 3:30 5:00 – 6:30

Even though you are signing up for multiple spots, you will only be asked to come in once on the 20th & once on the 27th Contact Information: Shannon Ferrucci sferru13@mercyhurst.edu (315) 525-3967

 

   

59   

Appendix 2: IRB Research Proposal

Date Submitted: 9/24/2008

Advisor’s Name (if applicable): Kristan Wheaton

Investigator(s): Shannon Ferrucci

Advisor’s Email: kwheaton@mercyhurst.edu

Investigator Address: 5117 Belle Village Drive East Erie, PA 16509

Advisor’s Signature Of Approval: [X]

Investigator(s) E-mail: Sferru13@mercyhurst.edu

Title Of Research Project: Conceptual Modeling And Intelligence Analysis

Investigator Telephone Number: 315-525-3967

Date Of Initial Data Collection: October 20, 2008

   

60   

Please describe the proposed research and its purpose, in narrative form:

The purpose of this study is to assess whether or not explicit conceptual modeling improves collection and the subsequent analysis The more complicated a question is and the more concepts that play a part in the answering of that question the harder it is to recall all of those concepts simply from memory. Therefore, putting these concepts and their relationships to each other down on paper can be extremely useful. Explicit conceptual modeling prior to collection should help to improve the efficiency of the collection process as the model provides you with a basis for what types of information to look for. Also, since the conceptual model is not static, but a fluid diagram that evolves as you learn more about a certain topic, the model should help to highlight and minimize gaps in knowledge. These improvements in collection should then improve the subsequent analysis. Additionally, this experiment is designed to test a specific method for developing conceptual models. The approach starts at the individual level and then moves to the group collaboration level. This method supports the age old idea that two minds are better than one. Furthermore, this method should limit the problems associated with group think by encouraging equal participation from all individuals, even those who may be less inclined to voice their opinions or participate in group settings.

Indicate the materials to be used: Consent Form Debriefing Form Conceptual Modeling Training Information Research Question Free Online Conceptual Modeling Software Analytic Confidence And Source Reliability Information Format For Written Product Writing Utensils Post-Test Questionnaire Procedure:
   

61   

One week prior to the start of the experiment, I will make an appearance in various undergraduate and graduate intelligence studies classes in order to promote my experiment and have students sign up. The students will be asked to provide their availability on two separate dates. I will then email them a designated time slot for both dates. Date and time assignments, as well as group assignments, will be random. On the first day of the experiment, the control group will show up for half an hour. They will be provided with a research question regarding upcoming 2008 Zambian presidential elections and a semi-structured format for a written intelligence product. I will also give them explanations of both source reliability and analytic confidence. At the end of the half hour they will be sent home and given one week to research and analyze the question posed to them. The experimental group on the other hand will be asked to come in for an hour and a half on the first day of the experiment. They will be provided with the same research question as the control group, will be given the same format for a written intelligence product and will also receive the same information regarding source reliability and analytic confidence. However, this group will undergo a small training session regarding what conceptual modeling is and how to create one on the same piece of free conceptual modeling software, such as Mindomo. After the training session the participants will each be asked to make a list of concepts off the top of their heads that they feel are relevant to the research question I provided them with. Next, we will go through and make a master list, consolidating all of the participants’ individual lists, highlighting concepts that were commonly found, significant differences in opinion and uncommon but highly useful concepts. Finally, each individual will then be asked to use this master list to create a conceptual model using software that shows the perceived relationships between concepts. Participants will then discuss the models they have created with the individual sitting next to them in order to share their ideas and see how another person visualized the same information. They will then print out a copy for themselves and a copy for me. On the second day of the experiment, a week from the first day, both the experimental and control groups will come back in. The experimental group will come in for half an hour. They will hand in their written analysis and an updated conceptual model that they have modified to reflect what they learned through their research. After doing this they will answer a short post-experiment questionnaire and will then be debriefed. The control group on the other hand will come in for an hour and a half on this day. They will hand in their written analysis and will then simply be told to visualize the concepts that were important in answering the research question using bubbles and lines. After completion they will hand in their visualization, complete a post-experiment questionnaire and then be debriefed.

   

62   

Participants who successfully complete all experiment responsibilities will receive extra credit from intelligence professors and pizza and soda will be offered to all those who participate in the study. Three second-year intelligence studies graduate students, who were not experiment participants, will then evaluate the written analyses using the same criteria that students are graded against in the Intelligence Communications class that these graduate students have already successfully completed.

1. Do you have external funding for this research (money coming from outside the College)? Yes[ ] No[X] Funding Source (if applicable): N/A

2. Will the participants in your study come from a population requiring special protection; in other words, are your subjects someone other than Mercyhurst College students (i.e., children 17-years-old or younger, elderly, criminals, welfare recipients, persons with disabilities, NCAA athletes)? Yes[ ] No[X]

If your participants include a population requiring special protection, describe how you will obtain consent from their legal guardians and/or from them directly to insure their full and free consent to participate. N/A

Indicate the approximate number of participants, the source of the participant pool, and recruitment procedures for your research: I plan to have approximately 60-80 participants. I intend to recruit undergraduate and graduate students in the intelligence studies department through sign-up sheets which I will pass around to students when I visit their classes to promote the experiment.

Will participants receive any payment or compensation for their participation in your research (this includes money, gifts, extra credit, etc.)? Yes[X] No[ ] If yes, please explain: Students will obtain extra credit from the intelligence professors willing to grant it for participation in an experiment and all participants will be offered pizza and refreshments at the end of the experiment.
   

63   

3. Will the participants in your study be at any physical or psychological risk (risk is defined as any procedure that is invasive to the body, such as injections or drawing blood; any procedure that may cause undue fatigue; any procedure that may be of a sensitive nature, such as asking questions about sexual behaviors or practices) such that participants could be emotionally or mentally upset? Yes[ ] No[X]

Describe any harmful effects and/or risks to the participants' health, safety, and emotional or social well being, incurred as a result of participating in this research, and how you will insure that these risks will be mitigated: None.

4. Will the participants in your study be deceived in any way while participating in this research? Yes[ ] No[X]

If your research makes use of any deception of the respondents, state what other alternative (e.g., non-deceptive) procedures were considered and why they weren't chosen: N/A

5. Will you have a written informed consent form for participants to sign, and will you have appropriate debriefing arrangements in place? Yes[X] No[ ]

Describe how participants will be clearly and completely informed of the true nature and purpose of the research, whether deception is involved or not (submit informed consent form and debriefing statement): Prior to the start of the experiments, participants will be provided with a general overview of what will occur during the session as well as the consent form, which will also describe what is expected of them. Following the experiment participants will be asked to fill out an administrative questionnaire and will then be provided with a

   

64   

debriefing statement that will explain how the results from the session will be used (please see forms at the end of this proposal).

Please include the following statement at the bottom of your informed consent form: “Research at Mercyhurst College which involves human participants is overseen by the Institutional Review Board. Questions or problems regarding your rights as a participant should be addressed to Mr. Tim Harvey Institutional Review Board Chair; Mercyhurst College; 501 East 38th Street; Erie, Pennsylvania 16546-0001; Telephone (814) 824-3372.”

6. Describe the nature of the data you will collect and your procedures for insuring that confidentiality is maintained, both in the record keeping and presentation of this data: Names are not required for my research and thus no names will be used in the recording of the results or the presentation of my data. Names will only be used to notify professors of participation in order for them to correctly assign extra credit.

7. Identify the potential benefits of this research on research participants and humankind in general. Potential benefits include: For participants: An opportunity to practice the intelligence analysis skills they have learned in the classroom in an experiment aimed at testing the value of explicit conceptual modeling as it applies to the quality of the analysis produced. Students are often asked to complete short written intelligence assignments with quick turnaround times in Intelligence Studies courses. This experiment hopes to validate a particular method for the creation of conceptual models, which if used by intelligence students should increase efficiency in collection and accuracy in analysis.

For the Intelligence Community: Currently, collection takes up quite a large amount of an analyst’s time due to information overload. This experiment hopes to demonstrate that pre-collection conceptual modeling will not only make the collection process more efficient, it will also
   

65   

help to minimize gaps in knowledge as those gaps will be recognized earlier on. This process will then in turn help to improve the analysis stemming from the collection.

   

66   

Appendix 3: Control Group Consent Form

The purpose of this research is to test the value of variations in analytic approaches as they apply to the quality of the analysis produced. Your participation involves the development of a short analytic product, the completion of a data visualization exercise and the filling out of a post-experiment questionnaire. This process will require your onsite attendance today and one week from today during the pre-determined timeslot that was designated to you. In total time spent onsite should not exceed two hours, however, some participation on your own time is required throughout the week. Your name WILL NOT appear in any information disseminated by the researcher. Your name will only be used to notify professors of your participation in order for them to assign extra credit. There are no foreseeable risks or discomforts associated with your participation in this study. Participation is voluntary and you have the right to opt out of the study at any time for any reason without penalty. I, ____________________________, acknowledge that my involvement in this research is voluntary and agree to submit my data for the purpose of this research.

_________________________________ Signature _________________________________ Printed Name

__________________ Date __________________ Class

Telephone Number: ____________________________________ Researcher’s Signature: ___________________________________________________

If you have any further question about this research you can contact me at sferru13@mercyhurst.edu Research at Mercyhurst College which involves human participants is overseen by the Institutional Review Board. Questions or problems regarding your rights as a participant should be addressed to Tim Harvey; Institutional Review Board Chair; Mercyhurst College; 501 East 38th Street; Erie, Pennsylvania 16546-0001; Telephone (814) 824-3372. tharvey@mercyhurst.edu    

67   

Appendix 4: Experimental Group Consent Form

The purpose of this research is to test the value of explicit conceptual modeling as it applies to the quality of the analysis produced. Furthermore, the following experiment will test the effectiveness of a structured individual to group conceptual modeling method. Your participation involves an instruction period and training exercise, the development of a short analytic product with accompanying model and the filling out of a post-experiment questionnaire. This process will require your onsite attendance today and one week from today during the predetermined timeslots that have been designated to you. In total, time spent onsite should not exceed two hours, however, some participation on your own time is required throughout the week. Your name WILL NOT appear in any information disseminated by the researcher. Your name will only be used to notify professors of your participation in order for them to assign extra credit. There are no foreseeable risks or discomforts associated with your participation in this study. Participation is voluntary and you have the right to opt out of the study at any time for any reason without penalty. I, ____________________________, acknowledge that my involvement in this research is voluntary and agree to submit my data for the purpose of this research.

_________________________________ Signature _________________________________ Printed Name Telephone Number: ____________________________________

__________________ Date __________________ Class

Researcher’s Signature: ___________________________________________________

If you have any further question about Conceptual Modeling or this research, you can contact me at sferru13@mercyhurst.edu Research at Mercyhurst College which involves human participants is overseen by the Institutional Review Board. Questions or problems regarding your rights as a participant should be addressed to Tim Harvey; Institutional Review Board Chair; Mercyhurst College; 501 East 38th Street; Erie, Pennsylvania 16546-0001; Telephone (814) 824-3372. tharvey@mercyhurst.edu    

68   

Appendix 5: Research Question

Due to the 19 August 2008 death of Zambia’s President Mwanawasa, early presidential elections will take place on 30 October 2008 in accordance with the Zambian constitution that requires new elections to be held within 90 days of a president's untimely departure from office. Who will win this upcoming Zambian presidential election (Rupiah Banda, Michael Sata or Hakainde Hichilema) and why?

   

69   

Appendix 6: Important Supporting Information
Source Reliability: Source Reliability reflects the accuracy and reliability of a particular source over time. Sources with high reliability have been proven to have produced accurate, consistently reliable, information in the past. Sources with low reliability lack the accuracy and proven track record commensurate with more reliable sources. o In this experiment source reliability will be measured on a low - high scale conveying the reliability of the sources used for that piece of intelligence/report. o For more information regarding internet source reliability please refer to: http://www.library.jhu.edu/researchhelp/general/evaluating/

Analytic Confidence: Analytic Confidence reflects the level of confidence an analyst has in his or her estimates and analyses. It is not the same as using words of estimative probability, which indicate likelihood. It is possible for an analyst to suggest an event is virtually certain based on the available evidence, yet have a low amount of confidence in that forecast due to a variety of factors or vice versa. o o In this experiment Analytic Confidence will be measured on a low - high scale. For more information regarding factors contributing to the assessment of analytic confidence see the Peterson Table of Analytic Confidence provided below.       Use Of Structured Method(s) In Analysis Overall Source Reliability Source Corroboration/Agreement: Level Of Conflict Amongst Sources Level Of Expertise On Subject/Topic & Experience Amount Of Collaboration Task Complexity Time Pressure: Time Given To Make Analysis Peterson Table Of Analytic Confidence Assessment

   

70   

Appendix 7: Experiment Answer Sheet

NAME:

FORECAST: It is (likely, highly likely, almost certain) that (Rupiah Banda, Michael Sata, Hakainde Hichilema) will win the 30 October 2008 Zambian presidential elections. BULLETED DISCUSSION:

SOURCE RELIABILITY (CIRCLE ONE): HIGH ANALYTIC CONFIDENCE (CIRCLE ONE): HIGH

LOW

MEDIUM

LOW

MEDIUM

NAME OF PROFESSOR(S) GIVING EXTRA CREDIT:

   

71   

Appendix 8: Control Group Expectation Sheet EXPECTATIONS FOR 27 OCTOBER 2008 GROUP B

Completed answer sheet ready to be turned in o Inclusive of: Name, Forecast, Bulleted Discussion, Source Reliability, Analytic Confidence, Name of Professor(s) Giving Extra Credit

Complete short data visualization exercise onsite, based on past week of collection and analysis

Order of Events for 27 October 2008 (45 minute maximum) o Complete short data visualization exercise & hand in o Hand in answer sheet o Answer short post-experiment questionnaire o Pass out debriefing sheets

                     

   

72   

Appendix 9: Experimental Group Expectation Sheet EXPECTATIONS FOR 27 OCTOBER 2008 GROUP A

Completed answer sheet ready to be turned in o Inclusive of: Name, Forecast, Bulleted Discussion, Source Reliability, Analytic Confidence, Name of Professor(s) Giving Extra Credit

Updated Bubbl.us Conceptual Model o Over the course of the week please update your models as you learn more about the topic (ex. fill in knowledge gaps in the model, highlight areas that turned out to be more important than originally thought, place less focus on those areas that turned out not to be as important as originally thought). Allow your model to evolve alongside your analysis o I will ask you to sign into the computer and share the final version of your conceptual model with me once you arrive on the 27th

Order Of Events for 27 October 2008 (30 minute maximum) o Hand in answer sheet o Share final conceptual model o Answer short post-experiment questionnaire o Pass out debriefing sheets

   

73   

Appendix 10: Pre-Experiment Questionnaire

 
Thank you for agreeing to participate in this study! Please take a few moments to answer the following questions. Your feedback is greatly appreciated.

1. Do you feel as though you will be able to dedicate a sufficient amount of time to working on this experiment over the next week? Yes Maybe No

2. What type of a learner do you primarily consider yourself to be? Auditory Visual Kinesthetic & Tactile Unsure

3. Please rate how interested you are in this study, with 1 being not interested and 5 being extremely interested. 1 2 3 4 5

4. Please rate how useful you feel structured approaches to the analytic process are, with 1 being not useful and 5 being extremely useful. 1 2 3 4 5

5. What do you think the purpose of this experiment is?

6. What are your reasons for participating in this experiment?
 

   

74   

Appendix 11: Contact Information

CONTACT INFORMATION

Please feel free to get a hold of me during the week if you have any questions or problems! Thank you again for your participation.

Name: Shannon Ferrucci Email: sferru13@mercyhurst.edu Telephone: (315) 525-3967 Location: CIRAT Lab        

 

   

75   

Appendix 12: Conceptual Modeling Lecture •

Everybody builds implicit models on a daily basis o We build models of the world around us For ex: when we drive to the grocery store we model the route in our minds and when we get ready each morning we model our routine o We also build models when we are faced with questions We try to come up with preliminary answers and conclusions However, we often recognize gaps in our knowledge signaling that we need more information o Our models are unique to each of us They draw from our experiences, interests, opinions, etc… For ex: If I were to ask all of you, a room of intelligence students, what first comes to mind when faced with the question of what intelligence is, your answers would likely be very different than if I went out into the community and asked the same question of the first ten people I saw. Models can become extremely complex o Intelligence requirements (questions or topics posed to the analyst by a decision maker) often entail understanding complex relationships between people, states, organizations, industries etc… o Therefore, it is very rare that an analyst will ever develop a complete model of the requirements right off the bat. Generally, an analyst is able to fill in pieces of their model with things they already know but are forced to fill in the rest with topics, which they recognize they need to understand more about. Use of Conceptual Models o Conceptual models highlight key concepts and their relationships to each other o These models appear to be a useful way for intelligence professionals to model knowledge and come to terms with compound requirements o Conceptual modeling within the field of intelligence must include both what an individual already knows about a topic and also what that individual thinks he or she needs to know about it in order to sufficiently answer the requirements posed Don’t be afraid to explore o The first construction of the conceptual model is not factual but exploratory We may identify areas that we think are important to answering the requirement, but until we have collected that information we have no way of knowing whether or not they truly are
 

 

76   

However, when this happens it should not be viewed as a setback or waste of time • Exploring various concepts helps the analyst recognize what is critical to answering the question and what is not

to a requirement, it often provides context, background and understanding of the requirement Importance of relationships o Concepts alone are not enough, relationships amongst concepts must be included in the model as well Imagine that you are modeling a drug trafficking organization. To learn about the suppliers and the runners separately is a good start. However, you must then learn about the relationship between the two. Need for explicit models o When dealing with such complex models as is usually necessary within the field of intelligence, the need to make the models explicit becomes obvious o Many psychological studies have been done suggesting that there are upper limits on our working memory Often it is cited that an individual can only hold 7 things (plus or minus 2) in their memory at any given time While there are certain exceptions to this rule, it is obvious that most intelligence models will eventually become too complex to be stored solely in memory and must instead be made explicit Importance of model to analysts o Share and compare models with fellow analysts and professionals o Assess level of confidence in analysis produced o From a managing standpoint can help in tasking a team of analysts and divvying up responsibilities o Aids efficiency in collection process and helps to identify gaps in knowledge o Useful for after the fact review o Provides a good starting point for future related questions

   

77   

Appendix 13: Bubbl.us Instruction Sheet For Experimental Group Experiment Day 1 – Group A Bubbl.us Instructional Sheet • • •

Please sign into computers and go to Bubbl.us Click start brainstorming Click save on the right hand side of the screen o Fill out the create account steps and hit submit o This will allow you to save and share you project with me later on To start you will see that there is a single bubble in the center of the screen o If you click on the text saying start here you can replace that with whatever words or concept you deem appropriate o In this case, since it is the 1st bubble it is important to start by entering the specific requirements you need to answer based on the research question provided to you Now if you simply place your cursor over the center of the bubble you will see a choice of 6 icons. Let’s start on the top left hand corner. o If you click on the cross with arrows you can move the bubble anywhere you like on the screen o Moving to the top right, if you click on the X your bubble will disappear (to get it back simply click the undo button on the top left of your screen) o Clicking on the middle icon on the right hand side of the bubble allows you to create a new sibling bubble (i.e. a bubble that does not spring from the 1st bubble, but is entirely separate) o The blue circles icon allows you to show directional relationships through the use of arrowed lines. By clicking on the icon and dragging your cursor to the sibling bubble you just made in the previous step you can see an example of this o Clicking on the middle bottom icon of one of the bubbles you have created allows you to make a child balloon (i.e. a bubble that does spring from a previous bubble, generally these bubbles have some sort of direct relationship to each other, with the concept in the child balloon being a sub-concept of the original parent balloon) o Lastly, clicking on the bottom left hand icon allows you to change the color of the balloon

   

78   

Just a few more tips: o If you click the center button at the upper left, your entire conceptual model will be centered on the page o Also, if you would like to print your conceptual model you can click the set print area button at the upper left. This will help you to ensure your entire conceptual model is within the printable area of the page o Also, to zoom in and out you can scroll up and down on your mouse or hit the plus and minus buttons on the upper left Now take about 5 minutes to familiarize yourself with the software on your own o To do this begin to craft a practice conceptual model on important things to consider when buying a new car (ex. gas mileage) o Practice using the different icons that we just went over and try to incorporate each function into your conceptual model at least once o I will walk around to offer suggestions and take questions

   

79   

Appendix 14: Bubbl.us Instruction Sheet For Control Group Experiment Day 2 – Group B Bubbl.us Instructional Sheet • • •

Please sign into computers and go to Bubbl.us Click start brainstorming Click save on the right hand side of the screen o Fill out the create account steps and hit submit o This will allow you to save and share you project with me later on To start you will see that there is a single bubble in the center of the screen o If you click on the text saying start here you can replace that with whatever words or concept you deem appropriate Now if you simply place your cursor over the center of the bubble you will see a choice of 6 icons. Let’s start on the top left hand corner. o If you click on the cross with arrows you can move the bubble anywhere you like on the screen o Moving to the top right, if you click on the X your bubble will disappear (to get it back simply click the undo button on the top left of your screen) o Clicking on the middle icon on the right hand side of the bubble allows you to create a new sibling bubble (i.e. a bubble that does not spring from the 1st bubble, but is entirely separate) o The blue circles icon allows you to show directional relationships through the use of arrowed lines. By clicking on the icon and dragging your cursor to the sibling bubble you just made in the previous step you can see an example of this o Clicking on the middle bottom icon of one of the bubbles you have created allows you to make a child balloon (i.e. a bubble that does spring from a previous bubble, generally these bubbles have some sort of direct relationship to each other, with the child balloon being subordinate to the original parent balloon) o Lastly, clicking on the bottom left hand icon allows you to change the color of the balloon Just a few more tips: o If you click the center button at the upper left, your entire conceptual model will be centered on the page o Also, to zoom in and out you can scroll up and down on your mouse or hit the plus and minus buttons on the upper left

   

80   

Appendix 15: Structured Conceptual Modeling Exercise Structured Conceptual Modeling Exercise

Give students 2 minutes to identify requirements and create a list of concepts they feel are relevant to those requirements o What do they think they need to learn more about in order to answer the question? Ex. Governmental Type And Structure o What are the big, moving pieces? Ex. Level Of Candidate Support Come together as a group and consolidate individual lists into group list on board o Go around the room with each student reading off their list o Emphasize commonalities with plus signs o Highlight legitimate differences in opinion as food for thought o Take note of “AHA” moments Very important concept that only a select few thought of, but all recognize as essential to the question Go back onto Bubbl.us and create your own conceptual model combining your individual list and thoughts with that of the collaborative group list o Remember to start with requirements and build from there o Highlight relationships between concepts and directional flow of those relationships where applicable Briefly look at the way someone sitting next to you has set up their conceptual model o Take away ideas for your own o Offer suggestions or alternatives

   

81   

Appendix 16: Control Group Post-Experiment Questionnaire Follow-Up Questionnaire B

Thanks for your participation! Please take a few moments to answer the following questions. Your feedback is greatly appreciated.

1. Please rate your understanding of conceptual modeling prior to this study, with 1 being extremely low and 5 being extremely high. 1 2 3 4 5

2. Please rate your understanding of conceptual modeling following this study, with 1 being extremely low and 5 being extremely high. 1 2 3 4 5

3. Please rate how often explicit conceptual modeling has been a part of your personal analytic process prior to this experiment, with 1 being never and 5 being every time you produce an intelligence estimate. 1 2 3 4 5

4. Please rate how often you plan to incorporate explicit conceptual modeling into your personal analytic process in the future, with 1 being never and 5 being every time you produce an intelligence estimate. 1 2 3 4 5

5. Were you able to dedicate a sufficient amount of time to working on this experiment over the past week? Yes Maybe No

   

82   

6. Please rate your interest in the study after having completed the experiment, with 1 being not interested and 5 being extremely interested. 1 2 3 4 5

7. Based on your experience in this experiment, how useful do you feel structured approaches to the analytic process are, with 1 being not useful and 5 being extremely useful. 1 2 3 4 5

8. What do you think the purpose of this experiment was?

9. Please provide any additional comments you may have regarding conceptual modeling in general or any particular part of this experiment.

   

83   

Appendix 17: Experimental Group Post-Experiment Questionnaire Follow-Up Questionnaire A

Thanks for your participation! Please take a few moments to answer the following questions. Your feedback is greatly appreciated.

1. Please rate your understanding of conceptual modeling prior to this study, with 1 being extremely low and 5 being extremely high. 1 2 3 4 5

2. Please rate your understanding of conceptual modeling following this study, with 1 being extremely low and 5 being extremely high. 1 2 3 4 5

3. Please rate how useful you found the conceptual modeling training provided at the beginning of this experiment to be, with 1 being not at all helpful and 5 being extremely helpful. 1 2 3 4 5

4. Please rate how often explicit conceptual modeling has been a part of your personal analytic process prior to this experiment, with 1 being never and 5 being every time you produce an intelligence estimate. 1 2 3 4 5

   

84   

5. Please rate how often you plan to incorporate explicit conceptual modeling into your personal analytic process in the future, with 1 being never and 5 being every time you produce an intelligence estimate. 1 2 3 4 5

6. Please rate whether or not you found that explicit conceptual modeling in this experiment aided you in developing a more thorough and nuanced intelligence analysis. Definitely Somewhat Not At All

7. Please rate how effective you think the conceptual modeling method used in this experiment, inclusive of both individual work and group collaboration, was. 1 2 3 4 5

8. Please rate how useful you found the use of the technology aid Bubbl.us to be in creating and updating your conceptual models, with 1 being not useful and 5 being extremely useful. 1 2 3 4 5

9. Were you able to dedicate a sufficient amount of time to working on this experiment over the past week? Yes Maybe No

10. Please rate your interest in the study after having completed the experiment, with 1 being not interested and 5 being extremely interested. 1 2 3 4 5

   

85   

11. Based on your experience in this experiment, how useful do you feel structured approaches to the analytic process are, with 1 being not useful and 5 being extremely useful. 1 2 3 4 5

12. What do you think the purpose of this experiment was?

13. Please provide any additional comments you may have regarding conceptual modeling in general or any particular part of this experiment.

   

86   

Appendix 18: Control Group Debriefing Sheet Participation Debriefing B

Thank you for participating in this research. I appreciate your contribution and willingness to support the student research process.

This experiment was designed to test the specific part of the analytic process termed conceptual modeling. Currently there has been little research done on the topic of conceptual modeling within the field of intelligence analysis, and this study hopes to take the first of many steps in establishing the importance of explicit conceptual modeling within the analytic process.

Within the Intelligence Community collection takes up a significant amount of an analyst’s time due to information overload stemming from both open and classified sources. This experiment hopes to demonstrate that pre-collection conceptual modeling will not only make the collection process more efficient, it will also help to minimize gaps in knowledge as those gaps will be recognized earlier on. This process should then in turn help to improve the subsequent analysis.

If you have any further questions about conceptual modeling or this research you can contact me at sferru13@mercyhurst.edu.

   

87   

Appendix 19: Experimental Group Debriefing Sheet Participation Debriefing A

Thank you for participating in this research. I appreciate your contribution and willingness to support the student research process.

The purpose of this study was to test the value of explicit conceptual modeling as it applies to the quality of the analysis produced. Furthermore, this experiment tested the effectiveness of a structured individual to group conceptual modeling method. Currently there has been little research done on the topic of conceptual modeling within the field of intelligence analysis, and this study hopes to take the first of many steps in establishing the importance of explicit conceptual modeling within the analytic process.

Within the Intelligence Community collection takes up a significant amount of an analyst’s time due to information overload stemming from both open and classified sources. This experiment hopes to demonstrate that pre-collection conceptual modeling will not only make the collection process more efficient, it will also help to minimize gaps in knowledge as those gaps will be recognized earlier on. This process should then in turn help to improve the subsequent analysis.

If you have any further questions about conceptual modeling or this research you can contact me at sferru13@mercyhurst.edu.

   

88   

Appendix 20: Significance Testing Results
 

***The following results are based on a 0.05 level of significance. However, due to the fact that this research is exploratory in nature, a 0.10 level of significance was deemed most appropriate and is therefore reflected in the text of this thesis.***

Results: Is there a difference between control and experimental for source reliability? Null: there is no difference between control and experimental for source reliability. Alternative: there a difference between control and experimental for source reliability. Will be using t-test for independent samples. Testing normality assumption as sample sizes are < than 30.
3.00 23 22 25 24

Response

2.50

2.00

1.50 1 Control Experimental

Box plot shows outliers for control group. Even with the presence of outliers, normality is satisfied (See below). Also these values are important for the analysis. Thus the decision is not to remove the outliers.

1.00

Group

   

89   

Normal Q-Q Plot of Response
Expected Normal
2 1 0 -1 -2 1.0 1.5 2.0 2.5 3.0

for group= Control
Most points are close to the line thus the assumption of normality is satisfied for the group Control.

Observed Value

Normal Q-Q Plot of Response
for group= Experimental
Expected Normal
1 0 -1 -2 1.0 1.5 2.0 2.5 3.0

Most points are close to the line thus the assumption of normality is satisfied for the group Experimental.

Observed Value
Group Statistics Group Control Experimental N 25 22 Mean 2.1200 2.2273 Std. Deviation .43970 .52841 Std. Error Mean .08794 .11266

Response

   

90   
Independent Samples Test Levene's Test for Equality of Variances F Response Equal variances assumed Equal variances not assumed 2.234 Sig. .142 t -.760 -.751 t-test for Equality of Means df 45 41.052 Mean Sig. (2-tailed) Difference .451 .457 -.10727 -.10727

Here need to check if the assumption of equal variances is satisfied. According to Levene’s test (P-value = 0.142) > ( variances is satisfied. = 0.05), thus assumption of equal

Independent Samples Test Levene's Test for Equality of Variances F Response Equal variances assumed Equal variances not assumed 2.234 Sig. .142 t -.760 -.751 t-test for Equality of Means df 45 41.052 Mean Sig. (2-tailed) Difference .451 .457 -.10727 -.10727

According to above table, t-test value = -0.76, P-value = 0.451. Since (P-value = 0.451) > ( = 0.05), null hypothesis is not rejected.

Conclusion: At 5% level, there is no difference between control and experimental for source reliability. Is there a difference between control and experimental for analytic confidence? Null: there is no difference between control and experimental for analytic confidence. Alternative: there a difference between control and experimental for analytic confidence. Will be using t-test for independent samples. Testing normality assumption as sample sizes are < than 30.

   

91   

Response for Analytical Confidence

3.00

24 25

47 45 46

2.50

2.00

1.50 21 27 Control 30 29 28 Experimental

1.00

Box plot shows outliers for both groups. Even with the presence of outliers, normality is satisfied (See below). Also these values are important for the analysis. Thus the decision is not to remove the outliers.

Group

Normal Q-Q Plot of Response for Analytical Confidence for group= Control
Expected Normal
2 1 0 -1 -2 1.0 1.5 2.0 2.5 3.0

Most points are close to the line thus the assumption of normality is satisfied for the group Control.

Observed Value
Normal Q-Q Plot of Response for Analytical Confidence for group= Experimental
Expected Normal
1.5 1.0 0.5 0.0 -0.5 -1.0 -1.5 1.0 1.5 2.0 2.5 3.0

Most points are close to the line thus the assumption of normality is satisfied for the group Experimental.

Observed Value
   

92   
Group Statistics Group Control Experimental N 25 22 Mean 1.9600 1.9091 Std. Deviation .45461 .61016 Std. Error Mean .09092 .13009

Response for Analytical Confidence

Independent Samples Test Levene's Test for Equality of Variances F Response for Equal variances Analytical Confidence assumed Equal variances not assumed 2.287 Sig. .137 t .327 .321 t-test for Equality of Means df 45 38.491 Sig. (2-tailed) .745 .750 Mean Difference .05091 .05091

Here need to check if the assumption of equal variances is satisfied. According to Levene’s test (P-value = 0.137) > ( variances is satisfied.
Independent Samples Test Levene's Test for Equality of Variances F Response for Equal variances Analytical Confidence assumed Equal variances not assumed 2.287 Sig. .137 t .327 .321 t-test for Equality of Means df 45 38.491 Sig. (2-tailed) .745 .750 Mean Difference .05091 .05091

= 0.05), thus assumption of equal

According to above table, t-test value = 0.327, P-value = 0.745. Since (P-value = 0.745) > ( = 0.05), null hypothesis is not rejected.

Conclusion: At 5% level, there is no difference between control and experimental for analytic confidence. Is there a difference between control and experimental for forecast results? Null: there is no difference between control and experimental for forecast results. Alternative: there a difference between control and experimental for forecast results. Will be using t-test for independent samples. Testing normality assumption as sample sizes are < than 30.

   

93   

Response for Forecast Results

2.00 1.80 1.60 1.40 1.20 1.00 Control Experimental

Box plot shows no outliers for both groups.

Group
Normal Q-Q Plot of Response for Forecast Results for group= Control

Expected Normal

0.25 0.00 -0.25 -0.50 -0.75 -1.00 1.0 1.2 1.4 1.6 1.8 2.0

Most points are not close to the line thus the assumption of normality is not satisfied for the group Control.

Observed Value
Normal Q-Q Plot of Response for Forecast Results for group= Experimental

Expected Normal

0.75 0.50 0.25 0.00 -0.25 -0.50 1.0 1.2 1.4 1.6 1.8 2.0

Most points are not close to the line thus the assumption of normality is not satisfied for the group Experimental.

Observed Value

   

94   

Cannot use independent samples t- test as normality is not satisfied. Wilcoxon Rank Sum test, non-parametric test.
Descriptive Statistics N Response for Forecast Results Group 47 47 Mean 1.5532 1.4681 Std. Deviation .50254 .50437 Minimum 1.00 1.00

Need to use

Maximum 2.00 2.00

Ranks Response for Forecast Results Group Control Experimental Total N 25 22 47 Mean Rank 26.98 20.61 Sum of Ranks 674.50 453.50

Test Statisticsa Response for Forecast Results 200.500 453.500 -1.844 .065

Mann-Whitney U Wilcoxon W Z Asymp. Sig. (2-tailed)

a. Grouping Variable: Group

Wilcoxon Rank Sum test value = -1.844, P-value = 0.065. Since (P-value = 0.065) > ( = 0.05), null hypothesis is not rejected.

Conclusion: At 5% level, there is no difference between control and experimental for forecast results. Is there a difference between control and experimental for question, “Please rate how interested you are in this study, with 1 being not interested and 5 being extremely interested.”? Null: There is no difference between control and experimental for question, “Please rate how interested you are in this study, with 1 being not interested and 5 being extremely interested.”

   

95   

Alternative: There is a difference between control and experimental for question, “Please rate how interested you are in this study, with 1 being not interested and 5 being extremely interested.” Will be using t-test for independent samples. Testing normality assumption as sample sizes are < than 30.

Response for Q. 1 in Pre-Experiment

5.00 4.50 4.00 3.50 3.00 2.50 2.00 Control Experimental

Box plot shows no outliers for both groups.

Group for Questionnaire Results

Normal Q-Q Plot of Response for Q. 1 in Pre-Experiment for groupq= Control
Expected Normal
0.5 0.0 -0.5 -1.0 -1.5 -2.0 2.0 2.5 3.0 3.5 4.0

Most points are close to the line thus the assumption of normality is satisfied for the group Control.

Observed Value

   

96   

Normal Q-Q Plot of Response for Q. 1 in Pre-Experiment for groupq= Experimental

Expected Normal

2 1 0

-1 -2 2.0 2.5 3.0 3.5 4.0 4.5 5.0

Most points are close to the line thus the assumption of normality is satisfied for the group Experimental.

Observed Value
Group Statistics Group for Questionnaire Results Control Experimental N 28 24 Mean 3.5357 3.5000 Std. Deviation .63725 .72232 Std. Error Mean .12043 .14744

Response for Q. 1 in Pre-Experiment

Independent Samples Test Levene's Test for Equality of Variances F Response for Q. 1 in Pre-Experiment Equal variances assumed Equal variances not assumed .516 Sig. .476 t .189 .188 t-test for Equality of Means df 50 46.352 Sig. (2-tailed) .851 .852 Mean Difference .03571 .03571

Here need to check if the assumption of equal variances is satisfied. According to Levene’s test (P-value = 0.476) > ( variances is satisfied.
Independent Samples Test Levene's Test for Equality of Variances F Response for Q. 1 in Pre-Experiment Equal variances assumed Equal variances not assumed .516 Sig. .476 t .189 .188 t-test for Equality of Means df 50 46.352 Sig. (2-tailed) .851 .852 Mean Difference .03571 .03571

= 0.05), thus assumption of equal

According to above table, t-test value = 0.189, P-value = 0.851. Since (P-value = 0.851) > ( = 0.05), null hypothesis is not rejected.
   

97   

Conclusion: At 5% level, there is no difference between control and experimental for question, “Please rate how interested you are in this study, with 1 being not interested and 5 being extremely interested.” Is there a difference between control and experimental for question, “Please rate your interest in the study after having completed the experiment, with 1 being not interested and 5 being extremely interested.”? Null: There is no difference between control and experimental for question, “Please rate your interest in the study after having completed the experiment, with 1 being not interested and 5 being extremely interested.” Alternative: There is a difference between control and experimental for question, “Please rate your interest in the study after having completed the experiment, with 1 being not interested and 5 being extremely interested.” Will be using t-test for independent samples. Testing normality assumption as sample sizes are < than 30.

Response for Q. 1 in Post-Experiment

5.00 4.00 3.00 2.00 26 1.00 Control Experimental

Box plot shows outliers for Experimental group. Even with the presence of outliers, normality is satisfied (See below). Also these values are important for the analysis. Thus the decision is not to remove the outlier.

Group for Questionnaire Results for Post

   

98   

Normal Q-Q Plot of Response for Q. 1 in Post-Experiment for grouppostq1= Control

Expected Normal

2 1 0

-1 -2 2.0 2.5 3.0 3.5 4.0 4.5 5.0

Most points are close to the line thus the assumption of normality is satisfied for the group Control.

Observed Value
Normal Q-Q Plot of Response for Q. 1 in Post-Experiment for grouppostq1= Experimental

Expected Normal

0.5 0.0

-0.5 -1.0 -1.5 -2.0 1.0 1.5 2.0 2.5 3.0 3.5 4.0

Except one, most points are close to the line thus the assumption of normality is satisfied for the group Experimental.

Observed Value
Independent Samples Test Levene's Test for Equality of Variances F Response for Q. 1 in Post-Experiment Equal variances assumed Equal variances not assumed .367 Sig. .548 t 1.124 1.107 t-test for Equality of Means df 43 38.079 Sig. (2-tailed) .267 .275 Mean Difference .26000 .26000

Here need to check if the assumption of equal variances is satisfied. According to Levene’s test (P-value = 0.548) > ( variances is satisfied. = 0.05), thus assumption of equal

   

99   
Group Statistics Group for Questionnaire Results for Post Control Experimental N 25 20 Mean 3.7600 3.5000 Std. Deviation .72342 .82717 Std. Error Mean .14468 .18496

Response for Q. 1 in Post-Experiment

Independent Samples Test Levene's Test for Equality of Variances F Response for Q. 1 in Post-Experiment Equal variances assumed Equal variances not assumed .367 Sig. .548 t 1.124 1.107 t-test for Equality of Means df 43 38.079 Sig. (2-tailed) .267 .275 Mean Difference .26000 .26000

According to above table, t-test value = 1.124, P-value = 0.267. Since (P-value = 0.267) > ( = 0.05), null hypothesis is not rejected.

Conclusion: At 5% level, there is no difference between control and experimental for question, “Please rate your interest in the study after having completed the experiment, with 1 being not interested and 5 being extremely interested.” Is there a difference between control and experimental for question, “Please rate how useful you feel structured approaches to the analytic process are, with 1 being not useful and 5 being extremely useful.”? Null: There is no difference between control and experimental for question, “Please rate how useful you feel structured approaches to the analytic process are, with 1 being not useful and 5 being extremely useful.” Alternative: There is a difference between control and experimental for question, “Please rate how useful you feel structured approaches to the analytic process are, with 1 being not useful and 5 being extremely useful.” Will be using t-test for independent samples. Testing normality assumption as sample sizes are < than 30.

   

100   

Response for Q. 2 in Pre-Experiment

5.00 4.00 3.00 2.00 1.00

28 2726 25

1 2 Control Experimental

Box plot shows outliers for Control group. Even with the presence of outliers, normality is satisfied (See below). Also these values are important for the analysis. Thus the decision is not to remove the outlier.

Group for Questionnaire Results for Pre
Normal Q-Q Plot of Response for Q. 2 in Pre-Experiment for grpreq2= Control

Expected Normal

1 0 -1 -2 1 2 3 4 5 6

Except one, most points are close to the line thus the assumption of normality is satisfied for the group Control.

Observed Value
Normal Q-Q Plot of Response for Q. 2 in Pre-Experiment for grpreq2= Experimental

Expected Normal

1.0 0.5 0.0

-0.5 -1.0 -1.5 3.0 3.5 4.0 4.5 5.0

Most points are close to the line thus the assumption of normality is satisfied for the group Experimental.

Observed Value

   

101   
Independent Samples Test Levene's Test for Equality of Variances F Response for Q. 2 in Pre-Experiment Equal variances assumed Equal variances not assumed .378 Sig. .541 t -1.554 -1.595 t-test for Equality of Means df 50 48.358 Sig. (2-tailed) .127 .117 Mean Difference -.38690 -.38690

Here need to check if the assumption of equal variances is satisfied. According to Levene’s test (P-value = 0.541) > ( variances is satisfied.
Independent Samples Test Levene's Test for Equality of Variances F Response for Q. 2 in Pre-Experiment Equal variances assumed Equal variances not assumed .378 Sig. .541 t -1.554 -1.595 t-test for Equality of Means df 50 48.358 Sig. (2-tailed) .127 .117 Mean Difference -.38690 -.38690

= 0.05), thus assumption of equal

According to above table, t-test value = -1.554, P-value = 0.127. Since (P-value = 0.127) > ( = 0.05), null hypothesis is not rejected.

Conclusion: At 5% level, there is no difference between control and experimental for question, “Please rate how useful you feel structured approaches to the analytic process are, with 1 being not useful and 5 being extremely useful.” Is there a difference between control and experimental for question, “Based on your experience in this experiment, how useful do you feel structured approaches to the analytic process are, with 1 being not useful and 5 being extremely useful.”? Null: There is no difference between control and experimental for question, “Based on your experience in this experiment, how useful do you feel structured approaches to the analytic process are, with 1 being not useful and 5 being extremely useful.” Alternative: There is a difference between control and experimental for question, “Based on your experience in this experiment, how useful do you feel structured approaches to the analytic process are, with 1 being not useful and 5 being extremely useful.” Will be using t-test for independent samples. Testing normality assumption as sample sizes are < than 30.

   

102   

Box plot shows outliers for Control group. Even with the presence of outliers, normality is satisfied (See below). Also these values are important for the analysis. Thus the decision is not to remove the outlier.

Most points are close to the line thus the assumption of normality is satisfied for the group Control.

Most points are close to the line thus the assumption of normality is satisfied for the group Experiment.

Group Statistics Group for Questionnai re Results for Post N

Mean

Std. Deviation

Std. Mean

Error

   

103   

Response for Q. 2 in Control 25 Post-Experiment Experiment 20 al

4.0800 4.2000

.81240 .76777

.16248 .17168

Independent Samples Test Levene's Test for Equality of Variances t-test for Equality of Means Sig. (2tailed) .617

F Response for Q. 2 Equal in Post-Experiment variances assumed Equal variances assumed not .123

Sig. .727

t -.504

df 43

-.508

41.758 .614

Here need to check if the assumption of equal variances is satisfied. According to Levene’s test (P-value = 0.727) > ( variances is satisfied. = 0.05), thus assumption of equal

Independent Samples Test Levene's Test for Equality of Variances t-test for Equality of Means Sig. (2tailed) .617

F Response for Q. 2 Equal in Post-Experiment variances assumed .123

Sig. .727

t -.504

df 43

   

104   

Independent Samples Test Levene's Test for Equality of Variances t-test for Equality of Means Sig. (2tailed) .617

F Response for Q. 2 Equal in Post-Experiment variances assumed Equal variances assumed not .123

Sig. .727

t -.504

df 43

-.508

41.758 .614

According to above table, t-test value = -0.504, P-value = 0.617. Since (P-value = 0.617) > ( = 0.05), null hypothesis is not rejected.

Conclusion: At 5% level, there is no difference between control and experimental for question, “Based on your experience in this experiment, how useful do you feel structured approaches to the analytic process are, with 1 being not useful and 5 being extremely useful.” Is there a difference between control and experimental for question, “Please rate your understanding of conceptual modeling prior to this study, with 1 being extremely low and 5 being extremely high.” Null: There is no difference between control and experimental for question, “Please rate your understanding of conceptual modeling prior to this study, with 1 being extremely low and 5 being extremely high.” Alternative: There is a difference between control and experimental for question, “Please rate your understanding of conceptual modeling prior to this study, with 1 being extremely low and 5 being extremely high.” Will be using t-test for independent samples. Testing normality assumption as sample sizes are < than 30.

   

105   

Box plot shows no outliers for both groups.

Most points are close to the line thus the assumption of normality is satisfied for the group Control.

Most points are close to the line thus the assumption of normality is satisfied for the group Experiment.

Independent Samples Test Levene's Test for Equality of Variances t-test for Equality of Means
   

106   

F Response for Q. 3 Equal variances .137 in Pre- assumed Experiment Equal variances not assumed

Sig. .713

t 1.340 1.335

df 43

Sig. (2tailed) .187

40.189 .189

Here need to check if the assumption of equal variances is satisfied. According to Levene’s test (P-value = 0.713) > ( variances is satisfied. = 0.05), thus assumption of equal

Group Statistics Group for Questionnaire Results for Q. 2, 3, and 4 N Response for Q. 3 in Control 25 Pre-Experiment Experimental 20

Mean 3.5600 3.1500

Std. Deviation 1.00333 1.03999

Std. Mean .20067 .23255

Error

Independent Samples Test Levene's Test for Equality of Variances t-test for Equality of Means

F Response for Q. 3 Equal variances .137 in Pre- assumed Experiment Equal variances not assumed

Sig. .713

t 1.340

df 43

Sig. (2tailed) .187

1.335

40.189 .189

   

107   

According to above table, t-test value = 1.34, P-value = 0.187. Since (P-value = 0.187) > ( = 0.05), null hypothesis is not rejected.

Conclusion: At 5% level, there is no difference between control and experimental for question, “Please rate your understanding of conceptual modeling prior to this study, with 1 being extremely low and 5 being extremely high.” Is there a difference between control and experimental for question, “Please rate your understanding of conceptual modeling following this study, with 1 being extremely low and 5 being extremely high.”? Null: There is no difference between control and experimental for question, “Please rate your understanding of conceptual modeling following this study, with 1 being extremely low and 5 being extremely high.” Alternative: There is a difference between control and experimental for question, “Please rate your understanding of conceptual modeling following this study, with 1 being extremely low and 5 being extremely high.” Will be using t-test for independent samples. Testing normality assumption as sample sizes are < than 30. Both Box plots show outliers. Even with the presence of outliers, normality is satisfied (See below). Also these values are important for the analysis. Thus the decision is not to remove the outliers.

   

108   

Most points are close to the line thus the assumption of normality is satisfied for the group Control.

Most points are close to the line thus the assumption of normality is satisfied for the group Experimental.

Independent Samples Test Levene's Test for Equality of Variances t-test for Equality of Means Sig. tailed) 1.000 (2-

F Response for Q. Equal 3 in Post- variances Experiment assumed Equal variances not assumed .944

Sig. .337

t .000

df 43

.000

43.000 1.000

Here need to check if the assumption of equal variances is satisfied.
   

109   

According to Levene’s test (P-value = 0.337) > ( variances is satisfied.

= 0.05), thus assumption of equal

Independent Samples Test Levene's Test for Equality of Variances t-test for Equality of Means Sig. tailed) 1.000 (2-

F Response for Q. Equal 3 in Post- variances Experiment assumed Equal variances not assumed .944

Sig. .337

t .000

df 43

.000

43.000 1.000

According to above table, t-test value = 0.00, P-value = 1. Since (P-value = 1) > ( = 0.05), null hypothesis is not rejected.

Conclusion: At 5% level, there is no difference between control and experimental for question, “Please rate your understanding of conceptual modeling following this study, with 1 being extremely low and 5 being extremely high.” Is there a difference between control and experimental for question, “Please rate how often explicit conceptual modeling has been a part of your personal analytic process prior to this experiment, with 1 being never and 5 being every time you produce an intelligence estimate.”? Null: There is no difference between control and experimental for question, “Please rate your understanding of conceptual modeling following this study, with 1 being extremely low and 5 being extremely high.” Alternative: There is a difference between control and experimental for question, “Please rate your understanding of conceptual modeling following this study, with 1 being extremely low and 5 being extremely high.” Will be using t-test for independent samples. Testing normality assumption as sample sizes are < than 30.
   

110   

Box plot shows no outliers for both groups.

Most points are close to the line thus the assumption of normality is satisfied for the group Control.

Most points are close to the line thus the assumption of normality is satisfied for the group Experimental.

Independent Samples Test
   

111   

Levene's Test for Equality of Variances t-test for Equality of Means Sig. tailed) .872 (2-

F Response for Q. Equal variances .002 4 in Pre- assumed Experiment Equal variances not assumed

Sig. .968

t .162 .162

df 43

41.211 .872

Here need to check if the assumption of equal variances is satisfied. According to Levene’s test (P-value = 0.968) > ( variances is satisfied. = 0.05), thus assumption of equal

Group Statistics Group for Questionnaire Results for Q. 2, 3, and 4 N Response for Q. 4 in Control 25 Pre-Experiment Experimental 20

Mean 2.8000 2.7500

Std. Deviation 1.04083 1.01955

Std. Mean .20817 .22798

Error

Independent Samples Test Levene's Test for Equality of Variances t-test for Equality of Means Sig. tailed) .872 (2-

F Response for Q. Equal variances .002 4 in Pre- assumed Experiment Equal variances not assumed

Sig. .968

t .162 .162

df 43

41.211 .872

   

112   

According to above table, t-test value = 0.162, P-value = 0.872. Since (P-value = 0.872) > ( = 0.05), null hypothesis is not rejected.

Conclusion: At 5% level, there is no difference between control and experimental for question, “Please rate how often explicit conceptual modeling has been a part of your personal analytic process prior to this experiment, with 1 being never and 5 being every time you produce an intelligence estimate.” Is there a difference between control and experimental for question, “Please rate how often you plan to incorporate explicit conceptual modeling into your personal analytic process in the future, with 1 being never and 5 being every time you produce an intelligence estimate.”? Null: There is no difference between control and experimental for question, “Please rate how often you plan to incorporate explicit conceptual modeling into your personal analytic process in the future, with 1 being never and 5 being every time you produce an intelligence estimate.” Alternative: There is a difference between control and experimental for question, “Please rate how often you plan to incorporate explicit conceptual modeling into your personal analytic process in the future, with 1 being never and 5 being every time you produce an intelligence estimate.” Will be using t-test for independent samples. Testing normality assumption as sample sizes are < than 30. Box plot shows outliers for Experimental group. Even with the presence of outliers, normality is satisfied (See below). Also these values are important for the analysis. Thus the decision is not to remove the outlier.

   

113   

Most points are close to the line thus the assumption of normality is satisfied for the group Control.

Except on point, most points are close to the line thus the assumption of normality is satisfied for the group Experimental.

Independent Samples Test Levene's Test for Equality of Variances t-test for Equality of Means Sig. tailed) .617 (2-

F Response for Q. Equal 4 in Post- variances Experiment assumed Equal variances assumed not .086

Sig. .770

t -.504

df 43

-.501

39.631 .619

Here need to check if the assumption of equal variances is satisfied.
   

114   

According to Levene’s test (P-value = 0.77) > ( variances is satisfied.

= 0.05), thus assumption of equal

Group Statistics Group for Questionnaire Results for Q. 2, 3, and 4 N Response for Q. 4 in Control 25 Post-Experiment Experimental 20

Mean 3.4800 3.6000

Std. Deviation .77028 .82078

Std. Mean .15406 .18353

Error

Independent Samples Test Levene's Test for Equality of Variances t-test for Equality of Means Sig. tailed) .617 (2-

F Response for Q. Equal 4 in Post- variances Experiment assumed Equal variances assumed not .086

Sig. .770

t -.504

df 43

-.501

39.631 .619

According to above table, t-test value = -0.504, P-value = 0.617. Since (P-value = 0.617) > ( = 0.05), null hypothesis is not rejected.

Conclusion: At 5% level, there is no difference between control and experimental for question, “Please rate how often you plan to incorporate explicit conceptual modeling into your personal analytic process in the future, with 1 being never and 5 being every time you produce an intelligence estimate.” Conceptual Modeling Results: Null: Number of bubbles for Pre and Post experimental CM are not different.

   

115   

Alternative: Number of bubbles for Pre and Post experimental CM are significantly different. Will be using t-test for independent samples. Testing normality assumption as sample sizes are < than 30.
80.00 36

60.00

21

40.00

20.00

0.00 Pre Post

Box plot shows outliers for both groups. Even with the presence of outliers, normality is satisfied (See below). Also these values are important for the analysis. Thus the decision is not to remove the outliers.

Bubbles

Group
Tests of Normality Kolmogorov-Smirnov Statistic df Sig. .183 24 .036 .193 23 .026
a

Bubbles

Group Pre Post

Statistic .927 .863

Shapiro-Wilk df 24 23

Sig. .086 .005

a. Lilliefors Significance Correction

Kolmogorov-Smirvov test gives p-values < ( = 0.05), thus normality assumption is not satisfied for both samples. So take a look at Shapiro-Wilk test. P-value for group Pre is > (( = 0.05), thus normality assumption is satisfied for group Pre. Shapiro-Wilk test gives p-values < ( = 0.05), thus normality assumption is not satisfied for group Post. Need to look at Normal probability plot for group Post.

   

116   

Normal Q-Q Plot of Bubbles
for group= Pre

Expected Normal

2 1 0 -1 -2 0 10 20 30 40 50 60

Most points are close to the line thus the assumption of normality is satisfied for the group Pre.

Observed Value

Normal Q-Q Plot of Bubbles
for group= Post

Expected Normal

2 1 0 -1 -2 0 20 40 60 80

Most points are close to the line with no pronounced curvature thus the assumption of normality is satisfied for the group Post.

Observed Value
Group Statistics Group Pre Post N 24 23 Mean 23.0000 30.9130 Std. Deviation 11.90542 15.60569 Std. Error Mean 2.43018 3.25401

Bubbles

   

117   
Independent Samples Test Levene's Test for Equality of Variances F Sig. Bubbles Equal variances assumed Equal variances not assumed 1.637 .207 t-test for Equality of Means df Sig. (2-tailed) 45 41.143 .056 .058

t

-1.960 -1.948

Here need to check if the assumption of equal variances is satisfied. According to Levene’s test (P-value = 0.207) > ( variances is satisfied. = 0.05), thus assumption of equal

Independent Samples Test Levene's Test for Equality of Variances F Sig. Bubbles Equal variances assumed Equal variances not assumed 1.637 .207 t-test for Equality of Means df Sig. (2-tailed) 45 41.143 .056 .058

t

-1.960 -1.948

According to above table, t-test value = -1.96, P-value = 0.056. Since (P-value = 0.056) > ( = 0.05), null hypothesis is not rejected.

Conclusion: At 5% level, Number of bubbles for Pre and Post experimental CM are not different. Null: Number of lines and arrows for Pre and Post experimental CM are not different. Alternative: Number of lines and arrows for Pre and Post experimental CM are significantly different. Will be using t-test for independent samples. Testing normality assumption as sample sizes are < than 30.

   

118   
70.00 36

Lines and Arrows

60.00 50.00 40.00 30.00 20.00 10.00 0.00 Pre Post

Box plot shows outliers for Post group. Even with the presence of outliers, normality is satisfied (See below). Also this value is important for the analysis. Thus the decision is not to remove the outliers.

Group
Tests of Normality Kolmogorov-Smirnov Statistic df Sig. .128 24 .200* .200 23 .018
a

Lines and Arrows

Group Pre Post

Statistic .942 .895

Shapiro-Wilk df 24 23

Sig. .181 .020

*. This is a lower bound of the true significance. a. Lilliefors Significance Correction

Kolmogorov-Smirvov test gives p-values > ( = 0.05), thus normality assumption is satisfied for group Pre. Kolmogorov-Smirvov test gives p-values < ( = 0.05), thus normality assumption is not satisfied for Post. So take a look at Shapiro-Wilk test for group Post. Shapiro-Wilk test gives p-value < ( = 0.05), thus normality assumption is not satisfied for group Post. Need to look at Normal probability plot for group Post.

   

119   

Normal Q-Q Plot of Lines and Arrows
for group= Post

Expected Normal

2 1 0 -1 -2 0 10 20 30 40 50 60 70

Most points are close to the line with no pronounced curvature thus the assumption of normality is satisfied for the group Post.

Observed Value
Group Statistics Group Pre Post N 24 23 Mean 26.5000 30.9565 Std. Deviation 12.39916 12.77952 Std. Error Mean 2.53097 2.66471

Lines and Arrows

Independent Samples Test Levene's Test for Equality of Variances F Sig. Lines and Arrows Equal variances assumed Equal variances not assumed .142 .708 t-test for Equality of Means df Sig. (2-tailed) 45 44.757 .231 .232

t

-1.213 -1.213

Here need to check if the assumption of equal variances is satisfied. According to Levene’s test (P-value = 0.708) > ( variances is satisfied. = 0.05), thus assumption of equal

Independent Samples Test Levene's Test for Equality of Variances F Sig. Lines and Arrows Equal variances assumed Equal variances not assumed .142 .708 t-test for Equality of Means df Sig. (2-tailed) 45 44.757 .231 .232

t

-1.213 -1.213

   

120   

According to above table, t-test value = -1.213, P-value = 0.231. Since (P-value = 0.231) > ( = 0.05), null hypothesis is not rejected.

Conclusion: At 5% level, Number of lines and arrows for Pre and Post experimental CM are not different. Null: Number of bubbles for Control and Experimental CM are not different. Alternative: Number of bubbles for Control and Experimental CM are significantly different. Will be using t-test for independent samples. Testing normality assumption as sample sizes are < than 30.
80.00 60

60.00

53

40.00 23 20.00

0.00 Control Experimental

Box plot shows outliers for both groups. Even with the presence of outliers, normality is satisfied (See below). Also these values are important for the analysis. Thus the decision is not to remove the outliers.

Bubbles

Group
Tests of Normality Kolmogorov-Smirnov Statistic df Sig. .202 24 .012 .191 47 .000
a

Bubbles

Group Control Experimental

Statistic .920 .893

Shapiro-Wilk df 24 47

Sig. .057 .000

a. Lilliefors Significance Correction

Kolmogorov-Smirvov test gives p-values < ( = 0.05), thus normality assumption is not satisfied for both samples. So take a look at Shapiro-Wilk test. P-value for group Control is > (( = 0.05), thus normality assumption is satisfied for group Control. Shapiro-Wilk test gives p-values < ( = 0.05), thus normality assumption is not satisfied for group Experimental. Need to look at Normal probability plot for group Experimental.
   

121   

Normal Q-Q Plot of Bubbles
for groupce= Experimental

Expected Normal

4 2 0 -2 -4 0 20 40 60 80

Most points are close to the line with no pronounced curvature thus the assumption of normality is satisfied for the group Experimental.

Observed Value
Group Statistics Group Control Experimental N 24 47 Mean 12.5833 26.8723 Std. Deviation 3.46306 14.25942 Std. Error Mean .70689 2.07995

Bubbles

Independent Samples Test Levene's Test for Equality of Variances F Sig. Bubbles Equal variances assumed Equal variances not assumed 17.117 .000 t-test for Equality of Means df Sig. (2-tailed) 69 55.753 .000 .000

t

-4.821 -6.504

Here need to check if the assumption of equal variances is satisfied. According to Levene’s test (P-value = 0.000) < ( variances is not satisfied. = 0.05), thus assumption of equal

   

122   
Independent Samples Test Levene's Test for Equality of Variances F Sig. Bubbles Equal variances assumed Equal variances not assumed 17.117 .000 t-test for Equality of Means df Sig. (2-tailed) 69 55.753 .000 .000

t

-4.821 -6.504

According to above table, t-test value = -6.504, P-value = 0.000. Since (P-value = 0.000) < ( = 0.05), null hypothesis is rejected.

Conclusion: At 5% level, Number of bubbles for Control and Experimental CM are significantly different. Null: Number of lines and arrows for Control and Experimental CM are not different. Alternative: Number of lines and arrows for Control and Experimental CM are significantly different. Will be using t-test for independent samples. Testing normality assumption as sample sizes are < than 30.
70.00 59

Lines and Arrows

60.00 50.00 40.00 30.00 20.00 10.00 0.00 Control Experimental 23

Box plot shows outliers for both groups. Even with the presence of outliers, normality is satisfied (See below). Also these values are important for the analysis. Thus the decision is not to remove the outliers.

Group

   

123   
Tests of Normality Kolmogorov-Smirnov Statistic df Sig. .129 24 .200* .174 46 .001
a

Lines and Arrows

Group Control Experimental

Statistic .942 .943

Shapiro-Wilk df 24 46

Sig. .182 .025

*. This is a lower bound of the true significance. a. Lilliefors Significance Correction

Kolmogorov-Smirvov test gives p-values > ( = 0.05), thus normality assumption is satisfied for group Control. Kolmogorov-Smirvov test gives p-values < ( = 0.05), thus normality assumption is not satisfied for Experimetnal. So take a look at Shapiro-Wilk test for group Experimental. Shapiro-Wilk test gives p-value < ( = 0.05), thus normality assumption is not satisfied for group Experimental. Need to look at Normal probability plot for group Experimental.

Normal Q-Q Plot of Lines and Arrows
for groupce= Experimental

Expected Normal

4 2 0 -2 -4 0 10 20 30 40 50 60 70

Most points are close to the line with no pronounced curvature thus the assumption of normality is satisfied for the group Experimental.

Observed Value
Group Statistics Group Control Experimental N 24 46 Mean 12.9167 29.3043 Std. Deviation 3.88885 12.03859 Std. Error Mean .79381 1.77499

Lines and Arrows

   

124   
Independent Samples Test Levene's Test for Equality of Variances F Sig. Lines and Arrows Equal variances assumed Equal variances not assumed 17.268 .000 t-test for Equality of Means t df Sig. (2-tailed) -6.475 -8.428 68 60.097 .000 .000

Here need to check if the assumption of equal variances is satisfied. According to Levene’s test (P-value = 0.000) < ( variances is not satisfied. = 0.05), thus assumption of equal

Independent Samples Test Levene's Test for Equality of Variances F Sig. Lines and Arrows Equal variances assumed Equal variances not assumed 17.268 .000 t-test for Equality of Means df Sig. (2-tailed) 68 60.097 .000 .000

t

-6.475 -8.428

According to above table, t-test value = -8.428, P-value = 0.000. Since (P-value = 0.000) < ( = 0.05), null hypothesis is rejected.

Conclusion: At 5% level, Number of lines and arrows for Control and Experimental CM are significantly different.

 

   

Sign up to vote on this title
UsefulNot useful