Policy Forum

COMPUTER SOFTWARE:

How to Validate Deployed Modeling and Simulation Programs

J. Horne*

Increasingly, modeling and simulation (M&S) programs are being used to guide decision making, the U.S. Department of Defense (DoD) being a leader in these efforts. Future Combat Systems (FCS) and Network-Centric Warfare are prime examples. Modern war relies on highly technologically based modeling and simulation of battlefields worldwide. Similar to using computers to diagnose and treat patients in medicine, M&S systems are used to assess battlefield situations and suggest solutions. Two examples are presented in a recent report to the Defense Advanced Projects Research Agency (DARPA) (1).
Effects Based Assessment Support System (EBASS) : A distributed operational assessment tool based on the principles of value focused thinking (VFT) and developed at the U.S. Military Academy to initially support the military command in Afghanistan in 2002. EBASS provides: 1) a qualitative value model which can account for the decision makers most important evaluation considerations and measures, and 2) quantitative scoring functions and weights to evaluate alternatives.

And
Senturion: Senturion is a predictive analysis software tool developed at the National Defense University (NDU) Center for Technology and National Security Policy (CTNSP). CTNSP has been testing the Senturion capability since 2002, and has begun to support the application of this new technology in DoD. Centurion is a simulation capability that analyzes the political dynamics within local, domestic, and international contexts and predicts how the policy positions of competing interests will evolve over time.

A critical transition is from simulating models of reality for guiding decision making to accomplish[ing] complex collaborative missions with no operator intervention (2) . Autonomous systems are here to stay, because for the DoD, there is The need for taking people out of dull, dirty, and dangerous missions is a primary motivator for UAS [Unmanned Autonomous Systems] technology (3). As the advent of new war fighting approaches, wherein ideally all military units ideally are coordinated as one entity, there is greater complexity beyond which humans may not be able to manage. While the 1983 movie Wargames was riddled with technical errors and produced for melodramatic purposes, the central idea of a human-created device with

AAAS Policy Forum Page 1 of 5

real-time decision making spinning out of control is not out of question, especially considering what can happen from autopoiesis, or self-organization. We now can have a synthetically-based war, in which arises a dynamic between human consciousness and a possible "artificial consciousness" in the M&S program (marked by emergent behavior). Such a conflict can be global and assume a life of its own beyond the capacity and control of the persons relying upon the M&S program to help manage the conflict. The DoD in its Broad Agency Announcement (BAA-0002) of 4 April 2008 states: The challenges however are increasingly a function of human awareness (cognition) and autonomous control levels (autonomy). Complexity at the systems level is also matched by even more complexity when these systems are aggregated with other UAS and manned systems in System of Systems and Complex System scenarios (3) meaning Current DoD T&E [Testing and Evaluation] capabilities and methodologies are insufficient to address testing of Unmanned and Autonomous Systems operating in non-deterministic and unscripted modes characteristic of future unmanned and autonomous systems (4). Failure to account for all the variables in M&S programs may result in a surprise by emergence, or unexpected behavior. Such was expressed in a Broad Agency Announcement (BAA) issued by the Office of Secretary of Defense (OSD) on 12 December 2006 as, unacceptable and unpredictable outcomes that can be detrimental to the UAS [Unmanned and Autonomous Systems], humans/systems (5). Such super-intelligent machines spinning out of control is being raised in the public consciousness by ABC televisions Last Days on Earth as one of the top seven threats to humanitys survival (6). The DoD's response to this concern has been checkered, at best. The DoD's Defense Modeling and Simulation Organization (DMSO) was established 22 June 1991 to address such issues, but it in the Fall of 2007 its office closed its doors, replaced by the Modeling and Simulation Coordination Office (MSCO). A major reason for the change was that the Modeling and Simulation Master Plan was not carried out, little progress had been made on the interoperability of modeling and simulation programs, and The senior leaders [of the DoD] thought that given 13 years, the better part of a billion dollars counting just the DMSO budget from 1994 to 2007, and many more billions of dollars throughout the Department, we should be much further along on that vision than we are (7). Overall, that vision was to conform with the DoDs main goal, ... the creation of a force that is dominant across the full spectrum of military operations persuasive in peace, decisive in war, preeminent in any form of conflict (8).

AAAS Policy Forum Page 2 of 5

Verification, Validation, and Accreditation (VV&A) is the DoD's method of helping to ensure that M&S programs have integrity and utility, the ...three interrelated but distinct processes that gather and evaluate evidence to determine, based on the simulations intended use, the simulations capabilities, limitations, and performance relative to the real-world objects it simulates (9). Failure to address meaningfully the integrity of M&S programs, especially ones used to drive combat situations, originates with a) the thinking upon which the DoD M&S world is based, and b) an unwillingness or inability to address the consequences of this thinking. A solution rests upon clarifying definitions, recognizing the limits of M&S in general, and formulating credible VV&A methods. The DoD says that validation is The process of determining the degree to which a model or simulation is an accurate representation of the real-world from the perspective of the intended uses of the model or simulation. a simulation being ... a method for implementing a model over time (10). Says the DoD, Reality is The quality or state of being actual or true, a definition that cites the 1995 Webster's II New College Dictionary. A popular dictionary definition cannot be expected to capture the complexity of this age-old problem still debated by philosophers, as a reading of the Encyclopedia of Philosophy or a web search will indicate (11). Various ideas of truth have different meanings and implications, such as theories of correspondence, consensus, and coherence. Furthermore, one needs to ask what is an accurate representation of reality from anyone's perspective (12). Any object of modeling is non-discrete and dynamic (i.e., open), and any attempt to represent it will involve identifying specifics variables and processes. The 2500-year old allegory of the cave in Platos seventh book of the Republic centers on the illusion of reality and the problems of representing it. From the modern perspective, Gdel proved that a system never can be complete, and the larger the system (more variables) the more intractable the problem of any description. A comprehensive and coordinated approach to enhancing a greater probability for safety of uninhabited autonomous systems driven by software is needed. Many new M&S programs coming on line, such as Simajin (with its management tool, Simanij) rely upon chaos generation engines to generate a simulation (13). Such a program, while fulfilling its purpose, cannot be validated, given the DoD's thinking and approach. Complexity theory further complicates issues, as well. Three considerations arise in finding a solution to the M&S validation problems. First, that the DoD seems to have deferred to the National Academies of Science for work of social science analysis and prediction ( 14) indicates the complexity of modeling social behavior and the M&S programs used to emulate it. While the DoD's action is a step in the right direction, there is a concern that the bureaucratic processes may encumber efforts at addressing the more immediate need of ensuring the safety of UAS through sufficient validation and management of emergent phenomena. To its credit, MSCO's year 2001 Human Behavior Representation (HBR) Literature Review, dated as it is (15),

AAAS Policy Forum Page 3 of 5

calls for more theoretical and cross-disciplinary research. However, the HBR approach is offset by current DoD practice of seeking mechanical ready-made solutions, as evidenced by the BAAs and the report to DARPA. The DoD admits that current methods of validation are inadequate, and that the more realistic approach would be human behavior representation, such as validating M&S programs by comparing them to human behavior models. Second, a meaningful coordinating activity has to be in place, headed by persons academically and experientially credentialed in the areas of validation, as well as the disciplines invoked in creating software used to drive complex systems. They need to call on philosophers of science, logicians, social systems experts (e.g.s: sociologists, political scientists, economists), as well as software engineers and mathematicians. Third, a higher than agency-level criticality must be assigned to interoperability, standardization, and validation. The Clinger-Cohen Act of 1996 came on the heels of the Granada debacle, when troops in the field could not communicate via incompatible equipment. Now that worldwide threats have become enormously more complex, the responses must be as equally sophisticated. The unit addressing our reliance upon synthetic war very well could be effective at the undersecretary level, where it would be better situated to receive the attention it needs. Ongoing Congressional monitoring needs to occur, as well. However jocular it may sound, at this time it would seem to be sound advice to at least figuratively to check to see if anyone is at home in that plane flying overhead or in front of the consoles controlling uninhabited autonomous systems.

References and notes

1. Verification, Validation, and Accreditation (VV&A), Final Report, Prepared for: Mr. Robert E.
Miller, Jr. Contract Officers Representative U.S. Army RDECOM, CERDEC and Dr. Alexander Kott Program Manager Defense Advanced Research Projects Agency, by: Evidence Based Research, Inc. 1595 Spring Hill Road, Suite 250 Vienna, Virginia 22182-2216, July 30, 2007, Contract: W15P7T-07-C-P209, [no author] pp. 123-125

2. Unmanned Autonomous Systems Test (UAST), Broad Agency Announcement <BAA-0002> (Test Resource Management Center, Office of the Secretary of Defense, 4 April 2008) , p. 32 3. Unmanned Autonomous Systems Test (UAST), Broad Agency Announcement <BAA-0002> (Test Resource Management Center, Office of the Secretary of Defense, 4 April 2008), p. 5 4. Unmanned Autonomous Systems Test (UAST), Broad Agency Announcement <BAA-0002> (Test Resource Management Center, Office of the Secretary of Defense, 4 April 2008), p. 17

AAAS Policy Forum Page 4 of 5

5. Unmanned Autonomous Systems Test (UAST), Broad Agency Announcement BAA UAST0001 (Test Resource Management Center, Office of the Secretary of Defense, 12 December 2006), p.6 6. 'Last Days on Earth,' Change How You See the World (ABC News,August 31, 2006; http://abcnews.go.com/2020/story?id=2319986 , accessed 14 July 2008)

7. DoD Changes Approach to Managing Modeling & Simulation (DoD Modeling and Simulation Steering Committee, MSSC News Release, 23 November 2007; http://www.msco.mil/files/MS_Mgmt_Structure_News_Release.pdf)., p. 1 8. Joint Vision 2020 ( http://www.dtic.mil/jointvision/jvpub2.htm, accessed 14 July 2008). 9. VV&A Recommended Practices Guide (DoD Modeling and Simulation Coordination Office, http://vva.dmso.mil/, accessed 9 September 2008). p. 1 10. VV&A Recommended Practices Guide Glossary, (http://vva.dmso.mil/, accessed 9 September 2008). 11. e.g., http://www.iep.utm.edu/ and http://en.wikipedia.org/wiki/Reality 12. A. Schmid, J. Artificial Societies and Social Simulation 8, (http://jasss.soc.surrey.ac.uk/8/4/5.html , 2005). 13. Simajin (http://www.rhinocorps.com/RhinoWeb/simajin/index.jsp, accessed 14 July 2008). 14. Broad Agency Announcement (BAA) Number W911NF-08-R-0007 (U.S. Department of Defense, Amery Research Office, 2008; http://www.arl.army.mil/www/DownloadedInternetPages/CurrentPages/DoingBu sinesswithARL/research/08-R-0007.pdf, accessed 9 September 2008). 15. Human Behavior Representation (HBR) Literature Review , (Modeling ans Simulation Coordination Office, 2001; http://vva.dmso.mil/, accessed 9 September 2008). 16. A. Cho, E. Kintisch, Science 321, 186 (2008).

Author for correspondence. E-mail: jhorne18@earthlink.net

AAAS Policy Forum Page 5 of 5