Marc L. Resnick Florida International University Miami, FL
With the advent of mass customization, global financial markets, and Internet-based business models, executive decision making has an increased need for speed, scope, and accuracy. To meet these needs, IT vendors have created executive dashboards, systems that display visualizations of critical data via interfaces that pull from corporate data warehouses. While the visualization techniques they use take advantage of the latest technology and may support complex data analysis, these systems often fall short because they do not match the schema of the decision maker nor do they support situation assessment or awareness. A design process that considers a more naturalistic decision making perspective and uses cognitive engineering techniques such as ecological interface design would provide a significant improvement to the design of executive dashboard interfaces. However there are several challenges to applying current cognitive engineering methods to the executive decision making domain. This paper will present the executive decision making domain, illustrate how cognitive engineering principles are necessary for the design of executive dashboards, and present some methodological challenges to applying these principles.

Naturalistic decision making (NDM) has been studied in a variety of domains. Some involve the use of complex interfaces, such as air traffic control (Rodgers, Mogford and Strauch, 2000), nuclear power plants (Roth, 2001) and commercial and military pilots (Amalberti and Deblon, 1992). Some involve very little technology, such as fireground commanders (Klein, 2000a). But regardless of the technological differences, these domains have many things in common, including complexity, uncertainty, significant consequences, competing goals, and time constraints (Zsambok and Klein, 1997). An additional domain that shares many of these characteristics is executive decision making. The speed and accuracy with which business executives’ decisions must be made has increased substantially over the past decade. Reasons for this acceleration vary for each industry, but include a decrease in the typical product development cycle, integration of processes such as production and development into enterprise systems, a shift to Internet-based business models, and the sensitivity of the stock market to fluctuations in a company’s financial position. The similarities between executive decision making and other NDM domains suggest that similar techniques can be used to evaluate the decision making processes and design support systems and interfaces to accommodate them. Sutcliffe and Weber (2003) provide a detailed analysis of the knowledge needed by executives and

come to a distressing conclusion. Often, the cost of acquiring knowledge exceeds its value toward executive decision making. The implications of raw data are seldom obvious and need to be interpreted within the context of collection circumstances and possible uses. They find that the key requirement of executive information systems is to provide a clear and consistent framework from which the executive can create a schema of the environment. As with pilots, fireground commanders, and other complex decision makers, identifying the general nature of the problem is key to selecting the proper approach. Sutcliffe and Weber (2003) directly correlated decision making with macro indicators of business performance and found that the accuracy of the executive’s knowledge of particular details is less important than having a structural schema that maximizes the executive’s situation awareness and his or her ability to improvise and adjust quickly. Becoming “lost” in the raw data was often worse than not having the data in the first place. This contrasts with the approach of many executive dashboard designers which is to maximize the amount of data users can access. Errors on the opposite end of the spectrum are also common. Bonabeau (2003) reports that executives frequently use intuitive decision making styles. He suggests that this stems from the natural pattern recognition capabilities of the human information processing system. The lack of specific training in decision making reduces the ability of the executive to recognize and overcome these processes. However, in



the complex environment of business decisions, intuition is frequently wrong. Notorious cases of successful intuition (such as Jack Welch of General Electric Corp.) often mask the more numerous cases of failures of intuition in the business environment. Premature commitment to a particular understanding of the environment narrows an executive’s thinlung and reduces the alternatives that will be considered. Ecological models can be used to identify the information sources that should be considered for each type of decision environment and focus the user more directly on the most diagnostic sources. This can increase the likelihood that natural pattern recognition processes and anchoring bias the user towards to the correct interpretation.

EXECUTIVE DASHBOARDS In order to assist executives in making these complex decisions, a new kind of computer system has been developed - the executive dashboard. Executive dashboards are systems that allow company executives to view key business facts, providing a circumspective view that is designed to support effective decision making. Functionality can include completely configurable displays based on ERP-connected data warehouses and analytic modules. Executive dashboards are intended to provide managers and executives with the business intelligence they need to benchmark and improve business processes, motivate employees, increase organizational learning, and improve decisionmaking to keep pace with or out-perform the competition (Gledhill, 2002). As with traditional dashboards for automobiles and aircraft, executive dashboards are composed of several gauges that represent various aspects of the system to be managed. Representations can include current status, history, rates of change, warnings and indicators, and other formats depending on the data type and source. The depiction of priority can be accomplished using typical screen design techniques such as location, size and color. However, unlike typical dashboards, the decisions that must be supported are much more complex and the amount of data that must be accessible can be enormous. Data from functions such as finance, operations, marketing, sales, distribution, competitive positioning, and others may be needed for certain kinds of high level decisions, and thus must all be available to the executive in an intuitive and intelligible fashion. For example, Figure 1 presents a hierarchical dashboard that is divided into quadrants based on the Balanced Scorecard method commonly used to manage

businesses (Kaplan and Norton, 2001). Each quadrant is divided into components , the number of which depends on the particular strategy selected by the company. Certain parameters are immediately visible, based on the color of the display, and within each component there are a number of indicators. Additionally, more detailed data, historical analysis and other advanced features are available by drilling down into each component. The kinds of decisions executive dashboards are designed to support vary widely in scope and depth. Some decisions focus on details from just one function while others require integration of data from several functions. Hung (2003) also includes trend analysis and multi-dimensional analysis as typical capabilities of an executive dashboard; thus the interface must support integrating substantial quantities of data in complex ways. However, the decisions are not the same for each business. Depending on the strategy, industry, market position and other factors, the information required to make decisions will vary. In order to support this considerable breadth of requirements, executive dashboards must be customizable for various types of users and tasks. Current designs often have powerful data analysis functionality, but are not based on cognitive engineering principles, either regarding the information content that is provided, how this information is organized for the user, or the design of specific displays and navigation controls. They often are used by executives with a great deal of business knowledge, but perhaps very little experience with executive dashboards or other decisionsupport technologies (Hung, 2003). And with the high turnover among corporate executives that has become a rule rather than a trend, experience with a particular implementation of an executive dashboard and familiarity with specific data structures cannot be assumed. However current executive dashboard designs do not always support the prospective users’ task specific needs.

It would be much more effective if executive dashboards were developed to maximize the situation awareness (Endsley, 2000a) of the executive user. It is critical that the interface draws attention to the parameters that are most influential in the current situation, facilitates the integration of these data into an assessment of the business’ current situation, and assists in the prediction of what is likely to occur next. These correspond to the three levels of situation awareness described by Endsley (2000b) as perception, comprehension, and prediction. Considering the tremendous amount of data that is contained within an




Figure 1. Sample screen from an executive dashboard (Qualitech Solutions, Inc.) executive dashboard, the interface must support a filtration process whereby executives’ attention is led to the appropriate data. One serious implication of this need is for the system to help the executive limit his or her consideration to those data that are most relevant (Pew, 2000). Wickens’ (2000) use of SA to refer to a ‘continuous extraction’ of information from the environment is perhaps the most relevant to the executive decision making domain. But while the research cited by Wickens generally refers to tasks that take minutes and hours, the executive is extracting information over months and years and must maintain a consistent schema of the competitive environment that evolves from some events but persists despite extreme shocks from others. For example, the situation awareness of executives at major book retailers such as Barnes and Noble regarding their competitive landscape went through major changes when entered the market. Amazon priced its books well below the prices found at bricks and mortar stores and made a much wider selection of books available. However the relationships that the retailers had with its distributors, publishers, franchisees, employees and other partners did not change. During the period between 1998 and 2003, the relative performance of Amazon and online retailing in general went through many stages. As this new data became available, the implications for bricks and mortar retailers needed to be constantly reevaluated. The extraction of information from data sources is often driven by the user’s goals and expectations (Endsley, 200 1). Considering the extended durations of executive decision making situations, expectations are more prevalent and more strongly ingrained into the executive’s schema, thus creating more opportunity for bias when the expectations are unfounded. Any support system must help executives identify these biases and perhaps overcome them. This is particularly relevant when a potentially large shift in the market occurs. For example, when Amazon began selling books over the Internet, the traditional retailers were forced to deal with this completely new model of competition. However, the ingrained marketing channels, distribution, vendor relationships, inventory control and other



business processes they had were not going to disappear either from reality or from the strong decision making schema that executives had developed over thousands of repetitions (Zsambok, 1997). Executive dashboards can help executives make better strategic decisions by focusing attention on those factors that are relevant to each kind of decision - some of which must be radically reconfigured in the new Internet-based market while others should not.

Unfortunately, many of the methods currently used to evaluate SA in other domains are not feasible in the executive decision making domain, both from content and methodological perspectives. Companies are very different from each other in terms of strategic goals, organizational structures, operational processes, and the products and services they offer. Within each company, there are only a few individuals who are responsible for executive level decisions. Even within each functional area (ie. marketing), there are only a few people who are involved in the highest level strategic decisions. Therefore creating an evaluation scenario with a large enough sample population to collect meaningful data is rarely possible. Whereas 747 pilots interact with very similar challenges using similar interfaces regardless of what airline they work for, this cannot be assumed for marketing executives selecting a brand strategy. Therefore new techniques that can be used with much smaller sample sizes must be developed. Pritchett and Hansman (2000) describe the advantages and disadvantages of several SA methods, but they do not consider the sample sizes required. A second difference is the timing of the decision making process. Although in many cases the actual decision must be made quickly, the development of the situation awareness occurs over months and years. Events that occurred in past times may not only contribute to general domain knowledge but also may be directly relevant to the current decision. Again considering the domain of book retailing, the ,JILranceof Amazon into the market affected pricing and delivery, but the marketing of each genre of book to particular user groups did not change. Selecting an online marketing strategy for each user group requires combining the new and old information into one integrated schema. This is problematic because running simulations that take months to complete is not a feasible analysis method. Direct measurement methods such as those described in Endsley (2000~) would not be feasible. Perhaps critical incident reports (Klein 2000b) and other

post-hoc methods (Rodgers, Mogford, and Strauch, 2000) are the most appropriate. The personalities of the executive users also may challenge the use of SA methods in ways that are not as problematic in traditional SA domains. Subjective methods require participants to reflect on how confident they are in their SA (Jones, 2000). However executives may be less able to introspect their SA than pilots and commanders, in part because of the personalities that tend to obtain executive level positions and in part because of an unwillingness to concede a lack of awareness. The long duration of each decision unit may also create challenges. SA may vary greatly between the beginning and end of a single decision, both in reality and in subjective perception.

As a result, there is a need to extrapolate the findings of SA research from other areas to gain insight into executive dashboard design. SA knowledge can be drawn from past research into the design of systems and displays for pilots and air traffic controllers, training of individuals and teams, and others to design executive dashboards that facilitate the development of SA for business executives. But this extrapolation must be done carefully by experts in both SA and the business domain to insure that it is applied appropriately. Reising and Sanderson (2002) describe a methodology using ecological design to create an interface for an industrial decision support system that accurately maps the semantics of the applied environment directly to the display. Similar techniques can be used specifically for the creation of executive dashboards. Testing of executive dashboard systems to evaluate their support of SA must also evolve to fit the needs of the business domain. One cannot test a generic implementation of an executive dashboard across several companies because the interface must be significantly customized for each one in order to be relevant. The effectiveness of these systems is limited out of context. Therefore more of a case study method that evaluates qualitative data must be used. Effective methods do not currently exist and must be developed specifically for this domain.

c -

The executive decision malung domain is a rich environment for cognitive engineering because of the complexity of the decision making environment and the importance of these decisions to the performance of the



company and often to national economies. However the current architecture of executive dashboards fails to take advantage of what cognitive engineering can provide. Cognitive engineering principles such as situation assessment and awareness are critical for successful executive decision making and can be used to improve executive dashboard design. But SA research methods must be modified to investigate executive decision making effectively. Executive decision making is a new and challenging domain for SA research and practice.

Amalberti R. and Deblon F. (1992). Cognitive Modeling of fighter aircraft process control: a step towards an intelligent on-board assistance system. International Journal of ManMachine Studies. 36,639-671. Bonabeau E. (2003). Don't trust your gut. Harvard Business Review, May, 116-122. Endsley M.R. (2000a). Theoretical underpinnings of situation awareness: A critical review. In M.R. Endsley and D.J. Garland (eds). Situation Awareness Analysis and Measurement. Lawrence Erlbaum Associates: Mahwah, NJ. Endsley M. R. (2000b). Situation Assessment Analysis and Measurement. Lawrence Erlbaum Associates: Mahwah, NJ. Endsley M.R. (2000~). Direct measurement of situation awareness: validity and use of SAGAT. In M.R. Endsley and D.J. Garland (eds). Situation Awareness Analysis and Measurement. Lawrence Erlbaum Associates: Mahwah, NJ. Endsley M.R. (2001) Designing for situation awareness in complex systems. Proceedings of the Second International Worhhop of Symbiosis of Humans, Artifacts and Environments, Kyoto, Japan. Gledhill J. (2002). Create value with IT investments: how to generate a healthy ROI across the enterprise Food Processing, Sept 2002 v63 i9 p76-81. Hung S.Y. (2003). Expert versus novice use of the executive support systems: an empirical study. Infirmation & Management, 40, 3, 177-189. Jones D.G. (2000). Subjective measures of situation awareness. In M.R. Endsley and D.J. Garland (eds). Situation Awareness Analysis and Measurement. Lawrence Erlbaum Associates: Mahwah, NJ.

Kaplan R.S. and Norton D.P. (2001). The Strategy-Focused Organization. Harvard Business School Press: Cambridge, MA. Klein G. (2000a). Sources of Power. MIT Press: Cambridge, MA. Klein G. (2000b). Analysis of situation awareness from critical incident reports. In M.R. Endsley and D.J. Garland (eds). Situation Awareness Analysis and Measurement. Lawrence Erlbaum Associates: Mahwah, NJ. Pew R.W. (2000). The state of situation awareness measurement: heading toward the next century. In M.R. Endsley and D.J. Garland (eds). Situation Awareness Analvsis and Measurement. Lawrence Erlbaum Associates: Mahwah, NJ. Pritchett A.R. and Hansman R.J. (2000). Use of testable response for performance-based measurement of situation awareness. In M.R. Endsley and D.J. Garland (eds). Situation Awareness Analysis and Measurement. Lawrence Erlbaum Associates: Mahwah, NJ. Reising D.V.C. and Sanderson P.M. (2002). Ecological Interface Design for Pasteurizer 11: A process description of semantic mapping. Human Factors, 44,2,222-241. Rodgers M.D., Mogford R.H. and Strauch B. (2000). Post hoc assessment of situation awareness in air traffic control incidents and major aircraft accidents. In M.R. Endsley and D.J. Garland (eds). Situation Awareness Analysis and Measurement. Lawrence Erlbaum Associates: Mahwah, NJ. Roth E.M. (2001). Situation Assessment Panel. Presented at the Human Factors and Ergonomics Society 45thAnnual Meeting. Human Factors and Ergonomics Society, Santa Monica, CA. Sutcliffe K.M. and Weber K. (2003). The high cost of accurate knowledge. Harvard Business Review, May, 75-82. Wickens C.D. (2000). The trade-off of design for routine and unexpected performance: implications of situation awareness. In M.R. Endsley and D.J. Garland (eds). Situation Awareness Analysis and Measurement. Lawrence Erlbaum Associates: Mahwah, NJ. Zsambok C.E. (1997) Naturalistic decision making: Where are we now? In Naturalistic Decision Making. C.E. Zsambok and G. Klein (eds). Lawrence Erlbaum Associates: Mahwah, NJ. Zsambok C.E. and Klein G. (1997). Naturalistic Decision Making. Lawrence Erlbaum Associates: Mahwah, NJ.