EVALUATION OF THE EFFECTIVENESS OF USING COMPUTER-BASED TRAINING SIMULATIONS TO DEVELOP MANAGERIAL COMPETENCY

A thesis submitted in partial fulfilment for the degree of Doctor of Business Administration
by John Kenworthy

Henley Management College/Brunel University September 2005

Abstract
Computer-based simulations and games are powerful tools to support learning environments (Swanson and Holton, 1999) and Gartner research suggests that simulations may be e-learning’s ‘killer application’ (Lundy et al., 2002). The multi-billion dollar business and management training industry and management education are beginning to turn more attention to using simulations and games but there are doubts about even the most fundamental claims of the efficacy of simulations (Feinstein and Cannon, 2002). This study tests a model in comparing a training programme using three different experiential activities, a simulation, a business game and case studies using Kirkpatrick’s (1959/60) familiar and ubiquitous (Russ-Eft and Preskill, 2001) four levels as a guiding model for evaluation. In particular, the study focuses attention on the development of managerial competencies and the differences in demonstrated competency before and after (May, 1993) a strategic management training programme (Baker et al., 1997). The literature on management learning provides insights into the multi-disciplinary nature of the research highlighting the many factors considered to influence and shape the way people learn and transfer their learning to the workplace. The literature includes consideration of enjoyment of the learning event (Prensky, 2000, Schank, 1997) and motivation (Holton, 1996), learning style (Kolb, 1976) and personality type (Patz, 1990, 1992), the potential effect of team working (Higgs, 1999) and personal background such as age, gender, cultural heritage and prior academic achievement (Sternberg, 1997). Managerial competency models are discussed and compared to establish the most fitting model for the location of the research and to provide a measure of change in behaviour and individual competencies, these are linked to organisational competence and performance – providing the links across the four levels of evaluation. An appropriate research methodology is discussed and the chosen quasiexperimental design fits the scientific research tradition demands for robust methodology and the pragmatic demands of research conducted in the business training world (Easterby-Smith et al., 1991). Operationalisation of suitable constructs and processes provide the empirical evidence repeatedly called for in previous studies. Data was collected from 266 participants working for private companies in Malaysia and Singapore. The results detected significant differences between both simulation and game groups and the case study groups in reaction, learning and learning transfer and business impact and, as implied by Kirkpatrick (1959/60), significant correlations between the different levels are observed but the strength of the relationship 2

falls well short of sufficiently explaining the results. Both programmes including the simulation or the game show higher levels of learning, change in demonstrated managerial competencies and in business performance than the case study based programme groups providing strong evidence that simulations and games are effective tools to employ within an experiential learning intervention. The continued importance of human tutors or facilitators to provide useful feedback and debriefing from the activities is strongly indicated. From the literature, factors that may influence learning are considered but the results failed to detect any differences between learning styles or personality type. However, some influence was detected with the age of participants with younger managers as suggested by Aldrich (2002), significantly enjoying the computer-based activities over older managers. The impact of team working and enjoyment of team working within the activities is discussed suggesting that the composition of the team (Belbin et al., 1976, 1981), the task and the context, whether competitive or collaborative may impact which competencies are developed. As suggested by previous research in this field (Wolfe and Guth, 1975, Keys and Wolfe, 1990, Brenenstuhl and Catalanello, 1979, Gopinath and Sawyer, 1999, Hannafin, 1992, Hannafin et al., 1996), the results provide empirical evidence that computer-based simulations and games are significantly more useful for learning and competency development than case studies.

3

Contents
CHAPTER 1 PURPOSE OF STUDY CHAPTER 2 LITERATURE REVIEW
2.1 2.2 2.2.1 2.2.2 2.2.3 2.3 2.3.1 2.3.2 2.4 2.4.1 2.5 2.5.1 2.5.2 2.5.3 2.6 2.6.1 2.6.2 2.6.3 2.6.4 2.6.5 2.7 Overview Management learning Schools of thought in management learning Evaluation models and taxonomies Management learning and evaluation summary Learning transfer Managerial competency models Learning transfer and competency frameworks summary Business results – linking individual competency models to organisation outcomes Business results summary Personality and personal background Learning styles Factors that shape and influence learning Personality and personal background summary Previous research in simulations and games Support for simulations and games Criticisms of simulation and game research Evaluation of simulations Evaluation of simulations for learning outcomes Previous research summary Literature review summary

10 13
13 15 15 24 38 41 42 52 53 57 59 60 61 67 69 70 71 72 74 76 76

CHAPTER 3 RESEARCH QUESTIONS AND HYPOTHESIS CHAPTER 4 METHODOLOGY
4.1 Key choices in methodology 4.1.1 Independence of the researcher 4.1.2 Sample size 4.1.3 Theory testing or generation 4.1.4 Experimental or fieldwork design 4.1.5 Universality 4.1.6 Verification or falsification 4.1.7 Summary key choices 4.1.8 Scientific method – ideal but inherently complex 4.2 Research model 4.2.1 Validity, reliability and generalisability 4.2.2 Issues with experimental design 4.2.3 Learning evaluation design 4.2.4 Quasi-experimental design 4.2.5 Summary research model 5.1 5.1.1 5.1.2 5.1.3 5.1.4 5.1.5 5.1.6 5.1.7 5.1.8 5.1.9 5.2 5.3 5.3.1 5.4 5.4.1 6.1 Operationalisation of constructs Personality type Preferred learning style Position in the organisation Cultural heritage Managerial competencies Bosses performance rating Reaction to the programme Learning Motivation to learn and transfer learning and transfer climate Programmes investigated Evidence collection Procedures Data analysis strategy Statistical procedures to be used in the research Effectiveness of simulations and games

78 82
82 83 83 84 84 85 86 86 87 87 88 89 90 90 93 94 94 95 95 96 96 97 98 98 98 99 102 102 106 106

CHAPTER 5 CONSTRUCTS AND PROGRAMMES INVESTIGATED 94

CHAPTER 6 RESULTS AND ANALYSIS 4

109
111

6.1.1 Correlation on Kirkpatrick’s levels of evaluation 6.2 Effect of learning styles 6.3 Effect of teams and groups 6.4 Effect of demographics

117 128 134 138

CHAPTER 7 DISCUSSION
7.1 7.2 7.3 7.4 8.1 8.2 Analysis summary Limitations Directions for further research Practitioner guide Summary of key findings and conclusions Personal learning reflection

149
149 154 156 158

CHAPTER 8 CONCLUSIONS BIBLIOGRAPHY
Appendix 1 - Strategy Programme Overview Appendix 2 – Online Version of Kolb’s LSI III Appendix 3 – Performance Rating Scale Appendix 4 – Simulation and Game Overviews Appendix 5 – MCQ Competency Descriptors Appendix 6 – Learning Scale Appendix 7 – Example Report Appendix 8 – Reaction Form

160
160 162

166
183 184 188 189 191 193 194 197

5

List of Tables
Table 1. Fourteen schools of thought about learning: Essence and use - Adapted from Burgoyne (2002) .................................................................................................................15 Table 2. Five key schools of learning - adapted from Houldsworth (2004) after Burgoyne (2002) ..................................................................................................................................17 Table 3. Four learning modes - Kolb (1976) .....................................................................22 Table 4. Four general purposes of evaluation (Easterby-Smith, 1994) ..............................25 Table 5. Russ-Eft & Preskill (2001) - Reasons to evaluate ................................................25 Table 6. Kirkpatrick's (1959/60, 1994) four levels of evaluation .......................................30 Table 7. Four stages of performance intervention (Brinkerhoff, 1987)..............................31 Table 8. Six stages of evaluation (Brinkerhoff, 1987)........................................................32 Table 9. Reasons that training is not evaluated with financial analyses (Mosier, 1990) ....39 Table 10. JCS competencies (Dulewicz and Herbert, 1992) ..............................................46 Table 11. Top Ten competencies in a KBE - Singapore HRD perspective (Lee Mei Ching et al., 2002) .........................................................................................................................48 Table 12. A study of the attributes of managerial effectiveness in Singapore - competency model - (Kenworthy and Wong, 2003) ...............................................................................49 Table 13. Managerial competencies (Spencer and Spencer, 1993) ....................................50 Table 14. Comparing MCQ (Spencer and Spencer, 1993), JCS (Dulewicz and Herbert, 1992), Lee et al. (2002), Kenworthy and Wong, (2003).....................................................51 Table 15. Kolb's four different learning styles (Johnson and Stratton, 1978) ....................60 Table 16. Possible learning outcomes for simulations (adapted from Anderson and Lawton, 1997) .....................................................................................................................75 Table 17. Summary research questions and hypotheses .....................................................81 Table 18. Key choices of research design (Easterby-Smith et al, 1991) ............................83 Table 19. MBTI dimensions (Myers and Myers 1980) ......................................................94 Table 20. Cronbach alpha reliability analysis on MCQ......................................................97 Table 21. Programme outcomes..........................................................................................99 Table 22. Linking MCQ to training programme...............................................................101 Table 23. Participant breakdown statistics in each group.................................................110 Table 24. Differences MCQ pre to post test t-test summary ............................................112 Table 25. Discriminant analysis Sim and Game group against Case Study MCQ ...........113 Table 26. Summary t test reaction ....................................................................................114 Table 27. ANOVA Reaction data by activity type ...........................................................115 Table 28. Summary t test learning ....................................................................................115 6

Table 29. Summary t tests MCQ differences....................................................................116 Table 30. Summary t test performance change.................................................................117 Table 31. Learning correlation with participant reaction..................................................118 Table 32. Correlation reaction to learning ........................................................................118 Table 33. Factor analysis reaction data.............................................................................119 Table 34. MCQ difference correlation with learning........................................................121 Table 35. MCQ difference correlation with activity reaction...........................................122 Table 36. Multiple Regression MCQ Differences, all groups and each Activity type .....123 Table 37. MCQ difference correlation with performance change ....................................124 Table 38. Correlation boss's performance rating change with boss's rating of change in MCQ .................................................................................................................................125 Table 39. Correlation participant reaction to activity and performance change...............125 Table 40. Multiple Regression - dependent boss performance rating...............................126 Table 41. Summary ANOVA table Kolb LSI preference on reaction, learning and learning transfer between groups ....................................................................................................129 Table 42. Correlation teamwork with MCQ and learning ................................................136 Table 43. Age and activity type ANOVA.........................................................................141 Table 44. Summary Research Questions and Hypotheses and Findings ..........................149

7

List of Figures
Figure 1. Kirkpatrick (1994) Four levels of evaluation and literature review overview ....14 Figure 2. Circles of learning - adapted after Mabey, Topham & Roland Kaye (1998), Binsted (1988) and, Houldsworth (1994, 2004) .................................................................18 Figure 3. Kolb's Experiential Learning Cycle (1976).........................................................22 Figure 4. Models and 'schools of thought' in evaluation (Easterby-Smith, 1986) ..............27 Figure 5. 'Chain of consequences' for a training event (Hamblin, 1974)...........................35 Figure 6. Evaluation of outcomes and link to decision making (adapted after Burgoyne and Singh, 1977) .................................................................................................................36 Figure 7. Use of evaluation style matrix (Easterby-Smith 1994) .......................................40 Figure 8. Hierarchical model of competence (Baker et al., 1997) ......................................54 Figure 9. Model of professional competence (Cheetham and Chivers, 1996)....................56 Figure 10. Individual variables of competency, competence and performance and organisation core competence (adapted from Young, 2002) ..............................................57 Figure 11. Combining LSI and MBTI dimensions (Kolb et al., 2000)...............................62 Figure 12. Three faces of simulation evaluation (adapted from Anderson, Cannon, Malik, and Thayikulwat, 1998) ......................................................................................................74 Figure 13. Research model .................................................................................................88 Figure 14. Research design .................................................................................................90 Figure 15. Mean Differences in Competency ...................................................................111 Figure 16. Correlation enjoyment and usefulness reaction on activity to learning...........120 Figure 17. Converging and other LSI ANOVA MCQ difference ....................................130 Figure 18. Converging and other MBTI LSI ANOVA MCQ difference .........................131 Figure 19. MBTI Learning style and enjoyment/usefulness of activity ...........................133 Figure 20. ANOVA MCQ and learning by activity type..................................................135 Figure 21. Enjoyment of Simulation or Game by age group ............................................139 Figure 22. Usefulness of Simulation or Game by age group ............................................139 Figure 23. Enjoyment of Case Study by age group ..........................................................140 Figure 24. Age and activity type ANOVA charts.............................................................142 Figure 25. Gender and achievement orientation ...............................................................143 Figure 26. Pre MCQ managers and senior managers........................................................144 Figure 27. Summary ANOVA MCQ differences on prior academic attainment..............145 Figure 28. ANOVA Learning difference Asian and Western heritage.............................146 Figure 29. ANOVA Simulation and Game Group - learning and developing others Asian-Western difference .................................................................................................147 Figure 30. ANOVA Cultural heritage and position, pre and post test MCQ....................148

8

Acknowledgements
I would like to thank the following people for making the completion of this draft thesis possible for me: Malcolm Higgs, for being the most encouraging, if sometimes cryptic, supervisor, always willing to take the time to share his honest academic advice and occasional steers with me. Vic Dulewicz, for his supporting supervision and steering me to fill the gaps and for laying the groundwork that a lot of this thesis is based on. David Price, who first allowed me to believe that I might be able to do this, encouraged me and took a risk in supporting my studies. Keith Gay for patient, kind guidance in my development through stage 1. Liz Houldsworth for kindly reading an early draft and her feedback and direction. ABSEL members, and in particular Jerry Gosen, Joe Wolfe and Andy Feinstein for their guidance and great insights into this research. My clients who were kind enough to support this research and in particular, Kamal, Thiru, Azman, Mei-Woo, Sureish, David, Eng-Kiat and Chris. My father, Wesley Kenworthy, for his encouragement and enthusiasm for my personal development and for his undying love and giving me the space to be who I want to be. Annie, for her love, patience throughout the many lost weekends and her support both personal and professional that has given me the opportunity and space to do this.

9

Chapter 1

Purpose of Study

The use of computer-based simulations and games has received attention more recently for both their increasingly sophisticated design and their promotion of participant interest (Mitchell, 2004). However, one of the major problems, according to Hays and Singer (1989), is how to evaluate the training effectiveness of simulations (Feinstein and Cannon, 2002). Although for more than 40 years, researchers have lauded the benefits of simulation (Wolfe and Crookall, 1998), very few of these claims are supported with substantial research (Miles et al., 1986, Butler et al., 1988). Many authors attribute the lack of progress in simulation evaluation to poorly designed studies and the difficulties inherent in creating an acceptable methodology of evaluation. Hence, this study will provide a benchmark and model to evaluate the effectiveness of the use of computer-based simulations in management development. The research compares the use of two different types of simulation and a case study approach in a quasi-experimental design assessing participant enjoyment, learning and behaviour change in the workplace following a development programme intervention. This research is designed to add to our understanding of experiential pedagogies and the evaluation of development interventions and hence the value of using specific delivery methods in management education and development to the business. It is this area in particular that is noted by many authors as lacking (Barnett, 1984, Anderson and Lawton, 1997b, Teach and Giovahi, 1988, Bedingham, 1997, Bee and Bee, 1994, Gopher et al., 1994, Phillips, 1998, Schank, 2002, Wolfe, 1990). In particular, this research considers the development of managerial competencies using computer-based business training simulations, as demonstrated learning transfer following the ubiquitous Kirkpatrick levels of training evaluation. The research includes consideration of each individual’s preferred learning style (Kolb, 1984, Honey and Mumford, 1982) to understand if this affects participant enjoyment, learning and behaviour change from a particular method of development intervention. The research will also consider each participants age and level of formal qualification to assess if there is a trend, as suggested by Aldrich (2002, 2005), that younger managers prefer and benefit more from computer-based, immersive training methods. It is important to clarify here, what the author means by simulations, as these come in many guises and the term is used ubiquitously in the field of education and management learning.

10

Kemmis et al. (1977) and MacDonald et al. (1977) usefully categorise computer assisted learning, describing four educational paradigms. 1. Instructional: Directed or programmed learning. The drill and practice of content acquisition (after Tolman, 1932, Skinner, 1950). 2. Revelatory: Discovery learning. Student exploration within parameters (after Bruner, 1973, Ausubel, 1978). 3. Conjectural: Social or constructivist learning. Modelling that allows full manipulation and testing of ideas and hypothesis (after Vygotsky, 1978, Kolb, 1984). 4. Emancipatory: Computer as a labour saving device. This cuts across the three paradigms above and deals with the degree to which student labour is authentic rather than inauthentic (Kemmis et al., 1977). Drawing also on Barr and Tagg (1995), management training simulations and games are classic examples of the Revelatory paradigm. MacDonald et al. (1977) define that within a simulation, the learner may only tinker within the parameters, not the central working of the system. Simulations and games are increasingly being introduced into educational programmes and business training (Thompson et al., 2002, Suqrue and Kim, 2004, Lane, 1995), and for over 40 years, researchers have praised the benefits of simulations (Keys and Wolfe, 1990), but very few of these claims are supported with substantial research regarding the learning benefits of the technique (Brenenstuhl and Catalanello, 1977, Hannafin et al., 1996, Gopinath and Sawyer, 1999). An article in The Wall Street Journal (Totty, 2005, Page R6), states that companies in the U.S. “spend about $60 billion a year on training their employees, but there's a good chance much of that is wasted”. Totty goes on to cite several examples of major US corporations using computer gaming in training employees effectively. According to Gartner Research, simulations may be the ‘killer application’ for e-learning (Lundy et al., 2002) and in 2005, Gartner estimate that annual spending for training worldwide is over $100 billion, and e-learning content accounts for only a little of that. When it comes to potential growth in the e-learning content market, Gartner’s Lundy suggests that much of the expected growth for e-learning will be driven by simulations (Boehle, 2005). Trade journals are replete with extolling the benefits of simulation-based learning and how it brings e-learning to a new level but there is doubt in the minds of organisation executives about using simulations. The idea of playing games at work, let alone with interactive cartoon-like characters does not strike well with many, and the branching-video and game11

style simulations are likely to encounter resistance from companies that question whether business soft skills can be taught on a computer. Summers (2004), drawing from a number of sources, estimates the size of the worldwide market for business simulations between $623 and $712 million. Estimating the number of business simulations in use by companies and academia is particularly difficult as there is little consistency in definition of business simulations, though Faria and Wellington (2004) report a thorough analysis of the academic market showing more than 52% of respondents had used a business game with expectations that use would increase. The ASTD State of the Industry Reports (Thompson et al., 2002, Suqrue and Kim, 2004) support the figures and show that these are all very positive trends for simulations, but realistically, the current use of business simulations represents a tiny proportion of overall spending on corporate training – perhaps as much as 1%. Part of the problem is that nobody has shown definitively that simulation training works in the business world. In gauging the impact of e-learning initiatives on sales, customer satisfaction, or overall company performance, training departments do not isolate simulation from other forms of online content, such as workbooks and lectures. Compounding a dearth of empirical evidence is a mindset at many companies that dismisses video minidramas, flash animation, and virtual characters as manifestations of pop culture, unsuitable for serious business instruction (Davies, 2003). However, the basic idea of simulation - the more realistic the computer experience, the more engaged the mind becomes, accelerating learning and retention - remains compelling. What the business world needs is compelling evidence that simulations and games are useful and effective – the purpose of this study is to go some way towards providing such evidence. This will be achieved through a review of the multi-disciplinary literature on management learning and competency development and how it may be evaluated, the factors that are thought to shape the way in which people learn and transfer their learning to the workplace, the appropriate measures for the business world in the form of demonstrated managerial competency and effectiveness and lastly a review on previous research in this field. An appropriate research methodology is discussed that will provide the empirical evidence demanded by business and academics and a suitable quasiexperimental design model is detailed. The thesis continues to outline the processes for evidence collection and the strategy for analysing the data. The results of the study answer the research questions and hypothesis relating to the effectiveness of business simulations in management learning put forward from the literature and the thesis concludes with the key findings and a discussion of the implications for practitioners. 12

Chapter 2
2.1 Overview

Literature Review

There is an increasing drive amongst professional training organisations (for example, the American Society for Training and Development, ASTD) for better evaluation of training and development intervention (Thompson et al., 2002, Suqrue and Kim, 2004). The basic principle driving this is for training to demonstrate its worth to organisations – whether this be its’ attributable Return on Investment (ROI) or its value in improving performance of individuals (such as gains in productivity or reduced accidents) or of the organisation (such as more efficient use of resources or demonstrable improvement in quality). Essentially, training and development costs time and money and needs to be shown to be worthwhile. The trend to evaluate the business impact of training and development programmes continues as increasing numbers of organisations worldwide undertake these evaluations (Phillips, 1999). Senior managers increasingly want to see the economic contributions - including ROI - that training and development programmes bring to their organisations. Interest in measurement and evaluation of training is spreading globally (Phillips, 1997) and increasingly as organisations implement e-learning technologies to deliver or support training in replacement of more traditional classroom instruction based programmes (Mantyla and Woods, 2001, Schank, 2002). Most organisations recognise the need and value of evaluating performance interventions though few undertake anything but basic training intervention evaluations (Russell, 1999). There is an increasing trend, particularly in the U.S.A., to measure the ROI for training and development (Phillips, 1998) whilst Reingold (1999) notes that the average U.S. company spent some $10 million on internal and external executive development in 1998 and that spending on U.S. corporate training and education for managers rose to $16.5 billion. Gartner research in 2005 puts the total global spend on training at $100 billion (Boehle, 2005). In the meantime, there is a transformation in the ways in which training and development is being delivered through e-learning, the Internet and computer-based simulations. Aldrich (2002) suggests that the next generation of learners, those aged 30 and below, having grown up with computer games, now expect to be engaged on multiple levels simultaneously with fast feedback, graphical, high simulation, immersive, user-centric learning environments. Yet, there is little empirical

13

research (Burns et al., 1990) that supports the notion of training effectiveness. Although people recognise the need to evaluate effectiveness, few do so (Phillips, 1998). Kirkpatrick’s (1959/60, 1994) four level model is the most familiar and frequently used in business training for evaluating training effectiveness, and although dated the model has still not been superseded. The literature review broadly follows these four levels. Starting with a review of the study of management learning and how it is evaluated, using Kirkpatrick’s and other evaluation models, and reflecting the multi-disciplinary approach of the research study, four further main sections: managerial competency models and holistic business competence models; a review of personality and personal background that cuts across all levels, and lastly a review of previous research in this particular field (Figure 1):

Previous research in simulations and games Competence
Business Impact

Behaviour change

Competency models

Holistic models of competency

Figure 1. Kirkpatrick (1994) Four levels of evaluation and literature review overview

Commencing with the literature on management learning positions this research study and allows the researcher to bring the multi-disciplinary threads together through the evaluation of management development, that highlight pertinent and important aspects of related disciplines on factors that are considered to shape the way people learn and how a researcher might operationalise useful constructs to better understand and observe the effectiveness of a training intervention over others to achieve the purpose of the study. Appropriate examples from research studies in the simulation and gaming literature 14

Personality & background

Management Learning
Learning

Reaction

Evaluation Models & taxonomies

illustrate connections from the broader fields of management learning, personality, background and teamwork, and this review’s final section deals explicitly with prior research in the use of simulations and games to develop managers. From this final section, the researcher can recognise the limitations and issues faced by others and heed the calls for further research to ensure that this study is a significant contribution to our understanding and knowledge.

2.2 Management learning
Learning theories abound and each show subtle and dramatic differences and combined with a vast variety of methods in teaching, training and learning facilitation (Huczynski, 2001), there is little doubt that the analysis of management learning remains a challenge. Perhaps due to the variety of models and approaches available, there doesn’t seem to be one, univocally accepted model of what constitutes management, let alone management learning, nor how to analyse it (Ivancewich and Matteson, 1996). Winterton and Winterton (1999) attempt to distil the meaning of management development through contrast of the most prevalent theories and concepts. The resulting confusion of what management development is, or is not, highlights the complexity of the field and suggests that an attempt to address the whole of management development, or management learning, may add to the confusion rather than provide a pragmatic model to fit most situations. As such, the aim of this section of the literature review is to provide a map of the theoretical field and focus on the (relatively) small part of the field as it can be applied to this research. Firstly, this section will outline the main schools of thought in management learning and associated approaches to management development, then review evaluation of management development and the dominant approaches.

2.2.1 Schools of thought in management learning
Describing fourteen schools of learning, Burgoyne (2002) provides a map of the alternative theoretical perspectives on management learning. Table 1 on page 15 shows these fourteen different theories in management learning and the different essence of how learning may be facilitated, what type of learning is likely to take place and a simple example of a learning situation that fits within the theory:

15

Table 1. Fourteen schools of thought about learning: Essence and use - Adapted from Burgoyne (2002)
Learning Theory 1. Conditioning and the connectionist approach The trait modification view View of Self Mechanical Essence to facilitate learning Clarity about desired behaviours, repetitive practice Profile of knowledge and behaviours identifying those capable of being influenced and design programme accordingly Effective communication of information in an organised fashion Help learners hone own mental models Learning is contextual adapting to environment and motivated by survival (success) Recognise individual complexity, wholeness and own mental models Creation of the process by which learners develop identity in doing or being in own eyes and others in social environment Realigning dynamic balance of the unconscious and conscious mind Learning of: Example use Simple or repetitive skills. Dog training Complex knowledge and behaviours Competency development

2.

Specification

3.

The information transfer approach The cognitive school The systems theory approach

Recorder

4.

Knowing

5.

Discovery

Knowledge and procedures Legal profession Personalised learning in complexity Research skills Contextual or situational learning Management simulations Individual learning of complex real-life existence NLP training Learning is about developing identity Leadership development

6.

7.

The humanistic and existential approach Social learning theory

Essential

Identity

8.

Psychodynamics and related approaches

Mystical

9.

Post-modernism

Decentred and fragmented

10. Situated learning theory 11. Post-structuralism

Communal

Help learners become comfortable with natural multiple identities, directions and lack of clarity Collective and informal learning in situations Learners are not fixed beings, learning can happen in many ways Activity or task based learning in context with personal and available resources Learners are part of an integrated system wide context including tools and materials Help learners become action researchers continuously learning and adapting

Learning through interactions between conscious and unconscious Psycho-therapy training and use Complex and accepting of self. Life coaching Learning takes place in the situation Apprenticeship Myriad ways of learning Self-motivated learning Learning of particular tasks or activities Pilot training Learning evolves as the whole system All encompassing Learning is reflection in context and adapting to new situations Action research

‘Vacant’

12. Activity theory

Contextualised

13. Actor network theory

Co-evolving

14. Critical realism

Hermeneutic

Since the aim of this research is to evaluate the learning and behaviour change in individuals (their competency as a manager) as a result of a computer-based simulation training event , considering Burgoyne’s 14 schools, this research falls less than neatly into two particular schools of thought, namely Trait Modification and Systems Theory Approach. There are elements of other schools of thought which partially fit – Actor 16

network theory for example and information transfer approach suggesting that the complexity of learning and any particular intervention is unlikely to fit neatly into one School of Thought. In comparing a management simulation, Strategy CoPilot® (Imparta, 2003), a management game, Strategy at the Edge (CELSIM, 2003), and case study programmes – this research may be considered to be comparing schools of thought about learning: trait modification, systems theory and information transfer respectively. Though as Burgoyne suggests, there are many different ways the arbitrary distinction between one theory and another may be drawn – however, it begins to provide a useful framework to consider what is being developed in an intervention, and how we might consider the learning entities in order to most suitably develop the person and/or their capabilities and/or knowledge. Houldsworth (2004) uses Burgoyne’s schools of thought but usefully distils them and suggests five key schools of management learning within organisations today, being recognised as the longest established: conditioning theories; experiential approach; information transfer approaches; and trait modification approach. Table 2 shows these five schools, their basis and learning principles together with example common usage:
Table 2. Five key schools of learning - adapted from Houldsworth (2004) after Burgoyne (2002)
Approach Conditioning theories Basis Stimulus/response view on learning (Burgoyne, 2002) Learner build complex maps assimilating or accommodating knowledge (Burgoyne and Stewart, 1977) Learning as a natural process of growth. (Knowles et al., 1998) Learning principles Reinforcement, practice and feedback Holistic learning, personal and subjective. Practical problem solving Sharing experience, practical problem solving around real life issues (Revans, 1971, 1980) Transmission, organisation, storage and retrieval Example common use Rote learning (times tables, verbs) Early school learning ‘the school of hard knocks’ Learning on the job

Cognitive theories

Experiential approaches

Information transfer

Product of learning as stored information

Trait modification

Learning causes change in profile of characteristics

Knowledge, skills and attributes as traits (Burgoyne, 2002) are learned through a cycle of learning (Kolb, 1976, Honey and Mumford, 1992)

Syndicate method, action learning group work and formalised on-the-job learning Subject and tutor focus. Lengthy reading lists (Burgoyne and Stewart, 1977) MBA programmes and much e-learning Competency development training. Blended learning including role play or simulations

The distinctions between the approaches build upon each other, taking elements that are understood to be most useful and incorporating these with new dimensions of understanding about human beings and learning. 17

Symons (1996) provides a simpler distinction whilst seeking to provide an underpinning understanding of the key theories and issues for those designing and delivering management development programmes. He develops characteristic types of programmes in business management, Type A and Type B. Type A programmes are more traditional in nature characterised by a teachercentric and analytical approach – similar to Houldsworth’s (2004) later grouping conditioning, cognitive and information transfer approaches. Type B programmes however, are learner centred and experiential in nature – experiential and trait modification approaches grouped from above. This classification may be too simplistic though Symons (1996) rightly suggests that there is a steady move within the management development community towards more experiential, or Type B programmes, reflecting a move away from Taylor’s (Scientific Management) division between “thinkers” and “doers” and the need for organisations to tap individuals at all levels. This move to more experiential approaches is borne out in the ASTD State of the Industry reports (Thompson et al., 2002, Suqrue and Kim, 2004). These schools of thought on management learning and programme types provide a framework from which to work but do not provide one that greatly assists the positioning of this research. Earlier work by researchers in the field of computer-assisted learning (Binsted, 1988, Topham, 1990, Houldsworth, 1994) condenses learning approaches into three circles of learning. Building on the work of Mabey et al. (1998), the author has added the trait modification approach – within which the simulation and management game of this research as part of a programme would then be placed (Figure 2).
High

Learner
autonomy

tion odifica Trait m

Experiential
Personal growth

over content

Conditioning
Basic data & Information Incl. Information Transfer approaches

Cognitive
Situation specific & response skills

Low Low

Learner autonomy over process

High

Figure 2. Circles of learning - adapted after Mabey, Topham & Roland Kaye (1998), Binsted (1988) and, Houldsworth (1994, 2004)

18

Figure 2 above, shows the trait modification approach placed where the learner has greater autonomy over the content of learning as one would anticipate that the learners would know their traits profile (Burgoyne, 2002) and the desired profile and hence what content they need. The trait modification approach includes elements from the cognitive, conditioning and experiential schools, with greater emphasis from the latter, and these are reviewed in the following paragraphs to contextualise how the development programmes under consideration in this research fit. Conditioning theories These are based on a stimulus/response view of learning. Practice, feedback and reinforcement are the main learning principles and considerable empirical research has been undertaken (Burgoyne, 2002). Although frequently criticised as being mechanistic in nature, there are some areas of training where these principles are appropriate where a Pavlovian response to a situation may be the most beneficial – for example in fire safety or emergency evacuation. It may not be appropriate as an approach to management development (Houldsworth, 2004). Information transfer Under this proposed model, information transfer approaches would be a sub-set of conditioning approaches having their roots in Greek monologue with a curriculum defined by research and tradition (Symons, 1996). Here, the teachers are subject experts transmitting information and the product of learning as stored information is tested by examination (Houldsworth, 2004). This type of learning is often regarded as ‘classroom learning’ though increasingly this approach has been the driver of the migration of much learning content onto technology-based delivery systems, commonly referred to as ‘elearning’ and probably the reason that e-learning is the subject of so much criticism in spite of the promise of anytime, anywhere accessibility. This type of learning has little to do with the individual self or the development of managers unlike the more experiential models or action learning models where managers take responsibility for their own learning. Symons (1996) however, suggests that transfer of information is a necessary precursor to other models which makes intuitive sense in that in order to use information or knowledge, it is necessary to have acquired such beforehand.

19

Cognitive theories In reaction to the inability of the behaviourism of the conditioning theories to adequately explain for much of human activity, cognitive theories arose from the concern that the stimulus/response link was not simple or straightforward (Winn and Snyder, 1996). Here, the learner builds up complex cognitive maps of the world, which can be assimilated or modified or accommodated according to the appropriateness of fit and this is discovered through a process of experimentation. The more traditional origins of cognitive psychology attach great importance to the use of psychometrics and the pivotal position of learning-styles theory (Burgoyne and Reynolds, 1997b). Unlike information transfer, cognition is holistic in approach in the Gestalt tradition (Symons, 1996) and knowledge here is considered personal and subjective where learners not only accumulate facts and data, but also the representation of patterns and relationships among and between them (Burgoyne and Stewart, 1977). Cognitive learning theories include three sub-categories of learning outcomes: Declarative, Procedural and Strategic Knowledge. Declarative knowledge The learner is typically required to reproduce or recognise some item of information. For example, White (1984) demonstrated that students who played a computer game focusing on Newtonian principles were able to more accurately answer questions on motion and force than those who did not play the game. Procedural knowledge This requires a demonstration of the ability to apply knowledge, general rules, or skills to a specific case. For example, Whitehall and McDonald (1993) found that students using a variable payoff electronics game achieved higher scores on electronics troubleshooting tasks than students who received standard drill and practice. Strategic knowledge This requires the application of learned principles to different contexts or deriving new principles for general or novel situations – referred to by others as constructivist learning (see Dede, 1997). Wood and Stewart (1987) for example, found that the use of a computer game to improve practical reasoning skills of students led to improvements in critical thinking.

20

Experiential approaches Such approaches are most often selected for management development (Suqrue and Kim, 2004, Thompson et al., 2002). Here, knowledge derives from concrete experience and each individual has the freedom of choice and the capacity to initiate action rather than just respond to circumstances (Symons, 1996). It is seen as a natural learning process of growth rather than a teacher led activity. Shared experiences within groups or syndicates are strongly emphasised over content. There are three main ideas associated with experiential approaches relevant to this study: andragogy; action learning and the learning cycle (and associated learning styles). Andragogy Knowles (1970, 1996, Knowles et al., 1998) popularised the notion that adults are considered to learn differently from children. The pedagogical model, designed for teaching children, places the teacher as responsible for all decision-making regarding learning content, delivery, timing and evaluation. The andragogical model focuses on the education of adults and is based on the precepts that adults need to know why they should learn something, that adults maintain responsibility for their own decisions and enter educational activity with greater volume and more varied experiences to children, that adults have a readiness to learn those things that they need to know in order to cope effectively with real-life situations, and adults are more responsive to internal motivators than external motivators. Burgoyne and Reynolds (1997a) consider that management learning theory has been strongly influenced by this humanistic ethic of adult education. Action learning Revans’ (1980, 1983) model of action learning shows learning reduced to a combination of programmed knowledge and questioning insight and links a cognitive philosophy with an experiential method. The basis of action learning uses the manager’s experience of the past and uncertainties of the future, refining the mental map together with a group of ‘fellows’ in a ‘set’ through tackling a particular practical problem. The expected outcomes of successful action learning are a change in behaviour following an exchange of ideas with others leading to a personal action plan adapted to the environment. Inherent within this is the belief that this will be for the better (McLaughlin and Thorpe, 1993).

21

Learning cycle A popular model of experiential learning is Kolb’s (1984) learning cycle. Here, learning is seen as a four-step process that Kolb identifies as: 1) watching, 2) thinking, 3) feeling (emotion), and 4) doing (muscle). Kolb defined learning as “… the process whereby knowledge is created through the transformation of experience.” (Kolb, 1984, p41) Drawing from three key sources: Dewey (1938) – who emphasised that learning needed to be grounded in experience; Lewin (1951) – who stressed the importance of being active in the learning; and Piaget – who considered that intelligence was the result of a person’s interaction with the environment. Learning, in Kolb’s model shown in Figure 3 below, involved four stages: the immediate concrete experience of an event which results in observations that are assimilated into theory and from which, the individual deduces new implications for future action.
Concrete Experience Experience

Active Experimentation

Testing implications of concepts in new situations Formation of abstract concepts & generalisations Abstract Conspetualisation

Observation Reflective & Reflection Observation

Figure 3. Kolb's Experiential Learning Cycle (1976)

The assumption underlying Kolb’s (1976) model is that in order to be effective, the learner needs four different kinds of abilities or basic learning modes as shown in Table 3:
Table 3. Four learning modes - Kolb (1976)

1. Concrete Experience (CE) – able to involve fully, openly, and without bias in new experiences. 2. Reflective Observation (RO) – able to reflect on and observe these experiences from many perspectives. 3. Abstract Conceptualisation (AC) – able to create concepts that integrate observation into logically sound theories. 4. Active Experimentation (AE) – able to use these theories to make decisions and solve problems.

22

This portrays the ideal learning cycle where the learner goes through each aspect – experiencing, reflecting, thinking and acting – in a recursive process responding to the learning situation, context and learning content. The learning cycle has intuitive appeal and is commonly used as a basis of experiential learning, however, situational learning theorists (Lave, 1998, Fox, 1998) propose a more radical model rejecting the cognitive structure and suggest that learning is more social in nature and that individual’s do not necessarily internalise experience as learning does not have to be explicit and declarative. Simplistically, it may be that it is possible to learn from experience without knowing what has been learned – something that Prensky (2000) suggests is a benefit of using games for learning. Trait modification Trait modification approaches to learning have a basis in the concept of the learner being describable as a broad set of traits – knowledge, skills and attitudes (Burgoyne, 2002) – and that learning is a change in the profile of these characteristics. These approaches assume, to a greater or lesser extent, the characteristics of the conditioning, cognitive and experiential approaches. Kraiger et al. (1993) synthesised the work of Gagne (1984), Anderson (1982) and others proposing very similar broad-based categories of learning outcomes: skill-based, cognitive and affective outcomes. Cognitive or knowledge outcomes are reflected in the cognitive theories section above, while skill-based learning outcomes address technical or motor skills. A number of game-based instructional programmes have been used for practice and drill of technical skills – for example, Gopher et al. (1994) investigated the effectiveness of the use of an aviation computer game by military trainees on subsequent test flights. Affective learning outcomes refer to attitudes – including feelings of confidence, self-efficacy, attitudes, preferences and dispositions. Some research have shown that games can influence attitudes, for example see Wiebe and Martin (1994) and Thomas et al. (1997). The trait modification approach is closely associated with psychometric traditions, such as Myers Briggs Type Indicators (MBTI) and the learning process is most often associated with the idea of the learning cycle and learning styles. Competency-based approaches to management development share the characteristics of the trait modification approach. Dubois and Rothwell (2004) suggest that competency-based approaches should be used when the resources and time are available

23

and the training has a significantly high strategic impact for a large population whose performance should be exemplary rather than merely successful. Management Learning Summary This research will compare the learning and learning transfer from a strategic management programme using three different activities, a management simulation, a management game and case studies. Designed to be experiential, Symons (1996) suggests that in such a Type B programme, learning will only take place when the environment encourages risk-taking, provides feedback sensitively and quickly and unambiguously. Whilst this research is concerned with the learning that is manifest by a training intervention, it is also concerned with the transfer of such learning to the workplace by means of behavioural change in the business, i.e. participants learn some things about strategy and management and use this learning in the workplace to the benefit of the business – often assumed and rarely measured (Thompson et al., 2002). This places this research firmly in the trait modification approach to management learning. The only difference between the programmes is the use of particular experiential delivery methods, namely a management simulation, a management game and case studies. The case method is the most used approach outside the traditional lecture/instruction format (Burgoyne and Mumford, 2002) and used here as a accepted standard comparison base, or control group (Wolfe and Guth, 1975, Jennings, 2000). The literature review now continues to consider how we might evaluate the development intervention and why such evaluation is important.

2.2.2 Evaluation models and taxonomies
Why evaluate training and development, and why it doesn’t always happen. Organisations place an increasing requirement to demonstrate the worth of training and development, particularly in terms of the impact on the business and often associated but rarely measured, Return on Investment (Phillips, 1991) as has been seen above. Training professionals and academics have emphasised other, non-monetary aspects of the value of training and in particular, why there is a need to evaluate training and development. Easterby-Smith (1994) notes four general purposes of evaluation (Table 4).

24

Table 4. Four general purposes of evaluation (Easterby-Smith, 1994)
1. 2. 3. 4. Proving the worth and impact of training: Designed to demonstrate conclusively that something has happened as a result of training or development activities. Improving: A formative process to explicitly discover improvements to a training programme. Learning: Where the evaluation itself is, or becomes, and integral part of the learning on a training programme. Controlling: Quality aspects in the broadest sense, both in terms of quality of content and delivery to established standards.

More recent literature uses different terminology and offers some variations along similar themes. Russell (1999), for example, does not include an explicit ‘learning’ purpose, but suggests that the evaluation of performance interventions produces benefits to determine if performance gaps still exist and if the performance intervention achieved its goals. Russell goes on to note that evaluation can assess the value of performance intervention against organisational goals and [evaluation] helps to identify necessary changes or improvements to the intervention in the future. Russ-Eft and Preskill (2001) review evaluation across the entire organisation, not just in the area of training and development, and suggest 6 distinct reasons to evaluate. Applying the comparatively simple reasons from Easterby-Smith (1994) to Russ-Eft and Preskill’s (2001) six reasons show strong similarities, suggesting that more is not necessarily better, as can be seen in Table 5 below:

Table 5. Russ-Eft & Preskill (2001) - Reasons to evaluate
• • • • • • Words Ensure quality (Controlling) Cont ribut es to increased organisation members’ knowledge (Learning) Helps prioritise resources (Improving) Helps plan and deliver organisational improvements (Improving) Helps organisation members be accountable (Controlling) Findings can help convince others of the need or effectiveness of various organisational initiatives (Proving) in brack ets author’s adaptation after Easterby-Smith (1994)

The complexity and breadth of management training and development and its cost, both in terms of time and effort as well as financial, has led to a continuing concern with evaluation (Burgoyne and Reynolds, 1997a). In organisations, the concern is more about proving and controlling whilst academia and professional consultants are often as preoccupied, if not more so, with learning and improving. The problem of evaluation is 25

multi-faceted and whilst there are useful models and processes to address these, issues remain, not least because evaluation requires time and effort. Simulation and game based performance interventions need to be evaluated for the same reasons outlined above, but pose particular additional problems with evaluation. Burns et al. (1990) consider the multi-fold problem with evaluating experiential pedagogies stating that there is firstly a need to compare the efficacy to ‘traditional’ approaches, and there is a need to compare alternative experiential pedagogies competing to achieve the same learning. Not surprisingly, they note a paucity of solid empirical evidence regarding the relative effectiveness of experiential techniques. Other authors (e.g. Pierfy, 1977) note two particular problems with respect to evaluating simulations or experiential techniques: the first being the conceptual problems pertaining to definitions, domain boundaries and the theoretical basis which underpin and frame pedagogical research. The second fundamental problem is that there remain significant methodological difficulties including experimental design, constraints within the organisations and institutions, time considerations and ethical questions associated with any comparative study. In an attempt to address these issues, it is useful to understand the models and schools of thought of evaluation to help provide a suitable framework to help conceptualise the problem and find potential solutions. Schools of thought in evaluation Easterby-Smith (1994) groups major approaches to evaluation, classifying them on two dimensions, the scientific-constructivist dimension, and the research-pragmatic dimension. The scientific-constructivist dimension represents the distinct paradigms, often incompatible ways of seeing and understanding the world (Filstead, 1979). The scientific end favours the use of quantitative techniques involving the attempt to operationalise all variables in measurable terms. This contrasts greatly with the constructivist end, which emphasises the collection of different views of various stakeholders before data collection begins. The constructivist process continues with reviewing largely, but not necessarily exclusively, qualitative data. The research-pragmatic dimension represents the contrasting styles of how evaluation is conducted. Easterby-Smith (1980) describes the two extremes as Evaluation and evaluation (author’s emboldening) – representing the research and pragmatic styles respectively. 26

Research style rightly stresses the importance of rigorous procedures, that the direction and emphasis of the evaluation study should be guided by theoretical considerations and that these are aimed at producing enduring generalisations, and knowledge about the learning and developmental process involved. The pragmatic style, in contrast, emphasises the reduction of data collection and other time-consuming aspects of the evaluation study to the minimum possible. EasterbySmith et al. (1991) point out, that when a researcher is dependant on the cooperation and given time of managers and other informants, they may, and will have, other, more important, priorities than the evaluator’s study. This usually means that the researcher is cornered into a less than rigorous process however good the intention, by balancing the desire for solid, defensible evidence against the practical problem of influencing respondents to answer all questions with little personal incentive to do so. Combining the dimensions into a useful matrix, Easterby-Smith uses arrows to show the influence of one ‘school’ on the development of the linked ‘school’ shown in Figure 4, below:

Research Experimental research Illuminative evaluation Goal-free evaluation Scientific Systems model Constructivist

Interventionalist evaluation

Pragmatic
Figure 4. Models and 'schools of thought' in evaluation (Easterby-Smith, 1986)

In order to understand the approaches to evaluation that may be considered, the review uses Easterby-Smith’s matrix, to highlight the key models and taxonomies that may be attributed to each ‘school’ of thought. Though division, in practice, is arbitrary as most evaluations contain elements of each point of view (Easterby-Smith, 1994). This provides a useful framework to divide the frequently overlapping considerations within 27

evaluation and we start with the earliest and seemingly still most utilised school of thought, especially in business training, before reviewing those that have developed since and offer some significant suggestions to consider. Experimental Research ‘school’ Experimental research has its roots in traditional social science research methodology and Easterby-Smith (1994) cites Kirkpatrick (1959/60) as the best known representative of this ‘school’ - (see also Campbell and Stanley, 1966, Campbell et al., 1970, Hesseling, 1966). The emphasis in experimental research is: 1. Determining the effects of training and other forms of management development. 2. Demonstrating that any observed changes in behaviour or state can be attributed to the training, or treatment, that was provided. This is strongly associated with the ROI approach of Philips (1997) where practitioners seek to demonstrate the monetary business impact in addition to Kirkpatrick’s levels of evaluation. On a more academic side, there is an emphasis on the theoretical considerations, preordinate designs and quantitative measurements and comparing the effects of different treatments. In a training evaluation study, this would involve at least two groups being evaluated before and after the training intervention. One group receives training, whilst the other group does not, or receives some previously known but different treatment. The evaluation would measure the differences between the groups in specific, quantifiable aspects related to their work. Russ-Eft and Preskill (2001) suggest three evaluation models of this ‘school’, in particular, are frequently used in management training: Kirkpatrick, Brinkerhoff and Holton. They do not include Bloom’s taxonomy, which tends to be associated more with education than with training, however, according to Burns et al. (1990), Bloom’s taxonomy of educational objectives (Bloom, 1956) is particularly relevant to business simulations learning assessment, as the scheme offers guidance in the pursuit of internal validity and provides a structure for the measurement of learning across studies helpful to external validity concerns. Hence, we first consider Bloom’s taxonomy before the other three models:
Bloom’s taxonomy of educational objectives: ideal but not practical

Bloom’s taxonomy identifies a hierarchy of outcomes in order to plan learning experiences, taking into account three domains of behaviour: cognitive, affective and psychomotor. They range from basic knowledge to higher level critical thinking skills. At 28

the bottom of the hierarchy, the objective is one of acquiring knowledge, which involves remembering facts, terms, and concepts. The progressive levels above are those of comprehension (explaining the material), application (using the concept to solve a problem), analysis (breaking the material down into component parts), synthesis (producing something new), and finally, evaluation (making a judgement based on a preestablished set of criteria) (Gopinath and Sawyer, 1999). Based on Bloom’s taxonomy and rigorous research standards, Gosen and Washbush (2004), stress the importance of understanding why such highest research standards have been relatively rare and proposed three reasons why Bloom’s taxonomy would be ideal, but is not practical for assessing the effectiveness of computer-based simulations: 1. Careful, rigorous research dedicated to developing a valid instrument and reflective of thought-out learning objectives is extremely time-consuming. 2. The criterion variable being used, which is learning, is illusive. In their knowledge, Gosen and Washbush suggest that every attempt to concretise this variable has failed. 3. Few researchers have called for validity studies to prove the value of lectures, cases, examinations, discussions, term papers, or any of the other typical class methods. “Why should we hold simulations to higher standards?” (Gosen and Washbush, 2004, pp 284-285) Gosen and Washbush’s (2004) research subjects are predominantly in educational environments and Bloom’s taxonomy may be ideal within such settings where learners can be coerced into ensuring that all tests are taken in a controlled environment, however, within organisations, participants have more pressing matters than answering the researchers tests (Easterby-Smith et al., 1991). A review of the management education literature shows a growing awareness of the taxonomy’s potential usefulness, particularly with college and university level educators where it is often used for curriculum design and student assessment (Athanassiou et al., 2003). The difficulties in assessing higher levels of learning and given the realities of the training world, Sackett and Mullen (1993) conclude that while one should generally take advantage of the opportunity to use a design that neatly controls for various threats, such as the rigorous use of Bloom’s taxonomy, situations will often arise when the best one can do is to implement some form of quasiexperimental design, and attempt a mix of good science with good practice. Hence, we review more commonly used models designed for the training world rather than the educational one. 29

Kirkpatrick Four Levels

Kirkpatrick created the most familiar taxonomy of a four step approach to evaluation (Kirkpatrick, 1959/60) – now referred to as a model of four levels of evaluation (Kirkpatrick, 1994). It is one of the most widely accepted and implemented models used to evaluate training interventions (Russ-Eft and Preskill, 2001). Table 6 below, shows Kirkpatrick’s four levels of measurement:

Table 6. Kirkpatrick's (1959/60, 1994) four levels of evaluation

1. Reaction to the intervention 2. Learning attributed to the intervention 3. Application of behaviour changes on the job 4. Business results realised by the organisation

This simple model is well-recognised and Russ-Eft and Preskill (2001) note that the ubiquity of Kirkpatrick’s model stems from its simplicity and understandability – having reviewed 57 journal articles in the training, performance, and psychology literature that discussed training evaluation models, they found that 44 (77%) of these included Kirkpatrick’s model. They go on to note that only in recent years, 1996 on, that several alternative models have been developed. Kirkpatrick (1998b), in an article written in 1977, considered how the evaluation at his four levels provided evidence or proof of training effectiveness and declares that such proof of effectiveness requires an experimental design using a control group to eliminate possible factors affecting the measurement of outcomes from a training programme. Without such a design, [the model] can only provide evidence of training effectiveness, but not proof (Kirkpatrick, 1998a). Kirkpatrick’s model is the basis of Phillips (1997) ROI model – often referred to as the fifth level and strongly endorsed as a preferred approach by the American Society for Training and Development (ASTD) and it’s sister organisation the ROI Network (www.astd.org). Others suggest that it would be possible and desirable to go beyond business impact and ROI and consider societal impact (Watkins et al., 1998). Whilst widely used, Kirkpatrick’s model is not without criticism. Alliger and Janak (1989) and Holton (1996) discuss the flaws of the model in detail – in essence, their critique is that the model only has limited use in education because it lacks explanatory

30

power. The model is useful in addressing broadly what happens, but not why it has happened. A useful question to ask about using the Kirkpatrick (or Phillips) model is for whom is the evaluation being conducted? At the first level of Reaction – usually called happy sheets – it is often the trainers who are most interested to discover how much they were liked, or the client (the persons accountable for the training investment) – who frequently use this as the only measure of training effectiveness (Thompson et al., 2002, Suqrue and Kim, 2004). This does not mean that reaction to training is unimportant, however, if its purpose is to demonstrate effectiveness it has questionable validity. Level 2, learning is clearly of interest to the trainees and trainers and may be of interest to others as well. Behaviour change on the job is likely to be of interest to the trainee’s managers – and to trainers as well, particularly if they are interested in understanding the impact of training. Though some argue that learning and behaviour change only occurs after failure (Schank, 1997) which is rarely an enjoyable experience of itself and hence the link from reaction to learning to behaviour change may not always be in place. On a personal note, I recall vividly not enjoying Latin class in school at all yet to this day I can recite any number of verb conjugations that I have never used in real life. Kirkpatrick’s model assumes that the levels represent a causal chain such that positive reactions lead to greater learning, which in turn produces greater transfer and hence to more positive business impact or results. Kirkpatrick is, however, vague about the actual nature of the linkages, his writings do imply that a simple causal relationship exists between the levels of evaluation (Holton, 1996). Other authors suggest that Kirkpatrick’s model is incomplete for other reasons. One example is Brinkerhoff whose model we shall consider next.
Brinkerhoff – adding prior evaluation stages

Brinkerhoff’s (1987, 1988) model has its roots in evaluating training and HRD interventions. Brinkerhoff’s cyclical model consists of six stages grouped into the following four stages of performance intervention (Table 7):

Table 7. Four stages of performance intervention (Brinkerhoff, 1987)

• • • •

Performance analysis Design Implementation Evaluation

31

Brinkerhoff’s model addresses the need for evaluation throughout the entire human performance intervention process (Table 8) and insists that the evaluation should be determined by the purpose for conducting the evaluation, i.e. that the purpose should reflect the audience’s concerns.

Table 8. Six stages of evaluation (Brinkerhoff, 1987)

1. Goal setting – identify business results and performance needs and determine if the problem is worth addressing 2. Program design – evaluation of all types of interventions that may be appropriate 3. Program implementation – evaluates the implementation and addresses the success of the implementation 4. Immediate outcomes – focuses on learning that takes place during the intervention 5. Intermediate outcomes – focuses on the after-effects of the intervention, some time following the intervention 6. Impacts and worth – how the intervention has impacted the organisation, the desired business results and whether it has addressed the original performance need or gap

Whilst using different terminology to ensure clarity and distinction between steps in the evaluation process, Brinkerhoff’s model essentially adds two prior stages to Kirkpatrick’s four levels evaluating that the performance intervention is necessary and appropriate. The process suggested by Brinkerhoff explicitly considers other stakeholders in the evaluation – a valuable addition to the implication in Kirkpatrick’s work – providing greater understanding for the purpose of confirmation and for continuously improving development programmes and performance.
Holton’s HRD evaluation research and measurement model- theory driven and less pragmatic

Holton (1996) criticised Kirkpatrick’s model arguing that the model is a taxonomy and not a fully developed theoretical model. He agrees with and identified three outcomes of training (learning, individual performance and organisational results) however suggests that these are affected by primary and secondary influences. Learning is affected by trainee’s reactions, their cognitive ability, and their motivation to learn. The outcome of individual performance is influenced by motivation to transfer their learning, the programme’s design and the condition of the training transfer. Organisational results are 32

affected by the expectations for return on investment, the organisation’s goals, and external events and factors. Holton’s model bears great similarity to Kirkpatrick’s and highlights an important factor not present – that of the motivation of a trainee to transfer their learning to the workplace. Holton’s model may be more testable than others, in that it is the only one that identifies specific variables that affect the impact of training through identification of various objects, relationships, influencing factors, predictions and hypotheses. However, as pointed out by Russ-Eft and Preskill (2001), it is related to a theory-driven approach to evaluation and less pragmatic than Kirkpatrick’s. Kirkpatrick’s model remains ubiquitous within the business HRD community in spite of potential flaws, its simplicity and intuitiveness perhaps at least encouraging evaluation practice however, on a more academic level there are other issues to consider with an experimental approach. Issues with experimental approach There are a number of reasons why the experimental approach to training evaluation may not work as well as it might promise, particularly with management training where sample sizes are limited. Easterby-Smith (1994), discusses four main reasons: sample size, control groups, causality and time & effort.
Sample size

Using statistical techniques, an essential component of the experimental approach, samples need to be large to discover statistical significances when evaluating management development. This becomes particularly problematic when group sizes are typically often less than ten and rarely greater than thirty.
Control groups

Achieving genuine control groups in management development is especially difficult. One study by Easterby-Smith and Ashton (1975) is pertinent here: the selection of the group to receive training were ‘closer’ to their bosses than those who were not selected – the result, counter-intuitively, was that the non-selected group outperformed the selected group, perhaps to deliberately upset the selected group or perhaps because the training intervention simply provided the non-selected group the time and opportunity to become ‘closer’ to their bosses and prove their worth.

33

Causality

The intention of experimental design is primarily to demonstrate causality between the training intervention and any subsequent outcomes. However, it is often hard to isolate the intervention from other influences on the manager’s behaviour – the study cited above provides a possible example of this. It may be possible to reduce the clutter of other influences for training interventions that have a clear, specific and identifiable skills focus. Interventions of a more complex nature may be subject to myriad external influences between evaluations.
Time & Effort

Kirkpatrick (1998b) identifies that in proving the effectiveness of training, the time of the post-test evaluation must be undertaken long enough after the intervention for the participant to have had an opportunity to change behaviour or for results to be realisable. The additional effort of evaluating training is regularly cited as being the main reason for not undertaking it (Phillips, 1991, 1998). It adds to the burden of the trainer or HRD team and the evaluation results are not the foremost priority of the organisation (Russ-Eft and Preskill, 2001). Illuminative evaluation The illuminative evaluation ‘school’ takes issue with the comparative and quantitative aspects of the experimental approach. Instead, this view tends to concentrate on the views of different people in a more qualitative way. The aim of illuminative evaluation is to discover not how a training intervention performs on standard measures, but what factors and issues are important to the participants in that particular situation (Parlett and Hamilton, 1987). However, formal illuminative evaluation has been noted to be more costly than anticipated (Jenkins et al., 1981), and has been rejected by sponsors (Easterby-Smith, 1994). Systems model The systems model ‘school’ of evaluation falls into the tradition of behavioural objectives approach (Russ-Eft and Preskill, 2001) and is similarly scientific to the experimental ‘school’ though more pragmatic. There are three main features of the systems model, starting with the objectives with an emphasis on identifying the outcomes of training and a stress on providing feedback about these outcomes to those involved in providing training inputs (Easterby-Smith, 1994). 34

Evaluation here assesses the total value of training in social as well as financial terms. Hamblin (1974), suggests that evaluating social as well as financial terms is overambitious. Also widely referenced, Hamblin (1974), devised a five-level model (Figure 5) similar to Kirkpatrick’s. Hamblin adds a fifth level that measures ‘ultimate value variables’ of human good.

Training Event

Reactions effects Learning effects Job behaviour effects Organisation effects Ultimate value effects

Figure 5. 'Chain of consequences' for a training event (Hamblin, 1974)

An important feature of Hamblin’s work is the emphasis on measurement of outcomes from training at different levels. It assumes that any training event will, or can, lead to a chain of consequence. Hamblin suggests that it would be unwise to conclude from an observed change at one of the higher levels of effect, that this was due to a particular training intervention, unless one has followed the chain of causality through the intervening levels of effect. Should a job change behaviour, for example, be observed, the constructivist take on this would be to ask the individual for his own views of why they were now behaving in a different way and then compare this interpretation with the views of one or two close colleagues. The stress on feedback to trainers and decision makers in the training process is an important feature of the systems model ‘school’. Warr et al. (1970) take a very pragmatic 35

view of evaluation, suggesting that it should be of help to the trainer in making decisions about a particular programme as it is happening - reflecting in action to continuously improve the process as it happens (Schon, 1983). Rackham (1973) builds on earlier work (Warr et al., 1970) making a further distinction between assisting decisions that can be made about current programmes and feedback that can contribute to decisions about future programmes. Rackham notes that the process of feedback from one programme to the next resulted in clear improvements when the programmes were non-participative in nature – but in programmes involving a lot of participation, there was no apparent improvement after feedback to improve future programmes. Feedback, as an important aspect of evaluation, is further developed by Burgoyne and Singh (1977) who distinguish between evaluation as feedback and feedback adding to the body of knowledge. The former they saw as perishable data of momentary value directly to decision-making and the latter, as generating permanent and enduring knowledge about education and training processes. Burgoyne and Singh relate evaluative feedback to a range of decisions about training in the broadest sense. Figure 6 shows an adaptation of their model with examples of decisions at each level. This not only highlights the critical importance of the evaluation and feedback process, but also the level of importance of each decision to the training and development process.

Policy
Role of the training institution Provision of funding and resources

Strategy Decision levels
Optimisation of resources Organisation of the institution

Body of knowledge about training and education

Programme
Longer or shorter programme? More or less structured? Internal or external speakers?

Method
Employ a lecture, case study or simulation to introduce topic?

Evaluation research

Intra-method
Straight delivery to lively debate? Programme Course/Event Learning Behavioural/ organisational outcomes

Figure 6. Evaluation of outcomes and link to decision making (adapted after Burgoyne and Singh, 1977)

36

The systems model has been widely accepted, especially in the U.K., but there are a number of problems and limitations with the systems model approach to evaluation – over feedback, the emphasis on outcomes and on the establishment of objectives. Easterby-Smith (1994) suggests that feedback, as data provided from an evaluation of a past event, can only contribute marginally to decisions about the future due to a legacy of the past training event. Thus, feedback can highlight incremental improvements based on a previous design, but not accentuate when radical change is needed. The emphasis on outcomes provides a good and logical basis for evaluation but it represents a mechanistic view of learning. In the extreme, this suggests that learning consists of facts and knowledge being placed in people’s heads and that this becomes internalised and gradually incorporated into behavioural responses. Indeed, this criticism is often levelled at many forms of e-Learning (Schank, 2002). MacDonald-Ross (1973) questions if there is any value in specifying formal objectives for training at all, since among other things, this might place unnecessary restrictions on the learning that could be achieved. This leads to the next major evaluation ‘school’: goal-free evaluation. Goal-free evaluation A radical view that the evaluator should take no notice of the formal objectives of a programme during its investigation was proposed by Scriven (1972). Goal-free evaluation leans more to the constructivist method where the evaluator should avoid discussing, or even seeing, published objectives of the programme, and through a discovery process with participants, establish what happened and what was learned. Goalfree evaluation avoids the risk of narrowly defining the scope of enquiry and missing unanticipated learning from a programme (Patton, 1990). This may not help the trainer improve the programme towards the intended objectives and is likely to be timeconsuming as an evaluation process; however, identifying learning that is incidental or extra can be of immense value. Interventionalist evaluation Contrasting responsive, or illuminative, evaluation with the preordinate approach of experimental research, Stake (1980) suggests that the latter requires the design to be clearly specified and determined before the evaluation begins – it makes use of objective measures evaluating these against criteria determined by programme staff. Responsive evaluation, on the other hand, is concerned with programme activities rather than intentions – taking into account different value perspectives. Additionally, Stake (1980) 37

stresses the importance of responding to audience requirements for information (unlike goal-free evaluators who distance themselves from principal stakeholders). Guba and Lincoln (1989) take this method further in what they call responsive constructivist evaluation. They recommend starting with the identification of stakeholders and their concerns, arranging for these concerns to be exchanged and debated before collection of further data. Patton (1978), stresses the importance of identifying the motives of key decision makers and their information needs and recognising that some stakeholders have more influence than others (which Guba and Lincoln (1989) argue should not be the case) but goes further by concentrating on the uses of the subsequent information. Interventionalist evaluation including it’s guises of responsive and utilisationfocussed evaluation (Easterby-Smith, 1994) is in danger of being so flexible in its approach that it may produce results that are weak and inconclusive as the whims and circumstance of each stakeholder changes. Impartiality and credibility may also be reduced as the evaluator becomes too close to the programme and stakeholders.

2.2.3 Management learning and evaluation summary
Many researchers and organisations measuring the effects of training have looked at one or more of the outcomes identified by Kirkpatrick (1959/60, 1994): reaction, learning, behaviour, and results. It is reasonable to assume that enjoyment is a precursor to learning, and that if a trainee enjoys the training, they are likely to learn (Russ-Eft and Preskill, 2001). However in a meta-analytic study combining the results of several other studies, Alliger et al. (1997) found no support for the assumption. Kirkpatrick’s second level, Learning, is the most commonly used measurement after trainee reaction to assess the impact of training. Studies investigating the relationship between learning and work behaviour have shown mixed results and offer little concrete evidence to support the notion that increased learning from a training programme relates to performance in the organisation (Collins, 2004). Transfer of Learning is the application of knowledge, skills and attitudes acquired during training to the work setting. Some research in this area has focused on comparison of alternative conditions to training transfer. A typical design of such research compares groups who receive different training methods and/or a control group that receives no training. Frameworks to investigate learning transfer have been developed by a number of researchers with some commonality that includes three phases: pre-training phase –including needs analysis; learning phase – the delivery of the training; and a post38

course phase – management of the work environment to promote transfer (see Huczynski and Lewis, 1980). Using the transfer factors highlighted in such studies, Lim and Johnson (2002) identified that the opportunity to use [the learning] on the job was the main reason for high transfer, conversely, lack of opportunity to apply on the job was the main reason for low transfer. The research into learning transfer indicates that some for of post-training transfer strategy facilitates transfer (Russ-Eft and Preskill, 2001). Results of training in terms of business results, financial results and return on investment is much discussed in the popular literature – most offering anecdotal evidence or conjecture about the necessity of evaluating training’s return on investment and methods that trainers might use to implement such an evaluation. Solid research on this topic is not, however, so voluminous. Mosier (1990) proposes a number of capital budgeting techniques that could be applied to evaluating training whilst recognising that there are common reasons why such financial analyses are rarely conducted or reported (Table 9):

Table 9. Reasons that training is not evaluated with financial analyses (Mosier, 1990)

• • • •

It is difficult to establish a monetary value for many aspects of training No usable cost-benefit tool is readily available The time lag between training and results is problematic Training managers and trainers are not familiar with financial models

Phillips (1991, 1998) used Kirkpatrick’s four levels and created a useful framework for measuring ROI – however, its use requires considerable time and effort and is typically used only for the more expensive training interventions and results most often are not released by the organisations conducting them to the public domain. Reviewing the history and development of training evaluation research shows that there are many variables that ultimately affect how trainees learn and transfer that learning to the workplace. Russ-Eft and Preskill (2001) suggest that a comprehensive evaluation of learning, performance and change would include the representation of a considerable number of variables. In such an approach, whilst laudable in terms of academic research, organisations need to recognise the trade-off between the cost and duration of the evaluation process and increasing the quality of the information it generates (Warr et al., 1970).

39

A considerable amount of research has been carried out with a great variety of focal theories, usually looking for consistent relationships between educational methods and learning outcomes. The underlying theory appears to be taken from behaviouralist psychology – where the view is that what the subject (patient) does (behaviour or response) is a consequence of what has been done to him (treatment or stimulus) (Hamblin, 1974). There are no forced rules about the style of evaluation to be used for a particular purpose, however, Easterby-Smith (1994) suggests that studies aimed at fulfilling the purpose of proving will tend to be located towards the research end of the dimension, and studies aimed at improving, will tend towards the pragmatic end. On the methodological dimension, there may be more concern with proving at the scientific end, and learning at the constructivist end (Figure 7).

Research Experimental research Illuminative evaluation Goal-free evaluation Constructivist Learning

Proving

Scientific

Proving Improving Systems model

Interventionalist evaluation

Pragmatic
Adapted from Easterby-Smith

Figure 7. Use of evaluation style matrix (Easterby-Smith 1994)

Management development which is intended to increase the effectiveness of managers can be evaluated at a number of levels and using any individual or combination of models. Kirkpatrick’s (1959/60) framework remains the most widely used and most organisations carry out evaluation at the reaction level, some measure learning but few attempt to assess changes in behaviour or the impact on the business. This research study aims to demonstrate how effective the use of management simulations or games are in developing managerial competency in comparison to the use of case studies – as such, the evaluation model of the training programme leans towards the proving ends of each of Easterby-Smith’s dimensions in the above matrix and strongly suggests that this is experimental research. We now need to review and understand in more detail the aspects of transfer of learning and how it impacts the business. 40

2.3 Learning transfer
The literature about learning transfer suggests multiple definitions though there is general agreement that transfer involves the application, generalisability and maintenance of new knowledge and skills (Ford and Weissbein, 1997). A number of writers identify specific factors that affect transfer particularly relating it to work environment factors (Awoniyi et al., 2002, Cromwell and Kolb, 2002, Lim and Johnson, 2002) and the associated transfer climate supporting or inhibiting transfer (Holton et al., 2001). Ruona et al. (2002) argue that the transfer climate is only one of a network of factors that affect transfer of learning to the workplace, other factors include the training design, personal characteristics, the opportunity to use the training, and motivational influences. A notable issue with many of the studies cited above is the opportunity for trainees to utilise the learning in the workplace – this is understandable in regard to highly specific technical or task-specific skills, but to a lesser extent to general management skills. In this research, exactly what is to be measured as part of the evaluation is a particularly problematic area. Aspects of behaviour or reaction that are relatively easy to measure are usually trivial. Measuring change in behaviour, for example, may require the observations to be reduced to simpler or more marginal characteristics. Alternatively, these characteristics may be assessed by the individual’s subordinates – however, the purist would no doubt claim that such holistic judgments are of dubious validity. The problem is that the general requirement for quantitative methods tends to produce a trivialisation in the focus of the evaluation. According to Bedingham (1997, p89), ultimately “the only criteria that make sense are those which are related to on-the-job behaviour change”. Bedingham (1997) advocates the use of 360º questionnaires that objectively measure competency sets and skills applicable to most organisations, functions or disciplines and making the results of the feedback taken immediately prior to the event available to trainees during their training – thus allowing individuals to easily see how they actually do something and the relevance of the training. Thus, they can then start transferring the learned skills immediately on return to the workplace. The downside of providing feedback on pre-test measurements during the training is that this may immediately skew the results, as will be discussed in the methodology section later. The inhibitors to learning transfer, particularly within the locale of this research, are well shown in research by Tsang (1997), who set out to discover what and how Singaporean companies learn from direct investment in China (FDIs) and from conducting joint ventures with Chinese companies. His research design involved a sample of 19 Singaporean companies with business experience in China, he then carried out 41

approximately 80 interviews with Singaporean and Chinese managers working for these companies. Tsang also examined meeting records and reports. On the basis of his data, he concluded that Singaporean companies rarely learn much from their business links in China, although there is considerable evidence of technological and managerial systems transfer to the Chinese partners. From the evidence, Tsang inferred a number of reasons • • Singaporeans felt that their systems were superior to those in China and therefore would learn little from their Chinese partners There was little transfer of learning back to the parent company because no institutional structures were set up for this purpose The interesting conundrum posed here is that whilst Singaporeans seek greater exposure to management abroad, once back in the home organisation, they may not put newly developed competencies and capabilities to effective use for reasons suggested by Holton et al. (2000) and Russ-Eft and Preskill (2001). It is a popular assumption that the purpose of management development is that it provides the means and the route to improving performance and in creating or maintaining competitive advantage, yet there remains little robust empirical evidence to support this. According to Mumford (1994) it is necessary to define effective management behaviour and an increasingly common way of doing so is by understanding and defining what managers do, how they behave and what they should do to be successful (Robotham and Jubb, 1996). Managerial competencies and the movement associated with their definition and understanding provides a way of defining effective management. Schroder (1989), for example, suggests that competences are simply personal effectiveness skills, whilst others consider competences as being linked to personality and therefore, within the context of the intended research, potentially impact on the understanding of managerial performance and effectiveness. The next section reviews managerial competency models and frameworks building on the literature above identifying this research to be within the trait modification approach to management learning - where competency models have become a common way to identify and plan development programmes.

2.3.1 Managerial competency models
There is a growing level of interest and focus on managerial competences and managerial performance with a wealth of literature (see Boyatzis, 1982, Finn, 1993, Sarawano, 1993, Spencer and Spencer, 1993, Higgs, 1999, Young, 2002), with considerable debate and confusion over the definition of competences and competencies 42

(Finn, 1993). The author will follow the distinction from Young (2002), in the meantime, the terms used are as used in the citation. Many researchers have attempted to identify and isolate the competencies or characteristics or dimensions of superior performers in the practice of management. McClelland is often cited as the father of the modern competency movement. In 1973, he challenged the then orthodoxy of academic aptitude and knowledge content tests, as being able to predict performance or success in life as being biased against minorities and women (Young, 2002). Identified through patterns of behaviour, competencies are characteristics of people that differentiate performance in a specific role or job (McClelland, 1973, Kelner, 2001). Competencies distinguish well between roles at the same level in different functions and also between roles at different levels (even in the same function) often by the number of competencies required to define the role. A competency model for a middle manager is usually defined within ten to twelve competencies, of those two are relatively unique to a given role. Kelner (2001), cites a 1996 unpublished paper by the late David McClelland were he performed a meta-analysis of executives assessed on competencies, where McClelland discovered that only eight competencies could consistently predict performance in any executive with 80 percent accuracy. Competence and Competency The concept of competence remains one of the most diffuse terms in the organisational and occupational literature (Nordhaug and Gronhaug, 1994). Exactly what does an author mean when using any of the terms of competence? The concept of individual competence is widely used in human resource management (Boyatzis, 1982, Schroder, 1989, Burgoyne, 1993). This refers to a set of skills that an individual must possess in order to be capable of satisfactorily performing a specified job. Although the concept is well developed, there is continuing debate about its precise meaning. Others take a job-based competence view that according to Robotham and Jubb (1996) can be applied to any type of business where the competence-based system is based on identifying a list of key activities (McAuley, 1994) and behaviours identified through observing managers in the course of doing their job. A useful view is to look at competence to mean a skill and the standard of performance, whilst competency refers to behaviour by which it is achieved (Rowe, 1995). That is, competence describes what people do and competency describes how people do it. 43

Rowe (1995, p16) further distinguishes the attributes an individual exhibits as “morally based” behaviours – these are important drivers of behaviours but especially difficult to measure – and “intellectually based” behaviours as capabilities or competencies. Capabilities are distinguished as these refer to development behaviours – i.e. are graded to note development areas to improve behaviours in how people undertake particular tasks. Young (2002) develops on a similar theme and builds on Sarawano’s (1993) model, linking competency and competence to performance and identifies competency as a personal characteristic (motives, traits, image/role and knowledge) and how the individual behaves (skill). Competence is what a manager is required to do – the job activities (functions, tasks). These in turn lead to performance of the individual [manager]. Jacobs (1989) considers a distinction between hard and soft competences. Soft competences refer to such items as creativity and sensitivity, and comprise more of the personal qualities that lie behind behaviour. These items are viewed as being conceptually different from hard competences, such as the ability to be well organised. Jacob’s distinction fits neatly into Young’s model with hard competences referring to identifiable behaviours, and soft competences as the personal characteristics of the individual. Further distinctions relate to the usefulness of measuring competenc[i]es. Cockerill et al. (1995) define threshold and high-performance competences. Threshold competences are units of behaviour which are used by job holders, but which are not considered to be associated with superior performance. They can be thought of as defining the minimum requirements of a job. High performance competences, in contrast, are behaviours that are associated with individuals who perform their jobs at a superior level. In the UK, the Constable and McCormick Report (1987) suggested that the skill base within UK organisations could no longer keep pace with the then developing business climate. In response, the Management Charter Initiative sought to create a standard model where competence is recognised in the form of job-specific outcomes. Thus, competence is judged on performance of an individual in a specific job role. The competences required in each job role are defined through means of a functional analysis – a top-down process resulting in four levels of description: • • • • Key purpose Key role Units of competence Elements of competence

44

Elements are broken down into performance criteria, which describe the characteristics of competent performance, and range statements, which specify the range of situations or contexts in which the competence should be displayed. The MCI model now includes personal competence, missing in the original, addressing some of the criticisms levelled at the MCI standards. Though the model tends to ignore personal behaviours which may underpin some performance characteristics, particularly in the area of management, where recent work has indicated the importance of behavioural characteristics such as self-confidence, sensitivity, proactivity and stamina. The US approach to management competence, on the other hand, has focused heavily on behaviours. Boyatzis (1982) identifies a number of behaviours useful for specifying behavioural competence. Schroder (1989) also offers insights into the personal competencies which contribute to effective professional performance. Personal competencies and their identifying behaviours form the backbone of many company-specific competency frameworks and are used extensively in assessment centres for selection purposes. This is because behavioural (or personal) competence may be a better predictor of capability – i.e. the potential to perform in future posts – than functional competence – which attests to competence in current post. The main weakness of the personal competence approach, according to Cheetham and Chivers (1996), is that it doesn’t define or assure effective performance within the job role in terms of the outcomes achieved. In his seminal work “The Reflective Practitioner”, Schon (1983) attempts to define the nature of professional practice. He challenges the orthodoxy of technical rationality – the belief that professionals solve problems by simply applying specialist or scientific knowledge. Instead, Schon offers a new epistemology of professional practice of ‘knowing-in-action’ – a form of acquired tacit knowledge – and ‘reflection’ – the ability to learn through and within practice. Schon argues that reflection (both reflection in action and reflection about action) is vital to the process professionals go through in reframing and resolving day-to-day problems that are not answered by the simple application of scientific or technical principles. Schon (1983) does not offer a comprehensive model of professional competence, rather he argues that the primary competence of any professional is the ability to reflect – this being key to acquiring all other competencies in the cycle of continuous improvement. There are criticisms of competency-based approaches to management and these tend to argue that managerial tasks are very special in nature, making it impossible to capture and define the required competences or competencies (Wille, 1989). Other writers 45

argue that management skills and competences are too complex and varied to define (Hirsh, 1989, Canning, 1990) and it is an exercise in futility to try and capture them in a mechanistic, reductionist way (Collin, 1989). Burgoyne (1988) suggests that the competence-based approach places too much emphasis on the individual and neglects the importance of organisational development in making management development effective. It has also been argued that generic lists of managerial competences cannot be applied across the diversity of organisations (Burgoyne, 1989, Canning, 1990). Considering the wealth of often contradictory literature, are there useful frameworks that will enable the measurement of behaviour as a means of assessing the effectiveness of learning transfer? The next section reviews different competency frameworks with this intended purpose. Competency frameworks One framework used frequently in the UK was developed by Dulewicz and Herbert (1992), who empirically identified twelve independent dimensions of managerial performance – ‘supra-competencies’ – based on research of assessment centres (Dulewicz and Fletcher, 1982, Fletcher and Dulewicz, 1984) and developed the Job Competency Survey questionnaire (JCS). Dulewicz and Herbert (1992) correlated the responses on the importance and performance of each competency in the JCS and cross-validated with the Occupational Personality Questionnaire (OPQ). The twelve competencies, outlined in (Table 10) below, are grouped into four broad clusters:

Table 10. JCS competencies (Dulewicz and Herbert, 1992)

Intellectual • Strategic perspective • Analysis and judgement • Planning and organising Adaptability • Adaptability and resilience

Interpersonal • Managing staff • Persuasiveness • Assertiveness • Interpersonal sensitivity • Oral communication Results-orientation • Energy and initiative • Achievement motivation • Business sense

The JCS provides a useful instrument having been derived from Dulewicz’s assessment centre work and literature and already been used to test the responses of 46

hundreds of managers. However, some authors (Sarawano, 1993, Gay, 1995, Birchall et al., 1996, Chong, 1997) have sought to check its suitability in an international context. As this research is being carried out in South East Asia, the author reviewed research pertaining to competency frameworks and their use outside the UK and US to identify if other frameworks might be more appropriate. Is the JCS suitable in the location of the research? Gay (1995) reviews generic competency models from McBer (Spencer and Spencer, 1993), Dulewicz and Herbert (1992), Coulson-Thomas (1992), Barham and Oates (1991), and Barham and Wills (1992) – with the intention of seeking differences in the relative importance that practicing international managers might give to the ranking of competences which had been carried out with respect to national managers. Gay’s (1995) research design used the JCS model as a basis, supplemented with specific competences for international managers derived from his literature review including consideration of the influence of cultural differences on the conduct of international business. Building on the work of Hutton (1988), Hofstede (1980), Elashmawi and Harris (1993) and Adler (1986), Gay (1995) develops three additional categories to supplement the JCS including Global Awareness, Cross-border Cultural Awareness and Foreign Language Skills. However, the instrument would contain some fifty competences which Gay sought to reduce to a more workable twenty, considered by a chosen sample of international managers that they regarded as being the most relevant to successful performance in the international context. Chong’s working paper (1997) builds upon a follow-up study by Dulewicz and Herbert (1996) comparing the managerial competences and performance of UK managers with Singaporean public sector managers. He notes similarities in ten factors and distinct competences peculiar to each nationality. Sarawano’s (1993) research focused on the effect of cultural differences on managerial style and compared competences exhibited by UK managers (from the Dulewicz and Herbert 1992 study) and those exhibited by Malaysian managers. Sarawano used Chattel’s 16PF and a modified JCS to facilitate comparison with the UK data. He puts forward several hypotheses to predict differences expected between the two groups, largely based on Hofstede’s (1980) work on cultural differences. Each of these studies suggests that a generic managerial competency framework could be used to assess both expatriate managers and managers in their own country –

47

allowing for differences in culture stemming from national cultures, the organisations culture and the individual’s cultural heritage. Is the JCS up-to-date? It is more than twelve years since the JCS was created and the world of business has seen considerable change during that period of time, not least is the mid-90’s drive towards a Knowledge Based Economy across the world and, of particular interest to this research, in South East Asian countries. The Singapore Training and Development Association (STADA) in conjunction with the Nanyang Technological University (NTU) undertook a study in 2001 on ‘Managerial Competencies in a Knowledge Based Economy (KBE)’ (Lee Mei Ching et al., 2002). This study sought the views of Human Resource Development (HRD) professionals in Singapore on the implications of the KBE on human resource development. Table 11 notes the top ten competencies in a KBE resulting from this study:
Table 11. Top Ten competencies in a KBE - Singapore HRD perspective (Lee Mei Ching et al., 2002)

1. Adaptability to changes 2. Ability to see the ‘big picture’ 3. Communication skills 4. Visioning skills 5. Knowledge of own strengths and weaknesses 6. Creative thinking skills 7. Relationship building skills 8. Leadership skills 9. Consulting skills 10. Understanding of improvement in human performance

Kenworthy and Wong (2003) undertook a qualitative study with focus groups of business leaders, senior managers and HRD practitioners based in Singapore to understand what makes for an effective manager and to compare the findings with the frameworks above – this to help establish if the existing frameworks are still current and applicable in South East Asia. The study identifies those competencies considered to be base or threshold and those behaviours that differentiate effective managers. Mapping the constructs across shows close similarities, though there does appear to be a newer explicit emphasis on creativity and passion (Table 12):

48

Table 12. A study of the attributes of managerial effectiveness in Singapore - competency model (Kenworthy and Wong, 2003)
Descriptor Behaviours Base / Differe ntiator B B D B B B B D B B B B B B B B B B B D D B B B D D B B B B B B B B D D

Achievement / Results Oriented Accountability and Responsibility

Executive Maturity

Cultural Sensitivity

Building and Managing Relationships

Communication

Team Leadership

Achieves the business results (goals, sales targets, financial targets) Completes tasks or targets on time Sets own and others’ goals or targets Shows strong commitment to organisation Prepared to make tough decisions and see them through Help to grow the business and grow with the business (intellectually and behaviourally) Able to stand on own feet to make decisions Take accountability and responsibility for decisions made Positive thinker Eye for detail Resourceful Analytical Attuned to own potential and limitations Prepared to learn from mistakes and open to continuous self-learning Unafraid of criticism or feedback Aware of ‘face’ or ‘guan xi’ issues in dealing with people from different races Works around cultural barriers Adopts and adapts the way business is conducted Respects different mindsets Enjoys mixed cultural environments and working in a ‘melting-pot’ Works well with foreigners and locals Invests time in building both formal and informal relationships Good at identifying others’ strengths and weaknesses Earns trust Develops and uses skills in influencing, networking and customer satisfaction Getting along upwards, downwards and laterally Able to package and present message Actively listens to others Shows strong interpersonal skills and always ready to understand other’s view Recognises, understands and uses team dynamics, social structure, culture, strengths and weaknesses Works and relates well with people around and aligned with the company culture and builds consensus Fits in and wants to belong and contribute Open communication style Supports and knows where to gain support Teaches or coaches others Motivates others and inspires team members

49

Measuring managerial effectiveness? There is a strong focus with the JCS on the competencies being predictors of superior performance measured by means of advancement in the organisation or success. However, in this research, the author is interested in measuring learning transfer as a result of a training programme, and whilst the JCS is recognised as being a good indicator, it is more future-oriented than historical. Historically, there has been little agreement on what exactly it is that constitutes managerial effectiveness or excellence (Koch and Cebula, 1994). Robotham and Jubb (1996) debate whether it is possible to identify elements of effective performance from observing practicing managers and consider that this is because the precise nature of management is, to a degree, contingent on the context or environment in which it is being practiced. Thus, behaviours that are considered effective in one sector may be less appropriate in another. Traditionally, the views surrounding the issue of managerial effectiveness have tended to be largely based on the assumptions about what managers do, and what they should do to be successful (Robotham and Jubb, 1996). Such assumptions are challenged (Luthans et al., 1985) in that rather than relying on an evaluation of managers’ performance that is based on the activities traditionally prescribed for managerial success, a focus on the activities managers actually perform has emerged. A useful set of scaled competencies – competencies that have sets of behaviour ordered into levels of sophistication or complexity – was developed by Spencer and Spencer (1993). The competencies found to be most critical for effective managers are shown in (Table 13):

Table 13. Managerial competencies (Spencer and Spencer, 1993)

• • • • • • •

Achievement orientation Developing others Directiveness Impact and influence Interpersonal understanding Organisational awareness Team leadership

This set of characteristics, or individual competencies, that a manager brings to the job need to match well to the job or additional effort may be necessary to carry out the job 50

or the manager may not be able to use certain managerial styles effectively. These in turn are affected by the organisational climate and the actual requirements of the job. Managerial effectiveness is the combination of these four critical factors: organisational climate, managerial styles, job requirements and the individual competencies that a manager brings to the job. Reddin (1970) points out that managerial effectiveness is not a quality but a statement about the relationship between his behaviour and some task situation. If the Spencer and Spencer (1993) framework of competencies are able predict performance in any executive (Kelner, 2001) and these competencies are ‘trainable’ – then any training programme designed to develop managerial effectiveness in any role can be evaluated by means of assessing the changes in behaviour of the participant that demonstrates the competency. Spencer and Spencer’s (1993) framework was used to develop the Hay/McBer Managerial Competency Questionnaire (MCQ) (McBer, 1997) and (Table 14) below shows a mapping across each of the frameworks discussed in detail above.

Table 14. Comparing MCQ (Spencer and Spencer, 1993), JCS (Dulewicz and Herbert, 1992), Lee et al. (2002), Kenworthy and Wong, (2003)
MCQ Achievement Orientation Developing Others Directiveness Impact and Influence Interpersonal Understanding Organisational Awareness Assertiveness & Decisiveness Persuasiveness Interpersonal Sensitivity Adaptability & Resilience Strategic Perspective Communication skills Consulting skills Creative thinking skills Relationship building skills Ability to see the “big picture” Knowledge of own strengths and weaknesses Leadership skills Visioning skills JCS Achievement Motivation Stada/NTU Adaptability to changes Understanding of improvement in human performance Kenworthy & Wong Achievement /Results Oriented Proactive Team Leadership Accountability & Responsibility Independent Communication Adaptability & Flexibility Cultural Sensitivity Communication Executive Maturity

Team Leadership

Professionalism & Judgement

Team Leadership Passion

Mapping the competencies across the frameworks was done by means of using the competency descriptors in each case that describe the behaviour demonstrated in each 51

competency. The MCQ maps well onto the JCS and brings in an important element of Developing Others – shown in the two later studies. Since the competencies are scaled, it also provides a means of measuring change in behaviour readily, and as such provides a suitable basis to measure managerial effectiveness (competence in doing the job of management). Appendix 5 shows the MCQ Competency descriptors.

2.3.2 Learning transfer and competency frameworks summary
The effectiveness of training programmes is not solely contingent upon either the learning content or the quality of delivery methods (Naquin and Holton, 2003) and the transfer of learning to the workplace may be inhibited by the lack of an appropriate transfer climate (Huczynski and Lewis, 1980) and/or the motivation of participants to use the knowledge and skills gained during the training (Ruona et al., 2002). It makes intuitive sense that motivation to learn would be an important precondition of learning, and a motivation to transfer learning to the workplace would also be an in important precondition of effective transfer (Naquin and Holton, 2003), however the debate does not contribute to assessing behaviour change. To assess the effectiveness of a management development programme, it is necessary that we define effective management behaviour according to Mumford (1994) and Winterton and Winterton (1999) and the literature on management competencies provides useful frameworks to describe the behaviour and skills and application of appropriate knowledge that is associated with effective management. This part of the review has considered a number of commonly used competency frameworks and assessed their suitability both at the time of the research and the location in which the study is taking place concluding that the Hay/McBer MCQ framework, based on Spencer and Spencer (1993) and associated instrument, the MCQ (Hay/McBer, 1997), provides the author with a means of measuring competencies before and after the training event to demonstrate any change in behaviour. Using a generic competency model has the benefit of not being job or situation specific and thus reducing the most prevalent inhibitor of transfer of a learned skill to the workplace, that of no opportunity to do so (Awoniyi et al., 2002, Cromwell and Kolb, 2002, Lim and Johnson, 2002). The next section will review how individual competencies link to the performance of the organisation and Kirkpatrick’s highest level of evaluation – business results.

52

2.4 Business results – linking individual competency models to organisation outcomes
Some writers have identified competencies that are considered to be generic and overarching across all occupations. Reynolds and Snell (1988) identify ‘meta-qualities’ – creativity, mental agility and balanced learning skill – that they believe reinforces other qualities. Hall (1986) uses the term ‘meta-skills’ – as skills in acquiring other skills. Linstead (1991) and Nordhaug and Gronhaug (1994) use the term ‘meta-competencies’ to describe similar characteristics. The concept of meta-competence falls short of providing a holistic, workable model, but it does suggest that there are certain key competencies that overarch a whole range of others. There is however, some doubt about the practicability of breaking down the entity of management into its constituent behaviours (Burgoyne, 1989). This suggests that the practice of management is almost an activity that should be considered only from a holistic viewpoint. Baker et al. (1997) link the various types of competence by first establishing a hierarchy of congruence as a backbone to the model. In broad terms, they describe the congruence of an entity to be the degree of match or fit between some external driver to the entity and the response of that entity to the driver. This method enables them to take into consideration the idea that management, as an entity, and the individuals who perform the function do so within a particular environment. Measurement of congruence or goodness of fit, has been attempted in studies of operations (Cleveland et al., 1989, Vickery, 1991). Baker et al.’s hierarchy is shown in Figure 8 below, with four levels of congruence: 1) Organisation level, 2) Core business process level, 3) Sub-process within core process level, and 4) Individuals level.

53

Figure 8. Hierarchical model of competence (Baker et al., 1997)

At the organisation level, there is congruence when a firm adopts a strategy that is consistent with the competitive priorities derived from the firm’s business environment. The strategy, in turn, determines the operational priorities of the firm, following Platts and Gregory (1990), Baker et al. (1997) using their own terminology, consider these operational priorities to drive the core processes of the firm. These, in turn, can be broken down into a number of sub-processes – and congruence is needed between the subprocesses and the core processes. At the individual level, the skills and knowledge should also match the priorities driven by the sub-processes. This hierarchical model follows a traditional approach that structure follows strategy (Vickery, 1991, Cleveland et al., 1989, Kim and Arnold, 1992). Others view that competences are a part of the structure of the firm and should influence strategy making, Bhattacharaya and Gibbons (1996) point out that Prahalad and Hamal (1990) and Stalk et al. (1992) take this approach. The hierarchical model has been tested analysing case studies of seventeen manufacturing plants that won Best Factory Awards during the period 1993-95 in the UK (Cranfield) and established benchmarks. Baker et al. (1997) found some direct causeeffect links between enabling competences at the sub-process level and competitive 54

performance (at the core process level). However, they also found many ‘best practices’ such as employee empowerment and team working which were harder to link to specific competitive competences. This model provides an insightful way to break down the complex issue of how individual performance influences the competitive competences of the firm. Baker et al.’s research is limited within the manufacturing sector where core processes are often easier to identify and define with a clear delineation of individual effort, technology and product. It is also established on the basis that structure follows strategy – whereas, most firms will already have structure and will be adapting their strategies continuously as the external environment changes. Cheetham and Chivers (1996) describe a model of competence that draws together the apparently disparate views of competence - the ‘outcomes’ approach and the ‘reflective practitioner’ (Schon, 1983, Schon, 1987) approach. Their focus was to determine how professionals maintain and develop their professionalism. In drawing together their model, they consider the key influences of different approaches and writers. The core components of the model are: Knowledge/cognitive competence, Functional competence, Personal or behavioural competence and Values/ethical competence with overarching meta-competencies include communication, self-development, creativity, analysis and problem-solving. Reflection in and about action (Schon, 1983) surround the model, thereby bringing the outcomes and reflective practitioner approaches together in one model shown in Figure 9 below:

55

Figure 9. Model of professional competence (Cheetham and Chivers, 1996)

Cheetham and Chivers model of professional competence is useful in bringing the concept of individual competence to bear on the competence of the organisation in a nonmanufacturing context, but it still falls short of providing a useful model to link an individuals behaviour with the business results of an organisation across industries – a generic model if you will. Young (2002) does so neatly, by developing his individual model further to the organisational perspective adopting the concept of core competence, as articulated by Prahalad and Hamal (1990) and further developed by Stalk et al. (1992) and Tampoe (1994), suggesting that the collection of individual competences within the organisation create the organisational core competence (Figure 10):

56

INDIVIDUAL Personal Characteristics •Motive •Trait •Image/role •Knowledge Job Activities •Functions •Tasks

JOB

Behaviour •Skill

COMPETENCY

COMPETENCE

PERFORMANCE

Organisation Activities Organisation Level •Functions •Tasks

CORE COMPETENCE

PERFORMANCE

Figure 10. Individual variables of competency, competence and performance and organisation core competence (adapted from Young, 2002)

This model provides a way to understand how developing competency (personal characteristics and behaviours) at the individual level enables an individual to demonstrate competence (the functions and tasks of the job) which in turn cascades through a hierarchy of the organisation (core competence and other activities supporting the organisation) to deliver business results.

2.4.1 Business results summary
Hierarchical models of competence within an organisation show how an individuals competencies link to the competence and hence performance of the organisation. Young’s (2002) model provides a neat and simple clarity to the linkages and then this research focuses on how the hierarchy is managed and what are the competencies that provide a manager the capability to determine strategy, design core and sub-processes (for many the value chain or value system) and develop the individual competencies (skills and knowledge) to better deliver the enabling competences of the sub-processes. A manager who has this capability may be considered to be an effective manager. It is, for this reason that the training programme chosen for this research is about strategy development – providing all participants with the knowledge and opportunity to develop the skills in developing strategy. Neither the simulation nor the game was designed specifically to develop the managerial competencies under investigation. Strategy as a 57

subject was also suitable for this research because the simulation and game existed and were directly comparable with a case study based programme. Given the time-frame for business results to be noticeable and measurable, this research does not measure business results directly but will ask participants’ bosses to rate the performance impact of the participants enabling the author to consider the research question that rated performance impact will improve following the training.

58

2.5 Personality and personal background
This section of the literature review explores possible factors that are often considered as influencing how an individual learns and transfers such learning into behaviours that have a positive impact in the workplace. Aspects of an individual’s personality and the influence on learning and competencies has been briefly covered in the sections on management learning and managerial competency frameworks above, this section of the review will consider how we might anticipate how an individual’s background, such as their age, gender, previous academic attainment, position in the organisation and cultural heritage (Sternberg, 1997) may affect their preference for a particular learning environment and effect on their learning and learning transfer. The author also considers the affect of groups of individuals working together as teams within the learning environment to foster teamwork and leadership in the workplace. First, the author examines in some detail the literature on personality, and in particular, the two most frequently considered aspects of personality, learning style and personality type. Young’s (2002) model linking competency, competence and performance considers personal characteristics to be a fundamental part of competency, this follows other researchers notably Boyatzis (1982) and McClelland (1973). The purpose of this section of the literature review is to determine what factors may need to be considered when evaluating managers’ behaviour and what influences might we expect about an individual’s personality and background on their behaviour and if personality measures should be included because they may affect performance. Boerlijst and Meijboom (1989) argue that human behaviour is continually subject to changes in human attributes and attributions. They discuss the differences between long and short term locus of concern suggesting that people adapt a task and identity for self in the long term, whilst they will concentrate on performance of a task and fulfilment of self needs in the short term. Evidence suggests that individual personality factors have a great effect on performance (Byrne and Wolfe, 1974, Brenenstuhl and Catalanello, 1977, Henke, 2001). When attempting to evaluate the effectiveness of a training intervention, we should consider the potential for an individual’s personality to affect outcomes. In management development there is a strong influence from humanistic psychology, and from this a growing practice of student-centred learning (Burgoyne and Reynolds, 1997b). In particular, the influence of experiential learning theory in management learning and how individuals learn, or their learning style, is a central 59

consideration in the design and potential success of learning interventions in achieving the desired outcomes.

2.5.1 Learning styles
As suggested by Sadler-Smith (1996), in addition to a consideration of factors such as organisational and environmental context, the characteristics of the learner and in particular, their learning styles, should be taken into account when designing, developing and facilitating learning experiences. According to Loo (2002), learning style is intimately related to cognitive style, the learner’s personality, temperament and motivations, and refers to the customary approach in which a learner perceives and processes information initially proposed by Kolb (1976). A close examination of Kolb’s (1976) four stage model of learning (Table 3 on page 22 above) reveals that learning requires abilities that are polar opposites or dimensions (action and reflection, concreteness and abstraction) and that the learner must continuously choose which set of learning abilities to bring to bear in any specific learning situation. Kolb (1976) argues that most people develop a learning style in which some of the learning abilities are emphasised over others and proposes that the combination of dominant abilities along the two dimensions defines an individual’s learning style. Table 15 below, shows Kolb’s four different learning styles highlighting strengths of the style and a common example.

Table 15. Kolb's four different learning styles (Johnson and Stratton, 1978)
Learning Style Converging Diverging Assimilating Accommodating Combination of abilities AC+AE CE+RO AC+RO CE+AE Characteristics strengths of style example Practical application of ideas – engineers Imaginative ability – managers with humanities or liberal arts background Creation of theoretical models – researcher Doing things – marketing or sales

It is argued that there may be no generalisable learning process or that an individual is particularly inclined to a preferred learning style (Symons, 1996) and Kolb’s learning styles are not without critiques. Beard and Wilson (2001) report that Kolb’s theory is extremely influential and rarely seen as problematic in the field of management education. Kolb’s theory overlooks or mechanically explains the social, historical and cultural aspects of self, thinking and action located as it is in the cognitive psychology 60

tradition (Reynolds, 1997a). Holman et al. (1997) suggest that the idea that a manager reflects like a scientist in isolation of events does not take sufficient account of the social interactions of a person that are important in the development of the self, thought and learning. They also question the sequential progress through the cycle – suggesting that it is rather, practical argumentation with oneself and others that forms the basis of learning. Reynolds (1997a) presented a critique of learning styles and the readiness of the HRD profession to use instruments such as Kolb’s Learning Style Inventory (LSI) (1984) and Learning Styles Questionnaire (LSQ) (Honey and Mumford, 1982). Although recognising the intuitive appeal of such instruments, Reynolds questions the validity of the theoretical and empirical evidence. Sadler-Smith (2001) in a reply to Reynold’s critique suggests that whilst the labelling for learning styles is justified, extending the argument to cognitive styles is not, and would close off a promising avenue of research and may deny some individuals access to greater self-awareness and, potentially, development. Honey and Mumford (1992) criticise Kolb’s LSI as not being meaningful for managers with whom they dealt. They developed instead the Learning Styles Questionnaire (LSQ) yielding four separate scores, purporting to measure a learning style equivalent to each of the four stages in Kolb’s learning cycle – noting that Kolb’s LSI being based on research with students rather than managers. Honey and Mumford's LSQ exhibits better validity and reliability than the LSI, though Duff (2000) finds that in reviewing studies on the reliability of the LSQ that it may not be a suitable alternative for education researchers. Learning from experience is often considered to produce tacit knowledge that affects performance in real-world settings (Nonaka and Takeuchi, 1995, Argyris, 1999). For example, in a study to explore the acquisition of tacit knowledge being influenced by managers’ learning styles, Armstrong and Mahmud (2004) found that Kolb’s Converging learners had significantly higher level of measured tacit knowledge than Divergers. Such studies suggest that an individual’s learning style may influence how well and how much they learn from a particular style of management development intervention and, following, how much and how well they can transfer to the workplace in the form of changed behaviour.

2.5.2 Factors that shape and influence learning
The importance of individual differences in ability to learn and transfer learning has long been a theme in educational psychology with a growing emphasis on how learners seek to learn and transform information to create and construct knowledge in their 61

own mind (Pintrich et al., 1986). Yet the literature on training effectiveness has paid little attention to the potentially dynamic interaction of individual differences with different segments of training and the ultimate success in terms of learning or transfer (Herold et al., 2002). Those factors of individual differences most frequently associated with experiential learning theory are discussed by Kolb et al. (2000) noting the factors that shape and influence learning styles noting five particular levels of behaviour: personality types, early education, professional career, job role and adaptive competencies. Sternberg (1997) also argues that national culture may be one of several variables (others including gender, age, schooling and occupation) that are likely to affect the development of an individual’s thinking styles. Personality types and motivation In particular, Kolb et al. (2000) link Myers Briggs Type Indicator (MBTI) with learning style, suggesting that the Accommodating learning style is Extroverted Sensing type, and the Converging style the Extroverted Thinking type. Figure 11 maps the MBTI types onto the LSI dimensions.

Feeling Concrete Experience Experience

Active Experimentation Extroversion

Testing implications of concepts in new situations Formation of abstract concepts & generalisations Abstract Conspetualisation Thinking

Observation Reflective & Reflection Observation Introversion

Figure 11. Combining LSI and MBTI dimensions (Kolb et al., 2000)

The apparently close linkage to experiential learning theory and learning styles means that the Myers-Briggs personality type has been frequently used in effectiveness studies. For example, Patz (1990, 1992) undertook a study to assess the influence of MBTI dominant personality on the performance of students in Total Enterprise simulations, 62

finding that Intuitive (N) and Thinking (T) dominance among team members was positively related to performance. However, Patz made no attempt to assess the complicating influence of group dynamics on his results. Anderson and Lawton (1993) eliminated this complication by having each individual operate his or her own Total Enterprise simulation company. They also related simulation performance to the student’s relative MBTI preferences rather than dominant personality type and found no significant differences between personality types at any stage of the simulation’s operation and the direction and differences found were contrary to Patz’s findings. Gosenpud and Washbush (1992) more directly tested Patz’s hypothesis concerning dominant personality types and found inconclusive evidence. In addition to the potential for an individual’s personality type to be an important influence on preferred learning style, personal characteristics may influence their motivation to learn and their actual performance (Naquin and Holton, 2003). Intuitively, motivation to learn seems to be an important precondition of learning, and the educational psychology literature discusses many different concepts of what motivation is and where it stems from. Well known conceptions of motivation include a broad-range of ideas from single-motive conceptions such as Freud’s notion of libido (Gage and Berliner, 1998) to hierarchical conceptions such as Maslow’s hierarchy of needs (1954). Despite many studies in education, motivation to learn as a variable in training has been largely neglected in training-related research (Clark et al., 1993) and fewer studies have examined motivation to transfer learning (Holton et al., 2000). There is a proliferation of conceptual frameworks, but with little agreement on practical issues of how personality and learning style are assessed, and few models have sound conceptual or empirical base (Sadler-Smith, 2001). There does appear to be agreement that an individual’s personality and their motivation are considered to be important factors that affect learning and transfer with most emphasis within the experiential learning and trait modification approaches towards personality type and learning style and the implicit suggestion that aligning the design of learning with an individual’s preference for learning style will be both more motivational and lead to greater performance. Early education and age According to Kolb et al. (2000), early educational experiences shape an individual’s learning styles through encouraging specific sets of learning skills. They go on to suggest links between particular undergraduate majors with preferred learning styles. 63

Of particular interest in this research study is the individual’s age at the time of the training event. Whilst simulations have been around for over 40 years, few people had the opportunity in their education to experience them. The power and flexibility of modern computer technology and its increasing accessibility and affordability means that computers and the Internet now form a significant part of the education curriculum – either learning about or with the assistance of technology. The simulation and management game are examples of comparatively new ways of experiential learning using modern computer technology and both Prensky (2000) and Aldrich (2002) suggest that younger individuals will prefer such learning methods to older individuals. According to Hannafin et al. (1996), the computer represents a unique learner-centred opportunity; learners are able to control a wide variety of instructional variables including the subject matter content, the context of instructional situation, amount of practice undertaken and the amount of advice to be provided during the instruction. In effect, computer based learning systems can encourage learners to build relationships among learning concepts through exploration, experimentation and manipulation much more easily than in a traditional classroom environment. This suggests that younger learners may be more accepting of and comfortable with computer based simulations or games than older learners, in part because computer technology has been more prolific in education in recent years. Academic attainment, gender and organisation position differences The influence of prior educational experience in developing a greater acceptance of technology supported learning has been discussed above, but the influence of other individual differences on an individual’s willingness to embrace learning technology is often overlooked (Hoskins and Van Hooff, 2005). Gender and age are frequently considered factors – in part possible because the data is easy to obtain. Duff (2004) cites many examples of previous research indicating that age, gender (e.g. Hayes and Richardson, 1995) and prior academic achievement (e.g. Eskew and Faley, 1988) have direct effects on individuals approaches to learning and their performance and progression. An individual with a higher level of academic attainment may demonstrate a greater ability and understanding of the concepts than an individual with a lower academic attainment, and is a strong predictor of performance and progression (Duff, 2004). Studies of the use of learning technology and gender have shown mixed results, earlier studies suggest that males are more comfortable with and active in the use of learning technology (Chmielewski, 1998, Sussman and Tyson, 2000), while more recent 64

studies suggest that this is not the case, and that there is essentially no difference between males and females (Hoskins and Van Hooff, 2005). This may be the simple effect of time and increasing ease of accessibility and suggests that an individual’s gender may affect their preference for using learning technology at an individual level as a result of their personal experience, but overall it is unlikely to be a significant factor. An individual’s position or grade within an organisation has long been associated with the demonstration of managerial competencies (Dulewicz and Fletcher, 1982) and it is reasonable to assume that an individual who is more senior in the same organisation, in a meritocracy, will demonstrate higher levels of managerial competencies than an individual in a lower position. It makes intuitive sense that an individual with a higher level of competency before a training intervention has less opportunity to improve, however there are some suggestions (Anton, 1992) that seniority in a learning-by-doing environment encourages the greater adaptive competencies that Kolb et al. (2000) suggest are the most immediate pressure that shapes learning. Thus, through greater experience, a more senior manager adapts the skills required to effectively complete a particular task congruent between personal skills and the demands of the task (Kolb, 1984, Ridley et al., 1995, 1996). Bertsche et al. (1980, 1991) suggest that computer-based simulations are important tools to support learning and an ideal way of leveraging the experience of senior managers and this may be explained by the adaptive competencies brought to bear on the learning situation. Cultural heritage Several authors argue that national culture may be related to differences in cognitive style and that such differences may be important in a management learning situation, especially in an increasingly internationalised environment (Abramson et al., 1996, Allinson and Hayes, 2000). Hofstede’s (1980) cultural dimensions are often used to explain differences in outcomes and behaviour from training (Hwang, 1987, De Vos, 1985). The four dimensions Hofstede operationalised, based upon research among IBM employees in various countries throughout the world, were power distance, collectivismindividualism, uncertainty avoidance and masculinity-femininity. Power distance refers to the extent to which members of a national culture group are willing to accept the uneven distribution of power. Collectivism is the tendency for people to belong to groups and look after each other. Individualism is the predisposition to look after oneself and immediate family only. Uncertainty avoidance is the extent to which people feel threatened in ambiguous situations. Masculinity-femininity refers to the distribution of roles between 65

genders. Masculinity is associated with competitiveness and assertiveness, while femininity is associated with modesty and a caring disposition. We might anticipate that national culture would affect an individual’s thinking style though Kirton (1994) reports a study by Thompson (1980) showing little difference in score using the Kirton Adaptive-Innovator between English speaking managers in Singapore and Malaysia and their UK counterparts. However, in a comparative study of cognitive styles, Savvas et al. (2001) found differences between cultural groups and found some cross-cultural differences at post-graduate level between UK and Hong Kong groups. It may be, that at the time of Thompson’s research, both Malaysia and Singapore had relatively recently gained independence from Britain and the dominant educational system remained English, while Hong Kong, although under British rule for longer, is physically and economically much closer to Asia and China in particular. Potentially, these studies suggest that national cultural differences may shape individual’s learning style and preference but such influence may not clearly be defined by something as simple as race or citizenship, but the influence of the environment at the time. In this research, we will consider the possible influence that a participant’s cultural heritage has on their reaction, learning and transfer. In particular, the concept of “face” is well established in differences of ways of doing business (Ho, 1976) and saving face by making judgements in private rather than public (Keys and Wolfe, 1990, p316) often makes for quieter training rooms in the east than those in the west. Computer-based simulations and games may provide a useful medium to offset this proclivity attributed to Asians (Hofstede, 1980), allowing participants to make their decisions within the privacy of the human-computer interface. The literature review on factors that shape learning has highlighted potential influences that may be considered through the comparative evaluation of training methodologies as is the intention of this research study. Another important factor to consider in relation to the way people learn and transfer that learning is the effect of team working and group dynamics. Teams, group dynamics and performance Management involves group problem solving and leveraging cross-functional expertise, and simulations provide management teams with enhanced means of learning from experience and from each other, as the “group must work together for optimal learning to occur” (Keys and Wolfe, 1990, p316).

66

A great deal of research in group dynamics and more recently, teams, suggests a range of potential benefits arising from team working in an organisational context. Higgs (1999) notes the benefits highlighted in the literature as: • • • Performance: suggesting that teams are more productive than individuals or competing groups, Satisfaction: that team members are more satisfied when working in a team, and Pooling information: that teams pool information and discuss to improve the quality of decisions. Belbin’s seminal work (Belbin et al., 1976, Belbin, 1981) on effective teams shows that the impact of the mix of differing ‘types’ within a team has a significant impact on team performance. Based on a long term study, Belbin presented evidence of the performance of different groups of managerial teams on a computerised management game included in the Henley General Management course. This work was developed in an experimental setting yet its findings have been widely upheld within organisations and supported by a range of follow-up studies demonstrating the organisational applicability of the findings. Margerison and McCann (1985) in similarly structured research have demonstrated similar results of comparative performance on differing mixes of team roles. In a study of some 100 managers, Dulewicz (1995) found evidence of clear linkages between Belbin’s team roles and the supra-competencies developed by Dulewicz and Fletcher (1982, Fletcher and Dulewicz, 1984). These competencies were also linked to the 16PF and OPQ instruments and both were shown to be related to the Belbin Team Roles. In developing competencies of the team and team members, Bal (1995) argues that effective development depends on the team members developing trust and collaboration, without which the team working environment is less conducive to the exchange of ideas and learning. Furthermore (p147) that teams working in close proximity engender a feeling of belonging encouraging freer debate and discussion. Such positive feelings are likely to be seen in an individual’s reaction to team working within the training programme and reflect greater performance both in learning and learning transfer – particularly those competencies closely associated with working with others and teams.

2.5.3 Personality and personal background summary
Acquiring competencies and knowledge and transferring them to the workplace is a complex process with many potential factors affecting each individual’s learning from a 67

particular training and application of the learning afterwards. The literature on personality and personal background provides some insight into the multitude of variables that can be considered in management learning. The question then is which factors are most relevant or considered to be the most influential with regard to improving effectiveness and then, in relation to the intended study, which of those factors can to an extent be controlled and/or measured. A number of studies, particularly those in the field of simulation or game based learning, consider that an individual’s learning style and/or their personality type are of great influence in the enjoyment and, usually perceived, usefulness for learning. The review has considered the literature in regard to learning style and personality type discussing the strong support from those closely associated with experiential learning theory, and those who critique the simplicity and, in particular, the measurement instruments commonly used (Mainemelis et al., 2002). The evidence from studies of performance in simulations suggests that individual personality factors, as measured by the MBTI instrument, are less reliable measures for predicting performance (Schneier and Beatty, 1977, Gosenpud and Washbush, 1992, Anderson and Lawton, 1993) and that group or team dynamics have a greater affect on performance in a simulation (Kickul, 2001, Dulewicz and Herbert, 1992, Dulewicz, 1995, Belbin, 1981). This suggests that when attempting to evaluate the effectiveness of a training intervention to develop managerial competencies, we should consider the potential of personality and team dynamics to affect the outcomes, particularly when measuring changes in behaviour. Motivation to learn and motivation to transfer of individuals is considered by many to be of particular importance and this may be considered to be closely associated with an individual’s personality but also is concerned with the learning environment and methods employed (Russ-Eft, 2002) as well as the potential effect of working in groups or teams (Higgs, 1999). There does not seem to be a clear definition of what motivation is that works in a generalisable way, though attempts to measure the transfer climate show potential for predicting the effectiveness of training (Holton et al., 2000, Holton et al., 2001). The section has also reviewed other factors considered to shape learning and influence the effectiveness of learning and transfer including some of the key factors identified by Sternberg (1997), gender, age, schooling and occupation, as well as the possible influence of national culture. In respect of the latter, the review has considered Hofstede’s (1980, 1991) work in particular in this area.

68

The final section of the literature review considers previous research in the field of study, drawing together the work of other researchers and noting the calls for further research and the limitations of studies previously undertaken in this field of study.

2.6 Previous research in simulations and games
Although for more than 40 years, researchers have lauded the benefits of simulations (Keys and Wolfe, 1990), very few of these claims are supported with substantial research regarding the learning benefits of the technique (Brenenstuhl and Catalanello, 1977, Keys and Wolfe, 1990, Hannafin et al., 1996, Gopinath and Sawyer, 1999). A large amount of the business gaming literature has dealt with how its method fared against traditional methods for delivering course material (Keys and Wolfe, 1990). For example, the studies by Kaufman (1976), McKenney (1962, 1963), Raia (1966), Wolfe and Guth (1975), and Kenworthy and Wong (2005) found superior results for game-based groups versus case groups either in course grades, performance on concepts, examinations, or goal-setting exercises. Yet a review of 68 studies that compared the instructional effectiveness of simulations with other instructional methods by Randel et al. (1992) revealed that a majority (56%) of them found no difference between simulations and other pedagogical methods, 32% found that simulations led to better student performance, and only 5% favoured traditional instruction. There are also a number of empirical studies that have examined the effects of game-based instructional programmes on learning. For example, both Whitehall and McDonald (1993) and Ricci et al. (1996), found that instruction incorporating game features led to improved learning. In a recent review, Druckman (1995) concludes that games seem to be effective in enhancing motivation and increasing student interest in the subject matter, yet the extent to which this translates into more effective learning is less clear. Business gaming’s largest set of literature has dealt with factors that concern the nature of the simulation itself, the particular aptitudes and abilities of those playing the business game, and the game administration by the instructor (Keys and Wolfe, 1990). Certo (1976), Keys (1977), and McKenney (1967) highlight the importance of instructor guidance during crucial stages during the use of a simulation or game and the debriefing stage to insure some degree of closure and summary insights gained from the experience. Garris et al. (2002) agree and found that the role of the instructor in debriefing learners is a critical component in the use of instructional games, as are other learner support strategies.

69

Education literature has further suggested that the use of simulation has different effects on different students due to variations in individual methods of processing information and subsequent learning (Snyder and Vaughan, 1996). As highlighted by Feinstein et al. (2002), computer simulation is not an educational panacea. Simulation design must be rooted in research on teaching and learning tools, and resources provided must support the learner’s efforts (Salomon, 1993).

2.6.1 Support for simulations and games
Hoberman and Mallick (1992) and Geber (1994) suggest an impressive number of benefits of training using simulations including: 1) Improved transfer of learning to the work venue; 2) Well-suited for teaching participants how to respond to change; 3) Relatively risk-free environment in which to try new behaviours; 4) Higher participant involvement and motivation; 5) Ability to manipulate several variables at once; and 6) Potential for immediate feedback. Researchers have identified benefits that are unique to simulation techniques: improved ability to teach teamwork (Keys et al., 1994); unique contribution to long-term strategy making (Gopinath and Sawyer, 1999); demonstrate the complexities of dynamic business systems (Romme, 2003); and positive relationship between business game experiences and outcomes such as income and organisational position (Wolfe and Roberts, 1993). It has also been proposed that simulations seem well suited to promoting what Argyris and Schon (1978) have termed ‘double-loop learning’, a learning climate that supports valid information, free and informed choice, and internal commitment: and strategic knowledge that requires applying learned principles to different contexts or deriving new principles from general or novel situations (Garris et al., 2002). Simulations have become widely accepted pedagogical techniques, in part because participants are more actively involved in the learning process and receive immediate feedback on the results of their actions (Brenenstuhl and Catalanello, 1979). This supports Senge’s (1990) view that human beings learn best through first-hand experience, particularly when feedback from actions is rapid and unambiguous. Garris et al. (2002) support this view and propose that the game cycle is iterative, in that game play involves repeated judgement-behaviour-feedback loops and user reactions to this lead to greater persistence or intensity of effort because it is enjoyable, interesting and builds confidence. Support for simulations in greater learning in and of itself is one aspect, but Swanson and Holton (1999) also argue that learning activities that recreate work situations, such as simulation enhanced learning activities, foster better transfer of learning. 70

Henke (2001) emphasised that a central characteristic of a game is that they are fun and a source of enjoyment. Schank (1997) and Prensky (2000) agree and stress that the best way to break through resistance and apathy to learning is with an opening that is immediately involving and fun.

2.6.2 Criticisms of simulation and game research
In spite of the extensive literature, many of the claims and counterclaims for the teaching power of business games and simulations rest on anecdotal evidence or inadequate or poorly implemented research. These research defects, according to Keys and Wolfe (1990), have clouded the business gaming literature and hampered the creation of a cumulative stream of research. Most studies have been conducted using college students as subjects for the experiments (Chang, 2003, Cook, 1999, Kim et al., 2002), yet Babb et al. (1966) stressed nearly 40 years ago, that there are striking differences in behaviour and game results between students and experienced managers. They concluded that these differences in behaviour raise serious questions as to the extent that results from student behaviour in management games can be generalised to behaviour patterns in the business world. The inability to make supportable claims about the efficacy of simulations can be traced to poorly designed studies, the lack of a generally accepted research taxonomy, and no well-defined constructs with which to assess learning outcomes (Feinstein and Cannon, 2001, Gosenpud, 1990). Sales and Cannon-Bowers (2001) highlight the somewhat misleading conclusion that simulation (in and of itself) leads to learning; unfortunately most of the evaluations rely on trainee reaction data and not on performance or learning data. There are also such a variety of stimuli (e.g. teacher attitudes, student values, teacher-student relationship) in the complex environment of a game that it is difficult to determine the exact stimuli to which learners are responding (Keys, 1977). Gosen and Washbush (2004) point out that although it seems appropriate to undertake research assessing the value of simulations, the majority of early studies have focused on performance in a simulation. However, research on the relationship between learning and performance suggests that the two variables do not co vary, performance is not a proxy for learning, and it is inappropriate to assess simulations using performance as a measure of learning (Washbush and Gosen, 2001, Wellington and Faria, 1992). There is evidence to suggest that simulations are effective, but the studies showing these results do not meet the highest standards of research design and measurement, and any conclusion about them must be tentative (Gosen and Washbush, 2004). 71

Gredler (1996) notes, as did Pierfy (1977) that a major design weakness of most studies evaluating simulation based training is that they are compared to regular classroom instruction. However, the instructional goals for each can differ. Similarly, many studies show measurement problems in the nature of the post-tests used. Rose (1995) and Whitelock et al. (1996) are amongst many (McKenna, 1996, Clark and Craig, 1992, Reeves, 1993) criticising the evaluation of learning effectiveness using interactive technologies (games, multimedia, simulations and virtual reality) and ask for more research to quantitatively evaluate the real benefits of these forms of learning and teaching. Since the early days of simulation and gaming as a method to teach, there have been calls for hard evidence that support the teaching effectiveness of simulations (Hays and Singer, 1989). Feinstein and Cannon (2002) suggest that in spite of the extensive literature, it remains difficult to support even the most fundamental claims for the efficacy of games as a teaching pedagogy. There is little hard evidence that simulations produce superior learning to other methodologies. They go on to review the reasons as being traceable to the selection of dependent variables and the lack of rigour with which investigations have been conducted.

2.6.3 Evaluation of simulations
The criticisms above show that it is important to develop a coherent framework for pursuing the evaluation problem. There are three prominent constructs appearing in the literature: fidelity, verification and validation. Validation is further split into two, validation that the simulation programme as a method of teaching and that the simulation programme as a training intervention, produces (or helps to produce) learning and transfer of learning (Hays and Singer, 1989). Yet fidelity and verification are easier to evaluate (but not necessarily to measure objectively) and often distract evaluators from the more tricky issue of validation. Simulation fidelity Fidelity is the level of realism that a simulation presents to a learner. This is how similar the training situation is, relative to the operational situation, in order to train most efficiently (Hays and Singer, 1989). Fidelity focuses on the equipment that is used to simulate a particular learning environment. In sophisticated (technologically) simulations that use virtual reality, for example, the construct of fidelity has an additional dimension, that of presence. Presence is more than the suspension of disbelief (Bailey and Witmer, 1994, Witmer and Singer, 1994, 72

Dede, 1997, Salzman et al., 1999) that users ignore ‘unreality’ but the degree to which an individual believes that they are immersed within the virtual world of the simulation (Witmer and Singer, 1994). The degree of fidelity or presence in a learning environment is a difficult element to measure (Feinstein and Cannon, 2001, 2002), but much research in the 1960’s and 70’s studied the relationship between fidelity and its effects on training and education. These studies, according to Feinstein and Cannon (2002) found that a higher level of fidelity does not translate into more effective training or enhanced learning (see Salzman et al., 1999, Stanney et al., 1998, Gagne, 1984, Alessi, 1988). In fact, it may be that lower levels of fidelity but with effective Human Computer Interface (HCI) or navigational simplicity yet with a significant degree of presence, can assist trainees in acquiring knowledge or skills within the simulation (Pegden et al., 1995). Simulation verification Verification is the process of assessing that a model is operating as intended. During the process, simulation developers test and debug the software through a series of tests under different conditions, verifying that the model works under such conditions. Often, developers are distracted by this process producing what appears to be brilliant models that work ‘correctly’ but with no appreciation of the educational effects and hence their validity. It is a crucial step but can be a trap (Feinstein and Cannon, 2002). Simulation validation Building on the work of Feinstein and Cannon (2002) and others (Cannon and Burns, 1999, Teach and Giovahi, 1988, Teach, 1989, Miles and Randolph, 1985, Anderson, 1982, Anderson and Lawton, 1997a), there are three main questions beyond those of the general issues of evaluation noted above: 1. Are we measuring the “right thing”? - Validity in the constructs 2. Does the simulation provide the opportunity to learn the ‘right thing’? – Verification of the simulation and the appropriate fidelity for the content and audience 3. Does the method of using a simulation deliver the learning? – Validity as a method of developing. Figure 12 shows these three faces of simulation validation:

73

Figure 12. Three faces of simulation evaluation (adapted from Anderson, Cannon, Malik, and Thayikulwat, 1998)

2.6.4 Evaluation of simulations for learning outcomes
An analysis of simulation literature identifies learning outcome instructors adopt as they strive to educate business students. These learning outcomes have been advanced as targeting the skills and knowledge needed by practicing managers (Gosenpud, 1990). Simulation researchers have speculated that the method is an effective pedagogy for achieving many of these outcomes (Table 16). It is clear from the table below that the vast majority of evaluations have relied on the learner’s perceptions of their learning outcomes. Objective measurements (for any learning intervention) are more difficult, however, there appears to be a need to bring in more objective measures to help understand if simulations are an effective method for people to learn business management skills and to transfer that learning to the workplace.

74

Table 16. Possible learning outcomes for simulations (adapted from Anderson and Lawton, 1997)
Facts and concepts of the business discipline Increase the student’s knowledge of basic principles and concepts of the discipline Interpersonal skills Improve the students ability to… Participate effectively in group problem solving Motivate coworkers Provide meaningful feedback to coworkers Resolve conflicts Communicate clearly with coworkers Develop people Lead Form coalitions Develop consensus Delegate responsibility Supervise Manage People Work as a member of a team Work in a group environment Appraise performance Increase the student’s knowledge of human behaviour in a group setting General analytical, critical thinking, problem-solving, or decision-making skills Improve the student’s ability to… Identify problems Frame problems Structure unstructured problems Analyse problems Use data to make better decisions Distinguish relevant from irrelevant data Interpret data Implement ideas and plans Make decisions using incomplete information Solve problems Solve problems creatively Solve problems systematically Make good decisions The interrelationships among things Improve the student’s ability to… Integrate material from vaiour functional areas of business See the ‘big picture’ Increase the student’s understanding of the complex interrelationships in a business organisation Increase the student’s understanding of why organisational subsystems must be integrated for organisational effectiveness Business specific knowledge and skills Improve the student’s ability to… Assess the situation quickly Plan effectively Plan business operations Schedule and coordinate Prioritize tasks Forecast Use spreadsheet for decision-making Increase the student’s understanding of… The decision process

P P P P P P P P P P P P P P P P

P P P P P P P P P P P P P

O

O

O

P P P P

P P P P P O P P

75

2.6.5 Previous research summary
The review on previous research in simulations helps clearly position this research such that it is concerned with validating (Feinstein and Cannon, 2002) the use of games and simulations as a method of delivery and we might expect, following Wolfe and Guth (1975) and others that both games and simulations would show greater learning than using case studies. We might also anticipate, following Hoberman and Mallick (1992) and Geber (1994), that games and simulations would show a greater transfer of learning into the workplace. However, most previous studies have been reliant on perceived benefits, and the literature frequently calls for [more] empirical research into the effectiveness of games and simulations. The concept of fidelity and associated sense of presence is an important aspect to consider in using simulations or games in training. The evidence from studies suggests differences regarding the enjoyment and perception of usefulness of the experiential activity within the programmes. Fidelity is thought to be an important factor in user enjoyment but not necessarily that greater fidelity causes greater enjoyment or learning (Feinstein and Cannon, 2001), nor is it the converse, it seems that the level of fidelity or realism is a fine balance that creates sufficient presence for the user (Salzman et al., 1999, Stanney et al., 1998).

2.7 Literature review summary
The need for research in this broad field is driven by the substantial spending by organisations, and by academic institutions, worldwide on technology that supports management development. The interest and use of computer-based simulations and games is growing significantly as a means of accelerating the learning process and the belief that such methods are more enjoyable for users, generate greater learning and hence are more effective than more traditional methods of training and teaching. Yet, as we have seen, there is little empirical evidence to support the notion. The management learning literature section has discussed the emphasis on what people learn and how such learning is transferred to the workplace and the different schools of management learning that help us understand the influences of different theorists and practice and what that means we, as trainers and educators, assume about the way people learn and behave. An important aspect of management learning is how we evaluate the effectiveness of development and the various schools of thought consider 76

what can, and should be measured. In the realm of business training, Kirkpatrick’s four levels of evaluation remain the most well known and used, in spite of the possible shortcomings of the model, its very simplicity has probably lent its continued ubiquity. Learning itself is a complex and widely debated construct and as researchers seek to transform it into an observable phenomenon there is an equally if not greater complexity of factors that may or may not affect the way an individual learns and applies or transfers learning to another situation. The educational and cognitive psychology literature show many competing and often contradictory theories to explain the process of learning and how it is affected by myriad influences at any given time though there is a seeming consensus that some factors have a greater effect on learning and transfer, notably learning style (or cognitive style), motivation and some key defining elements in a persons background, such as their national culture, gender, age and schooling, and their adaptive competencies. Some factors are easy to identify, whilst others are complex in their own right and may not be observable, however, as human beings we experience them ourselves even if we are unable to explain them. Finally, the literature review considered previous research in this field of study, what other researchers had found and the particular complexity of evaluation when some elements of computer-based simulations and games are difficult to precisely identify – fidelity in particular is believed to be an important factor in learning from simulations, yet it is a construct we have difficulty in observing. Computer-based simulations and games are regarded by many as an important tool in the educators and trainers armoury of methods. For some organisations, they represent a critical and only alternative to on-the-job learning (such as airline flight simulators) whilst, for others, they represent a new and more effective way to develop knowledge and skills. The former is a belief that anyone wishing to board an aircraft is happy to accept unequivocally, the latter has yet to be demonstrated. The purpose of this research is to at least, make a start on that by answering the research questions and hypothesis in the following chapter.

77

Chapter 3

Research Questions and Hypothesis

The author’s research interest is to establish if computer-based simulations and games used in management development programmes are as effective as more traditional, case-study, methods. After Schuman et al. (2001) to overcome some of the issues surrounding evaluation, particularly experiential learning events such as simulation-based training, assessing effectiveness utilising a holistic framework approach using Kirkpatrick’s 4 levels provides a means of making a useful generalised assessment. In particular, this research investigates learning and learning transfer – Kirkpatrick’s second and third levels of evaluation. The literature on Management Learning and Evaluation and Competencies helps develop the first and overriding research question: Do the development interventions show a positive change in learning and management behaviour? The overriding question of this research is: RQ1. Are computer-based business training simulations and games an effective way to develop management learning and learning transfer? The hypotheses underlying this question require that the training in all groups shows that participants have learned something and demonstrate a positive behaviour change representing the transfer of learning to the workplace. Continuing to use Kirkpatrick’s four levels of evaluation and the evidence from previous research (Wolfe and Guth, 1975, Wolfe, 1985, Swanson and Holton, 1999), this is broken down into four hypotheses: H1.1. H1.2. H1.3. H1.4. The simulation and game groups will show higher ratings in participant The simulation and game groups will show greater learning than the case The simulation and game groups will show greater change of demonstrated The simulation and game groups will show higher bosses rated reaction than the case study groups study groups managerial competency (learning transfer) than the case study groups. performance change between groups. We also anticipate that there will be a degree of positive correlation between each of the four levels of evaluation following the implication in Kirkpatrick (1959/60, 1974): H1.5. H1.6. H1.7. Participants reaction will correlate with learning Participant learning will correlate with change in managerial competency Participant change in managerial competency will correlate with bosses 78

rating of performance impact.

This research includes participant assessment of both the learning style and personality type for the purposes of comparison and to evaluate whether learning style preference reflects in the enjoyment, perceived usefulness, learning and behaviour change of individuals on the programmes. The research question developing from this following Kolb, is that we would expect Converging learners to enjoy and find more useful, the simulation or game activities more than other learning preferences. Kolb’s learning styles suggests that Converging learners (those combining the learning steps of Active experimentation and Concrete experience) will prefer formal learning situations where they can experiment with simulations (Kolb, 1999). RQ2. Do participants with different learning style preferences show differences in reaction, learning or learning transfer? Tentative hypotheses are developed from this: H2.1. Converging learners prefer and show greater improvement in managerial competency from a computer-based simulation or game intervention than individuals with other learning style preferences. Following Kolb et al (2000) and mapping MBTI types to the learning cycle we might expect that an individual’s personality type and their preferred learning style will show commonality, in particular, this research is interested in the converging learning style preference that Kolb (1999) suggests would also show a preference for using simulations in formal learning situations: H2.2. An individual with MBTI type EN will prefer a converging learning style on the LSI Following Patz (1990, 1992) studies that found differences between personality types in learning and perceived usefulness for application in the workplace: H2.3. An individual with MBTI type EN will prefer and show greater improvement in managerial competency from a computer-based simulation or game intervention than individuals with other dominant personality types. After Higgs (1999) and Keys and Wolfe (1990), teamwork is considered an important factor in participant enjoyment and learning and learning transfer. Following Higgs (1999) suggestion that teams are more productive than individuals or competing groups – develops a research question that greater learning may take place in the simulation or case study teams over the competing groups in the game. Also, from satisfaction and pooling information – that members’ who enjoy the team working and find it useful will learn more. The researcher would also anticipate that, after Bal (1995) 79

the close team working environment and exchange of ideas engenders greater feelings of belonging to the team and this would be associated with enjoyment and perceived usefulness of the team and subsequent performance. RQ3. Does participant rating of their enjoyment and perceived usefulness of team work reflect differences in learning or learning transfer? H3.1. Participant rating of enjoyment and perceived usefulness of teamwork will positively correlate with learning and change in managerial competency. This research will also consider the influence of personal characteristics including age, gender, cultural heritage and organisational position after Sternberg (1997): RQ4. How do age, gender, seniority, prior education and cultural background influence the results? Following Aldrich (2002) we may anticipate that younger managers will show a greater preference for computer-based simulation and game than older managers H4.1. H4.2. H4.3. Younger managers will enjoy the simulation or game more than older Male and female participants will show no distinguishable differences in Male and female participants will show no significant difference in change managers. competencies before or after the training intervention of managerial competency Following Spencer and Spencer (1993) and Dulewicz (1992) we might anticipate that senior managers will show a greater level of managerial competency than more junior managers before the training. H4.4. Senior managers show a greater level of managerial competency before the training. Following logical thought and the literature on management learning we might anticipate that: H4.5. Participants with higher prior academic achievement demonstrate higher levels of managerial competency Following Hofstede (1980) and Sarawano (1993) this research also considers how cultural heritage affects performance from the simulation and learning. Following Savvas et al. (2001), it is phrased as a null hypothesis: H4.6. There will be no difference between participants from a different cultural heritage at the four levels of evaluation. 80

A summary of the research questions and hypothesis is presented in Table 17 below:
Table 17. Summary research questions and hypotheses

RQ1 H1.1 H1.2 H1.3 H1.4 H1.5 H1.6 H1.7 RQ2 H2.1 H2.2 H2.3

RQ3 H3.1 RQ4 H4.1 H4.2 H4.3 H4.4 H4.5 H4.6

Are computer-based business training simulations and games an effective way to develop management learning and learning transfer? The simulation and game groups will show higher ratings in participant reaction than the case study groups The simulation and game groups will show greater learning than the case study groups The simulation and game groups will show greater change of demonstrated managerial competency (learning transfer) than the case study groups. The simulation and game groups will show higher bosses rated performance change between groups. Participants reaction will correlate with learning Participant learning will correlate with change in managerial competency Participant change in managerial competency will correlate with bosses rating of performance impact. Do participants with different learning style preferences show differences in reaction, learning or learning transfer? Converging learners prefer and show greater improvement in managerial competency from a computer-based simulation or game intervention than individuals with other learning style preferences An individual with MBTI type EN will prefer a converging learning style on the LSI An individual with MBTI type EN will prefer and show greater improvement in managerial competency from a computer-based simulation or game intervention than individuals with other dominant personality types Does participant rating of their enjoyment and perceived usefulness of team work reflect differences in learning or learning transfer? Participant rating of enjoyment and perceived usefulness of teamwork will positively correlate with learning and change in managerial competency How do age, gender, seniority, prior education and cultural background influence the results? Younger managers will enjoy the simulation or game more than older managers Male and female participants will show no distinguishable differences in competencies before or after the training intervention Male and female participants will show no significant difference in change of managerial competency Senior managers show a greater level of managerial competency before the training. Participants with higher prior academic achievement demonstrate higher levels of managerial competency There will be no difference between participants from a different cultural heritage at the four levels of evaluation.

81

Chapter 4

Methodology

4.1 Key choices in methodology
Management research is a rich and diffuse field of study and the variety of frameworks available to underpin the conduct of research makes the selection of approach potentially difficult. The literature review above includes discussion on the approaches to evaluation of management learning suggesting that a positivistic empirco-rational stance derived from the disciplines of sociology, education and psychology as the majority of books on research methods would indicate is appropriate (Easterby-Smith et al., 1991). In this section, this researcher develops a greater sense of reflexitivity into the research and firstly identifies a number of different epistemological and ontological stances that can be taken and continues to critique the approach identified in the evaluation of management learning literature that is essentially positivistic. The research questions and hypothesis developed from the literature in the previous chapter indicate that whilst essentially derived from the positivistic tradition of management research, the researcher recognises that the ontological stance is far from a single truth. The concepts and constructs are widely debated within the literature and there are many choices in methodological approach ranging from the traditions of positivism to social constructionism (Easterby-Smith et al., 1991). Hence, the ‘truth’ may be complex, and different researchers have widely differing views and the researcher is faced with a choice of simplifying a way of knowing a complex truth, or using a complex way of knowing a simple truth (Worral, 2004). The former, positivistic stance, means the researcher accepts the complexity of differing views and subjects but seeks to synthesise these into unified perspectives, while the latter, phenomenological, approach accepts complexity and allows the researcher to borrow or adapt the protocols of other established research disciplines. The research may also be positioned as a complex truth ontologically and require complex ways of knowing epistemologically – here the researcher adopts a more social constructivistic approach that accepts the multiple realities of the differing views of researchers exploring on many levels with established methods and protocols or creating new methods to make some sense of the complexities. Some may argue that this latter stance is also the domain of the postmodern that, paraphrasing Burr (2003, p204) means the researcher rejects “grand narratives in theory” and replaces “a search for truth with a celebration of the multiplicity of (equally valid) truths”. Whilst this may be a great intellectual exercise it may not provide the business training community with the evidence they and other researchers in this field call for. 82

The literature review on evaluation of management learning (see paragraph 2.2.2 Evaluation models and taxonomies on page 24 above) considers the methodological approaches to the intended study and the research aims, questions and hypotheses suggest that a positivistic approach would be considered suitable. However, there are considerable practical difficulties in firmly rooting the research within a positivistic epistemology. Easterby-Smith et al. (1991, p43) provide a useful checklist of key choices in research design which will be used to show the assessment against each criteria (Table 18).

Table 18. Key choices of research design (Easterby-Smith et al, 1991)

Researcher is independent Large samples Testing theories Experimental design Universal theory Verification

vs vs vs vs vs vs

Researcher is involved Small samples Generating theories Fieldwork methods Local theory Falsification

4.1.1 Independence of the researcher
In this study, the researcher is unable to be completely detached from the phenomena being observed as being both closely involved in the training events and the measurements of each individual, further, the training events are intended to change the organisation through changing the individuals participating and their involvement in the research process itself – this suggests an Action Research tradition. The researcher facilitates all programmes and the organisations sponsoring participants do so on a fully commercial basis – the researcher is dependent upon positive outcomes for all regardless of the organisation’s choice of activity method for business sustainability which goes some way to eliminate the effects of researcher bias identified by Argyris (1980).

4.1.2 Sample size
The positivistic stance suggests that the research study is across a reasonable sample of organisations and (and on a practical level) cultures in South East Asia (principally in Singapore and Malaysia) – crossing cultures is inevitable in such multicultural societies and across different organisations as this researcher believes, on anecdotal evidence, that different organisations (ownership, size, location of operations and location of parent organisation) show large differences in the way they operate and the organisational culture impact highlighted by Trompenaars and Hampden-Turner (1993). The sample size to carry this out would be at least 100 participants on the simulation based 83

programme and a further 100 on the game-based programme, from at least six different organisations, with a control group of 50 participants on a similarly designed training programme which utilises case studies rather than a simulation or game. This sample size should be sufficient for discriminant analysis on the main independent variable (activity type) being in excess of 20 observations in each programme run, and allow, where appropriate, factor analysis with groups of at least 50 observations (Hair et al., 1998). A case study approach, according to Yin (1989), may have an advantage over an experimental approach in this respect that allows the researcher to ask the questions of ‘how’ and ‘why’ about the training intervention and subsequent transfer to the workplace – i.e. a contemporary set of events over which the researcher may not have control. It may also allow for greater flexibility to understand the complexity of the ‘truth’ in a more constructionist way than the above positivistic approach. A case study approach with smaller numbers is likely to be practically easier to organise than the larger, and cross organisation sample required in a positivistic study, however, the case study approach may not be as generalisable – which will be discussed later.

4.1.3 Theory testing or generation
The theories surrounding experiential learning and trait modification are wellresearched and in a positivist paradigm, this study would attempt to test these rather than generate new theories. The wealth of literature, particularly in respect of experiential learning theory and in competency development, whilst replete with debate and conflicting views, has the robust tradition of education and psychology research, and a researcher can choose a particular standpoint and method from which to work. A more constructionist and qualitative approach may highlight more complexity in the understanding of how individuals learn and what it is about simulations or games that help or hinder that process and this may allow the researcher to develop new theories. For this researcher, this would mean a leaning to the approach of Strauss (1987), than Glaser (1992), as the researcher is already familiar with the prior research in this field.

4.1.4 Experimental or fieldwork design
The literature points towards a positivistic paradigm but the practical difficulties anticipated of pure experimental design (i.e. preventing external influences after the training and before the post-test) suggests a quasi-experimental design (Easterby-Smith et al., 1991) and the most common method is the pre-test/post-test comparison design. To anticipate potential flaws of using a related control group (Easterby-Smith and Ashton, 84

1975), this design would use an unrelated control group made up of participants with similar backgrounds and similar organisations. While this may not be truly random, the choices are made by the participating organisations and not the researcher, and whilst not identical, it recognises the practicalities of undertaking such research in the business world. In particular, a case study approach may be considered as an alternative and not attempt to compare the use of simulation or game methods with others which as Pierfy (1977) and Gredler (1996) have pointed out is the major design weakness of many previous studies. However, such comparisons were made with traditional classroom instruction that may have had entirely different objectives – which is not the situation with the comparison suggested above.

4.1.5 Universality
There is a concern that the research design assumes the universal applicability of a generic managerial competency model. Researchers have attempted to validate or build models appropriate to the assessment of managers in the location of this study, Singapore and Malaysia (Sarawano, 1993, Chong, 1997, Kenworthy and Wong, 2003) that show some variation in emphasis of the importance of particular competencies and particular traits that are more highly valued than those in UK and US studies. However, the suggested model is robust and provides a basis for comparison between similar groups, all of whom are in a similar external environment. The practical considerations of the research mean that the outcomes and focus will be on the local knowledge rather than universal and reflects a critical management research view (Easterby-Smith et al., 1991, Reynolds, 1997b). Adopting a more universal approach would mean increasing the breadth of the study to include other organisations in other countries across the world, and in a positivistic paradigm, attempting to control for the myriad influences of local economic and environmental impacts for fair comparison. A case study approach would place the applicability into a more localised context and be considerably easier for the researcher to take environmental and economics impacts into consideration. The majority of the research studies in this field, have been conducted in the US and to a lesser extent, Europe – hence by adding localised knowledge to this, even as a more rigorous and empirical study may allow researchers to extend the applicability of the findings.

85

4.1.6 Verification or falsification
Easterby-Smith et al. (1991) recommend a strategy of falsification rather than intending to seek evidence that supports the currently held views of the world. There is some difficulty with this approach as the currently held views are the subject of much debate and, in particular, contrasting the positivistic and the constructionist paradigms highlighted by Reynolds (1997) in his critique of learning styles. The evidence for previous research in this field is inconclusive, some, Wolfe and Guth (1975) for example found gaming to be superior to cases, while others suggest that it remains difficult to support the efficacy of games as a teaching pedagogy (Feinstein and Cannon, 2002). However, the general leaning in the existing research is that simulations and games have been evaluated to be more enjoyable, show greater learning and learning transfer. As such, this researcher is inclined to seek evidence that supports this current view.

4.1.7 Summary key choices
The methodological discussion above demonstrates that there is no clear and absolute choice. The research is being carried out in the real world and in a complex, multi-disciplinary field of study. Ontologically, the standpoint must be that there is not one universal truth but that there may be multiple truths. Epistemologically, a positivistic paradigm is certainly not the only option and many argue that with a complex phenomenon we call management learning, is demanding of a critical evaluation leaning to a more qualitative approach. However this researcher is keen to synthesise the complexities into unified perspectives such that the research may be applied in the real world and partly fulfil the repeated calls for empirical evidence. In addition, this researcher needs to consider the willingness of clients to participate in this research (and pay for the privilege) and provide them with results that are meaningful and useful. As such, this maintains the approach in a positivistic paradigm, accepting the complexity of truths and a scientific method. A case study approach is suitable for consideration and would both allow the researcher to look in depth at one, or maybe a few organisations, and discover how simulations or games support the management development activities in those organisations. However, the researcher is more interested in the phenomena at the individual level, and hoping to develop a greater understanding of how simulations and games may support learning and differences between individuals rather than the organisation and this lends itself more to an experimental approach, however, there remains complexities to consider. 86

4.1.8 Scientific method – ideal but inherently complex
Experiential learning research lacks rigorously designed studies (Gosenpud, 1990, Easterby-Smith, 1994) and there are relatively few studies attempting to assess the learning and transfer of learning effectiveness of experiential learning interventions and that which does exist, lacks sufficient rigour. The ideal research design is probably impossible to implement in experiential learning which includes random selection of treatment and control groups, full pre-testing, standardised appropriate post-testing and capturing all sources of learning that occur in an experiential learning environment for measurement purposes (Feinstein and Cannon, 2002). Additionally, modern business training simulations are inherently complex both in terms of learning content and fidelity (Gosenpud, 1990) and the intended outcomes are vague, since the focus is usually on very complex, abstract phenomena (Schumann et al., 2001). To overcome some of the issues surrounding evaluation of experiential learning events, Schumann et al. (2001) suggest a framework for assessing the effectiveness of simulations utilising a holistic approach using all Kirkpatrick’s 4 levels of evaluation to provide a means of making a generalised assessment. This concurs with the literature on evaluation of management learning across different approaches (Feinstein and Cannon, 2002).

4.2 Research model
While scientific method would suggest that the purest form of test of the experiential learning model would be one that isolates a single learning cycle, Gibbs (1988) suggests that may not be either possible or even desirable, as all experiences (and therefore the interpretation of those experiences) are influenced by the sum of the preceding experiences. Easterby-Smith (1994) suggests that the classic design of experimental research to assess the effectiveness of a particular training intervention would require two groups, one group to be trained (or given the treatment) and a comparable group not to be trained (receive no treatment). Individuals within the experiment would be assigned randomly to each group and both groups measured immediately before and after the training. The difference between the groups could then be attributed to the training received. In any evaluation of experiential learning, the existing portfolio provides the foundation upon which any test must be based (Morse, 2001). 87

This research design is based on the ‘before and after’ experimental design methodology commonly used in education and the social sciences (May, 1993). The test assumes that the background of each participant remains constant during the cycle and implicitly accepts the existing portfolio of knowledge, experience, motives, traits and values. Therefore, a pre and post test approach seems most appropriate. Figure 13 below shows an overview of the research model adopted:

Pre-test

Post-test Level 1 - Reaction Enjoyment Usefulness for learning Level 2 – Learning Presentation to SMs Affect? Training event Simulation Game Case Study Correlation?

Learning Style LSI III Personality & Background Gender Position Age Race Competencies MCQ 180 Performance Boss performance rating

Affect?

Difference?

Difference?

Level 3 – Transfer MCQ 180 Level 4 – Results Boss performance rating

Difference?

Figure 13. Research model

4.2.1 Validity, reliability and generalisability
The research design is based in the positivistic paradigm yet recognises the pragmatic realities of undertaking the research and as such, will lend itself more to the relativist viewpoint. This section summarises the relativist viewpoint based on EasterbySmith et al.’s (2002, p53) summary table of validity, reliability and generalisability: Validity The research design requires at least 50 appropriate participants undertaking a simulation based training programme, 50 on a game-based programmes and 50 on a case study based programme (acting as a control group) from at least 6 different organisations in Singapore and/or Malaysia. The organisations representing a cross-section of those found typically in the countries (both local and foreign MNCs, SMBs). This is considered 88

to be a sufficient number of perspectives to include for validity for statistical purposes (Remenyi et al. 1998). The Managerial Competency assessment used for pre- and post-testing will be on a 180º basis requiring participants to self-assess and nominate at least two third-parties to assess the participant independently (Higgs and Rowland, 2001). A full 360º process would be desirable but the time required by non-involved parties is considered to be too demanding (Wimer, 2002). Reliability The literature and on-going research (particularly in the US) will provide comparable observations to assess the reliability of the outcomes from the research. The statistical methods used will be rigorous and transparent. Generalisability The research will seek to observe patterns which may be applied to a particular group of the population and may be specific to a particular identifiable set of individuals – though the initial intention will be to present evidence to confirm or contradict the hypothesis presented above. The chosen research model allows the researcher to take an appropriate epistemological and ontological standpoint yet there are issues which need to be addressed or understood as potential limitations on the research.

4.2.2 Issues with experimental design
Easterby-Smith continues warning against experimental design (1994) suggesting that there are innumerable problems to achieve matching control groups and cites several studies where difficulties arise in interpreting results because either: the control group was not truly random (Easterby-Smith and Ashton, 1975); the accepted criterion was open to debate (Bligh, 1971); or the experiment may have been methodologically flawed (Partridge and Scully, 1979). There are, however, dangers in more qualitative methods as in a study by Argyris (1980) who found that despite best efforts to assess delivery methods of faculty, that the behaviour of faculty in practice was contrary to their espoused theories and values.

89

4.2.3 Learning evaluation design
Anderson and Lawton (1997b) suggest that there are two models to choose from regarding the assessment of the effectiveness of a simulation, a pre and post test design to measure the learning (using an objective measure) or an after-only test design using a random control group. They advocate the latter approach but recognise that whilst this may highlight the difference between different pedagogies used, it does not measure learning at an individual level. Since we are likely to be affecting the outcomes anyway by becoming involved (action research) and ethically, it is difficult to justify why one would deliberately give (even if randomly) a treatment that one believes is inferior (researcher bias) – such methodological approaches are unethical (Remenyi et al., 1998). For this reason, the design includes a measure of learning that is post-test only.

4.2.4 Quasi-experimental design
As discussed above, the difficulties in assigning individuals to random groups mean that a true experimental design is not feasible (Easterby-Smith et al., 1991) and precluded (Ross and Morrison, 2003). As such, this research will be a quasi-experimental design. Pre-testing of each individual presents the opportunity to qualify the similarities of each of the groups and provide a benchmark of the basis for the post-test to establish change in individuals’ behaviour at the workplace according to a 180º assessment. The programmes are scheduled to run within the same time frame in an attempt to minimise differences between groups caused by major economic environmental changes (Figure 14).
Game Group

Simulation Group

Training course

Case Study Group

Training course

Pre-test Time

Post-test

Figure 14. Research design

90

Internal validity – threats to the research There are threats to the internal validity of the research and these are considered using Campbell and Stanley’s (1966) classification of history, maturation, testing, instrumentation, statistical regression, selection, experimental mortality and diffusion of treatments. This provides a comprehensive view of potential and real threats to be considered and mitigated against where possible. Each factor is defined for clarity and how this may affect the research is discussed: History Events occurring, other than the training event, may influence the results. As the intervention occurs over time, participants will be exposed to other influences and learning or change may take place. It is considered that such is inevitable as the research cannot be conducted under laboratory conditions, but such events will be random and equitable to all groups. Major events occurring during the period discovered through observation or discussions with individuals will be noted and assessed if the impact does not affect all groups equally. For example, a rapid devaluation of currency in one country, as occurred in 1997 in South East Asia, may have greater impact on one or two groups than others as their currency of trade is adversely affected. Maturation This factor considers the physical or psychological changes taking place within the subjects during the experimental period. This would represent a greater concern for young subjects such as school children who change rapidly in a relatively short space of time. All subjects in this research are adults and the period over which subjects are measured in this research is over a three-month duration, hence, the researcher does not anticipate that subjects will show great maturation during this time-frame. Testing This considers that exposure to a pre-test, or intervening assessment influences performance in the post-test. This is anticipated to influence subjects and behaviour change in the workplace and because of this, all participants, in all groups will take the same pre-assessments and receive feedback only after the post-test in an attempt to nullify the argument that the process becomes a self-fulfilling hypothesis (Burgoyne and Cooper, 1975).

91

Instrumentation This refers to the inconsistent use of testing instruments or conditions. To alleviate the potential problem here, all instruments used are fixed, on-line assessments using standard web browser technology providing equitable and directly comparable results. Subjects are asked to complete the instruments under normal work conditions within a particular time-frame. Statistical regression This considers that subjects scoring very high, or very low on the pre-test naturally tend to score closer to the mean (regress) on the post-test (Ross and Morrison, 2003). This may show an effect on the scores as subjects are made more self-aware and become more open to learning from others – Johari window (Marsick and Watkins, 1990). By not revealing the pre-test scores until after the post-test as suggested under ‘Testing’ above, should alleviate this potential problem. Selection This refers to there being a systematic difference between the treatment groups under comparison. In this study, the subjects of the research are chosen by the client organisation and as such, personal details such as prior educational qualification attainment, age, position in the organisation, gender and race will enable the researcher to show the similarities and differences apparent between individual subjects and the groups. Experimental mortality This refers to the loss of subjects during the treatment period and potential measurement bias as a result. Such is not within the control of the researcher and subjects lost before the post-test will not form part of the analysed data. Diffusion of treatments This considers that particular treatment to one group influences the behaviour of the comparison group. In this study, unlike Easterby-Smith and Ashton’s (1975) problem, the individuals (or their organisation) have chosen which programme they will undertake without being aware of the experiment beforehand and comparison is not being made with a group who have deliberately not chosen or been chosen.

92

4.2.5 Summary research model
In this chapter, the researcher has reflected on the ontological and epistemological stances that may be taken with this research study concluding that while different standpoints and methods have potential advantages, the keen interest of this researcher is at the individual level, and hence an experimental approach is proposed. The choice of approach aligns with the literature on evaluation of management learning in the literature review and aligns with clients participating in the research as being both familiar to them, in the guise of Kirkpatrick’s four levels of evaluation, and useful to them in terms of the results emanating from the research of their programme. The chapter continued with identification of important issues with regard to the proposed approach and highlights the threats to the internal validity and how these would be mitigated or accepted as beyond the researcher’s control and declared limitations of the study. The next chapter will consider how, using this model, the constructs to be measured will be operationalised and details of the programmes to be investigated.

93

Chapter 5

Constructs and programmes investigated

This research in the positivist paradigm requires that the constructs to be measured are operationalised. As this research study intends to evaluate the effectiveness of the training programmes across Kirkpatrick’s four levels of evaluation to provide an holistic assessment, there are a number of different constructs within the research model: Personality Type; Preferred Learning Style; Position in the organisation; Cultural Heritage; Managerial Competencies; Performance Rating; Enjoyment of training programme; Perceived usefulness of training programme; and Learning. Other constructs that are identified in the literature as important factors but were eliminated after the two pilot studies include motivation to learn and motivation to transfer learning, and the transfer climate. Some of these constructs are clear and easily identified within the literature as being frequently used in positivist management learning research, others are subject to greater debate and greater attention is paid to these constructs in this section.

5.1 Operationalisation of constructs
5.1.1 Personality type
The MBTI instrument is commonly used to provide an assessment of an individual’s preferences for processing information and decision-making. The MBTI instrument measures an individual’s preference on four dichotomous scales (Table 19):
Table 19. MBTI dimensions (Myers and Myers 1980)

Extroversion (E) Sensing (S) Thinking (T) Judging (J)

---------versus ----------------versus ----------------versus ----------------versus --------

(I) Introversion (N) Intuition (F) Feeling (P) Perceiving

These four dimensions translate into 16 basic personality types, such as ESFP or INTJ. For each personality type a dominant and auxiliary personality pattern for information processing and decision-making can be identified: sensing, intuitive, thinking or feeling (Myers and Myers, 1980). In examining the internal consistencies based on Alpha coefficients, Myers and McCaulley (1989) report coefficients above 0.7 for the four scales and conclude that test-retest reliabilities are consistent over time. There are some questions over the validity of the MBTI instrument, however there is evidence indicating a 94

degree of validity for the scales and construct validity when compared with a number of instruments (Myers and McCaulley, 1989). Other instruments were considered and after Higgs (1996), Cattell’s 16PF was considered in particular as this may have provided the possibility to analyse the effect of team roles on any differences found. However, the researcher is less familiar with administration of the instrument and providing feedback to subjects but is trained and licensed in the use of MBTI.

5.1.2 Preferred learning style
Two particular instruments are frequently used within management learning research and have been discussed above, Kolb’s Learning Style Inventory (1999) and Honey and Mumford (1983) Learning Style Questionnaire. Both instruments are the subject of critiques in terms of validity of constructs and the validity of the instrument. Kolb (1981) reports test-retest reliability studies with coefficients ranging from a low 0.33 to 0.73 and argues that the LSI dimensions are not considered to be fixed traits but responses are variable and situation dependent. Duff (2000) reports Alpha coefficients for the LSQ instrument range from 0.52 to 0.71, indicating modest internal consistency reliability and concludes that the findings of his study indicate the LSQ may not be a suitable alternative to the LSI for education researchers. However, in spite of the arguments about the instruments respective reliability and validity there is also considerable support for their use and the final choice of instrument became a practical, financial consideration for the researcher where the Kolb LSI version III was considerably less expensive given the intended number of respondents than the Honey and Mumford LSQ.

5.1.3 Position in the organisation
The research model intends that a number of different organisations are included in the study to provide a breadth of different operations and provide for greater generalisability of the results. Unfortunately, especially in the location of study in Singapore and Malaysia, job titles and management grades are less than a clear way to identify different levels within different organisations – for example an ‘executive’ is most often a lower level employee, whilst in predominantly western organisations, an ‘executive’ is often at the top of that organisation. The training programmes are deliberately targeted for middle and senior managers in organisations for the client and,

95

following discussions with clients, the researcher chose a simple distinction between managers and senior managers. Hardy (1996) provides a useful definition that was agreed with client organisations: A manager is any person who has responsibility for people, money or resources. A senior manager is any person who has responsibility for other managers.

5.1.4 Cultural heritage
It is clear from research into the affect of national culture and anthropological studies, that the influence of culture on an individual is not as straightforward as it may have been when Hofstede undertook his seminal study with IBM in the late 1970’s. However, it is considered that national culture may have an effect that shapes the way in which individuals learn, their comfort level and potentially enjoyment of different training methods. As this study is being conducted in Singapore and Malaysia, there are four readily identifiable cultures in both countries, defined by race, they are, Chinese, Malay, Indian and Others. However, the effect of national culture may not be dominated by a person’s race but where they reside and/or the influence of their upbringing or education. Client organisations indicated a keen interest in being able to compare the ethnic groups, whilst this researcher is more concerned with comparison of those with a predominantly Asian upbringing, and those with a western upbringing – irrespective of race. Hence the use of the construct, cultural heritage. To satisfy both the clients desire to compare ethnically, and this researcher’s desire to explore the broader concept of east-west, the construct for ‘others’ includes a person of any race whose home base is not in Asia and who were predominantly brought up and educated in the west – as such this would also include the majority of Eurasians in the sample.

5.1.5 Managerial competencies
The literature review above include an extensive discussion about competency frameworks considered for this research, and concluded that the Hay McBer MCQ (McBer, 1997) instrument was chosen for it’s suitability in assessing levels of competency in general management. The competencies in the instrument are taken from Hay Group competency dictionary and based on the work of Spencer and Spencer (1993). Every core competency in the dictionary reliably differentiates performance in a variety of organisations. Psyfactor undertook a study using the MCQ and conducted analysis to

96

ensure the reliability of the instrument finding over 80% reliability except for team leadership which was tempered by distance management (Psyfactor, 2005). Hay Group were unable to provide technical data for the MCQ and personal communication with the principle author of the instrument (Kelner, 2005) led this researcher to test the reliability of the instrument. The method chosen, the Cronbach-alpha coefficient, produces the mean scores of all the possible split-half combinations. Normally accepted range for reliability coefficients lay between .60 and .80. Above .80 suggests that the scale items have similar wording or measure virtually the same behaviours – as we would expect in this case due to the scaled level of the competencies under scrutiny. Coefficients below .60 are likely to be excessively heterogeneous or ambiguous. The results of the reliability analysis conducted on the seven competency factors of the MCQ from all 266 participants are shown in Table 20 below:
Table 20. Cronbach alpha reliability analysis on MCQ
Reliability Section --------- It em Values --------Standard Variable Mean Deviation AO 15.66278 2.853559 DO 17.11917 3.485665 DI 15.10376 2.824753 II 15.95564 2.760583 IU 16.40714 2.262072 OA 17.22932 2.686803 TL 14.87331 3.981002 Total Cronbach's Alpha 0.898784 ------------------- If This Item is Omitted -----------------Total Total Coef Corr Mean Std.Dev. Alpha Total 96.68835 14.41401 0.8769 0.7678 95.23196 13.62304 0.8642 0.8563 97.24737 14.73271 0.8899 0.6492 96.39548 14.20925 0.8645 0.8864 95.94398 15.34614 0.8996 0.5537 95.1218 15.19556 0.9045 0.5014 97.47782 13.44179 0.8797 0.7712 112.3511 16.70521 0.8988

R2 Other Items 0.7307 0.7768 0.7916 0.8605 0.5212 0.733 0.7652

Std. Cronbachs Alpha 0.899645

The Alpha coefficients range from .86 to .90 with an overall standard Cronbach Alpha shown at .899 – as a widely used instrument, this shows an acceptable reliability (Carmines and Zeller, 1990).

5.1.6 Bosses performance rating
In order to measure the business impact of training, it is now common to undertake a Return on Investment approach such as the model developed by Phillips (1997). Typically, such an analysis is taken some 4 to 6 months after the training intervention. Following discussions with client organisations willing to participate in the study, this aspect was not included in the research for two reasons, firstly, clients considered the data to be confidential and did not want such data disseminated, and secondly the timescale to obtain results was prohibitive for this research. Instead, participants’ direct boss would 97

rate the participant’s business performance on a 5-point scale similar to those already used within the client organisation (Appendix 3), before and after the intervention.

5.1.7 Reaction to the programme
Reaction evaluation is the most commonly used in business training, often forming the only evaluation undertaken. Feedback from participants is sought most often using a rating on a five-point Likert scale. For the purposes of this research, participants’ reaction to the training programme would be assessed immediately following the event asking for their rating of their enjoyment and perceived usefulness of each method used in the programme delivery, i.e. the activity (simulation, game or case study), the theory sessions, the debrief and feedback sessions, and the teamwork during all sessions. Additional reaction data on the facilities, organisation and similar items were collected for the client organisation but do not form part of this research. Appendix 8 shows the common reaction feedback form.

5.1.8 Learning
Participant learning is measured on a post-test only basis (Anderson and Lawton, 1997b, Psyfactor, 2005). Two pilot studies were carried out in developing the final research model to establish suitable measures, in particular, for learning from training. Participants were not keen or comfortable with a formalised test and stated a preference for being assessed on the presentation at the end of the programme. Learning is thus assessed by participants’ bosses at the final presentation on a five-point scale which may be found in Appendix 6. The scale was developed from case analysis measures (O'Rourke, 2003, Schneider, 2001) to be easy for bosses to assess performance with 6 directive questions to provide a single score. Participants are required, as part of their final presentation, to demonstrate what they have learned through the programme by applying their learned understanding of strategic analysis to a real business problem previously identified by the client organisation and allocated to participants. This measure of learning suits the practical nature of the training programme rather than a knowledge test which may be more suitable in an educational environment.

5.1.9 Motivation to learn and transfer learning and transfer climate
The literature on management learning identifies an individual’s motivation to learn and motivation to transfer learning and the transfer climate within the organisation as being a factor in the effectiveness of management learning. Holton (1996) in particular 98

criticised Kirkpatrick’s four level model and considers motivation and transfer climate to be as important, if not more so, than the learning content or pedagogy adopted. This researcher considered the use of instruments to assess the motivation of participants to learn and transfer learning with Holton et al.’s (2000) Learning Transfer Systems Inventory (LTSI), a sixty-eight item instrument with sixteen factors regarding an individual’s attitude to training and motivation to implementing the training. In two pilot studies prior to the finalisation of the research model, participant feedback on the use of the LTSI suggested that the inclusion of this questionnaire, as well as all the others, was too time-consuming and the phrasing was less than ideal for the audience. It seemed that participants were willing to complete questionnaires that would provide them with useful, actionable feedback, such as the MCQ, but motivation may be overly demanding in a business setting.

5.2 Programmes investigated
This research study is designed to investigate three specific blended delivery methods of the same programme: The training programme is designed to develop strategic management capability with emphasis on strategy formulation and even more so on strategy implementation as this has been identified in the literature as an important component in a manager’s knowledge and skills armoury to be an effective manager (Rausch et al., 2001). The programme includes standard MBA Strategy foundations module content. Importantly, content elements are led by the same tutors in each programme and taught using examples. The programmes differ in the activity undertaken at each session. The simulation groups use the computer-based cooperative management simulation, Strategy CoPilot in small groups of three. The game groups use the computer-based competitive management game in teams of 6 to 8, Strategy at the Edge. The case study groups use case studies. Table 21 shows the programme outcomes and Appendix 1 shows an overview of the programme and the activities associated with each session for each of the three groups: Table 21. Programme outcomes
Strategy Programme Outcomes • • • • • • Identify and prioritise critical strategic issues Generate and evaluate creative ideas for new strategic directions Build the assets, relationships and capabilities required to sustain superior returns Plan an achievable implementation strategy How to align organisation strategy and stakeholder needs Present new strategic plans to senior management

99

Each of the programmes is highly pragmatic in nature and focuses on the application of the learning to the real business after the activity sessions, the tutor leads the debrief and links the learning to the organisation – the debrief sessions are naturally different in every programme run as the learning that takes place within the activities and the issues raised are closely linked to the participants interaction and their own business issues. Each programme culminates in a presentation of strategic recommendations to senior managers of the sponsoring organisation. The training programme was not designed directly to develop the managerial competencies but to develop strategic management capability amongst the participants – which one would anticipate would include the development of competencies for effective management. The theoretical content, interaction in the workshop and the activities undertaken were mapped to the MCQ competency framework identified. This process considered the key tasks and requirements in different parts of the programme and linked these to the competency definitions as being implicitly or explicitly part of the programme. For example, the programmes contained a theoretical workshop element on the topic ‘Shared Situational Awareness’ – this topic, based on strategic military training, considers how the interaction of command instruction, shared experience and shared situational awareness allow different units or people to work together more effectively, it explicitly includes a focus on developing others through sharing knowledge and experience in particular contexts with a focus on the intention to develop others knowledge and awareness – this maps well to the competency ‘Developing Others’ as does the requirement for participants to share experience with each other in groups or teams. Implicit mapping to the competency framework includes, for example, ‘Achievement Orientation’ which is identified as a concern for working well or for surpassing a standard of excellence. Implicitly this maps to the game activity because of the intention for the team to win in competition with others, in the simulation, to demonstrate their knowledge and skills to the simulation in order to gain favourable feedback from the system, and from the case study activities as the desire to show understanding, skills and knowledge. The mapping was discussed with fellow trainers and three client representatives and agreed as a fair representation of how and where the competencies would be developed as part of the training programme. It was also agreed and understood that this is not to suggest that the competencies would not be demonstrated or developed in other aspects of the programme, but to ensure that each competency is mapped to some part of the training programme such that there is a reasonable expectation that it would be developed. 100

Table 22 below provides a summary of the mapping of each competency and the mapping to the content, workshop activity or simulation, game or case study either as an explicit component or implicitly:

Table 22. Linking MCQ to training programme

Competency area

Implicit/ Key programme elements Explicit Game – compete against others to win Sim – to demonstrate knowledge and skills CS – desire to show understanding and knowledge Content – shared situational awareness Sharing expertise with team/group Team/group discussion on analysis and decisions Content – communicating strategy Team/group discussions Final presentations Working with others Debriefing and application to business Final presentations Content – Belbin team roles Rotating roles in activities Working in teams/groups

Achievement orientation

I

Developing others Directiveness Impact and influence Interpersonal understanding Organisational awareness Team Leadership

E I E I E E

101

5.3 Evidence collection
Following the insights from the literature on evaluation of management development, this research will measure participants at three levels of Kirkpatrick’s model, Reaction, Learning and Transfer. The fourth level, Results, is not measured directly partly because some client organisations were not willing to agree to publication and partly because the time necessary for business results to be realised. Results is thus assessed by means of a proxy, the participants boss’s rating of their performance before and after the training event.

5.3.1 Procedures
The same procedures were adopted across the programmes; Figure 13 (page 88) above provides an overview of the procedures adopted. Pre-test Instruments used were reproduced in an online, web based format for easier distribution and completion. Each included a brief description of the instrument and the appropriate background information. Participants complete a self-assessment of their learning styles using an online version of the Kolb LSI III (Kolb, 1999) which may also be found in Appendix 2 and complete a self-assessment MBTI Profile. Participants undertake an online 180º Managerial Competency Assessment based on the Hay/McBer Managerial Competency Questionnaire (MCQ) instrument (McBer, 1997), nominating at least two third-party assessors each (boss and staff or peer). Whilst a 360° assessment may provide greater objectivity in assessment, the administrative difficulties and potential for greater data mortality (Wimer, 2002) is considered too high. A 180° assessment is more manageable and should provide good correlation of assessment with performance (Beehr et al., 2001). The mean rating of MCQ will be used predominantly to provide an objective measure (Higgs and Rowland, 2001). Training programme The training programme was designed for managers to enhance their strategic analysis and thinking skills. All participants were provided with a general programme overview and appropriate workshop materials. The general outcomes are shown above in Table 21 and Appendix 1 shows the programmes and the activities. 102

The emphasis on all programmes was on gaining practical knowledge and skills applied to each group’s own business culminating in a presentation to senior managers on strategic recommendations and a plan of action for implementing the business transformation. The following sections provide an overview of each of the activities used as an experiential learning activity and simulation and game overviews may be found in Appendix 4. Management Simulation The management simulation used was Imparta’s Strategy CoPilot®, a highly sophisticated, animation-based computer simulation that combines interactive tutorials, application exercises, built-in intelligent coaching, and tailored feedback to provide an accelerated experience of applying key strategy tools within a realistic and engaging business setting. Working in a richly simulated bottle-making company environment, participants worked in groups of three and began defining their objectives and identifying the biggest issues and opportunities facing the business. In subsequent phases, teams refined the value delivered to customers, sought value capturing positions, and pushed back against the competitive forces acting on the company. Teams also explored different ways of building competitive advantage, taking into account the dynamic nature of most strategic resources and exploring creative ways of building capability. Finally, teams had to plan a strategy and recommend a series of consolidated moves. Management Game The management game used was Celsim’s Strategy at the Edge™ - a total enterprise management game based on the airline industry. Data and algorithms for this game are based on real market data and projections for economic growth, inflation, interest rates and passenger numbers supplied by British Airways, Airbus Industrie, Singapore Airlines, Malaysian Airlines System and the Economist Intelligence Unit. Participants worked in teams of 6 to 8 to establish a new company and launch a new business into the global market. To assist participants with their evaluation of decisions about the business, they had access to a computer loaded with Microsoft Excel and a full working template business model to explore scenarios and decisions. Participants were provided with an initial (virtual) funding base and evaluated product service launch into the market, considering the market dynamics, cost of deployment and operations, and assumptions on customer and competitor action and 103

reaction. The team’s strategic direction is encapsulated in the number and configuration of aircraft, hub and routes together with additional services and pricing to reflect their marketing strategy from budget, no-frills airline to first-class only. Through strategic and financial analysis teams made a separate set of decisions each round (year of operation) of the company. Once decisions are entered from all teams, the game computes market share and profitability based on a market algorithm. Teams competed to achieve their own objectives and must be profitable to win the game. Case Study The case studies used were Boo.com (Kunnath and Sedick, 2001), Cooley’s Distillery (O'Gorman, 1997) and NTT DoCoMo (Times, 1998). These were chosen from case studies used in the Henley MBA Strategic Direction module as appropriate vehicles for practising the use of strategic models and application to real life and allowed for the consideration and application of the same concepts as from the different phases or rounds of the simulation or game respectively. Working in non-competing teams of 6 to 8, participants would analyse the case study and discuss and agree courses of action based on tutor instruction of the objectives in each session. No computer models were provided but participants were free to use any technology to assist their analysis and presentations. The lead tutor for the case studies is an approved Henley MBA, INSEAD and Open University tutor in strategic management. Post-test Reaction Participants’ reaction to the training programme was assessed immediately following the event asking for their rating on a five-point Likert scale about their enjoyment and perceived usefulness of each method used in the programme delivery, i.e. the activity (simulation, game or case study), the theory sessions, the debrief and feedback sessions, and the teamwork during all sessions. Additional reaction data on the facilities, organisation and similar items were collected for the client organisation but do not form part of this research. Appendix 8 shows the common reaction feedback form. Learning Participant learning is measured on a post-test only basis (Anderson and Lawton, 1997b, Psyfactor, 2005). Participants were not keen or comfortable with a formalised test and stated a preference for being assessed on the presentation at the end of the programme. 104

Participants were required, as part of their final presentation, to demonstrate what they have learned through the programme by applying their learned understanding of strategic analysis to a real business problem previously identified by the client organisation and allocated to participants. The presentations were assessed on a five-point scale (Appendix 6) by participants’ bosses. Learning transfer Participants undertake the same 180º online MCQ as in the pre-test, eight to ten weeks after the training programme. This provides a direct comparison with the pre-test on their demonstrated managerial competencies in a timeframe that is sufficient for behaviour change to be demonstrated in a range of everyday situations and considered short enough for other intervening events that may affect any change to be recalled easily and noted (e.g. another training intervention, a new project, a new team). Business Results Participants’ bosses were asked to rate the performance of each individual using the same rating scale used in the pre-test. This allows a direct comparison with the pre-test and was conducted at the same time as the boss’s completion of the post-test MCQ. This rating is used as a proxy for business results which would otherwise require considerably more time and effort to be measured for each individual (Phillips, 1991, 1998). Feedback on results Feedback to participants on their performance rating and competencies assessment forms an important, if not critical, aspect of the evaluation process. However, in order to help ensure that the research is evaluating the training programme and methodology of delivery, and not a self-fulfilling hypothesis (Burgoyne and Cooper, 1975) because individuals become more aware of their competency gaps, each participant received a full report of their assessments and individual feedback on the results after the post-test to facilitate personal development understanding and future planning. An anonymous example feedback report may be found in Appendix 7.

105

5.4 Data analysis strategy
In planning this research study, choices have been made about the research design to be quasi-experimental, the setting in the real-world environment, and the measures of the constructs to be investigated. This section outlines the intended strategies for data analysis to answer the research questions and hypotheses identified as appropriate from the literature. Several techniques will be employed and the principle data analysis software chosen is NCSS (Number Cruncher Statistical System – www.ncss.com). This software was chosen as an alternative as suggested in Remenyi et al. (1998) to using SPSS and SAS as it offers an excellent range of analytical procedures, publication quality graphics and has two distinct advantages over both packages: ease of use through a comprehensive help/tutorial system and, a substantial cost saving at less than half the cost of the alternatives. However, like other software packages, the reports are very comprehensive and the author chose to re-format output using Microsoft Excel to ease the creation of summary tables and some charts. Ross and Morrison (2003) provide a useful table of commonly used techniques in research in educational technologies and this was used as the starting point for the statistical techniques to be employed in the analysis of the data. The next section indicates the statistical tests that will be undertaken with the data based on commonly used techniques in research in educational technologies:

5.4.1 Statistical procedures to be used in the research
t test Independent samples Types of data: Nominal independent variable, one-interval ratio measure Features: Test the differences between treatment groups – can test for causal effects Example: Does the simulation treatment group surpass the game or case study group? – can test for causal effects t test Dependent samples Types of data: Interdependent variable nominal (repeated measure), dependant one-interval ratio measure Features: Test the difference between treatment means for a given group– can test for causal effects Example: Will participants change their behaviour in demonstrating particular competencies from pre-test to post-test following the treatment?

106

Analysis of variance (ANOVA) Types of data: Interdependent variable = nominal, dependent variable = one interval-ratio measure Features: Test the differences between 3 or more treatment means. If ANOVA is significant, follow-up comparison of means are performed– can test for causal effects Example: Will there be differences in learning among three groups that learn from simulation, game and case study Multivariate analysis of variance (MANOVA) Types of Data: Independent variable = nominal, dependent = two or more interval-ratio measures Features: Test the difference between 2 or more treatment groups means on 2 or more learning measures. Controls type 1 error rate across the measure. If MANOVA is significant, an ANOVA on each individual measure is performed– can test for causal effects Example: Will there be differences among three different learning methods on problem solving and knowledge learning? Analysis of covariance (ANCOVA) or multivariate analysis of covariance (MANCOVA) Types of Data: Independent variable = nominal, dependent variable = one or more interval ratio measures. Covariate = one or more measures Features: Replicates ANOVA or MANOVA but employs an additional variable to control for treatment group differences in aptitude and/or to reduce error variance in the dependent variable(s) – can test for causal effects Example: Will there be differences in concept learning among learnercontrol, program-control and advisement strategies, with differences in prior knowledge controlled? Pearson r Types of Data: Two ordinal or interval ratio measures Features: Test relationship between two variables Example: Is age related to test performance? Multiple linear regression Types of Data: Independent variable = two or more ordinal or interval-ratio measures, dependent = one ordinal or interval-ration measure Features: Test relationship between set of predictor (independent) variables and outcome variable. Shows the relative contribution of each predictor in accounting for variability in the outcome variable. 107

Example:

How well do experience, age, gender, and educational qualification predict demonstration of managerial competencies?

Discriminant analysis Types of Data: Nominal variable (groups) and 2 or more ordinal or interval-ratio variables Features: Test relationship between a set of predictor variables and subjects’ membership in particular groups. Example: Do students with different learning style preferences or MBTI types differ with regard to ability, age and enjoyment of sessions? Chi-square test of independence Types of Data: Two nominal variables Features: Test relationship between two nominal variables Example: Is there a relationship between gender and attitudes towards the instruction?

The data analysis considers the impact of multiple variates and variables on the results and as such following the advice of Hair et al. (1998), the analysis of data will commence with examination of the data to observe basic relationships that may be apparent. This will be undertaken through a graphical examination of the data, an evaluation of the process for missing data, identification of outliers and how they may distort the relationships by their uniqueness, and, as has been shown above, the analytical methods appropriate to analysing the data. The data analysis process is iterative in nature and thus allows the researcher to further investigate interesting findings through other techniques such as factor analysis or data filtering that may helpfully explain phenomena that may not have been anticipated.

108

Chapter 6

Results and analysis

Data was gathered from a total of 12 separate groups of participants across three distinct programmes. Specifically, six groups participated in a programme that utilised the management game, four groups participated in a programme that utilised the management simulation and three groups used case studies. Programmes were organised for private companies in Malaysia and Singapore by their respective Human Resource Development departments. Participants on the programmes were either volunteers or nominated by their managers to attend, and were targeted as high potentials in their respective companies for strategic thinking development. The researcher had no influence on the choice of participants. All participants were pre-determined to be computer-literate at time of programme registration – and this information served to establish that the participants were capable of operating a computer and should be relatively comfortable in using a computer-based simulation or game when appropriate. A total of 266 participants completed all assessments and the training programme, a further 35 participants on the programmes did not complete one or more of the assessments (28 the MCQ post-test) or had left the organisation before the post-test was administered. Data was analysed using NCSS (www.ncss.com) and Microsoft Excel. Table 23 shows a breakdown of the percentage of personal characteristics across each group.

109

Table 23. Participant breakdown statistics in each group
Average by Group Sim Game Case All Groups

# of Groups Avg pax/group Male Female Chinese Western* Indian Malay <30 30-35 36-40 >40 Manager Senior Manager Undergrad Graduat e Post Grad

4 20.5 69% 31% 49% 12% 16% 23% 12% 36% 29% 22% 18% 82% 15% 45% 40%

6 21.2 70% 30% 49% 24% 11% 16% 16% 32% 33% 19% 24% 76% 12% 48% 40%

3 19.0 62% 38% 57% 21% 6% 16% 27% 45% 19% 9% 74% 26% 23% 57% 20%

13 20.5 68% 32% 51% 19% 11% 18% 17% 36% 29% 18% 33% 67% 15% 48% 36%

*Western includes Caucasians and other races with main upbringing in the west.

The principle aim of this section is to provide empirical analysis of the data collected to answer the research questions posed following the literature review using appropriate techniques as outlined in the data analysis strategy above. The interpretation of the results was carried out using the convention that a significance level of 0.05 and below is statistically significant (Hair et al., 1998) and follows the dominant thinking within the field of educational psychology and whilst in business research it may be acceptable to choose a less restrictive Type I error, the calls for empirical research in this field demand robust design and analysis and a level of 0.05 is judged appropriate by this researcher. This means that less than 5% of the variation is likely to relate to chance than rater unreliability. Such a level represents a challenging demand and provides a relatively rigorous basis for evaluating change and difference (Higgs and Rowland, 2001). When significance is shown at the 5% level, the researcher undertakes a further test at the 1% level in many instances providing a greater assurance that the test in question is rigorous, and on occasion, when significance is apparent close to but above 5%, the researcher undertakes the same routine at the 10% level. This provides less rigour but may indicate that something is worth further investigation. 110

6.1 Effectiveness of simulations and games
RQ1 Are computer-based business training simulations and games an effective way to develop management learning and learning transfer? The difference between mean pre and post-test scores for each of the groups was reviewed and the result may be found in Figure 15 below. All groups show mean higher levels of demonstrated managerial competency in the post-test MCQ. Figure 15 shows the mean differences across the three groups.
Mean Differences Competency Change
6.5

6.0

5.5

5.0

4.5

4.0

3.5

3.0

Achievement orientation 5.1395 5.7200 4.8127

Developing Others 4.1395 4.7360 3.8709

Directiveness 4.9070 6.3840 5.3345

Impact & Influence 5.5349 5.6800 5.0291

Interpersonal Understanding 5.2791 4.1440 4.4436

Organisational Awareness 4.1628 4.6400 3.3909

Team Leadership 5.4186 5.7760 3.3891

Simulation (n=86) Game (n=125) Case Study (n=55)

Figure 15. Mean Differences in Competency

If we consider that case studies are an effective way to develop managerial competency, then both the game and the simulation are also effective based on the mean difference shown. As such, this indicates that there is a difference worth investigating and as the pre and post-test MCQ scores are carried out on the same individual, a more powerful test, a paired t test, is suitable. The paired t test calculates the difference between each pair of means for pre and post-test MCQ rating and calculates the standard deviation. The paired t test was run for all groups and showed significant difference for all competencies. As the particular interest of this research question is regarding the difference for the simulation and game groups, the paired t-test was re-run filtering the data to include only those subjects on the simulation or game supported programmes. The results of this analysis are summarised in Table 24 below, which shows a paired t test 111

summary of the differences of average MCQ ratings between pre and post-test for the 211 subjects in the game and simulation groups. All clearly show a significant difference suggesting that computer-base business training simulations and games result in positively significant work-related behaviours:
Table 24. Differences MCQ pre to post test t-test summary
95% Confidence Interval in the Difference Lower -5.971 -5.042 -6.321 -6.154 -5.079 -4.889 -6.229 Upper 4.996 3.944 5.243 5.088 4.134 4.002 5.032 Alternative Hypothesis Average PREAverage Post>0 T-Value -22.048 -16.044 -21.029 -20.679 -19.112 -19.632 -18.444 Sig 0.000* 0.000* 0.000* 0.000* 0.000* 0.000* 0.000*

Difference Pre to Post Test Mean Achievement orientation Developing Others Directiveness Impact & Influence Interpersonal Understanding Organisational Awareness Team Leadership -5.483 -4.493 -5.782 -5.621 -4.607 -4.445 -5.630 SD 3.613 4.068 3.994 3.948 3.501 3.289 4.434 SE 0.249 0.280 0.275 0.272 0.241 0.226 0.305

* Significant .05
In order to provide a comparison with the case study groups, we are able to use a two-group discriminant analysis by combining the simulation and game groups and using the case study groups for comparison. The results of the analysis, seen in Table 25, show superior performance improvement in the MCQ factors of ‘Directiveness’ and ‘Team Leadership’ for the simulation and game group combined. Other factors, whilst showing a greater difference than the case study group are not significant. Directiveness and team leadership alone are able to reduce classification errors of 71.4%:

112

Table 25. Discriminant analysis Sim and Game group against Case Study MCQ
Discriminant Analysis Sim and Game Group against Case Study Removed Variable Difference Directiveness Difference Team Leadership Lamb da 0.987 0.966 F-Value 3.46 9.27 FProb 0.064 0.003 Lamb da 0.998 0.956 Alone FValue 0.56 12.29 FProb 0.456 0.001 RSquared Other X's 0.108 0.092

Linear Discriminant Functions Sim and Game Group Variable Sim or Game Case Study Constant -23.436 -14.161 Difference -0.006 0.004 Directiveness Difference 0.259 0.125 Team Leadership

Classification Count Table for Sim and Game Group Predicted Actual Sim or Case Total Game Study Sim or 207 4 211 Game Case 34 21 55 Study Total 241 25 266 Reduction in classification error due to X's = 71.4%

The data suggests that simulations and games are an effective way to develop managerial competency, they may also be superior to the use of case studies in such a training programme. The analysis above in answering the research question suggests that the hypotheses stated in the positive following the literature and previous studies is appropriate.

113

H1.1 The simulation and game groups will show higher ratings in participant reaction than the case study groups. It is appropriate to use a two sample t test of independent samples and comparing each group with each other to establish if the difference in means, seen in the descriptive statistics of the data, are significant. Table 26 below summarises t tests of reaction data across the three groups. Both the simulation and game groups show significant differences compared with the case study group on enjoyment and usefulness of the activity at the 1% level. Similarly the enjoyment and usefulness of teamwork were rated significantly more highly than the case study group at the 1% level.
Table 26. Summary t test reaction
Summary T-Test Simula tion
(n=86)

Mean Game
(n=125)

Standard Deviation Case Study
(n=55)

Significance ** = 0.01, * = .05, = .10 SimGame 0.165 0.301 0.000** 0.521 0.004** 0.691 0.000** 0.000** SimCase 0.000** 0.000** 0.091 0.011* 0.000** 0.000** 0.000** 0.000** GameCase 0.000** 0.000** 0.309 0.062 0.000** 0.000** 0.404 0.838

Simula tion 0.565 0.427 0.687 0.391 0.450 0.404 0.786 0.669

Game 0.582 0.682 0.601 0.674 0.618 0.629 0.595 0.742

Case Study 0.727 0.914 0.934 0.867 0.904 0.893 0.988 0.980

Enjoy Activity Enjoy Teamwork Enjoy Debrief Enjoy Theory Useful Activity Useful Teamwork Useful Debrief Useful Theory

4.657 3.973 4.507 3.944 4.742 4.081 3.611 3.558

4.545 4.059 4.156 3.996 4.518 4.112 4.356 4.120

2.858 3.469 4.275 4.220 3.206 3.416 4.256 4.093

The results show support for the hypothesis as suggested by previous research studies in this field and also Henke (2001) , Schank (1997) and Prensky (2000) amongst many others suggest that games and simulations are, and should be, by design more fun than more traditional pedagogies – the data concurs and as Dixon (1990) emphasised, enjoyment as a by-product of the learning process may enhance effectiveness, which may explain why participants also rated the perceived usefulness more highly. We will return to the aspect of teamwork in RQ3 below. There are obvious and notable differences and further analysis to understand if there is an interaction between the groups by activity type and each of the reaction variables is appropriate, shown in Table 27 this analysis uses Analysis of Variance or ANOVA of the reaction data. The results show that there is a significant difference between the three groups and together with the results in the t test above, supports hypothesis H1.1 in that participant reaction from the simulation and game groups for the activity is rated significantly higher than the case study groups. 114

Table 27. ANOVA Reaction data by activity type
Activity Type F-Ratio Sig Power 177.13 0.000* 1.000 15.36 0.000* 0.999 6.28 0.002* 0.894 3.30 0.038* 0.623 107.09 0.000* 1.000 25.32 0.000* 0.999 26.51 0.000* 1.000 14.86 0.000* 0.999

Summary ANOVA Enjoy Activity Enjoy Teamwork Enjoy Debrief Enjoy Theory Useful Activity Useful Teamwork Useful Debrief Useful Theory * Significant 0.05

However, some reaction variables are contrary to the hypothesis – where the case study groups’ reaction was higher than the simulation or game groups. These are in the enjoyment and perceived usefulness of the theory sessions, and the usefulness of the feedback/debrief sessions. H1.2 The simulation and game groups will show greater learning than the case study groups The same procedures adopted above for the reaction variable were used for participant learning across the groups. Table 28 below shows a summary of the t-tests between different groups on assessed learning at the end of the training programme. This shows a significant difference between the simulation and game groups and the case study group.
Table 28. Summary t test learning
Summary T-Test Simula tion
(n=86)

Mean Game
(n=125)

Standard Deviation Case Study
(n=55)

Sig SimGame 0.557 SimCase 0.000* GameCase 0.000*

Simula tion 0.425

Game 0.534

Case Study 0.975

Learning Significant * 0.05

4.233

4.192

3.291

The data supports the hypothesis and supports the views of Brenenstuhl and Catalanello (1979) that the greater involvement in and enjoyment of the learning activity will lead to greater learning, and also the findings in Wolfe and Guth’s (1975) comparative study where they found that business games showed superior learning to case studies.

115

H1.3 The simulation and game groups will show greater change of demonstrated managerial competency (learning transfer) than the case study groups. As has been seen in Figure 15 above, there are differences between groups in the change of managerial competency. Again, the same statistical techniques were used as for the reaction and learning variables to compare the mean difference in managerial competency. Table 29 below shows a t-test summary of the differences of average MCQ ratings between pre and post-test:
Table 29. Summary t tests MCQ differences
Summary TTest Simul ation
(n=86)

Mean Game
(n=125)

Standard Deviation Case Study
(n=55)

Significance ** = .01, * = .05 SimGame 0.252 0.296 0.008** 0.794 0.020* 0.302 0.566 SimCase 0.587 0.656 0.503 0.406 0.191 0.171 0.012* GameCase 0.137 0.211 0.111 0.340 0.569 0.020* 0.000**

Simul ation 3.344 3.351 3.603 3.289 3.847 3.254 5.306

Game

Case Study 3.682 3.682 3.818 3.835 3.414 3.246 3.271

Mean Difference Achievement orientation Developing Others Directiveness Impact & Influence Interpersonal Understanding Organisational Awareness Team Leadership 5.134 4.140 4.907 5.535 5.279 4.163 5.419 5.720 4.736 6.384 5.680 4.144 4.640 5.776 4.813 3.871 5.335 5.029 4.444 3.391 3.389 3.782 4.492 4.150 4.356 3.177 3.313 3.735

There are two comparisons showing a significant difference at the 1% level: between the simulation group and the game group in Directiveness; and between the game group and case study group in Team Leadership. Three more at the 5% level: Simulation to game group in Interpersonal understanding; Simulation to case study group in Team leadership and Game to Case study group in Organisational awareness. The difference in Directiveness may be explained by the size of the teams during the training, the simulation group of three compared to the game and case study teams of 6 to 8 participants. Larger teams may require individuals to be more directive to ensure that their argument is considered. In addition, the way the simulation was used was cooperative in nature, and this might suggest a higher level of Interpersonal Understanding being developed, which is shown in the data. The difference in Organisational awareness is more difficult to explain other than the game uses real data and forecasting future environments, whilst case studies, by their very nature, are historical. Organisational awareness as a competency is future-oriented

116

including the “[ability to identify] and to predict how new events or situations will affect individuals and groups within the organisation” (Hay/McBer, 1997, p12). Again, we will return to Team leadership in RQ3. H1.4 The simulation and game groups will show higher bosses rated performance change between groups. Table 30 below shows significant difference using t tests between the different groups – and both the simulation and game groups show a higher performance improvement than the case study group. Thus accepting the hypothesis that there is a difference between the groups.

Table 30. Summary t test performance change
Summary T-Test Simula tion
(n=86)

Mean Game
(n=125)

Standard Deviation Case Study
(n=55)

Significance ** 0.01 SimGame SimCase GameCase

Simula tion

Game

Case Study

Performance improvement

0.837

0.528

0.436

0.717

0.589

0.601

0.000**

0.001**

0.000**

Of particular interest is that the simulation group performance improvement is significantly greater than that of the game group and pre-test performance ratings for each group at 3.314, 3.504 and 3.491 for the simulation, game and case study groups respectively were not significantly different. The conjecture from the competency improvement seen above, is that this may relate to greater improvement in Interpersonal Understanding for participants in the simulation group. This makes intuitive sense as the higher levels of Interpersonal Understanding competency demonstration is about understanding underlying issues of other people (including the boss), which may be appreciated and be reflected in perceived greater performance. Correlation of the variables does not however, show a significant relationship between Interpersonal Understanding and Boss’s performance rating.

6.1.1 Correlation on Kirkpatrick’s levels of evaluation
H1.5 Participants reaction will correlate with learning After Kirkpatrick (1959/60, 1974) we might assume that higher ratings for reaction would correlate with higher levels of learning. Correlation routines allow the researcher to see and understand if a relationship exists between variables, how strong that relationship 117

may be and whether it is a significant relationship. Table 31 below shows each of the reaction variables correlating significantly with learning from the programme. Enjoyment of theory sessions and perceived usefulness of the debrief sessions are significant at the 5% level and all others significant at the 1% level.

Table 31. Learning correlation with participant reaction

Correlation participant reaction and learning
Correlation Significance Enjoyment Activity Teamwork 0.505 0.356 0.000** 0.000** Debrief 0.232 0.000** Theory 0.129 0.035* Usefulness Activity Teamwork 0.557 0.337 0.000** 0.000** Debrief 0.149 0.014* Theory 0.175 0.004**

Significance ** .01, * .05

Individually, aspects of participant enjoyment and usefulness are significantly correlated to learning, so using Canonical correlation is the multivariate extension of correlation analysis and allows the researcher to determine the overall correlation, in this case of reaction variables, through finding a weighted average of the reaction scores to learning, shown in Table 32 below, to establish the overall affect of reaction variables to learning. The results of canonical correlation shows that whilst the linear relationship is not strong – R2 – 0.39, it is significant at the 1% level, and the data suggest that enjoyment and usefulness for the activity and for teamwork have the strongest correlation of all reaction data.
Table 32. Correlation reaction to learning
Correlation Section Learning 0.505 0.356 0.232 0.129 0.557 0.337 0.150 0.175 1.000

Enjoy Activity Enjoy Teamwork Enjoy Debrief Enjoy Theory Useful Activity Useful Teamwork Useful Debrief Useful Theory Learning

Canonical Correlations Section Variate Number 1 Canonical Correlation 0.626 RSquared 0.392 Num DF 8 Den DF 257 Sig .01 - ** 0.000** Wilks' Lambda 0.608

F-Value 20.74

118

This result is consistent with previous studies showing significant though relatively weak correlation between reaction and immediate learning (Warr et al., 1999) but these results show a stronger link than those studies (R2 at .07) and led the researcher to use factor analysis in a confirmatory way to help define the underlying structure in the data matrix. This addresses the problem of analysing the structure of the interrelationships among the large number of variables by defining a set of common underlying dimensions or factors. The results from the canonical correlation suggest, a priori, that two factors should be extracted. Factor analysis using varimax rotation simplifies the columns of the factor matrix such that the maximum possible simplification is reached (Hair et al., 1998) of the reaction variables, Table 33 suggests also that reaction on the activity and teamwork will show the strongest correlation with learning.
Table 33. Factor analysis reaction data
Communalities after Varimax Rotation Factors Variables Factor1 Factor2 Communality Enjoy 0.603 0.009 0.613 Activity Enjoy 0.366 0.168 0.534 Teamwork Enjoy 0.133 0.146 0.279 Feedback Enjoy 0.017 0.422 0.439 Theory Useful 0.717 0.012 0.729 Activity Useful 0.319 0.151 0.470 Teamwork Useful 0.000 0.496 0.497 Feedback Useful 0.004 0.748 0.752 Theory

Bar Chart of Communalities after Varimax Rotation Factors Variables Factor1 Factor2 Communality Enjoy Activity ||||||||||||| | ||||||||||||| Enjoy Teamwork |||||||| |||| ||||||||||| Enjoy Feedback ||| ||| |||||| Enjoy Theory | ||||||||| ||||||||| Useful Activity ||||||||||||||| | ||||||||||||||| Useful Teamwork ||||||| |||| |||||||||| Useful Feedback | |||||||||| |||||||||| Useful Theory | ||||||||||||||| ||||||||||||||||

119

The factor loadings seen in the results for enjoyment and usefulness of the activity and teamwork show practical significance, the former above 0.50 are considered practically significant, the latter (teamwork) are considered to meet the minimum level (Hair et al., 1998). To assess the statistical significance of these indications, the researcher created a variate of the average of reaction to the activity and teamwork, and using canonical correlation, the results show R2 of 0.324 significant at the 1% level. A variate of reaction of the activity alone against learning shows R2 of 0.321 significant at the 1% level (Figure 16).

5 4.5 4 3.5 3

R2 =0.3213

2.5 2 1.5 1 0.5 0 0 0.5 1 1.5 2 2.5 3 3.5 4 4.5

Enj oy m e n t a n d U se f ul ne ss of A c t i v i t y A v e r a ge

Variate Number 1

Canonical Correlation 0.567

RSquared 0.321

F-Value 125

Sig ** .01 0.000**

Wilks' Lambda 0.679

Figure 16. Correlation enjoyment and usefulness reaction on activity to learning

The results indicate that participant enjoyment and perceived usefulness of the activity undertaken can explain 32% of the increase in learning supporting the notion that greater enjoyment of experiential activities is an important precursor to learning as suggested by Russ-Eft and Preskill (2001) amongst others. However, as the results above indicated, while the enjoyment and usefulness of the activity reaction data show the strongest significant correlation to learning, the communalities of the variables seen in the factor analysis above for enjoyment and usefulness of the activity being above a general guideline of 0.50 (Hair et al. 1998) suggests that it does not meet an acceptable level of 120

explanation. The indications are that other factors may be in play, and the researcher will investigate this in more detail with multiple linear regression after considering the correlations for learning and learning transfer, and for learning transfer and performance. Other elements that may influence learning, and potentially the results seen above will be investigated in RQ2. H1.6 Participant learning will correlate with change in managerial competency To assess the potential affect of any factor correlating with the demonstrated change in behaviour. In viewing the raw data, it is reasonable to assume that an individual with a high pre-test MCQ is likely to show a smaller increase and vice-versa is possible. Hence, the researcher chose to remove the influence of pre-test MCQ scores and ‘partial’ out using linear regression to remove the influence of prior competency differences. Table 34 below shows a summary correlation matrix of learning against each of the MCQ factors and against the sum total of MCQ difference.
Table 34. MCQ difference correlation with learning
Correlation with Learning - pre-test scores ‘partialled out’ AO Correlation Significance 0.197 0.001** DO 0.061 0.332 DI 0.153 0.014* II 0.075 0.229 IU 0.040 0.521 OA 0.121 0.051 TL 0.210 0.001** Sum MCQ 0.328 0.000**

Significance ** .01, * .05

The results show that Achievement Orientation and Team Leadership and the sum of MCQ differences correlate significantly at the 1% level with learning, whilst Directiveness correlates significantly at the 5% level. Organisational Awareness is significant at 10%. Since the training programme is holistic in nature and designed to enhance managerial effectiveness as a whole, the literature on managerial competency frameworks suggests that an effective manager has a balance of strengths across all the competencies and thus the sum of MCQ differences represents a useful variate to measure overall effectiveness. The results show that learning from the training programme is significantly related to overall (sum) change in behaviour of demonstrated managerial competency and may be interpreted to be 32% of the factors causing overall change in behaviour. Following H1.5, we might expect that participant reaction to the activity would also correlate with the change in behaviour. Table 35 shows a similar significance in Achievement Orientation, Team Leadership and sum of all MCQ factors, Directiveness is less significant at the 10% level. 121

Table 35. MCQ difference correlation with activity reaction
Correlation with activity reaction - pre-test scores partialled out AO DO DI II Correlation Significance 0.167 0.007** 0.058 0.353 0.109 0.081 0.019 0.765

IU -0.049 0.434

OA 0.077 0.218

TL 0.297 0.000**

Sum MCQ 0.263 0.000**

Significance ** .01

From this, the results suggest that participant enjoyment and perceived usefulness of the activity are a precursor to learning, and also a precursor to learning transfer. To help understand the factors that appear to have the greatest influence on the development of managerial competencies, multiple linear regression identifying the sum of all MCQ differences as the dependent variable, and participant reaction and learning as predictors. The objective of the multiple regression undertaken here was to establish if there is a general linear model that can be used to explain the data. The dependent variable is the difference in the sum of all MCQ differences, and the analysis included all reaction variables and the learning variable. Using a model of hierarchical forward with switching, the calculations ultimately establish those variables that explain most of the difference, the relationship and the significance of the relationship. After several iterations, the full model providing the most useful observation included a maximum of 5 subset terms which when reached terminated the algorithm. The routine was run for all groups and then for each of the activity type groups in turn through data filtering and the results are shown in Table 36.

122

Table 36. Multiple Regression MCQ Differences, all groups and each Activity type
R-Squared Section - all groups Total R2 for this I.V. and those above R2 Increase when this I.V. added to those above R2 Decrease when this I.V. Is removed R2 when this I.V. is fit alone Partial R2 adjusted for all other I.V.'s

Sig * .05

Independent Variable Enjoy Feedback 0.004 0.004 0.001 0.004 0.001 Enjoy Theory 0.010 0.006 0.013 0.010 0.014 Learning 0.070 0.060 0.021 0.048 0.022 Enjoy Feedback * 0.078 0.009 0.003 0.012 0.003 Enjoy Theory Enjoy Theory*Learning 0.088 0.010 0.010 0.005 0.011 R-Squared Section - Simulation groups 0.006 0.006 0.152 0.006 0.176 Enjoy Feedback 0.089 0.083 0.130 0.078 0.154 Enjoy Theory 0.170 0.081 0.044 0.066 0.058 Enjoy Activity 0.246 0.076 0.114 0.025 0.138 Useful Feedback 0.290 0.044 0.044 0.005 0.058 Useful team R-Squared Section – Game groups 0.060 0.061 0.095 0.061 0.107 Learning 0.127 0.067 0.005 0.020 0.006 Useful Theory 0.146 0.018 0.055 0.044 0.065 Useful Team 0.164 0.019 0.054 0.000 0.064 Learning*Useful Team Useful Theory* Useful 0.210 0.045 0.045 0.040 0.054 Team R-Squared Section – Case Study groups 0.005 0.005 0.028 0.005 0.040 Enjoy Feedback 0.005 0.000 0.039 0.002 0.054 Useful Feedback 0.100 0.095 0.113 0.062 0.142 Useful Theory 0.258 0.158 0.094 0.167 0.122 Useful Team Enjoy Feedback * 0.319 0.062 0.062 0.000 0.083 Useful Feedback Linear Regression – Full Model up to 5 subsets – Hierarchical Forward with Switching

0.548 0.058 0.016* 0.366 0.088 0.000* 0.000* 0.029* 0.001* 0.029* 0.410 0.005* 0.005* 0.010* 0.000* 0.161 0.101 0.006* 0.012* 0.040*

123

The multiple regression for all groups show that learning has the strongest significant relationship to the increase in managerial competencies. When the results are analysed for each activity type group on their own, the results show interesting differences. For the simulation groups, the strongest influence is the participant enjoyment of the activity – something already indicated above. However, the game and case study groups do not feature this reaction factor, the game groups show that usefulness of teamwork and learning as showing the strongest relationship, while the case study groups show usefulness of theory sessions and usefulness of teamwork and the enjoyment and usefulness of the feedback and debrief sessions as having the strongest relationship. Considering all of these results, it is clear that it is not only the type of experiential activity and how much participants enjoy this or find it useful but a combination of different aspects of the programme and the theory sessions and feedback/debrief sessions and working in teams are important factors in both learning and development of the competencies and transfer to the workplace. The results provide strong support for the use of a blended approach to management development. H1.7 Participant change in managerial competency will correlate with bosses rating of performance impact. Using the same techniques as described above, Table 37 shows each of the MCQ competencies correlation with change in boss’s performance rating. Achievement Orientation and Team Leadership and the sum of MCQ change show significance at the 1% level able to represent some 47% of the differences observed.
Table 37. MCQ difference correlation with performance change

Correlation – MCQ behaviour change and boss performance rating change Prior MCQ and prior performance rating ‘partialled out’
AO Correlation 0.553 Significance 0.000** Significance ** .01 DO 0.084 0.176 DI 0.007 0.916 II 0.048 0.439 IU 0.048 0.445 OA 0.039 0.532 TL 0.474 0.000** Sum MCQ 0.474 0.000**

It appears from these results that participant boss’s performance rating of individual business impact reflects demonstration of managerial competency with a particular emphasis on Achievement Orientation and Team Leadership. As the performance rating is the boss’s perception, it is reasonable to assume that the boss’s performance rating change would reflect their own rating of the MCQ and the difference from pre to post-test. Table 38 shows this correlation, with Organisational Awareness 124

significant at the 5% level, however, the correlation is negative14% – suggesting that higher levels of Organisational Awareness may actually have a negative impact on performance. Achievement Orientation is positively significant at the 1% level, as is overall MCQ change able to represent for some 21% of the variance observed.
Table 38. Correlation boss's performance rating change with boss's rating of change in MCQ

Change in performance rating and change in MCQ - correlation
AO Change in Performance Rating Significance 0.337 0.000** DO 0.077 0.221 DI -0.112 0.072 II -0.025 0.687 IU 0.053 0.397 OA -0.137 0.026* TL -0.029 0.645 Sum MCQ 0.218 0.000**

Significance ** .01, * .05

It makes intuitive sense that Achievement Orientation, being a competency concerned with achieving results and competitiveness, has a significant impact on business results and this concurs with Young’s (2002) model discussed above in the literature review on business results and linking individual managerial competency to organisation competence and performance. Achievement Orientation development is implicit in the training programme as has been discussed in the Operationalistion chapter, and considered to be developed in particular in the experiential activity. It follows from these results, that if a participant enjoys and finds the activity useful, they will, ultimately, have a higher rating for their performance at work after the training event. Table 39 shows this correlation as being significant at the 1% level:

Table 39. Correlation participant reaction to activity and performance change

Correlation of participant reaction to the activity and performance rating change
Enjoy Activity Performance Rating change Significance Significance ** .01 0.273 0.000** Useful Activity 0.236 0.000**

The results from the analysis on correlation across each of Kirkpatrick’s four levels of evaluation suggest that there is a significant relationship, and in particular, that enjoyment and usefulness reaction to the activity is positively correlated to learning, transfer and results. Other factors are certainly involved and these will be considered in more detail in the remaining sections of this chapter. However, this being a possible and significant explanation of the results the researcher uses multiple linear regression to 125

analyse the relationship with single dependent or criterion variables and several independent or predictor variables. The objective of the technique is to confirm the results indicated above. The objective of the multiple regression undertaken here was to establish if there is a general linear model that can be used to explain the data. The dependent variable is the difference in boss’s performance rating, and the analysis included all reaction variables, learning variable and the MCQ difference variables. Using a model of hierarchical forward with switching, the calculations ultimately establish those variables that explain most of the difference, the relationship and the significance of the relationship. After several iterations, the full model providing the most useful observation included a maximum of 3 subset terms which when reached terminated the algorithm. The results are shown in Table 40

Table 40. Multiple Regression - dependent boss performance rating
Analysis of Variance Independent Variable Model Enjoy Theory Enjoy Activity Usefulness Feedback Dependent = Difference Boss Performance R2 0.078 0.009 0.069 0.002 Mean Square 12.443 4.207 32.714 0.908 F-Value 7.423 2.510 19.517 0.542 Sig * .05 0.000* 0.114 0.000* 0.462 Power of Test at 5% 0.985 0.352 0.993 0.114

The best fit model included the factors of reaction of enjoyment of the theory, enjoyment of the activity and usefulness of the feedback. The only significant relationship found was with the enjoyment of the activity, able to explain less than 7% of the difference. When we consider the effectiveness of a training intervention holistically across all four levels of Kirkpatrick’s model, the results suggest that enjoyment of the activity is an important factor ultimately in the business results realised, and has already been seen above, is also important to learning gained, and learning transfer. However, clearly, it does not explain close to everything and there are other, more substantive factors that affect the effectiveness of the training programme. The data analysis shows significant supporting evidence to show that the simulation and game groups’ participants rated a more favourable reaction to the training 126

programme, particularly the enjoyment and perceived usefulness of the activity, and this in turn shows a significant correlation with learning, learning transfer and performance impact - something that the supporters of simulations and gaming have long claimed in the literature. However, the correlations suggest that other factors have an influence and the analyses move on to consider learning styles and personality in RQ2, the effect of teamwork in RQ3 and personal background in RQ4.

127

6.2 Effect of learning styles
The literature on learning styles and how an individual’s preference may shape learning suggests that an individual with a convergent learning style, the combination of Active Experimentation and Abstract Conceptualisation would, according to Kolb (1976) have a characteristic strength in the application of ideas and enjoy formal learning situations that include the use of simulations. Whilst there is considerable debate about how learning style can be and is measured (Mainemelis et al., 2002) and whether learning style is the same as cognitive style (Sadler-Smith, 2001, Reynolds, 1997a) – the concept of learning style is well-understood in the business training community and provides some benefit in an individual’s self-awareness that may help them in their learning. The researcher is interested to establish in this analysis if, as Kolb (1997) suggests a link between learning style preference and participants reaction to the different aspects of the training programme, and the activity in particular. RQ2 Do participants with different learning style preferences show differences in reaction, learning or learning transfer? The summary results of ANOVA of the participants KOLB LSI preference for each variable in reaction, learning and learning transfer between the groups is shown in Table 41 below:. Participants’ reaction to the usefulness of the activity, teamwork and debrief show significant difference at the 5% and 1% levels respectively, whilst in learning transfer, Achievement Orientation and Developing Others are significant at 5% and Interpersonal Understanding at 1% level. There are differences worth investigating and particularly to understand if, as Kolb et al (2000) suggest, converging learning preference will prefer and show greater improvement in learning, and learning transfer than other LSI preferences.

128

Table 41. Summary ANOVA table Kolb LSI preference on reaction, learning and learning transfer between groups

Summary ANOVA table Kolb LSI preference on reaction, learning and learning transfer between groups
Summary ANOVA Enjoy Activity Enjoy Teamwork Enjoy Debrief Enjoy Theory Useful Activity Useful Teamwork Useful Debrief Useful Theory Achievement orientation Developing Others Directiveness Impact & Influence Interpersonal Understanding Organisational Awareness Team Leadership Performance Improvement Learning Kolb LSI Preference F-Ratio Significance Power 1.29 0.280 0.342 0.78 0.506 0.217 0.78 0.508 0.216 1.21 0.306 0.323 3.01 0.031* 0.706 4.90 0.003** 0.906 4.20 0.006** 0.854 0.62 0.603 0.179 3.35 3.05 0.14 1.34 9.39 1.67 2.06 1.27 1.54 0.020* 0.029* 0.936 0.261 0.000** 0.173 0.106 0.286 0.204 0.756 0.713 0.076 0.356 0.997 0.436 0.524 0.337 0.404

Significance ** .01, * .05

H2.1 Converging learners prefer and show greater improvement in managerial competency from a computer-based simulation or game intervention than individuals with other learning style preferences. The data subjects were recombined according to their preferred learning style, grouping all the Diverging, Accommodating and Assimilating participants into one group - named ‘Other LSI Preferences’ leaving those with a Converging style in their own group. The chart and combined table in Figure 17 provides a summary of ANOVA of Convergent and Other LSI Preferences for the simulation and game groups only and the mean difference of MCQ competencies. The competency factors Achievement Orientation and Developing Others are significant at the 5% level supporting the hypothesis. However, Interpersonal Understanding is significant at the 5% level counter to the hypothesis. Other competency factors show marginal and non-significant differences. Given the results, the hypothesis is not supported. 129

Summary Means & Effects
7.000

6.000

5.000

MCQ Difference

4.000

Mean Converging Mean Other LSI

3.000

2.000

1.000

0.000
Achievement orientation Developing Others Directiveness Impact & Influence Interpersonal Understanding Organisational Awareness Team Leadership

Mean Converging Mean Other LSI Mean Square F-Ratio Sig Power (ά=0.05)

6.387 5.107 71.700 5.61 0.0187* 0.655

5.532 4.060 94.847 5.87 0.016* 0.674

5.790 5.778 0.006 0.00 0.984 0.050

5.532 5.658 0.689 0.04 0.834 0.055

2.887 5.322 259.604 23.44 0.000** 0.997

4.581 4.389 1.604 0.15 0.701 0.067

5.661 5.617 0008 0.00 0.948 0.050

Significance ** .01, * .05

Figure 17. Converging and other LSI ANOVA MCQ difference

The LSI has been subject to much criticism as has been discussed, partly because it is a self-assessment and partly because it may not be a suitable instrument for working managers (Honey and Mumford, 1982). This may explain why the results of the analysis do not support the hypothesis, or it may indicate that the critiques for learning style as a concept would consider this supporting evidence for their position. Either way, the results do not appear to provide much insight into the observed differences in learning, transfer or results. The more robust MBTI instrument may provide better insight.

130

H2.2 An individual with MBTI type EN will prefer a converging learning style on the LSI The analysis here requires simple comparison between the results of the LSI and MBTI for each individual. The results of this simple comparative analysis show that 78 individuals within the groups show MBTI type EN, of these, only 26 show an LSI preference for the converging learning style. If the MBTI instrument has greater construct validity than the LSI III instrument, it may be more accurate in reflecting an individual’s learning style. This may explain why the comparative ANOVA in H2.1 does not show the same results. H2.3An individual with MBTI type EN will prefer and show greater improvement in managerial competency from a computer-based simulation or game intervention than individuals with other dominant personality types. Using the MBTI learning style indication that type EN would prefer a converging learning style shows similar differences to those shown above for LSI preferences, however, none of the differences are significant. The combined chart and table in Figure 18 below provides a summary overview of this analysis:
Summary Means & Effects
7.000

6.000

5.000

MCQ Difference

4.000

Mean Converging Mean Other LSI

3.000

2.000

1.000

0.000 Achievement orientation Developing Others Directiveness Impact & Influence Interpersonal Organisational Understanding Awareness Team Leadership

Mean Converging Mean Other LSI Mean Square F-Ratio Sig Power (ά=0.05)

6.572 5.412 2.857 0.22 0.641 0.075

5.000 4.301 20.569 1.24 0.266 0.199

5.552 5.869 4.241 0.26 0.607 0.081

5.345 5.725 6.094 0.39 0.533 0.095

3.983 4.843 31.133 2.56 0.111 0.357

4.690 4.353 4.768 0.44 0.508 0.101

5.776 5.575 1.694 0.09 0.769 0.059

Figure 18. Converging and other MBTI LSI ANOVA MCQ difference

131

Reviewing the results of ANOVA of all reaction, learning and MCQ data by activity type and Kolb LSI (or MBTI) learning style preferences shows significant difference in every factor, often as a combination of preference and the activity type. However the results are mixed, showing converging and other preferences apparently in counter positions to the literature. This may be due to the fact that all programmes are blended and designed, as a whole, to appeal to, and be useful to all learning preferences. Future research may consider comparison with a group who only undertake the simulation, with no tutor input or interference – however, this researcher’s personal experience suggests that such a group would not gain the intended learning or behaviour change intended and was ruled out as unethical during the research design (Remenyi et al., 1998). An interesting finding regarding converging MBTI preference over other preferences was found to be evident in the enjoyment and usefulness of the activity. It was found that other learning preferences rated their enjoyment and usefulness of the activity significantly higher than converging learners, and this was in turn found to be only significant in the simulation group. Figure 19 below, shows a summary of the ANOVA and charts for the simulation group.

132

Analysi s of Variance Table Mean FSource Square Ratio Term Enjoy 4.892 5.76 Activity Useful 2.970 4.04 Activity Simulation Group only ANOVA by MBTI Converging and Enjoy 1.389 4.53 Activity Useful 2.133 11.89 Activity Sig * .05 Level 0.017* 0.046* other LS 0.036* 0.001* 0.557571 0.926216 Power (Alpha=0.05) 0.666607 0.516595

Means of ENJSIM
4.8

4.7

E NJS IM

4.6

4.5

4.5

Convergi

OtherLSI

MBTI_Converge

Simulation group only
Means of USEFUL SM
4.9

4.8

U SEF UL SM

4.7

4.6

4.5

Convergi

OtherLSI

MBTI_Converge

Simulation group only

Figure 19. MBTI Learning style and enjoyment/usefulness of activity

Further analysis showed that Assimilators and Accommodators rated their enjoyment of the simulation higher than converging and diverging, and all three other preferences rated higher than converging for usefulness of the simulation – significant at the 5% level. From this, we might surmise that the management simulation appeals to other learning style preferences contrary to literature – this may be explained by the high fidelity of the simulation causing high levels of presence (Salzman et al., 1999, Stanney et al., 133

1998) in the simulation for the participants who consider the activities within the simulation to be concrete experience and reflective observation of events – certainly the simulation’s designers intended this to be the case. The game, on the other hand, has less fidelity and is more obviously a game and not reality and appealed almost equally to all participants. For the purposes of this research, learning style does not appear to have a significant influence on learning or learning transfer in such blended training programmes but the design of the simulation or game in terms of its fidelity may be viewed as increasingly important (see Mowbray et al., 2003, for more on evaluating fidelity). The analysis has highlighted a number of potentially interesting avenues for further research but any conclusion drawn from these results should be considered suspect and in need of both verification and validation. We will now consider the effect of teams and teamwork on reaction, learning and transfer of learning.

6.3 Effect of teams and groups
The literature on how individual performance is affected by their membership of teams or groups suggests that teams produce superior performance to individuals (Higgs, 1999). This researcher is interested to observe if this is reflected in participant reaction to working in a team during the training, suggesting that if an individual enjoys working in his or her team, they will gain greater performance shown in learning and transfer. RQ3 Does participant rating of their enjoyment and perceived usefulness of team work reflect differences in learning or learning transfer? We have already seen in RQ1 that the participant’s reaction to the experiential activity correlates with learning and learning transfer and noted in the same section that participant’s reaction to the teamwork element of the training programme was the next most influential factor. This question relates more directly to the literature that greater learning may take place in the simulation or case study teams over the competing teams in the game (Higgs, 1999). The combined chart and table Figure 20 below shows the differences in MCQ and learning by activity type.

134

Means by Activity type
6.500

6.000

5.500

DI
5.000

II AO

4.500

IU

4.000

OA
3.500

DO TL Learning

3.000 Simulation Game Case study

Thicker lines highlight those significant at 1% level and interpersonal understanding (IU) whose profile is counter to all others.
ANOVA by Activity type Mean F-Ratio Square AO 18.40 1.40 DO 17.50 1.10 DI 59.94 3.92 II 8.18 0.53 IU 33.40 2.80 OA 30.06 2.79 TL 112.83 6.31 Learning 18.41 47.43 Significance ** .01, * .05 Prob Level 0.248 0.335 0.021* 0.589 0.063 0.063 0.002** 0.000** Power (Alpha=0.05) 0.299 0.242 0.702 0.136 0.547 0.546 0.895 1.000

Figure 20. ANOVA MCQ and learning by activity type

The data are counter to Higgs (1999) suggestion excepting Interpersonal Understanding which, however, is not significant. Learning, Directiveness and Team Leadership all show a significant difference and all show the game groups fairing better than the simulation or case study groups. Interpersonal Understanding development in the simulation group may be because the teams of three discuss their decisions both with each other, and effectively, with the simulation – being co-operative in design and requiring participants to consider the reactions of the simulation characters (players). The case study teams typically work for consensus in decision making and hence this may be a better medium to develop such skills over a competitive game. 135

The game, being competitive (and sometimes extremely competitive!) may provide a better medium for participants to develop their Team Leadership and Directiveness skills. H3.1 Participant rating of enjoyment and perceived usefulness of teamwork will positively correlate with learning and change in managerial competency. Table 42 below shows a summary correlation matrix of the reaction variables enjoyment and perceived usefulness of teamwork with each of the MCQ competency changes and learning.
Table 42. Correlation teamwork with MCQ and learning
Correlation Enjoyment and usefulness of teamwork and MCQ and learning AO DO DI II IU OA TL Enjoy 0.090 -0.062 -0.005 -0.029 -0.058 0.007 0.181 Teamwork Sig 0.148 0.322 0.943 0.640 0.355 0.915 0.003** Usefulness 0.062 -0.052 0.029* 0.084 -0.095 0.074 0.225 Teamwork Sig 0.320 0.405 0.641 0.176 0.129 0.234 0.000** Learning 0.372 0.000** 0.332 0.000**

Significance ** .01, * .05

Following Bal (1995) those who enjoy the teamwork and find it more useful may have a greater sense that they belong to a team, engendering freer debate and discussion leading to greater performance. However, only Team Leadership and Learning are significant at the 1% level whilst the remaining competencies show no significant relationship. The hypothesis is only partially supported and – considering the evidence above, counter intuitive regarding Interpersonal Understanding. We might expect someone who enjoys and finds teamwork useful to also want to understand others and, particularly in these inter-cultural groups and teams, to demonstrate greater inter-cultural sensitivity (Hofstede, 1980, 1991, Trompenaars and Hampden-Turner, 1993) – such does not appear to be the situation here. This supports the notion that it is not only the team’s composition (Belbin, 1981, Belbin et al., 1976), but also the task and the context in which the task takes place that shapes performance and the results suggest that a different learning context such as cooperation, collaboration or competition with others may influence which behaviours or skills are most developed through the training (Higgs et al., 2003). Given the simplicity of the constructs measured, all be it ones that are frequently seen in training programme feedback forms, the evidence is somewhat sparse and the multitude of variables that are considered to affect a team’s effectiveness and how the mix of individuals in the team affects performance suggests that many factors should be taken into consideration before passing judgment on the results seen here. However, it may help focus further research 136

efforts to determine, if, as is suggested here, that the nature and context of the team task within a learning environment affects the development of particular competencies (Higgs et al., 2003). This should be considered carefully, as the number of team members is variable within any particular programme group and different between programmes under study. As the team size was not pre-determined nor able to be enforced rigorously by the researcher this may be a limitation on the results.

137

6.4 Effect of demographics
The final area of analysis considers the demographic data collected as part of the study. The literature review highlights a number of ways in which an individual’s personal background and current situation may shape learning. RQ4 How do age, gender, seniority, prior education and cultural background influence the results? It can be seen from above that the experiential activity and to a significant, but lesser extent teamwork, are significant precursors to learning and learning transfer, but they do not account for all learning or change in behaviour. Nor, from the literature, would we expect this to be the case. Other factors influence how an individual learns and, importantly, their choice to transfer such learning to the workplace. This research does not intend to be comprehensive by attempting to cover all possible variables that may influence an individual to learn and transfer learning, and most notably does not include a measure for motivation to learn or transfer learning for reasons seen above in the operationalisation of the research. However, from the literature and life experience, we might anticipate some commonly referred differences in individuals to show differences in their own learning or behaviour change. H4.1 Younger managers will enjoy the simulation or game more than older managers. Participants reaction on their enjoyment and usefulness of the simulation or game activity during the training programme shows that younger manager (those under 30 years) significantly enjoy the activity more than older managers (Figure 21). Similarly, younger participants significantly rated the usefulness of the simulation or game activity higher than older managers (Figure 22)

138

Mean Under 30 30 to 35 36-40 Over 40 4.879 4.521 4.609 4.479

Standard Error 0.105 0.007 0.007 0.008 Sig 0.018*
Means of ENJSIM

Mean F-Ratio Square 1.103 3.44 Significance * .05
4.9

Power (Alpha=0.05) 0.766

4.8

EN JSIM

4.7

4.5

4.4

Under 30

30 to 35

36-40

Over 40

AGE

Figure 21. Enjoyment of Simulation or Game by age group

Mean Under 30 30 to 35 36-40 Over 40 4.831 4.454 4.725 4.531

Standard Error 0.102 0.007 0.007 0.008 Sig 0.003**
Means of USEFULSM

Mean F-Ratio Square 1.441492 4.75 Significance ** .01
4.9

Power (ά=0.05) 0.896

4.8

U SEFU LSM

4.7

4.6

4.5

Under 30

30 to 35

36-40

Over 40

AGE

Figure 22. Usefulness of Simulation or Game by age group

139

The anomaly with both ratings appears to be with managers in the 30 to 35 age group, so by means of comparison, the same ANOVA test was applied to the case study groups only. Usefulness of the case study does not show significant results, though Figure 23 indicates that the 30 to 35 age group significantly (at the 10% level) enjoyed the case study activity more than other age groups.

Mean Under 30 30 to 35 36-40 Over 40 2.688 3.128 2.556 2.600

Standard Error 0.175 0.014 0.023 0.031 Sig 0.085*
Means of ENJSIM

Mean F-Ratio Square 1.148 2.33 Significance * .05
3.2

Power (Alpha=0.05) 0.681

3.0

EN JSIM

2.9

2.7

2.5

Under 30

30 to 35

36-40

Over 40

AGE

Figure 23. Enjoyment of Case Study by age group

Such results may be counter-intuitive to the indication by Aldrich (2002, Aldrich, 2005), however, in the societies in which this research was conducted, and only on anecdotal evidence and the authors personal experience, those in the 30 to 35 age group have a tendency to prefer a very safe environment, enjoying training that is familiar in style and methodology to their own higher education. Such will be investigated in part when we consider the influence of culture and prior academic achievement in the 140

following analyses H4.5 and H4.6. In the meantime, we shall consider the vexing question of gender difference. Further analysis of the differences of enjoyment and usefulness rated by participants, their age and the activity type was undertaken to understand why the results for all groups are counter-intuitive. Table 43 shows ANOVA of enjoyment and usefulness and learning by age and activity type with significant differences for each evident.
Table 43. Age and activity type ANOVA
Age and Activity Type Analysis of Variance Table Source Mean FTerm Square Ratio Enjoyment Activity A:Age 0.493 1.39 B:Activity 71.142 200.06 Type AB 0.733 2.06 Usefulness Activity A:Age 0.431 1.09 B:Activity 38.951 98.47 Type AB 1.006 2.54 Learning A:Age 0.165 0.44 B:Activity 12.179 32.83 Type AB 1.475 3.98 Significance ** .01, * .05

Sig

Power (ά=0.05) 0.366 1.000 0.741 0.293 1.000 0.839 0.139 1.000 0.970

0.245 0.000** 0.058 0.354 0.000** 0.021* 0.722 0.000** 0.001**

Figure 24 shows the charts of the same ANOVA providing a clear view that the simulation and game rated higher enjoyment, usefulness and learning than the case study group and that the simulation was found enjoyable by both the younger managers and the over 40’s managers. The over 40’s managers show higher learning from the case study than their younger counterparts and found it more useful.

141

Means of ENJSIM
5.0

4.4

SIM TYPE Simulation Game Case Study

EN J SIM

3.8

3.1

2.5

Under 30 to 35 36-40 Over 40 30

AGE

Means of USEFULSM
5.0

4.4

SIM TYPE Simulation Game Case Study

U SEF U LSM

3.8

3.1

2.5

Under 30 to 35 36-40 Over 40 30

AGE

Means of Learning_Post
5

4

Learning_Pos t

SIM TYPE Simulation Game Case Study

4

3

3

Under 30 to 35 36-40 Over 40 30

AGE

Figure 24. Age and activity type ANOVA charts

A plausible reason for such a finding is that these older learners find the game less enjoyable than the simulation due to their possible preference for the ‘softer’ strategic level decision-making attributes or the non-competitiveness of the simulation. This is consistent with Drew and Davidson (1993) claims suggesting that these learners seem to prefer practical and relevant training environments which draw on their experience in promoting strategic thinking and understanding of complex business systems.

142

H4.2 Male and female participants will show no distinguishable differences in competencies before or after the training intervention After Kenworthy and Wong (2005) we might anticipate differences in learning and learning transfer between male and female participants. The data show a significant difference in learning across all groups at the 10% level and significant difference at the 5% level in the case study and game groups, but not the simulation group. Form this and following the correlation seen between learning and behaviour change we might anticipate that female managers from the case study and game groups will show a greater improvement in learning transfer than males contrary to the next hypothesis: H4.3 Male and female participants will show no significant difference in change of managerial competency In spite of the evidence above, pre and post test MCQ results show only one significant (at 5%) difference between male and female participants – Achievement Orientation (Figure 25).
Mean Male Female 5.044 5.985 Standard Error 0.268 0.039 Sig 0.049*
Means of DIFAVGAO
6.0

Mean F-Ratio Square 51.16111 3.92 Significance * .05

Power (ά=0.05) 0.506

5.8

D IF AVGAO

5.5

5.3

5.0

Male

Female

GENDER

Figure 25. Gender and achievement orientation

Considering that participant’s bosses appear to rate performance highly when an individual demonstrates high Achievement Orientation, this change in behaviour may not be simply a nice to have, but a critical differentiator to business results. So why should this be the case? Anecdotal evidence of discussions between this researcher and the 143

participants suggests that the training programme provided them an opportunity, otherwise not available, to practice and demonstrate this competency which would otherwise be frowned upon in these more patriarchal (Hofstede, 1980) societies. Something for further research using a case study approach and more qualitative approach would seem more appropriate for this. H4.4 Senior managers show a greater level of managerial competency before the training. It is reasonable to assume that managers in more senior organisation positions would demonstrate higher levels of managerial competency as measured by the MCQ. Figure 26 shows that this is not the case, managers show small, non-significant higher levels of managerial competency before the training programme than senior managers.

Pre MCQ Managers and Senior Managers
20 18 16 14 12 10 8 6 4 2 0 Achievement orientation Developing Others Directiveness

Managers Senior managers

Impact & Influence

Interpersonal Understanding

Organisational Awareness

Team Leadership

Mean square 14.954 32.152 23.084 4.852 5.607 7.618 34.616

F-Ratio 1.84 2.66 2.91 0.64 1.10 1.06 2.19

Sig 0.176 0.104 0.089 0.426 0.296 0.305 0.140

Power (ά .05) 0.272 0.369 0.397 0.124 0.180 0.175 0.314

Figure 26. Pre MCQ managers and senior managers

We will investigate this apparent anomaly further in H4.6. 144

H4.5 Participants with higher prior academic achievement demonstrate higher levels of managerial competency The summary chart and table Figure 27 below shows the post-test means across the MCQ competencies by prior academic attainment. The error bars on each show the difference from pre to post-test MCQ:

23 22 21 20 Undergrad 19 Graduate PostGrad 18 17 16 15 14
Mean Difference Undergrad Graduate PostGrad Mean Square F-Ratio Sig Power (ά=0.05) Pre-Test significant difference Post-test significant difference
Achievement orientation Developing Others Directiveness Impact & Influence Interpersonal Understanding Organisational Aw areness Team Leadership

4.220 5.649 5.417 32.169 2.47 0.087 0.493

4.902 4.278 4.250 7.041 0.44 0.645 0.121

5.878 5.476 5.896 5.713 0.36 0.696 0.108

4.122 5.578 5.979 50.346 3.32 0.037* 0.626

4.415 4.957 4.125 19.639 1.63 0.198 0.343 0.0025**

3.854 3.988 4.708 17.651 1.63 0.199 0.342 0.0134* 0.002**

4.585 5.600 4.833 24.372 1.31 0.270 0.283

0.022*

Figure 27. Summary ANOVA MCQ differences on prior academic attainment

Across 6 of the 7 competencies – graduates were rated lowest on the pre-test with Organisation Awareness and Interpersonal Understanding showing a significant 145

difference. Graduates were rated higher than both post-grads and undergrads in Directiveness though not significantly. Post-test ratings show increments across all groups across all competencies with Impact and Influence and Organisational Awareness showing a significant difference. The data does not support the hypothesis and it appears that prior academic attainment does not explain differences in managerial competencies prior to the training programme, nor the change in competency levels observed following the training programme. H4.6 There will be no difference between participants from a different cultural heritage at the four levels of evaluation. After Hofstede (1980, 1991) we might anticipate that participants with different cultural heritage would rate their enjoyment of each aspect of the training programme differently. The data showed that western participants consistently rate their enjoyment lower than the others, and when grouping the Asian participants together and comparing the results, enjoyment of the activity and debrief sessions were significantly (5%) rated higher by Asian participants. At the level of learning, Asian participants showed significant greater learning than their western counterparts (Figure 28):

Asian-Western Learning Difference Mean Prob F-Ratio Square Level 4.727 9.3 0.003** Significance ** .01
Means of Learning_Post
4

Power (ά=0.05) 0.859

4

Learning_Post

4

4

4

Asian

Western

Asian_Western

Figure 28. ANOVA Learning difference Asian and Western heritage

146

Significant differences are seen between participants based on their cultural heritage in the simulation and game groups. This might be anticipated when taking the concept of “face” into consideration (De Vos, 1985, Ho, 1976, Hwang, 1987) where both the simulation and game provide a setting in which participants are making judgements in a safe environment and thus offset the relative proclivity of Asians to dwell on face-saving in the presence of an audience. At the learning transfer level, Asian participants show a significant greater improvement in Developing Others (Figure 29).
Asian-Western Sim and Game Group Developing Others Mean Power Square (ά=0.05) F-Ratio Sig 118.9208 7.41 0.007** 0.773 Significance ** .01
Means of DIFAVGDO
5.0

4.4

D IF AVGD O

3.8

3.1

2.5

Asian

Western

Asian_Western

Asian-Western Sim and Game Group Learning Mean Power Square F-Ratio Sig (ά=0.05) 1.951081 8.34 0.004** 0.819 Significance ** .01
Means of Learning_Post
4

4

Learning_Post

4

4

4

Asian

Western

Asian_Western

Figure 29. ANOVA Simulation and Game Group - learning and developing others - AsianWestern difference

147

An interesting aspect uncovered during the analysis suggests evidence that Asian managers show higher ratings in five MCQ factors at pre-test, and three MCQ factors at post-test, than Western managers and Western senior managers. The Western senior managers are predominantly expatriates, and the data suggests that they do not demonstrate as high a level of managerial competency as their Asian, predominantly local, subordinates in Achievement Orientation, Developing Others and Interpersonal Understanding. The combined table and chart Figure 30 below shows this finding which is somewhat surprising given that reasoning frequently given for employing expatriate senior managers is due to the shortage of local talent. The data suggests that this may not be the situation and that bringing expatriates in could be causing local talent to be overlooked for promotion in spite of their demonstration of superior levels of managerial competency. It may suggest that Organisational Awareness is a critical competency factor, for participating organisations, when deciding whether to promote local managers or bring in expatriates.
Asian Manager Achievement orientation Developing Others Directiveness Impact & Influence Interpersonal Understanding Organisational Awareness Team Leadership Significance ** .01. * .05
23.00 23.00

Pre MCQ Western FSenior Ratio Manager 15.10 17.45 15.10 16.13 16.87 18.06 15.48 3.95 4.12 7.36

Sig 0.007**

Asian Manager 21.60 21.52

Post MCQ Western FSenior Ratio Manager 20.00 21.00 21.06 22.77 21.13 22.00 20.48 8.11 9.82

Sig 0.005** 0.002**

16.33 17.67 15.90 16.19 16.57 16.62 16.00

0.043*

20.57 21.19 21.86 21.52

0.047*

20.07

22.00

22.00

21.00

21.00

20.00

20.00

19.00

19.00

18.00

18.00

17.00

17.00

16.00

16.00

15.00

15.00

14.00 Achievement orientation Developing Others Directiveness Impact & Influence Interpersonal Understanding Organisational Awareness Team Leadership

14.00

Pre MCQ Western Senior Manager

Post MCQ Western Senior Manager

Pre MCQ Asian Manager

Post MCQ Asian Manager

Figure 30. ANOVA Cultural heritage and position, pre and post test MCQ

148

Chapter 7

Discussion

7.1 Analysis summary
Table 44 below presents a summary of the results for the research questions and hypotheses:
Table 44. Summary Research Questions and Hypotheses and Findings
Research Question/Hypothesis RQ1 H1.1 H1.2 H1.3 H1.4 H1.5 H1.6 H1.7 RQ2 H2.1 H2.2 H2.3 RQ3 H3.1 RQ4 H4.1 H4.2 H4.3 H4.4 H4.5 H4.6 Are computer-based business training simulations and games an effective way to develop management learning and learning transfer? The simulation and game groups will show higher ratings in participant reaction than the case study groups The simulation and game groups will show greater learning than the case study groups The simulation and game groups will show greater change of demonstrated managerial competency than the case study groups. The simulation and game groups will show higher bosses rated performance change between groups. Participants reaction will correlate with learning Participant learning will correlate with change in managerial competency Participant change in managerial competency will correlate with bosses rating of performance impact. Do participants with different learning style preferences show differences in reaction, learning or learning transfer? Converging learners prefer and show greater improvement in managerial competency from a computer-based simulation or game intervention than individuals with other learning style preferences An individual with MBTI type EN will prefer a converging learning style on the LSI An individual with MBTI type EN will prefer and show greater improvement in managerial competency from a computer-based simulation or game intervention than individuals with other dominant personality types Does participant rating of their enjoyment and perceived usefulness of team work reflect differences in learning or learning transfer? Participant rating of enjoyment and perceived usefulness of teamwork will positively correlate with learning and change in managerial competency How do age, gender, seniority, prior education and cultural background influence the results? Younger managers will enjoy the simulation or game more than older managers Male and female participants will show no distinguishable differences in competencies before or after the training intervention Male and female participants will show no significant difference in change of managerial competency Senior managers show a greater level of managerial competency before the training. Participants with higher prior academic achievement demonstrate higher levels of managerial competency There will be no difference between participants from a different cultural heritage at the four levels of evaluation. Summary Finding Yes Accept at .01 Accept at .01 Reject Accept at .01 R2 0.39 Accept at .01 R2 0.33 Accept at .01 R2 0.47 Accept at .01 Partially shown Reject Reject Reject Partially shown Reject

Partially accept Reject at .10 Accept except AO Not supported Not supported Not supported

149

The simulation and game groups demonstrate participant reaction to the training programme, learning, learning transfer and business performance impact comparable, and often superior to that demonstrated by participants in the case study group. It is clear from the data, that both simulations and games can be said to be effective and perhaps superior methods to develop managerial competencies to the case study method as found by Wolfe and Guth (1975). Thirty years later on this study adds an important dimension to their study, firstly it has been conducted in the business training world rather than an academic environment and secondly, the research has considered not just the learning gained by participants but also the learning transfer of managerial competencies into the workplace and to an extent, the business impact of the training. The results show some support for the idea that each of Kirkpatrick’s levels of evaluation are precursors to the next higher level, with significant correlations from participant reaction to learning and from learning to transfer of learning – noting in particular, that participant enjoyment and perceived usefulness of the experiential activity correlates most strongly and with business performance impact. The results provide empirical support for the advocates of simulation and game-based learning that they are deemed more enjoyable and more useful than more traditional methods such as case studies or lecture by participants and partly through greater enjoyment, lead to greater learning and to greater transfer of learning in terms of demonstrated behaviours in the workplace after the training event. Whilst strong and significant correlations are seen, the use of one method over another does not explain all the learning, transfer or business performance impact seen. Other factors have been considered and in particular, the research has considered the impact of personal learning preference and personality type with inconclusive results. The results do not provide support that computer-based simulations are more effective for converging than any other learning styles. The failure to detect a significant influence of learning styles may be due to issues relating to the use of the Kolb (1999) LSI. Although extensively used in training settings, the evidence for the construct validity and reliability of this measure has not been convincing (Towler and Dipboye, 2003) and this may be reflected in the results seen. The literature having forewarned this possible issue led the researcher to also include the use of the well-known and widely used MBTI instrument (Hodgkinson and Sadler-Smith, 2003). The MBTI instrument is well validated (Myers and McCaulley, 1985) and shows similarly inconclusive results to the LSI preferences. The fact that the MBTI and LSI instruments did not concur with each other may have led us to consider that one or the other may be more useful, however the LSI 150

showed converging style (not significantly) enjoying the simulation or game more than other LSI preferences, the MBTI showed other preferences (significantly) than converging preference. The use of computer-based simulations may have different effects on different learners and these differences can transcend to differences in the preferred way of encountering and assimilating new information (Snyder and Vaughan, 1996). The inconclusive results may highlight the importance of fidelity in simulations and games (Alessi, 1988, Mowbray et al., 2003), the concept of presence (Salzman et al., 1999, Stanney et al., 1998) and/or the suitability of simulations and games, in a blended training programme, for participant enjoyment, learning and learning transfer regardless of indicated learning preference. Other factors considered about participants’ background have highlighted some surprising results, in particular that females show a greater improvement in Achievement Orientation than their male counterparts and that this competency, above others seems to be foremost driver of bosses’ performance ratings. The research also shows that Western senior managers show lower levels of competency than Asian managers, excepting Organisational Awareness – suggesting that this competency may be considered more important when choosing who will fill senior management posts in the region. The findings about participant age and their enjoyment and perceived usefulness and learning within simulations and game based training programmes have important implications for the design and delivery of such interventions. It suggests that simulations used in the present study may be well-suited for older, senior managers as learning is grounded in a meaningful experience of ‘doing’. Garris et al. (2002) proposed that the key features of such advanced, dynamic simulations are that they represent real-world systems, and contain rules and strategies that allow flexible and variable learning activity to evolve. Comparatively, in a Game, what we are triggering is the ‘competitive spirit’ (i.e. playfulness, achievement, greed, and victory) (Prensky, 2000), and the weakness of games is their inability to provide the learner with a dynamic environment (Feinstein et al., 2002). This finding should, however, be interpreted carefully as results could be biased by the fewer representatives in both the youngest and oldest groups defined. The data show that younger managers find the competitive nature of the game more enjoyable since they enter the workforce with a lot more competitive computer-gaming experience (Lundy, 2003, Prensky, 2000, Aldrich, 2005); while older, more senior managers, find the simulation more enjoyable as it provides an ideal way of learning from leveraging their business experience (Bertsche et al., 1996).

151

In short, the results provide evidence to support that using a management simulation or game within a blended learning training programme show greater participant enjoyment and perceived usefulness, greater learning and higher levels of demonstrated managerial competency than the same programme using case studies. It could be that knowledge learned from simulations and games is more likely to be integrated into the cognitive structure of learners because of the higher level of active participation, interest, enjoyment or involvement (Randel et al., 1992). As highlighted by Feinstein et al. (2002), these advanced computer-based simulations involve immersing learners in an environment in which they actively participate in acquiring knowledge. By allowing participants to “practice their skills of decision-making and skills of planning alternative strategies” and evaluate the outcome of their decisions (Hyman, 1978, p155), such simulations provide dynamic visual environments to see the results of manipulating variables that cannot be duplicated in typical turn-based strategies for games (Feinstein et al., 2002). There may also be further benefits for Asian participants over their Western counterparts because there is no issue of losing face with a computer simulation or game, and for female participants in Asian societies because the environment provides them an opportunity to demonstrate traditionally more masculine traits of Achievement Orientation and leadership. The results offer support for the learning benefits of computer-based simulations and games and is consistent with the findings of recent research in cognitive science and situated learning which indicate that simulations and games promote the ‘what-if?’ reasoning and research (Millman, 2000). As emphasised by Romme (2003), business simulations create opportunities to build substantial synergy between learning to think in relevant theoretical frameworks and learning how to deal with the complexity of actual settings. Similarly, Gopinath and Sawyer (1999) suggested that the goal in business education of producing managers who can function effectively in complex business environments requires learning experiences that lead to higher levels of learning, and computer simulations and games can help achieve the desired higher-level conceptual processes of relativism and comprehension through application, analysis, synthesis and evaluation. The significance of the effect of teamwork and debriefing has wider implications for training and development. The importance attributed to feedback and teamwork suggests that computer-based simulations supplement, but do not replace, immediate involvement in real settings (Dede, 1996). Henke (2001) emphasises that the use of computer simulations, particularly those with modern graphical user interfaces, may have 152

much to commend it, but the simulation is still only a supportive tool, albeit a very powerful one, and not an end in itself, it is not an educational panacea (Feinstein et al., 2002). Similarly, Aldrich (2005) supports the notion that simulations as part of training programmes support the learning process and learning transfer and Fripp (1993, p54) concluded that “the best results are achieved when simulations are used in conjunction with other learning methods” and that “no one learning method is able to provide all the knowledge and skills required by managers.” Computer-base simulations have been, and continue to be, valuable tools for many current and future challenges in management learning and development. To enhance learning from simulations, it is essential to look beyond and either/or approach to its use alongside traditional methods in order to understand how and why computer simulations work or do not work. This study has highlighted the importance of several individual factors on the effectiveness of computer-based simulations, and demonstrated its usefulness for learning and learning transfer in developing managerial competency. Although this study is limited and further research is recommended, a step has been taken towards the development of a more comprehensive and empirical understanding of the effectiveness of computer-based simulations in management learning and development. It is not possible to make management training perfect, but over time, it is possible to reduce our errors and gradually improve the quality of our professional efforts (Athanasou, 1998).

153

7.2 Limitations
Although this study has led to some important results, several limitations should be noted: First, the relatively small size of the sample in this research limits the degree of confidence we can have in drawing generalisable conclusions from the present results. The research is limited in generalisability for two principle reasons, geographic location and sample size. Haseman et al. (2002) argue that a larger sample size is necessary to ensure adequate statistical power even though the sample size achieves more than the minimum appropriate for the statistical techniques employed (Hair et al., 1998). As outlined above, this was beyond the resources of the present study and due to pragmatic constraints and the real development needs of client organisations willing to participate in the study. Easterby-Smith (1986, 1994) highlighted that it is often quite difficult in practice to achieve samples of sufficient size. Moreover, the present results are necessarily conditional, specific to the sample characteristics, computer-based simulations and games, and the instructional tools used in the training programmes. Caution is advised in projecting the results to other settings. As suggested by Chapman and Sorge (1999), although computer-based simulations can be effective learning tools, all simulations are not created equal, and it is the responsibility of individual instructors to ensure the quality and efficacy of the one chosen to support learning. Secondly, the instruments used were either entirely self-reporting or included selfreport measures, and data based on such requires caution in its interpretation. Albertson (1995) suggests common problems with self-report measures, including: 1) ratings may not correlate with training application; 2) data are sensitive to participant mood at the time of completion; 3) ratings are sensitive to wording nuances; 4) surveys are often completed quickly and without a great deal of thought; 5) the surveys are often given once only, and thus, cannot assess training concept retention. Indeed, a possible issue with this research could be that certain terminology in the LSI, for example “hunches”, “quiet and reserved”, may have negative connotations in an Asian cultural context, which may bias self-scoring and reduce the extent to which we can obtain significant relationships. Such issues have been mitigated in regard to the measures for learning and learning transfer and business impact, however, drawing conclusions relating self-reported measures with more objective measures still require caution. Learning in this study was demonstrated through a practical exercise and assessed by participant bosses – it is possible to include more formal measures of knowledge learned either by means of testing or, perhaps, through qualitative investigation to establish knowledge learned and used after the training. It was not 154

appropriate for the client organisations in this study, but may be included in studies wishing to understand a more complete picture of the aspect of learning itself. Thirdly, geographically, the research was undertaken in Singapore and Malaysia, with a mixture of local and international client organisations and different industries. Though similar studies in the USA and Europe have shown similar results (e.g. Wolfe and Guth, 1975, Wolfe and Roberts, 1986, Wolfe and Roberts, 1993), many have previously been criticised for relying on perceptions of participants only and not on empirical data (Anderson and Lawton, 1997b). Inclusion of organisations from other countries would enable greater generalisability. As has been outlined above, this was not within the resources available to the researcher and, as the training programmes are run by a commercial entity, the study is restricted to willing client organisations to participate. Fourthly, the participants were chosen by the sponsoring client organisation in each case based on their needs (and not those of the researcher), as such the groups are not absolutely equal in terms of age, experience, gender etc and thus do not allow for an ideal comparison. Further, the team sizes in each group varied according, in part, to the total number of participants in the group, the available facilities at the training location. As such, the diversity in each team may not be apparent and the potential to analyse the effects of teamwork on performance is more difficult to compare equitably. The results also clearly show that factors other than those identified in this research model are likely to be having an influence. This is not surprising given the complexity of the human mind and the ongoing debate about what learning is, let alone how people learn. However, the researcher considers that the limitation of not including motivation to learn or motivation to transfer learning may be important. The use of a potentially suitable instrument, the LTSI (Holton et al., 2001, Holton et al., 2000) proved too much for participants on the pilot programmes, and this may have been more useful than the Learning Style or Personality Type questionnaires in retrospect.

155

7.3 Directions for further research
Additional research replicated on other training settings and larger sample sizes would strengthen the argument for the effectiveness of computer-based simulations and games across different learning styles, in different countries. A deeper understanding of the process of learning will also require information about what has happened longitudinally as a result of the training and observe how participants change over time consequent to being exposed to the training programme. Certainly, an optimal way would be to implement strict experimental designs, obtain conclusive empirical evidence and replicate the results found in the present study. Wolfe (1990) argued that maintaining rigour in simulation-based learning assessment is important because there is controversy surrounding its learning value, and because nonrigorous evaluation research has only added to the confusion surrounding the controversy; but Wolfe also noted that many features of the ideal research design are probably impossible to implement in management training, a view shared by Easterby-Smith (1994). It is thus recommended that for any training evaluation to be meaningful, training criteria must be psychometrically sound, valuable for decision-makers, and must be able to be collected within typical organisational constraints (Tannenbaum and Woods, 1992). An alternative approach for researchers would be to consider a case study approach and include qualitative assessment to provide a richer source of data that may help identify more clearly the factors that drive learning and learning transfer when using simulations or games. Future research should attempt to increase the validity and credibility of evaluation data by using multiple instruments and data sources to assess the effectiveness of computer-based simulations and games. In general, the more data sources used to evaluate a training programme, the more complex is the picture of its effectiveness (Carnevale and Schulz, 1990). It would, for example, be useful to also assess participant personality with a robust instrument (such as the OPQ or 16PF) before the training to potentially explain the impact of, for example, team roles on the team leadership measurement. It may also be useful to understand participant motivation and personal values which are often considered critical in transferring the learning to the workplace (Zalatan and Mayer, 1999, Whitehall and McDonald, 1993, Garris et al., 2002, Tannenbaum et al., 1991). This study has provided useful insights that could facilitate further research to examine the impact of factors concerning the nature of the simulation itself (i.e. the fidelity and veracity) on learning and learning transfer. For example Hannafin et al. (1996) 156

argue that few researchers have questioned the benefits of high-fidelity simulation design, suggesting that more is not necessarily better, and that we should be aware of the cognitive demands that computer-based simulations place on learners and thoughtfully apply techniques that support, not interfere with, learner effort. While it is believed that multimedia provides a tool and medium to accommodate different learning styles, it remains a question as to whether this accommodation influences the retention, recall and application of subject content (Snyder and Vaughan, 1996). Finally, future consideration should be given to the comparative effectiveness of various computer-based training (CBT) methods (e.g. CD-ROM vs. web-based, synchronous vs. asynchronous delivery). According to Hannafin et al. (1996), the computer represents a unique learner-centred opportunity, learners are able to control a wide variety of instructional variables including the subject matter content, the context of instructional situation, amount of practice undertaken and the amount of advice to be provided during the instruction. In effect, computer based learning systems can encourage learners to build relationships among learning concepts through exploration, experimentation and manipulation much more easily than in a traditional classroom environment. The relatively recent ease of access to the Internet has already fuelled the proliferation of CBT to both industry and university (Henke, 2001, Schank, 2002, Aldrich, 2005). Drew and Davidson (1993) proposed that in the near future, new types of technology, such as artificial intelligence and virtual reality, will provide expanded possibilities for CBT – we don’t appear to be in that near future yet, however examples of the implementation of such technologies abound in research laboratories (Witmer and Singer, 1994, Stanney et al., 1998, Salzman et al., 1999, Amstutz et al., 2004) and as recommended by Feinstein (2004) rather than comparing CBT to traditional forms (i.e. lectures or case studies), future research should focus on evaluating the educational effectiveness of CBT on its own merits.

157

7.4 Practitioner guide
What does this mean for the practitioner? Inclusion of computer-based simulations or management games is becoming increasingly common in both management education and management training. The claims for such programmes include that they accelerate the learning process and provide a realistic environment for managers to practice and hone their skills, often in a holistic fashion in Total Enterprise Simulations or capstone programmes. The first problem for practitioners is that the term ‘simulation’ includes a plethora of computer-based and noncomputer-based activities, as such, this practitioner guide is focused towards the use of computerbased ‘business’ or ‘management’ simulations of a non-competitive nature which have a reasonable degree of realism and utilise graphical interfaces and interactive characters who approximate artificial intelligence through the inclusion of complex decision trees and multiple paths to achieving an outcome. A business or management game, adds the intention to compete with others (including computer players) using the same game at the same time but not necessarily synchronously and tend to be largely driven by numbers and market-like algorithms that simulate reality. A game may be less interactive in and of itself and usually has a clear objective to win. Simulations are suitable in particular, for the younger generation – those who have grown up with and use computer games, and the older generation, those born in the 60’s and before who are computer literate and providing the simulation has a degree of behavioural realism and requires a deeper analysis of the situation. Competitive games appear to be most suitable for the group of managers in between – and, depending upon the sophistication of the interactivity, appeal to the younger computer game players as well. The results of this research suggest that simulations and games have advantages over more traditional teaching or training methods, such as case studies, in the development of practical knowledge and the skills of using such in the workplace and participants will enjoy these activities more. However, the results of this research suggest that it is the inclusion of such activities within a blended development programme that includes human tutors, in particular for feedback and debriefing sessions, and with participants working together in teams is likely to benefit both the individual in the development of his or her competencies and also, benefit the organisation in terms of the impact on the business. In more direct terminology, practitioners should not simply replace training with a simulation-based method, but consider how such technology can enhance training effectiveness and ensure that tutors or facilitators are on-hand to help draw out the learning and apply it to real life. From this research there are five key questions for the practitioner to answer based on the objectives of the training for the participants and for the business:

Use a game or a simulation? Essentially the difference here is competition over deeper
analysis and decision-making. Games are likely to develop Directiveness over a simulation, and a simulation is likely to develop Interpersonal Understanding over a game.

158

Team-based or individual? This research has not compared directly with individuals, but
team working is likely to benefit development of Team Leadership and Directiveness competencies in particular and participants are likely to learn more.

Fidelity? This is a difficult construct to define and even more difficult to measure but it
refers to the visual, auditory (and in virtual reality, kinaesthetic) qualities of the representation in the simulation or game. Greater fidelity is likely to lead to greater enjoyment of the activity, i.e. if it looks, sounds and feels like a real situation but it is known not to be, then participants will enjoy the activity more and find it more useful.

Realism? Other than how realistic the virtual environment is, the amount of realism in the
simulation or game is considered important. The game, in this research, has more realism in that it is competitive, but importantly it is a simplified realism. If a simulation or game has near complete realism the number of variables influencing the participant makes the tasks near impossible to achieve and takes longer than in real-life to achieve and frustration is likely to take over learning. For an example you might like to play The Sims 2.

Stand-alone or blended? Some simulations and games are designed for an individual to
use, and may be very effective in developing the learning objectives for which they were designed. However, this research suggests that using simulations or games to enhance learning environments rather than replace them, is likely to be more effective. The (human) tutor or facilitator is able to help draw out the learning from the activity and help participants apply the learning to real-life situations that are personalised and internalised. The artificial intelligence or AI embedded into an increasing number of simulations and games has vastly improved and goes part way to doing this, but the AI does not have the flexibility of creative thought…yet!. Lastly, can simulations or games be used for anyone? The research here suggests that yes, anyone and everyone will benefit from using a simulation or game in an enhanced learning environment, some more so than others and this author’s advice to anyone choosing a simulation or game to enhance learning environments is to ask the question of the designer or supplier “what evidence do you have to support the claims of meeting the objectives?”

159

Chapter 8

Conclusions

8.1 Summary of key findings and conclusions
This research set out to assess the effectiveness of using simulations and games as a means of developing managerial competency, heeding the call for rigorous empirical research to understand effectiveness at each of Kirkpatrick’s four levels of evaluation. In the scientific yet pragmatic tradition of quasi-experimental design the aim has been achieved. This research has considered the effectiveness across different learning styles and the effect of participant background on the results as well as the importance of team and group effects. Simulation and game activities were found to be more enjoyable by participants and they perceived the activities to be more useful than those using case studies. Enjoyment of learning activities and perceived usefulness (Warr and Bunce, 1995, Warr et al., 1999) is often considered to be motivational, and importantly, intrinsically motivational and lead to greater performance (Druckman, 1995, Hoberman and Mailick, 1992, Geber, 1994). The t-test and ANOVA results show that there are significant differences in enjoyment and perceived usefulness between the different programmes and this is similar to findings in other studies (e.g. Wolfe, 1990, Wolfe and Guth, 1975). In demonstrating the potential link of enjoyment and perceived usefulness to the next level of evaluation, the results show that the enjoyment and usefulness of the activity has been seen to correlate significantly with learning and learning transfer and business impact in this study, i.e. across all four levels of Kirkpatrick’s (1959/60) model of training evaluation supporting the notion that higher ratings of participant reaction do lead to greater learning, transfer and results. However, whilst the correlations are significant, they are not strong and other factors have been shown to have an effect including team work, participant age, gender and cultural heritage. Important considerations of personal learning style preferences and personality type have shown inconclusive results and cannot be deemed to either support the theories in the literature or explain the observed results. Reaction and Learning level evaluation studies are relatively common in business education and in the evaluation of business simulations. This study is more comprehensive, and evaluated across all four levels of Kirkpatrick’s model and has clearly shown differences in change of demonstrated managerial competencies and in performance rating. The principle aim of this research was to understand if computerbased simulations or games are an effective way to develop managerial competencies. By 160

comparing three strategic management training programmes using a simulation or a game or case studies it has been found that both simulations and games are effective in enhancing the development of managerial competencies. This research has been conducted in the real business training world and shows that the results of previous studies in education are reflected in this environment and has gone further to provide empirical evidence that simulations and games are effective in enhancing learning, transfer and business impact. The realities of conducting research in the business environment have limited the scope of this research and it is recommended that future research should include other factors believed to shape learning, and in particular motivation to learn and motivation to transfer, in a more qualitative way. This research is an important contribution to the ongoing debate on the efficacy of business simulations and games as a teaching pedagogy and a learning medium. Using a robust methodology that recognises real-world issues, the research provides empirical evidence in support of the use of business simulations and games to develop managers. By evaluating the programmes across all Kirkpatrick’s four levels, the study provides a holistic perspective with strong indications that there are causal links between levels but also highlights that other factors are important. In particular, the role of motivation to learn and to transfer learning is highlighted as a potentially critical factor to consider. There is no evidence in this study to support or contradict the effect of an individual’s learning style preference or their personality type in their demonstrated learning or behaviour change that links to a particular intervention method but there are aspects of an individual’s demographics that are noteworthy. The evidence in this study suggests that female managers may benefit more, than their male counterparts, from a simulation-based training programme – it is also noted that this may be peculiar to the more patriarchal societies in which the research was conducted; however, it may be worth further investigation. There is also evidence that Asian managers may have found the simulation-based training more beneficial for them to develop particular competencies than their western counterparts. The literature on the concept of ‘face’ provides some insight as to why this may be and, in short, suggests that computer-based simulation training may provide a genuinely safe environment where fear of ‘loss of face’ has no place and allows people to experiment freely as they learn. Several authors have criticised previous simulation research that compares simulations with other, more traditional, methods because the objectives and purpose may be different for each method. This is not the situation with this research, and the 161

comparison with case study groups allows this researcher to compare against a standard, accepted base that provides a useful benchmark. In doing so, this research shows that using business simulations or games in training is as effective as using case studies, and almost certainly more effective. Perhaps efforts to compare them with other methodologies can now be redirected to better understand how to design and use them to greatest effect.

8.2 Personal learning reflection
When I started my DBA journey, I considered that my background and competencies and continuous quest for personal development meant that I had the right blend of relevant skills, practical experience and knowledge and motivation to bring my twin passions - the development of people and the use of technology to enhance learning together. I chose to undertake the DBA at Henley as a vehicle for pulling these strands together and consolidate them establishing the effectiveness of using simulations to develop managers during training events. Increasingly my consultancy work has been driven by a move towards strategic training and development, and price sensitive services. This has required me to compete with many other management trainers, business schools and management consultancies. I have had to originate, design, develop, manage and deliver practical, added value and easily understood people interventions that managers, senior managers and directors can own, use and promote within their organisations. I hope to extend myself further, utilising the DBA in parallel with my business role to obtain an international market niche within the management development arena. Discussions with fellow Henley DBA and other Doctoral researchers made me realise, sometimes late in the day, that I could have chosen a much easier route than the one on which I embarked. Notwithstanding, the changes in my life and career during the period have often caused me to re-consider the route I was taking. On occasions, it became evident that the hoops I had created to jump through were too stretching and that I was unlikely to achieve the targets if I couldn’t gain some control over my weakest personal competency cluster, organising and planning. Strongly linked to this was prioritisation. The fact is that my DBA studies often took a back seat, usually to work commitments and I was unable to dedicate sufficient blocks of time to write. I discussed the problem with friends and colleagues who were either still in the process of completing their own doctorates, or had successfully completed. 162

After the first four months, I was finding it impossible to keep up the initial pace of study, my work commitments increased and I was finding it difficult to find an appropriate route and academic hook that would give my research appropriate structure and achievability. I found myself reading seemingly vast amounts of literature just trying to find out what I needed to do and how others had gone about it, but I wasn’t finding anything in the specific area of my interest. Researchers who had written about simulations either concentrated on the technology aspect or were talking about simulations of a very different nature. I visited the Singapore Airlines flight simulator training centre to see if I could use the idea of learning to fly with learning to manage a business but it seemed to me to be similar but too removed and was tempting me further down a technology route that I was determined was not going to be my focus. In the first year I was becoming downbeat, and on one of my very infrequent visits to the UK and Henley for a Research Techniques workshop I, at last, began to share my frustrations with my mentor and fellow DBA cohort. The good news was that I was not alone in my frustrations, in fact a couple of my fellows made me feel that I had actually been making positive progress. The bad news was that I still hadn’t found the hook I needed until a casual drink in the bar with the person who was eventually (after another beer) to be persuaded to be my supervisor. The amazing thing to me was he simply mentioned a name as in ‘you might take a look at John Burgoyne’s work on evaluation’. What evaluation had to do with simulations was, it transpired, for me to find out and reviewing this led me to management learning, which led to others and experiential learning theory and developing competencies and then, to a new seeming dead-end. I recall being warned in the opening programme of the DBA that anyone attempting to research a new field of study should stop and rethink, because it wasn’t going to be achievable. The new brick wall was an apparent complete lack of research in the evaluation of simulations outside of the US Armed Forces, NASA and a couple of interesting but deep in the world of educational and cognitive psychology studies in Cambridge. The USAF reports bore promise but access to more than tantalising glimpses was strictly controlled. So should I postpone the DBA, study Psychology and come back later, or was there another route? I was convinced that somebody, somewhere was researching simulations and then I met a fellow trainer for lunch. She knew my interest and passed me a book ‘The Guide to Business Gaming and Experiential Learning’ published by the Association for Business Simulation and Experiential Learning. Not only was someone doing research in this field, there was an Association of people doing it – they just happen to be rather keen on referring to ‘Gaming’ rather than ‘Simulations’ and 163

do not tend to get published in the more ‘respected’ academic journals. I joined the association and received a CD ROM complete with some 30 years of research papers. For me, this was the researcher’s birthday and Christmas rolled together. It was another year before I went to the US ABSEL conference and met with these similarly interested academics and researchers and was able to make substantial progress in understanding the pitfalls and issues that they had faced and that I would contend with. Following this, my Research Proposal had been accepted and forewarned that I really did realise just how much I was undertaking, Malcolm Higgs formally agreed to be my first supervisor after a half pint of Tiger – I’ve often wondered what he’d do for a pint? My lit review and proposal had guided my efforts and now all that remained was to line up as many clients as possible to agree to the training and the associated research. In a highly competitive market and within economies that were still officially in recession, this proved to be traumatic both professionally and personally. I had to retain as much academic rigour as possible and I was forced to compromise some in order to be able to carry this out in the business world with all the associated pressures of constant change and external forces impacting on the businesses, then on a professional and economic level, I was being forced to cut prices and potentially undermine the long-term sustainability of my business. Some clients were only willing to participate in the research if the training was free but I felt that this compromised my ethics and they were not included, nor are they still clients but something had to give. Once I was into the fieldwork fully, the difficulties were mostly collecting post-test data – in spite of regular automated emails asking people to complete their MCQ, or get their boss to do so, the vast majority needed a phone call and a personal pleading to please help. Most of the participants were great in completing all the questionnaires themselves, but getting their boss to do so was often exasperating as the time pressure to try and ensure everything was done within a small window took hold. Nevertheless, the fieldwork was completed within the time frame, a few data subjects had either moved or changed jobs but sufficient numbers had all completed. I wrote my second working paper before all the data had been collected and submitted it for the ABSEL 2005 conference to be held in Florida the following March. It was accepted and I received some excellent feedback from peers on how to improve it. I had been unable to get over to the UK for the Research Colliquia at Henley – the timing was always inconvenient – and I hadn’t really had feedback on what I was producing as a result of my efforts, so the feedback from ABSEL members was welcomed. OK, that’s not quite how I felt at the time as it made me realise just how little I had really done. It was 164

only through pulling everything together that I noticed gaping holes in my original paper and also therefore, my proposal. The last part, at this point, is the writing up of the thesis. Taking the feedback I had received into account I wrote, what I thought was a pretty decent first full draft. Both Malcolm and Vic were fabulously swift in their feedback and suggested that I had the main components in place and gave me great direction on what else I needed to do. Once again, I discovered that there was a lot left to do – it seemed initially that it would be pretty straightforward but, thanks to their feedback, I found that actually there were still holes in the literature, the method, the results, the conclusions, oh, and the introduction… in short, everywhere. Not to be disheartened, I have, I believe, plugged the gaps and realise that when I write, there’s an awful lot that I have assumed and usually consider unnecessary to include and am beginning to realise that when you take out the assumption and try to support it, sometimes you know it’s anecdotal, other times, you’ve read it somewhere and filing and organisation come to the fore for those that have it, for me, it means a re-search through piles of paper, books and hundreds of pdf files. So, what have I learned that I would pass onto others? Like the research above, motivation is key – sometimes you have to dig very deep inside and re-find the purpose. Then you need friends – talk to peers and colleagues and share your research as much as you can – someone will provide a snippet of an idea, or a whole lot, and the way ahead becomes clearer and it is highly likely that someone else, somewhere has done something very similar and the Internet makes it much easier to find them. Lastly you need support from your partner (and family) – every weekend, many mornings and many long nights are spent with your head in books, reading journals, getting data and then you develop CTS as you finish – and having someone who encourages, cajoles and totally supports your study just makes it all possible.

165

Bibliography

ABRAMSON, N. R., KEATING, R. J. & LANE, H. W. (1996) Cross-national Cognitive Process Differences: A Comparison of Canadian and Japanese Managers. Management International Review, 36, 123-148. ADLER, N. J. (1986) International Dimensions of Organisational Behaviour, Boston, Mass., Kent. ALBERTSON, D. S. (1995) Evaluating Experiential Training: Case Study and Recommendations. Developments in Business Simulation & Experiential Exercises, 22, 166-171. ALDRICH, C. (2002) A Field Guide to Educational Simulations, Alexandria, VA., American Society for Training & Development. ALDRICH, C. (2005) Learning by Doing: The Essential Guide to Simulations, Computer Games, and Pedagogy in e-Learning and other Educational Experiences, San Francisco, John Wiley & Sons, Inc. ALESSI, S. M. (1988) Fidelity in the Design of Instructional Simulations. Journal of Computer-Based Instruction, 15, 40-47. ALLIGER, G. M. & JANAK, E. A. (1989) Kirkpatrick's Levels of Training Criteria: Thirty Years Later. Personnel Psychology, 42, 331-342. ALLIGER, G. M., TANNENBAUM, S., BENNET, W., TRAVER, H. & SHOTLAND, A. (1997) A Meta-Analysis of the Relations among Training Criteria. Personnel Psychology, 50, 341-358. ALLINSON, C. W. & HAYES, J. (2000) Cross-national Differences in Cognitive Style: Implications for Management. International Journal of Human Resource Management, 11, 161-170. AMSTUTZ, P., HEDGES, R. & OTTO, K. (2004) Creating Interreality: The Virtual Object System. ANDERSON, J. R. (1982) Acquisition of Cognitive Skills. Psychology Review, 89, 369406. ANDERSON, P. H. & LAWTON, L. (1993) Dominant Personality Types and Total Enterprise Simulation Performance: A Follow-up Study. Developments in Business Simulation & Experiential Learning, 20, 1-3. ANDERSON, P. H. & LAWTON, L. (1997a) Demonstrating the Learning Effectiveness of Simulations: Where We are and Where We Need to Go. Developments in Business Simulation & Experiential Exercises. ANDERSON, P. H. & LAWTON, L. (1997b) Demonstrating the Learning Effectiveness of Simulations: Where We are and Where We Need to Go. Developments in Business Simulation & Experiential Exercises, 24, 68-73. ANTON, J. J. (1992) Seniority and Efficiency. Scandinavian Journal of Economics, 94, 425-442. ARGYRIS, C. (1980) Some Limitations of the Case Method: Experiences in a Management Development Program. Academy of Management Review, 5, 291298. ARGYRIS, C. (Ed.) (1999) Tacit Knowledge and Management, Mahwah, NJ, Erlbaum. ARGYRIS, C. & SCHON, D. (1978) Organizational Learning: A Theory Action Perspective, Reading, MA, Addison-Wesley. ARMSTRONG, S. J. & MAHMUD, A. (2004) The Influence of Learning Styles on the Creation of Actionable Knowledge in Public Sector Managers. Academy of 166

Management Best Conference Paper 2004. New Orleans, USA, Academy of Management. ATHANASOU, J. A. (1998) A Framework for Evaluating the Effectiveness of Technology-assisted Learning. Industrial and Commercial Training, 30, 96-103. ATHANASSIOU, N., MCNETT, J. M. & HARVEY, C. (2003) Critical Thinking in the Management Classroom: Bloom's Taxonomy as a Learning Tool. Journal of Management Education, 27, 533-555. AUSUBEL, D. (1978) In Defence of Advance Organizers: A Reply to the Critics. Review of Educational Research, 48, 251-257. AWONIYI, G. M., GRIEGO, O. V. & MORGAN, G. A. (2002) Person-environment Fit and Transfer of Training. International Journal of Training and Development, 6. BABB, E. M., LESLIE, M. A. & VAN SLYKE, M. D. (1966) The Potential of BusinessGaming Methods in Research. The Journal of Business, 39, 465-472. BAILEY, J. & WITMER, B. (1994) Proceedings of Human Factors & Ergonomics Society. BAKER, J. C., MAPES, J., NEW, C. C. & SZWEJCZEWSKI, M. (1997) A Hierarchical Model of Business Competence. Integrated Manufacturing Systems, 8, 265-272. BAL, S. (1995) The Interactive Manager, London, Kogan Page. BARHAM, K. & OATES, D. (1991) The International Manager, London, London Business Books. BARHAM, K. & WILLS, S. (1992) Management Across Frontiers: Identifying the Competences of Successful International Managers. Ashridge, Ashridge Research Group and the Foundation for Management Education. BARNETT, T. (1984) Evaluations of Simulations and Games: A Clarification. Simulation/Games for Learning, 14, 37-44. BARR, R. B. & TAGG, J. (1995) From Teaching to Learning: A New Paradigm for Undergraduate Education. Change, 13-25. BEARD, C. & WILSON, J. P. (2001) The Power of Experiential Learning: A Handbook for Trainers and Educators, London, Kogan Page. BEDINGHAM, K. (1997) Proving the Effectiveness of Training. Industrial and Commercial Training, 29, 88-91. BEE, R. & BEE, F. (1994) Training Needs Analysis and Evaluation, London, Institute of Personnel and Development. BEEHR, T. A., IVANITSKAYA, L., HANSEN, C. P., EROFEEV, D. & GUDANOWSKI, D. M. (2001) Evaluation of 360 degree feedback ratings: relationships with each other and with performance. Journal of Organizational Behavior, 22, 775-789. BELBIN, R. M. (1981) Management Teams, Why They Succeed or Fail, Oxford, Heinemann. BELBIN, R. M., ASTON, R. R. & MOTTRAM, R. D. (1976) Building Effective Management Teams. Journal of General Management, 3, 23-29. BERTSCHE, D., CRAWFORD, C. & MACADAM, S. E. (1996) Is Simulation Better Than Experience. The McKinsey Quarterly, 1, 15-22. BHATTACHARAYA, A. K. & GIBBONS, A. M. (1996) Strategy Formulation: Focussing on Core Competences and Processes. Business Change and Reengineering, 3, 47-55. BINSTED, D. (1988) The Key to the Use of Interactive Video for Management Education. IN H., M., RUSHBY, N. & BUDGETT, R. (Eds.) Aspects of Educational Technology Vol XXI Designing New Systems and Technologies for Learning. BIRCHALL, D., TAN, J. H. & GAY, K. (1996) Competences for International Management. Singapore Management Review, 18, 1-13. BLIGH, D. (1971) What's the Use of Lectures? Harmondsworth, Penguin. 167

BLOOM, B. S. (1956) Taxonomy of Educational Objectives, New York, David McKay Company. BOEHLE, S. (2005) Simulations: The Next Generation of E-learning. trainingmag.com. VNU. BOERLIJST, G. & MEIJBOOM, G. (1989) Matching the Individual and the Organisation. IN HERRIOT, P. (Ed.) Assessment and Selection in Organisations. London, John Wiley and Sons. BOYATZIS, R. E. (1982) The Competent Manager: A Model for Effective Performance, New York, NY, John Wiley. BRENENSTUHL, D. C. & CATALANELLO, R. F. (1977) An Analysis of the Impact upon the Learning Effectiveness of Traditional Instruction, Simulation, Gaming and Experiential Teaching Methodlogies: An Experimental Design. Computer Simulation and Learning Theory, 3, 463-473. BRENENSTUHL, D. C. & CATALANELLO, R. F. (1979) The Impact of Three Pedagogue Techniques on Learning. Journal of Experiential Learning and Simulation, 1, 211-225. BRINKERHOFF, R. O. (1987) Achieving Results from Training, San Francisco, JosseyBass. BRINKERHOFF, R. O. (1988) An Integrated Evaluation Model for HRD. Training & Development, 42, 66-68. BRUNER, J. (1973) Going Beyond the Information Given, New York, Norton. BURGOYNE, J. (1988) Management Development for the Individual and the Organisation. Personnel Management, 40-44. BURGOYNE, J. (1989) Creating the Managerial Portfolio: Building on Competency Approaches to Management Development. Management Education and Development, 20, 56-61. BURGOYNE, J. (1993) The Competence Movement: Issues, Stakeholders and Prospects. Personnel Management, 22, 6-13. BURGOYNE, J. (Ed.) (2002) Learning Theory and the Construction of Self: What kinds of people do we create through the theories of learning that we apply to their development? John Wiley & Sons Ltd. BURGOYNE, J. & COOPER, C. L. (1975) Evaluation Methodology. Journal of Occupational Psychology, 48, 53-62. BURGOYNE, J. & MUMFORD, A. (2002) Learning from the Case Method. European Case Clearing House. BURGOYNE, J. & REYNOLDS, M. (1997a) Introduction. IN BURGOYNE, J. & REYNOLDS, M. (Eds.) Management Learning; Integrating Perspectives in Theory and Practice. London, Sage Publications Ltd. BURGOYNE, J. & REYNOLDS, M. (Eds.) (1997b) Management Learning: Integrating Perspectives in Theory and Practice, London, Sage Publications. BURGOYNE, J. & SINGH, R. (1977) Evaluation of Training and Education: Micro and Macro Perspectives. Journal of European Industrial Training, 1, 17-21. BURGOYNE, J. & STEWART, R. (1977) Implicit Learning Theories as Determinants of the Effects on Management Development. Personnel Review, 6, 5-14. BURNS, A. V., GENTRY, J. W. & WOLFE, J. (Eds.) (1990) A Cornucopia of Considerations in Evaluating Effectiveness of Experiential Pedagogies, London, Kogan Page. BURR, V. (2003) Social Constructionism, London, Routeledge. BUTLER, R. J., MARKULIS, P. M. & STRANG, D. R. (1988) Where Are We? An Analysis of the Methods and Focus of the Research on Simulation Gaming. Simulation & Games, 19, 3-26. 168

BYRNE, E. T. & WOLFE, D. E. (1974) The Design, Conduct and Evaluation of a Eomputerized Management Game as a Form of Experiential Learning. Simulations, Gaming and Experiential Learning Techniques, 1, 22-30. CAMPBELL, D. T. & STANLEY, J. C. (1966) Experimental and Quasi-Experimental Design for Research, Chicago, Rand-McNally. CAMPBELL, J. P., DUNNETTE, M. D., LAWLER, E. E. & WEICK, K. E. (1970) Managerial Behaviour, Performance and Effectiveness, Maidenhead, McGrawHill. CANNING, R. (1990) The Quest for Competence. Industrial and Commercial Training, 122, 12-16. CANNON, H. M. & BURNS, A. V. (1999) A Framework for Assessing the Competencies Reflected in Simulation Performance. Developments in Business Simulation & Experiential Learning, 26. CARMINES, E. G. & ZELLER, R. A. (1990) Raliability and Validity Assessment. Sage University Paper. CARNEVALE, A. P. & SCHULZ, E. R. (1990) Evaluation Framework, Design, and Reports. Training and Development Journal, 44, 15-23. CELSIM (2003) Strategy at the Edge. Singapore, Corporate Edge Ltd. CERTO, S. C. (1976) The Experiential Exercise Situation: A Comment on Instructional Role and Pedagogy Evaluation. Academy of Management Review, 1, 113-116. CHANG, J. (2003) Strategic Management: An Evaluation of the Use of Three Learning Methods in Hong Kong. Developments in Business Simulation and Experiential Learning, 30, 146-151. CHAPMAN, K. J. & SORGE, C. L. (1999) Can a Simulation Help Achieve Course Objectives? An Exploratory Study Investigating Differences Among Instructional Tools. Journal of Education for Business, 225-230. CHEETHAM, G. & CHIVERS, G. (1996) Towards a Holistic Model of Professional Competence. Journal of European Industrial Training, 20, 20-30. CHMIELEWSKI, M. A. (1998) Computer Anxiety and Learner Characteristics: Their Role in the Participation and Transfer of Internet Training. Dissertation Abstracts International Section A: Humanities and Social Sciences, 59. CHONG, E. (1997) A Aomparative Study of the Managerial Competences and Performance of British Managers and Singaporean Public Sector Managers. Henley-on-Thames, Henley Management College. CLARK, C. S., DOBBINS, G. H. & LADD, R. T. (1993) Exploratory Field Study of Training Motivation. Group & Organization Management, 18, 292-307. CLARK, R. & CRAIG, T. (1992) Research and Theory on Multi-Media Learning Effects. IN GIARDINA, M. (Ed.) Interactive Learning Environments; Human Factors and Technical Consideration on Design Issues. Berlin, Springer-Verlag. CLEVELAND, G., SCHROEDER, R. G. & ANDERSON, J. C. (1989) A Theory of Production Competence. Decision Sciences, 20, 655-68. COCKERILL, T., HUNT, J. & SCHRODER, H. M. (1995) Managerial Competencies: Fact or Fiction? Business Strategy Review, 6, 1-12. COLLIN, A. (1989) Managers' Competence: Rhetoric, Reality and Research. Personnel Review, 18, 20-25. COLLINS, D. B. (2004) Performance-Level Evaluation Methods Used in Management Development Studies from 1986-2000. Human Resource Development Quarterly, 15, 217. CONSTABLE, C. & MCCORMICK, R. (1987) The Making of British Managers. A Report for the BIM and CBI into Management Training, Education and Development. London, British Institute of Management and Confederation of British Industry. 169

COOK, J. E. (1999) Assessing Effectiveness of an Experiential Oriented Course Over Time. Developments in Business Simulation and Experiential Learning, 26, 160164. COULSON-THOMAS, C. (1992) Creating the Global Company, Maidenhead, McGrawHill. CROMWELL, S. E. & KOLB, J. A. (2002) The Effect of Organisational Support, and Peer Support on Transfer of Training. IN EGAN, T. & LYNHAM, S. A. (Eds.) 2002 Academy of Human Resource Development Annual Conference. Bowling Green, OH, Academy of Human Resource Development. DAVIES, P. (2003) Simulation: Bringing e-learning to a New Level. ComputerUser.com. DE VOS, G. A. (1985) Dimensions of the self in Japanese culture. IN MARSELLA, G., DE VOS, G. A. & HSU, F. L. K. (Eds.) Culture and self: Asian and Western perspectives. New York, Tavistock. DEDE, C. (1996) Emerging Technologies in Distance Education for Business. Journal of Education for Business, 71, 197-205. DEDE, C. (1997) The Evolution of Constructivist Learning Environments. Educational Technology, 52, 54-60. DEWEY, J. (1938) Experience and Education, New York, MacMillan. DIXON, N. (1990) Evaluation: A Tool for Improving HRD Quality, San Diego, CA., University Association. DREW, S. A. W. & DAVIDSON, A. (1993) Simulation-based Leadership Development and Team Learning. The Journal of Management Development, 12, 39-52. DRUCKMAN, D. (1995) The Educational Effectiveness of Interactive Games. IN CROOKALL, D. & ARAI, K. (Eds.) Simulation and Gaming Across Disciplines and Cultures: ISAGA at a Watershed. Thousand Oaks, CA., Sage. DUBOIS, D. & ROTHWELL, W. J. (2004) Competency-Based or a Traditional Approach to Training? T&D, 58, 46-58. DUFF, A. (2000) Learning Styles of UK Higher Education Students. Bristol Business School Teaching and Research Review. DUFF, A. (2004) Understanding Acdemic Performance and Progression of First-year Accounting and Business Economics Undergraduates: The Role of Approaches to Learning and Prior Acdemic Achievement. Accounting Education, 13, 409. DULEWICZ, V. (1992) Assessment Centres as the Route to Competence. Personnel Management. DULEWICZ, V. (1995) A Validation of Belbin's Team Roles from 16PF and OPQ using Bosses' Ratings of Competence. Occupational and Organisational Psychology, 68, 81-100. DULEWICZ, V. & FLETCHER, C. A. (1982) The Relationship Between Previous Experience, Intelligence and Background Characteristics of Participants and their Performance in an Assessment Centre. Journal of Occupational Psychology, 55, 197-207. DULEWICZ, V. & HERBERT, P. (1992) Personality, Competences, Leadership Style and Managerial Effectiveness. Henley-on-Thames, Henley Management College. DULEWICZ, V. & HERBERT, P. (1996) General Management Competencies and Personality: A 7-year follow-up study. Henley-on-Thames, Henley Management College. EASTERBY-SMITH, M. (1980) The Evaluation of Management and Development: an Overview. Personnel Review, 10, 28-36. EASTERBY-SMITH, M. (1986) Models and 'Schools of Thought' in Evaluation. EASTERBY-SMITH, M. (1994) Evaluating Management Development, Training and Education, Aldershot, Gower. 170

EASTERBY-SMITH, M. & ASHTON, D. J. L. (1975) Using Repertory Grid Technique to Evaluate Management Training. Personnel Review, 4, 15-21. EASTERBY-SMITH, M., THORPE, R. & LOWE, A. (1991) Management Research: An Introduction. London, Sage. EASTERBY-SMITH, M., THORPE, R. & LOWE, A. (2002) Management Research: An Introduction, London, Sage. ELASHMAWI, F. & HARRIS, P. R. (1993) Multicultural Management: New Skills for Global Success, Houston, Gulf Publishing. ESKEW, R. K. & FALEY, R. H. (1988) Some Determinants of Student Performance in the First College-Level Financial Accounting Course. The Accounting Review, 63, 137-148. FARIA, A. J. & WELLINGTON, W. J. (2004) A Survey of Simulation Game Users, Former-users and Never-users. Simulation & Gaming, 35, 178-207. FEINSTEIN, A. H. (2004) A Model for Evaluating Online Instruction. Developments in Business Simulation and Experiential Learning, 31, 32-39. FEINSTEIN, A. H. & CANNON, H. M. (2001) Fidelity, verifiability, and validity of simulation: constructs for evaluation. Working Paper. Detroit, Wayne State University. FEINSTEIN, A. H. & CANNON, H. M. (2002) Constructs of Simulation Evaluation. Simulation and Gaming, 33, 425,440. FEINSTEIN, A. H., MANN, S. & CORSUN, D. L. (2002) Charting the experiential territory: clarifying definitions and uses of computer simulation, games, and role play. The Journal of Management Development, 21, 732-744. FILSTEAD, W. J. (1979) Qualitative Methods: A Needed Perspective in Evaluation Research. IN COOK, T. D. & REICHARDT, C. S. (Eds.) Qualitative and Quantitative Methods in Evaluation Research. Beverly Hills, Sage. FINN, R. (1993) A synthesis of current research on management competencies. Henley Working Paper Series. Henley-on-Thames, Henley Management College. FLETCHER, C. A. & DULEWICZ, V. (1984) An Empirical Study of a U.K. based Assessment Centre. Journal of Management Studies, 21, 83-97. FORD, J. K. & WEISSBEIN, D. A. (1997) Transfer of Training: An Update Review and Analysis. Performance Improvement Quarterly, 10, 22-41. FOX, S. (1998) Situated Learning Theory Versus Traditional Cognitive Learning Theory: Why Management Education Should Not Ignore Management Learning. Systems Practice, 10, 727-748. FRIPP, J. (1993) Learning Through Simulations: A Guide to the Design and Use of Simualtions in Business and Education, New York, McGraw-Hill. GAGE, N. L. & BERLINER, D. C. (1998) Educational Psychology, Boston, Houghton Mifflin Company. GAGNE, R. M. (1984) Learning outcomes and their effects: Useful categories of human performance. American Psychologist, 39, 377-385. GARRIS, R., AHLERS, R. & DRISKELL, J. E. (2002) Games, motivation, and learning: a research and practice model. Simulation & Gaming, 33, 441-467. GAY, K. (1995) Competences for International Management. Henley Management College, Brunel University. GEBER, B. (1994) Let the games begin. Training, 31, 10-15. GIBBS, G. (1988) Learning by Doing: a guide to teaching and learning methods, London, Kogan Page. GLASER, B. G. (1992) Basics of Grounded Theory Analysis: Emergence versus Forcing, Mill Valley, CA, Sociological Press. GOPHER, D., WEIL, M. & BAREKET, T. (1994) Transfer of skill from a computer game trainer to flight. Human Factors, 36, 387-405. 171

GOPINATH, C. & SAWYER, J. E. (1999) Exploring the learning from an enterprise simulation. Journal of Management Development, 18, 477-489. GOSEN, J. & WASHBUSH, J. (2004) A review of scholarship on assessing experiential learning effectiveness. Simulation & Gaming, 35, 270-293. GOSENPUD, J. (Ed.) (1990) Evaluation of Experiential Learning, London, Kogan Page. GOSENPUD, J. & WASHBUSH, J. (1992) The influence of Myers-Briggs type and group dynamics factors on total enterprise performance. Developments in Business Simulation & Experiential Learning, 19, 64-67. GREDLER, M. E. (1996) Educational Games and Simulations: A Technology in search of a (Research) Paradigm. IN JONASSEN, D. H. (Ed.) Handbook of Research for Educational Communications and Technology. New York, Simon & Schuster Macmillan. GUBA, E. G. & LINCOLN, Y. S. (1989) Fourth Generation Evaluation, London, Sage. HAIR, J. F., ANDERSON, R. E., TATHAM, R. L. & BLACK, W. C. (1998) Multivariate Data Analysis, Upper Saddle River, NJ, Prentice Hall. HALL, D. T. (1986) Career Development in Organisations, San Francisco, CA, JosseyBass. HAMBLIN, A. C. (1974) Evaluation and control of training, London, McGraw Hill. HANNAFIN, M. J. (1992) Emerging technologies, ISD, and learning environments: critical perspectives. Educational Technology Research and Development, 40, 4963. HANNAFIN, M. J., HANNAFIN, K. M., HOOPER, S. R., RIEBER, L. P. & KINI, A. S. (1996) Research on and research with emerging technologies. IN JONASSEN, D. H. (Ed.) Handbook of Research for Educational Communications and Technology. New York, Simon and Schuster. HARDY, C. (1996) Understanding Power: Bringing about Strategic Change. British Journal of Management, 7. HASEMAN, W. D., NUIPOLATOGLU, V. & RAMAMURTHY, K. (2002) An Empirical Investigation of the Influences of the Degree of Interactivity on User-outcomes in a Multimedia Environment. Information Resources Management Journal, 15, 31-48. HAY/MCBER (1997) MCQ Profile and Interpretive Notes, TRG Hay/McBer. HAYES, K. & RICHARDSON, J. T. E. (1995) Gender, subject and context as Determinants of Approaches to Studying in Higher Education. Studies in Higher Education, 20, 19-31. HAYS, R. T. & SINGER, M. J. (1989) Simulation fidelity in training systems design: Bridging the gap between reality and training., New York, Springer-Verlag. HENKE, H. (2001) Learning theory: applying Kolb's learning style inventory with computer based training. Project Paper. HEROLD, D. M., DAVIS, W., FEDOR, D. B. & PARSONS, C. K. (2002) Dispositional Influences on Transfer of Learning in Multistage Training Programs. Personnel Journal, 55, 851-869. HESSELING, P. (1966) Strategy of Evaluation Research in the Field of Supervisory and Management Training, Anssen, Van Gorcum. HIGGS, M. (1996) A Comparison of Myers Briggs Type Indicator Profiles and Belbin Team Roles. Henley-on-Thames, Henley Management College. HIGGS, M. (1999) Teams and Team Working: What do we know? (A Literature Review). Henley Working Paper Series. Henley-on-Thames, Henley Management College. HIGGS, M., PLEWNIA, U. & PLOCH, J. (2003) Influence of Team Composition on Team Performance and Dependence on Task Complexity. Henley Working Paper Series. Henley-on-Thames, Henley Management College. HIGGS, M. & ROWLAND, D. (2001) Developing change leaders: Assessing the impact of a development programme. Journal of Change Management, 2, 47-64. 172

HIRSH, W. (1989) Defining Management Skills. Brighton, University of Sussex, Institute of Manpower Studies. HO, D. Y. F. (1976) On the Concept of Face. American Journal of Sociology, 81, 867890. HOBERMAN, S. & MAILICK, S. (1992) Experiential Management Development, New York, Quorum. HODGKINSON, G. P. & SADLER-SMITH, E. (2003) Complex or Unitary: A Critique and Empirical Re-assessment of the Allinson-Hayes Cognitive Style Index. Journal of Occupational and Organizational Psychology, 76, 243-269. HOFSTEDE, G. (1980) Culture's consequences: international differences in work-related values, Beverly Hills, Sage. HOFSTEDE, G. (1991) Cultures and organisations: software of the mind, London, McGraw-Hill. HOLMAN, D., PAVLICA, K. & THORPE, R. (1997) Rethinking Kolb's theory of experiential learning in management education: The Contribution of Social Constructionism and Activity Theory. Management Learning, 28, 135-148. HOLTON, E. F., III (1996) The Flawed Four-Level Evaluation Model. Human Resource Development Quarterly, 7, 5-21. HOLTON, E. F., III, BATES, R. & RUONA, W. E. A. (2000) Development and Validation of a Generalized Learning Transfer Climate Questionnaire. Human Resource Development Quarterly, 11, 333-360. HOLTON, E. F., III, CHEN, H. & NAQUIN, S. S. (2001) An Examination of Learning Transfer System Characteristics across Organisational Settings. IN ALIAGA, O. (Ed.) Proceedings of the 2001 Academy of Human Resource Development Conference. Baton Rouge, LA, Academy of Human Resource Development. HONEY, P. & MUMFORD, A. (1982) The Manual of Learning Styles, Maidenhead, Peter Honey. HONEY, P. & MUMFORD, A. (1992) The Manual of Learning Styles, Maidenhead, Peter Honey. HOSKINS, S. L. & VAN HOOFF, J. C. (2005) Motivation and Ability: Which Students Use Online Learning and What Influence Does it have on their Achievement? British Journal of Educational Psychology, 36, 177-192. HOULDSWORTH, E. (1994) Multimedia in Education: Theoretical Framework and Methodological Choices. IN HIGGS, M. (Ed.) Henley Working Paper. Henley-onThames, Henley Management College. HOULDSWORTH, E. (2004) The Learning Manager. IN REES, D. & MCBAIN, R. (Eds.) People Management: Challenges and Opportunities. Palgrave Macmillan. HUCZYNSKI, A. A. (2001) Encyclopedia of Development Methods, Aldershot, Gower. HUCZYNSKI, A. A. & LEWIS, J. W. (1980) An Empirical Study into the Learning Transfer Process in Management Training. Journal of Management Studies, 17, 227-240. HUTTON, J. (1988) The world of the international manager, Oxford, Phillip Allan. HWANG, K. K. (1987) Face and Favor: The Chinese Power Game. American Journal of Sociology, 92, 944-974. HYMAN, R. T. (1978) Simulation Gaming for Values Education: The Prisoner's Dilemma, New Brunswick, NJ, University Press of America. IMPARTA (2003) Strategy CoPilot. London, Imparta Ltd. IVANCEWICH, J. M. & MATTESON, M. T. (1996) Organisational Behaviour and Management, London, Irwin. JACOBS, R. (1989) Getting the measure of managerial competence. Personnel Management, 21, 32-7. 173

JENKINS, D., SIMMONS, H. & WALKER, R. (1981) Thou Nature are my Goddess. Naturalistic Enquiry in Educational Evaluation. Cambridge Journal of Education, 11, 169-89. JENNINGS, D. (2000) Strategic Management: An Evaluation of the Use of Three Learning Methods. ABSEL: Developments in Business Simulation & Experiential Learning, 27, 20-27. KAUFMAN, F. L. (1976) An empirical study of the usefulness of a computer-based business game. Journal of Educational Data Processing, 13, 13-22. KELNER, S. P. (2001) A Few Thoughts on Executive Competency Convergence. Center for Quality of Management Journal, 10, 67-71. KELNER, S. P. (2005) Email response to questions on technical data for MCQ. IN KENWORTHY, J. M. (Ed.) Singapore. KEMMIS, S., ATKIN, R. & WRIGHT, E. (1977) How Do Students Learn? Working Papers on Computer Assisted Learning. Norwich, Centre for Applied Education in Research, UEA. KENWORTHY, J. M. & WONG, A. (2003) A Study of the Attributes of Managerial Effectiveness in Singapore: Implications for a Competency model for Managers in Singapore. Singapore, Corporate Edge White Paper. KENWORTHY, J. M. & WONG, A. (2005) Developing Managerial Effectiveness: Assessing and Comparing the Impact of Development Programmes using a Management Simulation or a Management Game. IN LEDMAN, R. (Ed.) ABSEL National Conference. Orlando, FL., ABSEL. KEYS, J. B. (1977) The management of learning grid for management development. Academy of Management Review, 2, 289-297. KEYS, J. B., WELLS, R. A. & EDGE, A. G. (1994) The multinational management game: a simuworld. The Journal of Management Development, 13, 26-37. KEYS, J. B. & WOLFE, J. (1990) The role of management games and simulations in education and research. Journal of Management, 16, 307-336. KICKUL, G. (2001) Antecedents of work team performance in a business simulation, personality and group interaction. Developments in Business Simulation & Experiential Learning, 28, 128-136. KIM, B., WILLIAMS, R. & DATTILO, J. (2002) Students' perception of interactive learning modules. Journal of Research on Technology in Education, 34, 453-473. KIM, J. S. & ARNOLD, P. (1992) Manufacturing competence and business performance: a framework and empirical analysis. International Journal of Operations and Production Management, 13, 4-25. KIRKPATRICK, D. (1959/60) Techniques for evaluating training programs: Parts 1 to 4. Journal of the American Society for Training and Development, November, December, January and February. KIRKPATRICK, D. L. (1994) Evaluating Training Programs: The Four Levels, San Francisco, Berret-Koehler. KIRKPATRICK, D. L. (1998a) Another Look at Evaluating Training Programs, Alexandria, VA., American Society for Training and Development. KIRKPATRICK, D. L. (1998b) Evaluating Training Porgrams: Evidence vs. Proof. IN KIRKPATRICK, D. L. (Ed.) Another Look at Evaluating Training Programs. Alexandria, VA, ASTD. KIRTON, M. J. (1994) Adaptors and Innovatrors at Work. IN KIRTON, M. J. (Ed.) Adaptors and Innovators: Styles of Creativity and Problem Solving. London, Routeledge. KNOWLES, M. (1970) The Modern Practice of Adult Education, New York, Association Press. 174

KNOWLES, M. (Ed.) (1996) Adult Learning, New York, McGraw-Hill. KNOWLES, M. S., HOLTON, E. F. & SWANSON, R. A. (1998) The Adult Learner, Houston TX, Butterworth Heinemann. KOCH, J. V. & CEBULA, R. J. (1994) In Search of Excellent Management. Journal of Management Studies, 31, 681-99. KOLB, D. A. (1976) Management and the learning process. California Management Review, 18, 21-31. KOLB, D. A. (1981) Experiential learning theory and the learning style inventory: a reply to Freedman and Stumpf. Academy of Management Review, 6, 289-296. KOLB, D. A. (1984) Experiential Learning: Experience as a source of learning and development, Englewood Cliffs, NJ, Prentice Hall. KOLB, D. A. (1999) Learning Style Inventory, HayGroup. KOLB, D. A., BOYATZIS, R. E. & MAINEMELIS, C. (2000) Experiential Learning Theory: Previous Research and New Directions. IN STERNBERG, R. J. & ZHANG, L. F. (Eds.) Perspectives on cognitive, learning, and thinking styles. NJ, Lawrence Erlbaum. KRAIGER, K., FORD, J. K. & SALAS, E. (1993) Application of cognitive, skill-based, and affective theories of learning outcomes to new methods of training evaluation. Journal of Applied Psychology, 78, 311-328. KUNNATH, G. & SEDICK, R. (2001) Boo.com: The Path to Failure. ECCH - Garduate School of Management University of Western Australia. LANE, D. C. (1995) On a Resurgence of Management Simulations and Games. The Journal of the Operational Research Society, 46, 604-626. LAVE, J. (1998) The Practice of Learning. IN CHAIKLIN, S. & J., L. (Eds.) Understanding Practice: Perspectives on Activity and Context. Cambridge, Cambridge University Press. LEE MEI CHING, W., CHIAN, K. Y. & TAN, Y. H. (2002) Managerial competencies in a knowledge-based economy: Its implications for human resource development. Singapore, Nanyang Technological University, STADA. LEWIN, K. (1951) Field Theory in Social Sciences, New York, Harper and Row. LIM, D. H. & JOHNSON, S. D. (2002) Trainee Perceptions of Factors that Influence Learning Transfer. International Journal of Training and Development, 6, 36-48. LINSTEAD, S. (1991) Developing management meta-competence: can distance learning help? Journal of European Industrial Training, 6, 17-27. LOO, R. (2002) A meta-analytic examination of Kolb's learning style preferences among business majors. Journal of Education for Business, 77, 252-256. LUNDY, J. (2003) E-learning simulation: putting knowledge to work. Gartner U.S. Symposium/Itxpo. LUNDY, J., LOGAN, D. & HARRIS, K. (2002) Simulation May be your E-Learning 'Killer Application'. Gartner. LUTHANS, F., ROSENKRANTZ, S. A. & HENNESSEY, H. W. (1985) What do successful managers really do? An observation study of managerial activities. The Journal of Applied Behavioural Science, 21, 255-70. MABEY, C., TOPHAM, P. J. & ROLAND KAYE, G. (1998) Computer-based Courseware: a Comparative Review of the Learner's Experience. Accounting Education, 7, 51-65. MACDONALD-ROSS, M. (1973) Behavioural Objectives: A Critical Review. Instructional Science, 2, 1-52. MACDONALD, B., ATKIN, R., JENKINS, D. & KEMMIS, S. (1977) Computer Assisted Learning: its Educational Potential. IN HOOPER, R. (Ed.) The National Development Programme in Computer Assisted Learning. The Final Report of the Director. London, Council for Educational Technology. 175

MAINEMELIS, C., BOYATZIS, R. E. & KOLB, D. A. (2002) Learning Styles and Adaptive Flexibility: Testing Experiential Learning Theory. Management Learning, 33, 5-33. MANTYLA, K. & WOODS, J. A. (2001) Evaluating Program Success. IN MANTYLA, K. & WOODS, J. A. (Eds.) The 2001/2002 ASTD Distance Learning Yearbook. New York, McGraw-Hill. MARGEISON, C. & MCCANN (1985) Team Management Index and Description of Team Roles, Bradford, MCB University Press. MARSICK, V. J. & WATKINS, K. E. (1990) Informal and Incidental Learning in the Workplace, London, Routeledge & Kegan Paul. MASLOW, A. H. (1954) Motivation and Personality, New York, Harper and Row. MAY, T. (1993) Social Research: Issues, Methods and Process, Buckingham, Open University Press. MCAULEY, J. (1994) Exploring Issues in Culture and Competence. Human Relations, 47, 417-30. MCBER (1997) Managerial Competency Questionnaire, London, TRG Hay/McBer. MCCLELLAND, D. C. (1973) Testing for Competence Rather Than Intelligence. American Psychologist, 28, 1-14. MCKENNA, S. (1996) Evaluating IMM: Issues for Researchers. Charles Stuart University. MCKENNEY, J. L. (1962) An Evaluation of Business Game in an MBA Curriculum. Journal of Business, 35, 278-286. MCKENNEY, J. L. (1963) An Evaluation of a Decision Simulation as a Learning Environment. Management Technology, 3, 56-67. MCKENNEY, J. L. (1967) Simulation Gaming for Management Development, Boston, MA., The Division of Business, Harvard College. MCLAUGHLIN, H. & THORPE, R. (1993) Action Learning - A Paradigm in Emergence: the Problems Facing a Challenge to Traditional Management Education and Development. British Journal of Management, 4, 19-27. MILES, R. H. & RANDOLPH, W. A. (1985) The Organisation Game: A Simulation, Glenview, Il, Scott, Foresman and Company. MILES, W. G., BIGGS, W. D. & SCHUBERT, J. N. (1986) Students Perceptions of Skill Acquisition Through Cases and a General Management Simulation: A Comparison. Simulation & Games, 17, 7-24. MILLMAN, I. J. (2000) Flight Simulators for Business'. 7th Annual EDINEB Conference: Educational Innovation in Economics and Business. Newport Beach, USA. MITCHELL, R. C. (2004) Combining Cases and Computer Simulations in Strategic Management Courses. Journal of Education for Business, 79, 198-204. MORSE, K. (2001) Assessing the Efficacy of Experiential Learning in a Multicultural Environment. Developments in Business Simulation & Experiential Learning, 28, 153-159. MOSIER, N. R. (1990) Financial Analysis: The Methods and Their Application to Employee Training. Human Resource Development Quarterly, 1, 45-63. MOWBRAY, C. T., HOLTER, M. C., TEAGUE, G. B. & BYBEE, D. (2003) Fiedlity Criteria: Development, Measurement and Validation. American Journal of Evaluation, 24, 315-340. MUMFORD, A. (Ed.) (1994) Effectiveness in Management Development, Aldershot, Gower. MYERS, I. B. & MCCAULLEY, M. H. (1985) A Guide to the Development and Use of the Myers-Briggs Type Inidcator, Palo Alto, Consulting Psychologists Press. MYERS, I. B. & MCCAULLEY, M. H. (1989) A Guide to the Development and Use of the Myers-Briggs Type Inidcator, Palo Alto, Consulting Psychologists Press. 176

MYERS, I. B. & MYERS, P. B. (1980) Gifts Differing, Palo Alto, Consulting Psychologists Press. NAQUIN, S. S. & HOLTON, E. F., III (2003) Motivation to Improve Work through Learning in Human Resource Development. Human Resource Development International, 6, 355-370. NONAKA, I. & TAKEUCHI, H. (1995) The Knowledge-creating Company, Oxford, Oxford University Press. NORDHAUG, O. & GRONHAUG, K. (1994) Competencies as Resources in Firms. The International Journal of Human Resource Management, 5, 89-106. O'GORMAN, C. (1997) Cooleys Distillery plc: A New "Spirit" in the World Whiskey Industry. Dublin, University College. O'ROURKE, J. (2003) Management Communication: A Case Analysis Approach, Upper Saddle River, NJ, Prentice Hall. PARLETT, M. & HAMILTON, D. (1987) Evaluation as Allumination: A New Approach to the Study of Innovatory Programmes. IN MORPHY, R. & TORRANCE, H. (Eds.) Evalauting Education: Issues and Methods. Milton Keynes, Open University Press. PARTRIDGE, S. E. & SCULLY, D. (1979) Cases Versus Gaming. Management Education and Development, 10, 172-180. PATTON, M. Q. (1978) Utilization-Focussed Evaluation, Beverly Hills, Sage. PATTON, M. Q. (1990) Qualitative Evaluation and Research Methods, London, Sage Publications. PATZ, A. L. (1990) Group Personality Composition and Total Enterprise Simulation Performance. Developments in Business Simulation & Experiential Exercises, 17, 132-137. PATZ, A. L. (1992) Personality Bias in Total Enterprise Simulations. Simulation & Gaming, 23, 45-76. PEGDEN, C. D., SHANNON, R. E. & SADOWSKI, R. P. (1995) Introduction to Simulation using SIMAN, Hightstown, NJ, McGraw-Hill. PHILLIPS, J. (1991) Handbook of Evaluation and Measurement Methods, London, Gulf Publishing. PHILLIPS, J. (Ed.) (1997) Measuring Return on Investment, Alexandria, VA., ASTD. PHILLIPS, J. J. (Ed.) (1998) ROI: The Search for Best Practices, Alexandria VA, American Society for Training & Development. PHILLIPS, P. P. (1999) The ROI Process: Trends and Issues. IN PHILIPS, J. (Ed.) Measuring Return on Investment. Alexandria, VA., ASTD. PIERFY, D. A. (1977) Comparative Simulation Game Research: Stumbling Blocks and Steppingstones. Simulation and Gaming, 8, 255-68. PINTRICH, P. R., CROSS, D. R., KOZMA, R. B. & MCKEACHIE, W. J. (1986) Instructional Psychology. Annual Review of Psychology, 37, 611-651. PLATTS, K. W. & GREGORY, M. J. (1990) Manufacturing Audit in the Process of Strategy Formulation. International Journal of Operations and Production Management, 10. PRAHALAD, C. K. & HAMAL, G. (1990) The Core Competences of the Corporation. Harvard Business Review, 63, 79-91. PRENSKY, M. (2000) Digital Game Based Learning, McGraw Hill. PSYFACTOR (2005) Management Development in Chemical Manufacturing. Psyfactor. RACKHAM, N. (1973) Recent Thoughts on Evaluation. Industrial and Commercial Training, 5, 454-61. RAIA, A. P. (1966) A Study of the Educational Value of Management Games. Journal of Business, 39, 339-352. 177

RANDEL, J. M., MORRIS, B. A., WETZEL, C. D. & WHITEHALL, B. V. (1992) The Effectiveness of Games for Educational Purposes: A Review of Recent Research. Simulation & Gaming, 23, 261-276. RAUSCH, E., HALFHILL, S. M., SHERMAN, H. & WASHBUSH, J. B. (2001) Practical Leadership in Management Education for Effective Strategies in a Rapidly Changing World. The Journal of Management Development, 20, 245-257. REDDIN, W. J. (1970) Managerial Effectiveness, London, McGraw Hill. REEVES, T. (1993) Research Support for Interactive Multimedia: Existing Foundations and New Directions. IN LATCHEM, C., WILLIAMSON, J. & HENDERSONLANCETT, L. (Eds.) Interactive Multimedia. London, Kogan Page. REINGOLD, J. (1999) Exec Ed: Learning to Lead. Business Week. REMENYI, D., WILLIAMS, B., MONEY, A. & SWARTZ, E. (1998) Doing Research in Business and Management, London, Sage Publishing. REVANS, R. (1983) The ABC of Action Learning, Bromley, Chartwell-Bratt. REVANS, R. W. (1971) Developing Effective Managers, London, Longman. REVANS, R. W. (1980) Action Learning, London, Blond and Briggs. REYNOLDS, M. (1997a) Learning Styes: A Critique. Management Learning, 28, 115134. REYNOLDS, M. (1997b) Towards a Critical Management Pedagogy. IN BURGOYNE, J. & REYNOLDS, M. (Eds.) Management Learning: Integrating perspectives in theory and practice. London, Sage Publications Ltd. REYNOLDS, M. & SNELL, R. (1988) Contribution to Development of Management Competence, Sheffield, Manpower Services Commission. RICCI, K., SALAS, E. & CANNON-BOWERS, J. A. (1996) Do Computer-based Games Facilitate Knowledge Acquisition and Retention? Military Psychology, 8, 295-307. RIDLEY, M., LASCHINGER, H. & GOLDENBERG, D. (1995) The Effect of a Senior Preceptorship on the Adaptive Competencies of Community College Nursing Students. Journal of Advanced Nursing, 22, 58-65. ROBOTHAM, D. & JUBB, R. (1996) Competences: Measuring the Unmeasurable. Management Development Review, 9, 25-29. ROMME, A. G. L. (2003) Learning Outcomes of Microworlds for Management Education. Management Learning, 34, 51-61. ROSE, H. (1995) Assessing Learning in VR: Towards Developing a Paradigm. HITL. ROSS, S. & MORRISON, G. R. (2003) Experimental Research Methods. IN JONASSEN, D. H. (Ed.) Handbook of Research Methods in Educational Communications and Technologies. Second ed., AECT. ROWE, C. (1995) Clarifying the Use of Competence and Competency Models in Recruitment, Selection and Staff Development. Industrial and Commercial Training, 27, 12-17. RUONA, W. E. A., LEIMBACH, M., HOLTON, E. F., III & BATES, R. (2002) The Relationship between Learner Utility Reactions and Predicted Learning Transfer among Trainees. International Journal of Training and Development, 6, 218-228. RUSS-EFT, D. (2002) A Typology of Training design and Work Environment Factors Affecting Workplace Learning and Transfer. Human Resource Development Review, 1, 45-65. RUSS-EFT, D. & PRESKILL, H. (2001) Evaluation in Organizations A Systematic Approach to Enhancing Learning, Performance, and Change, Cambridge, MA., Perseus Publishing. RUSSELL, S. (1999) Evaluating Performance Interventions. Info-line. SACKETT, P. R. & MULLEN, E. J. (1993) Beyond Formal Experimental Design: Towards an Expanded View of the Training Evaluation Process. Personnel Psychology, 46, 613-627. 178

SADLER-SMITH, E. (1996) Learning Styles: A Holistic Approach. Journal of European Industrial Training, 20, 29-39. SADLER-SMITH, E. (2001) A Reply to Reynold's Critique of Learning Style. Management Learning, 32, 291-304. SALES, E. & CANNON-BOWERS, J. A. (2001) The Science of Training: A Decade of Progress. Annual Review of Psychology, 52, 471-499. SALOMON, G. (Ed.) (1993) On the Nature of Pedagogic Computer Tools: The Case of the Writing Partner, Hillsdale, NJ., Erlbaum. SALZMAN, M. C., DEDE, C., R., B. L. & CHEN, J. (1999) A Model for Understanding How Virtual Reality Aids Complex Conceptual Learning. Presence: Teleoperators and Virtual Environments, 8, 293-316. SARAWANO, R. (1993) Assessment of Managerial Competencies and Potential. Henleyon-Thames, Henley Management College/Brunel University. SAVVAS, M., EL-KOT, G. & SADLER-SMITH, E. (2001) Comparitive Study of Cognitive Styles in Egypt, Greece, Hong Kong and the UK. International Journal of Training and Development, 5, 64-74. SCHANK, R. (1997) Virtual Learning: A Revolutionary Approach to Building a High Skilled Workforce, New York, McGraw-Hill. SCHANK, R. (2002) Designing World-Class E-Learning, New York, McGraw-Hill. SCHNEIDER, M. (2001) A new test for MBA wannabes? The GMAT is good at evaluating analytical skills. But what about “common sense” or right-brain skills? Enter the SIA. BusinessWeek Online. SCHNEIER, C. E. & BEATTY, R. W. (1977) Predicting Participants' Performance and Reactions in an Experiential Learning Setting: An Empirical Investigation. New Horizons in Simulation Games and Experiential Learning, 4, 291-298. SCHON, D. (1983) The Reflective Practitioner: How Professionals Think in Action, London, Maurice Temple Smith. SCHON, D. (1987) Educating the Reflective Practitioner, San Francisco, CA, JosseyBass. SCHRODER, H. M. (1989) Managerial Competence: The Key to Excellence, Iowa, Kendall/Hunt. SCHUMANN, P. L., ANDERSON, P. H., SCOTT, T. W. & LAWTON, L. (2001) A Framework for Evaluating Simulations as Educational Tools. Developments in Business Simulation & Experiential Learning, 28, 215-220. SCRIVEN, M. (1972) Pros and Cons about Goal-Free Evaluation. Evaluation Comment, 3, 1-4. SENGE, P. (1990) The Fifth Discipline, New York, Doubleday Currency. SKINNER, B. F. (1950) Are Theories of Learning Necessary? Psychology Review, 57, 193-216. SNYDER, L. T. & VAUGHAN, M. J. (1996) Multimedia & learning: Where's the connection? Developments in Business Simulation & Experiential Exercises, 23, 179-180. SPENCER, L. M. & SPENCER, S. (1993) Competence at Work: Models for Superior Performance, New York, John Wiley & Sons. STAKE, R. E. (1980) Responsive Evaluation. University of Illinois. STALK, G., EVANS, P. & SHULMAN, L. E. (1992) Competing on Capabilities: The New Rules of Corporate Strategy. Harvard Business Review, 70, 57-69. STANNEY, K., MORRANT, R. & KENNEDY, R. (1998) Human Factor issues in Virtual Environments. Presence: Teleoperators and Virtual Environments, 7, 327-351. STERNBERG, R. J. (1997) Thinking Styles, Cambridge, Cambridge University Press. STRAUSS, A. L. (1987) Qualitative Analysis for Social Scientists, Cambridge, Cambridge University Press. 179

SUMMERS, G. J. (2004) Today's Business Simulation Industry. Simulation & Gaming, 35, 208-241. SUQRUE, B. & KIM, K. (2004) 2004 State of the Industry Report, ASTD. SUSSMAN, N. M. & TYSON, D. H. (2000) Sex and Power: Gender Differences in Computer-Mediated Interactions. Computers in Human Behaviour, 16, 381-394. SWANSON, R. A. & HOLTON, E. F., III (1999) Results: How to Assess Performance, Learning in the Art of Systematic Change, San Francisco, Berret-Koehler Publications, Inc. SYMONS, J. (1996) What Type of Learning? A Review of Learning Approaches in the Context of the Design and Delivery of Academic Management Development Programmes in the UK. Henley-on-Thames, Henley Management College. TAMPOE, M. (1994) Exploiting the Core Competences of Your Organization. Long Range Planning, 27, 66-77. TANNENBAUM, S. I., MATHIEU, J. E., SALAS, E. & CANNON-BOWERS, J. A. (1991) Meeting Trainees' Expectations: The Influence of Training Fulfillment on the Development of Commitment, Self-efficacy, and Motivation. Journal of Applied Psychology, 76, 759-769. TANNENBAUM, S. I. & WOODS, S. B. (1992) Determining a Strategy for Evaluating Training: Operating within Organizational Constraints. Human Resource Planning, 15, 63-81. TEACH, R., D. & GIOVAHI, G. (Eds.) (1988) The Role of Experiential Learning and Simulation in Teaching Management Skills. TEACH, R. D. (Ed.) (1989) Using Forecast Accuracy as a Measure of Success in Business Simulations. THOMAS, R., CAHILL, J. & SANTILLI, L. (1997) Using an Interactive Computer Game to Increase Skill and Self-efficacy Regarding Safer Sex Negotiation: Field Test Results. Health Education & behavior, 24, 71-86. THOMPSON, C., KOON, E., WOODWELL, W. H. J. & BEAUVAIS, J. (2002) Training for the Next Economy: An ASTD State of the Industry Report on trends in employer-provided training in the United States. Alexandria, VA., ASTD. THOMPSON, D. (1980) Adaptors and Innovators: A Replication Study on Managers in Sinagpore and Malaysia. Psychological Reports, 47, 383-7. TIMES, F. (1998) NTT DoCoMo "Questions hang over record-breaking issue". TOLMAN, E. C. (1932) Purposive Behaviour in Animals and Men, New York, AppletonCentury-Crofts. TOPHAM, P. J. (1990) Humanistic Computing in the Management Learning Field. University of Lancaster. TOTTY, M. (2005) Business Solutions: Better Training Through Gaming. The Wall Street Journal. WSJ.com ed. TOWLER, A. J. & DIPBOYE, R. L. (2003) Development of a Learning Style Orientation Measure. Organisational Research Methods, 6, 216-235. TROMPENAARS, F. & HAMPDEN-TURNER, C. (1993) Riding the Waves of Culture: Understanding Cultural Diversity in Business, Nicholas Brealey Publishing. TSANG, E. W. K. (1997) Learning from Joint Venturing Experience: the case for foreign direct investment by Singapore companies in China. University of Cambridge. VICKERY, S. K. (1991) Theory of Production Competence Revisited. Decision Sciences, 22, 635-43. VYGOTSKY, L. S. (1978) Mind in Society, Cambridge, Mass, Harvard University Press. WARR, P. B., ALLAN, C. & BIRDI, K. (1999) Predicting Three Levels of Training Outcome. Journal of Occupational and Organizational Psychology, 72, 351-375. WARR, P. B., BIRD, M. W. & RACKHAM, N. (1970) Evaluation of Management Training, Aldershot, Gower. 180

WARR, P. B. & BUNCE, D. (1995) Trainee Characteristics and the Outcomes of Open Learning. Personnel Psychology, 48, 347-375. WASHBUSH, J. & GOSEN, J. (2001) Learning in Total Enterprise Simulations. Simulation & Gaming, 32, 281-296. WATKINS, R., LEIGH, D., FOSHAY, R. & KAUFMAN, R. (1998) Kirkpartick Plus: Evaluation and Continuous Improvement with a Community Focus. Educational Technology Research and Development, 46, 90-96. WELLINGTON, W. J. & FARIA, A. J. (1992) An Investigation of the Relationship Between Team Cohesion, Player Attitude, Performance Attitude on Player Performance. Developments in Business Simulation & Experiential Exercises, 19, 184-189. WHITE, B. (1984) Designing Computer Games to Help Physics Students Understand Newton's Laws of Motion. Cognition and Instruction, 1, 69-108. WHITEHALL, B. & MCDONALD, B. (1993) Improving Learning Persistence of Military Personnel by Enhancing Motivation in a Technical Training Program. Simulation & Gaming, 24, 294-313. WHITELOCK, D., BIRNA, P. & HOLLAND, S. (1996) Proceedings. IN EDITIONS, C. (Ed.) European Conference on AI in Education. Lisbon Portugal, Colibri Editions. WIEBE, J. H. & MARTIN, N. J. (1994) The Impact of a Computer-based Adventure Game on Achievement and Attitudes in Geography. Journal of Computing in Childhood Education, 5, 61-71. WILLE, E. (1989) Managerial Competencies and Management Development. Training Officer, 25, 326-328. WIMER, S. (2002) The Dark Side of 360-degree. Training & Development, 37-42. WINN, W. & SNYDER, D. (1996) Cognitive Perspectives in Psychology. IN JONASSEN, D. H. (Ed.) Handbook of Research for Educational Communications and Technology. First ed., Simon and Schuster. WINTERTON, J. & WINTERTON, R. (1999) Developing Managerial Competence, London, Routledge. WITMER, B. & SINGER, M. J. (1994) Measuring Immersion in Virtual Environments. ARI Technical Report 1014. WOLFE, J. (1985) The Teaching Effectiveness of Games in Collegiate Business Courses: A 1973-1983 Update. Simulation & Games, 16, 251-288. WOLFE, J. (Ed.) (1990) The Evaluation of Computer-based Business Games: Methodology, Findings, and Future Needs, USA, Association for Business Simulation and Experiential Learning (ABSEL). WOLFE, J. & CROOKALL, D. (1998) Developing a Scientific Knowledge of Simulation/Gaming. Simulation & Gaming, 29, 7-19. WOLFE, J. & GUTH, G. (1975) The Case Approach vs. Gaming in Evaluation. Journal of Business, 48, 349-364. WOLFE, J. & ROBERTS, C. R. (1986) The External Validity of a Business Management Game: A five-year longditudinal study. Simulation & Games, 17, 45-49. WOLFE, J. & ROBERTS, C. R. (1993) A Further Study of the External Validity of Business Games: Five year peer group indicators. Simulation & Gaming, 24, 2133. WOOD, L. E. & STEWART, P. W. (1987) Improvement of Practical Reasoning Skills with a Computer Game. Journal of Computer-Based Instruction, 14, 49-53. WORRAL, L. (2004) A Perspective on Management Research. Wolverhampton, University of Wolverhampton. YIN, R. K. (1989) Case Study Research: Design and Methods, Newbury Park, CA., Sage. YOUNG, M. (2002) Clarifying Competency and Competence. Henley Working Paper. Henley, Henley Management College. 181

ZALATAN, K. A. & MAYER, D. F. (1999) Developing a Learning Culture: Assessing changes in student performance and perception. Developments in Business Simulation & Experiential Learning, 26.

182

Appendix 1 - Strategy Programme Overview
Programme Over vie w
Mission Objectives Strategy Tactics

Conte nt Introduction to the programme Proble m solving Mission and Objectives Finance Value chain Industry attractiveness External forces (PEST) Debrie f and applicat ion to business Identifying customer needs Industry segmentation Attractiveness and achievability of new positions Debrie f and applicat ion to business Strengthening value chain Pushing back against 5 forces Using partnerships and alliances Exp lo it co mpetitor wea knesses Debrie f and applicat ion to business Selecting strategy SFA Strategic staircase

Strategy CoPilot – Simulati on Introduction to the simu lation Phase 0 – Proble m Solving Phase 1 – Diagnosis – analysis of current situation

Strategy at the Edge Game Introduction to the game Principles of decision ma king and input Round 1 – Establish objectives and strategy to achieve mission

Case Study

Introductory case study. How to read and analyse cases. Case: NTT DoCoMo “Questions hang over record-breaking issue” Case: Cooleys Distille ry: A Ne w “Spirit” in the World Whiskey Industry Case: Boo.Co m The Path to Failure

Internal Analysis Finance Value C hai n Industry attractiveness External Analysis PEST Industry segmentati on Customer ne eds

SWOT

Phase 2 – Customer Interviews and content e xerc ise

Round 2 – Assess attractiveness and achievability of chosen strategy against competitors Round 3 & 4 Mergers and acquisitions and alliances allowed.

Competitive Advantage Competitiv e Positio ns Attractiveness and Achieva bil ity ANSOF matrix

Phase 3 – Co mpetit ive Advantage

Options M&A Generic Strategi es

Phase 4 – Se lect and Refine Strategy

Choice Suitabi lity, Feasibi lity Acceptabi lity Scenari os

Round 5 Consultancy round – each team consults another to evaluate decisions and effectiveness

Case: Cooleys and Boo.Co m

Debrie f and applicat ion to the business Group presentation on Business unit strategy

183

Appendix 2 – Online Version of Kolb’s LSI III
Instructions The learning styles inventory describes the way you learn and how you deal with ideas and day-to-day situations in your life. Below are 12 sentences with a choice of four endings. Rank the endings for each sentence according to how well you think each one fits with how you would go about learning something. Try to recall some recent situations where you had to learn something new, perhaps in your job. Then, using the bullets provided, rank a "4" for the sentence describing how you learn best, down to a "1" for the sentence ending that seems least like the way you would learn. Be sure to rank all the endings for each sentence unit. Please do not make ties. Please provide the following (* required): First Name * Last Name * Company * Email *

When I Learn: I like dealing with my feelings I like to think about ideas I like to be doing things I like to watch and listen

Least like me

2

3

Most like me

I Learn Best When: I listen and watch carefully I rely on logical thinking I trust my hunches and feelings I work hard to get things done

Least like me

2

3

Most like me

184

When I am Learning: I tend to reason things out I am responsible about things I am quiet and reserved I have strong feelings and reactions

Least like me

2

3

Most like me

I Learn By: Feeling Doing Watching Thinking

Least like me

2

3

Most like me

When I Learn: I am open to new experiences I look at all sides of issues I like to analyze things, break them down into their parts I like to try things out

Least like me

2

3

Most like me

When I am Learning: I am an observing person I am an active person I am an intuitive person I am a logical person

Least like me

2

3

Most like me

185

I Learn Best From: Observation Personal relationships Rational theories A chance to try out and practice

Least like me

2

3

Most like me

When I Learn: I like to see results from my work I like theories and ideas I take my time before acting I feel personally involved in things

Least like me

2

3

Most like me

I Learn Best When: I rely on my observations I rely on my feelings I can try things out for myself I rely on my ideas

Least like me

2

3

Most like me

When I am Learning: I am a reserved person I am an accepting person I am a responsible person I am a rational person

Least like me

2

3

Most like me

When I Learn:

Least like me

2

3

Most like me

186

I get involved I like to observe I evaluate things I like to be active

I Learn Best When: I analyze ideas I am receptive and open-minded I am careful I am practical

Least like me

2

3

Most like me

187

Appendix 3 – Performance Rating Scale
Generalised example of the boss’s Performance Rating Scale

Name: • • • 2 • • • • 4 • • • 5 • Fails to meet objectives Demonstrates poor skills in working with others or hits own targets at the expense of others Below target – meets objectives with detailed supervision Demonstrates limited level of knowledge and problem solving skills to meet objectives independently On-target – meets the key objectives of the role Demonstrates strong ability to deliver to targets and takes a proactive approach to problem solving Exceeds the demands of the role with minimum supervision or assistance Shows high levels of initiative and problemsolving Demonstrates thorough knowledge and ability to achieve the role requirements Outstanding – substantially exceeds demands of KPI’s and the role in general Demonstrates high levels of pro-activity, insight and creativity

1

3

188

Appendix 4 – Simulation and Game Overviews
STRATEGY CoPilot™
The simulation is divided into five phases, which follow the same conceptual structure as the tutorial. In outline: • Phases 0 and 1: An introduction to Acme Bottle and its competitive context, based on the issue of whether the parent company should accept a bid for the division. In answering this question the user is also encouraged to diagnose the current issues around Acme’s positioning and sources of advantage. The user gathers data through interviews with the management team and an outside broker, and is guided in their problem solving and communication efforts by the Mentor.

• Phase 2: In this phase the user develops ideas for new competitive positions for Acme. They interview a range of customers to understand their needs, and then work with the management team to turn this data into a needs-based segmentation. Mapping competitors onto this segmentation identifies potential new positions for Acme to serve. The programme then generates a long list of possible positions using the creative idea generation techniques in the theory section and the phase finishes with an initial assessment of the relative attractiveness and achievability of a shortlist of these options. • Phase 3: The next phase revolves around investigating how Acme can develop or strengthen the capabilities it needs in order to serve the favoured competitive positions. Again, users must ensure they are asking the right questions to elicit valuable insights, and they practice a range of powerful techniques for generating ideas about competitive advantage. On the basis of these findings, the user may adjust their previous analyses of the attractiveness of potential new positions. However, the pressure is now building from the CEO of the parent company. In this and the final phase, the user must keep their team happy and involved, as well as ensuring thorough consideration of the key issues. • Phase 4: In the final phase the user selects and presents the chosen strategy for Acme Bottle, and lays out the initial outline for the key steps to implementation. This includes understanding the financial implications of the strategy, as well as identifying internal barriers to implementation and the implications of uncertainty. As in real life, there are political and emotional challenges as well as analytical issues to address.

189

STRATEGY at the Edge
The simulation is divided into five rounds or years, which follow the same conceptual structure as the tutorial. In outline: • Round 1: Your team is the new management team of an International Airline. With capital raised by the founding shareholders, you need to establish your strategy and objectives to provide the return on equity expected. You need to make fundamental choices about your airline and how it will be run. Decisions in this round include: Your hub airport, the number and configuration of aircraft (you are limited to two aircraft type in the first round, the Airbus 200 and 400 series, for short and long haul flights respectively), the seating configuration in each aircraft, the in-flight catering, and your seat pricing in each class of service you offer. The team gathers data through reviewing market research information and using the simulation to test your assumptions. Your tutor will guide you through any difficulties. At the end of this round, the simulation calculates the market share of each team based on a market engine that accurately predicts the effects of interactions. • Round 2: In this round, teams develop ideas for new competitive positions for their airline. They review new market research to understand customer needs, and then work with the simulation to turn this data into a needs-based segmentation. Mapping competitors onto this segmentation helps identify potential new positions for your airline. Using a number of different combinations of services, routes, pricing and incentives (such as air miles), teams may use the simulation to establish likely effects based on assumptions. • Round 3: The next round revolves around investigating how the team can develop or strengthen the capabilities it needs in order to serve the favoured competitive positions. Again, users must ensure they are asking the right questions to elicit valuable insights, and they practice a range of powerful techniques for generating ideas about competitive advantage. On the basis of these findings, the user may adjust their previous analyses of the attractiveness of potential new positions. However, the pressure is now building from the founding shareholders. In this and the final rounds, the team needs to show profitability or face the consequences of potential merger or acquisition. • Round 4: In the final round of the competitive game, teams may form alliances, merge with each other or mount friendly (or even hostile) takeovers. Or teams may elect to go it alone • Round 5: Teams rotate to another team and act as consultants to assess the selection of chosen strategies and reviews the implementation of the strategy. This includes understanding the financial implications of the strategy, as well as identifying internal barriers to implementation and the implications of uncertainty. As in real life, there are political and emotional challenges as well as analytical issues to address.

190

Appendix 5 – MCQ Competency Descriptors
Achievement Orientation Core: Does the person think about meeting and surpassing goals and taking calculated risks for measured gains? A concern for working well of for surpassing a standard of excellence. The standard may be one’s own past performance (striving for improvement); an objective measure (results orientation); outperforming others (competitiveness); challenging goals one has set; or even what no one has ever done (innovation). Thus a unique accomplishment also indicates achievement orientation. Developing Others Core: Does the person work to develop the long-term characteristics (not just skills) of others? Involves a genuine interest to foster the long-term learning or development of others, with an appropriate level of need analysis and other thought or effort. Its focus is on the developmental intent and effect rather than on a formal role of training. Directiveness Core: Does the person set firm standards of behaviour and hold people accountable for them? Implies the intent to make others comply with one’s wishes by appropriate and effective use of personal power or the power of one’s position, with the long-term good of the organisation in mind. It includes a theme or tone of “telling people what to do.” The tone ranges from firm and directive to demanding or even to threatening. Impact and Influence Core: Does the person use deliberate influence strategies or tactics? Implies an intention to persuade, convince, influence, or impress others in order to get them to go along with or to support one’s own agenda. It is based on the desire to have a specific impact or effect on others: the person has his or her own agenda – a specific type of impression to make or a course of action that he or she wants others to adopt. Interpersonal Understanding Core: Is the person aware of what others are feeling and thinking, but not saying? Implies wanting to understand other people. It is the ability to accurately hear and understand the unspoken or partly expressed thoughts, feelings, and concerns of others. It measures increasing complexity and depth of understanding of others and may include cross-cultural sensitivity. Organisational Awareness Core: Is the person sensitive to the realities of organizational politics and structure? The ability to learn and understand the power relationships in one’s organisation or in other organisations (customers, suppliers. etc.). This includes the ability to identify the real decision-makers and the individuals who can influence them, and to predict how new events or situations will affect individuals and groups within the organisation.

191

Team Leadership Core: The intention to take a role as leader of a team or other group. It implies a desire to lead others. Team leadership is generally, but certainly not always, shown from a position of formal authority. The “team” here should be understood broadly as any group in which the person takes on a leadership role.

192

Appendix 6 – Learning Scale
Assessment of presentation in demonstrating learning Please consider the following questions when assessing the learning of you staff at their presentation: • • • • • • Have they demonstrated analysis of the business problem in question? Have they defined the problem? Have they defined the goals and objectives? Have they shown how facts were analysed and how the problem developed? Have they considered realistic alternatives? Have they chosen the best solution based on the alternatives?

Please rate your staff member’s learning, in your opinion that they have demonstrated from the training programme on the following scale Participant name Demonstrated learning from the training programme A little Somewhat A lot A great deal

Not at all

Please add any comments for feedback to the participant:

193

Appendix 7 – Example Report
Managerial Effectiveness Competency Assessment Personal Profile Ann Onymous
Thursday July 14th 2004

Interpreting the Competency Frequency profile The profile you and your third party nominees have completed illustrates the frequency with which you demonstrate managerial behaviours. In other words, if you "Occassionally" demonstrate each of the different behaviours that compose Developing Others, your frequency graph would be at 50% of the way up the graph. When reviewing your profile you should consider four key questions: 1. Do I demonstrate each competency at least some of the time? It is easier to develop an existing competency than to start one from nothing. If you score 25% or higher, it means you have demonstrated the competency at least once or twice, and would find it easier to increase your use of it. 2. How often are each of the competencies seen? The higher the frequency, the more often you have demonstrated all these behaviours. If a competency is scored 50% or higher, it has been seen multiple times; 75% or higher, frequently. 3. Which are strengths and which areas for growth? All seven competencies are needed to be the most effective manager. To reasch that point, all seven should be at the 50% mark. For development, you should concentrate on those rated below 50%. 4. How do my thrid party nominees see me demonstrate these competencies? Often, our own assessment is different from those we work with. Remember the question asked for each competency is: "How often have you seen.... in the last six months". You may consider that you do demonstrate something, others however, may not see this - this suggests that you may need to clearly demonstrate particular behaviours that you are comfortable with. Conversely, others may see behaviours that you don't recognise. Consider your own scores with your thrid parties - what does this tell you and how might you go about changing this perception? The consolidated level profile will help you identify particular behaviours that may guide your self-awareness and/or what you might undertake to develop yourself in competency areas. The following page describes each of the comptencies measured within this assessment. If you wish to discuss your profile in confidence, please log on to the Corporate Edge Mentor Forum - we will be pleased to review the profile with you by email, phone or in person. Thank you for taking the time to complete the MECA, we hope that you find it beneficial and a helpful guide for your further personal development.

Corporate Edge Asia www.ce-asia.com

194

Competency Level profile for Ann Onymous Bachelors degree
38 year old Female, Chinese. Malaysian national, Manager Achievment Orientation A stable part of your repertoire: you are comfortable enough with these behaviours that people see them frequently. Average Level Level Level Level Level Frequency Frequency Frequency Frequency Frequency Frequency Percentage Self Boss Peer Staff Friend Achievment Orientation Level Profile

Scale Level 1 2 3 4 5 6

Behaviour Wants to do job well (not scored)

Frequency Percentage
0.0% Wants to do job well (not scored) Creates own measures of excellence (not scored) Improves performance Sets and works to meet challenging goals Makes cost-benefit analyses Takes calculated entrepreneurial risks 25.0% 50.0% 75.0% 100.0%

Not Scored Creates own measures of excellence (not scored) Not Scored Improves performance Sets and works to meet challenging goals Makes cost-benefit analyses Takes calculated entrepreneurial risks 30.0% 37.5% 37.5% 27.5% 50% 63% 50% 63% 63% 75% 63% 25% 38% 50% 75% 50%

Takes calculated entrepreneurial risks: Commits significant resources and/or time (in the face of uncertainty) to increase benefits (i.e., improve performance, reach a challenging goal, etc.). Boss assessment higher than your own Peer assessment higher than your own Not assessed by Staff Not assessed by Friend

Developing Others
Scale Level 1 2 3 4 5

A stable part of your repertoire: you are comfortable enough with these behaviours that people see them frequently. Level Level Level Level Level Average Frequency Frequency Frequency Frequency Frequency Frequency Friend Staff Peer Boss Self Percentage Not Scored 50.0% 37.5% 45.0% 40.0% 100% 63% 88% 75% 75% 63% 75% 63% 75% Level 63% 63% 63%
Developing Others Level Profile

Behaviour Expresses positive expectations of person (not scored) Gives how-to directions Gives reasons, other suport Gives feedback to encourage Does longer-term Coaching or Training

Level

Frequency Percentage
0.0% Expresses positive expectations of person (not scored) Gives how-to directions Gives reasons, other suport Gives feedback to encourage Does longer-term Coaching or Training 25.0% 50.0% 75.0% 100.0%

Does longer-term coaching or training: Arranges appropriate and helpful assignments, formal training, or other experiences for the purpose of fostering a person's learning and development. Has people work out answers to problems themselves so they really know how, rather than simply giving the answer. This does not include formal training done simply to meet corporate requirements, May include identifying a training or developmental need and establishing new programmes or materials to meet it. Boss assessment not as high as your own Peer assessment higher than your own Not assessed by Staff Not assessed by Friend

Impact and Influence
Scale Level 1 2 3 4 5 6

A stable part of your repertoire: you are comfortable enough with these behaviours that people see them frequently. Average Level Level Level Level Level Frequency Frequency Frequency Frequency Frequency Frequency Percentage Self Boss Peer Staff Friend Impact and Influence Level Profile

Behaviour States intention but takes no specific action (not scored) Takes a single action to persuade Takes multiple actions to persuade Calculates the impact of one's actions or words uses indirect influence Uses complex influence strategies (not scored)

Frequency Percentage
0.0% 25.0% 50.0% 75.0% 100.0%

Not Scored 22.5% 35.0% 32.5% 37.5% Not Scored 38% 63% 88% 63% 63% 50% 50% 50% 13% 63% 25% 75%
Level

States intention but takes no specific action (not scored) Takes a single action to persuade Takes multiple actions to persuade Calculates the impact of one's actions or words uses indirect influence Uses complex influence strategies (not scored)

Uses indirect influence: Uses chains of indirect influence: "Get A to show B so B will tell C such-and-such" OR takes two steps to influence, with each step adapted to the specific audience. Uses experts or third parties to influence.

Boss assessment not as high as your own Peer assessment higher than your own Not assessed by Staff Not assessed by Friend Team Leadership A stable part of your repertoire: you are comfortable enough with these behaviours that people see them frequently. Average Level Level Level Level Level Frequency Frequency Frequency Frequency Frequency Frequency Percentage Self Boss Peer Staff Friend Not Scored Not Scored 30.0% 30.0% 40.0% 42.5% 50% 88% 88% 75% 63% 63% 63% 75% 50% 63% 38%
Positions self as the leader Communicates a compelling vision

Team Leadership Level Profile
0.0% Manages meetings well (not scored) Keeps people informed (not scored)

Frequency Percentage
25.0% 50.0% 75.0% 100.0%

Scale Level 1 2 3 4 5 6

Behaviour Manages meetings well (not scored) Keeps people informed (not scored) Promotes team effectiveness Takes care of the group Positions self as the leader Communicates a compelling vision

Level

Promotes team effectiveness Takes care of the group

Communicates a compelling vision: Has genuine charisma; communicates a compelling vision that generates excitement, and commitment to the group mission.

Boss assessment not as high as your own Peer assessment same as own Not assessed by Staff Not assessed by Friend

195

Competency Frequency Profile for Ann Onymous Bachelors degree post assessment
38 year old Female, Chinese. Malaysian national, Manager

Self 100% 90% 80% 70% 60% 50% 40% 30% 20% 10% 0% Achievment orientation
Used Self Self Boss Peer Staff Friend Seen Multiple times Apparent Strength Seen Multiple times Seen Multiple times

Boss

Peer

Staff

Friend

Developing others
Used Seen Frequently Apparent Strength Seen Multiple times Seen Multiple times

Impact and Influence
Used Seen Multiple times Apparent Strength Seen Multiple times Seen Occasionally

Team leadership
Used Seen Frequently Apparent Strength Seen Multiple times Seen Occasionally

196

Appendix 8 – Reaction Form
Name
(please use block capitals)

Please rate your opinion about each session/tutor and please add comments. Your feedback will help us to improve the programme. Please rate each session during the programme on two scales: How much you enjoyed the session and How useful you found the session for your development Session
Strategy Co-Pilot Simulation Strategy Co-Pilot Debrief and application Theory sesions Team Work I enjoyed this se ssion Not at all Greatly 1 1 1 1 2 2 2 2 3 3 3 3 4 4 4 4 5 5 5 5 I found thi s se ssion useful Not at all Very 1 1 1 1 2 2 2 2 3 3 3 3 4 4 4 4 5 4 5 5

Please rate the programme overall and each of the tutors
Not at all 1 1 1 1 Not at all 1 1 1

Programme overall
Did thi s programme Meet your ex pectations? Meet stated objectives? Reflect your organisational business issues? Please rate your overall satisfaction with this programme Programme Content Please state the extent to which: You have inc reased your understanding of strategy The programme was relevant to your needs The programme was of practical value

Some 2 2 2 2

OK 3 3 3 3

Well 4 4 4 4

Very well 5 5 5 5

Some 2 2 2

OK 3 3 3

Well 4 4 4

Very well 5 5 5

197

Master your semester with Scribd & The New York Times

Special offer for students: Only $4.99/month.

Master your semester with Scribd & The New York Times

Cancel anytime.