Using Games Based Technology in Formal Assessment of Learning

Mhairi McAlpine, Linn van der Zanden and Veronica Harris, Scottish Qualifications Authority, Glasgow, Scotland mhairi.mcalpine@sqa.org.uk linn.vanderzanden@sqa.org.uk veronica.harris@sqa.org.uk

Abstract While scoring is a routine part of most computer games, and games based learning continues to grow, the formal assessment of learning within a gaming context has been somewhat neglected. The opportunities afforded within games to control the environment have advantages for both learning and the assessment of learning in that context. Allowing a serious game to produce output which can then be used for the assessment of prior learning may overcome practical and validity issues where simulated. In 2008, the Scottish Qualifications Authority with the support of the European Social Fund, started developing Games Based assessments for the hurdle tests which form part of their pre-existing vocational qualifications. By embedding the assessment into a simulated work environment it was aimed to provide a more authentic and valid assessment than traditional pencil and paper. To ensure parity with existing assessments and the maintenance of standards, a report was produced at the end of the game with the responses given in the game used to complete the answers to questions. This report was identical to the outcome of a completed pencil and paper test. Initial trials based on a retailing environment suggested that this approach would be popular with vocational candidates and the teachers who assessed them. There were however a number of practical and organisational difficulties encountered which need to be overcome before we could mainstream this innovative assessment methodology. This paper examines whether Games Based Assessment can provide a practical, cost effective and acceptable method of assessing candidates in vocational subjects, whether it can provide a more authentic assessment experience and as the impact that the change in mode has on the validity and reliability of the qualification. It also goes on to look at the natural progression of games based learning into games based assessment once the medium becomes more mainstream Keywords: Assessment, SQA, vocational education, outcome based learning, games based assessment, formal assessment, hurdle test, vocational assessment.

Introduction SpaceWar developed in 1961 at MIT is widely accredited as being the first computer game and by just the following year, the Chicago Tribune was reporting on the use of computer games for learning. In the 40 years since, computer game technology has advanced enormously and its potential for use in Teaching and Learning has an increasing profile. Within Scotland we have the Consolarium at Learning and Teaching Scotland (Independent, 2010) a dedicated Games Based Learning Thinktank which aims to explore how computer games can be used to best effect to support the new Scottish Curriculum for Excellence, while in the wider UK both FutureLab (FutureLab, 2004) and the now defunct BECTA (BECTA, 2006) have commissioned large scale research into the potential for the use of games to support and enhance learning in classroom, informal and domestic situations. One area which has not been so widely explored is the potential for computer games to assess learning. On the face of it, it looks like an obvious use for the technology especially as the concept of assessment through scoring has been built in from the very earliest computer games. However, in practice there are considerable challenges to be overcome at the assessment design and practical implementation stages as well as ensuring that the conditions required for formal certified assessment of learning are upheld. Development of Skills for Work Qualifications In 2008 the Scottish Qualifications Authority successfully applied to the European Social Fund for additional monies to support the development of the new Skills agenda in particular the new Skills for Work qualifications. It had been recognised for some time that there was a need for vocational courses for young people in the 14-16 age group, which will give them the opportunity to develop knowledge and skills which will help to prepare them for the world of work. To meet this need, in 2006 SQA started to develop Skills for Work courses aimed at pupils in the 14-18 age range with a firm focus on vocational rather than academic skills. These courses differed from other vocational provision because they focused on the generic employability skills needed for success in the workplace, offering opportunities for learners to acquire these critical generic employability skills through a variety of practical experiences that are linked to a particular vocational area such as construction, hairdressing, hospitality etc. The first qualifications were offered as school/college partnerships and intially aimed at students who were not engaging with school. These students were likely to become poor attenders and disengage from learning. In many cases they would re-engage with learning after their formal school leaving date, in a college or a work based learning setting frequently entering with no qualifications. The hope was that these new qualifications could engage their interest and meet their needs as they embarked on the transitition from school to college or employment while offering them a qualification en route. These qualifications became very successful not only for the disengaged learners they were aimed at, but also for more motivated learners, who had a clear career focus and wanted an insight into the industry they wished to enter. The qualifications conformed to the Scottish National Qualifications design rules, which ensure a common structure across different subject areas. These design rules specify that National Qualifications should be comprised of three hurdle tests ungraded internally marked tests which candidates are required to pass to continue with their course, as well as an externally marked final component. In discussing with the qualification developers how they felt that computer assisted assessment could enhance the Skills for Work experience the primary issues that were raised was the difficulty of maintaining a physical portfolio where educational provision and assessment could be undertaken in a number of different locations, including school, college and workplace as well as the barriers that some students found in the requirement for hurdle tests. SQA requires these tests to be passed to ensure the appropriate standards are being applied and that candidates entering the final assessment are suitably

prepared. Many students, however, felt that the writing that was expected from them in these tests was out of kilter with the type of learning that they were doing in the workplace based setting and also the primary form of assessment which was folio based. Rationale for the Use of Games Based Assessment Vgotsky (1962) focused on the role of the interpersonal in learning and development and the importance of the socio-cultural setting of learning. This is further developed in situated learning theory as developed by Lave (1988) which suggests that learning is primarily unintentional and occurs only when it is situated within authentic activity, context and cultural setting. He suggests that knowledge and skills are only acquired through the inclusion in a community of practice which embodies the beliefs and behaviours that the student is to acquire. Brown, Collins & Duguid (1989) have further developed social learning theory through the concept of a cognitive apprenticeship which supports learning in a domain by enabling students to acquire, develop and use cognitive tools in authentic domain activity. Learning, both outside and inside school, advances through collaborative social interaction and the social construction of knowledge. The difficulties of a physical portfolio were easily solved by the provision of an e-portfolio system, however the difficulties encountered by the hurdle tests were more challenging. The attractiveness of the qualification in the first place was premised on their practical nature and written assessments undermined that attractiveness. To overcome this, some of the assessments of the hurdle test involved role play and simulated situations, however it was difficult to make that standardised with limited control over the nonassessed character in the role play. It was thus desirable to reinforce the situational backdrop of Skills for Work courses in the assessment design. SQA decided to develop Games Based Assessment as it could provide the look and feel of being in a workplace environment with the opportunity to interact with another character within it in a standardised manner. It was hoped that this could increase the validity and reliability of the assessment, while also making it more authentic for the candidates. However, while games have been used extensively in learning activities for many years, there is little literature on the use of Games for formal assessment purposes. Indeed a major review of games based learning conducted for JISC (de Freitas, 2006) mentioned assessment and accreditation as one of the pitfalls for the use of games based learning and a potential stumbling block for its widespread adoption. Description of the Games Based Assessments Developed Classified by the Hertz System (Hertz, 1997), the games produced for Skills for Work qualifications are a mix of role play and simulation. In the games you adopt the character of a worker in the environment the exact role is undefined - however it would probably be most akin to a work placement student being assessed. At the same time you are asked to do specific tasks within the environment although not a true simulation such as a flight simulator - there is still the immersive approach. Candidates were assessed on exactly the same performance criteria as per the traditional assessment and a report was produced (see Appendix 1) detailing the result. This report was designed to be as close as possible to the test script the candidate would have produced, had they been assessed in the traditional manner. Two games were developed for launch in Autumn 2010. The first, in Retailing, was designed for candidates at SCQF1 Level 5 (EQF Level 3). In this assessment, the candidate found themselves in a shopping mall, which contained an electrical store, a fashion store and a food retailer. Candidates could go into the stores in any order where they would be met by a supervisor who would explain the organisational guidelines specific to that store. In the electrical store, they were asked to approach and assist customers and recommend them a product based on the results of their questioning; in the food store they were asked questions about key pieces of consumer legislation and in the fashion store they were asked to identify the features and benefits of products as well as select
1

Scottish Credit and Qualifications Framework

complimentary or alternative products. Once the candidate had completed all of the required tasks in the third store that they visited, they were approached by a customer wishing to complain. The candidate was then required to deal with the complaint by following the organisational guidelines for the store that they were in. The second, in Health Sector, was designed for candidates at SCQF Level 4 (EQF Level 2). In this assessment the candidate found themselves in a hospital. They were directed to a supervisor, who gave them general instructions on navigation and behaviour within a hospital environment before being directed to the radiologist. On the way they were intercepted by a nurse and asked questions about their personal responsibility for infection control before finding the radiologist who then questioned them about how biomedical devices, cross contamination and droplet spread can spread infection in a health sector environment. They were then directed to a nurse who asked them to perform a risk assessment of the hospital waiting area by walking round the environment, identifying hazards, the risk associated with each hazard and the corrective action that could be undertaken to minimise the risk. On completion they were directed back to the supervisor, who simulated a sneeze following which the candidates were asked what action they should now take. Developing the Framework of the Assessment The Conditions of Learning (Gagne, 1965) identifies a nine-step process for instructional design using events of instruction. This 9 step model has been much utilised in game based learning (Beker, 2005). Some studies have found that the connection between specified learning outcomes and the learning that has been undertaken in games can be less than linear (Egeneldt-Nielsen, 2005) and there are moves to consider other ways in which the learning undertaken in games may be documented. However, given that learning outcomes tend to be at the heart of current competence based qualifications the dominant form within the vocational sphere, an outcome based approach provides a starting point for exploration. Although not specifically designed for assessment, Gagnes framework allows us to consider the assessment event as a process and consider how a student can move through the event in a meaningful way. The figure below shows these instructional events in the left column and the associated event in the game based learning in the right column. The first six instructional events follow the Gagne sequence, however stage 7 providing feedback, was deliberately missed. The software allows for instant feedback to be generated as candidates respond to an activity, but a primary objective was to ensure comparability with a paper examination where candidates would not receive feedback until after the whole assessment was completed. Thus we removed the feedback option from the game. Table 1: Gagnes 9 events of instruction. Instructional Event Gain attention Inform of objective Stimulating recall Present stimulus Provide guidance Elicit performance Provide feedback The learner is introduced to the environment and given general directions about where they find themselves and what they will be doing At each stage the objective and performance criteria that would be assessed were clearly signaled to the learner The environment itself is designed to be a recall trigger, as they move about into different areas, they should encounter situations which are familiar to them from their learning The candidate is guided to a virtual area where a scenario has been set up with a character ready to interact with the candidate Guidence on how to response to the character is provided on screen through instructions Following the guidance provided This stage, although possible, is explicitly not done to replicate the conditions of a traditional assessment.

Assess performance Enhance retention

Performance is assessed by the tutor who receives a report of the candidates Again this stage was not considered to be suitable for a formal summative assessment

The games development engine, Thinking Worlds - which SQA used as a platform, had been selected partly on the low cost of development but also the potential for rich interactivity through a first person perspective. SQA felt this would have the desired effect of placing the candidate within the virtual environment. While the software had the possibility of automated marking, SQA chose not to implement that module. A critical test of the acceptability of the assessment was ensuring that the teacher remained in control of the assessment. The assessment is not graded, but assessed on a pass/fail basis reflecting the open ended nature of the original assessment. Many of the responses that were available for the candidates to choose were not strictly right or wrong, but closer to a correct response or further way. By providing a report and allowing the teacher to judge whether they have sufficiently demonstrated all of the required competencies the module can be assessed more holistically as well as retaining the principle of internal assessment for the hurdle tests. Development of the assessment It was critical to us that the assessment developed through the games based assessment was comparable to the hurdle tests generated on paper. As per a paper development, subject specialists were recruited to generate the assessment content while artwork the artwork was developed. The development of artwork prior to the content development proved to be erroneous, as unnecessary material had been developed on an ad hoc basis., while further development of essential artwork had to be undertaken once the content was written. Relating tasks in the game environment to the outcomes and performance criteria the fundamental building blocks of assessment units within SQA proved much more difficult in the game environment. Balancing the desire to utilize the environment fully, the need to adhere to the criteria and the limitations of the software, proved a difficult task. Subject specialists who develop assessment material for SQA tend to be experienced classroom practitioners, although as a mature demographic, less familiar with computer games (Social Gaming Monitor, 2010). It quickly became apparent that there was a tension between developing a good game which fully utilised the possibilities of the environment and ensuring a valid and reliable assessment. On a number of occasions the subject specialists deviated from the assessment specifications in order to enhance the playability of the game as they discovered new possibilities and explored different avenues. The subject specialists developments were overseen by the Qualifications Managers who take ultimate responsibility for assessment materials which are published by SQA. Qualification Managers usually take a hands-off approach to development; given the experience of the subject specialists, work is usually of high quality and any errors can be righted relatively quickly. At checking stage however it became apparent that the unfamiliarity of the medium had led the subject specialists to deviate from the specification. As the development had already been done, an expensive process of re-engineering the materials to fit with the learning outcomes and performance criteria had to be undertaken under a much higher level of supervision than normal as well as further revision. In moving to an operational development, it was decided that additional checkpoints were required over and above what would normally be required for a hurdle test development, given the potential for deviation from the specification, the unfamiliarity of the subject specialists with the medium for which they were developing content and the high cost of rework. The development of the artwork specification stage was moved to after content signoff to ensure that the artwork developed supported the assessment material written, rather than the material being written around the environments which had been developed.

Initial Piloting and Feedback In June 2009 a prototype of the game was taken to schools and evaluated in context with Skills for Work candidates. Candidates and tutors had the opportunity to play the game and then participated in individual semi-structured interviews with an external educational consultant. All of the candidates on which we tested the game played games recreationally, however none had prior experience in using a game as a learning or assessment tool. All of them found controlling the character easy and managed both movement and interactions with no difficulty, reporting them as easy to use. They did note that the game was slow and the graphics not as smooth as they were used to a product of a lower cost development engine than that which would be used in commercial games although they did consider them realistic. Pupils reported that they would be happy to have their knowledge, skills and abilities tested through the games based medium and they felt that it made the assessment more fair. They considered that the games based environment was helpful for them in creating a context for them to demonstrate their abilities adding that they were no longer required to imagine a context which was given to them through a written description. They also considered that this type of hurdle test would be less stressful than a traditional pencil and paper version. Teachers identified a number of barriers to introducing a game based assessment primarily practical. ICT provision in schools tends to be available only through specialist ICT suites. Implementing games based assessment requires obtaining a booking slot and ensuring that all required software is available on the machine prior to the assessment session. As this is controlled at a local authority level, this is potentially a time consuming process. They foresaw few barriers for candidates attempting an assessment in this manner, instead suggesting that the visual presentation may overcome some of the difficulties that candidates had with written material. A copy of the questions used to support the semi-structured interview are included as Appendix 2. Report Generation, Access Management and e-Portfolio Integration Once the game was developed and functional, it became apparent that there were a number of surrounding conditions associated with summative assessment through a game, which would not be so critical within learning activities. As it was decided not to automate the marking but instead to produce a report, SCORM compliant data was extracted from the game and a report generated from the output. In order to verify candidate details could be verified through a central access management system rather than locally, the game required an online rather than standalone environment. Furthermore, in a traditional hurdle test, candidates would be given as much time as they required and could change their answers. However the game was linear, so once candidates had undertaken a virtual task, they could not return and there was also a requirement that once the report was submitted by a candidate it was not viewable, just in case they showed it to another candidates who had not yet undertaken the assessment. Thus maintaining the associated assessment conditions of a hurdle test became a major focus of the latter part of the development. Ensuring accessibility, security and the ability of candidates to review and revise their script prior to submission required technical solutions. In order to replicate the conditions, it became clear that the game had to be housed within a larger environment which could ensure the authentication and script management services essential to operational use. As mentioned, the folio of evidence that candidates were required to produce as part of the assessment could be held in an e-portfolio, provided as part of the qualification. However while the game platform was SCORM compliant, the e-portfolio which housed the evidence generated by Skills for Work candidates was not. Consequently a Moodle installation hosting the game was integrated into the

e-portfolio, so that access to the game could be restricted and the final output would only be given to tutors once the candidate had approved it. Candidates were allowed to retake the game as many times as they wished within the one session and view the report produced at the end. Once they were happy with the report, they submitted it for assessment. Following this, the candidate had no further access to the report, it was only available to the tutor for online marking, ensuring the security of a completed script. Conclusion and Next Steps The formal assessment of learning through a games based approach has posed considerable challenges. The main issues and steps that we have taken to address them have been outlined below. 1. Aligning the game based activities to the learning outcomes and performance criteria particularly where possibilities are broad and errors may be expensive to correct. For future developments we are ensuring that the development of assessment content which is carefully aligned to outcomes and performance criteria is prioritised and that transparency over development is achieved through multiple signoffs. 2. There are challenges with subject specialists working in a new and unfamiliar medium. For new qualifications in the Skills for Work suite, we are working closely with the qualifications development staff to make them aware of the possibilities of using games based assessment, so that outcomes and performance criteria are written in a manner which does not preclude interactive virtual assessments and awareness is raised through demonstrating existing developments. 3. The conditions required for a formal assessment can be onerous once translated to an online environment, balancing the need for security with the need for candidates to have an accessible route to assessment. The integration of the e-portfolio system to manage the access to the assessment, report the outcome and ensure the security of the completed script should overcome these issues. The games piloted in 2009 have now been redeveloped and integrated into the e-portfolio product via an intermediate Moodle installation. The first developments in Retailing (SCQF Level 5/EQF Level 3) and Health Sector (SCQF Level 4/EQF Level 3) will be available for live candidate use from August 2010, with further developments in Energy (SCQF Level 5/EQF Level 3), Uniformed and Emergency Services (SCQF Level 4/EQF Level 2) and Health Sector (SCQF Level 5/EQF Level 3) following in early 2011. Additional games based assessments are planned to be made available for August 2011. Initially they are launched as a supplement to the existing provision and we are encouraging the centres to provide them as a revision test for the first year rather than as an acceptable form of assessment evidence, while we gather comparative data for the games based and traditional paper versions. We intend to compare the item level marks obtained by candidates on the paper based and on the game based version and use that data to check the validity and reliability of the games based assessments, as well as monitor the impact that they may have on any sub-group of candidates prior to a full launch as an assessment methodology.

References Becker, K. (2005). How are games educational? Learning theories embodied in games. Proceedings of DiGRA 2005 Conference. Retrieved February 15, 2007 from http://www.digra.org/dl/db/06278.23299.pdf Becta (2006) Computer games in Education Project report available at http://research.becta.org.uk/index.php?section=rh&&catcode=&rid=13595 Brown, J.S., Collins, A. & Duguid, S. (1989) Situated cognition and the culture of learning Educational Researcher, 18(1), 32-42. de Freitas, S. (2006) Learning in immersive worlds. A review of game-based learning. available at: http://www.jisc.ac.uk/media/documents/programmes/elearninginnovation/gamingreport_v3.pdf Egeneldt-Nielsen (2005) Can Education and Psychology join forces available at http://www.nordicom.gu.se/common/publ_pdf/222_egenfeldt_nielsen.pdf FutureLab (2004) Games and Learning available at http://www.futurelab.org.uk/resources/documents/lit_reviews/Games_Review.pdf Gagn, R. M. (1965) The conditions of learning and theory of instruction New York, NY: Holt, Rinehart & Winston Independent (2010) How computer games discovered virtuous reality available at http://www.independent.co.uk/life-style/gadgets-and-tech/gaming/how-computer-gamesdiscovered-virtuous-reality-1871927.html Lave, J. (1988) Cognition in Practice: Mind, mathematics, and culture in everyday life. Cambridge, UK: Cambridge University Press. Social Gaming Monitor (2010) available at http://4.bp.blogspot.com/_RHJCq8Jes1s/S6zp7o8TtmI/AAAAAAAABMA/m_hOgR0T4TM/s1600/ Social_Gaming_Monitor_International_tn51.png Vygotsky, L. S. (1962) Thought and language. Cambridge, MA: The MIT Press.

Appendix 1: Excerpts from Report

Appendix 2: Evaluative instrument for semi-structured interview It should be noted that the questions below were used as the basis for a semi-structured interview conducted individually with each of the teachers and pupils following the trial Questions for pupils to consider 1. What is the extent of your experience of playing digital games (for recreation or fun)? 2. What is the extent of your experience of playing digital games with a view to learning about something? 3. What was your overall experience of using the Thinking Worlds gaming environment (ease of use, error-free, speed of response)? 4. Ease of navigating through the game environment (both shopping mall, health centre)? 5. Ease of initiating / completing interactions with game characters? 6. Which of the following list of behaviours (interactions) did you experience?(Explain what they are to pupils): MCQ / Classify / Comprehend / Label /Contrast / Replace True/False; and Locate items 7. How easy was it to understand and respond to these behaviours? 8. Did you find consistency of experience within each behaviour type (e.g. within Multiple Choice Question)? 9. Did the environment represent a realistic interpretation of the sector you were studying? 10. Would you be confident that this games-based approach provides a fair assessment of your knowledge and skills in the subject? 11. Did the game environment help you understand the context of the questions you were being asked? Or was it a distraction? Or did it not matter? Questions for teachers to consider 1. To what extent have you been using online experiences (game-based or otherwise)to elp pupils learning in the subject? 2. Identify the key factors that have to be managed in order to carry out a gamesbased assessment in your school (e.g. booking computer suites, getting approval to load software)? 3. In the pilot did you observe any barriers to participation on the part of pupils (e.g. lack of basic IT skills, levels of comprehension of game behaviours, lack of other skills) 4. In your view are there specific pupil groups for whom this approach would not be suitable or appropriate? 5. In your view are the assessment outcomes generated through the game-play valid? 6. How well did the game environment reflect the scenarios you have been developing as part of your teaching the subject? 7. To what extent did the game environment reinforce the concepts being tested in the assessment? Or form a distraction from the concepts being tested? 8. What impact did the use of games-based assessment have on pupil motivation (if any)? 9. Did you notice any improvement in pupil performance when using the software versus a conventional approach?