Writing Multiple Choice Assessments

L & D Writing MC Evaluations

Version 1.0

Table of Contents
Course Description and Objectives ........................................................................................ 5 Introduction .................................................................................................................... 5 The Advantages and Disadvantages of .................................................................................. 5 MC Evaluations ...................................................................................................................... 7 The Advantages ............................................................................................................. 7 The Disadvantages......................................................................................................... 7 Components of a Multiple Choice Question............................................................................ 8 Overview ........................................................................................................................ 8 Writing the Stem ............................................................................................................. 9 Writing Distractors .........................................................................................................13 Distractor Activity ...........................................................................................................15 Good and Bad Practices ................................................................................................16 Test Validity ...................................................................................................................16 Writing a Multiple Choice Question Activity ....................................................................17 Summary .......................................................................................................................18

L & D Writing MC Evaluations

Version 1.0

L & D Writing MC Evaluations

Version 1.0

Course Description and Objectives

Introduction
Rationale There are people that describe themselves as expert test takers. They can write a multiple choice test and pass without having to learn the lesson. There are workshops on how to effectively take Multiple Choice tests. How does someone learn how to take a test? The reason why people are able to take multiple choice tests without having expert knowledge in a particular subject matter is largely because the Multiple Choice questions are poorly designed.

Purpose

The purpose of this lesson is to discuss and learn how to effectively write Multiple Choice questions for New Hire CSR training evaluations.

Learning Objectives

By the end of this session, you will be able to: Identify the components within a Multiple Choice question List the Advantages and Disadvantages of Multiple Choice questions and evaluations Explain why validity is important in designing a question Identify good practices and bad practices Write an effective Multiple Choice Question

L & D Writing MC Evaluations

Version 1.0

L & D Writing MC Evaluations

Version 1.0

The Advantages and Disadvantages of MC Evaluations

The Advantages
What are some advantages of MC evaluations? Some advantages of multiple choice evaluations are: Easy to score Provide statistical information Easily administered

Can you think of anymore?

The Disadvantages
What are some Some disadvantages of multiple choice evaluations are: disadvantages Instructors tend to favour recall type questions (easier to design) of MC Cannot easily assess synthesis and evaluation cognitive levels evaluations? A good MC evaluation is difficult to design and time consuming

Can you think of anymore?

L & D Writing MC Evaluations

Version 1.0

Components of a Multiple Choice Question

Overview
What makes up a MC question? A Multiple Choice question is made up of 3 different components: 1. The Stem 2. The Distractors 3. The Correct Answer Just Energy has offices in which of the following cities? __________ a) Honolulu _________________ b) Mississauga _________________ c) Rome _______________

The Stem

The stem is the question or the item you are testing. Every item should be based on important content that you want the student to know when the training has completed. There is no rule regarding the length of the question, however it should reflect specific content related to the objectives. The stem may be presented in one of 2 ways: a) A Question b) A Statement.

How many Distractors should I use?

Once you have written your stem, you must then have your correct answer and distractors to make up the rest of your question; but how many distractors should you use? More often than not, we use 3 distractors and 1 correct answer. Some studies suggest that using 2 distractors is enough, and heres why. While creating a question, it takes time to think of plausible answers. We can usually come up with 1 or 2 really good ones and the third one is usually lacking in substance. The 3 distractors listed below are examples of some common distractors used when we cant come up with better ones. All of the above None of the above A and B, etc.

We will discuss the use of the distractors listed above in more detail. The average number of distractors used is 3-5.

L & D Writing MC Evaluations

Version 1.0

Writing the Stem

Introduction We have identified what the stem is; now let us look at some best practices and things to avoid, when creating a stem.

Best Practices

Some things we want to keep in mind when writing the stem: Make the directions clear and simple Present one thought or problem per stem Use pertinent information Use realistic scenarios

Make Directions Clear and Simple

When writing the stem, we want to make the directions/question clear and simple. We can do this the following ways: 1. Minimize reading time and effort We can do this by keeping questions short and simple through the use of shorter sentences and avoiding unnecessary information and verbiage. Edit this Stem High temperatures and heavy rainfall characterize a humid climate. People in this kind of climate usually complain of heavy perspiration. Even moderately warm days seem uncomfortable. Which of the following describes this climate zone? a. Desert b. Tropical c. Arctic 2. Avoid Teaching in the stem The students should already have learned the information necessary to answer the question. If we are teaching in the stem, then we must ask ourselves if we covered this information in class? If we did not cover the content in class, it cannot be evaluated! Destruction of certain cortical tissue will lead to symptoms affecting our behavior. However, this destruction may lead to symptoms that are due to withdrawal of facilitation in other cortical areas. Thus, the tissue damage affects bother directly and indirectly cortical functioning. While such effects may be temporary, what is the typical recovery time?
Continued on next page

L & D Writing MC Evaluations

Version 1.0

Writing the Stem, Continued

Make Directions Clear and Simple, continued How would you re-write this question to be more effective?

3. Use Names for Clarity In a complex scenario, using names will provide more clarity and make the options clearer. They add a realistic component. The use of first names is all that should be used as the use of first and last names can be too much. You should try to use names that reflect the culture of your organization/community. Without Names During a team meeting, the team leader notices that one of the participants is uncomfortable with a team members views on JustGreen. After the meeting, he asks him why he did not raise his concerns at the meeting. He explains that he was worried that he might take is comments personally. With Names During a team meeting, the team leader, Nigel, notices that Fatima is uncomfortable with Normas views on JustGreen. After the meeting, Nigel asks Fatima why she did not raise her concerns at the meeting. Fatima explains that she was worried Norma would take her comments personally.
Continued on next page

L & D Writing MC Evaluations

Version 1.0

10

Writing the Stem, Continued

Present One Thought or Problem per Stem This can also be related back to providing clarity. We should avoid asking 2 or more questions in 1 question. This may cause confusion. The mind can only focus on one thing at a time. To be fair to the student, if we want to ask 2 questions, then we should ask 2 separate questions. Instructions should be clear. Which of the following, are authorized to sign the agreement (Residential or Commercial)? What problems can you foresee with the above question?

Use Pertinent Information

We often tend to add extra information in questions that are not necessarily required. We do this because we think if the end user has more information it will make it easier to understand and answer the question. This is not always the case, especially if it is not relevant to the question being asked. This will cause confusion. When an Independent Contractor completes an enrollment agreement, he must complete the sale with an FPRC call to our TPV, who will provide an FPRC number and reaffirm the sale with the customer. Which of the following statements is TRUE? a) IC may not use their cell phone for the FPRC call b) IC will provide the TPV rep with customer and product information c) IC may not be present during the reaffirmation/verification portion of the call d) The TPV rep will provide the customer with an FPRC code to be written on the contract Edit the stem.

Continued on next page

L & D Writing MC Evaluations

Version 1.0

11

Writing the Stem, Continued

Use Realistic Scenarios Whenever possible, we should always try to use realistic scenarios that are plausible in our line of business. As trainers and designers, we are typically aware of the most common scenarios and types of calls the CSRs receive; and if we are not, we are able to gather this info from other resources. The evaluations should focus on common scenarios instead of one-off situations.

Using Negatives in the Stem

Using negatives in the stem should be avoided. It is twice as hard to understand a question when there are negatives included. This can also discriminate against some learners (ESL learners, Test Anxiety, etc). Our questions should be learner friendly. Use of Negatives Which is not a reason to stop CPR? a. b. c. d. No one has ever survived your resuscitation efforts before The patient regains a pulse The on-line physician tells you to stop The patient meets the criteria for declaring death in the field

Instead, try using alternative wording to get your point across. Try using these: What mistake did she make? He should avoid What strategy would be unwise?

L & D Writing MC Evaluations

Version 1.0

12

Writing Distractors
Overview Writing distractors is very involved and time consuming. More often than not, distractors are poorly designed because adequate time and thought are not provided. Different techniques and strategies will help to create more effective distractors. In this section we will look at strategies to writing effective distractors.

Strategies

Some strategies to keep in mind when writing distractors are: All distractors should be plausible Use incorrect paraphrases from material Using familiar or verbatim quotes that are incorrect True statements that are not pertinent to the question Using common errors that learners might make Statements containing irrelevant jargon

Length of Distractors

The length of your distractors should be equal in length. When they are not, it can make guessing the correct answer that much easier. Guess the correct answer. Medical practitioners consider secondary gain as a. Synonymous with malingering b. A frequent problem with obsessive-compulsives c. A complication of a variety of illnesses and tends to prolong many of them especially in older patients d. Never seen in organic brain damage

Vocabulary in the Distractor

The distractors should parallel the vocabulary and grammar in the stem. The most effective strategy in playing winning tennis for a beginner is a. b. c. d. More pace in ground strokes To keep the ball in play Volley the net as often as possible Hit the ball as hard as possible

What is the correct answer?

Continued on next page

L & D Writing MC Evaluations

Version 1.0

13

Writing Distractors, Continued

Clang Associations Guess the correct answer. Which is the fundamental law that governs the slope of groundwater? a. Gibsons Formula b. Darcys Law c. Toricellis Theorem Clang Associations occur when a word in the stem is repeated in the correct answer, therefore making the correct answer easily identifiable. Remember, vocabulary should be parallel in all distractors.

Format

The format or order in which you present your stem and distractors may impact the learner. Typically, when distractors are listed horizontally instead of vertically, it makes it more difficult for the learner. You draw a card from a deck of 52 cards. What is the chance you will draw a face card? a. 8/52 b. 12/52 c. 16/52 d. 18/52

You draw a card from a deck of 52 cards. What is the chance you will draw a face card? a. b. c. d. 8/52 12/52 16/52 18/52

When placing distractors, they should be done so they are in logical or numerical order. A ladies shirt costs $9.99 and is reduced by 25%. What is the new price? Incorrect a. b. c. d. $5.00 $7.50 $2.50 $6.60 Correct a. b. c. d. $2.50 $5.00 $6.60 $7.50

L & D Writing MC Evaluations

Version 1.0

14

Distractor Activity
Instructions Read the Laying Off Employees handout. The Stem and the correct answer are provided. Create 3 plausible distractors. Scenario As a result of a prolonged strike, Consolidated Tool & Die decides in a toplevel meeting to reduce staff within 30 days. As he leaves the meeting, Production Manager Marco wonders how to break the news to those machinists on the shop floor who will soon be leaving. What should he do to best communicate the upcoming layoff to his workers? He should Correct Answer a. Share the news with affected workers at the earliest opportunity.

b. _______________________________________________________ _______________________________________________________

c. _______________________________________________________ _______________________________________________________

d. _______________________________________________________ _______________________________________________________

L & D Writing MC Evaluations

Version 1.0

15

Good and Bad Practices

Words to Avoid Guess the correct answer. Research on homework supports which of the following statements? a. Never assign homework on Fridays b. Homework should be consistent with class learning c. Always evaluate homework the next day Words that are definitive should be avoided. This includes words like Always and Never.

None/All of the Above

The None of the Above and All of the Above options should not be used too often. When using these options, keep the following in mind: They should be the correct answer some of the time (25%) They are used as a last resort when no other options come to mind

Trick Questions

Trick questions should not be used. We are not trying to trick the learners, only evaluating what they learned and ensuring there was a transfer of knowledge. For learners with disabilities or test anxiety, this will make answering the question that much more difficult. How many animals of each gender did Moses bring on the Ark?

Test Validity
Defending Questions At the end of the day, all of our questions must be defendable. This means, should a learner raise a concern about the result of a question, we must be able to defend why we have that questioned and where to find the answer. We cannot ask questions that are not included in our written material. To defend our evaluations, we must be able to demonstrate where we can find the answer. If there was a question that was ambiguous or less clear, the student may be correct in defending their answer. We must therefore, always ensure there is clarity in our questions as well.

L & D Writing MC Evaluations

Version 1.0

16

Writing a Multiple Choice Question Activity

Instructions Please read Deciding Whom to Lay Off handout. Once you have completed reading, create a question complete with a correct answer and distractors. Stem _____________________________________________________________ _____________________________________________________________ _____________________________________________________________ _____________________________________________________________

a. _______________________________________________________ _______________________________________________________ _______________________________________________________ _______________________________________________________

b. _______________________________________________________ _______________________________________________________ _______________________________________________________ _______________________________________________________

c. _______________________________________________________ _______________________________________________________ _______________________________________________________ _______________________________________________________

d. _______________________________________________________ _______________________________________________________ _______________________________________________________ _______________________________________________________

L & D Writing MC Evaluations

Version 1.0

17

Summary
In Conclusion Writing Multiple Choice evaluations is not as easy as most people believe to be. There are different techniques and strategies used to create effective MC evaluations. Some things to keep in mind when creating MC questions: Avoid Trick questions Keep questions relevant Distractors should be plausible Use parallel vocabulary and grammar Ensure the question is defendable

L & D Writing MC Evaluations

Version 1.0

18