Some perspectives on MCQ assessment

Stuart Palmer Mary Dracup

Source: http://www.dreamstime.com/item.php?imageid=7299038

1

Assessment generally

“Examinations are formidable even to the best prepared, for the greatest fool may ask more than the wisest man can answer.”
Charles Caleb Colton (1780 - 1832)

2

General thoughts
MCQs can be a useful element of an overall assessment design Can’t be abstracted from the context
No marks / diagnostic / formative / interactive Few marks / semi-formative / assignment High stakes / summative / examination

Design and evaluation approach should be appropriate for the context
3

No need to look too far from home Considine. Collegian. Design. validity and reliability of multiple choice questions for use in nursing research and education. J. 4 . M and Thomas. 12 (1): 19-24. 2005 Jan. S. Botti. format.

M and Thomas. S. format. 2005 Jan. Collegian. 12 (1): 19-24. Design. Botti.A proposed MCQ process From: Considine. 5 . J. validity and reliability of multiple choice questions for use in nursing research and education.

An alternative/pragmatic process Design Content/Face validity Use Construct validity Reliability Keep / Replace 6 .

Design – unit assessment plan Assignment No. 1 2 3 4 5 6 7 Exam Assignment details Introduction to DSO Information literacy & the Library Referencing Professional ethics report Multi-choice test 1 Multi-choice test 2 Major report and presentation Must pass! Marks 3% 5% 10 % 15 % 6% 6% 25 % 30 % 7 .

Knowledge 3.5% 3.Design – MCQ test plan Content Areas A B C D E % Processes 1.25% 10.75% 6.25% 3.25% 35 3.75% 25 2.75% 5.5% 5.75% 7. Comprehension 5. Application 6% 10% 6% 12% 6% 40 % 15 25 15 30 15 100 8 .25% 8.

Design – NCLEX-RN MCQ test plan Source: 2010 NCLEX-RN® Detailed Test Plan 9 .

Face validity A sub-set of content validity Use an ‘expert panel’ to : assess item clarity assess item readability assess consistency of style check for errors 10 .

Construct validity Key check – do ‘experts’ agree on the correct answer? Item difficulty – the proportion of examinees that get the correct answer Distracter evaluation Item discrimination – the ability of an item to distinguish between knowledgeable students and others 11 .

Correlation with final unit mark r5 = 0.3x10-10) r6 = 0.8x10-9) 12 .569 (p < 1.584 (p < 5.

8 t 0.3 o s r a e 0.1 0 0 5 10 15 20 25 30 35 Percenta eofunitm g arks Exam Ass4 Ass2 Ass5 Ass 7 Ass6 Ass3 y=0.2209 R² =0.Correlation with final unit mark 0.7 n e i c i f 0.9 0.2 P 0.5 o i t a l e 0.6 f e o c n 0.4 r r o c n 0.1854ln(x) +0.8698 Ass1 13 .

Correlation with final unit mark 0.9 0.4626 R² =0.3 o s r a e 0.1 0 0 5 10 15 20 25 30 35 P ercenta eofunitm rk g a s Exam Ass2 Ass5  Ass6 Ass4 Ass 7 Ass3 y=0.2 P 0.5 o i t a l e 0.7434 Ass1 14 .8 t 0.0139x +0.4 r r o c n 0.7 n e i c i f 0.6 f e o c n 0.

Reliability Really ‘internal consistency’ Typically. Cronbach’s alpha or KR-20 Provides a measure of the average correlation among items within a test scale If a test examines a range of ‘themes’. may need to divide the test data into ‘scales’ 15 .

they should only be discarded on the basis of clearly poor construct validity and reliability 16 .Review of items Where items have been developed based on a systematic test plan and good content validity.

Negative marking / formula scoring “Be the reasons for test-takers’ reluctance to answer under formula scoring what they may be. this reluctance is a psychometric Pyrrhic victory. individual differences in people’s risk attitudes are nothing but unwanted noise: they add another source of variance. From the perspective of educational measurement.” (Bar-Hillel. Budescu & Attali. 2005. which is likely to reduce the test’s reliability and validity. p. Furthermore. Neither test-takers’ risk attitudes nor their strategic reasoning are what multiple-choice tests purport to measure. the testmakers’ persisting inability to distinguish between a guessed correct answer and a known correct answer is compounded by their inability to distinguish between an omission deriving from ignorance and one deriving from risk aversion. 7) 17 .

Negative marking – discontinued by: The Royal College of General Practitioners – from 1992  discriminated against female candidates The Royal College of Obstetricians and Gynaecologists – prior to 2004  benefited bold and test-wise candidates. inhibited reasoning Examination Committee of the European Society of Anesthesiology – from 2008  does not reflect real practice where decisions must be made on incomplete knowledge. no effect on rank ordering of results 18 .

A useful resource? Source: https://www.htm 19 .org/1287.ncsbn.

Nursing care activities are then analyzed in relation to the frequency of performance. impact on maintaining client safety and client care settings where the activities are performed.Development of the NCLEX-RN Test Plan “A total of 12. This analysis guides the development of a framework for entry-level nursing practice that incorporates specific client needs.000 newly licensed RNs are asked about the frequency and importance of performing 155 nursing care activities. as well as processes fundamental to the practice of nursing.” 20 .

NCLEX-RN test plan structure Safe and Effective Care Environment Management of Care Safety and Infection Control Health Promotion and Maintenance Psychosocial Integrity Physiological Integrity Basic Care and Comfort Pharmacological and Parenteral Therapies Reduction of Risk Potential Physiological Adaptation 21 .

Design – NCLEX-RN MCQ test plan 22 .

Detailed test plan 23 .

Detailed test plan 24 .

Sample items – management of care For each of the eight content areas a sample item is provided 25 .

Write a key to represent important information the entrylevel nurse should know 6. misconceptions or irrelevant information 7. Use the previous information and write the distracters 8. Complete the item using the stem. Select a subcategory from the chosen area of the test plan 3. Identify common errors. Select an important concept within that subcategory 4. Use the concept selected and write the stem 5. Select an area of the test plan for the focus of the item 2. key and distracters 26 .Steps in item writing 1.

example Step 1.Steps in item writing . Select a subcategory from the chosen area of the test plan Standard Precautions/Transmission-Based Precautions/Surgical Asepsis Step 3. Select an area of the test plan for the focus of the item Safety and Infection Control Step 2. Select an important concept within that subcategory Evaluate infection control precautions implemented by staff members 27 .

Write a key to represent important information the entrylevel nurse should know Contact Isolation: Assisting the client to ambulate in the hallway 28 .example Step 4. Use the concept selected and write the stem The nurse and nursing assistant are caring for a client with vancomycin-resistant enterococci (VRE). Which of the following activities by the nursing assistant would require immediate follow-up? Step 5.Steps in item writing .

Steps in item writing . Identify common errors. misconceptions or irrelevant information Lack of understanding of isolation precautions Uncertainty related to specific diagnosis Step 7.example Step 6. Use the previous information and write the distracters Leaving a blood pressure cuff in the client’s room to be used by the client only Putting on a protective gown to assist the client to sit in a chair Taking the gloves off before leaving the client’s room 29 .

Steps in item writing .Putting on a protective gown to assist the client to sit in a chair c. Complete the item using the stem. Which of the following actions performed by the nursing assistant would require immediate follow-up from the nurse? a.Leaving a blood pressure cuff in the client’s room to be used by that client only b. key and distracters The nurse and nursing assistant are caring for a client with vancomycin-resistant enterococci (VRE).example Step 8.Taking the gloves off before leaving the client’s room d.Assisting the client to ambulate in the hallway (key) 30 .

Case scenario exercise Reduction of Risk Potential The nurse is caring for client who had a procedure three hours ago. 31 . Write an item that includes assessment data the nurse would observe in this client and which data should the nurse respond to first.

blood urea nitrogen (BUN). 22 mg/dL (key) c.Reduction of Risk Potential – model The nurse is caring for a client who had a cardiac catherization 3 hours ago. 101 d.blood pressure increase from 103/68 to 110/70 over the past one hour b. Which of the following findings would be essential for the nurse to follow-up? a.decrease in respiratory rate from 18 to 16 over the past one hour 32 .pulse.

DSO assessment tools .edu.edu.resources       Online assessment strategies: http://www.deakin.deakin.au/itl/dso/strategies-teaching/assessm Blackboard assessment tools guides: http://www.au/itl/dso/guides/guide-list-respondus 33 .deakin.edu.au/itl/dso/guides/guide-list-assessme Respondus guides: http://www.

34 . mumbling and occasionally yelling aloud “Stop it!” Write an item describing the action the nurse should take in this situation.Case scenario exercise Psychosocial Integrity A nurse on an inpatient psychiatric unit observes a client pacing the hallway.

c. (key) 35 . pacing in the hallway.Remove other clients from the area. and occasionally yelling “Stop it!” Which of the following actions should the nurse take? a.Psychosocial Integrity – model MCQ The nurse is caring for a client who is mumbling. d.Request that the client be quiet and not disrupt others.Escort the client back to the client’s room. b.Use distraction to re-focus the client to reality.

Thank you for your time 36 .