You are on page 1of 17

Assessment Office                                                                                                             (808) 956‐4283

Hawai‘i Hall 107 2500 Campus Road,                                                                                            (808) 956‐6669
Honolulu, HI 96822                                                                                                            manoa.hawaii.edu/assessment

HOW GOOD IS YOUR MULTIPLE-CHOICE TEST?

Making Program Assessment Happen Workshop Series

INTENDED OUTCOMES 1

1. Articulate the purpose and need of item analysis
2. Define the statistics below in either words or math
functions.
1. Item facility
2. Difference index
3. B-index
3. Explain the function of distractor analysis
4. Calculate item analysis indices in Excel
5. Identify the proper index to use in pre- & post-test and
contrasting group scenarios.
6. Evaluate items based on analysis results.
2

You may use these materials only for nonprofit educational purposes. Please give credit/cite appropriately
1

Purpose of item analysis 2. Introducing item analysis software 7. Questions & Evaluation 3 Mission: Improve student learning through program assessment 4 You may use these materials only for nonprofit educational purposes. & B- index 3. Excel practice 1 4. Introducing item facility. Excel practice 2 6.edu/assessment WORKSHOP AGENDA 1. HI 96822                                                                                                            manoa. Assessment Office                                                                                                             (808) 956‐4283 Hawai‘i Hall 107 2500 Campus Road.hawaii. Introducing distractor analysis 5. Please give credit/cite appropriately 2 . difference index.                                                                                            (808) 956‐6669 Honolulu.

                                                                                            (808) 956‐6669 Honolulu. Please give credit/cite appropriately 3 .edu/assessment PROGRAM ASSESSMENT OVERVIEW Assessment Cycle Learning Outcomes Learning Improvement Plan Opportunities Collection & Analysis Assessment Results of Evidence 5 WHY SCRUTINIZE TEST ITEMS? • Ensure accurate measurement of knowledge/skill • Increase confidence in drawing conclusions • Outcome achievement • Student knowledge/skill level • Teaching effectiveness • Enhance student learning • Increase student engagement • Avoid demoralizing students 6 You may use these materials only for nonprofit educational purposes.hawaii. HI 96822                                                                                                            manoa. Assessment Office                                                                                                             (808) 956‐4283 Hawai‘i Hall 107 2500 Campus Road.

S.hawaii. capital? Stem (A) Seattle (B) New York distractors Options/choices (C) Los Angeles (D) Washington DC KEY √  1. Assessment Office                                                                                                             (808) 956‐4283 Hawai‘i Hall 107 2500 Campus Road. only” options? 8 You may use these materials only for nonprofit educational purposes. Please give credit/cite appropriately 4 . HI 96822                                                                                                            manoa. x  0 7 BEFORE STATISTICAL ITEM ANALYSIS Editorial Review of content: • Unintentional clues avoided? • Distractors plausible? • Redundancy avoided in the options? • Ordering of the options carefully considered? • Correct answers randomly assigned? • Eliminated “none of the above” and “a.edu/assessment ANATOMY OF A MULTIPLE CHOICE ITEM Which city is the U.                                                                                            (808) 956‐6669 Honolulu. and b.

intuition. (Post-test) • 0. • Background knowledge.hawaii. 5. Please give credit/cite appropriately 5 . guessing 2. Assessing unintended knowledge/skills. Assessment Office                                                                                                             (808) 956‐4283 Hawai‘i Hall 107 2500 Campus Road. 6. HI 96822                                                                                                            manoa. Confusing stem or options. 10 You may use these materials only for nonprofit educational purposes. 3.50: half answered correctly.                                                                                            (808) 956‐6669 Honolulu. A distractor is too close to being correct. Test-taking strategies alone can answer an item correctly 4. (Pre-test) • 1: everyone answered correctly. Not assessing target outcomes/knowledge/skills. • Range: 0 – 1 • 0: no one answered correctly. ITEM FACILITY 9 • Definition: % of students who answered the item correctly. Multiple correct answers.edu/assessment How Can a Multiple-Choice Item Go Wrong? 1.

7 1.hawaii.9 0.edu/assessment ITEM FACILITY = NO.                                                                                            (808) 956‐6669 Honolulu. Please give credit/cite appropriately 6 . CORRECT / TOTAL Students Item1 Item2 Item3 Item4 Item5 01_Robert 1 1 1 1 1 02_Millie 1 0 1 1 1 03_Dean 1 0 0 1 1 04_Shenan 1 1 0 1 1 05_Cuny 1 9 1 1 1 1 06_Corky 10 1 0 1 1 1 07_Randy 1 1 0 1 1 08_Jeanne 1 1 0 0 1 09_Iliana 1 1 1 0 1 10_Lindsey 0 0 0 0 1 Item Facility = 0.5 0.and post-test comparison – Difference index • Contrasting group comparison – B-index 12 You may use these materials only for nonprofit educational purposes.6 0. HI 96822                                                                                                            manoa. Assessment Office                                                                                                             (808) 956‐4283 Hawai‘i Hall 107 2500 Campus Road.0 9/10 11 COMPARING PERFORMANCE ON ITEMS • Pre.

1 to 1 • -1: Post – all x.IFPRE 13 Items 1 2 3 4 5 Pre Post Pre Post Pre Post Pre Post Pre Post Monica 0 1 1 1 0 0 1 0 0 1 Yao 0 1 0 1 0 0 1 0 1 1 Dick 0 1 1 1 0 0 1 0 0 1 Karen 0 1 0 1 0 0 1 0 0 1 IF 0.0 0.0 DI 1.25 1. HI 96822                                                                                                            manoa.0 0.                                                                                            (808) 956‐6669 Honolulu. DI = IFpost .0 1. Please give credit/cite appropriately 7 . Pre – all √  • 1 : Post – all √. Pre – all x  • Acceptable value: higher than 0 DIFFERENCE INDEX = IFPOST .hawaii.5 1.0 1.0 0.0 0.75 14 You may use these materials only for nonprofit educational purposes.IFpre • Possible range: .and post-tests.0 -1.edu/assessment DIFFERENCE INDEX (DI) • Definition: The difference in item facility between the pre. Assessment Office                                                                                                             (808) 956‐4283 Hawai‘i Hall 107 2500 Campus Road.5 0.0 0.0 0.0 0.

0 IFnonmaster 0.1 to 1 • -1: Masters – all x.edu/assessment B-INDEX • Definition: The difference of item facility between the those who succeeded (masters) and those who failed the test (non-masters).0 0.0 0.8 1. Non-masters – all x  • Acceptable value: higher than 0 15 B-INDEX = IFMASTER – IFNON-MASTER Students Item1 Item2 Item3 Item4 Total 01_Robert 1 0 1 1 90 02_Millie 1 0 1 1 85 03_Dean 1 0 1 1 80 04_Shenan 1 0 0 1 80 05_Cuny 1 0 1 1 70 70% cut-point 06_Jeanne 0 1 0 1 60 07_Iliana 0 1 1 1 40 08_Lindsey 0 1 0 1 20 IFmaster 1.                                                                                            (808) 956‐6669 Honolulu.0 0.5 0.0 -1. HI 96822                                                                                                            manoa.hawaii. B-index = IFmaster – IFnon-master • Possible range: . Non-masters – all √  • 1 : Masters – all √.0 1.0 B‐index 1. Assessment Office                                                                                                             (808) 956‐4283 Hawai‘i Hall 107 2500 Campus Road. Please give credit/cite appropriately 8 .0 0.3 1.0 16 You may use these materials only for nonprofit educational purposes.

• Attribute: plausible but incorrect • Distractor efficiency index: % who select that option • A good distractor will: • Attract none of the masters.edu/assessment EXCEL PRACTICE 1 DISTRACTOR ANALYSIS 17 • Function of a distractor: to attract students who do not know the correct answer. Please give credit/cite appropriately 9 .hawaii. Assessment Office                                                                                                             (808) 956‐4283 Hawai‘i Hall 107 2500 Campus Road. or fewer masters than non-masters • Attract non-masters at a random chance level (33% for a 4-choice item) 18 You may use these materials only for nonprofit educational purposes.                                                                                            (808) 956‐6669 Honolulu. HI 96822                                                                                                            manoa.

 HI 96822                                                                                                            manoa.                                                                                            (808) 956‐6669 Honolulu. Assessment Office                                                                                                             (808) 956‐4283 Hawai‘i Hall 107 2500 Campus Road. Please give credit/cite appropriately 10 .edu/assessment DISTRACTOR ANALYSIS EXERCISE HANDOUT 1 19 EXCEL PRACTICE 2 20 You may use these materials only for nonprofit educational purposes.hawaii.

edu/people/brooksg/tap_download.assess. Please give credit/cite appropriately 11 .php?productid=417 &download=1&url=Iteman4212. CITAS (free): http://www.hawaii.zip .edu/assessment ITEM ANALYSIS SOFTWARE .aspx 21 22 You may use these materials only for nonprofit educational purposes. TAP (free): http://www.php?productid=407 .com/xcart/product.assess.ohio. HI 96822                                                                                                            manoa.                                                                                            (808) 956‐6669 Honolulu. Iteman 4 (demo version limited to 50 items and 50 examinees): http://www. Web-based Attainment Calculator: http://attainmentcalculator.net/Start. Assessment Office                                                                                                             (808) 956‐4283 Hawai‘i Hall 107 2500 Campus Road.htm .com/xcart/product.fancyfoxpublications.

Assessment Office                                                                                                             (808) 956‐4283 Hawai‘i Hall 107 2500 Campus Road.                                                                                            (808) 956‐6669 Honolulu.edu/assessment Input keys for each item Number of options per item “Y” to include items for analysis 23 24 You may use these materials only for nonprofit educational purposes. Please give credit/cite appropriately 12 .hawaii. HI 96822                                                                                                            manoa.

p (proportion) value • Difference Index  Instruction sensitive item analysis (Crocker & Algina. Assessment Office                                                                                                             (808) 956‐4283 Hawai‘i Hall 107 2500 Campus Road. 1996) • B-index  Differential group strategy (Brown.                                                                                            (808) 956‐6669 Honolulu. 2008). HI 96822                                                                                                            manoa. Please give credit/cite appropriately 13 .edu/assessment ALTERNATIVE TERMS 25 • Item Facility  item difficulty. intervention strategy (Brown.hawaii. 1996) • Distractor analysis distractor efficiency analysis 26 You may use these materials only for nonprofit educational purposes.

 2013  Excel Practice 1  Part 1: Calculating Item Facility (IF)  1.  2. Each item occupies a column.  Enter this formula in Cell B18. Calculate the difference by entering  the following formula for Item 1 in Cell F3: =B3‐D3. Copy and paste this cell to C18 to K18. Observe that the students’ scores are already sorted in the descending order.  3.      . and paste to the cells for the rest of the items. type the formula: =AVERAGE(B19:B24). Select cells B25. Calculate IFmaster for Item 1: In Cell B25. Student names or IDs should be in Column A. The column ( : ) translates as “to. The last column  shows the total score for each student. B2:B17 specifies the data range  starting from B2 and ending at B17. The formula says: deduct the value in D3  from the value in B3.   3.   4.  4. Calculate B‐index for Item 1: In Cell B27. Sort the data on the Total variable in the descending order (largest to smallest). type the formula: =AVERAGE(B4:B17).  An item is scored 1 if it is correctly answered and 0 for an incorrect answer.   5. and B27. There are several ways to calculate item facility in Excel. The more intuitive way is to use  formula:  = SUM(B2:B17)/16.  2. Open the sheet ID_Data in your data file. SUM is a  function to add up all the values specified in the parenthesis. type the formula: =B25‐B26  6. Copy the formula in Cell F3 to the rest of the items (ranging from F4 to F22).  7.” The back slash ( / )is a  division symbol. B26.  Part 3: Calculating B‐Index  1.                 Assessment Office   Item Analysis Workshop   Mar 1. Examine the results and identify the best and worst items if these items are used on a post‐test. Prepare your data. The IFs are already calculated for the pre‐ and post‐tests.   4. Calculate IFnon‐master for Item 1: In Cell B26. An Excel formula starts with the equal sign ( = ). Examine the results and identify the best and worst items. copy.  3.  5. Open the sheet B‐Index_Data in your data file. Examine the results and identify the best and worst items.  Part 2: Calculating Difference Index  1.  2. Those who passed  the course and those who failed have been identified and separated into two groups: masters  and non‐masters. Open the sheet IF_Data in your data file: Item_Analysis_Practice1.

 You can see the results automatically updated at the bottom portion. 2013  Excel Practice II: Distractor Analysis  1.  3. Copy the distractor efficiency indices (Row 1008 to Row 1011) and paste special as values and  number format + Transpose in the sheet Report. The template allows for up to 1000  examinees and up to 300 items.  2. The top window from Row 1 to  Row 1001 is the section that you can paste your own data. Paste the data in the top portion of the second sheet  calculation_template.                   .                 Assessment Office   Item Analysis Workshop   Mar 1. Open the sheet distractor in the file distractor analysis template. Be sure to include the first row  with column headers.   4. Copy the data including ids and responses for Item 1 to Item 10. This sheet has the raw data. The distractor efficiency indices are calculated automatically in  row 1008 to Row 111. The sheet calculation_template was split into two windows.

00  0.60*  0.37  0.00  0.80*  0.00  Low  0.60*  10  0.63  High  0. 72).34  0.20*  0.08  0.00*  0.00  2  0.32  0.  c.80  0.00  0.12  Low  1.20  3  0.07  0.13  0.  Item Number  1  0.00  1.00  Low  0. p.00  0.      .11  Low  0.38  High  0.84  0.60*  0.50  High  0.  d.00  0.00  0.08  0.32  4  0.20*  Low  0.13  0.34  5  1.20  0.49  0.00  High  0.00*  0.13  0.00  High  0.72  0.03        Low  0.56  High  0.00  0.44  High  0.60*  0.                 Assessment Office   Item Analysis Workshop   Mar 1.00*  6  0.12  0.27  0.00  Low  0.    Adapted from Brown (1996.  b.08  0.00  0.13  0.21  9  0.40*  0.44  High  0.00*  Low  0.17  0.80*  0.00*  0.00  0.00  8  0.50  High  1.00  0.21  0.31  7  0.20  0.15  Low  0.19  0.40  0.00  0. 2013  Distractor Analysis Exercise Handout  Distractor Efficiency  Options  IF  Group  a.00*  0.94  High  1.14  0.00  Low  0.46  *Correct option.00*  0.13  0.06  0.80*  0.00  0.20*  0.00  0.00  0.00*  0.00  1.00  0.00*  0.12  0.

  Crocker. Testing in language programs.).”A”)  Count all the ocurrencies of text A in the  RANGE. Specify data range in the parenthesis.D.). Upper Saddle River.info/statistics/excel.d. (n. Test item analysis using Microsoft Excel spreadsheet program.d.d. Mason. Basic concepts in item and test analysis. J. C. OH: Cengage  Learning.html  Matlock‐Hetzel. Retrieved from  http://www. (2008). & Algina.net/ft/tamu/Espy.  =CELL A/CELL B  =B2/B17  The value in B2 divided by the value in B17  Common rules of Excel formula:  1.).html  Fulcher. NJ: Prentice Hall. (n.    Resources  Brown. don’t use the  quotation mark around it. Excel spreadsheets for classical test analysis. J. Item & distracter analysis[PowerPoint slides].slideshare. Introduction to classical & modern test theory..   =COUNTIF(DATA  =COUNTIF(B2:B17. Retrieved from  http://languagetesting.com/elvin/publications/2003/itemanalysis. 2013  List of Item Analysis Indices Covered in the Workshop  Item Analysis  Definition in Words  Calculation Formula  Index  Item Facility (IF)  % of students who answered the item    correctly  Difference Index  Difference in IFs between the pre and post‐ = IFpost ‐ IFpre  (DI)  tests  B‐Index  Difference in IFs between the masters and  = IFmaster – IFnon‐master  non‐masters  Distractor  % of examines who chose that option    Efficiency Index    List of Excel Formulas Covered in the workshop  Formula  Example  Explanation  =SUM(DATA RANGE)  =SUM(B2:B17)  Add up the values in the data range from  B2 to B17  =AVERAGE(DATA RANGE)  =AVERAGE(B2:B17)  Average the values in the data range from  B2 to B17  =COUNTA(DATA RANGE)  =COUNTA(B2:B17)  Count the number of text values in the  data range from B2 to B17. (1997).  2.net/suequirante/item‐and‐distractor‐analysis  . (1996).htm  Quirante. Retrieved from  http://www. Always start with the equal sign ( = ). L. Retrieved from  http://www. S.  Elvin.ericae.  If the criterion is a number. G.                 Assessment Office   Item Analysis Workshop   Mar 1. S.CRITERION)  data range from B2 to B17.(n.eflclub.