MS&E 252 Handout #3

Decision Analysis I September 30
th
. 2004


Page 1 oI 5 Probabilistic Question InIo.
& Grading Policy

Probabilistic Question Information & Grading Policy

Multiple-choice questions are oIten administered in large classes to assess students` abilities to
recognize correct answers to questions that are essentially deIinitions or calculations. Students
presumably pick the answer that they believe is most likely to be correct. When you pick your
answer. you are making a decision about which answer is most likely to be correct.

In MS&E 252. we will encourage you to practice and reIlect on the decision processes you
employ to choose your responses. The multiple choice Iormat we employ as part oI this
encouragement works because it does more than iust ask you to pick the answer you believe most
likely to be correct. That would only give us a crude picture oI what you know or don`t know.
On a typical multiple choice test with Iour answers. Ior example. your response indicates what
you believe about the correctness oI one answer. In the case where you picked a wrong answer.
we don`t learn much about what you believed about the correct answer.

In some settings. knowledge assessors have used 'strictly proper¨ scoring Iunctions to elicit the
belieIs oI respondents Ior each answer on a multiple choice question. The respondents are
invited to assign a probability oI each answer being correct. Their scores are calculated as a
Iunction oI their likelihood or probability assignment; when the Iunction is 'strictly proper.¨
(e.g.. logarithmic). the respondents maximize their expected score when their probability
assignments reIlect their true belieIs.

Using such a probabilistic multiple choice Iormat. we intend to provide opportunities Ior you to
reveal how well you know the course material; and opportunities to practice deciding how to
allocate your resources on quiz and exam questions. guided by new insights Irom the course.

Example (Ior illustration only; real questions will Ieature concepts learned in the course):

The title oI MS&E listed in the 2004 Time Schedule is:

a. Intro to Decision Analysis
b. Decision Analysis I
c. Introduction to Decision Analysis
d. None oI the above


|Thinking time: resist the temptation to look up the answer. Instead. rank the answers in order
Irom least to most likely to be correct. then assign weights to them that indicate the relative
likelihood oI correctness oI a. b. c. and d. What probabilities are associated with the relative
likelihood you assigned? Answer beIore you turn the page!|

Your response

Question a. b. c. d.
Probability

MS&E 252 Handout #3
Decision Analysis I September 30
th
. 2004


Page 2 oI 5 Probabilistic Question InIo.
& Grading Policy

The way this question might be thought about by Allen. our sample student. is the Iollowing:
answer b strikes Allen as most likely to be correct. Iollowed by c. Allen also believes a and d to
be equally likely to be correct. Furthermore. b is about twice as likely to be correct as c. and c is
about twice as likely to be correct as a and d. This reasoning leads Allen to assign probabilities
that correspond to the ratios in the Iollowing manner:


Question a. b. c. d.
Probability 0.125 0.5 0.25 0.125

The strictly proper scoring Iunction used in MS&E 252 is a logarithmic Iunction parameterized
so that the top score Ior any given question is 1 and so that a uniIorm distribution oI uncertainty
(reIlecting a claim oI ignorance) gets zero. Notice that regardless oI the parameters chosen.
putting zero probability on an answer that is actually 'correct¨ earns a score oI negative inIinity.
This is a consequence oI the scoring Iunction`s property not to provide any incentive Ior
someone to misrepresent their belieIs in order to maximize their score. II your answers do not
add up to one. they will be normalized so that they do add up to one.

The scoring Iunction used to assess perIormances on probabilistic homework and exams in
MS&E 252 is:
Score ÷ 1 + ln(p)/ln(4)
Where p is the probability assigned to the correct answer.

In the example above. iI the correct answer is a. Allen would earn a score oI -0.5
In the example above. iI the correct answer is b. Allen would earn a score oI 0.5
In the example above. iI the correct answer is c. Allen would earn a score oI 0
In the example above. iI the correct answer is d. Allen would earn a score oI -0.5

Now would be a good time to look up the real answer and grade yourselI. How well did you do?
A part oI every homework assignment will be in the probabilistic Iormat. In addition. the
midterm will consist oI 15 probabilistic multiple-choice questions and the Iinal will be comprised
oI both probabilistic multiple-choice questions and short answer questions. The homework is
designed Ior you to have a chance to practice assigning probabilities to these types oI questions
and become Iamiliar with the process in order to prepare you Ior both the midterm and the Iinal.
We will be going over this inIormation in section and throughout the quarter. You will have
many opportunities to ask questions and seek advice beIore the midterm. We believe your
learning experience oI decision analysis will be signiIicantly enhanced by these opportunities to
make these personal decisions about how to portray your knowledge.

MS&E 252 Handout #3
Decision Analysis I September 30
th
. 2004


Page 3 oI 5 Probabilistic Question InIo.
& Grading Policy

Probaility Assignment Score Probaility Assignment Score
0.0001 -5.6439 0.51 0.514285
0.01 -2.3219 0.53 0.542032
0.03 -1.5294 0.55 0.568752
0.05 -1.161 0.57 0.594517
0.07 -0.9183 0.59 0.619393
0.09 -0.737 0.61 0.643441
0.11 -0.5922 0.63 0.666712
0.13 -0.4717 0.65 0.689256
0.15 -0.3685 0.67 0.711117
0.17 -0.2782 0.69 0.732334
0.19 -0.198 0.71 0.752945
0.21 -0.1258 0.73 0.772984
0.23 -0.0601 0.75 0.792481
0.25 0 0.77 0.811465
0.27 0.05552 0.79 0.829962
0.29 0.10706 0.81 0.847997
0.31 0.15517 0.83 0.865592
0.33 0.20027 0.85 0.882767
0.35 0.24271 0.87 0.899544
0.37 0.2828 0.89 0.915939
0.39 0.32077 0.91 0.931969
0.41 0.35685 0.93 0.947651
0.43 0.3912 0.95 0.963
0.45 0.424 0.97 0.978028
0.47 0.45537 0.99 0.99275
0.49 0.48543 1 1
Logarithmic Score
-6
-4
-2
0
2
0 0.2 0.4 0.6 0.8 1
Probability Assigned
S
c
o
r
e
MS&E 252 Handout #3
Decision Analysis I September 30
th
. 2004


Page 4 oI 5 Probabilistic Question InIo.
& Grading Policy
Grading Policy

Homework Policy
• We will tell you which sections you are to turn in with every homework assignment.
• ' Food Ior thought¨ and ' Probabilistic¨ questions are to be turned in on the class home page.
• ' Food Ior thought¨ questions are to provide you with the opportunity to ponder on some oI
the concepts covered in class. They will also give you the chance to apply some oI those
concepts practically. They are optional (i.e.. you do not need to turn them in. but they will
help us grade you on participation).
• To allow time Ior your learning curve Ior answering probabilistic questions. the second
homework score will be max(0. your score).
• There will be no dropping oI any homework assignment score.

Probabilistic Multiple Choice Grading Policy
For each probabilistic multiple-choice question. your score will be computed as Iollows:
Let be the probability you assign to the correct answer. II ~0
Score ÷ 1 ¹
4 ln
ln


If vou assign a probabilitv of zero to a correct answer, vou will receive a
score of zero for that particular homework question or exam question.
Additionallv, vour final course grade will be lowered 2 marks. For
example, if vou initiallv earned a B+, vour final grade will be lowered to a
B-. Please be aware that a grade strictlv less than C- will result in failing
the class.
The scoring Iunction is designed such that the best strategy is to assign the probabilities that most
accurately represent your state oI inIormation. WE STRONGLY RECOMMEND NEVER
ASSIGNING PROBABILITY ZERO TO ANY ANSWER!

Each probabilistic multiple-choice question will be graded as Iollows:

II your probabilities do not add up to one. the teaching team will normalize them. II you leave
boxes blank and your probabilities add up to less than one. we will distribute the remaining
probability equally among the empty answer boxes. II your probabilities add up to more than one
and you have leIt boxes blank we will normalize the non-blank boxes and place zeros in the
empty boxes. II you leave an entire question blank it will be interpreted as an assignment oI 0.25
Ior each oI the Iour possible choices.
MS&E 252 Handout #3
Decision Analysis I September 30
th
. 2004


Page 5 oI 5 Probabilistic Question InIo.
& Grading Policy
For example.
Question Number a b c d
I 0.3 0.5 0.3 0.3
J 0.7
K 0.9 0.2
L

Would become.
Question Number a b c d
I 0.214 0.357 0.214 0.214
J 0.1 0.1 0.7 0.1
K 0.82 0.18 0 0
L 0.25 0.25 0.25 0.25

We will distribute an Excel macro where you can practice assigning diIIerent
probabilities and obtain your grade.