This action might not be possible to undo. Are you sure you want to continue?

**Decision Analysis I September 30
**

th

. 2004

Page 1 oI 5 Probabilistic Question InIo.

& Grading Policy

Probabilistic Question Information & Grading Policy

Multiple-choice questions are oIten administered in large classes to assess students` abilities to

recognize correct answers to questions that are essentially deIinitions or calculations. Students

presumably pick the answer that they believe is most likely to be correct. When you pick your

answer. you are making a decision about which answer is most likely to be correct.

In MS&E 252. we will encourage you to practice and reIlect on the decision processes you

employ to choose your responses. The multiple choice Iormat we employ as part oI this

encouragement works because it does more than iust ask you to pick the answer you believe most

likely to be correct. That would only give us a crude picture oI what you know or don`t know.

On a typical multiple choice test with Iour answers. Ior example. your response indicates what

you believe about the correctness oI one answer. In the case where you picked a wrong answer.

we don`t learn much about what you believed about the correct answer.

In some settings. knowledge assessors have used 'strictly proper¨ scoring Iunctions to elicit the

belieIs oI respondents Ior each answer on a multiple choice question. The respondents are

invited to assign a probability oI each answer being correct. Their scores are calculated as a

Iunction oI their likelihood or probability assignment; when the Iunction is 'strictly proper.¨

(e.g.. logarithmic). the respondents maximize their expected score when their probability

assignments reIlect their true belieIs.

Using such a probabilistic multiple choice Iormat. we intend to provide opportunities Ior you to

reveal how well you know the course material; and opportunities to practice deciding how to

allocate your resources on quiz and exam questions. guided by new insights Irom the course.

Example (Ior illustration only; real questions will Ieature concepts learned in the course):

The title oI MS&E listed in the 2004 Time Schedule is:

a. Intro to Decision Analysis

b. Decision Analysis I

c. Introduction to Decision Analysis

d. None oI the above

|Thinking time: resist the temptation to look up the answer. Instead. rank the answers in order

Irom least to most likely to be correct. then assign weights to them that indicate the relative

likelihood oI correctness oI a. b. c. and d. What probabilities are associated with the relative

likelihood you assigned? Answer beIore you turn the page!|

Your response

Question a. b. c. d.

Probability

MS&E 252 Handout #3

Decision Analysis I September 30

th

. 2004

Page 2 oI 5 Probabilistic Question InIo.

& Grading Policy

The way this question might be thought about by Allen. our sample student. is the Iollowing:

answer b strikes Allen as most likely to be correct. Iollowed by c. Allen also believes a and d to

be equally likely to be correct. Furthermore. b is about twice as likely to be correct as c. and c is

about twice as likely to be correct as a and d. This reasoning leads Allen to assign probabilities

that correspond to the ratios in the Iollowing manner:

Question a. b. c. d.

Probability 0.125 0.5 0.25 0.125

The strictly proper scoring Iunction used in MS&E 252 is a logarithmic Iunction parameterized

so that the top score Ior any given question is 1 and so that a uniIorm distribution oI uncertainty

(reIlecting a claim oI ignorance) gets zero. Notice that regardless oI the parameters chosen.

putting zero probability on an answer that is actually 'correct¨ earns a score oI negative inIinity.

This is a consequence oI the scoring Iunction`s property not to provide any incentive Ior

someone to misrepresent their belieIs in order to maximize their score. II your answers do not

add up to one. they will be normalized so that they do add up to one.

The scoring Iunction used to assess perIormances on probabilistic homework and exams in

MS&E 252 is:

Score ÷ 1 + ln(p)/ln(4)

Where p is the probability assigned to the correct answer.

In the example above. iI the correct answer is a. Allen would earn a score oI -0.5

In the example above. iI the correct answer is b. Allen would earn a score oI 0.5

In the example above. iI the correct answer is c. Allen would earn a score oI 0

In the example above. iI the correct answer is d. Allen would earn a score oI -0.5

Now would be a good time to look up the real answer and grade yourselI. How well did you do?

A part oI every homework assignment will be in the probabilistic Iormat. In addition. the

midterm will consist oI 15 probabilistic multiple-choice questions and the Iinal will be comprised

oI both probabilistic multiple-choice questions and short answer questions. The homework is

designed Ior you to have a chance to practice assigning probabilities to these types oI questions

and become Iamiliar with the process in order to prepare you Ior both the midterm and the Iinal.

We will be going over this inIormation in section and throughout the quarter. You will have

many opportunities to ask questions and seek advice beIore the midterm. We believe your

learning experience oI decision analysis will be signiIicantly enhanced by these opportunities to

make these personal decisions about how to portray your knowledge.

MS&E 252 Handout #3

Decision Analysis I September 30

th

. 2004

Page 3 oI 5 Probabilistic Question InIo.

& Grading Policy

Probaility Assignment Score Probaility Assignment Score

0.0001 -5.6439 0.51 0.514285

0.01 -2.3219 0.53 0.542032

0.03 -1.5294 0.55 0.568752

0.05 -1.161 0.57 0.594517

0.07 -0.9183 0.59 0.619393

0.09 -0.737 0.61 0.643441

0.11 -0.5922 0.63 0.666712

0.13 -0.4717 0.65 0.689256

0.15 -0.3685 0.67 0.711117

0.17 -0.2782 0.69 0.732334

0.19 -0.198 0.71 0.752945

0.21 -0.1258 0.73 0.772984

0.23 -0.0601 0.75 0.792481

0.25 0 0.77 0.811465

0.27 0.05552 0.79 0.829962

0.29 0.10706 0.81 0.847997

0.31 0.15517 0.83 0.865592

0.33 0.20027 0.85 0.882767

0.35 0.24271 0.87 0.899544

0.37 0.2828 0.89 0.915939

0.39 0.32077 0.91 0.931969

0.41 0.35685 0.93 0.947651

0.43 0.3912 0.95 0.963

0.45 0.424 0.97 0.978028

0.47 0.45537 0.99 0.99275

0.49 0.48543 1 1

Logarithmic Score

-6

-4

-2

0

2

0 0.2 0.4 0.6 0.8 1

Probability Assigned

S

c

o

r

e

MS&E 252 Handout #3

Decision Analysis I September 30

th

. 2004

Page 4 oI 5 Probabilistic Question InIo.

& Grading Policy

Grading Policy

Homework Policy

• We will tell you which sections you are to turn in with every homework assignment.

• ' Food Ior thought¨ and ' Probabilistic¨ questions are to be turned in on the class home page.

• ' Food Ior thought¨ questions are to provide you with the opportunity to ponder on some oI

the concepts covered in class. They will also give you the chance to apply some oI those

concepts practically. They are optional (i.e.. you do not need to turn them in. but they will

help us grade you on participation).

• To allow time Ior your learning curve Ior answering probabilistic questions. the second

homework score will be max(0. your score).

• There will be no dropping oI any homework assignment score.

Probabilistic Multiple Choice Grading Policy

For each probabilistic multiple-choice question. your score will be computed as Iollows:

Let be the probability you assign to the correct answer. II ~0

Score ÷ 1 ¹

4 ln

ln

If vou assign a probabilitv of zero to a correct answer, vou will receive a

score of zero for that particular homework question or exam question.

Additionallv, vour final course grade will be lowered 2 marks. For

example, if vou initiallv earned a B+, vour final grade will be lowered to a

B-. Please be aware that a grade strictlv less than C- will result in failing

the class.

The scoring Iunction is designed such that the best strategy is to assign the probabilities that most

accurately represent your state oI inIormation. WE STRONGLY RECOMMEND NEVER

ASSIGNING PROBABILITY ZERO TO ANY ANSWER!

Each probabilistic multiple-choice question will be graded as Iollows:

II your probabilities do not add up to one. the teaching team will normalize them. II you leave

boxes blank and your probabilities add up to less than one. we will distribute the remaining

probability equally among the empty answer boxes. II your probabilities add up to more than one

and you have leIt boxes blank we will normalize the non-blank boxes and place zeros in the

empty boxes. II you leave an entire question blank it will be interpreted as an assignment oI 0.25

Ior each oI the Iour possible choices.

MS&E 252 Handout #3

Decision Analysis I September 30

th

. 2004

Page 5 oI 5 Probabilistic Question InIo.

& Grading Policy

For example.

Question Number a b c d

I 0.3 0.5 0.3 0.3

J 0.7

K 0.9 0.2

L

Would become.

Question Number a b c d

I 0.214 0.357 0.214 0.214

J 0.1 0.1 0.7 0.1

K 0.82 0.18 0 0

L 0.25 0.25 0.25 0.25

We will distribute an Excel macro where you can practice assigning diIIerent

probabilities and obtain your grade.

- ProblemSession7
- ProblemSession1
- ProblemSession3
- ProblemSession5
- ProblemSession4
- HO15-HW5
- ProblemSession6
- Midterm
- ProblemSession2
- HO10-PracticeMidterm
- HO28-HW7Solutions
- HO29-OtherDAClasses
- MidtermSolutions
- HO26-PracticeFinal1Sol
- HO8-HW2Solutions
- HO25-PracticeFinal1
- HO19-HW5Solutions
- HO2-HW1
- HO12-HW3Solutions
- HO5-HW2 v2
- HO17-CaseInstructions
- HO23-HW6Solutions
- HO26b-PF1Sols Erratum
- Final Instructions Part2
- HO27-PracticeFinal2WS

Sign up to vote on this title

UsefulNot usefulMS&E 252

MS&E 252

- ProblemSession4
- ProblemSession3
- HO15-HW5
- MidtermSolutions
- HO5-HW2 v2
- HO18-HW6
- ProblemSession2
- ProblemSession6
- ProblemSession8
- HO29-OtherDAClasses
- HO13-HW4
- HO10-PracticeMidterm
- HO16-HW4Solutions
- HO22-HW7
- HO8-HW2Solutions
- HO17-CaseInstructions
- HO28-HW7Solutions
- HO7-HW3
- HO19-HW5Solutions
- HO12-HW3Solutions
- ProblemSession7
- HO20-Decision Diagrams
- ProblemSession1
- ProblemSession5
- HO1-guide
- HO27-PracticeFinal2WS
- HO23-HW6Solutions
- HO26-PracticeFinal1Sol
- hmk2 sol
- HO25-PracticeFinal1
- HO3-ProbQ