Eliciting Expert Judgment: ENMG 698c: Special Topics - Project Risk Management - Dr. Rayan Khraibani

Lecture 4
Eliciting Expert Judgment
ENMG 698c: Special Topics - Project Risk Management - Dr. Rayan Khraibani
Copyright © 2020-2021, American University of Beirut, All Rights Reserved

Outline
▪ Role of experts in project risk management
▪ Expert opinions from the class
▪ Best practices
▪ Estimating parameters
▪ Assessing subjective probabilities
▪ Heuristics and bias
▪ Obtaining risk and time preferences
▪ Combining opinions from various experts
▪ Assessment protocol
2
Role of Experts in Project Risk
Management
3
4
Experts
▪ Technical experts
▪ Knowledge experts
▪ Data availability / quality / relevance
▪ Risk, uncertainty and probability
▪ Information on preferences of stakeholders
▪ Communication, confirmation, acceptance
▪ Conflicting beliefs / objectives / vested interests
5
Elicitation
▪ A data collection technique used to gather knowledge or
information from people
▪ Consists of interactions (dialogue) between the elicitor (facilitator)
and the expert(s)
▪ Requires a significant investment – even when data is available
▪ Provides subjective, scientific data.
6
Elicitation Techniques
▪ Interviews
▪ Brain Storming
▪ Focus Groups
▪ Workshops
▪ Prototypes
▪ Observation
▪ Surveys
7
Objective
▪ Represent the knowledge of one or more persons (experts) concerning an
uncertain quantity as
▪ A range of possibilities
▪ A probability distribution
▪ Represent the preferences of one or more persons (stakeholders) as
▪ A parametric value
▪ A trade-off.
8
Expert Opinions from the Class
9
Best Practices
10
Steps for Eliciting Experts Judgments
▪ Identify and train experts
▪ Build an evidence dossier
▪ Organize the interaction (interview, workshop) with a trained
facilitator
▪ Confirm the results
▪ Record and document the process.
11
Resources
▪ Well-established literature
▪ Decision making
▪ Clinical research
▪ Defense & civil protection
▪ Psychology
▪ Recognized cognitive bias
12
Best Practices
▪ Experts
▪ Explicit selection process and criteria
▪ Various domains and outlooks
▪ Avoid prejudice, bias, …
▪ Careful, honest, objective, informed
▪ Questions
▪ Clearly defined
▪ Decomposed into simple, easy to grasp elements
13
Best Practices
▪Documentation
▪ Log and summary
▪ Include experts’ conceptual model and assumptions
▪ List references
▪ Document changes (if any) in the evidence dossier
14
Best Practices
▪ Facilitation
▪ Follow a rigorous protocol
▪ Eliminate prejudice, bias, superstition, wishful thinking, hastiness, guessing,
manipulation…
▪ Dialogue
▪ Avoid cognitive biases
▪ Obtain quantitative judgments, avoid ambiguity
▪ Facilitate discussion
▪ Provide the « rational impartial observer opinion »
▪ Strive for consensus
15
Estimating Parameters
16
Uncertain Parameters
▪Meaning?
Proportion of defective matches in the batch
17
Assessing Subjective Probabilities
18
Various Approaches
▪ Fixed value
▪ estimate the probability of being higher or lower than a selected value,
or of discrete events
▪ Fixed probability
▪ estimate the value associated with a specific probability (a fractile)
19
Probabilities
▪ Translating beliefs into probabilities is a difficult exercise and
experts usually need to be trained
▪ cognitive references
▪ visual aids
▪ expressions
▪ Rare events are even more difficult to assess
▪ experience, heuristics
▪ The impact of rare event probabilities is often significant.
20
Assessing Discrete Probabilities
▪ The probability wheel
▪ measures probabilities for people who are uncomfortable with assigning numbers to uncertainty
▪ consists of a disk shaded in two colors, with a spinning pointer
▪ the colors, blue and orange, can be adjusted so that the orange portion goes from being a tiny sliver, 1 percent, all the way up to
99 percent of the disk
▪ Example: the probability assessor is trying to assess his feelings about the probability that the monarchy in Spain will be
overthrown next year
▪ Ask him to adjust the amount of orange (either decreasing or increasing it) until the wheel is configured so that the
assessor believes that it is as likely that the monarchy in Spain will be overthrown as that the pointer on the wheel when
spun will stop on the orange color.
▪ Then, look at the back of the wheel where a scale shows what proportion of the wheel is orange.
▪ This value is taken to be the probability that the wheel will land on orange and is taken to be the assessor's probability
that the Spanish monarchy will be overthrown.
21
Assessing Continuous Probabilities
▪Build the cumulative distribution

▪ What is the probability of getting a grade higher than 80?
▪ What is the grade that I have 95% chances of getting better?
▪ What is the grade which is as probable of being under than
being over?
22
Assessing Continuous Probabilities
▪ Often, three points are enough to approximate the cumulative
distribution
▪ In some cases, tails are important and care should be taken in

estimating extreme cases
▪ Because of some cognitive biases, it is better to work with fractiles

than with quantities
23
A common “simplistic" approach to sensitivity analysis is to vary uncertain variables by the
same percentage (e.g., +/-10%). A probabilistic/ DA approach is preferred.
The simplistic approach produces the sensitivity to changes in the uncertainties, but does not show how wide the
uncertainty ranges are.

For continuous uncertainties, the “low” &“high” values are defined using 10-90
percentiles.
Value for Sensitivity

Definition Name
Analysis
There is only a 10 percent P10

Low probability that the variable will 10th percentile
be less than or equal to this
value.
High There is only a 10 percent P90

probability that the variable will 90th percentile
be greater than this value.
Mid (Base case) There is a 50 percent probability P50

that the variable will be less than 50th percentile (median)
or equal to this value.
*The P10-90 range is the expert’s 80% confidence range.
25
Uncertainty Assessments
Why Assessment?
▪ Because many of our uncertainties are subjective, we need to elicit
information about them using an assessment process (i.e., you can’t look it
up in a book).
Ways to do it
▪ Interviews, surveys, interpretation of historical data...
Role of “Experts”
▪ Because we rely on expert knowledge, and most experts are human, we
need to concern ourselves with psychology and biases as well as the
technical merits of the expert.
26
Heuristics & Bias
29
Heuristics
▪ Mental processes used to function efficiently (short cuts)
▪ Run before thinking
▪ Make decisions in familiar situations

▪ Itinerary to go to AUB from home
▪ Sources of errors when assessing critical data

▪ Disregard the possibility of rare events
30
Cognitive Biases
▪A large number of documented instances
▪Often the result of heuristics
▪Can be minimized by using well-established methods

▪ The way a question is framed can influence the answer
▪ The format of the answer can influence the result
31
Availability
▪ Probability is related to the memory of similar events
▪ More memorable = more probable
▪ Memory can be due to media coverage
▪ Are car accidents more probable than stomach cancer?
▪ Or impact
▪ Are car accidents more probable just after you had one?
▪ Mitigating technique: provide evidence
32
Other Memory Biases
▪ Anchoring: the probability of an event depends on a single piece of
information (possibly the question itself)
▪ Imaginability: the probability is related to the ease with which it
can be imagined
▪ Illusory correlations: the probability is based on a fortuitous
occurrence of events
▪ Hindsight: the probability of an event seems higher after it
happened
33
Confidence Biases
▪ Desire: probability of a desired outcome is overestimated
▪ Selectivity: disregarding information that is inconsistent with the
expert’s experience
▪ Conservatism: new information is ignored
▪ Regression: probability is based on recent events
▪ Shy from extremes
35
Motivation Biases
▪ Assessments that are made and do not reflect the experts’ true
beliefs
▪ Second-guess the answer wanted by the facilitator
▪ Please the stakeholders or prepare them for bad outcomes
▪ Appear knowledgeable
▪ Conflict of interest.
36
Readings
▪ Brown, G. E. (2019). Eliciting Expert Opinion in Acquisition Cost and
Schedule Estimating.
▪ Szwed, P. (2016, April). Expert Judgment in Project Management: Narrowing
the Theory-Practice Gap. Project Management Institute.
▪ Hora, S. C. (2007). Eliciting probabilities from experts. Advances in decision
analysis: From foundations to applications, 129.
37

Eliciting Expert Judgment: ENMG 698c: Special Topics - Project Risk Management - Dr. Rayan Khraibani

Uploaded by

Document Information

Original Title

Copyright

Available Formats

Share this document

Share or Embed Document

Sharing Options

Did you find this document useful?

Is this content inappropriate?

Copyright:

Available Formats

Eliciting Expert Judgment: ENMG 698c: Special Topics - Project Risk Management - Dr. Rayan Khraibani

Uploaded by

Copyright:

Available Formats

Lecture 4

Eliciting Expert Judgment

Copyright © 2020-2021, American University of Beirut, All Rights Reserved

▪ Recognized cognitive bias

Proportion of defective matches in the batch

▪ consists of a disk shaded in two colors, with a spinning pointer

▪Build the cumulative distribution

▪ In some cases, tails are important and care should be taken in

▪ Because of some cognitive biases, it is better to work with fractiles

Copyright © 2020-2021, American University of Beirut, All Rights Reserved

Value for Sensitivity

There is only a 10 percent P10

High There is only a 10 percent P90

Mid (Base case) There is a 50 percent probability P50

▪ Make decisions in familiar situations

▪ Sources of errors when assessing critical data

▪A large number of documented instances

▪Often the result of heuristics

▪Can be minimized by using well-established methods

You might also like