7-Measurements-Drawing Valid Conclusions From Numbers

Railway engineering economy and project management
Measurements-Drawing Valid
Conclusions From Numbers
11/15/2021
1
Invalid data and poor statistical methods can

lead to bad decisions!
How you can instead use valid methods and

strong statistics so as to create credible
predictions for all of our project
management roles and measures.
2
measurement
process
3
The Fallacy of the Silent Evidence
4
5
So, in making design decisions for our engineered

systems, we must take steps so as to be sure that we
are in fact seeing all, or at least a properly
representative sample, of the evidence, else we will
fall victim to the fallacy of the silent evidence too.
6
Logical Flaws in the Organization of System Testing
We designed our tests deliberately so that the system would

pass! That is, we deliberately looked only for positive evidence.
Your test program should not be content looking only for

confirmation that – under a carefully scripted set of
conditions – the system works. You must also find the
situations where your system fails!
7
8
9
10
The point of gathering information about failures is

not to confirm our design decisions, but to give us
the earliest possible indication of the nature of
their flaws. So, don’t sweep bad news “under the
rug,” or interpret it away.
11
The Problem of Scale
Data or processing that deviates materially from the nominal

conditions, values, and/or sequences expected.
Since your system cannot avoid having to process these
outliers, there are two aspects of design that become
magnified in importance:
• 1.Getting the system to do what you want under
approximately normal conditions.
• 2.Preventing the system from doing horrible things under
severely off‐nominal conditions.
12
Signal and Noise
When we measure something, the value returned by the measurement is affected by

two factors:
1.The actual value of the item being measured.
2.Errors in the measurement process.
For example, if we measure something on 1 March and the answer is 301, and we
measure it again on 1 April and the answer is 302, then:
●●All of the change might be due to a real change in the item being measured.
●●All of the change might be due to errors in the measurement process.
●●Or (most likely) … some combination of the two!
13
• That part of the change in the measurement that is due to an

actual change in the underlying item is called the signal.
• That part of the change in the measurement that is due to
errors and variation in the measurement process is called
noise.
Every measurement contains both noise and signal; in order to get
meaningful data from our measurement, and to allow us to draw valid
conclusions from the measurements, we must learn to separate the signal
from the noise.There are many sources of noise; they combine to create a
random variation.Most people, unfortunately, do not separate signal from
noise. 14
To separate signal from noise, you look at the most recent set of
measurements (e.g. the item labeled “monthly results”) for five different
effects, which I call the Wheeler/Kazeef tests
Here are the five Wheeler/Kazeef tests:
• 1.There is one point outside the upper or lower natural process limits.
• 2.Two out of any three consecutive points are in either of the 2‐sigma to
3‐sigma zones.
• 3.Four out of any five consecutive points are outside the 1‐sigma zones.
• 4.Any seven consecutive points are on one side of the average.
• 5.Any measurement of the item called moving range is above the item
called moving range limit.
15
16
The Decision Tree: A Method That Properly Accounts For Conditional Probabilities
There is a method called the decision tree that allows one

properly to assess the expected cost of a set of alternative
courses of action.
17
The tree has two types of nodes, one called a decision node
and another called a chance node.
• Using these two types of nodes, a step‐by‐step depiction
of multiple complex courses of action can be created from
left to right, across the page.
• At the decision nodes, multiple candidate next steps in the
courses of action arise from the right‐hand side of those
nodes.
• Each line (e.g. candidate next step) that emerges from the
decision node can have a cost assigned to it.
18
19
What Engineering Project Managers Need to Measure
20
As managers of engineering projects, we need to

have our team make three types of measurements:
Technical performance measures
Operational performance measures
Management measures.
21
Implications for the Design and Management Processes
We Need Measurements in Order to Create Good

Designs ：
we can never create good designs based solely on theory, models,
and algorithms. In order to create effective designs via tinkering,
we need measurements from our models and prototypes,
providing data that allows us to refine the design, and to guide
the tinkering process.
22
Projects Provide an Opportunity for Time Series ：

On an engineering project, we automatically obtain the data
needed to create a time series. I showed you that we can
analyze the data in a time series using the process behavior
chart (also called the control chart); this chart – and the
associated five Wheeler/Kazeef tests – will help us separate
signal from noise, so we can try to make decisions based only
on the signal.
23
Interpreting the Data:
It gets progressively far more expensive to fix things as

you move from stage to stage of your project’s
development cycle. It costs a lot less – about 100 times
less – to fix something during the requirements or
design stage, as compared to fixing it during the
integration or test stage.
24
Avoid “Explaining Away” the Data
When in fact the data are a surprise.Try to obtain as

much of the data that goes into your operational and
technical performance measures as you can from
experimentation, prototyping, and other methods of
measuring actual operation.
Decrease your dependence where you can on the
predictions of theoretical models and algorithms.
25
Social Aspects of Measurement:
Good methods create credibility with your customers, and

pride inside your team. It is also the case that creating some of
these measures – certainly most of the operational
performance measures, and several of the management
measures – requires us to understand the customer and (at
times) people in general.
26
Non‐linear Effects
• In mathematics, we say that an effect is linear if a change in an input
results in a change in the output that is, at most, a multiplicative
effect (this is called linear because, if you plotted the relationship
between the input and the output, the graph would be a straight
line).
• But the behaviors of our engineered systems are seldom dominated
by such linear effects. Instead, our engineered systems usual display
multiple types of complex, unpredictable, non‐linear behavior, where
small changes in an input may result in huge changes to an output or
a behavior.
27
Sensitivity Analysis:
How can we find such non‐linear effects?
We can find them by looking for variations of the inputs that
cause the predictions to curve strongly up or down on one side as
we vary the inputs. When we find asymmetry and big variations
in predictions due to fairly small variations in inputs, this should
be investigated. Such non‐linear behavior could become a
catastrophic problem for our system.
28
Model:
These model builders often spend years collecting

data, and gathering real measurements and
observations.
29
Risk and Opportunity
Management
11/15/2021
30
Theory and Practice Railway engineering economy and project management
Text Book Definition: Risk Management is the systematic

process of identifying, analyzing, and responding to project risk.
It includes maximizing the probability and consequences of
positive events and minimizing the probability and
consequences of adverse events to project objectives.
Simple Terminology: Risk Management is the identification of

an unborn issue (a risk) that will have a negative impact on the
project that must be eliminated with proactive planning,
monitoring, and activities or tasks.
31
Technical and engineering risks can arise from all of the obvious sources:
• The invention required to create the desired system.
• The lack of maturity of some of the selected technologies and
products; there is often a gap between what a product is advertised as
capable of achieving, and what it actually can achieve.
• The scale and complexity of the project, including the problem of
managing the dynamic behavior of the system .
• All of the uncertainty and errors that can arise from quantitative
measures and analyses .
• … and many others too.
32
Technology and engineering, however, are not the exclusive sources of risk on
engineering projects. Risks can arise from other matters too, such as:
• Poorly defined or changing goals and requirements.
• Tensions between stakeholders: the buying customer, the eventual users, the
paying customer, the regulators, and other stakeholders may want slightly
different things, and trying to satisfy all of them may over‐constrain your
solution.
• Tensions within your development team: people may have different ideas about
the appropriate design; if you have subcontractors, they may have different
business aspirations from you and your company; there will always be some of
the ordinary issues of people not getting along, and/or having conflicting
personal goals and aspirations; and so forth.
33
Text Book - Risk Management Processes
Risk Identification
Risk Assessment Risk Analysis
Risk Prioritization
Risk
Management
Risk Mgmt Planning
Risk Control Risk Resolution
Risk Monitoring
34
Text Book - Risk Management Planning
DEFINITION: The process of deciding how to approach and plan

the risk management activities for a project.
Inputs Output Tools & Techniques

1. Project Charter • Risk Management Plan • Planning Meetings
2. Organizations Risk Mgmt Policy
3. Defined Roles & Responsibilities
4. Stakeholder Risk Tolerances
5. Template for the Organizations risk
management plan
6. Work breakdown structure
• Organizational Risk Management Policy: Predefined approaches to risk analysis and resolution that needs tailoring to a
particular project.
• Defined roles and responsibilities: Predefined roles, responsibilities, and authority levels for decision making will
influence planning.
• Stakeholder Risk Tolerance: Different organizations and different individuals have different tolerances for risk. Policies or
historical actions may communicate this.
• Planning Meetings: Project teams hold risk management planning meetings to develop the plan.
35
Text Book - Risk Management Plan
Risk Management Planning output is the “Risk Management Plan”
The plan describes how risk identification, qualitative and quantitative analysis,
resolution planning, monitoring, and control will be structured and performed during
the project life cycle.
The plan may include:

Methodology
Roles & Responsibilities
Budgeting
Timing
Scoring and interpretation
Thresholds
Reporting formats
Tracking
36
Text Book - Risk Identification
DEFINITION: The process of determining which risks might affect

the project and document the characteristics.
** Risk identification is an iterative process **

1. Risk management plan 1. Risks 1. Documentation reviews
2. Project planning outputs 2. Triggers 2. Information gathering
3. Risk categories 3. Inputs to other processes 3. techniques
4. Historical information 4. Checklists
5. Assumption analysis
6. Diagramming techniques
INPUTS
Risk Management Plan: See previous slide
Project Planning Outputs: Items to be reviewed, but not limited to: project charter, WBS, product description, schedule and cost estimates, resource
plan, procurement plan, assumption and constraints.
Risk Categories: Risks that may affect a project for better or worse can be identified and organized into risk categories. Categories include:
 Technical, quality, and performance risks
 Project Management risks
 Organizational Risks – cost, time, and scope
 External risks – shifting legal or regulatory environment, labor issues, natural events.
Historical Information: Information on prior projects may be available to help leverage lessons learned.
37
Risk Identification- Tools/Techniques and Outputs
Outputs
Risks: Yep! This an output!
Triggers: Sometimes called risk assumptions or warning signs, these are indications that a risk has
occurred or is about to occur.
Inputs to other processes: Risk identification may identify a need for a further action in another area or to
other projects.
Tools and Techniques

Documentation Reviews: Performing a structured review of the project plans and assumptions.
Information gathering techniques: Examples of information gathering include:
 Brainstorming – most common
 Delphi technique – Consensus of experts on a subject area
 Interviewing – Seek input from project managers or subject matter experts
 Strengths, Weaknesses, Opportunities, and Threats (SWOT) analysis
Checklist: Using historical information and knowledge is a quick and simple way to identify risk.
.
Assumption analysis: Every project is conceived and developed based on a set of hypotheses, scenarios,
or assumptions.
Diagram techniques: Cause-and-effect, System or process flow charts, and Influence diagrams
38
Text Book - Risk Analysis Railway engineering economy and project management
• Quantitative: The process of measuring the probability and consequences of

risks and estimating their implications for project objectives.
– Determine the probability of achieving a specific project objective.
– Quantify the risk exposure for the project, and determine the cost and schedule
contingency reserves that may be needed.
– Identify risks requiring the most attention by quantifying their relative contribution to
project risks.
– Identify realistic and achievable cost, schedule, and scope targets.
• Qualitative: The process of performing a qualitative analysis of risks and

conditions to prioritize their effects on project objectives.
39
Qualitative Risk - Tools and Techniques Railway engineering economy and project management
Risk probability and consequences: This is generally described in

qualitative terms such as VERY HIGH, HIGH, MODERATE, LOW, VERY
LOW.
• Risk Probability is the likelihood a risk will occur.
• Risk Consequences is the effect on the project objectives if the risk event occurs.
Probability/impact risk rating matrix: The use of risk’s probability scale

and risk’s impact scale is generally used.
• Risk scale falls between 0.0 (no probability) and 1.0 (certainty) – See next slide
• Risk impact scale reflects the severity of its effect on the project objective.
40
Rating Impacts for a Risk
Evaluating Impact of a Risk during Major Project Milestones

Project Very Low Low Moderate High Very High
Objective .05 .1 .2 .4 .8
Cost Insignificant <5% 5-10% 10 – 20% >20%
Cost Increase Increase Increase Increase Increase
Schedule Insignificant Schedule Slip Overall Prj Slip Overall Prj Slip Overall Prj
Slip
Schedule Slippage <5% 5-10% 10-20% Slips>20%
Scope Scope Reduction Minor areas of Major Areas Scope Reduction Project end item
scope affected of scope affected unacceptable to is not meeting
client business need
Quality Quality Only very Quality Reduction Quality Reduction Project end item
Degradation small demanding Apps requires client unacceptable to is useless
affected approval client
41
Qualitative and Quantitative Risk Outputs
Qualitative
Overall risk rating for the project: Risk ranking is used to rank the risk of the project under evaluation
against other projects.
List of prioritized risks: Risk can generally ranked as low, moderate, or high.
List of risks for additional analysis and management: Risk categorized as moderate or high would
be prime candidates for more analysis, including quantitative risk analysis, and for risk management.
Trends in qualitative risk analysis results: As the analysis is repeated, a trend of results may
become apparent, and can make risk resolution or further analysis more or less urgent and important.
Quantitative
Prioritized list of quantified risks: This list of risks include those that pose the greatest risk threat or
present the greatest opportunity to the project together with a measure of their impact.
Probabilistic analysis of the project: Forecasts or potential project schedule and cost results listing
the possible completion dates or project duration and costs with their associated confidence levels.
Probability of achieving the cost and time objectives: The probability of achieving the project
objectives under the current plan and with the current knowledge of the risks facing the project.
42
Text Book - Risk Resolution
DEFINITION: The process of developing procedures and

techniques to reduce threats to the project objectives.

1. Risk Management Plan 1. Risk resolution plan 1. Avoidance
2. List of prioritized risks 2. Residual risks 2. Transference
3. Risk ranking of the project 3. Secondary risks 3. Mitigation
4. Prioritized list of quantified risks 4. Contractual agreements 4. Acceptance
5. Probabilistic analysis of the project 5. Contingency reserve
6. Probability of achieving the cost and time amounts needed
objectives 6. Inputs to other processes
7. List of potential resolutions 7. Inputs to a revised project
8. Risk thresholds plan
9. Risk owners
10.Common risk causes
11. Trends in qualitative and quantitative risk
analysis results
43
Risk Resolution – Outputs
Risk Resolution plan: This is also called “risk register” and should include some or all of the following:
 Identify risks, their description, the area(s) of the project effected, their causes, and how they may affect project objectives
 Risk owners and assigned responsibilities
 Results from the qualitative and quantitative risk analysis
 Agreed to response including avoidance, transference, mitigation, acceptance for each risk in the risk resolution plan
 The level of residual risk expected to be remaining after the strategy is implemented
 Specific actions to implement the chosen resolution strategy
 Budget and times for resolution
 Contingency plans and fallback plans
Residual risk: Residual risk are those that remain after avoidance, transfer, or mitigation resolutions have
been taken. They also include minor risks that have been accepted and addresses.
Secondary risks: These risks arise as a direct result of implementing a risk resolution.
Contractual agreements: Contractual agreements may be used to specify each party’s responsibility for risks.
Contingency reserve amounts needed: Probabilistic analysis of the project and risk thresholds help the
project manager determine the amount of contingency needed to reduce risk.
Inputs to other processes: Most resolutions to risk involve expenditure of additional time, cost, or resources
and require changes to project plans.
Inputs to a revised project plan: The results of the resolution planning process must be incorporated into the
project plan, to ensure that agreed actions are implemented and monitored as part of the ongoing project.
44
Risk Resolution –Tools/Techniques
Tools and Techniques

Avoidance: Risk avoidance is changing the project plan to eliminate the risk or
condition or to protect the project objectives from its impact.
Transference: Risk transference is seeking to shift the consequences of a risk to a
third party together with ownership of the resolution. (Financial risk is most common)
Mitigation: Mitigation seeks to reduce the probability and or consequences of an
adverse risk event to an acceptable threshold.
Acceptance: This technique indicates that a project team has decided not to change
the project plan to deal with a risk. Developing a contingency plan is a way to
managing known risks. The contingency plan adopt contingency allowance or
reserve that includes:
 Time
 Money
 Resources
45
Text Book - Risk Monitoring & Control
DEFINITION: The process of monitoring residual risks, identifying new risks, executing
risk reduction plans, and evaluating their effectiveness throughout the project life cycle.
The purpose of risk monitoring is to determine if:
• Risk resolution have been implemented as planned
• Risk resolution actions are as effective as expected, or if new resolutions should be developed
• Project assumptions are still valid
• Risk exposure has changed from its prior state, with analysis and trends
• A risk trigger has occurred
• Proper policies and procedures are followed
• Risks have occurred or arisen that were not previously identified
1. Risk Management Plan 1. Workaround plans 1. Project Risk resolution

2. Risk resolution plan 2. Corrective action audits
3. Project communication 3. Project change request 2. Periodic project risk
4. Additional risk 4. Updates to the task reviews
identification and analysis resolution plan 3. Earned Value analysis
5. Scope Changes 5. Risk database 4. Technical performance
6. Updates to risk measurements
identification checklist 5. Additional risk resolution
planning
46
47
48

7-Measurements-Drawing Valid Conclusions From Numbers

Uploaded by

Document Information

Original Title

Copyright

Available Formats

Share this document

Share or Embed Document

Sharing Options

Did you find this document useful?

Is this content inappropriate?

Copyright:

Available Formats

7-Measurements-Drawing Valid Conclusions From Numbers

Uploaded by

Copyright:

Available Formats

Railway engineering economy and project management

Invalid data and poor statistical methods can

How you can instead use valid methods and

So, in making design decisions for our engineered

We designed our tests deliberately so that the system would

Your test program should not be content looking only for

The point of gathering information about failures is

Data or processing that deviates materially from the nominal

When we measure something, the value returned by the measurement is affected by

• That part of the change in the measurement that is due to an

There is a method called the decision tree that allows one

What Engineering Project Managers Need to Measure

As managers of engineering projects, we need to

Technical performance measures

Operational performance measures

We Need Measurements in Order to Create Good

Projects Provide an Opportunity for Time Series ：

It gets progressively far more expensive to fix things as

Avoid “Explaining Away” the Data

When in fact the data are a surprise.Try to obtain as

Social Aspects of Measurement:

Good methods create credibility with your customers, and

These model builders often spend years collecting

Text Book Definition: Risk Management is the systematic

Simple Terminology: Risk Management is the identification of

Risk Assessment Risk Analysis

Risk Control Risk Resolution

DEFINITION: The process of deciding how to approach and plan

Inputs Output Tools & Techniques

Risk Management Planning output is the “Risk Management Plan”

The plan may include:

DEFINITION: The process of determining which risks might affect

Inputs Output Tools & Techniques

Tools and Techniques

• Quantitative: The process of measuring the probability and consequences of

• Qualitative: The process of performing a qualitative analysis of risks and

Risk probability and consequences: This is generally described in

Probability/impact risk rating matrix: The use of risk’s probability scale

Evaluating Impact of a Risk during Major Project Milestones

DEFINITION: The process of developing procedures and

Inputs Output Tools & Techniques

Tools and Techniques

Inputs Output Tools & Techniques

1. Risk Management Plan 1. Workaround plans 1. Project Risk resolution

You might also like