USING MEASUREMENT AND THE SAATY

METHOD TO CHOOSE THE BEST DECISION

ALTERNATIVE

Peter Baxter, DISTRIBUTIVE MANAGEMENT

Hampton Roads INCOSE Decision Analysis Conference - Nov 2009
 2
OBJECTIVES

Learn:
 How to construct and weight evaluation criteria.

 How to make pair-wise comparisons of

alternatives.
 How to expand the example for more complex,
nested types of criteria.
 How a measurement process can support the
Saaty method.

DISTRIBUTIV
E
MANAGEMENT
WHAT IS THE
AHP/SAATY METHOD?
 4
SIMPLE DEFINITION

The Analytic Hierarchy Process (AHP)

DISTRIBUTIV
Compute the matrix values for a and … voila!
E
MANAGEMENT
 5
COMPARE THE SIZES
OF THESE CIRCLES

A B C D E
A B C D E
A 1 7 9 5 3
B 1/7 1 3 1/5 1/7
C 1/9 1/3 1 1/7 1/9
D 1/5 5 7 1 1/3
DISTRIBUTIV
E E 1/3 7 9 3 1
MANAGEMENT
 6
SCALE OF
COMPARISON
 Don’t need an absolute scale
(if you already know the size of all but one)
 Relative scale approximates difference

DISTRIBUTIV
E
MANAGEMENT
 7
SIMPLE EXPERIMENT

DISTRIBUTIV One test is worth 1000 expert opinions

E
MANAGEMENT
 8
WEBER’S EXPERIMENT

In 1846 Weber found, for example, that people while holding in

their hand different weights, could distinguish between a
weight of 20 g and a weight of 21 g, but could not if the
second weight is only 20.5 g. On the other hand, while they
could not distinguish between 40 g and 41 g, they could
between 40 g and 42 g, and so on at higher levels. We need
to increase a stimulus s by a minimum amount Δs to reach a
point where our senses can first discriminate between s and
s+Δs. Δs is called the just noticeable difference (jnd). The
ratio r = Δs/s does not depend on s.

Weber’s law states that change in sensation is noticed when the stimulus
is increased by a constant percentage of the stimulus itself. This law
holds in ranges where Δs is small when compared with s, and hence
DISTRIBUTIV in practice it fails to hold when s is either too small or too large.
E
MANAGEMENT
 9
WHY IT WORKS

 People are inconsistent at providing an absolute

scale to evaluate objects.
How much does Rock A weigh?
How much does Rock B weigh?

 There are better at comparing pairs of objects.

Does Rock A weigh more than Rock B?

DISTRIBUTIV
E
MANAGEMENT
 10 APPLYING THE TECHNIQUE

1. Select criteria(s) to evaluate

2. Define comparison scale
3. Perform pair-wise comparison
4. Check consistency
5. Calculate values

DISTRIBUTIV
E
MANAGEMENT
APPLYING THE METHOD
 12
EXAMPLE #1
SOFTWARE ESTIMATION
 A controlled test where existing software sizes are known.

 Ask 30 grad students to estimate the SLOC of common data

structures like stack, queue, list.

 Estimate SLOC three ways:

1. Guess a number.

2. Compare to one reference structure using numeric

scale.

3. Compare to one reference structure using relative scale.

 Plot results

DISTRIBUTIV From “Establishing Software Size Using Pair-wise Comparison Method”

E
by Eduardo Miranda
MANAGEMENT
 13
EXAMPLE #1
SOFTWARE ESTIMATION

DISTRIBUTIV
E
MANAGEMENT
 14
EXAMPLE #1
OBSERVATIONS
Miranda’s Observations
 The high variability of the “finger in the wind approach”,
which is almost two to three times bigger than the
corresponding paired comparisons method.
 The high correlation, r = .979, existing between the
relative sizes of modules independent of the estimation
method employed. This seems to corroborate the
premise that the human mind is better at establishing
differences than at guessing absolute values.

DISTRIBUTIV
E
MANAGEMENT
 15
EXAMPLE #2
SYSTEM REQUIREMENTS

 From an SEI study “Requirements Prioritization Case Study

Using AHP” by Nancy Mead.
 Given nine security requirements, decide which one(s) have
greatest cost-benefit.
 Benefits

“By using AHP, the requirements engineer can

also confirm the consistency of the result. AHP can
prevent subjective judgment errors and increase the
likelihood that the results are reliable.”

DISTRIBUTIV
E
MANAGEMENT
 16
EXAMPLE #2
APPROACH

Approach
1. Review requirements for completeness.
2. Apply pair-wise comparison for value.
3. Apply pair-wise comparison for implementation
cost.
4. Calculate AHP matrix and diagram for value & cost.
5. Use resulting diagram for analyzing requirements.

DISTRIBUTIV
E
MANAGEMENT
 17
EXAMPLE #2
COMPARISON SCALE

DISTRIBUTIV
E
MANAGEMENT
 18
EXAMPLE #2
COMPARISON
Compare one attribute of each requirement to another
requirement using a relative scale.
#1 is same #1 is greater
as #1 than #2

#2 is much
less than #8
DISTRIBUTIV
E
MANAGEMENT
 19
ABOUT CONSISTENCY

 If: A>B>C
 Then: C > A is wrong

AHP contains a technique to calculate the extent of

pair-wise consistency, which can then be
compared to a consistency tolerance.
AHP can also indicate which pair-wise comparison
(like the one above) is inconsistent.

DISTRIBUTIV
E
MANAGEMENT
 20
EXAMPLE #2
COMPARISON SUMMARY

DISTRIBUTIV
E
MANAGEMENT
 21
EXAMPLE #2
FINDINGS

Client feedback
 It may be beneficial to see the consistency matrix.
 Understand weight of cost and value.
 Difficult to understand the motivation of each
reviewer.

DISTRIBUTIV
E
MANAGEMENT
 22
OTHER EXAMPLES

From Saaty:
 Evaluate the best city in China for Disney
to build a new theme park.
 Determine optimum foreign relations
policy for dealing with Iran.
 Estimate market share of “super” retail
stores.
 Selecting a school.
DISTRIBUTIV
E
MANAGEMENT
MEASUREMENT AND
AHP
 24
MEASUREMENT AND
AHP
Goal is for Measurement to support AHP estimation

Use AHP Estimate

Capitalize Plan

Monitor Measurement Process

DISTRIBUTIV
E
& Control
MANAGEMENT
25
TYPICAL MEASUREMENT
PROCESS
USER FEEDBACK
Technical
and Management
Processes ANALYSIS RESULTS
INFORMATION
NEEDS

Core Measurement Process

Establish
Capability
Plan Perform
MEAS-
UREMENT
PLAN

Experience
Base ANALYSIS
RESULTS AND
PERFORMANCE
IMPROVEMENT MEASURES
ACTIONS
Evaluate
Measurement Process
 26
WHAT TO MEASURE?

Measurement “Requirements” = information needs

 Based on your business needs.
(not a pre-defined list)
 Information needs are:
 “Input” to the measurement process.
 Provided by the management and technical process that
need information to perform their jobs.
 Become the requirements for measurement process.
 Refined into measures and then resulting information
products are provided to the “users”.
DISTRIBUTIV
E
MANAGEMENT
 27
COMMON PROJECT
MEASURES

Estimation Project/Program Store

 Cost
 Staffing , Staff Hours
Factors estimated
using AHP  Functional / Requirements Size
 SLOC
 Defects

Usually there is opportunity for measurement to support AHP since

DISTRIBUTIV
there is overlap between what is estimated and what is measured
E
MANAGEMENT
 28
MEASUREMENT GUIDE
AHP
 Measurement contains project attributes:
 Lifecycle model
 Principle architecture
 Application domain

 AHP estimation of projects with similar attributes.

 Reference sizes should not differ by more than an
order of magnitude.

DISTRIBUTIV
E
MANAGEMENT
 29
AHP IN USE

Estimate total staff hours in person years using data from

five completed projects sharing similar attributes.

DISTRIBUTIV
E
MANAGEMENT
 30
RESOURCES

“Requirements Prioritization Case Study Using AHP”

by Nancy Mead Software Engineering Institute

“Establishing Software Size Using the Paired

Comparisons Method” by Eduardo Miranda

“Relative Measurement and Its Generalization in

Decision Making Why Pairwise Comparisons are
Central in Mathematics for the Measurement of
Intangible Factors The Analytic
Hierarchy/Network Process” by Thomas Saaty
DISTRIBUTIV
E
MANAGEMENT
 31
QUESTIONS

DISTRIBUTIV
E
MANAGEMENT
 32

Peter Baxter
Distributive Management
www.distributive.com
pbaxter@distributive.com

DISTRIBUTIV
E
MANAGEMENT