Chapter 9

Model Calibration

John Hourdakis
Center for Transportation Studies, U of Mn
Hourd001@tc.umn.edu
 Why Calibrate?

• Computers Cannot Magically Replicate Reality!

• Simulation Models Are Designed to be General
• Driver Behavior and Road Characteristics Depend on
ƒ Location i.e. Minnesota vs California
ƒ Vehicle Characteristics (Horsepower, Size, etc.)
ƒ Weather Conditions (Dry, Wet, Ice, etc.)
ƒ Day or Night

• Microscopic Simulators Can Adapt and Replicate

Almost Any Condition if the Model Parameters Are
Properly Adjusted
• What is Realistic and What is Not?
2
 Example of a Disaster

• Half of an Ordinary Double Cloverleaf

• Freeway Speed 65 mph, Ramps 45 mph
• Turning Warnings More Than 2000 Feet Away
• Lot’s of Traffic: 6500 veh/hour on Mainline
• Three Vehicle Types: Car, Truck, and Semi-Trailer3
 Simulation Results From 5 Scenarios
• Same Demands, Speed Limits, Turnings, Most of
Model Parameters.
• Changed Car and Semi-Trailer :
• Acceleration
• Normal Deceleration
• Maximum Deceleration (Emergency Stop)
Average Average Flow Total Travel Time Av. Delay LOS
Speed
+2f/s2 45.5 mph 6831 vph 257 hours 9 sec/veh D

+1f/s2 29.1 mph 6200 vph 484 hours 41 sec/veh F

BASE 46.7 mph 6396 vph 234 hours 9 sec/veh D

-1f/s2 35.3 mph 6152 vph 393 hours 30 sec/veh E

4
-2f/s2 27.4 mph 5500 vph 536 hours 57 sec/veh F
 Calibration Procedure
H is t o r ic a l
H is t o r ic a l s y s te m
s y s te m in p u t
in p u t d a ta
d a ta

A c tu a l S im u la tio n
sy ste m M odel

C om pare
S y stem M o d el o u tp u t
o u tp u t d ata d a ta

No
V a lid ?
C a lib r a te

5
 REALITY
Current Traffic Measurements, etc.

• Need to KNOW the Network to be Replicated

• Depending on the Scope of the Project, Two
or More of the Following Are Needed :
• Mainline Volumes (Every X Feet)
• Mainline Speeds (Every X Feet)
• Travel Times (Link or Between O/D Pairs)
• Bottleneck Capacity (Measured, Not Theoretical)
• Entrance Ramp Queues
• Intersection Queues and Queue Discharge Rates
6
 Modeling Parameters
• Global Parameters
• Vehicle Parameters
• Size and Power
• Driver Behavior
• Car-Following Model Parameters
• Reaction Time
• Speed Distibution Among Lanes (Overtaking
Manuevers)
• Local Section Parameters
• Curvature and Grade
• Speed Limit or Free Flow Speed
• Lane Changing Distances
• Headway/Hesitation Factor
7
 Model Parameter Issues

• Driver/Vehicle Characteristics
• Literature Does Not Provide Adequate Information
• Not Common to All Simulators
• No Info on Effects of Weather/Pavement/Ambient
Conditions
• Car-Following Parameters
• Depend on the Simulator’s Car-Following Model
Employed.
• Most Simulators Do Not Adequately Describe the
Modeling Process (if at all).
• User Has No Sense of Model Parameter Effects on Results.

8
 Calibration Issues

• Very Important For Model Accuracy and

Robustness
• Accuracy Depends on Measurement Granularity
• Averages Over Several Days is a Bad Choice
• Might Need Additional Information to be Collected in
Turbulent Sections (Bottlenecks, Weaving Areas, etc.)

• Simulation Objective Affects Calibration

• When Adaptive Control Strategies Are Simulated,
Stricter Validation is Needed
• Modeling of an Isolated Interchange in Rural
9
Minnesota Will be Restrictive
 Calibration Issues

• VERY TIME CONSUMMING PROCESS

• Currently Simulators Do Not Provide a Methodology or
Tools to Assist in Calibration
• Often Users End Up in Endless Trial-and-Error Cycles
• Sporadic Attempts Made in Literature to Streamline the
Process But:
• Focused on a Particular Simulator
• Too Complex or too Naive to be Effectively Used in
Practice
• No Widely Accepted Methods/Standards Currently
Available

10
 Before Calibration!

• Check Geometry For Correctness

• Disjoined Sections
• Stuck Vehicles (Sizes of Accel/Decel Lanes)
• Verify Location of Detectors
• Check Input For Accuracy
• Entrance Volume Comparison (Perfect Match)
• Exit Volume Comparison (Match Sum Over All
Hours)
• Volume Totals on Mainline Stations Should Match
11
 Practical Calibration Methodology
(Employed on Twin Cities Freeways)
• Need Simultaneous Boundary and Mainline Station Measurements.
• Technique:

upstream mainline detector stations

• Objective is to Match the Simulated and Actual Mainline Traffic

Measurements
• Traffic Measurements Used: Volume and Speed
• Occupancy Affected by Detector Sensitivity (Unknown)
• Perform Calibration in Stages:
• First 2 Stages Based on Volume and Speed in That Order
• Further Improvements in Optional 3rd Stage: 12
ƒ Depending on Objective i.e. For Ramp Control -> Queue Length
 Goodness-of-Fit Test Measures
• Recommended Goodness-of-Fit Measures:
2
1 n  xi − yi 
1. RMS Percent Error = ∑ 
n i =1  yi 

(Measures Overall % Error)
1 n ( xi − x)( y i − y )
2. Correlation Coefficient = CORL = ∑ σσ
n − 1 i =1 x y
(Measures Linear Association)

Where xi is the Simulated Traffic Measurement at Time i

x’ is the Mean of the Simulated Traffic Measurements
yi is the Actual Traffic Measurement at Time i
y’ is the Mean of the Actual Traffic Measurements
σx is the Standard Deviation of the Simulated Traffic Measurement
σy is the Standard Deviation of the Actual Traffic Measurements
n is the Number of Traffic Measurement Observations
13
 Goodness-of-Fit Measures (Cont.)
3. Theil’s U (Considers the Disproportionate Weight of
Large Errors) 1
n
∑(y − x )
n

i i
2

i =1

• 3 Components of Theil’s U 1
∑y + ∑x
1 n
2
i
n
2
i
n n
¾ Us i =1 i =1

(Measure of Variance Proportion, Close n(σ y − σ x ) 2

Us = n
to 1 Satisfactory) ∑(y i − xi ) 2
i =1
¾ Uc
2(1 − r )nσ yσ x
(Measure of Covariance Proportion Uc = n

or Unsystematic Error, Close to 0 Satisfactory) ∑(y

i =1
i − xi ) 2

¾ Um n( y − x ) 2
Um =
(Measure of Bias Proportion or n

∑(y i − xi ) 2
Systematic Error, Close to 0 Satisfactory) i =1

Where σy and σx are the Standard Deviations of the Actual and Simulated
Series
r is the Correlation Coefficient Between the Two Series
14
 Examples
200 200

vo lu m e (v eh ic le s/5- m in)
vo lu m e (v eh ic le s/5- m in)

150 150

ac tual volume actual v olume

simulated volume s imulated volume

100 100

50 50
25

31

55

61

13

19

37

55
1

7
13

19

37

43
49

67

25

31

43

49

61

67
tim e in 5-m in incr em en ts fro m m idnigh t tim e in 5-m in increm e nts fr om midn ig ht

Figure 1(a): Illustration of unsatisfactory Um Figure 1(b): Illustration of unsatisfactory Uc

200 2 00
vo lu m e (v eh ic le s/5- m in)

vo lu m e (v eh ic le s/5- m in)
150 1 50

actual v olume s imulate d volu me

s imulated volume a ctua l v olume

100 1 00

50 50
19

25

49

55
13

19

37

55

13

31

37

43

61

67
1

25

31

43

49

61

67

tim e in 5-m in increm e nts fr om midn ig ht t im e in 5 -m in in cr e m e nt s f rom m idnigh t

15
Figure 1(c): Illustration of unsatisfactory Uc Figure 1(d): Illustration of unsatisfactory Us
 Stage 1: Volume-Based Calibration

• Objective is to Match Simulated and Actual Mainline

Station Volumes
• Simulation Model Calibrated Beginning Upstream
and Proceeding Downstream
• Global Parameters Are Calibrated First:
• Usually Accomplished in First Few Stations
• Trial & Error Iterative Process For Each Parameter
• RMSP, r and U are the Metrics Used in Each Iterations
• Local Parameters Calibrated at All Stations
• Um, Uc and Us are the Metrics Used at This Point.

16
 Stage 2: Speed-based Calibration
• Objective is to Match Simulated and Actual Mainline
Station Speeds and Bottleneck Locations
• Actual Speeds Derived From Volume, Occupancy,
and Effective Vehicle Length (For Single Loop
Detectors)
• Speed Contour Graphs Used For Comparing
(Visually) Simulated and Actual Speeds
1. If Speed Contours Exhibit Significant Discrepancy, Revise
Global Parameters From Stage 1
2. Beginning Upstream and Proceeding Downstream,
Calibrate Local Parameters Until Mainline Speeds and All
Bottleneck Locations in the 2 Contour Graphs Match

17
 Stage 3: Application Dependent
• For Adaptive Ramp control:
• Compare Queue Lengths
• Local Simulation Parameters Affecting Detector Output
Need Further Calibration in This Stage

• Simulated Entrance Ramp Queues Should Match Actual Ones

• Queue Measurements From 30 sec Detector Counts
• Queue = Metered Demand - Actual Demand (Upstream)

• Over-Calibration to be Avoided to Ensure Generality

• Repeat Simulation With Different Random Seeds
• Simulate Additional Days

18
 Example of Calibration
• AIMSUN Microsimulator
• Mn/DOT Ramp Metering Evaluation.
• 2 Test Sites (Only One Presented)
• TH 169 Northbound in Minneapolis, MN
ƒ 12 Miles Long: From I-494 to 63rd Avenue N
ƒ 24 Entrance Ramps, 25 Exit Ramps
ƒ 30 Detector Stations.
ƒ 5-Minute Volume and Occupancy
ƒ March 21st to 23rd, 2000
ƒ 14:00 to 20:00 hrs

19
Test site 1: TH 169NB
A A B B

A A B B
20
 Stage 1 Results (Volume)
Goodness-of- Values
Fit Measure
r 0.960
RMSP 7.39%
U 0.002
Um 0.088
Us 0.031
Uc 0.881

• 500 Simulator Iterations Required, 2 Months (!)

• Irregularities in Input Data Observed Due to
Sensor Misplacement
21
 Actual Speed Contour

40-50
50-60
Downstream 60-70
767
70-80
80-90
769
90-100
443
100-110
441

439

437

434

432

430

428

426
171

175

181

185

187

191

197

201

203

207

209

213

219

223

225

229

231

235
169

173

177

179

183

189

193

195

199

205

211

215

217

221

227

233

237

239
TIME Upstream

22
 60-70

Stage 2 Results
70-80
767
80-90
769 90-100

(Speed) 443 100-110

441

439

437

434

Simulated Speed 432

430

After Stage 1 428

426

169

171

173

175

177

179

181

183

185

187

189

191

193

195

197

199

201

203

205

207

209

211

213

215

217

219

221

223

225

227

229

231

233

235

237

239
Sp e e d (s im u la tio n )
40- 50
50- 60
60- 70

7 67 70- 80

Simulated Speed 7 69 80- 90

4 43
90- 10 0

After Stage 2
100 -1 10
4 41
4 39

(About 200 4 37
4 34

Iterations) 4 32
4 30
4 28

4 26
16 9

17 2

17 5

17 8

18 1

18 4

1 99

2 02

2 05

2 08

2 11

2 14

2 17

22 0

22 3

22 6

22 9

23 2

23 5

23 8
187

190

193

196

40-50
50-60
60-70
767
70-80
80-90
769
90-100
443
100-110
441

Actual Speed
439

437

434

432

430
23 428

426
171

175

181

185

187

191

197

201

203

207

209

213

219

223

225

229

231

235
169

173

177

179

183

189

193

195

199

205

211

215

217

221

227

233

237

239
 Stage 3 Results (Queues)
• Simulated and Actual Queues Did Not Match Before This Stage
• 3rd Stage Required About 100 Iterations
Example Ramp

80

70

60

50
real queue

40 simulation queue after

30 stage 3
simulation queue before
20
stage 3
10

-10
t ime ( 5 - min inc re me nt s f ro m mid nig ht )

Simulation vs Real Queues 24

 Validation Accuracy
(Volume)

RMS% r U Um Us Uc

March 21st 10.620 0.980 0.004 0.309 0.011 0.681

(validation)
March 22nd 6.420 0.970 0.001 0.124 0.054 0.823
(validation)
March 23rd 7.390 0.960 0.002 0.088 0.031 0.881
(Calibration)
Over All Stations.

25
 Results (Calibrated Parameters)
AIMSUN Microscopic Simulator
Parameter Initial After stage 1 After stage 2 After stage 3
Max. desired 105.000 110.000 110.000 110.000
speed (kmph)

Max. acc. rate (m/s2) 4.500 3.000 3.000 3.000

Normal dec. rate (m/s2) -4.500 -5.000 -5.000 -5.000

Max. dec. rate (m/s2) -5.000 -5.500 -5.500 -5.500

Reaction time (sec) 0.700 0.590 0.610 0.610

Percent overtake 0.950 0.950 0.940 0.940
Percent recover 1.000 1.000 0.990 0.990
Max. speed difference 40.000 40.000 60.000 60.000
(kmph)

Max. speed difference 50.000 50.000 70.000 70.000

on-ramp (kmph)

Av. section speed 110.000 100.000 105.000 105.000

(regular section, kmph)

Av. section speed 90.000 75.000 70.000 70.000

(weaving section, kmph)

Av. section speed (ramp 60.000 60.000 55.000 55.000

section, kmph)
26
 Conclusion
• Garbage In >>> Garbage Out

• Simulation Useless/Dangerous Without

Calibration

Questions?

27