Bayesian Approach for Uncertainty Analysis of an

Urban Storm Water Model and Its Application to

a Heavily Urbanized Watershed

Misgana K. Muleta, M.ASCE 1; Jonathan McMillan 2; Geremew G. Amenu 3; and Steven J. Burian, M.ASCE 4

Abstract: The significance of uncertainty analysis (UA) to quantify reliability of model simulations is being recognized. Consequently,
literature on parameter and predictive uncertainty assessment of water resources models has been rising. Applications dealing with urban
drainage systems are, however, very limited. This study applies formal Bayesian approach for uncertainty analysis of a widely used storm
water management model and illustrates the methodology using a highly urbanized watershed in the Los Angeles Basin, California. A flexible
likelihood function that accommodates heteroscedasticity, non-normality, and temporal correlation of model residuals has been used for the
study along with a Markov-chain Monte Carlo-based sampling scheme. The solution of the UA model has been compared with the solution of
the conventional calibration methodology widely practiced in water resources modeling. Results indicate that the maximum likelihood sol­
ution determined using the UA model produced runoff simulations that are of comparable accuracy with the solution of the traditional
calibration method while also accurately characterizing structure of the model residuals. The UA model also successfully determined both
parameter uncertainty and total predictive uncertainty for the watershed. Contribution of parameter uncertainty to total predictive uncertainty
was found insignificant for the study watershed, underlying the importance of other sources of uncertainty, including data and model struc­
ture. Overall, the UA methodology proved promising for sensitivity analysis, calibration, parameter uncertainty, and total predictive uncer­
tainty analysis of urban storm water management models. DOI: 10.1061/(ASCE)HE.1943-5584.0000705. © 2013 American Society of Civil
Engineers.
CE Database subject headings: Uncertainty principles; Calibration; Bayesian analysis; Stormwater Management; Parameters; Markov
process; Monte Carlo method; Urban areas.
Author keywords: Uncertainty analysis; Calibration; Bayesian approach; Storm water management; Parameter uncertainty; Predictive
uncertainty; Markov-chain Monte Carlo.

Introduction interaction between SCMs of various types, sizes, and relative

Primarily by increasing imperviousness, urbanization alters natural
hydrology of a watershed and negatively impacts ecology, geomor­
phology, water quality, and socioeconomic functions of the receiv­
ing waters (National Research Council (NRC) 2008). Structural
and nonstructural methods, generally referred to as storm water
control measures (SCMs), are often used to mitigate these impacts.
To improve effectiveness of the SCMs, watershed-scale design so­
lutions are advocated as opposed to the conventional approach of
selecting and designing SCMs site-by-site (EPA 2007; NRC 2008).
Watershed-scale design requires understanding hydrologic and
water quality characteristics of individual SCMs as well as the
1
Dept. of Civil and Environmental Engineering, California Polytechnic
State Univ., San Luis Obispo, CA 93407 (corresponding author). E-mail:
mmuleta@calpoly.edu
2
Dept. of Civil and Environmental Engineering, California Polytechnic
State Univ., San Luis Obispo, CA 93407.
3
Dept. of Civil Engineering and Construction Engineering Manage­
ment, California State Univ., Long Beach, CA 90840.
4
Dept. of Civil and Environmental Engineering, Univ. of Utah,
Salt Lake City, UT 84112.
locations in a watershed, consequently making the design process
more challenging.
Computer models could be used for effective design of SCMs at
watershed-scale. Models can simulate responses of the watershed
and the SCMs considering the factors relevant to the generation and
routing of runoff and contaminants. Models, however, must be
properly calibrated before their use for planning and management
of water resources. The traditional calibration method seeks to
identify an optimal set of parameters that forces model simulations
closer to the observed counterparts. The basis of this calibration
approach is the assumption that the inputs used to build the model,
the observations used to evaluate goodness of model simulations,
and structure of the model that describes physics of the watershed
are all error free. Recent contributions to the water resources
literature have seriously questioned the continued usefulness of this
classic calibration method (Beven and Freer 2001; Muleta and
Nicklow 2005; Kavetski et al. 2006; Vrugt et al. 2008; Montanari
et al. 2009). It is acknowledged that hydrologic predictions are
plagued with uncertainties arising from errors associated with forc­
ings (inputs), observations, parameters, and model structural inad­
equacies. Consequently, a prudent evaluation technique is to
recognize these uncertainties and to quantify predictive uncertainty
and parameter posteriors (Vrugt et al. 2005; Moradkhani and
Sorooshian 2008; Gotzinger and Bardossy 2008; Montanari
et al. 2009).
During the past two decades, the Generalized Likelihood Uncer­
tainty Estimation (GLUE) technique of Beven and Binley (1992)
and Beven (2006) has found widespread application for un­
certainty analysis in water resources. This informal Bayesian ap­
proach is simple to implement but has been criticized for being
statistically incoherent (Mantovan and Todini 2006; Stedinger et al.
2008; Vrugt et al. 2009b). In response to this, various authors have
proposed formal uncertainty analysis (UA) methods that use proper
statistics and employ valid likelihood measures (Kuczera and
Parent 1998; Thiemann et al. 2001; Vrugt et al. 2003; Schoups
and Vrugt 2010). These techniques attempt to provide estimates
of the probability density function (PDF) of the model parameters
as well as total predictive uncertainty, e.g., through Monte Carlo
simulations. For computational efficiency reason, Markov-chain
Monte Carlo (MCMC) schemes are preferred to classic Monte
Carlo simulations that rely on random sampling.
In addition to efficient and robust sampling schemes, successful
UA entails appropriate formulation of the likelihood function.
The formal UA applications often reported in the literature make
unrealistic assumptions regarding the structure of the residuals
(errors) between model simulations and the observed watershed
response. Common assumptions include that residuals are: (1) tem­
porally independent (i.e., no correlation between errors of succes­
sive time steps); (2) normally distributed; and (3) homoscedastic
(i.e., variance of the residuals does not depend on magnitude).
Addressing these unrealistic assumptions, a flexible and general
formal likelihood (GL) function has been recently described by
Schoups and Vrugt (2010).
The objective of this study is to examine effectiveness of a
formal Bayesian approach for uncertainty analysis and calibration
of the U.S. EPA Storm Water Management Model (SWMM5)
(Rossman 2010). The GL function and a recently developed effi­
cient MCMC sampling scheme known as DREAMðZSÞ (Schoups
and Vrugt 2010) has been used to identify parameter posteriors
and to estimate runoff prediction uncertainty. The methodology
is illustrated using the Ballona Creek watershed, a heavily urban­
ized watershed located in the Los Angeles Basin, California. Effec­
tiveness of the UA method in removing heteroscedasticity and
temporal correlation, and in identifying representative PDF for
the residuals has been scrutinized. To examine robustness of the
UA method for identifying the optimal solutions typically sought
by classic calibration approaches, the UA solution [i.e., the maxi­
mum likelihood (ML) parameter set and the associated runoff pre­
dictions] has been compared with the solution determined by an
automated calibration algorithm known as dynamically dimen­
sioned search (DDS) (Tolson and Shoemaker 2007).
Most UA studies in water resources that applied MCMC tech­
nique within the Bayesian framework used lumped conceptual
models for rainfall-runoff analysis of rural watersheds (Kuczera
et al. 2006; Vrugt et al. 2009a; Schoups and Vrugt 2010). Few stud­
ies have been reported in spatially distributed modeling (Feyen et al.
2008). Applications to urban watersheds are very limited. Ball
(2009) underlined the need for UA-based approaches for evaluation
of urban drainage models in the discussion of the conventional
calibration effort reported on Ballona Creek watershed by Barco
et al. (2008). Freni et al. (2008, 2009a) applied GLUE to an urban
drainage model and tested sensitivity of the solutions to likelihood
measures (Freni et al. 2009a) and acceptability thresholds (Freni
et al. 2008). In a separate study, Freni et al. (2009b) compared
performance of Bayesian Monte Carlo method to that of GLUE.
Mannina and Viviani (2010) applied GLUE for UA of storm water
quality using a conceptual, urban drainage model developed in­
house. All these applications of UA to urban drainage models used
GLUE, an informal approach whose statistical validity has been
questioned (Mantovan and Todini 2006; Stedinger et al. 2008;
Vrugt et al. 2008).
Working under the Joint Committee on Urban Drainage estab­
lished by the International Water Association and the International
Association for Hydro-Environment Engineering and Research
(IWA/IAHR), the International Working Group on Data and
Models has recently published findings of its effort to develop a
framework for defining and assessing uncertainties in urban drain­
age models (Dotto et al. 2012; Deletic et al. 2012). The article by
Deletic et al. (2012) underscored the need for consistent use of
terminologies and methods for UA of urban drainage models.
The authors defined various sources of uncertainties, presented
the linkages between the different uncertainty sources, and pro­
posed a framework for UA of urban storm water models. The article
by Dotto et al. (2012) compared four different UA methods (three
non-Bayesian and one Bayesian) in terms of the posterior PDFs and
prediction intervals determined by the methods and their relative
computational efficiencies. They showed that the non-Bayesian
methods required subjective decisions that affected the UA results,
whereas the Bayesian method used erroneous assumption regard­
ing structure of the residuals.
This study is the first, to the best knowledge of the authors, to
apply an MCMC scheme that works within formal Bayesian frame­
work for UA of urban watersheds and to apply Bayesian approach
to UA of SWMM. Besides demonstrating application of state­
of-the-art in UA to urban storm water modeling, the ensuing
model will be used for an ongoing study that attempts to develop
a simulation-optimization model for watershed-scale design of
SCMs for urban watersheds.
Methods and Materials
Uncertainty Analysis and the MCMC Algorithm
The watershed response (e.g., runoff) simulated by a storm water
management model, f, such as SWMM5 can be described as
Y^ ¼ fðI; θÞ ð1Þ
where Y^ ¼ n × 1 vector representing the runoff time series
(y^ 1 ; : : : ; y^ n ); I = matrix of model forcings (e.g., precipitation);
and θ signifies a d-dimensional vector of model parameters. To test
how well f describes runoff from a watershed, the common practice
is to compare the model predictions, Y^ n ¼ fy^ 1 ; : : : ; y^ n g with the
corresponding observations, Y n ¼ fy1 ; : : : ; yn g. The difference
between the two time series can be represented by a residual vector,
En ðθÞ
En ðθÞ ¼ Y − Y^ ¼ fy1 − y^ 1 ; : : : ; yn − y^ n g ¼ fe1 ðθÞ; : : : ; en ðθÞg
ð2Þ
The traditional model calibration technique searches for a
single optimal combination of parameter values that minimizes this
residual time series. With the recognition that model simulations
are plagued by many sources of uncertainty, validity of this conven­
tional parameter estimation method has been questioned. A plau­
sible alternative is to account for the various sources of uncertainty
and to determine posterior PDF of the parameters, for example,
using the Bayesian approach.
From Bayes theorem, the posterior PDF of the parameters,
pðθjI; Y n Þ), can be given as
pðθjI; Y n Þ ∝ LðθjY n ; IÞpðθÞ ð3Þ
where LðθjY n ; IÞ denotes the likelihood function that measures
how well the model fits the data; and pðθÞ = prior distribution
of the model parameters. Different likelihood functions have been
proposed, depending on the assumptions made about the statistical
properties of the residual vector, En ðθÞ. If the residuals are assumed
to be temporally uncorrelated and normally distributed with zero
mean and a homoscedastic error standard deviation, σe , the like­
lihood function takes on the well-known simple least-square
(SLS) form (Box and Tiao 1992) as
Pn
1 ei ðθÞ2
LðθjY n ; IÞ ∝ exp − i¼1
2 σ2e
Limitations of the SLS assumptions for hydrologic models have
been documented by several authors (Sorooshian and Dracup 1980;
Kuczera 1983; Thyer et al. 2009; Schoups and Vrugt 2010), and
different proposals have been suggested to relax the assumptions.
One of the latest recommendations is the formal likelihood function
proposed by Schoups and Vrugt (2010), who described a general
error model that embraces temporal correlation, heteroscedasticity,
and non-Gaussian nature of the model residuals.
The generalized log-likelihood function of Schoups and Vrugt
(2010) can be written as
2σξ ωβ X n Xn
Lðθ; φjY; IÞ ¼ n log −1
− log σt − cβ jaξ;t j2=ð1þβÞ
ξþξ t¼1 t¼1
where φ signifies parameters of the error model. Temporal corre­
lation between the residuals is accounted for using a pth order
autoregressive polynomial [ϕp ðBÞ] as
X
p DDS requires minimal algorithmic parameter tweaking because
ϕp ðBÞet ¼ σt at ; where ϕp ðBÞ ¼ 1 − ϕ i Bi
i¼1
and Bi et ¼ et−i
where σt = standard deviation at time t, and to account for hetero­
scedasticity, it is assumed to increase linearly with the streamflow
yt as
σ t ¼ σ 0 þ σ 1 yt
where σ0 and σ1 are inferred from the data along with model
parameters, (θ). Finally, at represents an independent and identi­
cally distributed random error with zero mean and a unit standard
deviation, whose probability is described by a skew exponential
power (SEP) density with skewness (ξ) and kurtosis (β)
parameters
2σξ
pðat jξ; βÞ ¼ ωβ expf−cβ jaξ;t j2=ð1þβÞ g
ξ þ ξ −1
aξ;t ¼ ξ −signðμξ þσξ at Þ ðμξ þ σξ at Þ
where μξ , σξ , cβ , and ωβ are computed as a function of ξ and β as
described by Schoups and Vrugt (2010).
Besides the likelihood function, a sampling scheme that effi­
ciently identifies posterior PDFs is crucial for effective application
of Bayesian-based UAs. Markov-chain Monte Carlo (MCMC)
schemes are often used for this application, and improving
efficiency of MCMC schemes has been one of the focuses of UA
research the past few years. In this regard, Laloy and Vrugt (2012)
developed DREAMðZSÞ , an MCMC algorithm that capitalized on
the strength of the DiffeRential Evolution Adaptive Metropolis
(DREAM) (Vrugt et al. 2008). Effectiveness and efficiency of
DREAMðZSÞ for posterior sampling has been reported in several
studies. Schoups and Vrugt (2010) applied the GL function and
DREAMðZSÞ for rainfall-runoff analysis of two watersheds by using
a lumped conceptual model. This study examines DREAMðZSÞ and
GL for UA of SWMM5 using the Ballona Creek watershed, which
is one of the most urbanized watersheds in the world with approx­
imately 83% developed (Bay et al. 2003). Schoups and Vrugt
(2010) provides further description of DREAMðZSÞ .
ð4Þ
Single-Objective Calibration
Single-objective automated calibration was performed, primarily,
to compare solutions of the conventional model calibration tech­
nique to those identified by GL and DREAMðZSÞ . The dynamically
dimensioned search (DDS) (Tolson and Shoemaker 2007) was used
to identify optimal values of SWMM5 runoff parameters. DDS has
been developed to improve efficiency of calibrating computation-
ally demanding models. DDS is a simple, single-objective, heuris­
tic search method that starts by globally searching the feasible
region and incrementally localizes the search space as the number
of simulations approaches the maximum allowable number of sim­
ulations (the only stopping criteria used by the algorithm). Progress
from global to local search is achieved by probabilistically reducing
the number of model parameters modified from their best value
ð5Þ
obtained thus far. New potential solutions are created by perturbing
the current parameter values of the randomly selected model
parameters only. The best solution identified thus far is maintained
and is updated only when a solution with superior value of the
objective function is found.
the only parameters to set are the maximum number of model eval­
uations and the scalar neighborhood size perturbation parameter (r)
ð6Þ that defines the random perturbation size standard deviation as a
fraction of the decision variable range. The recommended value
of 0.2 (Tolson and Shoemaker 2007) has been used for r in this
study. Efficiency and effectiveness of DDS has been reported
by Tolson and Shoemaker (2007) and Muleta (2010), who com­
pared its performance to that of widely used optimization methods
ð7Þ including the Shuffled Complex Evolution-University of Arizona
(SCE-UA) (Duan et al. 1992) and the Genetic Algorithms (Holland
1975). For this study, DDS has been integrated with SWMM5 to
calibrate runoff for the study watershed.
EPA Storm Water Management Model
SWMM was first developed in 1971, and it continues to be widely
used throughout the world for planning, analysis and design of
storm water runoff, combined sewers, sanitary sewers, and other
ð8Þ
drainage systems (Rossman 2010). SWMM5, the latest version of
SWMM, simulates hydrology, hydraulics, and water quality of
ð9Þ urbanized and nonurbanized watersheds. The hydrologic processes
modeled include precipitation (rainfall or snow fall), evaporation,
surface runoff, infiltration, groundwater flow, and snowpacks and
snowmelt. Both single event and continuous simulations can be
performed, accounting for spatial and temporal variability in the
climate, soil, land use, and topography in the watershed. Surface
runoff is estimated using the nonlinear reservoir method in which
surface runoff occurs only when the depth of the overland flow
exceeds the maximum surface storage provided by initial abstrac­
tions, including depression storage and interception, in which case
the runoff rate is estimated using Manning’s equation. Horton
(1937), Green and Ampt (1911), and the Curve Number methods
(Soil Conservation Service 1964) are available to model infiltration
losses.
Runoff quality, including buildup and washoff of pollutants, can
be simulated by using various approaches from both developed and
nondeveloped land uses. The runoff quantity and quality simulated
from a subwatershed and the wastewater loads (if any) assigned to
the receiving nodes are added and then transported by using either
steady, kinematic wave, or dynamic wave routing through a con­
veyance system of pipes, channels, storage/treatment devices,
pumps, and hydraulic regulators such as weirs, orifices, and other
outlet types. Hydraulic conditions of any level of complexity, in­
cluding those experiencing backwater effect, flow reversal, and
pressurized flow, can be accommodated. In addition, the capability
to model the commonly used low impact developments (LIDs),
including porous pavements, bioretention cells, infiltration
trenches, vegetative swales, and rain barrels has been recently
added to SWMM5. For this study, source code of SWMM5 has
been integrated with the UA and the single-objection calibration
methods previously described.
Application Watershed and Data
The Ballona Creek watershed (Fig. 1) is used to illustrate the meth­
ods described in this study. Total drainage area of the Ballona Creek
watershed is approximately 337 km2. For this study, the upper
230 km2 of the Ballona Creek watershed (i.e., the portion that
drains to the streamflow gauging station used in this study) has
been modeled. Approximately 90% of the modeled watershed is
Fig. 1. Location map of the Ballona Creek Watershed
developed, and its land-use distribution consists of 60% residential,
10% commercial, 3.5% industrial, and 11% open space (Amenu
2011). The open spaces are in the Santa Monica Mountains, located
in the northern part of the watershed. The drainage system is char­
acterized by extensive networks of storm drains that collect storm
water from the watershed and convey it to the Ballona Creek, a
14.5 km (9-mi)-long flood protection channel that discharges to
the Santa Monica Bay [Los Angeles County Department of Public
Works (LACDPW) 2011]. The watershed has been identified as the
major source of non-point-source pollution to the Santa Monica
Bay (Stenstrom and Strecker 1993; Stein and Tiefenthaler 2005).
The data needed to build SWMM5 have been collected from
various sources. A digital elevation model, land-use map, and an
imperviousness map were obtained from the USGS seamless data
warehouse (USGS 2013), and soil survey geographic (SSURGO)
soil map has been obtained from the Natural Resources Conserva­
tion Service (NRCS) soil data mart (NRCS 2013). Because of the
difficulty to accurately delineate urban subwatersheds from digital
elevation models alone (Gironás et al. 2010), the subwatershed
delineation obtained from the LACDPW were used for this study.
The LACDPW delineation (which was created through a compre­
hensive hydrologic study based on USGS topo quads, as built
drawings, and field surveys) divided the watershed into 134 sub-
watersheds. For this study, the number of subwatersheds was fur­
ther reduced to 92 by merging smaller subwatersheds (area less
than 0.41 km2 ) to the adjoining subwatershed. Subcatchment infor­
mation such as area, slope, and flow length were extracted from the
digital elevation model. The soil, land-use, and imperviousness
maps were superimposed onto the subwatersheds to extract
SWMM5 runoff parameters including percent imperviousness,
infiltration parameters, and Manning’s roughness coefficient.
Rainfall data at three gauges (Fig. 1) and streamflow data for a
monitoring station that drains approximately 70% of the Ballona
Creek watershed were obtained from the LACDPW for 15 years
(i.e., 1996–2010) at 15-min intervals. Both rainfall and streamflow
data were collected using automatic gauges that are equipped with
real-time data telemetry and electronic data loggers (Amenu 2011).
Proximity and altitude criteria were used to define the rain gauge
that represents each subcatchment. The climate of the watershed
can be characterized as semiarid, with average annual rainfall of
approximately 380 mm and temperature of approximately 18°C.
Rainfall season for the region spans from October to April. The
elevation of the watershed varies from 750 m above mean seal level
(AMSL) at the Santa Monica Mountains to 0 m AMSL at its dis­
charge to the Santa Monica Bay.
The watershed consists of an extensive network of storm drains
(underground pipes and open channels) designed for flood protec­
tion purposes. With the assumption that overland flow from each of
the 92 subwatersheds directly flows to a storm drain inlet located at
the outlet of the subwatershed, only 72 larger storm drains were
considered in this model. In reality, each subwatershed may contain
numerous streets, swales, and minor storm drains that can play
significant roles in routing of runoff and contaminants within
the subcatchment. Neglecting the minor drainage systems and mod­
eling only the major drainage systems, as done in this study, can
have an effect on the hydraulics of the drainage system. For exam­
ple, the travel time could get longer as the faster conduit flows
are replaced with the slower overland flows under this assumption.
The approach used in this study is, however, commonly used to
simplify modeling complexity and to reduce the cost associated
with data collection (Gironás et al. 2010). The study by Burian et al.
(2000) was used for storm drain information, including shape, size,
slope, and length.
Methodology
Simulation Durations and Analysis Methods
For both the DDS and GL-DREAMðZSÞ methods, rainfall and
streamflow data from January 17, 2010, to January 23, 2010, were
used for calibration. Verification of the solutions was performed
using data from January 23, 2008, to January 28, 2008. The event
used for calibration had rainfall depth of approximately 12.4 cm
in 7 days, and the verification event produced 12.6 cm of rainfall
in 6 days. According to National Oceanic and Atmospheric
Administration (NOAA) Atlas 14 (NOAA 2013), for a station in
Ballona Creek, mean rainfall depth for a 2-year, 7-day event is
approximately 12.1 cm, which is comparable to the calibration/
verification storm events used for this study. Both streamflow and
rainfall data are available at 15-min intervals. The curve number
method was selected for infiltration modeling as the CN values
(primary parameter for the curve number method) can be deter­
mined more readily, compared to Horton or Green-Ampt parame­
ters, from the land cover and soil maps available for the watershed.
Because the GL-DREAMðZSÞ algorithm used for the UA requires
running SWMM5 repetitively (up to hundreds of thousands of
runs) to converge, computational time is a significant concern.
As such, kinematic wave routing was selected to reduce computa­
tional burden of the dynamic wave routing option.
Likewise, continuous (long-term) simulation was not consid­
ered because of computational concern. Short-term simulation
(i.e., duration of 7 days for calibration and 6 days for verification)
were used for both single-objection calibration and uncertainty
analysis. Most studies that reported on calibration of urban drain­
age models used single-event simulations with a typical duration of
a day or less (Barco et al. 2008; Fang and Ball 2007). The simu­
lation durations considered in this study for both calibration and
verification cases are therefore, significant improvements com­
pared to single-event simulations. Although single-event and the
short-duration simulation pursued in this study may suffice for cer­
tain applications such as flood control, continuous (e.g., multiple
year duration) simulation models are more appropriate for applica­
tions that are sensitive to long-term watershed characteristics
(e.g., contaminant buildup and washoff processes).
Parameters
A total of 11 SWMM5 runoff parameters were considered for
calibration and uncertainty analysis. Values of the parameters vary
from subwatershed to subwatershed depending on soil, land use,
imperviousness, topography and/or other characteristics of the
subwatershed. Initial values of the parameters have been extracted
for each subwatershed from the soil, land-use, imperviousness, and
topography maps using geographic information system (GIS).
During both calibration and uncertainty analysis, these initial (base­
line) values were altered by multiplying the parameters by the
respective adjustments proposed by the calibration and the UA
algorithm. This way, the initial values would be scaled up or down
while preserving the heterogeneity determined from watershed
characteristics. The parameters were assumed to follow uniform
distribution as done in Muleta and Nicklow (2005), and lower and
upper percentage adjustment bounds were assigned based on liter­
ature and engineering judgment (Rossman 2010; Barco et al. 2008).
A list of the parameters and the assumed adjustment ranges are
given in Table 1.
Objective Function and Efficiency Criteria
The streamflow measured at the Swatelle station, shown in Fig. 1,
was used for calibration and uncertainty analysis. Mean absolute
error (MAE), Eq. (10), was used as objective function for the
single-objection calibration. MAE was selected as the objective
function based on the findings of Muleta (2012), which compared
relative effectiveness of the efficiency criteria commonly used in
hydrologic modeling to describe goodness of model performances.
According to the study, efficiency criteria such as MAE that
describe the absolute deviation between observations and model
simulations were found to be robust. Goodness of the calibration
result was further assessed using additional efficiency criteria
including the Nash-Sutcliffe efficiency (NSE) criterion (Nash
and Sutcliffe 1970) described in Eq. (11), percent bias (PBIAS)
[Eq. (12)], and total volume of runoff
1X N
MAE ¼ jY − Oi j ð10Þ
N i¼1 i
PN
ðY − Oi Þ2
NSE ¼ 1 − PN i¼1 i 2
ð11Þ
i¼1 ðOi − Omean Þ
PN
ðO − Y i Þ
PBIAS ¼ 100 PN i
i¼1
ð12Þ
i¼1 Oi
where Y = model simulated output; O = observed runoff; Omean =
mean of the observations, which the NSE uses as a benchmark
Table 1. Model Parameters and Ranges Used for Calibration and data, and calibration data uncertainty. Total predictive uncertainty
Uncertainty Analysis was calculated using the methodology described in Schoups
Name Description Minimum Maximum and Vrugt (2010) based on the error model parameters determined
by GL-DREAMðZSÞ. The ML parameter values determined by
SWMM parameters Percent adjustment
Width Subcatchment width (m) −90 150
GL-DREAMðZSÞ are benchmarked against the calibration results
Slope Subcatchment slope (%) −20 20 obtained using DDS and are also compared in terms of their ability
%Imperv Percentage of impervious −15 15 to fit different parts of the hydrograph.
area (%)
N-Imperv Manning n for impervious −50 100
area Results and Discussion
N-Perv Manning n for pervious area −50 100
Dstore-Imperv Depression storage for −95 100
impervious area (mm) Single-Objective Calibration
Dstore-Perv Depression storage for −95 300
Results of the single-objection calibration are summarized in Fig. 2
pervious area (mm)
%Zero-Imperv Percent of the impervious −100 100
and Tables 2 and 3. Fig. 2 compares the streamflow simulated using
area with no depression the optimal parameter values identified by DDS with the stream-
storage (%) flow observed at Swatelle station for both calibration [Figs. 2(a–d)]
Curve number NRCS runoff curve number −20 20 and verification [Figs. 2(e–h)] periods. Performance of the cali­
Drying Time Time for a fully saturated soil −95 100 brated model was also tested using the efficiency criteria given
to completely dry (days) in Table 2. The graphical comparison and the efficiency criteria
Conduit n Manning’s roughness −50 100 indicate that the single-objection calibration performed very well
coefficient for conduit for both calibration and verification periods. However, as shown
Error parameters Parameter values in Fig. 2, a closer look into the characteristics of the residuals
σ0 Heteroscedasticity intercept 0 100 (i.e., the difference between the observed streamflow and the
(m3 =s) streamflow simulated by the calibrated model) depicts that the as­
σ1 Heteroscedasticity slope 0 1 sumptions (i.e., homoscedasticity, Gaussian distribution, and tem­
ϕ1 Lag one autocorrelation −1 1 poral independence) made by the objective functions almost always
coefficient used in model calibrations, including the MAE used for this study,
ϕ2 Lag two autocorrelation −1 1
are unjustified. The residuals exhibited heteroscedasticity (i.e., they
coefficient
ϕ3 Lag three autocorrelation −1 1
increase with the magnitude of streamflow) as shown in Fig. 2(b),
coefficient they do not follow Gaussian distribution [Fig. 2(c)], and they are
β Kurtosis parameter −1 1 temporally correlated [Fig. 2(d)] for both calibration and verifica­
tion periods. Similar findings have been reported by other studies
including those by Schoups and Vrugt (2010) and regarding char­
against which performance of the model simulations is evaluated; acteristics of the residuals generated from solutions of the tradi­
and N = number of data points (observations). tional calibration methods.
In addition to relying on unrealistic assumptions on residuals,
the conventional calibration models attempt to identify a single best
Parameter Posteriors and Predictive Uncertainty solution based on the assumption that data, model structure, and
Posterior distributions of the 11 runoff parameters were estimated model parameters are all error-free. It is now a common knowledge
with DREAMðZSÞ and the GL function. Six additional error model that input and output data are subjected to substantial uncertainty;
parameters were considered for the GL. These include σ0 and σ1 in no model structure is a true representation of the watershed being
Eq. (7), to explicitly account for heteroscedasticity of model resid­ studied, and optimal parameter sets are not unique for a given
uals, the kurtosis parameter, β, in Eq. (5) to account for non- watershed (i.e., multiple set of parameters can be equally good for
normality of the residuals, and ϕ1 , ϕ2 , and ϕ3 in Eq. (8) to allow the watershed). As such, these calibration methods provide no in­
for autocorrelation of the residuals. It was initially assumed formation on reliability of the optimal solution. The uncertainty
that the residual distribution is not skewed, setting skewness analysis model described in this study has been developed to
parameter, ξ, equal to 1, and then this assumption was checked a address these vital limitations.
posteriori. A total of 100,000 SWMM5 simulations were used to
sample the posterior distribution of the parameters. Convergence of
Parameter Uncertainty
GL-DREAMðZSÞ to a stable posterior PDF was monitored using the
R statistic of Gelman and Rubin (1992). Convergence is declared As previously described, 100,000 SWMM5 simulations were run
when Rj ≤ 1.2 for all j ¼ 1; : : : ; d, where d is the number of model for the GL-DREAMðZSÞ UA model. Fig. 3 shows progress of the R
parameters being analyzed (i.e., 17 in this study). The last 5,000 statistic of Gelman and Rubin (1992) that has been used to test
GL-DREAMðZSÞ runs that meet the convergence criteria were ex­ convergence of the UA runs. The plot indicates that the 100,000
tracted, and parameter posteriors were determined and reported for model runs used for the analysis were sufficient to meet the
each individual parameter. convergence criteria of R ≤ 1.2 for all the SWMM5 and error
Once the posterior distribution of the model parameters is model parameters considered for the study. Fig. 4 shows the pos­
known, runoff predictive uncertainty can be estimated by propa­ terior histograms obtained for the parameters using the last 3,000
gating the different samples of the posterior distribution through simulations. As shown in Fig. 3, the convergence criteria was met
the SWMM5 model and reporting the respective prediction uncer­ after approximately 62,000 model simulations, implying that the
tainty ranges (e.g., 95% confidence interval). This prediction last 3,000 simulations used to generate the posterior histograms
interval, however, represents parameter uncertainty only; it doesn’t have met the convergence criteria, and each of these parameter sets
consider other sources of error, including model structure, forcing represents a reasonable SWMM5 model for the watershed.
 500
(a)

Streamflow [m /s]
400

3
300

200
100

0
0 100 200 300 400 500 600
Time Count [15 minute interval]

40 1 1

Partial autocorrelation
(b) (c) (d)
0.8
Residuals (a)

20
0.5

Density
0.6
0
0.4
0
-20 0.2
-40 0 -0.5
0 200 400 -2 0 2 0 10 20 30 40 50
3 Residuals (a) Lag [15 minute interval]
Simulated flow [m /s]

400
(e)
Streamflow [m /s]
3

300

200

100

0
0 50 100 150 200 250 300 350 400 450 500 550
Time Count [15 minute interval]

50 1 1
Partial autocorrelation
(f) (g) (h)
0.8
Residuals (a)

25
0.5
Density

0.6
0
0.4
0
-25 0.2
-50 0 -0.5
0 100 200 300 -2 0 2 0 10 20 30 40 50
3 Residuals (a) Lag [15 minute interval]
Simulated flow [m /s]

Fig. 2. Comparison of observed and simulated streamflow and diagnostic plots of the residuals obtained using DDS for both calibration (a–d) and
verification (e–h) periods: (a and e) comparison between observed (dots) and simulated (solid line) streamflow; (b and f) illustration of heterosce­
dasticity of the residuals; (c and g) comparison of observed PDF of the residuals to normal distribution; (d and h) illustration of autocorrelation of the
residuals; dashed lines in (d) and (h) show lower and upper bounds of the 95% confidence interval

Fig. 4(a) offers important information regarding the relative sets did not show significant variability as shown in Fig. 5.
importance of the SWMM5 parameters considered for the analysis. This suggests that the SWMM5 parameters that exhibited wide
Except for percentage imperviousness (Imperv), depression storage uncertainty range do not have substantial effect on rainfall-runoff
for impervious subareas (Dstore-Imperv), and percentage of the im­ characteristics of the Ballona Creek watershed. This has practical
pervious subarea with no depression storage (% Zero Imperv), the implication, for example, in terms of prioritizing resources on data
uncertainty bound of the other SWMM5 parameters is very wide. collection. Availability of more accurate data that characterize the
Nonetheless, the streamflow simulated by the last 3,000 parameter insensitive parameters may not help in improving accuracy of run­
off simulation for the watershed. On the contrary, uncertainty of the
Table 2. Efficiency Criteria Values Obtained Using DDS and model predictions can be further reduced from having more reliable
GL-DREAMðZSÞ imperviousness data. Unlike the SWMM5 parameters, the error
Volume (mm) model parameters given in Fig. 4(b) produced narrow uncertainty
MAE Bias range indicating their identifiablity.
Method Period (m3 =s) NSE (%) Observed Simulated Tables 2 and 3 compare the solution of the GL-DREAMðZSÞ
DDS C 6.89 0.94 11.15 59.6 66.2 model with the single-objection calibration results. Performance
V 9.79 0.90 −18.82 83.6 68.9 of the maximum likelihood solution has been summarized using
GL-DREAMðZSÞ C 7.74 0.93 10.94 59.6 66.1 several efficiency criteria in Table 2. The results show that the
V 10.57 0.88 −20.40 83.6 67.6 ML solution performed very well but not as well as the DDS sol-
Note: C= calibration; V= verification. ution. This is not surprising because the objective function used by
Table 3. Percentage Adjustments Obtained by DDS, the ML, and the 95% This shows that the insensitive parameters have minimal impact on
Confidence Interval Obtained by GL-DREAMðZSÞ and Optimal Parameter runoff from the watershed and thus on the likelihood function. The
Values Determined for One of the Subcatchments in the Ballona Creek ML adjustments recommended for these insensitive parameters are
Watershed meaningless as any adjustment within the wide range of the pos­
Percentage adjustments Parameter valuesa terior PDFs will produce almost identical runoff and likelihood
GL-DREAMðZSÞ function value.
One interesting observation is the recommendation by both ML
Lower Upper Initial Optimal
and DDS solutions to decrease percent imperviousness of the
Parameter DDS ML bound bound value value
watershed by approximately 15% so that model simulations closely
Width 137.1 117.0 −12.6 126.3 460 998 match observed runoff. Given that only the largest 72 storm drains
Slope 2.7 −7.4 −40.1 99.5 1.79 1.66 were considered in the study by ignoring hundreds of smaller storm
% Imperv −14.9 −15.0 −15.0 −12.7 70 59 drains and channels in the watershed, one would intuitively expect
N-Imperv 32.1 79.3 5.8 99.5 0.01 0.02
a solution that speeds up travel time (e.g., increase in percent
N-Perv −42.8 38.6 −46.5 99.0 0.02 0.03
Dstore-Imperv 99.8 98.2 84.9 99.9 0.25 0.50 imperviousness and steeper slope) to compensate for impact of
Dstore-Perv 234.7 1.8 72.2 293.6 1.27 1.29 the ignored storm drains on travel time. The suggested decrease
% Zero-Imperv −99.9 −100.0 −100.0 −94.5 100 0 in percentage imperviousness is believed to be related to the
Curve Number −3.2 8.4 −17.5 0.7 69 75 assumption made regarding connectivity of the impervious and
Drying Time −6.9 −93.1 −68.0 89.7 4 0.28 pervious subareas in each subwatershed. Both subarea types were
Conduit n 31.0 28.0 34.0 96.0 0.013 0.016 assumed to directly flow to the outlet of each subwatershed,
a
Initial values refer to the actual initial parameter values assigned to one of whereas in reality only a fraction of the impervious subareas
the subcatchments, and the optimal values are determined from the ML may be directly connected to the engineered drainage system. This
percent adjustments and the initial parameter values. See Table 1 for units. assumption might have led to overestimation of the effective per­
centage imperviousness of the watershed in the model.
2

Predictive Uncertainty
1.8
Understanding the total predictive uncertainty associated with
model simulations is very essential for decision makers. Modeling
1.6 uncertainties are believed to arise from imprecise knowledge of the
R Statistic

temporal and spatial variability of input and observed system

1.4
1.2
1
0 50,000
Simulation #
Fig. 3. Convergence plot of the Gelman and Rubin R statistic for the
parameters analyzed using GL-DREAMðZSÞ ; dashed line shows conver­
gence threshold of R ¼ 1.2
the two methods is different. Although DDS attempts to minimize
the MAE, the GL-DREAMðZSÞ model uses the GL function as ob­
jective function to remove heteroscedasticity and autocorrelation
from the residuals while also attempting to minimize the residuals.
The ML parameter percentage adjustments, the upper and lower
bounds of the 95% confidence interval extracted from the posteri­
ors, and the optimal adjustments identified by the DDS are given
in Table 3. The initial parameter values assigned to one of the
subwatersheds in the study area and the optimal parameter values
calculated from the initial values and the ML percent adjustments
are also given in Table 3.
The optimal parameter adjustments identified by the ML and the
DDS for corresponding SWMM 5 parameters are substantially
different except for the results of Imperv, Dstore-Imperv, % Zero
Imperv, and Conduit n (i.e., Manning’s n for conduits). This
further confirms insensitivity of runoff from the watershed to the
majority of SWMM5 parameters considered in the analysis. The
95% confidence interval is very wide for the insensitive model
parameters. For some of the insensitive parameters (i.e., Dstore-
Perv, Curve Number, Drying Time and Conduit N), the ML solu­
tions are outside the 95% confidence interval as shown in Table 3.
response, from the assumptions and simplifications made in the
simulation model to represent physical processes in the watershed,
and owing to the parameter uncertainty described in the previous
section. In the past, parameter uncertainty has been assumed to
represent all uncertainty sources (Beven and Freer 2001; Muleta
and Nicklow 2005). However, as shown in Fig. 5, parameter
100,000 uncertainty for the study watershed is very minimal indicating that,
at least for the Ballona Creek watershed, parameter uncertainty
alone cannot represent the total predictive uncertainty associated
with simulation models. Similar findings have been reported
by previous studies including Kuczera et al. (2006) using different
hydrologic models and application watersheds.
Fig. 6(a) shows a 95% confidence interval for the total predic­
tive uncertainty generated for the calibration [Figs. 6(a–d)] and
verification [Figs. 6(e–h)] periods. The predictive uncertainty
has been determined using the last 3,000 parameter sets identified
by the GL-DREAMðZSÞ model as previously described. Fig. 6 in­
dicates substantial uncertainty for the watershed especially for low
flow simulations. Wider uncertainty bound for low flows might
have been obtained because the likelihood function used in the
UA (i.e., the GL function) is biased towards peak flows that seem
to have been simulated with a higher degree of reliability (i.e., nar­
row bounds). Overall, the total predictive uncertainty bounds seem
reasonably accurate because they bracketed more than 75% of the
observed data, albeit lower than the theoretically expected value of
95%, for both calibration and verification periods. This indicates
that the MCM-based formal, Bayesian methodology pursued in this
study is promising for UA of urban drainage models.
Fig. 6 also shows diagnostic plots of the residuals derived from
the ML solution. Fig. 6(b) shows that the residuals are not sensitive
to the magnitude of streamflow, indicating that heteroscedasticity
has been removed by the GL function. Fig. 6(c) clearly shows
the Laplace (i.e., double-exponential) distribution used by the
error model is consistent with the PDF of the residuals of the ML
 1200 2500 1000 1200

0 0 0 0
-20 Width 150 -100 Slope 125 -15 Imperv -12 -20 120
N-Imperv

2000 1500 1000 1500

0 0 0 0
-75 100 80 100 0 320 -100 % Zero Imperv -94
Dstore-Imperv Dstore-Perv
N-Perv

800 1200 1500

0 0 0
-20 CN 10 -100 Drying Time 100 30 Manning's n 100
(a)

1200 1800 1500

600 900 750

0 0 0
0 1 0.117 0.12 0.96 1
s0 x 10-6 s1 β

2000 1500

750
1000 750

0 0 0
0.845 0.85 -0.105 φ2 -0.1 0.185
φ3
0.205
(b) φ1

Fig. 4. Posterior PDFs determined using GL-DREAMðZSÞ for (a) SWMM5 runoff parameters; (b) GL error model parameters; ordinate of the plots
show frequency of occurrence

solution. Temporal dependence of the residuals is shown in Conclusions

Fig. 6(d), which indicates that the residuals still exhibit substantial
dependence at lag-one and lag-two autocorrelations. However,
the temporal correlation has been significantly reduced compared
to the DDS solution shown in Fig. 2(d). Given the short simulation
time interval (i.e., 15-min) used for the study, which is typical
in urban drainage modeling, the difficulty of removing temporal
correlation in its entirety is understandable. Generally, the diag­
nostic plots demonstrate that the assumptions made by the
GL-DREAMðZSÞ model regarding the characteristics of the resid­
uals are consistent with properties of the residuals derived from
the ML solution.
This paper describes an MCMC-based formal, Bayesian method­
ology for parameter uncertainty and total predictive uncertainty
analysis of a widely used urban storm water management model.
The methodology has been illustrated using the Ballona Creek
watershed, a heavily urbanized watershed located in the Los
Angeles Basin, California. Solution of the UA model has been
compared with the optimal solution typically derived by using
the traditional calibration methods widely used in water re­
sources modeling. Furthermore, validity of the assumptions com­
monly made with regard to characteristics of model residuals in the
 objective functions often used for model calibration has been
examined. Flexibility of the likelihood function used in the UA
model to accommodate the characteristics of the residuals has been
Calibration Period Verification Period
demonstrated. The subsequent paragraphs summarize major con-
Streamflow [m3/s]

clusions of the study.

Rainfall [mm]
The runoff simulated using the optimal solution identified by the
single-objection calibration attempt was in good agreement with
the observed counterparts when evaluated graphically and by using
several goodness-of-fit measures. However, diagnostic analysis of
the residuals indicates that the assumptions of homoscedasticity,
temporal independence, and Gaussian distribution commonly made
in such traditional calibration models are unjustified. On the other
hand, the maximum likelihood solutions determined using the UA
model produced runoff simulations that are of comparable accuracy
Time Count [15 Minute Interval] with that of the single-objection calibration solutions while accu­
rately characterizing the structure of the model residuals. The
Fig. 5. Comparison of observed runoff (dotted) to the 95% confidence
assumptions made by the error model used in the UA methodology
interval bounds (lines) determined using GL-DREAMðZSÞ , considering
were found consistent with the characteristics of the residuals
only parameter uncertainty for calibration and verification periods
generated from the ML solution.

600
(a)
Streamflow [m /s]

400
3

200

-200
0 100 200 300 400 500 600
Time Count [15 minute interval]

40 1 1
Partial autocorrelation
(b) (c) (d)
20 0.8
Residuals (a)

0.5
Density

0.6
0
0.4
0
-20 0.2
-40 0 -0.5
0 100 200 300 400 -2 0 2 0 10 20 30 40 50
Residuals (a) Lag [15 minute interval]
Simulated flow [m3/s]

600
(e)
Streamflow [m /s]

400
3

200

-200
0 50 100 150 200 250 300 350 400 450 500 550
Time Count [15 minute interval]

20 1 1
Partial autocorrelation

(f) (g) (h)

10
Residuals (a)

0.5
Density

0 0.5
0
-10

-20 0 -0.5
0 100 200 300 -2 0 2 0 10 20 30 40 50
Simulated flow [m3/s] Residuals (a) Lag [15 minute interval]

Fig. 6. Total predictive uncertainty (95% confidence interval) and diagnostic plots of the residuals obtained using GL-DREAMðZSÞ for calibration
(a–d) and verification (e–h) periods: (a and e) comparison between observed streamflow (dots) to the 95% confidence bounds (solid line); (b and f)
show that the residuals are homoscedastic; (c and g) comparison of observed PDF (dotted) of the residuals to the assumed distribution (solid line) in
GL; (d and h) show autocorrelation of the residuals; dashed lines in (d) and (h) show lower and upper bounds of the 95% confidence interval
 In addition to accurately simulating runoff and properly charac­
terizing the residuals, the UA model has successfully determined
parameter uncertainty and total predictive uncertainty. The param­
eter posteriors showed that eight of the 11 SWMM5 parameters
considered for the analysis exhibited wide uncertainty bound,
whereas the runoff simulated for the watershed considering param­
eter uncertainty alone showed no appreciable variability. This sug­
gests that runoff is sensitive only to three (i.e., Imperv, Dstore
Imperv, % Zero Imperv) of the 11 parameters. Additionally, the ML
solution identified by the UA model and the optimal solution
determined by DDS showed good agreement only for four (Imperv,
Dstore Imperv, % Zero Imperv, Conduit n) of the 11 SWMM5
parameters confirming nonidentifiablity of the insensitive parame­
ters. Results also suggest that contribution of parameter uncertainty
to total predictive uncertainty is insignificant for the study water­
shed, underlying the importance of the other sources of predictive
uncertainty for Ballona Creek watershed.
The 95% confidence interval determined for total predictive
uncertainty using the UA model bracketed the majority of the
observed data, demonstrating reasonable accuracy of the UA
result. Satisfactory total predictive uncertainty bounds were gen­
erated for both calibration and verification periods although the
verification period results seem less adequate. Overall, the UA
methodology proved promising for sensitivity analysis, calibra­
tion, parameter uncertainty, and total predictive uncertainty
analysis of urban storm water drainage models at least for the
short-simulation durations considered in this study. Applications
to additional watersheds in other hydroclimatic regions can help
further examine this potential. The subsequent study will investi­
gate application to continuous simulations and on ways to use the
predictive uncertainty in decision making such as for optimal
SCM design applications.
References
Amenu, G. G. (2011). “A comparative study of water quality conditions
between heavily urbanized and less urbanized watersheds of Los
Angeles Basin.” Proc., World Environmental and Water Resources
Congress 2011: Bearing Knowledge for Sustainability, ASCE, Reston,
VA, 680–690.
Ball, J. E. (2009). “Discussion of ‘Automatic calibration of the U.S. EPA
SWMM model for a large urban catchment’ by J. Barco, K. M. Wong,
and M. K. Stenstrom.” J. Hydraul. Eng., 135(12), 1108–1110.
Barco, J., Wong, K. M., and Stenstrom, M. K. (2008). “Automatic calibra­
tion of the U.S. EPA SWMM model for a large urban catchment.”
J. Hydraul. Eng., 134(4), 466–474.
Bay, S., Jones, B. H., Schiff, K., and Washburn, L. (2003). “Water quality
impacts of stormwater discharges to Santa Monica Bay.” Marine Envi­
ron. Res., 56(1–2), 205–223.
Beven, K. (2006). “A manifesto for the equifinality thesis.” J. Hydrol.,
320(1–2), 28–36.
Beven, K., and Freer, J. (2001). “Equifinality, data assimilation, and uncer­
tainty estimation in mechanistic modeling of complex environmental
systems using the GLUE methodology.” J. Hydrol., 249(1–4), 11–29.
Beven, K. J., and Binley, A. M. (1992). “The future of distributed models:
Model calibration and uncertainty prediction.” Hydrol. Processes, 6(3),
279–298.
Beven, K. J., and Freer, J. (2001). “Equifinality, data assimilation, and un­
certainty estimation in mechanistic modeling of complex environmental
systems using the GLUE methodology.” J. Hydrol., 249(1–4), 11–29.
Box, G. E. P., and Tiao, G. C. (1992). Bayesian inference in statistical
analysis, Wiley, New York.
Burian, S. J., McPherson, T. N., Brown, M. J., and Turin, H. J. (2000).
“Development of a stormwater model for the Ballona Creek watershed.”
Los Alamos Unclassified Rep. (LA-UR-00-1849), Los Alamos National
Laboratory, Presented at 1st Ballona Wetlands Symp., Los Angeles,
CA.
Deletic, A., et al. (2012). “Assessing uncertainties in urban drainage
models.” Phys. Chem. Earth, Parts A/B/C, 42–44(2012), 3–10.
Dotto, C. B. S., et al. (2012). “Comparison of different uncertainty
techniques in urban stormwater quantity and quality modelling.” Water
Res., 46(8), 2545–2558.
Duan, Q., Sorooshian, S., and Gupta, V. K. (1992). “Effective and efficient
global optimization for conceptual rainfall-runoff models.” Water
Resour. Res., 28(4), 1015–1031.
EPA. (2007). “National water quality inventory: Report to congress; 2002
reporting cycle.” Document No. EPA-841-R-07-001, Washington, DC.
Fang, T., and Ball, J. E. (2007). “Evaluation of spatially variable parameters
in a complex system: An application of a genetic algorithm.” J. Hydro-
inform., 9(3), 163–173.
Feyen, L., Khalas, M, and Vrugt, J. A. (2008). “Semi-distributed parameter
optimization and uncertainty assessment for large-scale streamflow sim­
ulation using global optimization.” Hydrol. Sci. J., 53(2), 293–308.
Freni, G., Mannina, G., and Viviani, G. (2008). “Uncertainty in urban
stormwater quality modelling: The effect of acceptability threshold
in the GLUE methodology.” Water Res., 42(8–9), 2061–2072.
Freni, G., Mannina, G., and Viviani, G. (2009a). “Uncertainty in urban
stormwater quality modelling: The influence of likelihood measure
formulation in the GLUE methodology.” Sci. Total Environ., 408(1),
138–145.
Freni, G., Mannina, G., and Viviani, G. (2009b). “Urban runoff modelling
uncertainty: Comparison among Bayesian and pseudo-Bayesian meth­
ods.” Environ. Modell. Software, 24(9), 1100–1111.
Gelman, A., and Rubin, D. B. (1992). “Inference from iterative simulation
using multiple sequences.” Stat. Sci., 7(4), 457–472.
Gironás, J., Niemann, J. D., Roesner, L. A., Rodriguez, F., and Andrieu, H.
(2010). “Evaluation of methods for representing urban terrain in storm-
water modeling.” J. Hydrol. Eng., 15(1), 1–14.
Gotzinger, J., and Bardossy, A. (2008). “Generic error model for calibration
and uncertainty estimation of hydrological models.” Water Resour.
Res., 44(12), W00B07.
Green, W. H., and Ampt, G. (1911). “Studies of soil physics. Part 1. The
flow of air and water through soils.” J. Agr. Soc., 4, 1–24.
Holland, J. H. (1975). Adaptation in natural and artificial systems, Uni­
versity of Michigan Press, Ann Arbor, MI.
Horton, R. E. (1937). “Determination of infiltration-capacity for large
drainage basins.” Trans. Am. Geophys. Union, 18, 371–385.
Kavetski, D., Kuczera, G., and Franks, S. W. (2006). “Bayesian analysis of
input uncertainty in hydrological modeling: 1. Theory.” Water Resour.
Res., 42(3), W03407.
Kuczera, G. (1983). “Improved parameter inference in catchment models,
1. Evaluating parameter uncertainty.” Water Resour. Res., 19(5),
1151–1162.
Kuczera, G., Kavetski, D., Franks, S., and Thyer, M. (2006). “Towards a
Bayesian total error analysis of conceptual rainfall runoff models: Char­
acterizing model error using storm-dependent parameters.” J. Hydrol.,
331(1–2), 161–177.
Kuczera, G., and Parent, E. (1998). “Monte Carlo assessment of parameter
uncertainty in conceptual catchment models: The Metropolis algo­
rithm.” J. Hydrol., 211(1–4), 69–85.
Laloy, E., and Vrugt, J. A. (2012). “High-dimensional posterior exploration
of hydrologic models using multiple-try DREAM(ZS) and high-perfor­
mance computing.” Water Resour. Res., 48, W01526.
Los Angeles County Department of Public Works (LACDPW). (2011).
“Ballona Creek Watershed.” 〈http://ladpw.org/wmd/watershed/bc/〉
(Jul. 26, 2011).
Mannina, G., and Viviani, G. (2010). “An urban drainage stormwater
quality model: Model development and uncertainty quantification.”
J. Hydrol., 381(3–4), 248–265.
Mantovan, P., and Todini, E. (2006). “Hydrological forecasting uncertainty
assessment: Incoherence of the GLUE methodology.” J. Hydrol.,
330(1–2), 368–381.
Montanari, A., Shoemaker, C. A., and van de Giesen, N. (2009).
“Introduction to special section on uncertainty assessment in surface
 and subsurface hydrology: An overview of issues and challenges.”
Water Resour. Res., 45(12), W00B00.
Moradkhani, H., and Sorooshian, S. (2008). “General review of rainfall-
runoff modeling: Model calibration, data assimilation, and uncertainty
analysis.” Hydrological modeling and water cycle: Coupling of the
atmospheric and hydrological models, Vol. 63, Springer, Water Science
and Technology Library, Berlin, Heidelberg, 1–23.
Muleta, M. K. (2010). “Comparison of model evaluation methods to
develop a comprehensive watershed simulation model.” World Environ­
mental and Water Resources Congress 2010, ASCE, Reston, VA,
2492–2501.
Muleta, M. K. (2012). “Sensitivity of model performance to the efficiency
criteria used as objective function during automatic calibrations.”
J. Hydrol. Eng., 17(6), 756–767.
Muleta, M. K., and Nicklow, J. W. (2005). “Sensitivity and uncertainty
analysis coupled with automatic calibration for a distributed watershed
model.” J. Hydrol., 306(1–4), 127–145.
Nash, J. E., and Sutcliffe, J. V. (1970). “River flow forecasting through
conceptual models. Part I- A discussion of principles.” J. Hydrol.,
125, 277–291.
National Oceanic, and Atmospheric Administration (NOAA). (2013).
“Atlas 14 point precipitation frequency estimates.” Silver Spring,
MD. 〈http://hdsc.nws.noaa.gov/hdsc/pfds/pfds_map_pr.html〉 (Jun. 25,
2013).
Natural Resources Conservation Service (NRCS). (2013). “Soil data mart.”
U.S. Dept. of Agriculture, Washington, DC. 〈http://soildatamart.nrcs
.usda.gov/〉 (Jun. 25, 2013).
National Research Council (NRC). (2008). Urban stormwater management
in the United States, The National Academies Press, Washington, DC.
Rossman, L. A. (2010). “Stormwater management model user’s manual,
version 5.” EPA/600/R-05/040, U.S. EPA, Cincinnati.
Schoups, G., and Vrugt, J. A. (2010). “A formal likelihood function
for parameter and predictive inference of hydrologic models with cor­
related, heteroscedastic, and non-Gaussian errors.” Water Resour. Res.,
46(10), W10531.
Soil Conservation Service. (1964). “National engineering handbook.” Sec­
tion 4, Hydrology, Department of Agriculture, Washington, DC, 450.
Sorooshian, S., and Dracup, J. A. (1980). “Stochastic parameter estimation
procedures for hydrologic rainfall-runoff models: Correlated and heter­
oscedastic error cases.” Water Resour. Res., 16(2), 430–442.
Stedinger, J. R., Vogel, R. M., Lee, S. U., and Batchelder, R. (2008).
“Appraisal of the generalized likelihood uncertainty estimation (GLUE)
method.” Water Resour. Res., 44(12), W00B06.
Stein, E. D., and Tiefenthaler, L. L. (2005). “Dry-weather metals and bac­
teria loading in an arid, urban watershed: Ballona Creek, California.”
Water Air Soil Pollut., 164(1–4), 367–382.
Stenstrom, M. K., and Strecker, E. (1993). “Assessment of storm drain
sources of contaminants to Santa Monica Bay: Annual pollutants load­
ings to Santa Monica Bay from stormwater runoff.” Rep. No. UCLA­
ENG-93-62, Vol. 1, Univ. of California, Los Angeles, 1–248.
Thiemann, M., Trosset, M., Gupta, H. V., and Sorooshian, S. (2001).
“Bayesian recursive parameter estimation for hydrologic models.”
Water Resour. Res., 37(10), 2521–2535.
Thyer, M., Renard, B., Kavetski, D., Kuczera, G., Franks, S. W., and
Srikanthan, S. (2009). “Critical evaluation of parameter consistency
and predictive uncertainty in hydrological modeling: A case study
using Bayesian total error analysis.” Water Resour. Res., 45(12),
W00B14.
Tolson, B. A., and Shoemaker, C. A. (2007). “Dynamically dimensioned
search algorithm for computationally efficient watershed model calibra­
tion.” Water Resour. Res., 43(1), W01413.
USGS. (2013). “The national map.” USGS, Washington, DC. 〈http://
seamless.usgs.gov/〉 (Jun. 25, 2013).
Vrugt, J. A., Diks, C. G. H., Gupta, H. V., Bouten, W., and Verstraten, J. M.
(2005). “Improved treatment of uncertainty in hydrologic modeling:
Combining the strengths of global optimization and data assimilation.”
Water Resour. Res., 41(1), W01017.
Vrugt, J. A., Gupta, H. V., Bouten, W., and Sorooshian, S. (2003).
“A Shuffled Complex Evolution Metropolis algorithm for optimization
and uncertainty assessment of hydrologic model parameters.” Water
Resour. Res., 39(8), SWC 1-1–SWC 1-14.
Vrugt, J. A., Ter Braak, C. J. F., Clark, M. P., Hyman, J. M., and Robinson,
B. A. (2008). “Treatment of input uncertainty in hydrologic modeling:
Doing hydrology backward with Markov chain Monte Carlo simula­
tion.” Water Resour. Res., 44(12), W00B09.
Vrugt, J. A., Ter Braak, C. J. F., Diks, C. G. H., Higdon, D., Robinson,
B. A., and Hyman, J. M. (2009a). “Accelerating Markov chain Monte
Carlo simulation by differential evolution with self-adaptive random­
ized subspace sampling.” Int. J. Nonlinear Sci. Numer. Simul., 10(3),
271–288.
Vrugt, J. A., Ter Braak, C. J. F., Gupta, H. V., and Robinson, B. A. (2009b).
“Equifinality of formal (DREAM) and informal (GLUE) Bayesian ap­
proaches to hydrologic modeling?” Stoch. Environ. Res. Risk Assess.,
23(7), 1011–1026.