See discussions, stats, and author profiles for this publication at: https://www.researchgate.

net/publication/221936175

Development and Verification of an Online Artificial Intelligence System for

Detection of Bursts and Other Abnormal Flows

Article  in  Journal of Water Resources Planning and Management · May 2010

DOI: 10.1061/(ASCE)WR.1943-5452.0000030

CITATIONS READS

105 369

3 authors:

Steve Mounce Joby B Boxall

The University of Sheffield The University of Sheffield
78 PUBLICATIONS   874 CITATIONS    205 PUBLICATIONS   2,848 CITATIONS   

SEE PROFILE SEE PROFILE

John Machell
The University of Sheffield
63 PUBLICATIONS   790 CITATIONS   

SEE PROFILE

Some of the authors of this publication are also working on these related projects:

Assessing the Underworld View project

Water age and quality in drinking water distribution networks View project

All content following this page was uploaded by Steve Mounce on 15 March 2019.

The user has requested enhancement of the downloaded file.

 Development and Verification of an Online Artificial
Intelligence System for Detection of Bursts and
Other Abnormal Flows
S. R. Mounce1; J. B. Boxall2; and J. Machell3

Abstract: Water lost through leakage from water distribution networks is often appreciable. As pressure increases on water resources,
Downloaded from ascelibrary.org by University of Sheffield on 03/15/19. Copyright ASCE. For personal use only; all rights reserved.

there is a growing emphasis for water service providers to minimize this loss. The objective of the work presented in this paper was to
assess the online application and resulting benefits of an artificial intelligence system for detection of leaks/bursts at district meter area
共DMA兲 level. An artificial neural network model, a mixture density network, was trained using a continually updated historic database that
constructed a probability density model of the future flow profile. A fuzzy inference system was used for classification; it compared latest
observed flow values with predicted flows over time windows such that in the event of abnormal flow conditions alerts are generated.
From the probability density functions of predicted flows, the fuzzy inference system provides confidence intervals associated with each
detection, these confidence values provide useful information for filtering and ranking alerts. Additionally an accurate estimate of
abnormal flow magnitude is produced to further aid in ranking of alerts. A water supply system in the U.K. was used for a case study with
near real-time flow data provided by general packet radio service. The online burst alert system was constructed to operate alongside an
existing flat-line alarm system, and continuously analyze a set of 144 DMAs every hour. The new system identified a number of events
and alerts were raised prior to their detection in the control room; either through flat-line alarms or customer contacts. Examples are given
of alert correlation with burst reports and subsequent mains repairs for a 2-month trial period. Forty four percent of alerts were found to
correspond to bursts confirmed by repair data or customer contacts, 32% of alerts were confirmed as unusual short-term demand from
manual analysis, 9% were related to known industrial events, and only 15% were ghosts. The results indicate that the system is an
effective and viable tool for online burst detection in water distribution systems with the potential to save water and improve customer
service.
DOI: 10.1061/共ASCE兲WR.1943-5452.0000030
CE Database subject headings: Water distribution systems; Water management; Leakage; Artificial intelligence; Neural networks.
Author keywords: Water distribution systems; Water management; Leakage; Artificial intelligence; Neural networks.

Introduction
Water supplies are under great duress as a result of factors such as
rapid population growth, unsustainable consumption patterns,
poor management practices, inadequate investment in infrastruc-
ture, and low efficiency in water use. This situation is likely to
become more acute in the future as a result of the more extreme
events predicted to occur as a consequence of climate change and
described in the GEO-4 scenarios 关United Nations Environment
Programme 共UNEP兲 2007兴. Leakage losses from distribution net-
1 sustainable business practice in recent years have led water ser-
Pennine Water Group, Dept. of Civil and Structural Engineering,
Univ. of Sheffield, Sheffield S1 3JD, U.K. 共corresponding author兲.
E-mail: S.R.Mounce@sheffield.ac.uk
2
Pennine Water Group, Dept. of Civil and Structural Engineering,
Univ. of Sheffield, Sheffield S1 3JD, U.K. E-mail: J.B.Boxall@sheffield.
ac.uk
3
Pennine Water Group, Dept. of Civil and Structural Engineering,
Univ. of Sheffield, Sheffield S1 3JD, U.K. E-mail: j.machell@sheffield.
ac.uk
Note. This manuscript was submitted on September 29, 2008; ap-
proved on April 30, 2009; published online on May 15, 2009. Discussion
period open until October 1, 2010; separate discussions must be submit-
ted for individual papers. This paper is part of the Journal of Water
Resources Planning and Management, Vol. 136, No. 3, May 1, 2010.
©ASCE, ISSN 0733-9496/2010/3-309–318/$25.00.
works are often high, and it is not unusual for more than one-third
of the water to be lost before delivery, even in many Central and
Eastern European countries. In the United Kingdom, often seen as
a European leader, leakage as a proportion of water supplied has
been consistently around 22–23% during the current decade
共OFWAT 2006兲, as this is close to the “economic” level of leak-
age using current tools, techniques, and technologies. The eco-
nomic level of leakage 共ELL兲 is defined as “the level at which it
costs more to reduce leakage further than to produce that water
from an alternative source” 共House of Lords 2006兲. This in-
creased pressure on water resources and a growing emphasis on
vice providers to consider water loss in treated water supply sys-
tems as a key “water supply system performance indicator.” It is
no longer simply an economic issue, and the environmental and
social costs must also be factored in. Hence, technological devel-
opments to reduce losses in water distribution systems are essen-
tial.
An important issue for leaks and bursts is that of timely detec-
tion minimizing water losses. Burst duration can be divided con-
ceptually into awareness, location, and repair times 关Water
Research Centre 共WRC兲 1994兴. Often there will be a gap between
a leak commencing and the water company becoming aware of it.
One of the primary aims of the described research is to minimize
the time prior to awareness, as shown conceptually in Fig. 1. The

JOURNAL OF WATER RESOURCES PLANNING AND MANAGEMENT © ASCE / MAY/JUNE 2010 / 309

J. Water Resour. Plann. Manage., 2010, 136(3): 309-318


Fig. 1. Life cycle of an example burst
system is designed to provide detection of bursts and leaks as they
occur but not existing or background leakage.
This paper presents the results of applying an online artificial
Downloaded from ascelibrary.org by University of Sheffield on 03/15/19. Copyright ASCE. For personal use only; all rights reserved.
intelligence 共AI兲 system for automated hourly analysis to a pilot
case study consisting of 144 district meter areas 共DMAs兲 and an
evaluation of system performance with direct feedback to and
from the water company control room. This self-learning AI sys-
tem is not reliant on any special hardware or network configura-
tion and produces intelligent “smart alarms.”
Background
Leakage
Water utilities generally first obtain awareness of bursts through
customer contacts via call centers where the details of service
problems are collected such as complaints of low pressure or
discoloration and possibly signs of visible surface water. Al-
though large bursts in a water supply system are usually found
and fixed quite rapidly due to multiple customer contacts, not all
bursts result in visible surface water and can go undetected. Re-
lying solely on customer contacts adds to the poor image of water
utility companies, and a more proactive approach is needed in
which the control room and field teams are alerted to failure be-
fore customers make contact; hence minimizing limitation of sup-
plies, maintaining service to customers, and limiting water losses.
Leaks that do not impact on customers are hence undetected and
become background leakage. Water companies typically address
background leakage when cumulative background leakage be-
comes apparent in flow data and is above the ELL for a given
area. Leakage detection activities may be classified in two groups:
reactive and proactive.
Reactive leak location is carried out by water company person-
nel once it is deemed that a sufficiently serious problem has de-
veloped, identified through customer contacts, or via other
information. This process commonly entails application of many
different techniques ranging from listening rods or sticks through
step testing to geophones and noise correlators. Several research
and development projects have explored the potential for leak
location by searching for deviation in pressure signals across mul-
tiple pressure monitoring points. Misiunas et al. 共2005兲 presented
such a technique, validated for both laboratory and field condi-
tions and then subsequently demonstrated the potential for detec-
tion and location of medium to large bursts within a DMA from
continuous monitoring of network inflow and pressure for a case
study 共Misiunas et al. 2006兲. Shinozuka and Liang 共2005兲 devel-
oped a technique to identify the location and severity of leak
damage using a damage detection approach based on a neural
network inverse analysis method. Stoianov et al. 共2007兲 reported
results of field trials to explore leakage detection using decentral-
ized “intelligent” sensors for a near real-time monitoring system
for water transmission pipelines. While all these techniques dem-
310 / JOURNAL OF WATER RESOURCES PLANNING AND MANAGEMENT © ASCE / MAY/JUNE 2010
J. Water Resour. Plann. Manage., 2010, 136(3): 309-318
onstrate potential, they are not widely used in practice due to the
necessary larger than normal industry practice deployment of in-
struments and the accuracy of hydraulic models required. Real-
time detection of burst and leak incidences direct from analysis of
hydraulic data is not common practice in the water industry.
Proactive leakage detection. District metering is a widely
adopted practice in the United Kingdom and its use has increased
internationally. It is based on the subdivision of distribution sys-
tems into discrete zones by the permanent closure of valves and
the measurement of the flows into 共and in some cases out of兲 each
zone. A DMA will generally comprise 500–3,000 properties. Gen-
erally, import and export points of a DMA, water storage inlet/
outlets, pumping stations, large industrial users, a low pressure
point in the DMA, and important locations on trunk mains are
monitored with permanently installed flow meters and pressure
transducers 关UK Water Industry Research 共UKWIR兲 2006兴. By
monitoring average night flows, unusual changes in water vol-
umes can be detected. If the night flow minus the legitimate flow
is close to zero, leakage is assumed to be close to zero. In con-
trast, a positive residual will signify leakage in the absence of any
other factors. The identification and quantification of distribution
losses in this way relies on the accurate estimation of the expected
night flows 共McKenzie and Seago 2005兲. System operators will
generally decide, based on experience with the particular water
system, if night flow increase is due to a leak or other factors.
Although this technique can be effective, night flow analysis is
not necessarily conducted routinely or consistently due to reliance
on manual interpretation. Without customer contact, a leak may
run undetected for several months, or longer. Hence there is scope
for automated systems to address the issue of leak/burst identifi-
cation and reduce water losses.
Artificial Neural Networks
Recent developments in the field of computational intelligence
variously called soft computing, machine learning, or data-driven
modeling are helping to solve various problems in the water re-
sources domain. These techniques are ideal for analyzing engi-
neering problems where solutions need to be developed without
solving the microscale interactions that actually occur and may be
poorly understood, but where measured data are readily available.
Artificial neural networks ANNs have been widely applied in
different fields of engineering and are a modeling approach based
on how biological neural systems are believed to work. It has
been proved mathematically that ANNs are universal computing
machines capable of arbitrary nonlinear function approximation
provided they are given sufficient training data 共Hornik et al.
1989兲. ANNs have been successfully applied to a range of water
modeling problems and have displayed particular promise for
forecasting applications. They can be evaluated based on physical
model outcomes and experimental/field data can be further inte-
grated in order to enhance their performance. An ANN-based
model was effectively applied to reservoir inflow prediction and
operation 共Jain et al. 1999兲. The ANN was found to model the
high flows better, whereas low flows were better predicted
through the autoregressive integrated moving average model.
ANNs were used for predicting runoff over three medium-sized
watersheds in Kansas 共Anmala et al. 2000兲, they concluded that
recurrent ANNs with feedback generally resulted in best perfor-
mance. Bowden et al. 共2002兲 demonstrated two methodologies for
optimal division of data for ANN models by means of an example
of forecasting salinity in the River Murray at Murray Bridge
共South Australia兲 14 days in advance. ANNs have been applied to
 both short-term demand forecasting for distribution systems, i.e.,
for 24 h demand forecasts 共Greenaway et al. 2006兲 and for much
longer term demand prediction, such as over a 10-year horizon
共Chang and Makkeasorn 2006兲. Maier and Dandy 共2000兲 provide
a comprehensive review of 43 papers dealing with the use of
ANNs for the prediction and forecasting of water resources vari-
ables, as well as a useful protocol for developing such models.
Their study found that in all but two papers reviewed, feed-
forward networks were used, and that most used the back-
propagation training algorithm.
There are many forecasting problems where simple point pre-
dictions are not adequate to guide action. In the realm of water
resources management it is often important to know the probabil-
Downloaded from ascelibrary.org by University of Sheffield on 03/15/19. Copyright ASCE. For personal use only; all rights reserved.
ity of a variable falling within 共or outside兲 different ranges. It
could be that a 10% probability of flooding may be enough to
place emergency services on alert. Problems such as these have
generated an interest in the area of probability density forecasting.
The standard solution to regression problems is to apply a sum-
of-squares error function to N training data pairs, 共x1 , t1兲 , . . . ,
共xn , tn兲, to predict a conditional mean of a new unseen data vector,
具tn+1 兩 xn+1典. This form of point prediction is often inadequate in
practice because there is no indication of the level of uncertainty
in the model’s predictions. Uncertainty of predictions can be ex-
pressed in terms of probability distributions. A model of the pro-
cess can hence be created as a sequence of probability density
functions. This means that for any given input value, e.g., xn+1,
the model produces a conditional density function p共tn+1 兩 xn+1兲.
One mechanism for achieving this is described later in the meth-
odology.
Fuzzy Logic
A time series prediction on its own will not provide a classifica-
tion. Forecasting can be very useful, but many applications re-
quire event detection. ANNs can be used for classification in this
way, but often water resource data sets do not have a classifica-
tion vector required for the various training and test sets. For
example, in the case of burst detection the actual start date and
time of the burst is unknown 共or at least has high uncertainty,
except for extreme events兲. In this case, some approach is re-
quired to detect discrepancies between actual and predicted val-
ues, over some time period. Even where a classification ANN is
possible, the knowledge obtained is not readily accessible to
human inspection and is, in any case, virtually incomprehensible
共a “black box”兲. Fuzzy logic is a useful technique for building
knowledge based systems that represent the impreciseness asso-
ciated with human reasoning and also allows a degree of trans-
parency. Fuzzy sets can contain elements with only partial degree
of membership. A fuzzy inference system 共FIS兲 can be used to
enable a degree of certainty to be placed on a classification or
output 共propositions can be true to a certain degree–somewhere
between 0 and 1兲. The main components of the FIS are the fuzzy
membership functions defined for the set of variables in the sys-
tem, the set of fuzzy rules, and the algorithms and strategies for
defuzzification 共obtaining a single valued output兲. Mamlook and
Al-Jayyousi 共2003兲 use a fuzzy logic approach for analysis and
identification of factors that affect leakage, based on the weights
determined for each of the factors. The study revealed that the
major factors that affect leakage are: pipe age, pipe material, op-
erational aspects, and demand patterns. Hybrid systems using the
strengths of both ANNs and FIS 共or other expert systems兲 are
becoming increasingly popular.
JOURNAL OF WATER RESOURCES PLANNING AND MANAGEMENT © ASCE / MAY/JUNE 2010 / 311
J. Water Resour. Plann. Manage., 2010, 136(3): 309-318
Algorithms
The viability of using ANNs to analyze DMA flow meter data for
the identification of burst events from historical data sets was
presented in Mounce et al. 共2002兲. An ANN model called the
mixture density network 共MDN兲 as presented by Bishop 共1994兲
was used as the time series predictor, to produce a probability
density forecast rather than a point prediction as discussed earlier.
A MDN is a mixture density model combined with an ANN. It
consists of a two-hidden layer network; the first layer with Sig-
moidal units, and the second with Gaussian units. Empirical in-
vestigations indicated that 10–15 hidden units gave marginally
beneficial accuracy for time series prediction. Further, one or two
Gaussians and 800–1,000 training cycles gave good general per-
formance 共Mounce 2005兲. The target vector is clamped directly to
the Gaussian nodes and the network is trained using maximum
likelihood 共minimizing the negative log of the likelihood兲. The
mechanism for adapting the network parameters is the back-
propagation of error as in a standard MLP ANN; the only differ-
ence being the error function. Various parameters of the mixture
models 共mixing coefficients, means, and variances兲 are governed
by the outputs of the neural network which takes xt as its input.
The network is trained with a scaled conjugate gradient optimizer
with the error function E being the negative log-likelihood L of
再兺 冎
the training data
N N M
E = − ln L = − 兺
n=1
ln p共xn兲 = − 兺
n=1
ln
j=1
p共xn兩j兲P共j兲 where
再 冎
1 储x − ␮j储2
p共x兩j兲 = exp − 共1兲
共2␲␴2兲d/2 2␴2j
In Eq. 共1兲, x is a data point, p共x兲 is a mixture distribution with M
component density functions p共x 兩 j兲 and P共j兲 the mixing param-
eters 关the density functions are Gaussian in this case, and having
the adjustable parameters P共j兲, ␮ j, and ␴ j兴 and where d is the
dimensionality of the target space t 共for univariate time series
prediction, this equals 1兲.
The input is a lag vector of 96 past 共flow兲 values at a 15-min
time step and the target is a 1-day ahead prediction. In many
applications, the distance into the future is simply the next time
step—this is the usual approach for optimal accuracy of predic-
tion a short period into the future. In this system, a one day ahead
prediction allows a series of predictions to be built up for the last
24 h. This series of predictions can then be analyzed by the sys-
tem in conjunction with the actual observed last 24 h data 共facili-
tating at worst daily downloading of data兲. The classification
window used by the FIS is 12 or 24 h for normal event detection
and thus the one day ahead prediction enables the detection of
discrepancy caused by abnormal flow. A full theoretical justifica-
tion appears in Mounce et al. 共2003兲. Once trained, the MDN
network can predict the conditional probability density function
of the target data for any given value of the input vector, rather
than just a point prediction, which is the case for most other ANN
models used for time series prediction. An 85% confidence inter-
val for the predicted value is computed from the mixture model,
which is a linear combination of component densities 共general
normal distributions兲 and this is used in conjunction with the
observed data to form a residual that is analyzed by a FIS classi-
fication module. The main variable is the number of values in the
window for which the actual value lies above the upper range of
the confidence interval with the membership functions and fuzzy
rules as set out in Mounce 共2005兲. The FIS gives a fuzzy value on

Downloaded from ascelibrary.org by University of Sheffield on 03/15/19. Copyright ASCE. For personal use only; all rights reserved.
Fig. 2. High level system scheme
the presence of a flow abnormality within a windowed time pe-
riod 共configurable to between 2 and 24 h兲 at a particular confi-
dence level 共since deviation outside the confidence interval for a
particular singular time step does not, in practice, indicate a
burst兲. The window of analysis used by the FIS in the online
system was 12 h, selected on the basis that a good rule of thumb
for abnormal flow persisting before it can be potentially classified
as a burst is around this period 共other events, such as industrial
usage or fire fighting, can have a similar appearance especially at
night but tend to last for less time兲. The system was then further
developed to provide burst size estimation 共Mounce et al. 2007兲.
Online System
Significant modifications to the offline AI software previously
developed were necessary in order to make it suitable for auto-
mated online analysis, including strategies for dealing with data
quality, multiple sites, and alarm handling. A version of the sys-
tem, including a new graphical user interface 共GUI兲, was devel-
oped in MATLAB, which makes use of the Database toolbox, the
Fuzzy Logic toolbox and NETLAB 共Nabney 2001兲. Fig. 2 pro-
vides a scheme of the system integration for the online AI system.
The automated analysis system is data-driven starting from the
logger units which initiate calls to the telemetry software. The
transmitted data are then mirrored to a server and export function-
ality in the communications software is used to automatically
update 共every hour兲 a set of comma-separated values 共CSV兲 files.
An open database connectivity 共ODBC兲 text driver is used to
interface to an MS Access database storing the time series history,
current logger values are appended by a data warehouse applica-
tion. The online AI system accesses the database via an ODBC
driver. The system runs analysis at predefined intervals 共e.g.,
every hour兲 and any classifications by the FIS result in automated
text alerts which can be e-mailed or processed and stored in a
database.
Data Handling
Data obtained from water distribution systems can, in general, be
classed as “dirty” in comparison to other fields. This manifests as
large chunks of missing data, data from faulty loggers and the
presence of erroneous date stamps. Therefore, a methodology for
dealing with these issues was necessary in order to be able to
handle a large number of loggers with potentially varying data
312 / JOURNAL OF WATER RESOURCES PLANNING AND MANAGEMENT © ASCE / MAY/JUNE 2010
J. Water Resour. Plann. Manage., 2010, 136(3): 309-318
quality. The system assembles a potential data file and then con-
ducts several tests for amount and quality of data. First, the data
obtained via ODBC from the data warehouse is transformed into
internal MATLAB time series objects consisting of both time se-
ries data and events 共the events are loaded from the event data-
base兲. Events that can be applied to the time series include “reset”
共do not use any data for this logger before this date, e.g., in the
case of a network change such as rezoning兲, “alarm on,” and
“alarm off.” For a new site there would not necessarily be any
alarm states recorded, but over time these would be automatically
populated by the system itself. The data are then subjected to a
number of logic checks based on overall amount of valid data
with user definable parameters and representing a trade off be-
tween stringency and a satisfactory training set size. For the case
study, the rules were as follows when analyzing three months
training data 共where x is the highest number of consecutive days
with all raw data present, that is, data not missing or periods
corresponding to an alarm state, and y is the total of all days with
all raw data present兲:
IF x ⬍ 14 days
INVALID
ELSEIF 14⬍ x ⬍ 28
VALID, FLAG= ’ Low data period’
ELSEIF (y/TOTAL DAYS) ⬍0.15
INVALID
ELSEIF (y/TOTAL DAYS) ⬍0.30
VALID, FLAG= ’ Low data period’
ELSE
VALID
The site is either then passed, failed, or tagged as “low data
period—treat with caution.” The maximum period that the system
utilizes to train the ANN is a user definable value on the GUI, the
default setting is 3 months 共derived empirically as providing a
compromise between data quantity and data variability兲.
Flow sensor data obtained from a water distribution system is
usually in the form of time series—that is, a data stream whose
value is a function of time. In U.K. industry standard practice, the
data stream has been sampled at a regular time interval of 15 min
to produce a discrete series xt. A full theoretical basis for modeling
time series in this context using the state-space model can be
found in Mounce et al. 共2003兲. Time series data of real-world
phenomena is inherently nonstationary 共for example, in the
financial world such as shares and derivatives兲. If the series is
nonstationary, then all the typical results of the classical
regression analysis are not valid and results will be spurious. For
a stationary signal, the signal properties do not change much over
time. However, most interesting signals contain numerous
nonstationary or transitory characteristics: drift, seasonal trends,
abrupt changes, and beginnings and ends of events 共Kendall and
Ord 1990兲. These fluctuations may not be limited only to the mean
of the series, but may also affect its overall variance structure. The
time domain must be considered when adopting an MDN network
for a temporal prediction task. If the generator of the data itself
evolves with time, then time must be considered as an additional
variable in the joint probability density 共Bishop 1995兲. However,
the MDN can be extended to nonstationary problems, provided
that the model is treated as continuously adaptive. In other words,
for nonstationary data, the model must be reestimated within a
relatively short time interval. In the case of flow data for
 distribution systems, good quality time series are “almost”
stationary in that they exhibit stationarity over extended periods—
in effect, a set of quasi-stationary models. Therefore, by retraining
the networks with a slightly updated data set for the DMA, current
conditions can be continually built into the model.
A second data quality or assurance test is thus applied to assess
if training sets are “almost stationary” in order to restrict the use
of spurious or bad quality data. The training data period is
analyzed to assess how the mean and variance change over time,
with the mean ␮n and variance ␴2n calculated for each week’s data.
These values are compared to the average mean ␮n and variance
␴2n over the entire period. If either measure is an outlier of a range
specified by a user defined multiplier, e.g., ␮n / ␮MULT ⬍ ␮i or ␮i
⬎ ␮N ⴱ ␮MULT then week i is defined as failing the test, similarly
Downloaded from ascelibrary.org by University of Sheffield on 03/15/19. Copyright ASCE. For personal use only; all rights reserved.
for variance. If a user defined number of failures is exceeded for
mean 共␮LIM兲 and/or variance 共␴LIM 2
兲 then the training data are
rejected on this pass. For the case study, with 12 weeks of training
data and as a consequence of empirical testing, ␮MULT = 1.5,
␴MULT
2
= 1.5, ␮LIM = 3, and ␴LIM
2
= 6. In practice, around 25% of the
worst quality data sets were excluded at any particular time.
Each data stream/instrument/site is thus in one of four
automatically controlled states at any one time: fail stationary test;
normal; sufficient but low period training data; insufficient
training data. A site can also be manually activated/deactivated
using the GUI 共for example, in the event of known sensor failure
or planned operational work兲.
Training
Training sets which pass the screening tests are preprocessed 关see
Mounce et al. 共2006兲兴, including a simple linear transformation of
the data. Bowden et al. 共2003兲 found that when transforming data
for use with ANNs in water resources applications, taking a loga-
rithmic transformation of the data, removing the seasonality of
the data and transforming the inputs and outputs to normality
were all found to give significantly larger forecasting errors than a
simple linear transformation of the data. Missing data are filled in
using ARIMA modeling 共Vandaele 1983兲.
An ANN is trained for each valid logger when the system is
initialized and thereafter at some predefined time period in order
to capture the usual distribution of values. The retraining interval
is selectable on the GUI and by default is weekly, found to pro-
vide good compromise between computational expense, system
variability, and daily and weekly patterns of water use. If the
regime for the supply system being monitored changes drastically
共e.g., system alterations such as valve position modifications兲,
then the training set will need to be started anew 共reset event兲.
Testing
Each valid logger is also tested, with the latest data, at some
regular interval controlled by a timer 共selectable on the GUI and
by default hourly, dictated by the data update period兲. In a similar
manner to training sets, test data are checked for availability and
quality. If passed, the data are presented to the appropriately
trained neural network. If events are detected based on a user
definable minimum level of confidence, then an alarm event is
added to the event database for the DMA, and an automated
e-mail is sent out describing the alert 共SCADA ID, DMA name,
the FIS output level and confidence, the classification period, and
an estimate of the leakage flow rate兲.
The GUI 共see Fig. 3兲 allows an operator to review sites, and an
“alarm” state can be cleared if it is judged that the event has been
JOURNAL OF WATER RESOURCES PLANNING AND MANAGEMENT © ASCE / MAY/JUNE 2010 / 313
J. Water Resour. Plann. Manage., 2010, 136(3): 309-318
dealt with 共e.g., either a burst has been repaired or the alarm
related to an unusual demand兲. An option allows the automatic
clearing of alarms after a set period 共such as one week兲.
Harrogate and Dales Case Study
The Harrogate and Dales 共H&D兲 administrative area in North
Yorkshire in the U.K. stretches from Harrogate northwards to-
ward Richmond, Darlington, and Middlesbrough. The area con-
sists of nearly 200 DMAs 共excluding trunk main and industrial
user DMAs兲 and includes approximately 122,000 properties.
Yorkshire Water Service, the water utility company responsible,
conducted a network service pilot in the H&D area by installing
450 Cello loggers equipped with a general packet radio service
共GPRS兲 communications infrastructure to dramatically improve
data transfer for both flow and pressure data. Data are communi-
cated every 30 minutes and two readings are obtained 共15-minute
sampled data兲. In addition to telemetry, the pilot system incorpo-
rates high 共for flow兲 and low 共for pressure兲 flat-line alarms, the
original alarm values being set according to the high and low
average values over 12 months, plus or minus 20% respectively
共these levels were then manually adjusted as the pilot progressed兲.
This system 共the combination of GPRS and flat-line alarm levels兲
represents a significant step forward in detecting events and is
helping demonstrate a step change in awareness of system perfor-
mance. Faster detection of bursts/leaks in the system enables
Yorkshire Water to react to help minimize the inconvenience of
interruption to customers as well as decrease the damaging con-
sequences to infrastructure, in particular for detecting sudden
catastrophic bursts. Around 60–80 alerts are received per week on
this system, a significant number being ghosts, and many events
are still not detected prior to customer contact.
Yorkshire Water Services’ pilot provided the case study data
共flow only兲 sources to trial the AI data analysis system for detect-
ing more subtle change from DMA flow data. The online system
was operational and generating alerts from August 2007. As a
result of promising initial results, the utility company agreed to
allocate the additional resources required to perform validation of
alerts within an operational environment. Access to all DMAs
within the H&D region was provided, resulting in a 2-month trial
on 144 sites. It should be emphasized that this trial was on data
from a “live” network and not a desktop study using historical
data sets. Alerts from the trial were e-mailed automatically both to
the research team and to a dedicated e-mail account accessible by
network controllers. In many cases, leakage teams were notified
about a problem in a DMA following AI alerts, which conducted
a leak survey and located a leak. Sources of information for cor-
relating alerts included the following:
• Direct liaison with the control room;
• Daily e-mailed briefings on major events;
• Customer contact database 共customer reports of visible leak-
age兲;
• Comparison with flat-line alerts 共the AI system was used to
feedback to the flat-line system and help tune alarm levels兲;
and
• Work management system 共WMS兲 record of repairs database.
Proformas were completed by control room staff to bring to-
gether the information from these sources for each alert and hence
it was possible, in many cases, to build up an accurate picture of
the cause of an alert and the subsequent sequence of events as
illustrated in the following results. The live nature of the pilot
meant that complete information was not always available.
Downloaded from ascelibrary.org by University of Sheffield on 03/15/19. Copyright ASCE. For personal use only; all rights reserved.

Fig. 3. Online AI system GUI

Results
Example events for the automated, online AI system are now
provided and summary results are then presented from the two
month rigorous evaluation period 共January 26–March 29, 2008兲.
No flat-line alert was generated as the increase was below the high
alarm level 共30 L/s for this DMA兲. The control room received
contacts from customers, the first more than 29 h after the AI alert,
at 08:16 on February 27. WMS confirmed that a service pipe
repair was conducted on February 27. Note that the burst start

Example Events
Fig. 4 illustrates a situation in which the AI system correctly
produced an alert for a burst that the flat-line system did not
detect. The alert was received at 03:11 on February 26, with a size
estimate of just over 2 L/s. As can be seen from the graph, the
flow was abnormal in the 12 h window up to this time 共the period
of FIS analysis兲. The DMA has 3,057 domestic and 474 commer-
cial properties.

SCADA ID: 3193_02

ⴱ
ALERTⴱ %CONFIDENCE: 80.00 Fuzzy
output: 0.82 Dates: From 25-Feb-2008
13:45:00 To 26-Feb-2008 01:45:00
Burst size estimate: 2.1 L/s Fig. 4. Event 1—AI alert for burst 共February 22–29, 2008兲

314 / JOURNAL OF WATER RESOURCES PLANNING AND MANAGEMENT © ASCE / MAY/JUNE 2010

J. Water Resour. Plann. Manage., 2010, 136(3): 309-318

Downloaded from ascelibrary.org by University of Sheffield on 03/15/19. Copyright ASCE. For personal use only; all rights reserved.

Fig. 5. Event 2—alerts, contact, and repair for burst 共November 19–
December 8, 2007兲

time, indicated on Fig. 4, is an estimate based on retrospective

visual inspection.
The next example shows an AI alert for a smaller leak which
precedes a customer contact but for which no WMS information
was available. No flat-line alert was generated as the increase was
below the high alarm level. The DMA has 96 domestic and 7
commercial properties.

SCADA ID: 3278_02

ⴱ
ALERTⴱ %CONFIDENCE: 85.00 Fuzzy
output: 0.763 Dates: 23-Nov-2007
23:00:00 24-Nov-2007 11:00:00 Burst
size estimate: 0.3 L/s

As can be seen in Fig. 5, the DMA had raised flow rates for nearly
2 weeks. There is no WMS record, so it is unclear what caused the
return to normal conditions. It is initially somewhat surprising that
the customer contact appears to be as the flow is returning to
normal rather than slightly prior to this as would be expected if a
Fig. 6. Event 3—AI alerts for industrial usage, cascaded meters
repair had been initiated following customer contact. It is
therefore feasible that an investigation and repair was initiated
following night line analysis and that the contact was triggered by
a presence and activity within the DMA. This example the information could be valuable for defining and checking
demonstrates how small leaks such as this, which are not customer compliance with contractual/billing arrangements for industrial
impacting, are often not identified by current systems in a timely users.
manner and become part of background leakage. However the
automated AI system accurately identifies such small leaks and
hence could redefine the ELL or the “sustainable level of leakage” SCADA ID: 3109_02
共House of Lords 2006兲. ⴱ
ALERTⴱ %CONFIDENCE: 99.7 Fuzzy
The final example illustrates how the system can detect output: 0.729 Dates: 06-Nov-2007
unusual industrial events. Postanalysis revealed that a large 17:15:00 07-Nov-2007 05:15:00 Burst
industrial user in a DMA was extracting more water than usual size estimate: 5.7 L/s
because they were having problems with their on-site borehole.
This resulted in three AI alerts: for the instrument monitoring the
Summary of All Alerts
industrial user, for DMA1 共containing the industrial user兲, and for
DMA2 共supply water to DMA1 in a cascade fashion, rather than A total of 59 alerts were produced by the AI system for the period
DMA1 being directly connected to the trunk main system兲. Fig. 6 of January 26–March 29 2008. One hundred forty four sites were
shows the time series plots for these three locations around the in the system but due to data quality and subsequent failure of
period of the event. Alerts of this type provide added value, data checks typically 90–100 were under analysis at any one time
previously unavailable knowledge and warning concerning in the two month period. Of the 59 alerts:
system activity of which water utilities would like to be aware. • Twenty six are leakage or suspected leakage: Eighteen have
For example, there are suspected incidences of illegal 3rd party been correlated with WMS 共16 unique bursts, 3 detected by
use 共e.g., tankering which has resulted in discoloration events兲 and the flat-line alarm—a given burst event can effect multiple

JOURNAL OF WATER RESOURCES PLANNING AND MANAGEMENT © ASCE / MAY/JUNE 2010 / 315

J. Water Resour. Plann. Manage., 2010, 136(3): 309-318


Fig. 7. Summary of alerts for trial period
flow meters depending on network configuration兲; and eight
Downloaded from ascelibrary.org by University of Sheffield on 03/15/19. Copyright ASCE. For personal use only; all rights reserved.
have been correlated to customer reports of bursts, and appar-
ent repairs on nightline.
• Five have been correlated with known industrial events 共1 de-
tected by the flat-line system兲;
• Nineteen are “abnormal”—large unusual demands or short-
term increases in nightline 共0 on flat line兲; and
• Nine unknown/ghosts 共generally result of spurious data兲.
This data are shown graphically in Fig. 7. The AI system de-
tected many more valid events 共confirmed as above兲 than the
flat-line system. The flat-line system generated many more ghosts.
While complete details on flat-line alerts were not available, for
the 5 events picked up by both systems the AI system had an
average 1 h 45 min advantage in detection time, even with the an
imposed delay due to the data route.
The abnormal classification covers situations where the AI
system produced an alert and subsequent manual data investiga-
tion confirmed that an event of some type did occur, but for which
there was no correlation with any further information from the
utility company. Examples include large unusual demands 共un-
known industrial activity, for example兲, and those related to
known network maintenance or unexplained but significant short-
term increases in nightline. The signature of some of these abnor-
mal events such as large industrial demands, or closure and
opening of valves, could not be differentiated from bursts in the
current analysis system. Unexpected or unlicensed water use such
as the filling of private fire tanks, increased industrial use for new
processes, unauthorized filling of street cleaning equipment and
bowsers or illegal connections can all generate abnormal high
flow and hence alerts from the AI system. These are all activities
of which water service providers should be aware of. For ex-
ample, the AI system detected an unauthorized hydrant opening at
around 3 L/s in a rural DMA, which was located and closed
within a matter of hours. Unexpected events can also be produced
by operational activity such as network rezoning or changes to
valve arrangements or pump schedules. Their detection by this
system provides an additional check for the company of the tim-
ing, magnitude, etc., of such activities, particularly important
when such activities are frequently outsourced to third party con-
tractors.
Yorkshire Water has won the U.K. “Utility Company of the
Year” award an unprecedented 3 years in a row and the latest
figures from OFWAT show that the company has met or out per-
formed agreed leakage targets for 10 years running and overall
leakage has been cut by more than 45%. Despite this, the com-
pany is committed to reducing leakage even further and improv-
ing customer service. A major step in achieving this was the
introduction of GPRS technology for communication of hydraulic
performance data and the associated flat-line alarm system. This
demonstrably brought about a step change in the business aware-
316 / JOURNAL OF WATER RESOURCES PLANNING AND MANAGEMENT © ASCE / MAY/JUNE 2010
J. Water Resour. Plann. Manage., 2010, 136(3): 309-318
ness and response to system events. The work presented in this
report demonstrates how a further step change in awareness of
system performance and hence response can be achieved through
the application of self-learning, automatic data analysis routines.
The system developed provides “sensitive” detection of abnormal
flow events as they occur and due to the FIS system provides a
confidence estimate, in the form of a percentage, of how unusual
the flow event is together with an estimate of the burst flow rate
that can be very effectively use to prioritize events and response.
Discussion
Data from online instruments potentially provides a wealth of
information about what is happening in a water distribution net-
work. The AI system presented in this paper adopts a “bottom up”
approach to analyzing data as this is most appropriate to the dirty
nature of hydraulic time series data. ANNs are particularly suited
to dealing with missing or spurious sensor readings and exhibit
graceful degradation in such situations 共i.e., their performance
will only slowly decrease as data quality deteriorates兲. The ap-
proach mirrors the bottom level of the data information knowl-
edge wisdom pyramid used in knowledge management 共Ackoff
1989兲. The AI system converts flow sensor data into usable infor-
mation 共abnormal flow兲. However, more intelligent algorithms are
needed to turn information into knowledge for decision makers.
For example, some form of knowledge base is needed to interpret
a number of alerts for set of related DMAs 共which may include
flow and pressure兲 and deliver a system wide analysis of events.
Future work will explore the development of this type of ap-
proach, especially with the expected increase in density of cover-
age of pressure monitoring locations which will enable more
accurate location of incidents by means of a “grid” of pressures.
In the longer term, there exists the possibility of combining hy-
draulic data with other signals 共for example, water quality兲 to
allow the preemptive management of events and thus reduce their
impact and increase levels of service of the utility company.
It should be noted that the AI system did not always detect
catastrophic failures that were repaired very quickly 共the com-
pany reports that on average the lag time, from awareness to
repair, for these is 1.5 h in the day and 3.5 at night兲. The AI
system does not detect such rapidly repaired incidents because
they do not have time to impact on the FIS analysis window.
However, in this situation the flat-line alarm system and the AI
system can be seen as complementary because the flat-line alarms
would quickly identify the higher magnitude bursts. In such a
combined scenario the settings of the flat-line alarms could be
“relaxed” reducing the number of false alarms they currently pro-
duce. In combination, the two systems would identify sudden
large events and slowly developing or long lasting events thereby
covering the majority of possible scenarios.
Conclusions
There are a range of applications for machine learning where
simple point prediction is not adequate to guide decision making,
and a classification 共for event detection兲 is not provided. The
developed hybrid ANN and FIS that utilizes a MDN architecture
overcomes this problem for DMA flow data analysis. The AI sys-
tem was modified and extended from an offline analysis tool to an
automated online application which performs data-driven self-
learning 共with no need for classification examples兲 and updating
 共retraining兲. Significant modifications were made including being
able to deal with multiple data sites; implementing strategies for
coping with poor quality or spurious data; adding functionality for
alarm handling; and the temporal synchronization and handling of
ANN training and testing. The system was used to monitor and
analyze flow time series data from DMA flow meters in a live
water distribution network. The aim was to detect flow anomalies,
especially burst events, and to assess the potential operational and
customer service benefits to be gained from its application by a
water company 共by control room staff aggregating relevant data
sources a posterior兲.
A trial was conducted in which the system analyzed continu-
ally updated 共every hour兲 data originating from 144 DMAs, pro-
Downloaded from ascelibrary.org by University of Sheffield on 03/15/19. Copyright ASCE. For personal use only; all rights reserved.
viding timely alerts via automated e-mails. These alerts have been
used by the network controllers and compared off line with flat-
line alarms, repair information, customer contacts, and on the
ground information. The results of the trial demonstrate that the
system is able to detect abnormal flow events in a timely and
effective manner. Major findings include:
• Enables early detection and fast awareness 共Fig. 1兲 with the
potential to initiate response to events before customers are
impacted;
• Provides a confidence estimate, in the form of a percentage, of
the abnormality of the flow event and an estimate of the burst
size;
• Low number of ghosts versus genuine event detections 共de-
pending on confidence threshold selected兲;
• Ability to detect medium to small events which a flat-line
system cannot, i.e., more sensitive to changes in the system;
• Technique proven in a live environment of a real system; and
• Forty four percent of alerts were correlated to definite bursts,
using WMS and customer contact information, with the ma-
jority of other alerts likely to be unknown leakage, high ab-
normal demands 共32%兲 or other notable network or industrial
events 共9%兲.
It is concluded that the system is an effective and viable pro-
active tool for online burst detection in water distribution systems
with the potential to providing significant advantage over current
practice.
Acknowledgments
The writers wish to acknowledge the support given for this re-
search by Yorkshire Water Services Ltd., U.K., and the time, en-
ergy and enthusiasm of the specific individuals involved.
References
Ackoff, R. L. 共1989兲. “From data to wisdom.” Journal of Applied Systems
Analysis, 16, 3–9.
Anmala, J., Zhang, B., and Govindaraju, R. 共2000兲. “Comparison of
ANNs and empirical approaches for predicting watershed runoff.” J.
Water Resour. Plann. Manage., 126共3兲, 156–166.
Bishop, C. M. 共1994兲. “Mixture density networks.” Technical Rep. No.
NCRG/94/004, Dept. of Computer Science and Applied Mathematics,
Aston Univ., Birmingham, U.K.
Bishop, C. M. 共1995兲. Neural networks for pattern recognition, Oxford
University Press, New York.
Bowden, G. J., Dandy, G. C., and Maier, H. R. 共2003兲. “Data transfor-
mation for neural network models in water resources applications.” J.
Hydroinform., 5, 245–258.
Bowden, G. J., H. R. Maier, and G. C. Dandy 共2002兲. “Optimal division
JOURNAL OF WATER RESOURCES PLANNING AND MANAGEMENT © ASCE / MAY/JUNE 2010 / 317
J. Water Resour. Plann. Manage., 2010, 136(3): 309-318
of data for neural network models in water resources applications.”
Water Resources Research, 38共2兲, 2.1–2.11.
Chang, N., and Makkeasorn, A. 共2006兲. “Water demand analysis in urban
region by neural network models.” Proc., 8th Water Distribution Sys-
tem Analysis Symp., USEPA/Univ. of Cincinnati, Cincinnati.
Greenaway, G., Guanlao, R., Bayda, N., and Zhang, Q. 共2006兲. “Water
distribution systems demand forecasting with pattern recognition.”
Proc., 8th Water Distribution System Analysis Symp., USEPA/Univ. of
Cincinnati, Cincinnati.
Hornik, K., Maxwell, S., and Halbert, W. 共1989兲. “Multilayer feedfor-
ward networks are universal approximators.” Neural Networks, 2,
359–366.
House of Lords. 共2006兲. “Water management.” HL Rep. No. 191-I, Vol. 1,
House of Lords Science and Technology Committee, London.
Jain, S. K., Das, A., and Srivastava, D. K. 共1999兲. “Application of ANN
for reservoir inflow prediction and operation.” J. Water Resour. Plann.
Manage., 125共5兲, 263–271.
Kendall, M., and Ord, J. K. 共1990兲. Time series, 3rd Ed., Hodder and
Stoughton Limited, Ky.
Maier, H. R., and Dandy, G. C. 共2000兲. “Neural networks for the predic-
tion and forecasting of water resources variables: A review of model-
ling issues and applications.” Environ. Modell. Software, 15, 101–
124.
Mamlook, R., and Al-Jayyousi, O. 共2003兲. Fuzzy sets analysis for leak
detection in infrastructure systems: A proposed methodology, Clean
Technology Environmental Policy, Vol. 6, No. 1, Springer Berlin,
Heidelberg, 26–31.
McKenzie, R. and Seago, C. 共2005兲. “Assessment of real losses in po-
table water distribution systems: Some recent developments.” Water
Supply, 5共1兲, 33–40.
Misiunas, D., Vítkovský, J., Olsson, G., Lambert, M., and Simpson, A.
共2006兲. “Failure monitoring in water distribution networks.” Water
Sci. Technol., 53共4–5兲, 503–511.
Misiunas, D., Vítkovský, J., Olsson, G., Simpson, A. and Lambert, M.
共2005兲. “Pipeline burst detection and location using a continuous
monitoring of transients.” J. Water Resour. Plann. Manage., 131共4兲,
316–325.
Mounce, S. R. 共2005兲. “A hybrid neural network fuzzy rule-based system
applied to leak detection in water pipeline distribution networks.”
Ph.D. thesis, Univ. of Bradford.
Mounce, S. R., Boxall, J. B., and Machell, J. 共2007兲. “An artificial neural
network/fuzzy logic system for DMA flow meter data analysis pro-
viding burst identification and size estimation.” Water management
challenges in global change, B. Ulanicki et al., eds., Taylor and Fran-
cis, London, 313–320.
Mounce, S. R., Day, A. J., Wood, A. S., Khan, A., Widdop, P. D., and
Machell, J. 共2002兲. “A neural network approach to burst detection.”
Water Sci. Technol., 45共4–5兲, 237–246.
Mounce, S. R., Khan, A., Wood, A. S., Day, A. J., Widdop, P. D., and
Machell, J. 共2003兲. “Sensor-fusion of hydraulic data for burst detec-
tion and location in a treated water distribution system.” Inf. Fusion,
4共3兲, 217–229.
Mounce, S. R., Machell, J., and Boxall, J. 共2006兲. “Development of arti-
ficial intelligence systems for analysis of water supply system data.”
Proc., 8th Water Distribution System Analysis Symp., USEPA/Univ. of
Cincinnati, Cincinnati.
Nabney, I. 共2001兲. NETLAB: Algorithms for pattern recognition,
Springer, New York.
Office of Water Services 共OFWAT兲. 共2006兲. 具www.ofwat.gov.uk典 共Sept.
28, 2008兲.
Shinozuka, M., and Liang, J. 共2005兲. “Use of SCADA for damage detec-
tion of water delivery systems.” J. Eng. Mech., 131共3兲, 225–230.
Stoianov, I., Graham, N., Madden, S., Odenwald, T., and Stein, M.
共2007兲. “WaterSense: Integrating sensor nets with enterprise decision
support.” Water management challenges in global change, B. Ulan-
icki, et al., eds., Taylor and Francis, London, 123–126.
UK Water Industry Research 共UKWIR兲. 共2006兲. “Integrated network
 management roadmap.” Rep. No. 07/WM/18/4, UK Water Industry Vandaele, W. 共1983兲. Applied time series and Box-Jenkins models, Aca-
Research Ltd, London. demic, San Diego.
United Nations Environment Programme 共UNEP兲. 共2007兲. “Fourth global Water Research Centre 共WRc兲. 共1994兲. “Managing leakage.” Rep. A,
environment outlook: Environment for development assessment re- U.K. Water Industry Research Ltd./WRc, Wiltshire, England.
port.” United Nations Environment Programme, Progress Press Ltd,
Malta.
Downloaded from ascelibrary.org by University of Sheffield on 03/15/19. Copyright ASCE. For personal use only; all rights reserved.

318 / JOURNAL OF WATER RESOURCES PLANNING AND MANAGEMENT © ASCE / MAY/JUNE 2010

View publication stats J. Water Resour. Plann. Manage., 2010, 136(3): 309-318