You are on page 1of 14

Available online at www.sciencedirect.

com

Journal of Cleaner Production 16 (2008) 517e530


www.elsevier.com/locate/jclepro

Environmental performance, indicators and measurement uncertainty


in EMS context: a case study
Eleonora Perotto a,*, Roberto Canziani b, Renzo Marchesi c, Paola Butelli d
a

Technical University of Milan, CQA, DIIAR, P.za Leonardo da Vinci 32, 20133 Milan, Italy
b
Technical University of Milan, DIIAR, P.za Leonardo da Vinci 32, 20133 Milan, Italy
c
Technical University of Milan, CQA, Department of Energetics, P.za Leonardo da Vinci 32, 20133 Milan, Italy
d
Technical University of Milan, TTO, P.za Leonardo da Vinci 32, 20133 Milan, Italy
Received 2 November 2006; accepted 20 January 2007
Available online 23 March 2007

Abstract
EMS is a tool for managing the interaction between the organization and the environment. The aim of an EMS is to improve the overall
environmental performance of the organization. The performance should be monitored through measurements, and managed by indicators. Indicators are variables that summarize or otherwise simplify relevant information about the state of a complex system. A correct evaluation of
environmental performance arises from the choice of adequate raw data and from the relationships among raw data.
This paper, after a short excursus concerning the rule of indicators in environmental performance evaluation and the mean of uncertainty,
proposes an approach to the study and the evaluation, through indicators and indices, of the environmental aspect wastewater discharges
of a Local Authority who is involved in EMS implementation. Particularly, the critical analysis of one of the indices that has been used is
reported. The role of the uncertainty of measurements has been stressed. The results showed that measurement uncertainty is essential for
an efficient data comparison and for a correct evaluation of environmental performance, which, in turn, is essential to guarantee the effectiveness
of the EMS application.
2007 Elsevier Ltd. All rights reserved.
Keywords: Environmental Management System; Environmental performance; Indicators; ISO 14001; Uncertainty of measurements

1. Introduction
An Environmental Management System (EMS) is a part of
an organizations management system (including all human,
economical and infrastructural assets) which aims to manage
the environmental aspects related to its activities, products
and services. Its main and ultimate scope is to improve the
environmental performance of the organizations. The new
ISO 14001:2004 standard [1] defines the environmental
performance as measurable results of an organizations
management of its environmental aspects. To help the

* Corresponding author. Tel.: 39 02 23996433/6522; fax: 39 02 23996499.


E-mail address: eleonora.perotto@polimi.it (E. Perotto).
0959-6526/$ - see front matter 2007 Elsevier Ltd. All rights reserved.
doi:10.1016/j.jclepro.2007.01.004

organizations in the process of performance measurement, International Standard Organization (ISO) has developed the
specific standard ISO 14031:00 [2]. Indicators are the main tools
of this standard, and are defined as the specific expression that
provide information about an organizations environmental
performance. Their main scope is to make measurement of
the environmental performance easier for organizations.
Unfortunately, the measurement of environmental performance remains one of the greatest difficulties for the organizations and for the certification/competent bodies [3e6]. In
particular, a factor which is often neglected is the uncertainty
of measurements related to the indicators and indices. On the
contrary, the uncertainty that affects raw data is a crucial issue,
since an indicator can yield a reliable picture of the environmental aspects or performance only if it is based on good-quality
data [7].

E. Perotto et al. / Journal of Cleaner Production 16 (2008) 517e530

518

2. Environmental Management System


Environmental Management System (EMS) is a problemidentification and problem-solving tool, based on the concept
of continual improvement, that can be implemented in an
organization in many different ways, depending on the sector
of activity and the needs perceived by management [8]. In particular, standards for EMS have been developed by the International Organisation for Standardisation (ISO 14001) and
by the European Commission e Eco-Management and Audit
Scheme (EMAS Regulation) [9].
The standard ISO 14001:04 (and likewise the EMAS Regulation 761/2001) defines the EMS a part of an organizations
management system used to develop and implement its environmental policy and manage its environmental aspects,
where the environmental aspects are element of an organizations activities or products or services that can interact with the
environment. So, an Environmental Management System is
a method of incorporating environmental care throughout the
corporate structure. It is a useful tool to improve compliance
with legislation, address stakeholder pressure, improve corporate image and raise awareness of environmental issues within
the organization. Most EMSs are built on the Plan, Do,
Check, Act model. So, an EMS is a continual cycle of planning, implementing, reviewing and improving the processes
and actions that an organization undertakes to meet its environmental obligation [10] and to permit the continuous improvement of the global environmental performance. Following the
model, the organization that will correctly have applied the
principles of the standard will come to be in a higher step of
the virtual spiral in comparison to the preceding cycle. The
standard ISO 14001 (and likewise the EMAS Regulation) developments the path according to the phases shown in Table 1
( process approach). For more details see Refs. [1,9,11e13].
3. Environmental performance
Many organizations are seeking ways to understand, demonstrate and improve their environmental performances. An
Table 1
EMS process approach ex ISO 14001:04
Continual
Environmental
improvement policy

Overall intentions and direction of an


organization related to its environmental
performance as formally expressed by top
management
Planning
Establish the objectives and processes
necessary to deliver results in accordance
with the organizations environmental
policy
Implementation implement the processes
and operation
Checking
Monitor and measure processes against
environmental policy, objectives, targets,
legal and other requirements, and report
the results
Management
Take actions to continually improve
review
performance of the Environmental
Management System

organization with an EMS should assess its environmental performance against its environmental policy, objectives, targets
and other environmental performance criteria. In fact, an
EMS gives an organized and coherent scheme to properly
deal with environmental issues in organizations, with the
main purpose to improve their environmental performances.
Many authors refer environmental performance and problems related to its evaluation (e.g. [14e19]). Therefore, in
1999, ISO published the standard ISO 14031 that gives guidance on the design and use of environmental performance
evaluation within an organization. However, this issue is still
matter for discussion, as reported in many papers [4e6,20,21].
Several definitions exist for the expression environmental
performance, e.g.:
 measurable results of an organizations management of its
environmental aspects (results can be measured against the
organizations environmental policy, environmental objectives, environmental targets and other environmental performance requirements [1]); and
 results of an organizations management of its environmental aspects (results may be measured against the organizations environmental policy, objectives and targets)
[2,9].
In any case, in order to evaluate the environmental performance it is necessary to assess the environmental aspects
(element of an organizations activities or products or services
that can interact with the environment [1]). Changes to the environment, either adverse or beneficial, resulting wholly or
partially from environmental aspects, are defined as environmental impacts. The relationship between environmental
aspects and impacts is one of cause and effect.
Besides, a significant environmental aspect is an environmental aspect that has or can have a significant environmental
impact [1]. The organization shall ensure that the significant
environmental aspects are taken into account in establishing,
implementing and maintaining its EMS: identifying significant
environmental aspects and associated impacts is necessary in
order to determine whether and where control or improvement
is needed and to set priorities for management action. In particular, it is necessary to define some significance criteria, that
shall be comprehensive, suitable for independent checking, reproducible and verifiable, in order to identify the significant
environmental aspects of the organizations activities, products
and services. The role of the significant aspects and the related
problems are illustrated in the ISO guidelines (ISO 14004:04
[22]) and those of EMAS Regulation (Recommendation 680/
2001 [23] and Recommendation 532/2003 [24]) and by
some authors (e.g. [3,25e30]).
Usually, to assist organizations in the management of their
environmental significant aspects and impacts it is necessary
to use a tool such as an environmental indicator. The rule of
environmental indicators in the environmental performance
evaluation is essential for many authors (e.g. [16,31e35]). Indicators will support organizations in quantifying and reporting their environmental performances: in fact, it is necessary

E. Perotto et al. / Journal of Cleaner Production 16 (2008) 517e530

to associate one or more indicators to each environmental aspect. Particularly, indicators allow to classify and summarize
data concerning environmental aspects, returning an immediate and representative picture of the company situation in relation to its environmental situation, comparable with the
territorial context of the organization and with the objectives
that have been stated. Indicators should address those environmental impacts that are most significant and which the company can directly influence by its operations, management,
activities, products and services. They should also be sensitive
enough to reflect significant changes in environmental impacts. These values are essential, because they represent the
term of reference for all future environmental performance
evaluations.
4. Indicators and indices
4.1. Indicators, environmental indicators and
environmental performance indicators
The definitions of indicators are particularly confusing [36].
Some specific definitions of indicator in the literature are:
measure of system behaviour in terms of meaningful and perceptible attributes [37]; measure that summarizes information relevant to particular phenomenon, or to reasonable
proxy for such to measure [38]; parameter, or value derived
from parameters, which points to/provides information about/
describes the phenomenon/environment/area with significance
extending beyond that directly associated with parameter
(property that is measured or observed) value [39]; and variable that describes the system, where to variable is an operational representation of an attribute (quality, characteristic,
property) of the system and it represents our image of an attribute defined in terms of the specific measurement or observation procedures [40].
In general, indicators should be able to [41]: (i) assess conditions and trends; (ii) compare across places and situations;
(iii) assess conditions and trends in relation to goals and targets; (iv) provide early warning information; and (v) anticipate
future conditions and trends. So, it is possible to say that desirable indicators are variables that summarize or otherwise
simplify relevant information, make phenomena of interest
visible or perceptible to the managing staff, and are able to
quantify, measure, and communicate relevant information.
Some of those properties are not universal requisites (e.g.
qualitative indicators may be used in some situations), but
a matter of convenience [41]. Most definitions of environmental indicators rule out the possibility of qualitative indicators,
by restricting the concept to numerical variables, either explicitly or implicitly [32,36,37,39,42]. Indeed, it is maintained that
one of the essential functions of indicators is to quantify an
item. Qualitative indicators may be preferred to quantitative
indicators in at least three cases: (i) when quantitative information is not available; (ii) when the attribute of interest is inherently non-quantifiable; and (iii) when cost is a crucial issue,
overwhelming all other considerations. In some cases, qualitative assessments can be translated into quantitative notation.

519

Only the more general requirements or desirable properties


are listed below [41]:
1. the values of the indicators must be measurable (or at least
observable);
2. data must be either already available or they should be
obtainable (through special measuring or monitoring
activities);
3. the methodology for data gathering, data processing, and
construction of indicators must be clear, transparent and
standardized;
4. means for building and monitoring indicators should be
available;
5. the indicators or sets of indicators should be cost effective,
an issue often overlooked;
6. political acceptability at the appropriate level (local, national, and international) must be fostered (indicators
that are not acceptable by decision-makers are unlikely
to influence decisions); and
7. participation of, and support by, the public in the use of indicators is highly desirable, as one element of the general
requirement of participation of the broader society in the
quest for sustainable development.
As for the environmental indicators many definitions have
been proposed. For EPAs Report on the Environment
(2003), an environmental indicator is a numerical value derived from actual measurements of a pressure, state or ambient
condition, exposure or human health or ecological condition
over a specified geographic domain, whose trends over time
represent or draw attention to underlying trends in the condition of the environment [43]. The Asian Development Bank
(1999) tells that the environmental indicators represent an efficient way of measuring the environment issues in a country:
potentially, indicators can signal the health of the environment
and can help in formulating actions to serve the long-term
needs of the environment and the community [44]. The
UNEP (2004) defines environmental indicators as a way to
improve the delivery of information for decision-making [8],
while OECD (2004) defines them as an essential tools for
tracking environmental progress, supporting policy evaluation
and informing the public [45].
Regarding the performance indicators, they can represent
a finite set of quantities chosen to reflect certain aspects in an
organization. One possible definition of this type of indicators
is a number, absolute or relative, that facilitates management,
communication and follow-up of an organisations performance [46]. Bartolomeo (1995) defines environmental performance indicators as the quantitative and qualitative
information that allow the evaluation, from an environmental
point of view, of company effectiveness and efficiency in the
consumption of resources [47]. The ISO 14031 [2] defines
the environmental performance indicators as specific expression that provide information about an organizations environmental performance and their scope is to make measurement
of the environmental performance easier for organizations. According to ISO 14031 indicators should be divided as follows.

E. Perotto et al. / Journal of Cleaner Production 16 (2008) 517e530

520

1. Environmental Performance Indicators (EPI): specific expression that provides information about an organizations
environmental performance, which are divided into:
 Management Performance Indicators (MPI): that provide information about the management efforts to influence an organizations environmental performance;
and
 Operational Performance Indicators (OPI): environmental performance indicator that provides information about the environmental performance of an
organizations operations;
2. Environmental Condition Indicators (ECI): specific expression that provides information about the local, regional, national or global conditions of the environment.
The ECls provide information about the condition of the environment. This information can help an organization to better
understand the actual impact or potential impact of its environmental aspects, and thus assist in the planning and implementation of EMS.
It is generally difficult to choose the suitable performance
indicators, as well as to define their suitable number which
can describe thoroughly what one wants to know. Also,
when different specific needs for information are to be fulfilled, then different indicators should be chosen, or built.
In EMS, the classification of indicators recommended by
standard ISO 14031 and the examples of indicators provided
should be considered first, even though they are not complete
or comprehensive [7,21,48,49]. Particularly, an organization
should make a list of indicators by following the general
guidelines of the ISO 14031 standard and linking them to
the corresponding environmental aspects. Table 2 shows an example of a table that can be filled to make it easier the connection between an environmental aspect and the corresponding
indicators.
4.2. Indices
The distinction between indices and indicators is not clear
yet. Regarding this issue, there are two different opinions.
Many authors [32,50,51] put indices on a higher level of aggregation than indicators (Information Pyramid ). Other authors [41] report that indices and indicators differ because of
different complexities of the function by which they are obtained, not because of their hierarchical level. In most cases,

indicators are variables representing complicated functions


of the primary data, while indices are simple functions of
lower level variables (sometimes called subindices). For the
Asian Development Bank [44], an index combines a number
of variables into a single value. The ability of an index to provide information at a level that encompasses information on
a number of variables in the form of a single value makes
the concept of an index attractive for a number of functions.
An environmental index is necessary:
(i) to reflect the state of an environmental resource in order
to understand the dynamics of an environmental system
or the relationship between different environmental
components;
(ii) to facilitate the analysis of trade-offs between objectives (development and environmental protection,
etc.); and
(iii) to assist in making resource allocations and policy
decisions.
Once indicators have been chosen, they can be aggregated
into indices, which can return very concise and readily understandable information. This, in turn, can be used to compare
the evolution of a situation over time, but also to compare different situations. Typically, aggregation involves indicators
which refer to the same area (e.g. economical and environmental) or compartments (in the case of environment: air, water,
soil, and noise). Aggregation of data and indicators and the
process of weighing is a very critical point. Weighs should
be proportional to the importance assigned to each single indicator. It is evident that subjectivity plays a very important role
in assigning weights. Therefore it is crucial that the selection
of weights should be highly transparent and weighs should be
defined according to measurable criteria. During the process of
elaborating raw data to produce indicators and indices, information may be partly lost, since it is transformed into
a more concise form; however, it gains in ease of communication as it should be understood even by non-experts [53]. As
for the issue of aggregation Wall and others [54] note
that: .the development of highly aggregated indicators is
confronted with the dilemma that, although a high level of aggregation is necessary in order to intensify the awareness of
problems, the existence of desegregated values is essential in
order to draw conclusion for possible courses of action. This
dilemma particularly affects highly aggregated approaches

Table 2
Example of a table for linking activities (products or services) and their environmental aspects to the corresponding indicators
Activity/product/service

Combustion

Environmental aspect

Condition

Indicators
OPI

MPI

ECI

No. of calls for maintenance


per year
.
.

NOX concentration
in air
.
.

Emission into atmosphere

N, A, E

Emission NOX/unit product

.
.

.
.

.
.

N normal operating conditions, A abnormal conditions, E emergency situations and accidents.


OPI Operational Performance Indicators; MPI Management Performance Indicators, ECI: Environmental Condition Indicators.

E. Perotto et al. / Journal of Cleaner Production 16 (2008) 517e530

which do not have a disaggregated substructure. Moreover,


highly aggregated systems still have substantial conceptual
problems. In approaches that envisage an aggregation of individual elements, it constitutes a methodological barrier. Distance-to-target methods only appear to bypass the problems
of valuation; in addition, they are often dependent on the existence of target values..
A big attention should be given to the choice of the relationships between data: aggregation should be made according
to the methods which should be as objective as possible, and
the procedure should be repeatable. Particularly, for the creation of the point scale and for the choice of the indicators
weights, it is essential to be able to guarantee objectivity and
transparency. For this reason it is important to consider [7]:
-

data of the organization at a definite time;


literature values;
values which refer to similar cases; and
statistical analysis techniques.

Another important issue is related to the last point: the assurance of primary-data quality, paying particular attention to the
choice and to the metrological quality of the raw data [44,52,53].
5. Uncertainty of measurement
When you can measure what you are speaking about, and
express it in numbers, you know something about it; but
when you cannot express it in numbers, your knowledge is
of a meagre and unsatisfactory kind. It may be the beginning of knowledge, but you have scarcely, in your thoughts,
advanced to the stage of science
(Lord Kelvin)
The knowledge of the uncertainty is very important because
it implies increased confidence in the validity of the result of
a measurement.
Uncertainty, as defined in Guide to the Expression of Uncertainty in Measurement (GUM e [55]) and in VIM (International
Vocabulary of Basic and General Terms in Metrology e [56]), is
a parameter, associated with the result of a measurement, that
characterizes the dispersion of the values that could reasonably
be attributed to the measurand The parameter may be, for
example, a standard deviation,1 or the width of a confidence
interval; while the measurand is a particular quantity subjected
to measurement that is a set of operations having the object of
determining a value of a quantity [56].
The science of measurement, embracing both experimental
and theoretical determinations at any level of uncertainty in
any field of science and technology, is the metrology (International Bureau of Weights and Measures e IBPM e [57]).

1
The standard deviation of the mean X of n values taken from a population
p
is given by: SX S= n.

521

There are two type of uncertainty: standard uncertainty and


expanded uncertainty.
The standard uncertainty, however, evaluated, is represented by an estimated standard deviation, termed standard uncertainty with suggested symbol ui, and equal to the positive
square root of the estimated variance. The ISO Guide [55] refers that the uncertainty can be evaluated by Type A o Type B
evaluation:
Type A: This uncertainty component is represented by a statistically estimated standard deviation si, equal to the
positive square root of the statistically estimated variance S2i , and the associated number of degrees of
freedom vi. For such a component the standard uncertainty is ui si.
Type B: This uncertainty component is represented by a quantity uj, which may be considered an approximation to
the corresponding standard deviation; it is equal to
the positive square root of u2j , which may be considered an approximation to the corresponding variance
and which is obtained from an assumed probability
distribution based on all the available information.
Since the quantity u2j is treated like a variance and
uj like a standard deviation, for such a component
the standard uncertainty is simply uj.
The expanded uncertainty is a quantity defining an interval
about the result of a measurement that may be expected to encompass a large fraction of the distribution of values that could
reasonably be attributed to the measurand.
It should be born in mind that:
(i) the fraction of the distribution of values that could reasonably be attributed to the measurand may be viewed
as the coverage probability or level of confidence of the
interval; and
(ii) one must define which is the probability distribution of
measurement results, when a specific level of confidence
has to be associated to the interval defined by the expanded
uncertainty and its combined standard uncertainty; the
level of confidence that may be attributed to this interval
can be known only to the extent to which the assumption
on the probability distribution may be justified.
The coverage factor is a numerical factor that is used as
a multiplier of the combined standard uncertainty in order to
obtain an expanded uncertainty.
If the purpose of the uncertainty statement is to provide
coverage with a high level of confidence, an expanded uncertainty is computed as:
U ku

where k is chosen to be the a/2 critical value from the t-table


(Students t-distribution) for v degrees of freedom. For large
degrees of freedom, it is suggested to use k 2 to approximate
95% coverage.

E. Perotto et al. / Journal of Cleaner Production 16 (2008) 517e530

522

The expanded uncertainty defined above is assumed to provide a high level of coverage for the unknown true value of the
measurement of interest, so that for any measurement result, Y:
Y  U  True value  Y U
For further information see Refs [55,56,58e60].
The evaluation of measurement uncertainty is essential for
the metrological quality of the results. Without such evaluation
the results of the measurements cannot be compared neither
between themselves nor with literature or standard values.
Another important application of the assessment of the
uncertainty is for benchmarking. Benchmarking should be at
the base of EMSs. In fact, it is not only an important tool for
boosting improvements, as it allows to compare and rank
organizations performances, but it also goes beyond the
establishment of benchmarks, standards and norms, since it
investigates the practices that support the benchmark itself
(e.g. [61e64]).
Sometimes, the word uncertainty is interchanged with
accuracy, but they are actually different. In fact, the Accuracy of measurement is the closeness of the agreement between
the result of a measurement and a true value of the measurand and it refers only to systematic error [56]. Therefore,
the word accuracy should not be used for quantitatively describing the characteristics of measuring instruments or other
entities. Even ignoring this point, the term accuracy is a partial contribute of the metrological term uncertainty, which
refers to both systematic and random errors.
6. Case study: the role of uncertainty of measurement in
environmental performance evaluation of municipal
wastewater discharges
Following the trend in the private sector, EMSs have been
introduced in several Local Authorities around the world

[65], at the national level [66] as well as the local government


level e municipal administrations [67,68].
In Italy, Local Authorities which most frequently apply for
EMSs are Municipalities (Sincert site [69]). Since an EMS
should ensure the management of the environmental aspects
of the organization that are under its direct control and influence, then an EMS for a Local Authority can help in approaching the goal of sustainable development of the entire territory.
Therefore, for a Municipality the Environmental Analysis
(EA) is often very important since it should cover not only
the environmental aspects related to its specific activities,
products and services, but also the environmental aspects affecting the territory. For this purpose, it is crucial to define indicators and indices which are able to effectively depict both
the environmental conditions of the territory and the relevant
environmental aspects which can exert negative effects on
the environment.
This paper proposes the use of indicators and indices for
the study and the evaluation of a specific environmental aspect
(wastewater discharges) of a Local Authority which is involved
in EMS implementation. The results of the Environmental
Analysis showed that wastewater discharges negatively
affected water quality of the receiving water body, and that
this aspect was significant. Therefore, it was decided to report
the environmental performance evaluation of wastewater
discharges.
A list of some possible indicators that have been used during the initial step of EMS implementation (Environmental
Analysis e EA) is reported. Indicators and indices have
been chosen by considering the importance of monitoring performances in an EMS context. Also, it has been considered
that sustainability management tools like benchmarking [63]
are strongly recommended in the Aalborg Charter (1994)
[70] for the exchange of experience and best practices among
local authorities. This issue has been considered as crucial,
since local authority actions play an essential role in

Table 3
Indicators that have been used to describe (i) the environmental quality of the environmental compartment hydrosphere and (ii) how it is managed by an
organization [76]
Indicator
Hydrosphere

Water Management
Watershed area (km2)
Main water bodies (No., km2)
Mean flow rate in main rivers (m3 s1)
Artificial water basins and capacity of reservoirs (No., km2, m3)
Water supply sources and abstractions (L s1)
River banks preservation areas (No., % of watershed area, km2)
Indicator of congruity of preservation areas with the Provincial Master Plan (%)
No. of works carried out on artificial reservoirs on a yearly basis
No. of abstractions and flow-rate diversions (L s1)
Environmental quality
Extended Biotic Index (EBI)
Ecological State of the Water Body (according to Italian law n. 152/06, Part 3, Addendum 1)
Ammonium nitrogen concentration (mg L1)
Dissolved Oxygen (mg L1)
E. coli (UFC in 100 mL)
Conformity to regulations for bathing and swimming (%)
No. of complaints per year

ECI

MPI

E. Perotto et al. / Journal of Cleaner Production 16 (2008) 517e530

improving the state of the environment, not only as a general


policy issue, but also for planning water quality objectives and
providing high-quality services to the public.
Finally, the critical analysis of one of the indices that have
been adopted is presented. The role of the uncertainty of measurements has been stressed, because in EMS context it is essential to assess the uncertainty of the raw data and correctly
interpret the information given by indices.
6.1. Indicators for wastewater discharge as an
environmental aspect
Wastewater discharge is an environmental aspect which
exerts its ultimate impact on the receiving water body, as
wastewater discharges can heavily affect the quality of natural
waters. In order to properly understand this aspect, it is necessary to assess which activities/products/services are involved
in wastewater production and treatment and to evaluate the environment where this aspect takes place.
Here below, some examples of indicators that are used to
assess this aspect are reported, according to the ISO 14031
classification. The choice of indicators reflects some literature
guidelines: [71e73]. Particularly, Table 3 shows some possible
indicators for the assessment of the hydrosphere, that is, the
environmental compartment where the impact wastewater
discharge takes place. In this case:
- ECI are Environmental Condition Indicators which give
a picture of the quality and the state of the hydrosphere;
and
- MPI describe how hydrosphere is managed by the local
authority, municipality in primis.
Some of the proposed indicators are the result of cartographical digital map elaborations based on Geographical
Information Systems (GIS). As an example, the Indicator
of congruity of preservation areas with the Provincial Master
Plan is expressed in terms of percent actually protected

523

land along the main water bodies compared with the requirements reported on the Master Plan.
Many indicators, instead, have been obtained by actual
measurements, such as the concentration of specific elements
and pollutants in the water body (e.g.: dissolved oxygen and
ammonium nitrogen concentrations). Actually, most indicators
that are commonly used for the characterization of this environmental aspect have been obtained through direct measurements. In Table 4 some examples of performance indicators
referred to the activity management of a sewer system and
the related wastewater treatment plant are reported. They
are classified according to ISO 14031 as follows:
- operational performance indicators (OPI) give information
about the efficiency of the sewer system and the wastewater treatment plant with particular care to the related environmental impacts;
- Management Performance Indicators (MPI) give information about management practices of the sewer system and
the wastewater treatment plant.
By applying and evaluating the proposed indicators to the
specific case of a Municipality, it appeared very clearly that
the compartment water presented some critical elements,
mainly related to the environmental aspect wastewater
discharges.
Particularly, the indicators have shown the existence of
a microbiological contamination of the water body, which
was most probably linked to the discharge of the wastewater
treatment plant. In fact:
- indicator Concentration of Escherichia coli in river
(ECI e CFU/100 mL) (Fig. 1) shows that microbiological
contamination of river is growing between 1999 and 2000:
the limit stated by law (700) is largely overcome (1300).
Escherichia coli (usually abbreviated to E. coli), in fact,
is one of the main species of bacteria that live in the lower
intestines of warm-blooded animals, including mammals,

Table 4
Examples of indicators that can be considered for the environmental aspect defined as wastewater discharges for the activity/service defined as management of
sewers and wastewater treatment plant [76]
Environmental aspect: wastewater discharges
Activity

Indicator

Management of sewers and


wastewater treatment plant
(WWTP)

Per capita wastewater discharge into sewers (L per capita and day)
COD concentration at inlet/outlet from WWTP (mg L1)
BOD5 concentration at inlet/outlet from WWTP (mg L1)
SST concentration at inlet/outlet from WWTP (mg L1)
Nitric/nitrous nitrogen concentration at inlet/outlet from WWTP (mg L1)
Ammonium nitrogen concentration at inlet/outlet from WWTP (mg L1)
Phosphorus concentration at inlet/outlet from WWTP (mg L1)
Mass of biosolids that are disposed off yearly (t per year)
Ratio of population equivalent served to total population (%)
Compliance to limits at sampling points (%)
Ratio of No. of actions to No. of sampling (yearly base, %)
Beaches where bathing was declared forbidden by ordinance (No. per year)
Maintenance actions (No. per year)
Complaints concerning malfunctions or nuisance originated by the sewers/WWTP (No. per year)

OPI

MPI

E. Perotto et al. / Journal of Cleaner Production 16 (2008) 517e530

524

Escherichia Coli

1600
1400

ufc/100 ml

1200

1999
2000
2001

1000
800

Law limit

600
400
200
0

Municipalities
Fig. 1. Indicator E. coli for the Municipality considered (shown by the solid arrow).

and its presence in water and groundwater is a common


indicator of fecal contamination.
- indicator Conformity to regulations for bathing and
swimming (MPI e expressed as percentages) that considers the results of conformity for the bathing and swimming, confirmed that microbiological contamination in the
receiving water body was real: in fact, the average of
the last 5 years was as low as 50% and this was due to
the fact that the conformity (ex DPR 470/82 e Italian
law) is not met for three parameters, all of them indicators
of fecal contamination, that are: Fecal Coliform bacteria,
Total Coliform bacteria and Streptococcus fecalis. The
presence of those bacteria in aquatic environments shows
that the water has been contaminated by fecal material.

6.2. Definition of SWTI (State of Wastewater Treatment


Index) and its application as a possible descriptor of the
environmental aspect wastewater discharges
After finding that the environmental aspect wastewater
discharges was the main cause of the bad microbiological
water quality of the river, it was devised to develop an index
for monitoring the state of wastewater treatment, which could
trace the trend of the performance of the municipal wastewater
treatment plant over time, and, consequently to allow its better
management. This index was named State of Wastewater
Treatment Index (SWTI) and has been applied to the Municipality considered.
6.2.1. The index SWTI
The index has been developed aiming at a better understanding of the performance of a wastewater treatment plant
by all stakeholders. In fact, it is important that the information
given by this indicator is understandable not only to technicians but also to decision-makers, so that they can easily
take actions to deal with the problem. Particularly, the scope
of such indicator should be to make more evident whether
sewerage and wastewater treatment are appropriate.
Directive 91/271/EEC [71], as amended by Directive 98/15/
EC [72], states that Member States shall ensure that, by 31

December 2005, urban waste water entering collecting systems


shall before discharge be subject to appropriate treatment in
the following cases:
 for discharges to fresh-water and estuaries from agglomerations of less than 2,000 P.E. (population equivalent),
 for discharges to coastal waters from agglomerations of
less than 10,000 P.E.,
whereas for all discharges from agglomerations of more than
2000 P.E., Member States shall ensure that urban waste water
entering collecting systems shall before discharge be subject to
secondary treatment or an equivalent treatment.
Furthermore, the European legislation defines an appropriate treatment as any process and/or system of disposal
which enables the waters receiving the discharges to meet
the specified quality objectives and to comply with the relevant
provisions of Directive 91/271/EC and any other Community
directives.
A specific Italian regulation states that appropriate
treatment is any simple biological treatment such as simplified activated sludge processes (as extended aeration), trickling filters, rotating biological contactors, lagooning and
phytodepuration systems, provided that they allow to meet
the required effluent quality standards stated by the competent Authority. Therefore, a definition of inappropriate
treatment should also be given. Following the European
legislation, inappropriate treatment means a treatment
which does not allow to meet the required discharge limits.
For example, inappropriate WWTPs can be all those who
have been sanctioned by the competent Authority. However,
by doing so, a rough approximation is made, as it happens
that final effluent limits are exceeded not because the plant
is not adequately designed or built, but because of improper
disposal practices (e.g. discharge of toxic substances into
sewers instead of proper disposal as special liquid wastes).
In this case, the discharge is inappropriate, not the plant.
However, the management staff of the WWTP should trace
the source of the improper discharge and take technical
and/or administrative actions to avoid other similar
occurrences.

E. Perotto et al. / Journal of Cleaner Production 16 (2008) 517e530

525

Table 5
Population equivalent (P.E.) in five Municipalities belonging on the same
watershed

Table 6
Values of the State of Treatment Index(SWTI) referring to data reported in
Table 5

Municipality

Sewered
P.E.

P.E. served
by WWTP

P.E. served by
inappropriate
WWTP

Municipality

P.E. served by
WWTP/Sewered
P.E. (%)

SWTI (%)

A
B
C
D
E (implementing an EMS)

9073
452
1355
997
3764

8478
452
186
997
3205

227
0
0
686
1100

P.E. served by
inappropriate
WWTP/P.E.
served by
WWTP (%)

A
B
C
D
E (implementing an EMS)

93.4
100.0
13.7
100.0
85.1

2.7
0.0
0.0
68.8
34.3

90.8
100.0
13.7
31.2
50.8

The SWTI index is defined as [73]:



Tot P:E: served by WWTP
SWTI
Tot P:E: served by sewers



P:E: served by inappropriate WWTP
100
Tot P:E: served by WWTP

SWTI is calculated as the difference between the ratio of


the total population equivalent (P.E.) served by Waste Water
Treatment Plants (WWTPs) to total P.E. served by the sewer
system and the ratio of P.E. which are inadequately served
by a WWTP to total P.E. served by WWTP (the P.E. represents
the unit of measure used to describe the size of a waste water
discharge).
If SWTI scores high and water quality remains poor, then
causes other than inadequate municipal sewerage systems and
wastewater treatment should be investigated (e.g. diffuse pollution such as surface runoff, poor cattle-breeding techniques).
Moreover, this index expresses the combined overall efficacy of the sewer system and the WWTP service and can be
used to compare the situation of each Municipality with others
in the same watershed. This is extremely important because
the contamination of the water body could not only be caused
by the Municipality under investigation (implementing an
EMS; from now on, named E), but also from Municipalities
which are located in the same watershed. Therefore, the geographic scale should be wider than municipality level in order
to identify and plan the most appropriate actions, and also to
determine the role of each administrative body, from the single
municipality level (low) to the regional level (high). For

instance, mapping the index on a watershed map over time


may be very interesting for tracing the progress of implementation of Plans for Preservation of Waters Resources.
Population equivalent and the other data that are used to
calculate SWTI are reported in Table 5. All data have been
found on the Provincial Report on Surface Water Quality.
The calculated values of SWTI for Municipality E and for
neighbouring ones (that have been named AeD) are shown
in Table 6 and Fig. 2.
Municipalities AeD have been considered as neighbouring as they belong to the same watershed. The neighbouring
municipalities have the same environmental, economic and social characteristics of Municipality under investigation, but are
not implementing an EMS.
As one can see, the index SWTI has immediately evidenced
(to non-experts, too) which Municipalities could be responsible for the poor quality of the receiving water body. As a matter
of fact, SWTI has allowed the classification of treatment practices in five Municipalities, and allowed to point out the
benchmark in that area, represented by Municipality B.
Second ranks Municipality A, while other Municipalities score
poor to very poor index values.
In particular, SWTI for Municipality E shows that wastewater collection and treatment is not comparable to the standards
of neighbouring Municipalities. Therefore, its sewerage/treatment services should be improved and specific actions should
be planned. However, the microbiological quality of the river
is not likely to improve until Municipalities C and D greatly
improve their performances.

Municipality E

Municipality D

Municipality C

Municipality B

Municipality A
0

10

20

30

40

50

60

70

80

90

100

110

Fig. 2. SWTI for Municipality E (implementing an EMS) and for other Municipalities in the same watershed.

E. Perotto et al. / Journal of Cleaner Production 16 (2008) 517e530

P:E:

3000
2500

Type

1
A
2
A
3
NA
4
NA
5
NA
6
NA
Total P.E. served by WWTPs
A appropriate; NA not appropriate.

BOD5 Load
(BL, kg d1)

P.E.

66.00
60.30
32.40
18.00
12.00
3.60

1100
1005
540
300
200
60
3764

2000
1500
1000
500
0

32 34 36 38 40 42 44 46 48 50 52 54 56 58 60 62 64 66 68

SWTI (%)
Variable: Municipality E, wc(BL) = 10%, PDF: Normal
Chi-square Test = 44,51302, gl = 8 (adgiust.), p = 0,0586
3000
2500
2000
1500
1000
500
0

5 10 15 20 25 30 35 40 45 50 55 60 65 70 75 80 85

SWTI (%)
Variable: Municipality E, wc(BL) = 20%, PDF: Normal
Chi-squareTest = 15,45378, gl = 8 (adgiust.), p = 0,0592
3000
2500

No. of obs

WWTP

Variable: Municipality E, wc(BL) = 5%, PDF: Normal


Chi-square Test= 16,81930, gl = 8 (adgiust.), p = 0,06205

6.2.3. Assessment of the uncertainty associated with


the Index
It was decided to assess the effect of uncertainty associated
with the biological measurement (BOD5) on SWTI numerical
values.
In this specific case, for the assessment of the uncertainty of
SWTI, we started from the definition of population equivalent
(P.E.):
Table 7
Characteristics of WWTPs which are present in Municipality E

BL
0:06

where BL total BOD5 load, expressed as kgBOD5 per day,


which is fed to the WWTP; 0.06 kg of BOD5 produced daily
by one person (kg per capita and day).

No. of obs

6.2.2. Critical observations about SWTI


The index cannot not directly provide any precise suggestion about which are the best actions to be adopted, since
the it provides an aggregated information. Only an analysis
of the disaggregated data (in this case: absence of plants or
the presence of inappropriate plants) can explain which is
the problem causing low index scores. One of the first actions
which can be devised as necessary may be in-depth studies
about the quality of the service. For example, it may be found
advisable to investigate which are the causes of WWTP inadequacy, by elaborating mass balances of pollutants.
Other aspects involved in the definition of SWTI should be
analysed more deeply, in order to prevent its improper use.
First of all we should define how reliable a measurement is.
Accuracy and precision are fundamental in order to calculate
reliable indicators and indices that give a correct description
of reality. Just take the definition of population equivalent
as it is in the European Directive 91/271/EC [71]. It is
a unit of measurement of biodegradable organic pollution representing a load of 60 g BOD5 produced on average per person
and day. The size of the agglomeration, expressed in P.E., corresponds to the organic load produced in the agglomeration
during an average day during the week of the year with maximum production. It is calculated from the sum of the organic
load produced by permanent and seasonal residential establishments and services and the organic load produced on the
same day by the industrial wastewater which must be collected
by a collecting system (European Commission, 2001 e [74]).
Measuring BOD5 with precision and accuracy is a challenge.
BOD5 is a highly variable measure related to biological activity, which is difficult to standardize, since it relies on bacterial
seeding (see official method for BOD5 [75]), which can vary
greatly for different wastewaters. SWTI is affected by the uncertainty of BOD5 measurements to evaluate the organic load
of a plant and the corresponding population equivalent and this
should be considered.

No. of obs

526

2000
1500
1000
500
0

10 15 20 25 30 35 40 45 50 55 60 65 70 75 80 85

SWTI (%)
Fig. 3. SWTI frequency distribution in Municipality E according to different relative standard combined uncertainties wc(BL) 5%, 10% and 20%.

E. Perotto et al. / Journal of Cleaner Production 16 (2008) 517e530

The BOD5 load (BL, kg per day), which is the startingpoint raw datum used to calculate P.E., is obtained from:
BL BQ

where B BOD5 concentration in wastewater (kg m3);


Q wastewater flow rate (m3 per day).
They both are experimental measurements.
In each Municipality the number of plants, the total BOD5
load and the corresponding P.E. have been considered. For
Municipality E, the corresponding relevant data are the following (details are reported in Table 7):
 3764 total sewered P.E.;
 3205 P.E. served by WWTPs (both appropriate and not appropriate); and
 1100 P.E. served by WWTPs which are not appropriate.
The study of uncertainty associated to STI has been
performed with the Monte Carlo Method (n 10,000; for
further information: [76e78]), by assuming the following
hypotheses:
 BL is normally distributed, as it usual as far as experimental measurements are concerning,
 values of 5%, 10% and 20% have been tested as relative
combined standard uncertainty associated to BL e i.e.
wc(BL); these values are rather common in this field as
supported by some references (inter alii: [79]).
Referring to Municipality E, results are shown in Fig. 3(Ae
C), where data from Monte Carlo simulation are represented
as histograms, overlayed with the best-fitting normal curve
(pdf, Chi-square test, p > 0.05). The three different pictures
are related to the value of relative combined standard uncertainty associated to BL (wc(BL)), respectively, equal to 5%,
10% and 20% for each case A, B, and C.
Finally, the expanded uncertainty (U) associated to SWTI
has been calculated according to the following expression:
U kw

527

where k is the coverage factor; w is the relative combined standard uncertainty.


The choice of the factor k is based on the level of confidence desired. For an approximate level of confidence of
95%, k is 2. The results are shown in Fig. 4.
As it can be clearly seen from Fig. 4, the classification of the
five Municipalities is not as certain as it was without considering
the uncertainty which affects the index, especially considering
case C, which is referred to a relative combined standard
uncertainty associated to [BL ewc(BL)] equal to 20%.
In fact, if the combined relative standard uncertainty
wc(BL) is equal to 5%, the classification based on the average
value as in Fig. 2 still holds. However, if wc(BL) is set to 20%,
the classification could not be taken as it appears: for instance,
it is not certain whether Municipality E is better than D, since
results can be mutually compatible.
Municipalities A and B are also in a very similar situation.
Finally, comparing Municipality B (optimum) with Municipality E (at wc(BL) 20%) it is clear that this is another case of
compatible measures.
Therefore, it appears that it is necessary that Municipality C
should plan and take actions to improve its wastewater treatment performances, while for Municipalities E and D it becomes difficult to establish any priority for intervention.
Finally, the use of an index as an instrument for planning
and/or benchmarking appears quite ineffective when uncertainty is high (third scenarios, wc(BL) 20%), while the
same index could be a good descriptor of reality if it is based
on good-quality raw data.
It can be concluded that an index should not be used without considering the uncertainty of the raw data used and the
uncertainty of the resulting index values.

7. Conclusions
In the case study described earlier, the number of served
population equivalent of the different Municipalities which
have been evaluated, can be highly affected by uncertainty
when based on BOD5 measurements, while it is often and

Municipality E
wc(BL) = 5%
Municipality D
wc(BL) = 10%
wc(BL) = 20%

Municipality C

Municipality B

Municipality A
0

10

20

30

40

50

60

70

80

90

100

110

STI U
Fig. 4. Representation of SWTI  U, where U is the expanded uncertainty that affects SWTI, according to three scenarios on the uncertainty wc(BL) which may
affect BOD5 measurements (5%, 10% or 20%).

528

E. Perotto et al. / Journal of Cleaner Production 16 (2008) 517e530

improperly considered as an absolute value unaffected by


uncertainty.
From data described before, Municipalities cannot be compared significantly through the evaluation of SWTI, since the
uncertainty of measurements is high. Any decision about taking or not remedial actions and about which remedial action is
appropriate becomes more and more susceptible of criticism
as uncertainty increases.
The importance of raw data in the field of Environmental
Management Systems is clear: environmental performance indicators can be strongly affected by uncertainty of raw data to
such an extent that results could be meaningless, or even misleading. Therefore raw data should be selected according to
the following rules:
(1) as for indicators: (i) choose the lowest possible number of
indicators which can adequately describe the situation under investigation; (ii) avoid redundant information;
(2) as for metrological traceability: (i) clearly specify reference conditions, analytical methods and proper calibration
of the instrumentation; (ii) assess the uncertainty of the
measurements.
It is essential to assess the uncertainty of the raw data in order to correctly interpret the information given by indicators
and indices in EMS context. In such a way, indices can be
very useful to decision makers provided that the uncertainty
related to index values is clearly stated and accounted for.
Therefore, in general, environmental data and information
can be comparable only if they are obtained by following strict
metrological specifications about both measuring methodology
and instrumentation. Also, the nature and characteristics of raw
data should be clearly considered when estimating their uncertainty. In other words, metrological quality of raw data should
always be considered as a basic requirement in order to use
raw data rationally for further elaborations in any area [7,80].

References
[1] International Standard Organisation. ISO 14001 e environmental management systems e requirements with guidance for use. International
Standard Organisation; 2004.
[2] International Standard Organisation. ISO 14031e environmental management e environmental performance evaluation e guidelines. International Standard Organisation; 1999.
[3] Johnston A, Hutchison J, Smith A. Significant environmental impact evaluation: a proposed methodology. Eco-Management and Auditing 2000;
7:186e95.
[4] Pojasek RB. How do you measure environmental performance? Environmental Quality Management 2001;79e88. Wiley Periodicals.
[5] Dias-Sardinha I, Reijnders L. Environmental performance evaluation and
sustainability performance evaluation of organizations: an evolutionary
framework. Eco-Management and Auditing 2001;8:71e9.
[6] Ammenberg J, Wik G, Hjelm O. Auditing external environmental auditors e investigating how ISO 14001 is interpreted and applied in reality.
Eco-Management and Auditing 2001;8(4):183e92.
[7] Butelli P, Marchesi R, Perotto E. Certificazione ambientale e riferibilita`
delle misure. In: Maggioli, editor. Atti dei seminari di Ricicla 2001,
Rimini Fiera, 26e29 settembre 2001; 2001. p. 126e33.

[8] UNEP/FIDIC/ICC United Nations environment programme e UNEP.


Environmental management system training resource kit, http://earth
watch.unep.net/about/docs/indicat.htm; 2001.
[9] European Community. Regulation (EC) No 761/2001 of the European Parliament and of the council of 19 March 2001 allowing voluntary participation by organisations in a community eco-management and audit scheme
(EMAS). Brussels: European Community; 2001 (OJEC, 24-04-2001,
L 114/1).
[10] Stapleton PJ, Glover MA, Davis SP. Environmental management systems: an implementation guide for small and medium-sized organizations. In: NSF International, editor. Glover-Stapleton Associates, Inc.,
http://www.nsf.org/NSF-ISR/; 2001.
[11] Whitelaw K. ISO 14001 environmental system handbook. Oxford: Butterworth-Heinemann; 1997.
[12] Environmental Protection Agency e US. EPA. Environmental management system manual. Version EM-01, Control No. ESC EM-01. The Environmental Science Center Board, http://www.epa.gov/; 2002.
[13] Caropreso G, Catto EG, Pernigotti D. La nuova UNI EN ISO 14001:
Guida pratica allo sviluppo e allapplicazione di un sistema di gestione
ambientale [UNI-Il Sole 24 Ore Ed]; 2006.
[14] Wehrmeyer W. Measuring environmental business performance: a comprehensive guide. In: Hillary, editor. Business and the environment practitioner series. Cheltenham; 1995. p. 1e110.
[15] Ditz D, Ranganathan J. Measuring up e toward a common framework
for tracking environmental performance. World Resources Institute;
1997.
[16] Ilinitch AY, Soderstrom NS, Thomas TE. Measuring corporate environmental performance. Journal of Accounting and Public Policy 1998;17:
383e408.
[17] Skillius A, Wennberg U. Continuity, credibility and comparability: key
challenges for corporate environmental performance measurement and
communication. IIIEE; 1998.
[18] Thoresen J. Environmental performance evaluation e a tool for industrial
improvement. Journal of Cleaner Production 1999;7:365e70.
[19] Young CW, Welford R. An environmental performance measurement
framework for business. In: Bennett M, James P, Klinkers L, editors. Sustainable measures: evaluation and reporting of environmental and social
performance. Sheffield: Greenleaf; 1999. p. 98e116.
[20] Bennett M, James P. ISO 14031 and the future of environmental performance evaluation. In: Bennett M, James P, Klinkers L, editors. Sustainable measures: evaluation and reporting of environmental and social
performance. Sheffield: Greenleaf; 1999. p. 76e97.
[21] OReilly M, Wathey D, Gelber M. ISO 14031: effective mechanism to
environmental performance evaluation. Corporate Environmental Strategy 2000;7(3):267e75.
[22] International Standard Organisation. ISO 14004. Environmental management systems e general guidelines on principles, systems and support
techniques. International Standard Organisation; 2004.
[23] European Community. Commission recommendation 2001/680/EC of 7
September 2001 on guidance for the implementation of Regulation
(EC) No 761/2001 of the European Parliament and of the Council allowing voluntary participation by organisations in a community ecomanagement and audit scheme (EMAS). European Community; 2001 [OJEC,
17-09-2001, L 247/1].
[24] European Community. Commission recommendation 2003/532/EC of 10
July 2003 on guidance for the implementation of regulation (EC) No.
761/2001 of the European Parliament and of the council allowing voluntary participation by organisations in a community eco-management and
audit scheme (EMAS) concerning the selection and use of environmental
performance indicators. European Community; 2003 [OJEC, 23-072003, L 184/19].
[25] Brown DJA. Evaluating environmental effects as part of an environmental management system. Clean Air 1995;25:108e15.
[26] Smith M. Methodologies for assessing environmental effects. ICHemE
Environmental Protection Bulletin 1996;41:12e22.
[27] Iraldo F. Valutare e gestire gli aspetti ambientali indiretti: un possibile approccio. Atti del seminario: Il nuovo Regolamento Emas: oltre i confini
aziendali. Milano: Universita` Bocconi; 2001.

E. Perotto et al. / Journal of Cleaner Production 16 (2008) 517e530


[28] Zobel T, Almroth C, Bresky J, Burman JO. Identification and assessment
of environmental aspects in an EMS context: an approach to a new reproducible method based on LCA methodology. Journal of Cleaner Production 2002;10:381e96.
[29] Zobel T, Burman JO. Factors of importance in identification and assessment of environmental aspects in an EMS context: experiences in Swedish organizations. Journal of Cleaner Production 2004;12:13e27.
[30] Perotto E. Una proposta metodologica per lidentificazione e la valutazione degli aspetti ambientali in ambito SGA. In: CIPA, editor. IA Ingegneria Ambientale; 2006. p. 245e55. no. 5.
[31] Azzone G, Manzini R. Measuring strategic environmental performance.
Business Strategy and the Environment 1994;3(1):1e14.
[32] Hammond A, Adriaanse A, Rodenburg E, Bryant D, Woodward R. Environmental indicators: a systematic approach to measuring and reporting
on environmental policy performance in the context of sustainable development. Washington: WR Institute; 1995.
[33] Azzone G, Manzini R, Noci G, Welford R, Young CW. Defining environmental performance indicators: an integrated framework. Business Strategy and the Environment 1996;5(2):69e80.
[34] Eagan P, Joeres E. Development of a facility-based environmental performance indicators related to sustainable development. Journal of Cleaner
Production 1997;5(4):269e78.
[35] Olsthoorn X, Tyteca D, Wehrmeyer W, Wagner M. Environmental indicators for business: a review of the literature and standardisation
methods. Journal of Cleaner Production 2001;9:453e63.
[36] Bakkes JA, Van den Born GJ, Helder JC, Swart RJ, Hope CW,
Parker JDE. An overview of environmental indicators: state of the art
and perspectives. UNEP/EATR.94-01; RIVM/402001001. In: Environmental assessment sub-programme. Nairobi: UNEP; 1994.
[37] Holling CS. Adaptive environmental assessment and management. John
Wiley & Sons; 1978. England.
[38] McQueen D, Noak H. Health promotion indicators: current status, issues
and problems. Health Promotion 1988;3:117e25.
[39] Organisation for Economic Co-operation and Development e OECD.
Core set of indicators for environmental performance reviews. A synthesis report by the group on the state of the environment, http://www.
oecd.org/; 1993.
[40] Walz R. Development of environmental indicators system: experience
from Germany. Environmental Management 2000;25(6):613e23.
[41] Gallopn GC. Indicators and their use: information for decision-making e
introduction. In: Sustainability indicators: report of the project of
indicators for sustainable development. Wiley/Scientific Committee on
Problems of the Environment (SCOPE); 1997.
[42] Adriaanse A. Environmental policy performance indicators. SDV Publishers; 1993.
[43] EPA. Draft report on the environment. Publication No. EPA 260-R-02006. EPA; 2003.
[44] Asian Development Bank. Development of environment statistics in developing asian and pacific countries. ADB; 1998.
[45] Organisation for Economic Co-operation and Development e OECD.
Key environmental indicators. Paris: OECD Environment Directorate,
http://www.oecd.org/; 2004.
[46] Ross H. Developing environmental performance indicators for business.
DNV; 1996.
[47] Bartolomeo M. Environmental performance indicators in industry. Milano: FEEM; 1995.
[48] Marsanich A. Environmental indicators in EMAS environmental statements. FEEM e ENEA 2000. Italy.
[49] Jasch C. Environmental performance evaluation and indicators. Journal
of Cleaner Production 2000;8:79e88.
[50] Sustainable Development of the scientific community on problems of
the environment e SCOPE. Environmental indicators: systematic approach to measuring and reporting on the environment in the context
of sustainable development. In: International workshop on indicators
of sustainable development for decision-making, 9e11 January, Ghent,
Belgium; 1995.
[51] World Resources Institute e WRI. Environmental indicators: a systematic
approach to measuring and reporting on environmental policy

[52]

[53]
[54]

[55]

[56]
[57]
[58]
[59]

[60]

[61]
[62]

[63]

[64]
[65]

[66]
[67]

[68]

[69]
[70]

[71]

[72]

[73]

[74]

[75]

529

performance in the context of sustainable development. Washington,


D.C.: WRI; 1995.
World Conservation Monitoring Centre e WCMC. In: Reynolds JH, editor. Information product design, series, vol. 3. Commonwealth Secretariat; 1998.
Perotto E, Marchesi R, Canziani R, Butelli P. Indici di valutazione per un
Sistema di Gestione Ambientale. Qualita` 2005;6:26e30.
Wall R, Ostertag K, Block N. Synopsis of selected indicator systems for
sustainable development. Report for the research project, Further development of indicator systems for reporting on the environment of the
Federal Ministry of the Environment. Karlsruhe: Fraunhofer Institute
for Systems and Innovation Research; 1995.
International Standard Organisation e ISO. Guide to the expression of
uncertainty of measurement. ISO, ISBN 92-67-10188-9; 1993. Paper.
p. 110.
International Standard Organisation e ISO. International vocabulary of
basic and general terms in metrology. 2nd ed. ISO; 1993. p. 59.
International Bureau of Weights and Measures e IBPM, http://www.
bipm.org/en/bipm/metrology/.
National Institute of Standards and Technologys e NIST. E-handbook of
statistical methods, <http://www.nist.gov/>; 2005.
Ellison SRL, Rosslein M, Williams A. Quantifying uncertainty in analytical measurement, Eurachem Citac Guide e QUAN. 2nd ed.. P1, Eurachem, Citac, 2000, http://www.measurementuncertainty.org/index.html;
2000.
Cox M, Harris P. Up a GUM tree? try the full Monte! Teddington,
Middlesex: Centre for Mathematics & Scientific Computing, National
Physical Laboratory (NPL). TW11 0LW, http://www.npl.co.uk/; 2003.
Andersen B, Pettersen PG. The benchmarking handbook: step-by-step instructions. London: Chapman & Hall; 1996.
Szekely F, Vollmann T, Ebbinghaus A. Environmental benchmarking.
Becoming green and competitive. In: Business and the environment e
practitioner series. Cheltenham: Stanley Thornes Ltd; 1996.
Bolli A, Emtairah T. Environmental benchmarking for local authorities:
from concept to practice. Environment issues report no. 20, http://themes.
eea.eu.int/indicators/all_indicators_box?sort_bydpsir; 2001.
Dattakumar R, Jagadeesh R. A review of literature on benchmarking.
Benchmarking: An International Journal 2003;10(3):176e209.
Noren H, von Malmoborg F. Are standardized EMSs useful in local authorities? A study of how a tool from the private sector in used in public
sector. Business Strategy and the Environment 2004;13:87e197.
Honkasalo A. Environmental management systems at the national level.
Eco-Management and Auditing 1999;6(4):170e3.
von Malmborg F. Environmental management systems: what is in it for
local authority? Journal of Environmental Policy and Planning 2003;
5(1):3e21.
Emilsson S, Hjelm O. Mapping environmental management system initiatives in Swedish local authorities e a national survey. Corporate Social
Responsibility and Environmental Management 2002;9(2):107e15.
SINCERT, <http://www.sincert.it/>.
Aalborg Charter. Charter of European cities & towns towards sustainability e as approved by the participants at the European conference on sustainable cities & towns in Aalborg. Denmark: Aalborg Charter; 1994.
European Council. Directive 91/271/CEE: council Directive of 21
May 1991 concerning urban waste water treatment [OJ L 135 of
30.05.1991].
European Commission. Directive 98/15/EC of 27 February 1998 amending Council Directive 91/271/EEC with respect to certain requirements
established in Annex I [OJ L 67 of 07.03.1998].
Perotto E. Metodologie e tecniche per i Sistemi di Gestione Ambientale
(Methodologies and techniques for Environmental Management Systems). Ph.D. Thesis, Politecnico di Milano; 2005 [in Italian].
European Commission. Implementation of Council Directive 91/271/EEC of
21 May 1991 concerning urban wastewater treatment, as amended by Commission Directive 98/15/EC of 27 February 1998 e situation at 31 December
1998 [European Commission web site]. European Commission; 2001.
APHA. Method 5210 B, 5-day BOD test, standard methods for the examination of water and wastewater. 20th ed. APHA; 1998.

530

E. Perotto et al. / Journal of Cleaner Production 16 (2008) 517e530

[76] Manly BFJ. Randomization and Monte Carlo methods in biology. In:
Statistical sciences series. Chapman & Hall Texts; 1998.
[77] Saltelli A, Chan KP, Scott M. Sensitivity analysis. In: Probability and
statistic series. John Wiley & Sons Publishers; 2000.
[78] Cox M, Harris P. The GUM and its planned supplemental guides.
Accreditation and Quality Assurance 2003;8:375e9.

[79] Butelli P, Menegozzi S. La valutazione dei carichi inquinanti addotti dai collettori fognari: caratterizzazione delle acque reflue ed incertezza di misura. In:
Atti del IV Congresso Metrologia & Qualita`, Torino; 2005. vol. 1. p. 48e52.
[80] Bottazzini N, Butelli P. Qualita` del dato analitico, Atti del 55 Corso di
Aggiornamento in Ingegneria Sanitaria, Il nuovo laboratorio per lambiente. Politecnico di Milano: DIIAR; 2001.

You might also like