Journal of Cleaner Production 9 (2001) 417–427

www.cleanerproduction.net

Using data envelopment analysis to evaluate environmentally

conscious waste treatment technology
a,* b
Joseph Sarkis , Jeffrey Weinrach
a
Graduate School of Management, Clark University, 950 Main Street, Worcester, MA 01610-1477, USA
b
JCS Novation Inc., 4501 Indian School Rd. NE, Suite 105, Albuquerque, NM 87110, USA

Received 8 February 2000; accepted 29 November 2000

Abstract

As environmental issues become increasingly important for organizational operations, the roles of organizational practices and
technology in improving environmental performance increase. Some of these new practices and technologies are difficult to evaluate
using traditional approaches. In this paper we analyze a decision making case study concerning the investment and adoption of
environmentally conscious waste treatment technology in a government supported agency. Data envelopment analysis, and some
of its extensions, will be used to evaluate the technologies. The results show that this tool could be useful in evaluating these
technologies. Managerial insights on the use of these models, including caveats on the use of DEA, are also presented.  2001
Elsevier Science Ltd. All rights reserved.

Keywords: Technology evaluation; Data envelopment analysis

1. Introduction the steel and iron industry [2], telecommunications [3]

Managing any organization requires the capability to
effectively measure and analyze information that is gen-
erated by business processes. Environmental manage-
ment of organizations and managing environmentally
focused business decisions from the perspective of the
natural environment is even more critical. These mana-
gerial practices may be reactive (e.g. responding to
environmental regulation) or proactive, (e.g. competitive
reasons and for the satisfaction of the stakeholders).
Developing and applying tools that have not been tra-
ditionally utilized for environmental management at stra-
tegic and operational levels could aid the managerial
decision making processes. One tool that has gained
wide acceptance at an industrial level of analysis (e.g.
organizational comparisons) is the use of data envelop-
ment analysis (DEA). For example, DEA has been used
to evaluate the relative performance of a broad range of
public and private organizations including banking [1],
and universities [4], to name a few. A thorough review
of DEA theory and application can be found in [5].
The use of DEA for evaluating intra-organizational
environmental decisions is the focus of this paper. An
evaluation of environmentally conscious waste treatment
technology for a governmentally supported institution is
the specific application introduced. The background of
the decision environment provides some of the issues
associated with the organizational environment and
DEA’s use. We have also identified practical issues asso-
ciated with DEA use that managers and researchers
should be aware of. These issues and a comparison of
DEA (and its variations) with the return on investment
(ROI) criterion, currently in use for evaluating the tech-
nologies, is also presented. A general description of how
DEA performs with respect to other multiple criteria
evaluation tools are discussed. The paper concludes with
a summary of the paper and findings along with
additional issues for practice and research consideration.

2. The decision environment

* Corresponding author. Tel.: +1-508-793-7659; fax: +1-508-793-
8822. The Department of Energy is comprised of production
E-mail address: jsarkis@clarku.edu (J. Sarkis). plants and laboratories whose primary missions have

0959-6526/01/$ - see front matter  2001 Elsevier Science Ltd. All rights reserved.
PII: S 0 9 5 9 - 6 5 2 6 ( 0 0 ) 0 0 0 8 4 - 6
418 J. Sarkis, J. Weinrach / Journal of Cleaner Production 9 (2001) 417–427
been to develop and maintain the nuclear weapons stock-
pile for the Department of Defense. These activities have
resulted in a forty-year legacy of radioactive and mixed
(a combination of radioactive and hazardous) waste
which must be managed. In order to prevent future waste
problems, a comprehensive pollution prevention pro-
gram has been adopted and is currently being
implemented at these sites.
As an integral part of any successful pollution preven-
tion program, goals must be set and goals must be met.
Since the Department of Energy consists of so many dif-
ferent types of facilities and operations, these goals
would naturally not be identical. However, there should
be (and needs to be) methods to compare the relative
success of meeting pollution prevention goals regardless
of what the goals are or what type of facility or operation
is being addressed. DEA is one of the methods that can
help address the problems that the Department of Energy
faces in realizing its pollution prevention goals.
The plutonium facility at Los Alamos National Lab-
oratory, known as TA-55, generates a significant portion
of the radioactive and mixed waste at Los Alamos. In
order to implement successful pollution prevention pro-
grams at TA-55, a variety of approaches are being taken.
The eight technologies and new processes that are being
used for this DEA study represent some of the process
modifications that TA-55 personnel are currently
developing and implementing. Each new process either
addresses a radioactive mixed solid waste stream or a
radioactive mixed aqueous waste stream (in a few
cases, both).
Here is a brief overview of the eight technologies
(processes) being evaluated in this study:
1) Nitric Acid Recovery and Recycle addresses the
need to recover the acidic aqueous waste stream that
is generated during the nitrate aqueous plutonium pro-
cessing stage.
2 and 3) Distillation (concentration) and Separation
of Actinides from Waste Salts addresses the need to
separate much of the plutonium from the bulk waste
salts that can be classified as transuranic waste which
would eventually be discarded at the Waste Isolation
Pilot Plant in Carlsbad, New Mexico. Plutonium sep-
aration would classify the resultant solid waste
streams as Low-Level waste (based primarily on
concentrations) and therefore could be discarded in a
much less costly low-level waste landfill. There are
two alternatives that have been identified for this tech-
nology; we have defined them as Distillation and Sep-
aration of Actinides from Waste Salts-1 and Distil-
lation and Separation of Actinides from Waste Salts-
2. The differences focus on the concentrations and
processing needed to alter the wastes to either the
transuranic waste level or the low level of concen-
trations. That is, Distillation and Separation of Actin-
ides from Waste Salts-1’s goal is to ultimately have
waste concentrations (outputs) at the transuranic lev-
els, while Distillation and Separation of Actinides
from Waste Salts-2’s goal is to have waste concen-
trations at the low level waste level.
4) Waste Acceptance for Nonradioactive Disposal is
a method to separate radioactive from non-radioactive
components of a solid waste stream using state-of-
the-art sensor technology. Much of the “radioactive”
waste that ends up in a low-level waste landfill is
actually not radioactive but “suspect” radioactive
which means that it cannot be definitively determined
that this waste is radioactive but since it came from
a site which historically has radioactive materials, it
is assumed to be radioactive. Waste Acceptance for
Nonradioactive Disposal allows personnel to separate
the radioactive components of waste from the non-
radioactive components and, therefore, greatly reduce
the volume (and the cost) of waste management.
5) Electrolytic Decontamination of Gloveboxes is an
improved technique to decontaminate gloveboxes
(metallic boxes that hold contaminated gloves and
other materials) from transuranic waste to low-level
waste that, if successful, would greatly reduce the vol-
ume of liquid and solid wastes generated during
decontamination.
6) Chloride Aqueous Waste Stream Polishing deals
with the other significant process waste stream at TA-
55 (the nitrate stream is the other). The polishing pro-
cess would make the neutralization of the chloride
solution and actinide removal much easier and less
costly through the use of new resins and adsorbents.
7) High-Gradient Magnetic Separation is a technique
to separate plutonium from other metals and solids
using a magnetic field. The paramagnetic plutonium
wastes can be separated from most of the other
materials that are diamagnetic. In a similar way to
Waste Acceptance for Nonradioactive Disposal, this
process would greatly reduce the amount of poten-
tially non-radioactive waste that ends up at a low-
level landfill.
8) Application of Freeze-Drying Technology to
Decontamination of Liquid Wastes uses freeze-drying
techniques to remove the acid and aqueous compo-
nents of the waste streams from the radioactive con-
taminants.
Even though these technologies may be quite different
in their approaches, they do have a few management
(environmental and general) characteristics in common:
앫 They all address significant waste issues and would,
if implemented, greatly reduce health risks and costs.
앫 They all require some technological advance; that is,
they could not be implemented through administrat-
ive controls.
 J. Sarkis, J. Weinrach / Journal of Cleaner Production 9 (2001) 417–427
앫 They all require some time to develop, test, and
eventually implement and, therefore, need to be
addressed in stages.
3. Data
The information used for this paper is used for
illustrative purposes and is adapted from actual reports
for the evaluation of these technologies. Additional
applications and more robust data may provide
additional insights.
To be able to use the data for the DEA evaluation
models, detailed in the next section, inputs and outputs
must be identified. The inputs are traditional cost meas-
ures that include operating and capital investment costs
(these are also used to calculate ROI values and are
initially represented as cost savings). Even though the
costs may be integrated into one value we felt that
allowing for their separation provided managerial
decision making flexibility (e.g. some management
decision making may focus more heavily on capital
investment costs than operational costs). The output data
will be estimates of wastes generated with the introduc-
tion of the technology. The wastes will include low level
and transuranic wastes measured in terms of drums. Fig.
1 shows the input and output relationships that will be
inputs for the DEA models.
There are eight technologies that are to be evaluated
in this model. For the basic DEA models, the number
technologies (or units under consideration) should be a
minimum of between 2 to 5 times the total number of
input and output factors. Thus, we have limited the total
number of inputs and outputs to four factors. The actual
values used in the data set for the problem we are
addressing will be described after discussion of the DEA
models that will be used in this paper.
4. Models and methodology
The analysis of technology evaluation will consist of
two major phases. We will use the standard ratio model
of DEA and a Ranking DEA model to determine the
Fig. 1. A graphical representation of the inputs and outputs and decision units (new technologies) for the DEA model.
419
initial efficiencies of the various systems under consider-
ation. The next phase will carry out a cross efficiency
analysis using “arbitrary”, or simple, and aggressive
cross efficiency measurement approaches. The cross-
efficiency and ranking efficiency scores will then be
compared to the return-on-investment (ROI) scores from
calculations provided by the Los Alamos Laboratories
[6].
5. DEA, cross efficiency, and ranking models
In this section we provide a review of basic DEA and
“cross-efficiency” extensions to the DEA models.
Productivity models have traditionally been used to
measure efficiency of systems. Typically, DEA pro-
ductivity models for a given “decision making unit” use
ratios based on the amount of output per given input. In
this paper a decision making unit will be a technology
under consideration within the organization. DEA allows
for the simultaneous analysis of multiple inputs to mul-
tiple outputs, a multi-factor productivity approach. The
general efficiency measure that is used by DEA can best
be summarized by Eq. (1).
冘 Osyvky
Eks⫽
y
(1)
冘x
Isxukx
where: (Eks) is the efficiency or productivity measure of
technology s, using the weights of “test” technology k;
(Osy) is the value of output y for technology s; (Isx) is
the value for input x of technology s; (vky) is the weight
assigned to technology k for output y; and (ukx) is the
weight assigned to technology k for input x.
For the traditional DEA ratio model developed by
Charnes et al. (CCR) [7] the objective is to maximize
the efficiency value of a test technology k, from among
a reference set of technologies s, by selecting the optimal
weights associated with the input and output measures.
The maximum efficiencies are constrained to 1. The for-
mulation is represented in Eq. (2).
420 J. Sarkis, J. Weinrach / Journal of Cleaner Production 9 (2001) 417–427

冘 Okyvky
ukx,vkyⱖ0 (4)

maximize Ekk⫽
y

冘
Eq. (4), which we will call the rank efficiency formu-
Ikxukx lation, allows for technically efficient scores to be greater
x than 1. This result, according to Andersen and Petersen,
subject to: will allow for a more discriminating set of scores for
technically efficient units.
Eksⱕ1 for all technologies s One of the difficulties with simple efficiency (the CCR
ukx,vkyⱖ0 (2) and rank efficiency models) is a resulting set of “false
positives.” A false positive is a unit that may be efficient
under CCR, but its cross-efficiency score (described
This non-linear programming Eq. (2) is equivalent to below) is less than a CCR inefficient unit. A false posi-
the following linear programming Eq. (3) (see [7]): tive technology typically has the characteristic of weigh-

冘
ing heavily on a single input or output, thus making it
maximize Ekk⫽ Okyvky more efficient than any other technology because of a
y dominant performance on a single factor. As mentioned,
subject to: a procedure for discriminating between “true” efficient
technologies and false positive technologies is by ana-
Eksⱕ1 for all technologies s lyzing the cross-efficiencies. Sexton, Silkman, and

冘
Ikxukx⫽1
x matrix provides information on the efficiency of a spe-
ukx,vkyⱖ0
The transformation is completed by constraining the
efficiency ratio denominator from Eq. (2) to a value of
1. This is represented by the constraint SIkxukx=1.
x averaged to get a mean cross efficiency measure (this
The result of Eq. (3) is an optimal “simple” or techni-
cal efficiency value (Ekk*) that is at most equal to 1. If
Ekk*=1, then it means that no other technology is more
efficient than technology k for its selected weights. That
is, Ekk*=1 has technology k on the optimal frontier and
is not dominated by any other technology. If Ekk*⬍1
then technology k does not lie on the optimal frontier
and there is at least one other technology that is more
efficient for the optimal set of weights determined by
Eq. (3). The Eq. (3) is executed s times, once for each
technology. The first method in the analysis will use the
CCR model to calculate the simple efficiency.
A variation of the CCR model proposed by Andersen
and Petersen [8], allows for the use of DEA efficiency
scores for ranking alternative units. In their model, they
simply eliminate the test unit from the constraint set. The
new formulation is represented by Eq. (4).
冘
maximize Ekk⫽ Okyvky
y minimize
subject to:
Hogan [9], introduced the concept of cross-efficiencies
and the cross efficiency matrix. The cross efficiency
cific technology with the optimal weighting schemes
(3)
determined for other technologies. Table 1 summarizes
a typical cross efficiency matrix. The kth row and the sth
column represent the efficiency measure of technology s
by the optimal weights for technology k (Eks.). Each of
the columns of the cross efficiency matrix may then be
calculation will be defined as the simple cross efficiency
measure when the optimal weights that are being used
are from the basic CCR model) for each technology (es).
A false positive technology is associated with an
efficient technology that has a relatively small cross
efficiency value or a cross efficiency value that is less
than the cross efficiency value of an initially inef-
ficient technology.
A pitfall in determining a cross efficiency score is that
the weights, derived from the CCR model, used to calcu-
late the optimal simple efficiencies (and eventually used
in cross efficiency measures) may not be unique. To
overcome this difficulty, a formulation developed by
Doyle and Green [10] — one that will help generate a
less ambiguous set of weights — may be used for cross
efficiency calculation and development of a cross
efficiency matrix. This formulation is represented by
Eq. (5).
冘冉 冘 冊
vky Osy ,
y s⫽k

Eksⱕ1 for all technologies s⫽k subject to:

冘x
Ikxukx⫽1 冘冉 冘 冊
x
ukx
s⫽k
Isx ⫽1,
 J. Sarkis, J. Weinrach / Journal of Cleaner Production 9 (2001) 417–427
Table 1
Generalized cross efficiency matrix
Mean cross efficiencies scores
Rating technology 1 2
1 E11 E12
2 E21 E22
3 E31 E32
: : : :
: : : :
n En1 En2
e1 e2
冘y
Okyvky⫺Ekk 冘 x
Ikxukx⫽0
Eksⱕ1 for all technology s⫽k
ukx,vkyⱖ0
The Doyle and Green formulation as presented in Eq.
(5) has a primary goal of obtaining a maximum simple
efficiency score for technology k (the test unit), and a
secondary goal of determining a set of weights that will
minimize the other technologies aggregate output. This
model has been defined as an “aggressive” formulation.
To make the Doyle and Green formulation a “benevol-
ent” formulation — one where the secondary goal is to
maximize the other technologies aggregate output —
requires changing the “minimum” to “maximum” in
Eq. (5).
Similar to the rank efficiency scores, the mean cross-
efficiency scores can be used to rank the technology
alternatives.
In summary the following DEA-based performance
measures will be calculated for each technology:
앫 Simple Efficiency — is the efficiency score calculated
by the basic CCR model for each technology.
앫 Ranked Efficiency — is the efficiency score calcu-
lated by the rank efficiency model for each tech-
nology.
앫 Simple Cross-Efficiency — is the efficiency score cal-
culated for a technology s by multiplying the optimal
weights of a technology k from the CCR model by
the original ratio of inputs and outputs of the tech-
nology s. The cross-efficiency scores are then aver-
aged for each technology to get a mean simple
cross-efficiency.
앫 Aggressive Cross-Efficiency — is the efficiency score
calculated for a technology s by multiplying the opti-
mal weights of a technology k from Doyle and
Green’s aggressive formulation to the original ratio
of inputs and outputs of the technology s. The cross
efficiency scores are then averaged for each tech-
421
Rated technology 3 ... n
E13 ... E1n
E23 ... E2n
E33 ... E3n
:
:
En3 ... Enn
e3 en
nology to get a mean aggressive cross-efficiency
value.
(5)
6. Data scaling and initial results
In the initial runs of the model we considered the dif-
ferential rates for the input and output data. That is, the
marginal impact of the technology on costs, transuranic
and low level wastes were used as the basic data points.
Table 2 shows the data as initially captured. The first
column identifies the technology. The second and third
columns, respectively, represent the capital and oper-
ational cost savings of the technology. Positive cost sav-
ings represent how much can actually be saved by a new
technology implementation. A negative value represents
an increase in costs. The final two columns represent the
amount of waste reduction in drums (a negative number
in these columns represents an increase in that waste
output).
DEA requires the data sets to be non-negative for the
outputs and strictly positive for the inputs. It is also
assumes that input values are improving as they get
smaller and output values are improving as they get
larger. Thus, this data needs to be scaled for two reasons.
The first reason is to eliminate any negative values in
the data set. The next is to make sure that data set has
the characteristic of having smaller values preferable for
inputs and larger values preferable for outputs. The
initial scaling required the cost categories (inputs) to be
altered such that lower values were preferable to higher
values. There was also a need to eliminate any negative
numbers in any of the columns. The outputs (last two
columns of Table 2) already have the increasing-value-
is-better characteristic.
The scaling was initially accomplished by subtracting
the values in the input columns from an arbitrarily large
number (10,000 in this case). The output columns were
scaled by adding a value of 10 to the transuranic output
factor and a value of 20 to the low level waste output
factor. This scaling allowed each of the inputs and out-
422 J. Sarkis, J. Weinrach / Journal of Cleaner Production 9 (2001) 417–427

Table 2
Unscaled data set of results for input and output factors after introduction of new technology

Technology Inputs Outputs

Capital Cost Saving Operational Cost Savings Transuranic Waste Low Level Waste
(000’s) (000’s) Decrease (Drums) Decrease (Drums)

Nitric acid recovery 240 128 2 0

Distill and separate actinides/salts 1 ⫺325 1006 0 59
Distill and separate actinides/salts 2 ⫺1400 1017 27 26
Waste accept. for non-rad. disp. ⫺363 500 0 75
Decon. gloveboxes ⫺55 256 77 66
Chloride waste stream polishing 4450 ⫺246 12 ⫺14
Magnetic separation 6520 79 ⫺3 ⫺14
Freeze drying technology 3900 341 20 23

puts to have the characteristics to run the DEA model.
These values are shown in Table 3. An initial execution
of the data using the CCR model of DEA provided us
the results as shown in the final column of Table 3. To
determine if there was any sensitivity from the arbitrary
additive scaling procedure used, we evaluated another
set of scaling values. This additional test scaling
included subtracting each element in the capital cost
input and operational costs input from values of 7000
and 1500, respectively. Values of 4 and 15 were added
to the transuranic and low level waste columns, respect-
ively. The new results are shown in Table 4. The
efficiency scores shown in Table 4 were significantly dif-
ferent than those in Table 3. For example, the Nitric
Acid Recovery and Recycle technology decreased in
efficiency when going from the first scaled data set to
the second set. Other technologies (e.g. Distillation and
Separation of Actinides from Waste Salts-1) increased
in efficiency when going from the first scaled data set
to the second set.
This initial finding represents an important caveat for
scaling of data using arbitrary absolute values. Two steps
were taken to make sure that data scaling errors did not
play a role in the final results of this analysis. First,
instead of using arbitrary numbers, actual values were
used for the initial base line values. This first step helped
eliminate any negative values from the data set. The
actual baselines were 7500, 1500, 280, and 1400 for
capital costs, operational costs, TRU and LLW respect-
ively. The values within each column were subtracted
from their respective baseline. These results provided
appropriate characteristics for the inputs, but the outputs
(TRU and LLW) still needed to have larger values as
more preferable. These columns were scaled by taking
the inverse of their actual values, thus maintaining the
integrity of the relationships of the data. The final data
set that will be used for the analysis are shown in
Table 5.
This section on scaling of data for the DEA models
is meant as an instructional point for those researchers
and practitioners who wish to apply these models. We
shall now discuss the results of the various models using
the data set in Table 5.

Table 3
Scaled data set 1. Data is arbitrarily scaled by subtracting cost savings from a value of 10,000 for capital and operational costs, and adding a value
of 20 to the transuranic and low level waste values

Technology Capital costs Operational costs Transuranic waste Low level waste Simple technical
efficiency (CCR)

Nitric acid recovery 9760 9872 12 20 0.232

Distill and separate 10325 8994 10 79 0.881
actinides/salts 1
Distill and separate 11400 8983 37 46 0.564
actinides/salts 2
Waste accept. for non- 10363 9500 10 95 1.000
rad. disp.
Decon. gloveboxes 10055 9744 87 86 1.000
Chloride waste stream 5550 10246 22 6 0.458
polishing
Magnetic separation 3480 9921 7 6 0.232
Freeze drying technology 6100 9659 30 43 0.805
 J. Sarkis, J. Weinrach / Journal of Cleaner Production 9 (2001) 417–427 423

Table 4
Scaled data set 2. Data is arbitrarily scaled by subtracting cost savings from a value of 7000 for capital costs and a value of 1500 from operational
costs, and adding a value of 4 to the original transuranic waste value and adding a value of 15 to the original low level waste values

Technology Capital costs Operational costs Transuranic waste Low level waste Simple technical
efficiency (CCR)

Nitric acid recovery 6760 1372 6 15 0.184

Distill and separate
7325 494 4 74 1.000
actinides/salts 1
Distill and separate
8400 483 31 41 1.000
actinides/salts 2
Waste accept. for non-
7363 1000 4 90 1.000
rad. disp.
Decon. gloveboxes 7055 1244 81 81 1.000
Chloride waste stream
2550 1746 16 1 0.546
polishing
Magnetic separation 480 1421 1 1 0.181
Freeze drying
3100 1159 24 38 1.000
technology

Table 5
Scaled data set 3. Data is scaled by actual absolute values and ratio scaled

Technology Capital costs Operational costs Transuranic waste Low level waste

Nitric acid recovery 7260 4372 3.597 0.7143

Distill and separate actinides/salts 1 7825 3494 3.571 0.7457
Distill and separate actinides/salts 2 8900 3483 3.953 0.7278
Waste accept. for non-rad. disp. 7863 4000 3.571 0.7547
Decon. gloveboxes 7555 4244 4.926 0.7496
Chloride waste stream polishing 3050 4746 3.731 0.7072
Magnetic separation 980 4421 3.534 0.7072
Freeze drying technology 3600 4159 3.846 0.7262

7. DEA models results and discussion
The efficiency score results of the various DEA mod-
els and runs are shown in Table 6. The first column of
results represent the simple technical efficiency scores
from the basic CCR DEA model. In this column we see
that the determination of which technology performs
best on the selected input and output factors is difficult
to determine. This difficulty arises from the number of
technically efficient technology (those with a score of
1.000). In this case five of the technologies had
efficiency scores of 1. Alone, the data in this first column
may not be useful. These results can be useful if
additional decision models are to be used, where this

Table 6
Efficiency score results of DEA model executions

Technology Simple tech. eff. Simple X-eff Aggressive X-eff Rank efficiencies ROI

Nitric acid recovery 0.833 0.807 0.698 0.833 128%

Distill and separate 1.000 0.956 0.846 1.054 299%
actinides/salts 1
Distill and separate 1.000 0.949 0.859 1.053 63%
actinides/salts 2
Waste accept. for non-rad. 0.915 0.872 0.762 0.915 133%
disp.
Decon. gloveboxes 1.000 0.970 0.856 1.118 455%
Chloride waste stream 0.887 0.877 0.739 0.887 205%
polishing
Magnetic separation 1.000 0.989 0.890 3.112 1960%
Freeze drying technology 1.000 0.993 0.837 1.016 166%
424 J. Sarkis, J. Weinrach / Journal of Cleaner Production 9 (2001) 417–427
simple technical efficiency data is used to filter out inef-
ficient technology (see [11], who applied multi-attribute
utility theory to the robot selection process after using
the basic DEA formulation to filter out inefficient units).
The second column of results shows the simple mean
cross-efficiency scores. These values were determined
by using the optimal weights from the basic CCR model.
There are no clear “false positive” candidates in these
measures. That is, those units that were initially techni-
cally efficient still performed better then those that were
technically inefficient, when the cross efficiency scores
in the second column are used to evaluate the tech-
nology. The Application of Freeze-Drying Technology
to Decontamination of Liquid Wastes (0.993) and High-
Gradient Magnetic Separation (0.989) technology alter-
natives had the highest simple mean cross-efficiency
scores. When compared to the ROI (in the last column
of Table 6), we see that 166% ROI for the highest ranked
alternative (Application of Freeze-Drying Technology to
Decontamination of Liquid Wastes) is only the fifth best
ROI. The relationships with ROI and the other
approaches will be further detailed below.
The next column in Table 6 is the aggressive mean
cross-efficiency measure as determined by the optimal
weights generated by the Doyle and Green formulation.
Again, there are no obvious “false positives” that occur
within this column of results. Yet, the ordering of the
technology alternatives does change slightly. For
example the High-Gradient Magnetic Separation alterna-
tive has the highest mean cross-efficiency score (0.890).
The Application of Freeze-Drying Technology to
Decontamination of Liquid Wastes technology which
was ranked highest under the simple mean cross
efficiency score is now the fifth best alternative. The Dis-
tillation and Separation of Actinides from Waste Salts-
alternative, which has the lowest ROI score (63%),
ranked as the second best technological alternative at
0.859.
The fourth column in Table 6 shows the rank
efficiency scores. High-Gradient Magnetic Separation is
the most clearly favored technology using the rank
efficiency approach. As we shall see later in Table 7
Table 7
Relationships of ROI and various DEA efficiency scores
ROI ranked measures
ROI Simple tech. eff. Simple X-eff
1960% 1.000 0.989
455% 1.000 0.97
299% 1.000 0.956
205% 0.887 0.877
166% 1.000 0.993
133% 0.915 0.872
128% 0.833 0.807
63% 1.000 0.949
there is a strong ranking correlation between the rank
efficiency scores and ROI results. Yet, Distillation and
Separation of Actinides from Waste Salts-2 still per-
forms better than other units that had higher ROI scores.
Based on these results, if there were only one tech-
nology to be selected, the clear favorite would be the
High-Gradient Magnetic Separation technology, since it
performed well in terms of financial and environmental
efficiency. If there were enough capital budgeted to
allow for more than one choice then the ranking of the
alternatives would prove more useful. A useful technique
to help analyze the selection of a portfolio of alternatives
under a capital rationing scheme could be modeled using
the efficiency scores in a mathematical programming
formulation. This extension is beyond the scope of the
presentation of this paper. A difficulty arises in
determining which ranking would be the most acceptable
to management. An average ranking could be determined
from among the various techniques, but it is difficult to
determine the importance assigned to one ranking
approach versus another. This approach would probably
require a weighting of how management feels about each
of the ranking approaches.
The ranking approaches, as we have mentioned, may
contain some similarities in the order of the ranked tech-
nology. Table 7 shows a full picture of the relationships
of the rankings based on the Best to Worst ROI ranks.
Looking at these data we see that the rankings do not
seem to be consistent between ROI and the simple
efficiency or either of the cross-efficiency scores. Yet,
there does seem to be a general relationship between the
rank efficiency scores and the ROI results. To test how
well the various approaches correlate with each other
a Pearson product moment correlation coefficient was
determined for each pair of rankings as shown in Table
8. Table 8 results show that there is a strong correlation
in the rankings of the ROI and ranked efficiency
approaches (0.987). And a strong correlation between
the simple technical efficiency, simple mean cross-
efficiency, and aggressive mean cross-efficiency scores,
which are all above a 0.95 correlation score.
These correlations may simplify the analysis slightly,
Aggressive X-eff Rank efficiencies
0.890 3.112
0.856 1.118
0.846 1.054
0.739 0.887
0.837 1.016
0.762 0.915
0.698 0.833
0.859 1.052
 J. Sarkis, J. Weinrach / Journal of Cleaner Production 9 (2001) 417–427 425

Table 8
Pearson rank correlations for various evaluation techniques

Evaluation techniquea ROI STE SXE AXE RCCR

ROI 1.000 0.338 0.441 0.521 0.987

STE 1.000 0.967 0.972 0.390
SXE 1.000 0.951 0.481
AXE 1.000 0.573
RCCR 1.000

a
ROI=Return on Investment; STE=Simple Technical Efficiency; SXE=Simple Cross Efficiency; AXE=Aggressive Cross Efficiency;
RCCR=Rank Efficiency.

where management may want to only weigh the rankings
of each of the groups that are correlated. Yet, these cor-
relations are only true for this data set, and additional
data may provide significantly different correlations
among the various approaches presented here.
8. DEA and other multiple criteria decision models
A number of other tools and multiple criteria decision
models may be used for evaluation of technology.
Included among these other tools are the analytical hier-
archy process (AHP), multiattribute utility theory
(MAUT), expert systems, multiple objective mathemat-
ical programming such as goal programming, out-
ranking, simulation, and scoring models. Table 9 pro-
vides some comparative information on technique
characteristics, as well as some representative references
of their applications to justification of technologies.
Much of the literature in this area focused on manufac-
turing and information technology. Clearly, these tech-
niques are also applicable to environmental technologies,
most of which are strategically focused.
In Table 9, we see that DEA does not dominate
(perform better than all other methods) in a number of
characteristics. From a managerial point of view it is
moderately costly to implement such an approach from
a training and data gathering context. The managerial
understanding of the approach is relatively low, prim-
arily due to its mathematical complexity and use of
mathematical programming approaches to arrive at an
answer. Thus, an expert analyst would be required to aid
decision makers in applying DEA. Eventually, effective
and user friendly support systems may allow for it to be
more broadly accepted. DEA’s data requirements, in
terms of actual managerial inputs are not as extensive
as many of the other approaches. The lack of a require-
ment for managerial preference input for DEA is advan-
tageous since requiring management effort and time can
be costly. The results of the approach are relatively easy
to understand, because it gives a quick evaluation among
the portfolio of technologies and is capable of ranking
the technology based on efficiencies. Even though a var-
iety of parameters and factors can be included in a DEA
analysis, categorical and qualitative factors are still dif-
ficult to integrate.
Overall, it is difficult to compare directly the many
types of tools for evaluation of technology, each has its
disadvantages and advantages. Many times it is depen-
dent on the decision environment where time, data avail-
ability, and costs all come into play when deciding on
the appropriate tool.
As we have seen, the results from DEA are very sensi-
tive to the data and the types of scaling that is used. This
data sensitivity is a major drawback in using DEA as a
decision tool, analysts, researchers and decision makers

Table 9
Summary of multiple criteria evaluation technique characteristics and exemplary referencesa

Evaluation technique Cost of Data Ease of Economic Management Mathematical Parameter References
implementation requirements sensitivity rigor understanding complexity mixing-
flexibility

DEA M M L M L H M [11,12]
AHP M M L L M L H [13–15]
Expert systems H H L H M H H [16,17]
Goal program M M M H L H L [18,19]
MAUT H H M M M M H [20,21]
Outranking M M L M L M M [22]
Simulation H H H H H H M [23,24]
Scoring models L L L L H L H [25,26]

a
H=High; M=Medium; L=Low (Adapted from [27]).
426 J. Sarkis, J. Weinrach / Journal of Cleaner Production 9 (2001) 417–427
should be aware of these limitations, as they should for
any tool. For a more complete evaluation of how these
and other multiple criteria evaluation techniques com-
pare and their underlying philosophies please see [28–
30].
9. Summary and conclusion
This paper provides an additional set of techniques to
help evaluate investments in environmentally conscious
organizational technology. The use of DEA is presented
and evaluated from the perspective of evaluating the
environmental and financial efficiency using a multi-fac-
tor productivity measurement. The small example
presented here uses actual data to evaluate environmen-
tally conscious waste treatment technology at a govern-
mentally supported national research laboratory. The
example showed the various types of DEA analyses that
could be used to evaluate these technologies. The
example as presented was based on two input and two
output factors. This limitation was due to the number of
alternatives that were to be considered and the
input/output data that was available. More complex
decision frameworks could be modeled as well.
Variations of the DEA analysis technique included the
use of cross-efficiency and ranking techniques. We
showed that in applying DEA, managers and researchers
should be aware of the sensitivity of DEA to data and
data scaling. Inappropriate scaling may provide very dif-
ferent answers resulting in misrepresentations of the
most appropriate technology path to pursue. Researchers
who wish to apply DEA should be aware of these limi-
tations, especially if scaling is required to adjust the data
for DEA execution. We found that DEA is a good tool
to discriminate among the technologies and can also be
used to rank the technologies if a portfolio of techno-
logies were to be selected based upon a definite budget.
The selection of the appropriate DEA model is not
always clear. Managers should determine what their
goals are when seeking to select one of the tools. For
example, the basic formulation may be adequate if all
they need is to filter the “best” performers from the worst
performers. If a manager wishes to discriminate among
all the technologies applying a cross-efficiency or rank-
ing efficiency approach may be more appropriate.
In this situation we also found that DEA and ROI
could be related, but it may depend on the data used and
the DEA model that is applied. For example, if cost data
were to be used in both ROI and the DEA models then
we can expect a significant relationship. If non-cost,
intangible or performance data were introduced into the
models, we could expect the DEA results and ROI to be
significantly different. Managers should be aware of the
differences and assumptions associated with the data and
models. Whereas ROI is a relatively simple ratio relying
on financial information, DEA is a multifactor ratio
approach that can incorporate a variety of factors.
For future work, additional extensions to DEA that
could be considered in the evaluation of environmentally
conscious technologies is inclusion of weight restrictions
on the input and output weights (see [31,32]). Weight
restrictions will require the preference structures of man-
agers to help determine the upper and lower bound con-
straints for each of the weights. This requirement means
that managers and decision makers would need to be
more actively involved in providing importance levels
for the various input and output factors. This is advan-
tageous in that it allows for more realistic weighting
schemes, but disadvantageous in that management must
spend time determining the importance levels of these
factors.
Unfortunately, the DEA approach was not available
at the time this decision was made. From a practical per-
spective, managerial acceptance of these evaluations
needs to be determined. But, we have illustrated that
these techniques may provide a viable set of tools for
evaluation of environmentally conscious waste treatment
technology and other technologies within an organiza-
tion.
Acknowledgements
This work was partially supported by NSF Grant
9320949 and Texas Higher Education Coordinating
Board ATP Grant Number 003656-036.
References
[1] Camanho ASR, Dyson RG. Efficiency, size, benchmarks and tar-
gets for bank branches: an application of data envelopment analy-
sis. J Operational Res Soc 1999;50(9):903–15.
[2] Oral M, Kettani O, Lang P. A methodology for collective evalu-
ation and selection of industrial R&D projects. Managmnt Sci
1991;37(7):871–85.
[3] Sueyoshi T. Measuring efficiencies and returns to scale of Nippon
Telegraph and Telephone in production and cost analysis. Man-
agmnt Sci 1997;43(6):779–96.
[4] Glass JC, McKillop DG, O’Rourke G. A cost indirect evaluation
of productivity change in UK universities. J Prod Anal
1999;10(2):153–75.
[5] Charnes C, Cooper WW, Lewin AY, Seiford LM. Data envelop-
ment analysis: theory, methodology and application. Boston:
Kluwer Academic Publishers, 1993.
[6] Boerigter ST. Cost/benefit analysis for waste minimization tech-
nologies at TA-55. Technical Report LA-UR-95-1977, Los
Alamos National Laboratory, Los Alamos, New Mexico 1995.
[7] Charnes C, Cooper WW, Rhodes E. Measuring the efficiency of
decision making units. Eur J Operational Res 1978;2:429–44.
[8] Andersen P, Petersen NC. A procedure for ranking efficient units
in data envelopment analysis. Managmnt Sci
1993;39(10):1261–4.
[9] Sexton TR, Silkman RH, Hogan AJ. Data envelopment analysis;
critique and extensions. In: Silkman RH, editor. Measuring
 J. Sarkis, J. Weinrach / Journal of Cleaner Production 9 (2001) 417–427
efficiency; an assessment of data envelopment analysis. San Fran-
cisco: Jossey-Bass, 1986:73–104.
[10] Doyle J, Green R. Efficiency and cross-efficiency in DEA: deri-
vations, meanings and uses. J Operational Res Soc
1994;45(5):567–78.
[11] Khouja M. The use of data envelopment analysis for technology
selection. Comp and Indust Engng 1995;28(2):123–32.
[12] Sarkis J. Evaluating flexible manufacturing systems alternatives
using data envelopment analysis. The Engng Economist
1997;43(1):25–48.
[13] Albayrakoglu MM. Justification of new manufacturing tech-
nology: a strategic approach using the analytical hierarchy pro-
cess. Product Inventory Managmnt J 1997;37(1):71–6.
[14] Kleindorfer PR, Partovi FY. Integrating manufacturing strategy
and technology choice. Eur J Operational Res 1990;47:214–24.
[15] Sarkis J. Evaluating environmentally conscious business prac-
tices. Eur J Operational Res 1998;107(1):159–74.
[16] Borenstein D. IDSSFLEX: an intelligent DSS for the design and
evaluation of flexible manufacturing systems. J Operational Res
Soc 1998;49(7):734–44.
[17] Padmanabhan S. A tandem expert support system as justification
for a flexible manufacturing system. J Manufacturing Syst
1989;8(3):195–205.
[18] Stam A, Kuula M. Selecting a flexible manufacturing system
using multiple criteria analysis. Int J Production Res
1991;29(4):803–20.
[19] Suresh NC. An extended multi-objective replacement model for
flexible automation investments. Int J Production Res
1991;29(9):1823–44.
[20] Chandler JS. A multiple criteria approach for evaluating infor-
mation systems. MIS Quarterly 1982;6(1):61–74.
[21] Pandey PC, Kengpol A. Selection of an automated inspection
system using multiattribute decision analysis. Int J Production
Economics 1995;39(3):289–98.
[22] Parsaei HR, Wilhelm MR, Kolli SS. Application of outranking
methods to economic and financial justification of CIM systems.
Comput Industrial Engng 1993;25(1/4):357–60.
[23] Primrose PL. Investment in manufacturing technology. London:
Chapman and Hall, 1991.
[24] Suresh NC, Meredith JR. Justifying multimachine systems; an
integrated strategic approach. J Manufacturing Syst
1985;4(2):117–34.
[25] Nelson CA. A scoring model for flexible manufacturing systems
project selection. Eur J Operational Res 1986;24:346–59.
427
[26] Semich JW. Here’s how to quantify IT investment benefits. Dat-
amation 1994;40(1):45–7.
[27] Sarkis J, Sundarraj RP. Factors for strategic evaluation of
enterprise Information technologies. Int J Physical Distrib Logis-
tics Managmnt 2000;30(3/4):196–220.
[28] Canada JR, Sullivan WG. Economic and multiattribute evaluation
of advanced manufacturing systems. Englewood Cliffs, NJ: Pren-
tice-Hall, 1989.
[29] Olson DL. Decision aids for selection problems. New York:
Springer, 1996.
[30] Zionts S. Multiple criteria decision making; an overview and sev-
eral approaches. Working Paper 454, SUNY at Buffalo School
of Management, Buffalo, NY 1980.
[31] Thompson RG, Langemeier LN, Lee CT, Thrall RM. The role
of multiplier bounds in efficiency analysis with application to
Kansas farming. J Econometrics 1990;46(1/2):93–108.
[32] Wong YH, Beasley JE. Restricting weight flexibility in data
envelopment analysis. J Operational Res Soc 1990;41(9):829–35.
Joseph Sarkis is currently an Associate Professor in The Graduate School
of Management at Clark University. He earned his PhD from the State
University of New York at Buffalo. His research interests include manu-
facturing strategy and management, with a specific emphasis on perform-
ance management, justification issues, enterprise modeling and environ-
mentally conscious operations and logistics. He has published over 120
articles in a number of peer reviewed academic journals and conferences.
He is a member of the American Production and Inventory Control Society
(APICS), Institute for Operations Research and Management Sciences
(INFORMS), the Decision Sciences Institute (DSI), and the Production
and Operations Management Society (POMS). He is also a certified pro-
duction and inventory manager (CPIM).
Jeffrey Weinrach has many years of experience in pollution prevention
and waste minimization (P2/WMin). He has provided leadership to the
Pollution Prevention Program Office, now called the Environmental Stew-
ardship Office, at Los Alamos National Laboratory (LANL). He has
worked to bring innovative approaches to LANL P2/WMin activities. He
has worked with the US Department of Energy (DOE), US Environmental
Protection Agency (EPA), US Department of Commerce, US Department
of Defense, the State of New Mexico, and the University of New Mexico
to bring ideas and projects together. Dr. Weinrach is co-Editor-in-Chief
of the “International Journal of Environmentally Conscious Design and
Manufacturing” and serves on the Editorial Board of the “Encyclopedia
of Life Support Systems.” Dr. Weinrach also serves as General Chairman
of the “International Congress of Environmentally Conscious Design
and Manufacturing.”