Information & Management 36 (1999) 151±164

Research
Evaluation models of information technology in
Spanish companies: a cluster analysis
E. Huerta Arribasa, P.J. SaÂnchez Inchustab,*
a
Dpt. Business Administration, Universidad PuÂblica de Navarra, Campus Arrosadia 31006, Pamplona, Spain
b
Dpt. of Mathematic and Computer Science. Universidad PuÂblica de Navarra, Campus Arrosadia 31006, Pamplona, Spain

Received 5 November 1997; accepted 21 January 1999

Abstract

The aim of this article is to study how Information Technology (IT) evaluation is carried out among a group of Spanish
companies, establish a typology or grouping of them which uses similar evaluation criteria, determine the relationship between
the level of IT maturity and the way in which it is evaluated and, ®nally, if there are any similarities in the forms of evaluation
carried out by Spanish and foreign companies. The analysis will be based on a detailed study of 20 Spanish ®rms, taking into
account the different stages of development of IT within each ®rm. # 1999 Elsevier Science B.V. All rights reserved.

Keywords: IT evaluation; Development level of IT; Comparative study of IT; Cluster analysis

1. Introduction ®nancial techniques are not suitable to evaluate invest-

The evaluation of Information Technology (IT) has
been seen as a complex task owing both to the
dif®culty in determining the value of information as
well as to the special economic characteristics of the
technology which produces it. One proof of this
complexity is the dichotomy which exists between
the practice and the theory.
Researchers even believe that a sound disparity
exists between the methods of evaluating investment
in IT, not only between different organizations but also
within the same company [13], based on the general
belief that investment in IT is different from other
forms of investment. This leads to the conclusion that
*Corresponding author. Tel.: +34-38-169-538; fax: +34-48-169-
521; e-mail: inchusta@upna.es
0378-7206/99/$ ± see front matter # 1999 Elsevier Science B.V. All rights reserved.
PII: S - 0 3 7 8 - 7 2 0 6 ( 9 9 ) 0 0 0 1 4 - 2
ment in IT [21, 33, 30, 28, 22].
However, little empirical evidence exists as to
whether both types of investment are treated by com-
panies in the same way. The few existing papers using
the survey method conclude that in practice compa-
nies use similar procedures to justify all types of
investment, whatever their nature [39, 1, 18]. Other
works do not offer such conclusive results and their
authors consider the reasons for similar or different
treatment to be extremely diverse [2].
The aim of this article is to study how IT evaluation
is carried out among a group of Spanish companies,
analyse the characteristics which unite or differentiate
them and determine whether we can establish a typol-
ogy or grouping of companies which uses similar
evaluation criteria. To do this, we pose ®ve main
questions to be answered: (a) Is evaluation of IT
investment viewed differently from other generic
152 E.H. Arribas, P.J.S. Inchusta / Information & Management 36 (1999) 151±164
investment? (b) Is the level of IT development in the
companies homogeneous? (c) Can the companies be
grouped into speci®c evaluation typologies? (d) Can
any relationship be established between the level of IT
maturity-co-ordination between organizational and IT
aims- and the way in which it is evaluated? (e) Are
there any similarities in the forms of evaluation carried
out by Spanish and foreign companies?
To answer these questions we have analysed 20
Spanish companies. Given the complex number of
cases and variables involved in the compilation of
quantitative and qualitative aspects, we have used
cluster and correspondence analysis.
The structure of the paper is as follow: Section 2
involves a justi®cation of the methodology used and
the group of companies chosen. In Section 3, the
results are analysed regarding the similarities between
IT evaluation and that of other investment, the levels of
maturity and the evaluation typologies. In Section 4,
we compare the results obtained with those of other
studies and ®nally, in Section 5, the basic conclusions
are drawn.
2. The empirical research
2.1. Methodology
Case studies have been consistently defended as a
suitable methodology for the study of IT and, more so,
for the analysis of their evaluation [4, 3, 24, 17, 37].
The main reasons given are the complexity of the task,
the multiple perspectives and interpretations which
can be adopted, the advantageousness of considering
content, context and evaluation process, the dynamic
and transitory nature of this technology and the need to
use quantitative and qualitative information to under-
stand these processes. The study of individual cases
would seem to be suitable for the investigation of
subjects which have not previously been studied,
while multiple cases are recommended when the
aim is to describe or construct theories, or to test them
as this allows the cross analysis of cases. What is
more, multiple cases are useful when the object of the
research is not to generalize but understand individual
processes or topics and recognize the capacity for
success. To this end, speci®c examples of excellence
are looked at and information is sought from quali®ed
groups [27]. However, when the number of cases is
high, drawing up conclusions can prove dif®cult if
some statistical tools are not used [14].
We have studied a group of 20 companies and
collected information by means of detailed interviews
in order to obtain qualitative information from differ-
ent levels and functions of the organizations. We have
used the cluster analysis which allows us to describe
common behavioural guidelines and bring companies
together according to the characteristics which distin-
guish them most from the other groups. This technique
has allowed us to establish niches or groups with
certain characteristics in common, not only of a
quantitative nature but also qualitative or nominal.
2.2. Characteristics of the companies
Twenty companies belonging to diverse economic
sectors with a variety of legal status, both national and
multinational were chosen. In terms of workforce size,
three companies are small (less than 100 workers), 14
are medium-sized (between 100 and 150) and three are
large (more than 500). However, weighing up their
incidence according to workforce size, 43% of work-
ers are found in large companies, 53.7% in medium
and 3.1% in small ones.
The companies were chosen with the aim to observe
behaviour and speci®c situations. Thus, the chosen
companies obey one or more of the following criteria:
(a) Innovative technology such as EDI or JIT has
been introduced (6 Companies).
(b) Signi®cant investment in IT has been carried out
(3).
(c) They are competitive and have a major involve-
ment in the foreign sector (4).
(d) They produce innovative and competitive pro-
ducts from an industrial point of view (4).
(e) Other characteristics (3).
The information concerning these companies was
principally obtained using open questionnaires as
stipulated in case study guidelines, and was completed
using information gathered from written documents,
both from internal company sources as well as pub-
lished material. Following this, the information was
put into closed question categories which allowed a
statistical analysis. As a result, we obtained a large
group of variables of which the majority are consid-
 E.H. Arribas, P.J.S. Inchusta / Information & Management 36 (1999) 151±164
ered to be active (60) and six illustrative and which
bring together primarily information of a qualitative
nature. As for the interviewees and their correspond-
ing hierarchical levels, four of the 45 people spoken to
correspond to General Management, 17 to Head of
Computing, 15 to Financial Director, one to Personnel
Manager, four to Manufacturing Manager and the
remaining four to other categories.
This information has been structured around three
axes relating to content, context and evaluation pro-
cess. The content refers to the values and criteria to be
considered and what should be measured. The process
is the way in which evaluation is carried out and who
are the actors. The context can be external-competitive
environment- or internal-organizational structure-.
This is in line with contextual and interpretative
theories since for a correct understanding of IT eva-
luation, not only must the values be analysed but also
the organizational setting and those who carry out the
process [7, 35] [36, 11, 12, 25, 29, 5, 20, 8, 23].
3. Empirical results
This research seeks to answer the questions which
concern us and which we set out in the introduction,
regarding how companies evaluate IT, what methods
are used and the in¯uence which the level of devel-
opment of IT can have on the way in which it is
evaluated. The main results are described.
3.1. Similarities between IT and other generic
investment evaluation
The main conclusion is that 11 companies out of 20
believe that the evaluation of IT investment is different
from other investment although no distinction is made
to the different kinds of IT and its wide range of
objectives. Other considerations which demonstrate
the peculiar nature of IT include the fact that evalua-
tion deals on the whole with generic projects (16 out of
20), but only in a minority of cases with IT; also,
within the ®nancial techniques only one is employed
(cost/bene®t analysis or net present value in 15 com-
panies) for generic investment while in the case of IT a
combination of techniques are used (13 companies).
Furthermore, different attitude is shown towards
expenditure on hardware or on software; the former
153
as investment and this as expenditure, which under-
lines this peculiar nature.
3.2. Maturity levels of IT
We understand as IT maturity the degree to which
companies incorporate IT in order to achieve organi-
zational aims. A multitude of theoretical models exist
although the one we have chosen [38, 15, 9, 16, 40] is a
simple but powerful one according to which we estab-
lish categories of IT for automation; IT to aid decision
making; and IT for strategic purposes. These models
are de®ned according to their objectives of mechan-
izing tasks which were previously carried out by hand
thereby achieving cost ef®ciency; obtaining the infor-
mation necessary to make decisions, and thus achiev-
ing effectiveness; and obtaining competitive advan-
tages, respectively.
The cluster analysis separates the companies stu-
died into four groups each with distinguishing mod-
alities. The variables and modalities in Table 1
characterise each of the four groups with the level
of con®dence shown in T-test and Prob. The ratio cla/
mod means the percentage of the companies who
display this modality and who are included in the
group; the ratio mod/cla means the contribution of the
companies with these modalities to the group. For
more details see [32].
The ®rst group is made up of companies in which
the management plays an important role as promoter
and motivator in the incorporation of IT. Furthermore,
IT is seen as a strategic factor which serves to link up
and co-ordinate with customers via the use of internal
and external networks. The IT department is of con-
siderable importance due both to its ®nancial
resources for expenditure and investment as well as
to the large number of staff who are employed there.
With reference to the development levels adopted, we
can broadly classify these systems as strategic systems
(SIS). The group comprises three companies.
The second group is made up of those companies
whose sales ®gures de®ne them as being large and yet
have no strategic plan. They have a high ratio of
computer equipment/indirect personnel, the equip-
ment is connected to an internal network and they
possess applications to aid decision making, albeit of a
personal nature and via of®ce automation applica-
tions. In other words, where IT is dedicated to the
154 E.H. Arribas, P.J.S. Inchusta / Information & Management 36 (1999) 151±164

Table 1
Maturity groups: special characteristics

Nominal characteristics Ratios t-test Prob.

cla/mod mod/cla
Variables Modalities

Group 1/4
Manager compromise Promoter 100.00 100.00 3.13
IT strategic planning Realized 60.00 100.00 2.38
IT expenses (1.5±2)% Turnover 100.00 66.67 2.15
IT competitive advantage Implication 42.86 100.00 1.87
Relation terminal/pc No preponderance 42.86 100.00 1.87
People IT department >5 people 66.67 66.67 1.69
IT evaluation Internal evaluation 37.50 100.00 1.65
Group 2/4
Decision applications Used 70.00 100.00 2.96
Strategic planning No strategic plan 83.33 71.43 2.45
Terminals computer No 53.85 100.00 2.01
Sales >40 millions $ 60.00 85.71 1.90
Computers/person Unitary 50.00 100.00 1.70
Internal networks Yes 54.55 85.71 1.57
Type of internal networks Business and factory 62.50 71.43 1.62
Group 3/4
Organizational dependance Do not exist 100.00 100.00 1.64
Functions Not specific 100.00 100.00 1.64
Group 4/4
Terminal computers Terminals yes 100.00 77.78 3.31
Relation terminals/PCs Terminals 100.00 55.56 2.40
Strategic planning Yes 64.29 100.00 2.26
Internal networks No 77.78 77.78 2.25
Computers/person (50±80) % 100.00 44.44 1.94
External networks No 70.00 77.78 1.81
Decision applications No 70.00 77.78 1.81
Type of external networks No 70.00 77.78 1.81

internal working of the company, strengthening inter-
connection and extending information and where the
system can be described as management decisions
support system (DSS). The group is made up of seven
companies.
The characterizing feature of Group 3 is that IT is
not considered strategic nor is there a computer
department, only users. The ratio of computer/indirect
personnel is very low and hardware structure is made
up by individual microcomputers. One out of the three
small companies is in this group.
In Group 4, IT is not considered important in the
strategy of the company. Guidelines for behaviour are
established which take no account of the role of IT
which continues to deal exclusively with the automa-
tion of tasks previously carried out manually. These
systems can be denominated systems of transaction
processing or automation. Group 4 is made up of nine
companies, almost half of those studied, and is the
most numerous group.
If we compare groups two and four, systems for
aiding decision making and automation systems, we
see that they display opposite characteristics (Fig. 1)
and that they obey the rationale of the models. The
only possible question mark is that companies in
group four possess a strategic plan, an advanced level
of formal planning, and yet the level of computeriza-
tion is relatively immature. This is a real, and quite
common situation and normally involves cases in
which a strategic plan has been developed as a result
of market needs and is explained by the manufacture
of a brand new product following the disappearance of
 E.H. Arribas, P.J.S. Inchusta / Information & Management 36 (1999) 151±164
Fig. 1. Modalities in groups 2 and 4.
demand (as in the case of spare parts for cars), by the
crisis caused by domestic or international competition
and/or the withdrawal of foreign capital. The company
directors have adopted a strategic plan which allows
the company to survive, but is normally developed by
external agents and without any involvement of IT in
the plan even when the lack of outside data has been
identi®ed as the weak point of the company.
3.3. IT evaluation typologies
In order to analyse the evaluation of IT, the different
techniques employed by the companies speci®cally in
IT investment are described together with their content
and their problems. We also look at the evaluation
process, dealing with such questions as who carries it
out, who makes the decision, what values are con-
sidered and what consideration is given to the evalua-
tion itself. In short, the aim is to deduce which
characteristics are held in common and which are
not, in order to establish possible, real evaluation
typologies.
The information used is contained in twenty nine
variables, the majority of which are of a qualitative
nature, 27 being considered active and two illustrative,
and which contain 102 modalities. This consideration
of active and illustrative variables is in our opinion the
most suitable in the context of the theories regarding
IT maturity and evaluation [31].
The result of the cluster analysis is the classi®cation
of all the companies studied into four groups whose
most outstanding modalities can be seen in Table 2
and which we will subsequently look at in more detail.
155
Group 1. Strategic evaluation. This group com-
prises ®ve companies which represent a quarter of
all the companies studied. The companies belonging
to this group can be de®ned using the following
characteristics which distinguish them from the other
groups: the use of diverse and varied techniques in the
evaluation of IT investment, depending naturally on
the different kinds of projects; the values observed are
in general strategic and bene®ts which may affect the
customer are included in a consideration of bene®ts to
be gained. Furthermore, those who carry out the
®nancial evaluation of investment are those who pro-
pose the investment and this evaluation is considered a
bene®cial process for the company.
The modalities which characterize this group from
others, together with the level of con®dence and the
class/modality and modality/class relation are seen in
the previously mentioned Table 2. It is worth pointing
out that, all in all, over 50% of the companies who
display each of these modalities are included in this
group, with less than a 5% probability of error. The
modality `maturity level 1' stands out dramatically,
emphasising the fact that all the companies which
correspond to this highest level of maturity are in this
group. The same percentage is found regarding `use of
different techniques to evaluate IT investment'. In
addition, of the companies whose characteristics
include a consideration of the bene®ts to the customer
brought about by IT, and/or an evaluation of invest-
ment carried out by the same person to propose it, 80%
are found in this group. However, those companies
who exclusively consider as a priority the values of
cost or necessity are not included here.
In short, the consideration of strategic values in the
evaluation of IT, the involvement of IT in company
strategy and the attention paid to of one of the
important competitive forces in this setting, the cus-
tomer, lead us to denominate this group Strategic
evaluation.
Group 2. Evaluation by standard ®nancial techni-
ques. This group consists of two companies. The
characteristics which most stand out from the rest
of the companies, and which in addition are held by
all components of the group, are the use of a combina-
tion of budgetary investment techniques in the evalua-
tion of generic investment; the importance given to
these techniques in the evaluation of IT investment
despite no written norm concerning its implementa-
156 E.H. Arribas, P.J.S. Inchusta / Information & Management 36 (1999) 151±164

Table 2
Evaluation models

Modalities and characteristics Ratios

cla/mod mod/cla T-test Prob.

Group 1/4
Evaluator Proposer Prop 80.00 80.00 2.58 0.005
Stakeholder Customers C9CL 80.00 80.00 2.58 0.005
Maturity Level 1 NIV1 100.00 60.00 2.38 0.009
Techniques Various B6DI 100.00 60.00 2.38 0.009
Values Strategic ESTR 50.00 100.00 2.14 0.016
Evaluation*2 Beneficial JBEN 57.14 80.00 1.87 0.031
Strategy Strategic SIES 41.67 100.00 1.63 0.051
Authority Owner Board COAD 100.00 40.00 1.62 0.053
Group 2/4
Importance Very important C2MU 100.00 100.00 2.56 0.005
Criteria Combination A4VA 50.00 100.00 1.86 0.032
Norms Do not exist B5NO 40.00 100.00 1.62 0.053
Benefits Ident.Intan ‡ Quant.Fi IICF 40.00 100.00 1.62 0.053
Gral. norms Do not exist A1NO 33.33 100.00 1.41 0.079
Problems EE C8EE 100.00 50.00 1.28 0.100
Techniques Feeling B7FE 100.00 50.00 1.28 0.100
Evaluation*2 Beneficial JBEN 28.57 100.00 1.22 0.111
Group 3/4
Extension Quantitative importa A3CU 75.00 85.71 2.62 0.004
Importance Insignificant B1IN 58.33 100.00 2.32 0.010
Frequency When required C1MA 100.00 42.86 1.87 0.031
Criteria Payback A4PA 66.67 57.14 1.42 0.078
Evaluator CEO FINA 55.56 71.43 1.27 0.102
Benefits Identif. intang. IINO 55.56 71.43 1.27 0.102
Values Necessity NECE 55.56 71.43 1.27 0.102
Similarity Similar B4SI 55.56 71.43 1.27 0.102
Group 4/4
Financial techniques Used A2SI 71.43 83.33 2.45 0.007
Maturity Level 4 NIV4 80.00 66.67 2.20 0.014
Evaluator General Manager GERE 100.00 50.00 2.11 0.018
Benefits Non intan-finan NINF 100.00 50.00 2.11 0.018
Criteria Others A4OT 62.50 83.33 2.10 0.018
Frequency None C1NO 55.56 83.33 1.78 0.038
Extension None C3NO 55.56 83.33 1.78 0.038
Extension None A3NO 75.00 50.00 1.55 0.061

tion; the evaluation of IT by quantifying economic
bene®ts and identifying those which are most quali-
tative or intangible; and ®nally the view that the formal
process of IT evaluation is bene®cial to the company.
In short, the enormous importance given to the use
of capital budgetary techniques to evaluate IT invest-
ment, together with the quantifying of economic
bene®ts lead us to denominate this group Standard
®nancial evaluation.
Group 3. Evaluation for reasons of necessity. This
group is made up of seven companies, representing
over a third of the companies in this study. It's
distinguishing characteristics are the following: The
companies only use capital budgetary techniques to
evaluate generic investment involving important quan-
tities, the most common of these techniques being the
recovery period; expenditure and investment in IT is
not a signi®cant sum in relation to turnover or total
 E.H. Arribas, P.J.S. Inchusta / Information & Management 36 (1999) 151±164
investment; thus, when budgetary techniques of
evaluation are used, they are used sporadically or as
they are required and not during the elaboration of
budgets.
Statistically, we can see that two-thirds of the
companies for whom IT investment is seen to be
insigni®cant are included in this group and all the
companies in the group display this characteristic. As
for the evaluation of generic investment, three quarters
of those who only evaluate large investments are
included here and this characteristic is dominant
among the companies of this group. With regard to
the frequency of evaluation, all those who carry out
evaluation sporadically belong to this group.
In this group there are no differences in the ways in
which investment in IT and generic investment are
evaluated. The ®nancial manager normally performs
the evaluation which, curiously, is limited to identify-
ing intangible bene®ts. Investment in IT is as a result
of necessity. 55% of those companies which display
these modalities are included in this group which
constitutes 70% of the group. For these reasons, we
have labelled this group Evaluation for reasons of
necessity.
Group 4. Cost evaluation. Six companies are
included in this group. The characteristics which
distinguish the group are the following: No ®nancial
techniques are used in the evaluation of generic
investment nor in that speci®cally of IT. Nor are the
criteria ®nancial: cost alone is considered, with no
attention paid neither to ®nancial nor qualitative ben-
e®ts. Furthermore, evaluation is normally carried out
by the General Manager.
All companies which only consider cost in their
evaluation without paying attention to any kind of
bene®t and all those in which the General Manager
carries out the evaluation are included in this group.
What is more, 80% of the companies in Group 4 of
maturity, or transaction processing are included here.
We will look more closely at this point in the following
section.
To sum up this epigraph, four different evaluation
typologies are found among the companies studied
which we have labelled strategic evaluation, standard
®nancial, reasons of necessity and cost criteria. Their
incidence within the group as a whole is unequal since
evaluation based on cost and on necessity is found in
13 of the 20 cases studied.
157
3.4. Correspondence between maturity and
evaluation levels
One question which was put forward in the intro-
duction, and which we will now analyse is whether a
relationship exists between the IT maturity level and
the different evaluation systems, assuming as a theo-
retical hypothesis that the most advanced levels of
maturity are accompanied by the best systems of
evaluation. To do this, we will use the results of the
cluster analysis from the previous epigraph and com-
plete them with the simple correspondence analysis
[19, 10].
Using the cluster analysis in the previous section we
noticed some interesting facts. Firstly, a correspon-
dence is con®rmed between the strategic information
systems and their own strategic evaluation. Thus, we
notice that all the companies pertaining to the Group 1
maturity level are included in the evaluation of Group
1. Secondly, in the other extreme, a correspondence
exists between those companies using IT for automa-
tion and the companies which evaluate investment in
IT according to cost. In this correspondence, half of
the companies situated in Group 4 maturity level are
represented in the evaluation Group 4 and make up
almost two-thirds of the companies in this evaluation
group.
We have completed these results of the cluster
analysis using simple correspondence analysis and
the elaboration of the ad hoc statistic (Table 3). We
are able to see that ®ve companies carry out a
strategic type of evaluation. However, the maturity
level of this category does not correspond entirely
to maturity Group 1 since two other companies
exist, one with DSS maturity level and the other,
automation.
The second evaluation group, that of the standard
®nancial techniques is carried out in two companies,
one corresponding to the DSS level of development
and the other to the automation level. The third
evaluation model, that of necessity, concerns seven
companies of which four are found at the DSS level
and three at the automation level. The fourth model,
that based on cost, includes six companies, one with
DSS level, four with automation level and one at an
initial stage of computerization. This demonstrates
that the evaluation only takes into account the cost
of the investment.
158 E.H. Arribas, P.J.S. Inchusta / Information & Management 36 (1999) 151±164

Table 3
Calculation and sign of the differences

Maturity

Evaluation G1-SIS G3-DSS/G2-ITP G4-TP LINE TOTAL

G1-strategic
Real value 3 1 1 5
Expected value 0.8 2 2.3 25%
Relative V and sign ‡2.9 ÿ0.5 ÿ0.57
G2-financial
Real value 0 1 1 2
Expected value 0.3 0.8 0.9 10%
Relative V and sign ÿ1.0 ‡0.25 ‡0.11
G3-necessity
Real value 0 4 3 7
Expected value 1.1 2.8 3.2 35%
Relative V and sign ÿ1.0 ‡0.43 ÿ0.03
G4-cost
Real value 0 2 4 6
Expected value 0.9 2.4 2.3 0%
Relative V and sign ÿ1.0 ÿ0.17 ‡0.48
Column 3 8 9 20
Total 15% 40% 45% 100%

To ®nd out whether in our subgroup, a higher IT
maturity level corresponds to a greater level or sophis-
tication of evaluation tools, we have elaborated an
approximate ad hoc statistic, and we have calculated
the sign and relative value of the differences between
the real and expected outcomes to determine the
existence of correspondence between the modalities,
sign and magnitude. Among the values and sign of this
ad hoc statistic we can point out 0 and 1 as the most
signi®cant values together with the positive and nega-
tive signs. The 0 value of this statistic indicates the
independence between both modalities or lack of
correspondence; the positive signs represent positive
correspondences or attraction between the modalities
and the negative signs, negative correspondence or
rejection. The value ÿ1 demonstrates the inexistence
of real values and instead estimated ones; the value ‡1
demonstrates positive correspondence and in it the real
values are double those expected.
We are aware of the inappropriateness of the usual
statistics such as 2, given that the size of the sample is
clearly insuf®cient to be representative, likewise its
randomness we have indicated the speci®c character-
istics of the chosen companies [6]. This will prevent us
from generalizing our conclusions but will allow us to
discover the correspondence in the subgroup studied.
From this analysis we can deduce that a positive
correspondence or attraction exists, one which is very
strong, between the level of strategic maturity and the
strategic evaluation system; a blatant attraction
between the DSS level of maturity and evaluation
of necessity; and equally between the automation level
of maturity and cost evaluation. However, a weak but
nevertheless positive correspondence exists between
the DSS level of maturity and the ®nancial level of
evaluation and almost complete independence
between the automation level of maturity and the
®nancial evaluation.
With regard to the rejection or negative correspon-
dence, the level of strategic maturity is clearly
opposed to the standard ®nancial types of evaluation
as well as those of necessity and cost. The DSS level of
maturity displays a slight rejection of strategic evalua-
tion and an almost complete independence to standard
®nancial evaluation. In short, within the group of
companies studied and bearing in mind the statistical
limitations mentioned, we can observe a positive
correspondence or attraction of considerable size,
 E.H. Arribas, P.J.S. Inchusta / Information & Management 36 (1999) 151±164
between the strategic evaluation form and the level of
strategic maturity. The second attraction of signi®cant
proportions is found between cost evaluation and the
automation level of maturity. A similar situation can
be found between evaluation of necessity and the DSS
level of maturity. However, standard ®nancial evalua-
tion has no obvious attraction with any level of
maturity, excepting its negative relationship with the
strategic level.
From an economic perspective, we can conclude,
®rstly, that a relationship exists between strategic
maturity and strategic evaluation. In other words, it
reveals that those companies in which the manage-
ment plays an active and dynamic role in the intro-
duction and planning of IT, where the function of the
computer department is linked to company strategy,
and where expenditure and investment in IT is both of
quantitative and qualitative importance, employ eva-
luation methods of IT investment where strategic
values take precedence, with the logical and subse-
quent medium and long term view. Furthermore, the
setting in which the company operates is taken into
account and the customer occupies an important posi-
tion as bene®ciary of the advantages which IT may
originate. The evaluation of such investment is viewed
differently from that of generic investment. Those who
propose the investment are the same people to evaluate
it and the process of evaluation itself is considered
bene®cial to the company. We have summed up these
aspects in the so called strategic evaluation.
Secondly, there is a positive correspondence
between the maturity level 2 or DSS and the evaluation
of necessity. That allows us to conclude that those
companies which do not possess a general strategic
plan, but do possess computer applications to aid
decision making, with a computer equipment structure
connected via internal corporate networks which link
up industrial plants, central of®ces and all areas of the
company are companies which only evaluate large
generic projects. In such cases the recovery period tool
is used in which expenditure and investment in IT is
not substantial, evaluation is sporadic rather than
regular and no difference is perceived between generic
evaluation and that of IT. The ®nancial manager who
is responsible for carrying out the investment cur-
iously identi®es the intangible bene®ts although it is
the principal of competitive necessity which has most
in¯uence on the decision.
159
In the third place, the correspondence between cost
evaluation and the automation level of maturity means
that companies who have established a strategic plan
for the purpose of emergency situations in the com-
pany, who are without an internal or external network
linking their hardware structure, and whose applica-
tions are used to carry out regular transactions but not
to make decisions, these companies do not employ
®nancial techniques or criteria either in the evaluation
of generic investment nor IT investment. The only
aspect to be taken into consideration is the cost.
To sum up, we can af®rm that a positive coincidence
exists within the group of companies studied between
strategic maturity and the strategic evaluation of IT;
between the DSS level of maturity and the evaluation
of necessity, and ®nally, between the transaction pro-
cessing system and that of cost evaluation. In the other
modalities, the independence, rejection or non-exis-
tence of correspondence can be maintained.
4. Comparison with other empirical results
Our ®nal task is to check our results with those
obtained in other international studies. In this area, the
empirical evidence as to whether both types of invest-
ment are dealt with by companies in a similar or
different way continues to be scarce. Work carried
out using the survey method indicates that in practice
companies use similar procedure to justify all types of
investment, independent of their nature [39, 18, 1].
Other authors believe that the reasons for similar or
different treatment are extremely diverse [2] produ-
cing even disparate results, normally when the
research is carried out by means of case study and
is therefore more comprehensive and qualitative.
Thus, research into 16 English companies [13] con-
cludes that few companies consider investment in IT
differently to other capital investment and, in general,
quanti®able, tangible and short term bene®ts are taken
into account. In nine Greek cases [26], IT investment
is considered different but evaluation is the same due
to the inability to ®nd an alternative method. In 14
South African cases [34], the difference lies in who is
the evaluator rather than the subject of the evaluation
and, thus the different types of IT projects are not
evaluated differently. However, a distinction is made
when the evaluators are the general management or the
160 E.H. Arribas, P.J.S. Inchusta / Information & Management 36 (1999) 151±164
IT manager, with qualitative criteria predominating in
the former and purely ®nancial criteria in the latter. In
three cases from the Low Countries [8] a great dif-
ference is acknowledged among the evaluation meth-
ods according to the objectives sought by the IT
projects. Thus, of the four studies carried out using
the case study method, three consider that IT evalua-
tion requires different procedures from all other invest-
ment, coinciding with our own conclusion (Fig. 2).
In contrast to the case study and survey methods, a
methodology which joins together both has been used:
the information has been compiled in a similar way to
the case study considering the complexity of the
evaluation in content, context and process aspects,
and a lot of qualitative information has been obtained.
Later, this information has been processed by using
cluster analysis. The conclusion of our work is that the
evaluation of IT is unlike the evaluation of generic
investments. This result is nearer that to the case study
that to the results of the survey method.
With regard to the relationship between the maturity
levels and evaluation models, the other works have not
studied this question. This is a particular and distinct
aspect of our work, which differs from the others and
which allows to bring a more general and wider vision
of the evaluation processes. We reach the conclusion
that there is a direct relationship between the maturity
levels and evaluation systems. Different maturity
levels and evaluation models have been found and,
so, our study is holistic. The perspective of other
studies focuses in a speci®c impact of IT (maturity
in a wide sense) and in a unique evaluation system. In
this way, we consider Farbey's study dealing with an
IT structure to aid decision making (DSS), Mirtidis'
study dealing with automation and Deitz's study with
strategy, and we compare them with characteristics
from our own groups of strategic evaluation, standard
®nancial, necessity and cost.
In Farbey's study a situation is described in which
no IT planning exists. This makes it impossible to
discover links with business strategy which means, as
a consequence, that the expected impact is low and not
strategic. Bene®ts, for this reason, are considered
tangible and short term. Farbey con®rms the dichot-
omy which exists between the potential impact of IT
and that which companies see as genuinely viable and
seeks to achieve. She also points to the progressive
abandonment of formal ®nancial techniques since
only four out of sixteen cases use them. If we compare
Farbey's study with our own necessity evaluation
group, so far as contextual aspects are concerned,
we can see that both studies display DSS or decision
support levels of development, and in neither of the
studies do we ®nd any indication that IT is considered
different from other types of investment. As for the
investment process, the department and IT managers
and not the users are those who have the biggest say in
its go ahead. Similarly, in our cases the ®nancial
managers are those who in addition list intangibles
among the bene®ts to be taken into consideration; so,
the evaluators come to the point of the hierarchical
level of dependency from the General Manager but
they do not carry out the same tasks.
In Mirtidis' study, the purpose of IT is automation
and the companies demonstrate a low level of matur-
ity. The evaluation of IT is perceived to be different
from that of other investment although the companies
do not know how to identify and quantify the intan-
gibles. Almost half the cases use DSS ®nancial tech-
niques and those who promote IT investment do not
have a decisive say in the process which instead is the
responsibility of the ®nancial manager. In our cost
study model, the aim of IT is also automation, or the
attainment of ef®ciency. The evaluation of IT is
considered different from other investment and there-
fore standard ®nancial techniques are not used. Only
the cost is taken into account. The ®nancial manager is
responsible for investment in IT although the users are
the main promoters of both its investment and use.
In comparison with Deitz's cases, a great similarity
is observed since strategic evaluation is related con-
textually to those companies in which strategy plays a
fundamental role in determining investment priorities
and IT has an important strategic role. As for content,
strategic criteria rather than ®nancial techniques are
important and as far as the process is concerned, the
role of the promoter which, in our group, coincides
with the management's vision of IT impact, is increas-
ingly relevant.
In brief, the main differences from other works in
the literature of the IT evaluation are the methodology
used, the diversity of IS studied and the analysis of
correspondences between the maturity levels and the
IT evaluation models. As far as other works use case
studies or surveys, we apply the cluster analysis in
order to form and process the information. The matur-
E.H. Arribas, P.J.S. Inchusta / Information & Management 36 (1999) 151±164 161
Fig. 2. Fundamental characteristics of other case studies.
162 E.H. Arribas, P.J.S. Inchusta / Information & Management 36 (1999) 151±164
ity level is more uniform in other studies in as much as
they reach companies where the IT impacts and aims
are very homogeneous, or automation, or aid decision
making or strategic; in our work, the group is hetero-
geneous. Finally, we have con®rmed a strong corre-
spondence between the IT maturity levels and the IT
evaluation methodologies.
5. Summary and conclusions
The questions put forward at the outset regarding
different aspects of IT evaluation have been dealt with
throughout this article and we will now conclude by
summarizing our observations.
Firstly, regarding the methodological aspects, we
have seen that when there is an abundance of quali-
tative information and the number of cases is con-
siderable, the cluster analysis is useful for gathering
companies together in groups with similar character-
istics.
Secondly, using this tool we have identi®ed four
maturity groups: strategic, supporting decision mak-
ing, automation and initial. The strategic maturity
group is characterized by the involvement of IT in
company strategy, by the management's role as pro-
moter in the introduction of IT and by the competitive
advantages of IT sought in relation to the customer. In
the IS group to aid decision making, IT is organized so
as to facilitate the internal working of the company,
support interconnection and spread the information
necessary to make decisions. The group containing
systems for automation uses IT without integrating it
into the organizational strategy and seeks to mechan-
ize tasks in order to achieve ef®ciency in the produc-
tion of information. However, 10 of the companies
studied display a low level of maturity, only three at
the strategic level and seven at that supporting deci-
sions.
Thirdly, we have also formed four groups regarding
evaluation typologies: strategic, formal-®nancial and
those of necessity and cost. The primary consideration
of strategic evaluation are the strategic values in the
long term; on the other hand, ®nancial evaluation uses
exclusively standard ®nancial techniques in the quan-
ti®cation of bene®ts. The so called necessity and cost
evaluation groups cannot be entirely considered eva-
luation inasmuch as they only take into account part of
the cost/bene®t binomial: the availability of ®nancial
resources to carry out investment or the attainment of
IT at the lowest price. Thirteen out of 20 companies
still use low level evaluation methodology although
®ve carry out strategic evaluation and two using
standard ®nancial methods.
Fourthly, we can conclude that within the group
studied a positive correspondence exists between the
level of strategic maturity and the strategic evaluation
of IT; between the level of DSS maturity and necessity
evaluation; and ®nally, between the systems for auto-
mation and cost evaluation. In the rest of the mod-
alities the independence, rejection or non-existence of
correspondence can be maintained. It is worth point-
ing out that the standard ®nancial techniques are of
little in¯uence and do not correspond exactly to any
speci®c context. This is justi®ed by the opinion of
eleven companies out of 20 that IT evaluation is
different from the evaluation of other investments.
Fifthly, in comparing our conclusions with those of
other studies, we have con®rmed differences exist
when it is used a survey methodology but not used
as case study. The Golden (1996) and Bacon (1992)
survey method reveals a signi®cant use of standard
®nancial techniques, reaching almost 70%, while in
the case study this use is seen as very much a minority.
Farbey quotes four out of 16 cases which use ®nancial
techniques and in our work we discovered two out of
20. With regard to the case studies, all agree that the
general level of IT maturity in companies is low and
that the evaluation methodologies used are also rudi-
mentary. This means that the theoretic and strategic
effects of the IT are not yet known by companies and,
subsequently, they are not considered in the evaluation
process. This con®rms the dichotomy between the
theory and the practice.
Finally, we are aware that the size of the sample is
clearly insuf®cient to be representative and to general-
ize the conclusions. So, for the immediacy future
research, we will study the maturity level of IT and
the evaluation methodology used by a representative
sample of Spanish companies of the service sector.
References
[1] C.J. Bacon, The use of decision criteria in selecting IST
investments, MIS Quarterly 16(3), 1992, pp. 335±349.
 E.H. Arribas, P.J.S. Inchusta / Information & Management 36 (1999) 151±164
[2] J.A. Ballantine, P. Powell, R.D. Galliers, Daring To Be
Different: Capital Appraisal and Technology Investments, in:
G. Doukidis, R. Galliers, T. Jelasi, H. Hremar, F. Land (Eds.),
Presented at European Conference On Information Systems
3rd, Athens, Greece, June, 1±3, 1995, pp. 87±97.
[3] J.J. Baroudi, W.J. Orlikowsli, The problem of statistical
power in MIS research, MIS Quarterly 13(1), 1989, pp. 87±
106.
[4] I.D.K. Benbasat, The case research strategy in studies of
information systems, MIS Quarterly 11(3), 1987, pp. 369±
386.
[5] A. Brown, Appraising Intangible Bene®ts From I.T. Invest-
ments, in: A. Brown, D. Remenyi (Eds.), Presented at
European Conference On IT Investment Evaluation, 1st,
Henley on Thames, Operational Research Society, Birming-
ham, England, 1994, pp. 187±199.
[6] G. Canavos, Probabilidad y EstadõÂstica: Aplicaciones y
MeÂtodos, McGraw-Hill, MeÂxico, 1988.
[7] J.G. Davis, J.R. Hamann, In Context Information Systems
Assessment: A Proposal and An Evaluation, in: N. Bjorn-
Andersen, G.B. Davis (Eds.), In Information Systems
Assessment: Issues And Challenges, North-Holland, The
Netherlands, 1988, pp. 283±296.
[8] R.M.H. Deitz, Deciding IT-Investment: Multiple Case
Description, Current Practice and Implication, in: A. Brown,
D. Remenyi (Eds.), Presented at European Conference On IT
Investment Evaluation, 1st, Henley on Thames, Operational
Research Society, Birmingham, 1994, pp. 79±89.
[9] M. Earl, Management Strategies for Information Technology,
Prentice-Hall, New Jersey, 1989.
[10] B. Esco®er, J. Pages, Analyses Factorielles Simples et
Multiples Objectifs, Methodes et Interpretations, Dunod,
ParõÂs, 1988.
[11] B. Farbey, F. Land, D. Targett, A Taxonomy of Evaluation
Methods, in: A. Brown, D. Remenyi (Eds.), Presented at
European Conference On IT Investment Evaluation, 1st,
Henley on Thames, Operational Research Society, Birming-
ham, UK, 1994, pp. 44±70.
[12] B. Farbey, Evaluating investments in IT, Journal of
Information Technology 7, 1992, pp. 109±122.
[13] B. Farbey, F. Land, D. Targett, IT Investment, A Study Of
Methods And Practice, Butterworth-Heinemann, Oxford,
1993.
[14] G.G. Gable, Integrating case study and survey research
methods: An example in I.S, European Journal of Informa-
tion Systems 3(2), 1994, pp. 112±126.
[15] R.D. Galliers, IT strategies: Beyond competitive advantage,
The Journal of Strategic Information Systems 2(4), 1993, pp.
283±292.
[16] R.D. Galliers, S.K. Somogyi, From Data Processing to
Strategic IS: A Historical Perspective, Toward Strategic
Information Systems, 5±25, Abacus Press, 1987.
[17] R.D. Galliers, Choosing Appropiate Inf Systs. Research
Approaches: A Revised Taxonomy, in: E. Nissen, K. Klein,
R. Hirschheim (Eds.), In Information Systems Research:
Contemporary Approach and emergent traditions, North-
Holland, Amsterdam, 1991, pp. 327±343.
163
[18] W. Golden, C. O¯aherty, Information Technology Invest-
ments Appraisal in Irish Organizations, in: J.D. Coelho, T.
Jelassi, W. Konig, H. Kremar, R. O'Callaghan, M. Saaksjar-
vi, Presented at European Conference On Information
Systems, 4th, Lisbon, Portugal, 2±4 July, 1996, pp. 283±296.
[19] Y. Grandes, E. Abascal, Fundamentos y TeÂcnicas de
InvestigacioÂn Comercial, Esic, Bilbao, 1995.
[20] C. Grif®ths, L. Willcocks, Are Mayor I.T. Projects Worth
The Risk? in: A. Brown, D. Remenyi (Eds.) Presented at
European Conference On IT Investment Evaluation, 1st,
Henley on Thames, England, Operational Research Society,
Birmingham, UK, 1994, pp. 256±269.
[21] R. Hayes, W. Albernathy, Our way to economic decline,
Harvard Business Review, July±August, 1980, pp. 67±77.
[22] B. Hochstrasser, C. Grif®ths, Regaining Control of IT
Investment: A Handbook for Senior UK Management,
Kobler Unit, London, 1990.
[23] E. Huerta, P. SaÂnchez, Evaluation of Information Technol-
ogies: Practical Cases in Spanish Firms, in: A. Brown, D.
Remenyi (Eds.), Presented at European Conference On
Evaluation of IT, 3rd, Bath University School of Manage-
ment, England, November, 1996, Bath, UK, pp. 43±53.
[24] A.S. Lee, A scienti®c methodology for MIS case studies,
MIS Quarterly, March, (1989), 32±50.
[25] N. McBride, C. Fidler, An Interpretative Approach to
Justi®cation of Investment in Executive Information Sys-
tems, in: A. Brown, D. Remenyi (Eds.), Presented at
European Conference On IT Investment Evaluation, 1st,
Henley on Thames, Operational Research Society, Birming-
ham, 1994, pp. 16±26.
[26] D. Mirtidis, V. Serafeimidis, Evaluating I.T. Investments in
Greece, in: A. Brown, D. Remenyi (Eds.) Presented at
European Conference On IT Investment Evaluation, 1st,
Henley on Thames, Operational Research Society, Birming-
ham, 1994, pp. 167±177.
[27] D. Remenyi, B. Williams, Some aspects of methodology for
research in information systems, Journal of Information
Technology 10, 1995, pp. 191±201.
[28] W. Robson, Strategic Management and Information Systems:
An Integrated Approach, Pitman Publising, 1992.
[29] V. Serafeimidis, S. Smithson, Evaluation of IS/IT Invest-
ments: Understanding and Support, in: A. Brown, D.
Remenyi (Eds.), Presented at European Conference On IT
Investment Evaluation, 1st, Henley on Thames, Operational
Research Society, Birmingham, 1994, pp. 135±147.
[30] J.K. Shank, V. Govindarajan, Strategic Cost Analysis of
Technological Investments, Sloan Management Review,
Autumn (1992), pp. 3±51.
[31] P.J. SaÂnchez, Modelos para la EvaluacioÂn de las Inversiones
en TecnologõÂas de la InformacioÂn: Estudio de casos en
empresas espan~olas. Ph.D. Thesis, Universidad PuÂblica de
Navarra, Spain, 1997.
[32] K. Simon, P. SaÂnchez, Information Technologies in Compa-
nies in Navarra: An Exploratory Study, in: J.D. Coelho, T.
Jelassi, W. Konig, H. Kremar, R. O'Callaghan, M. Saaksjar-
vi, Presented at European Conference On Information
Systems 4th, Lisbon, Portugal, 2±4 July, 1996, pp. 943±952.
164 E.H. Arribas, P.J.S. Inchusta / Information & Management 36 (1999) 151±164
[33] P. Strassman, Information Payoff, The Free Press, New York,
1985.
[34] F. Sutherland, Some Current Practices In The Evaluation IT
Bene®ts In S. African Organizations, in: A Brown, D.
Remenyi (Eds.), Presented at European Conference On IT
Investment Evaluation, 1st, Henley on Thames, Operational
Research Society, Birminghan, UK, 1994, pp. 27±43.
[35] V.J. Symons, Evaluation Of Information Systems: Multiples
Perspectives, Ph.D. Thesis, University of Cambridge, UK,
1990.
[36] V.J. Symons, G. Walsham, The Evaluation of Information
Systems: A critique, in: R. Veryard (Ed.), In The Economics
of Information Systems and Software, Butterworth-Heine-
mann, Oxford, 1991, pp. 70±88..
[37] G. Walsham, Interpreting Information Systems in Organiza-
tions, Weley Series in Information Systems, Chichester,
1993.
[38] J. Ward, P. Grif®ths, P. Whitmore, Strategic Planning for
Information Systems, Wiley Series in Information Systems,
Chichester, 1993.
[39] N. Wilner, Justi®cation of high technology capital investment
an empirical study, The Engineering Economist, Summer
37(4), 1992, pp. 341±353.
[40] C. Wiseman, Strategy and Computers: Information Systems
as a Competitive Weapons, Dow Jones-Irwin, Homewood,
Illinois, 1985.
E. Huerta Arribas is an associate professor
at State University of Navarre, teaching
Management Economics in the Business
School. His research interests include stra-
tegic decision making and organizational
architecture.
P.J. SaÂnchez Inchusta is an assistant
professor at State University of Navarre
(Spain) teaching Management Information
Systems in Business Administration Faculty.
He is member of Association for Informa-
tion Systems (AIS) and his research interests
include evaluation of information technol-
ogy, diffusion and implementation, and its
organisational impact and change.