A FRAMEWORK FOR MIS EFFECTIVENESS RESEARCH

Jamal Munshi, Sonoma State Univesity

All rights reserved

INTRODUCTION

Information systems are costly to purchase, deploy, and maintain.

Therefore, in a world where business enterprise is operated for value
maximization according to the theory of rational choice, it is natural to
suppose that MIS offers economic value and that this value overcomes
the costs. As such it has rightly been an objective of MIS research for at
least two decades to determine the economic role of MIS. Today
information system effectiveness continues to occupy the highest priority
of the MIS research agenda.

Yet much of the research on the effectiveness or success of information

systems has relied on psychometric measures of user satisfaction also
referred to as 'user information satisfaction' or UIS. These studies define
MIS success narrowly in only subjective terms. In a recent paper Melone
[14] argued that the UIS construct is inadequate as a surrogate of
effectiveness and that UIS questionnaire construction and methodology
are lacking in scientific rigor. MIS researchers have failed to use the
accumulated knowledge and practices of their reference disciplines.

An alternative measure offered by some is system usage which is based

on the principle that if the system is being used it must be effective and
the more it is used the greater the effectiveness. Srinavasan [15] argues
for usage and develops a correlation between usage and satisfaction.
More recently Barki and Huff [2] have combined usage and satisfaction
scores as dependent variables in an effort to measure success of DSS
implementations.

A more ambitious proposal is to define the effectiveness of the

information system as the degree to which the business goals, for which
the IS was deployed, are actually achieved. Although these ideas are
appealing from a viewpoint of rational objectivism, the difficulty of
defining and measuring such variables has forced researchers to once
again resort to questionnaires. The end result is that MIS effectiveness is
determined not in terms of observations made by the researcher but in
terms of an average of opinions of users, developers, and managers [6].

While satisfaction scores and opinions can be useful in comparing two

systems within the same user/developer community, they make inter-firm
comparisons difficult. These difficulties convinced many researchers to
abandon the questionnaire approach and make direct observations of
accounting and economic variables that would be indicative of the
achievement of business goals [1] [5].

However, at the firm and industry levels, these studies have been
frustrated by the concomitant effect of intervening and extraneous
variables that are much more powerful than the information system itself.
That is, the so called 'bottom line' of the business enterprise is so greatly
affected by economic and market conditions that only a small amount of
the variance may be attributed to the information system being tested.
Also, some organizations are more efficient than others in utilizing
information systems. Thus, the component of the total variance
contributed by information systems may not be easily detected in large
cross-sectional studies. The lack of statistical power possibly played a
role in the inconclusive nature of the results reported by Chismar and
Kriebel [4].

The problem with statistical power may be minimized if the study is

confined to a specific application or implementation or if it is confined to
a controlled laboratory experiment. The study of a sales information
system by Lucas [12] is an example of the former while the analysis of a
simulated strategic information system by King and Rodriguez [10] is
exemplary of the latter.

But what these studies gain in internal validity they lose in the ability to
externalize the findings to the 'real world'. Findings of the success of a
single implementation are usually viewed as anecdotal and do not lead to
convincing statements of a general nature; and the laboratory
environment such as one where hypothetical business problems are
solved by MBA students lack the realism of a business setting.

Some of these problems are overcome by the use of scientifically

designed case studies as described by Lee [11]. In these studies, a
carefully defined hypothesis is tested in a real-life situation in which
'fortuitous' circumstances allow rigorous control of concomitant
variables. If there is no error of measurement the hypothesis can be
rejected if it is violated by a single observation. A successful study of this
nature has been reported by Markus [13].

However the economic measure of IS effectiveness as the "conversion of

IS investment into real output" [16] is itself in question in organizational
paradigms where politics and conflict are important motivating factors.
Markus [13] found "re-distribution of power" to be an important impact
of IS implementation while other researchers have described the roles
played by leadership and managerial control issues in IS implementation
decisions.

The measurement of IS effectiveness at the firm level as proposed by

Crowston and Treacy [5] and Bakos [1] necessarily require that the
researcher have a theory of the firm that precludes conflict such as that
caused by agency issues [8]. These theories allow for managerial
behavior that is not consistent with profit or value maximization. Since
IS implementation decisions are made by managers, managerial behavior
and motivation are likely to be important variables in the study of IS
effectiveness.

It therefore appears that neither a purely subjective paradigm nor a purely

functional paradigm of the firm is adequate for understanding the
motivations for and the effectiveness of the deployment of information
system. There is a wide spectrum within which the concept of
effectiveness may be interpreted and the dimensions in which such an
interpretation may be placed. Definitional and measurement issues have
retarded the orderly and scientific accumulation of knowledge in this
field.

The purpose of this paper is to present a multi-dimensional 'framework'

of IS effectiveness that can accommodate many of these apparently
disjointed and conflicting views. It is hoped that the adoption of such a
framework will foster a cumulative tradition in effectiveness research in
which the complementary role of individual research efforts is
emphasized and the frictional losses of conflict are reduced.

Three dimensions of effectiveness are identified and explored in this

paper. These are the dimension of scope, the dimension of measurement,
and the dimension of social paradigm. The dimensions are first described
without reference to the placement of previous research within the
'effectiveness space' they define. An analysis of past research within this
framework is presented in the full paper available from the author.

THE DIMENSIONS OF EFFECTIVENESS

The dimension of scope describes how broadly the concept of

effectiveness is to be applied. In the narrowest sense it is applied to a
single implementation of a specific application program. This can be
expanded to include multiple implementations of the same program or to
an entire class of applications. We can refer to these as the application
level and we may interpret the effectiveness measure in terms of the
design, usability, and usefulness of the application in question.

An enhanced sense of generality can be realized by considering the

impact of IS on an entire firm regardless of application. At the firm level
the effectiveness measure can be related to the firm's MIS organization,
policy, budget, as well as attitudes and opinions. This generality can be
further increased by considering an entire class of firms in a
homogeneous line of business. An industry or in a broader sense, an
entire sector of the economy can be included in what can be termed the
industry level. At the economy level the impact of information
technology on the entire economy can be assessed as has been done by
Jonscher [9] in his famous paper. At the extreme generality of scope, the
society level, sociologists may consider the impact of information
technology on society at large.

The dimension of measurement addresses the type of data to be collected,

the method of their collection, and the manner of their interpretation.
This dimension can be broadly divided into two parts - direct
observations of business variables and psychometric measures of attitude
and behavioral variables. Psychometric measures are made by
constructing questionnaires designed to assess attitudes and opinions on
various types of ordinal scales. The psychometric measures can be
further sub-divided into two groups; those that measure attitudes (the
'perceptual level') and those that elicit opinions.

The perceptual level lies at the lowest end of the measurement dimension
in terms of objectivity. These questionnaires measures the subjective
evaluation of users, managers, and IS designers. The user information
satisfaction studies can be described by the properties of this level. The
influence of subjectivity is lower in the opinion level in which the
questions in the instrument are meant to gather data about behavior such
as IS utilization and business performance rather than attitude.

A much higher level of objectivity is attained when the researcher

directly observes variables that measure utilization and performance
instead of eliciting opinions about them. 'Utilization' is the extent to
which the IS is used by the intended users for the intended purpose.
'Performance' is the measure of improvement of the business function
supported by the IS.

The field observation level of utilization and performance measures are

obtained from business records or through visual or electronic inspection
and are therefore independent of personal opinions and attitudes. The
variables to be observed, however, may be difficult to identify and their
causal connection to the MIS may be questioned. This is because of the
presence of powerful intervening variables that relate to accounting,
marketing, economics, and managerial strategy and efficiency. These
problems are avoided at the highest level of measurement, the controlled
experiment. Although the experiment offers the highest degree of
precision, both the scope and the ability to extend the results beyond the
controlled environment of the experiment are limited.

The third dimension of IS effectiveness uses the organizational

paradigms described by Burrell and Morgan [3] and adapted to
information systems by Hirschheim [7]. This dimension adds generality
to the analysis by allowing for the possibility that not all activities of
business enterprises are interpretable as if they were rational
organizations objectively seeking to maximize the wealth of the owners.
Other paradigms of business organizations exist in which managers may
take action to increase their utility rather than the owners' wealth and
where various degrees of conflict exist between managers, workers, and
stakeholders.

The four paradigms presented by Hirschheim may be placed along a

single dimension according to the extent to which they approach the
normative model of rational choice. These are, from the lowest level to
the highest, Neohumanism, Social Relativism, Radical Structuralism, and
Functionalism.

Functionalism is characterized by a high degree of order and objectivity

and satisfies the normal assumption of rational choice. In this scenario,
the MIS supports rational decision makers who arrive at optimal
decisions for the firm and seek to increase valuation through objective
and scientific means. The effectiveness of the MIS can therefore be
correctly assessed by performance measures of the business or the so
called 'bottom line' as suggested by Crowston and Treacy. Such measures
of performance, however, may not be applicable in other paradigms.

Radical structuralism, for instance, would allow the managers, workers,

and owners to each have their own separate agenda. Although each group
attempts to maximize utility according to rational choice and objectivity,
their goals are allowed to be in conflict. Agency theory and the theory of
corporate control put forth by Jensen and Meckling [8] and others are
consistent with radical structuralism as is the important finding by
Markus [13] of the role of 'politics' in IS implementation success. In
firms where radical structuralism is important, performance measures
that subsume only functionalism would reveal only a part of the overall
impact of information systems. For example, managers may deploy
information systems to extend managerial control at the expense of the
wealth of the owners and the satisfaction of the workers. If such were the
case, the effectiveness of the system may not be assessed without an
objective measure of the additional control made possible by the system.

Social relativism and neohumanism emphasize subjective experiences of

individuals rather than the objective goals of groups . These two
organizational paradigms differ with respect to the degree of order and
conflict. Social relativism combines order and subjectivity. Each
individual is viewed as working toward company objectives for the
common good but in his own way. Users create their own reality and
MIS helps them create new realities. In neohumanism, divergent
individual goals create conflict. The IS supports the discourse between
individuals who are viewed as idealists seeking radical change. The
effectiveness of IS within the subjective paradigms may be measured best
with psychometric tests of attitude, interests, and opinions such as user
information satisfaction.

This description of paradigms is not meant to imply that any real

organization can be characterized by a single paradigm but only to
suggest that these concepts can help us identify the components of IS
effectiveness in a broad sense.

Figures 1 and 2 are graphical depictions of the IS effectiveness

framework described above. In each figure, rectangles are drawn to
encompass regions of research activity.

FIGURE 1

A FRAMEWORK FOR THE DEFINITION OF MIS EFFECTIVENESS

SHOWING THE SCOPE AND MEASUREMENT DIMENSIONS

A FRAMEWORK FOR THE DEFINITION OF MIS EFFECTIVENESS

SHOWING THE SOCIAL PARADIGM AND MEASUREMENT
DIMENSIONS

CONCLUSIONS

The framework of IS effectiveness presented may be used not only to

consolidate past research but to plan future research. It is hoped that such
a structured approach will prove a more constructive way in which
effectiveness research can build a cumulative tradition and lay a
foundation of knowledge. The ultimate aim is to construct a theory of the
firm from an IS perspective that can serve as a common point of
reference for research in information systems.
REFERENCES

[1] Bakos, J. Yannis, "Dependent Variables for the Study of Firm and
Industry Level Impacts of Information Systems", Proceedings of the Fifth
International Conference on Information Systems, 1985, pp. 10-23

[2] Barki, H. and S.L. Huff, "Implementing Decision Support Systems:

Correlates of User Satisfaction and System Usage", INFOR, Vol. 28 No.
2, May 1990, pp. 89-101

[3] Burrell, G. and G. Morgan, "Sociological Paradigms and

Organizational Analysis", Heinemann Press, London, 1979.

[4] Chismar, William G. and Charles H. Kriebel, "A Method for

Assessing the Economic Impact of Information Systems Technology on
Organizations", Proceedings of the Fifth International Conference on
Information Systems, 1985, pp. 45-56

[5] Crowston, Kevin and Michael E. Treacy, "Assessing the Impact of

Information Technology on Enterprise Level Performance", Proceedings
of the Sixth International Conference on Information Systems, 1986, pp.
299-310

[6] Gallagher, Charles A., "Perceptions of the Value of a Management

Information System", Academy of Management Journal, Vol. 17 No. 1,
1974, pp. 46-55

[7] Hirschheim, Rudy and Heinz K. Klein, "Four Paradigms of

Information System Development", Communications of the ACM, Vol.
32 No. 10, October 1989, pp. 1199-1215

[8] Jensen, Michael C. and William H. Meckling, "Theory of the Firm:

Managerial Behavior, Agency Costs, and Ownership Structure", Journal
of Financial Economics, Vol. 3, 1976, pp. 305-360

[9] Jonscher, C., "Information Resources and Economic Productivity",

Information Economics and Policy, 1983, pp. 13-35

[10] King, William R. and Jaime I. Rodriguez, "Evaluating Management

Information Systems", MIS Quarterly, September 1978, pp. 43-51

[11] Lee, Allen S., "A Scientific Methodology for MIS Case Studies",
MIS Quarterly, March 1989, pp. 33-50

[12] Lucas, Henry C., "Performance and the Use of an Information

System", Management Science, Vol. 21 No. 4, April 1975, pp. 908-918

[13] Markus, M.L., "Power, Politics, and MIS Implementation",

Communications of the ACM, June 1983, pp. 430-444

[14] Melone, Nancy Paule, "A Theoretical Assessment of the User

Satisfaction Construct in Information Systems Research", Management
Science, Vol. 36 No. 1, January 1990, pp. 76-91

[15] Srinavasan, A., "Alternative Measures of System Effectiveness:

Associations and Implications", MIS Quarterly, September, 1985, pp.
243-253

[16] Weill, Peter, and Margrethe H. Olson, "Managing Investment in

Information Technology: Mini Case Examples and Implications", MIS
Quarterly, March 1989, pp. 2-11.