The Relationship Between Participation in Information Systems Planning and Development and the Achievement of Performance Criteria in Australian

Commercial Organizations That Plan Strategically for Information Systems

Donald J. Falconer and R. Alan Hodgett School of Accounting and Information Systems University of South Australia Adelaide, Australia d.falconer@unisa.edu.au Abstract Senior management in three samples of Australian organizations were surveyed. Findings relating to the involvement of various categories of participants in information systems planning and development in Australian organizations are reported. Similarities were found across different sizes of organizations in terms of patterns of staff involvement in planning and development, the establishment of performance criteria and subsequent evaluation of systems against these criteria. The study has identified associations between the involvement of various categories of persons at different stages of planning and development and the achievement of pre-established performance criteria. Keywords IS Strategic Planning, IS Policy, IS Planning Issues, IS Project Development Policies, IS Performance Assessment, IS Performance Evaluation, User Involvement, User Participation

INTRODUCTION
Much has been written about who should be involved in the various stages of information systems planning and development. Such planning and development was once left to information systems professionals. The objective of this paper is to document the results of the project as they apply to the relationship between categories of people involved in the planning and development of a typical information systems project and the achievement of pre-established performance criteria for the project. The content of this paper is part of a larger study exploring the planning, management and use of information systems in Australian businesses. The overall study surveyed 1100 Australian organizations in three samples chosen to represent the business demographics of Australia. With the cost of new systems increasing and costly failures a business reality, organizations have had to consider who should be involved and the processes of involvement. There has been much research and publication in many countries related to this issue. Too often IS professional or other managers involved in systems development do not recognize systems development as a change process (Lederer and Nath 1991). Lack of user involvement can lead to systems that do not fully meet user needs, resulting in redesign and even sabotage (Tripp 1991). Lack of top management involvement and support can result in the development of systems that fail to meet the long term needs of the organization. (Doll 1985, Rademacher 1989). As early as 1986 Ewing explained that the MIS leader of the future would need managerial skills and a business perspective. Earl (1989)

Proc. 10th Australasian Conference on Information Systems, 1999

271

suggested the development of hybrid managers who would combine business literacy with sound IS/IT technical knowledge together with a third dimension of astuteness in the making of sound decisions for the use of IT in business. Tait and Vessey (1988) related the level and quality of user involvement in systems development to systems success. Jarvenpaa and Ives (1991) implicitly accepted the view that senior executives should participate in information systems planning and management and then explored the nature of appropriate participation. Yetton (1994) discussed a "federal" model that organizations might adopt to broaden the technical base of business staff and at the same time increase the business experience of its technical staff.

THE AUSTRALIAN CONTEXT

Australian businesses generally are smaller and employ fewer people than those in countries which have been the subject of much research; for example, Lederer and Mendelow (1988) surveyed organizations with up to 125,000 employees and Earl (1993) surveyed organizations having average annual revenues of 4.5 billion. Even the largest Australian organizations are small by comparison to these. The Australian Bureau of Statistics (1993) (ABS) reports on the Australian business sector in eight employment size groupings, the largest being those businesses employing more than 1000 people. In 1993 there were 26,060 organizations with 20 - 99 employees, 4218 with 100 - 499 employees and 1090 with more than 500 employees (op. cit.). Karpin (1995), in his report to the Australian Government makes extensive reference to a general lack of management skills, including the management of information technology, in Australian businesses. The report compares Australia unfavourably in these areas with other developed countries.

THE PROJECT
This paper is part of a project undertaken as a large mail survey of Australian businesses based on a stratified sample of members supplied by the ASCPA. A total of 1100 companies in three samples of Australian companies were surveyed: 500 companies employing from 20 to 99 employees, 400 companies employing from 100 to 499 employees and 200 companies employing 500 or more. These groups include Australian owned and managed companies as well as part or completely foreign owned or controlled. The Australian Bureau of Statistics (1993) (ABS) reports on the Australian business sector in eight employment size groupings, the largest being those businesses employing more than 1000 people. In 1993 there were 26,060 companies with 20 - 99 employees, 4218 with 100 - 499 employees and 1090 with more than 500 employees (op. cit.). The ABS (1993) includes statistics for a defined business unit titled "Management Units". Our survey used a term "organisation" which was defined in the questionnaire and is a close surrogate for the ABS Management Unit. This paper addresses the following issues in relation to categories of people involved in the planning and development of a typical information systems project and the achievement of pre-established performance criteria for the project: The categories of people involved in each stage of systems planning and development in different sized organizations. The establishment of performance evaluation criteria set for information systems projects. The evaluation of completed projects against the performance criteria. The percentage of projects that failed to meet established performance criteria. The sizes of projects, in monetary terms, considered to be failures.

272

The relationship between categories of people involved and the achievement of pre-established production performance criteria.

RESULTS
The response rates for the survey for the three samples were 33.4%, 35.6% and 35.5%. This compares favourably with the response rates achieved by many recently published surveys. (c.f. Steinbart and Nath 1992, Raymond et al. 1995 and Ward et al. 1996). Any sample should be as representative as possible of the population surveyed. The sample responses fit well with the Australian business population, except for over-representation of manufacturing and under representation of community services. This is almost certainly explained by manufacturing industries employing a higher percentage of accountants than do community services companies. The qualified accountants to whom this survey was addressed were asked to give the title of the positions currently held by them. The results are summarised in Table 1 and confirm that all respondents held executive or senior management positions. Any survey must necessarily accept that respondent bias will be reflected in the data gathered. The writers believed that because the respondents held senior management positions they would have the necessary knowledge to complete the questionnaire and that their answers would be sufficiently homogeneous to allow analysis and reporting in summary form.
Table 1. Descriptions of Positions* Currently Held by Respondents
Description Large* Co's (%) 34 15 9 9 8 5 3 17 Medium** Co's (%) 42 13 4 2 5 10 6 18 Small*** Co's (%) 25 12 5 0 11 14 9 24

Finance controller (Director) Accounting manager Divisional manager Information systems manager Management accountant CEO Director Other managerial positions
*n = 66, ** n = 126, *** n = 153.

Ninety-four percent of large organizations, 90% of medium organizations and 74% of small organizations asserted that they undertake strategic planning. The results presented in this paper relate to those organizations which indicated that they undertake strategic planning. Respondents were asked to report on one project completed in their organization in the last three years. No coaching was provided to respondents. They were free to select one system on which to report from all systems completed in their organization in the last three years. The reasons for selecting the systems reported were not requested and were not provided by any respondents. This approach was used to enable data to be gathered as randomly as possible. It is possible that some undetected systematic bias is present, but the authors have no means to detect this. It is considered more likely that, given the compete freedom to choose, the data is sufficiently random for analysis. To gauge approximate project size, respondents were asked for an estimate of the total project cost and development time. As was expected, smaller organizations generally reported on smaller projects, but large projects were well represented across all organization sizes with reported development costs ranging from $50,000 to in excess of $1m. Total elapsed development time was

273

generally in the range 6 to 24 months, with a shortest time of 3 months and with one project taking up to 48 months.

Table 2. Participants in the various stages of information systems planning and development
Participants Propos ed
(%)

Planne d
(%)

Design ed
(%)

Approv ed
(%)

Implemen ted
(%)

Large companies Bd of directors Chief executive Snr mgt group Org. planning group Financial accountant Mgt accountant In-house sys specialists Funct. Area mgr Funct. Area reps User/user group Internal auditors Consultants External auditors Other Medium companies Bd of directors Chief executive Snr mgt group Org. planning group Financial accountant Mgt accountant In-house sys specialists Funct. Area mgr Funct. Area reps User/user group Internal auditors Consultants External auditors Other Small companies Bd of directors Chief executive Snr mgt group Org. planning group Financial accountant Mgt accountant In-house sys specialists Funct. Area mgr Funct. Area reps User/user group Internal auditors Consultants External auditors Other *n = 42, **n = 82, ***n = 86

0 5 29 14 0 0 0 2 0 45 0 2 0 2 4 14 40 9 2 0 1 4 0 20 1 1 0 4 8 20 37 4 4 0 1 4 0 17 0 5 0 2

0 3 29 45 0 0 7 0 2 43 0 14 0 0 1 2 45 23 1 0 5 2 0 29 0 21 0 2 2 14 36 7 8 1 2 6 0 23 0 27 0 2

0 0 0 0 43 29 83 69 36 93 21 55 10 2 0 0 0 0 55 29 56 52 29 65 7 42 5 4 0 4 2 0 40 28 29 35 12 43 0 57 2 0

14 19 43 12 12 14 29 43 0 60 0 0 0 2 17 29 54 12 29 10 31 32 0 34 1 0 0 1 22 38 31 1 24 10 9 20 0 24 0 0 0 0

0 0 12 26 17 7 57 43 0 53 0 5 0 7 2 7 31 9 29 9 56 28 0 31 0 4 0 2 1 9 26 7 26 14 20 25 0 24 0 4 0 1

Respondents were asked to report on systems proposal and development processes in their organizations based on a traditional development life cycle framework. The questions related to involvement in the planning and development processes and to the establishment and use of performance criteria. Table 2 shows the participation rates of a range of individuals and groups in the various stages of information systems planning and development in regard to the systems project reported on by survey respondents. The table shows the percentage of occurrences of each participant type at each stage, for example, in large organizations, user groups were involved in the approvals stage in 60% of cases.

274

The patterns of involvement of participants between stages in planning and development were significantly different, while the patterns of involvement of participants in each stage across the three samples was confirmed statistically to be the same. Most systems were proposed by users/user groups, organizational planning groups or senior management groups. There is an increased representation by users and user groups in the sample of large organizations in the proposal stage. There are a number of noticeably low percentages including in-house systems specialists, financial accountants, consultants and internal auditors in this stage. The table shows fewer chief executives in large organizations propose information systems project than do in medium and small organizations. CEO proposals increase as the size of the organization decreases. Presumably chief executives in smaller organizations adopt a more hands on approach to their systems. The reverse situation can be noticed for organizational information systems planning groups, where it can be assumed that the smaller the organization the less likely it is to have a formal organizational planning group. The planning stage is quite similar, except for more involvement of consultants. Senior executives and the CEO hand over to functional area managers, in-house systems specialists and user groups at the design stage. The approvals stage sees fairly equal representation from all levels within the organization. External entities, such as consultants and auditors do not, as we would expect, play a part in the approvals stage. Implementation is the province of middle level managers, users and inhouse systems specialists. Few organizations relied on consultants at this stage. Respondents were asked whether performance criteria were established for the new system. Eightyone percent of large, 72% of medium and 70% of small organizations indicated that they prepare performance criteria for the project. They were not asked for details of these criteria. The results are somewhat similar for all three organization sizes, although large organizations appear to more often set performance criteria for new systems than the medium and small organizations. It is surprising that any respondents replied in the negative to this question. Respondents from organizations that did prepare performance criteria were asked whether the completed system was evaluated against the performance criteria and how well the system met the criteria. Sixty-seven percent of large, 67% of medium and 66% of small organizations indicated that they evaluated the completed system against established performance criteria. The results are almost identical for all three organizational size groupings. It is worth noting that about one third of organizations in each group do not evaluate their new systems' performance even though some of them set performance criteria. The survey design does not allow a conclusion that this result is representative of the population of all new systems produced, but the authors believe it is persuasive evidence that many organizations are neglecting this important step in systems development. Finally, respondents who indicated that performance criteria were set were asked to provide an opinion as to how well the completed system met the pre-established performance criteria. Respondents were asked to score their opinion using a Lickert Scale, choosing between: met all criteria, met most, met some, met few or met none. Table 3 shows the responses.
Table 3. Indicators of how well the completed system met the pre-established performance criteria for the project.
Percentages of Respondents Large* Medium** Small*** Co's Co's Co's (%) (%) (%) 21 41 17 46 13 49

Met criteria Met all criteria Met most criteria

275

Met some criteria Met few criteria Not evaluated

12 0 26

6 1 30

6 0 32

*n = 42, **n = 82, ***n = 86

Of all organizations which set performance criteria, only 21 percent of large organizations, 17 percent of medium and 13 percent of small organizations reported that their new systems met all criteria and nearly half the respondents felt their new systems met most criteria. Some respondents reported that their new systems had met only some or few performance criteria. No-one reported that their system met no criteria. Once again, the results are very similar across all three sample groups, which suggests that the size of the organization is not relevant when considering issues of systems success. An investigation was made to ascertain whether the size of the project was related to the success or failure of the project. There was no relationship found, except that the smaller organizations tended to report more on smaller projects. There were two projects costing in excess of $1m which were reported as meeting few performance criteria.

Participants

Participated in Planning Process

Participated in Design Process

Participate d in Approvals Process

Managed the Implementatio n Process

Direct Relationship Financial accountants Management accountants Organizational planning group Board of directors Chief executive officer Senior management Inverse Relationship Consultants In-house systems specialists Users/User groups * indicates a direct relationship

indicates an inverse relationship.

Figure 1. Indications of relationships* between certain categories of participants in information systems planning and development processes and the achievement of preestablished performance criteria.

As the patterns of involvement of participants in each stage across the three samples was the same, the final analysis for this paper was made combining the responses from all three samples into a single sample. The results were examined to determine whether a relationship exists between the categories of people involved and the reported achievement of all or most pre-established performance criteria or the achievement of some, few or no criteria. These two broad categories were labeled success and failure. Percentages of categories of people involved in each planning and development stage were prepared for both the success and failure groups. An examination of the percentages revealed different patterns of involvement for each group. There was from 10 to 20 percent difference in the levels of involvement of some participants between the success and failure groups. The pattern of these differences is reproduced in Figure 1.

276

The upward arrows in Figure 1 represent a direct relationship and the downward arrows an inverse relationship between the participants in a particular process and the success or "failure" of the project. Cells without arrows indicate that there was no identifiable differences between the groups. Relationships which showed a difference were subjected to chi-square tests for association for each pair. A significant association was found in all bar two cases. The two pairs involving financial and management accountants contained too little data to test for association. The indicators for these two pairs have been left in Figure 1 for completeness.

SUMMARY
This survey has found similar patterns of involvement in planning and development between all three organizational-size groups, the establishment and evaluation against systems performance criteria and completion success rates. This result was unexpected. It was thought likely that organizational size would be relevant in determining an organizations approach to the planning and development of information systems. For a considerable percentage of new systems, performance criteria are not established. In cases where they are established, the resulting system is not necessarily measured against the criteria. This should be of concern to all involved in the planning and development of information systems. Educators and trainers should note carefully that the messages delivered in management training are still not being extensively put into practice. It should also be of concern to all to learn that only about 20% of surveyed systems were considered to have met all established criteria and that the majority met only most or some criteria. If this result is representative of Australian organizations in general, then many organizations are failing to maximise rewards for their investment in information technology and systems. There is an opportunity here for researchers to inquire further, perhaps on an industry-by-industry basis, to help organizations identify reasons for their systems failing and to learn from past lessons of others. All business mangers should ensure that sound management practices are followed. The identified associations between the involvement of various categories of persons at different stages of planning and development and the meeting of pre-established performance criteria require careful consideration and analysis. No reasons for or explanations of the relationships were identified in this study. Perhaps the finding that the involvement of consultants at an early stage is associated with less likelihood of a successful outcome is related to the contracting process. It might also be related to the way in which organizations perceive and develop their relationships with consultants. This finding indicates that further study into the consultant-client relationship in the planning and design stages for new information systems will be of benefit to all parties. This study offers the broad suggestion that organizations which leave the bulk of planning and development to in-house systems specialists may be less likely to enjoy success with their new system than those organizations which involve a range of their business staff. This finding supports the contentions of much of the literature and the efforts of those organizations which seek to train and develop their inhouse information systems specialists in business matters and vice versa. Perhaps an explanation for the inverse relationship between the increased involvement of users in the approval stage and likely success might be that the approval process requires business managers with a more broadly-based understanding of the business than might be held by those close to the operation of the system.

277

REFERENCES
Australian Bureau of Statistics, (1993), Profiles of Australian Business 1992, Commonwealth of Australia. Doll, W.J. (1985) Avenues for top management involvement in successful MIS development, MIS Quarterly, Spring, 1735. Earl, M.J. (1989) Management Strategies for Information Technology, Prentice Hall. Earl, M.J. (1993), Experiences in strategic information systems planning", MIS Quarterly, Mar, 1 24. Ewing, T. (1986) The future of MIS, Information Week , Dec 1, 2637. Jarvenpaa, S.L. and Ives, B. (1991) Executive involvement and participation in the management of information technology, MIS Quarterly, June, 205227. Karpin, D.S. (Chairman), (1995), Enterprising Nation, Reviewing Australias Managers to Meet the Challenges of the Asia-Pacific Century, Commonwealth Government of Australia. Lederer, A. L. and Nath, R. (1991) Managing organizational issues in information systems development, Journal of Systems Management, 42:11, 2327. Lederer, A.L. and Mendelow, A.L. (1988) Convincing top management of the strategic potential of information systems, MIS Quarterly, Dec, 525534. Raymond, L., Par, G. and Bergeron, F. (1995) Matching information technology and organizational structure: an empirical study with implications for performance, European Journal of Information Systems, 4:1. Steinbart, P.J. and Nath, R. (1992) Problems and issues in the management of data communications networks: the experiences of American organisations, MIS Quarterly, March. Tait, P. and Vessey, I. (1988) The effect of user involvement on system success: a contingency approach, MIS Quarterly, March, 91108. Tripp, R.S. (1991) Managing the political and cultural aspects of large-scale MIS projects, Information resources Management Journal. Fall, 213. Ward, J., Taylor, P. and Bond, P. (1996) Evaluation and realisation of IS/IT benefits: an empirical study of current practice, European Journal of Information Systems, 4, 214225. Yetton, P.W. (1994) False prophecies, successful practice and future directions in IT management, TC8AUS IFIP Information Systems International Working Conference, Qld, May.

COPYRIGHT
Donald Falconer and R. Alan Hodgett 1999. The authors assign to ACIS and educational and non-profit institutions a non-exclusive licence to use this document for personal use and in courses of instruction provided that the article is used in full and this copyright statement is reproduced. The authors also grant a non-exclusive licence to ACIS to publish this document in full in the Conference Papers and Proceedings. Those documents may be published on the World Wide Web, CD-ROM, in printed form, and on mirror sites on the World Wide Web. Any other usage is prohibited without the express permission of the authors.

278

279