Perspectives on Improving Software Maintenance

Frank Niessink Software Engineering Research Centre P.O.Box 424, 3500 AK, Utrecht, The Netherlands Tel: +31 30 2545412, Fax: +31 30 2545948 E-mail: niessink@serc.nl Abstract
This paper presents a summary of the PhD thesis Perspectives on Improving Software Maintenance. The research described in this thesis investigates process improvement of software maintenance from two perspectives: measurement-based improvement and maturity-based improvement.

to improve IT service processes and using maturity-based improvement to improve IT service processes. With respect to the rst issue the usage of measurement for the improvement of IT services and IT service processes the thesis addresses three research questions: 1. How to introduce measurement in an IT service organization? What are the necessary steps to set up a measurement program and in which order should they be performed? 2. What are the prerequisites that need to be satised in order to improve the likelihood of success of the measurement program? 3. What is or what should be the relation between measurement and the maturity of the IT service organization? The research has been limited to software maintenance as one type of IT service. Furthermore, we limited the possible measurement applications to the planning and estimation of software maintenance effort. The second research issue maturity-based improvement of IT service processes is divided into the following three questions: 1. What arguments can we supply to support the notion of IT service process maturity? 2. What should mature IT service organizations look like? Which processes should a mature service provider implement? 3. How can we use the concept of IT service process maturity in practice to support the improvement of IT service providers? Again, we investigated software maintenance as one possible IT service to reduce the complexity of the research. 1

1. Introduction
This paper gives a short summary of the research questions and results from the PhD thesis titled Perspectives on Improving Software Maintenance by Frank Niessink [7]. The research described in this PhD thesis was done as part of two Dutch national research projects during the 19951999 timeframe. The goal of these projects was to develop methods, techniques and tools to help organizations provide better IT services. IT services are services provided by an IT service provider to its customers to manage, maintain or operate the software and hardware used by the customer. In this PhD thesis, we mainly focused on software maintenance as one type of IT service. This summary is structured as follows. The next section presents the research questions addressed in the thesis. Section 3 describes the research dealing with measurementbased improvement of software maintenance and section 4 presents the research into maturity-based improvement of software maintenance. Finally, section 5 presents the main contributions of this PhD thesis.

2. Research questions
The research described in this thesis addresses two research issues: applying measurement-based improvement
The research described in this paper was done while Frank Niessink was at the Vrije Universiteit Amsterdam.

analysis
Possible cause Problem Goal

analysis

Improvement
Possible solution

Measurement

Measurement program Experiment

implementation

implementation

Figure 1. A generic process model for measurement-based improvement

We have conducted four measurement program case studies to investigate the rst of the three measurementbased improvement research questions [9, 12]. The lessons learned from these four measurement program case studies have been used to develop a maturity model for software measurement processes [11]. This Measurement CMM provides an ordered set of measurement processes that, in our opinion, allows organizations to assess their own measurement maturity and that provides directions for the improvement of an organizations measurement processes. We have taken a rst step towards validation by comparing our measurement maturity model with other guidelines for implementing measurement programs [1, 2, 3, 4]. In order to facilitate this comparison, we developed a simple four-phase process model of measurement-based process improvement, see gure 1. The model contains the main activities an organization has to execute to employ measurement-based improvement: problem analysis, measurement program implementation, data analysis, and organizational change to implement the desired improvement. This process model was validated against the measurement program of an organization implementing level four of the Software CMM [5]. Next, we mapped the activities of several measurement program guidelines, including our Measurement CMM, onto the process model [13]. From this comparison we draw three conclusions:

3.2. Prerequisites for successful measurement programs

The comparison of our Measurement CMM with other related work led us to investigate external success factors for measurement programs, in addition to the well-known internal success factors, such as presented by Hall and Fenton [4]. Internal success factors are factors that inuence the correct implementation of the measurement program itself. External success factors inuence the usage of the measurement program results to improve the organization. We have identied four external success factors [14]: 1. The various assumptions underlying the measurement program should be made explicit. It should be decided if and when these assumptions are tested. 2. Different outcomes can result from a measurement program. An organization should consider all possible negative and positive outcomes and decide how to act on them. 3. The organization should act according to the outcomes of the measurement program, in order to reach the goals set or solve the problems identied.

3.1. How to implement measurement programs?

In the following three subsections we discuss the three measurement research questions for measurement-based improvement.

3. Measurement-based improvement

There is quite some consensus on the basic activities needed to successfully implement measurement programs, but at the same time, different frameworks emphasize widely different aspects of measurement program implementation, and, there is almost no consensus on, nor description of, activities needed to successfully use the results from measurement programs.

4. The organization should monitor the changes implemented, in order to verify that these changes indeed constitute an improvement for the organization. In addition, we have provided guidelines which could be used by organizations to adhere to these external success factors. The combination of the internal success factors and our four external success factors covers all four phases of the measurement-based improvement process model. With help of both the internal and external success factors we were able to explain the success or failure of the four measurement programs investigated [17]. Hence, these two sets of success factors provide an answer to the second research question: what are the prerequisites that need to be satised in order to improve the likelihood of success of the measurement program?

that cause differences between the service quality as expected by the customer and as perceived by the customer. We focused especially on software maintenance as one type of IT service. Hence, the arguments given lead to the conclusion that if we view software maintenance from a service perspective, other processes are needed for high maturity software maintenance organizations than those provided by the Software CMM. Or, put differently, high process maturity means different things for software development and software maintenance organizations [6, 16]. We have not by any means proven that the notion of IT service process maturity exists. However, we have argued that, if we accept that process maturity is a useful instrument to support process improvement activities, then the processes needed for a high maturity IT service provider are different from the processes needed for a high maturity software developer.

3.3. The relationship between measurement and process maturity

The last of the three questions concerns the relationship between process maturity and the success of measurement programs. We have found some indications that a more mature organization has a better possibility of a successful measurement program. However, there is no solid evidence. Moreover, we did not formally determine the maturity of the four organizations involved in the measurement program case studies. Whether a causal relationship exists, and if so, what the size of the effect is, remains an open question.

4.2. Mature IT service processes

To further investigate the differences between software product development and IT service provision we have developed an IT Service Capability Maturity Model [8, 10, 15]. This IT Service CMM was designed to capture the issues described in the previous section. The IT Service CMM includes the four processes identied using the service marketing literature that organizations can apply to help close the four service gaps.

4.3. IT service process maturity applied

We took some rst steps in applying the IT Service CMM in practice by doing two case studies in which the IT Service CMM was used as a reference framework for process assessment.

4. Maturity-based improvement
In this section we discuss the three maturity-based improvement research questions.

4.1. The notion of IT service process maturity

The rst of three research questions concerned the concept of IT service process maturity. We discussed the concepts of services and products, and we applied work from the service marketing literature to the software engineering domain, specically to software maintenance. Starting from the argument that the quality of service is judged differently from the quality of products, we provided arguments for the conjecture that, because of these differences, software maintenance organizations need different processes to produce high quality results than software development organizations need. We described several case studies to indicate what kind of processes that could be. These processes are aimed at supporting the functional quality of the services delivered, and at narrowing the organizational gaps

5. Conclusions
We have investigated four measurement program case studies. From these four case studies we learn several lessons with respect to success and failure factors of measurement programs. We have introduced an abstract process model of measurement-based process improvement. We have used this process model to compare different guidelines and frameworks for implementing measurement programs. The comparison reveals that these guidelines agree on the basic activities needed for successful measurement, but at the same time emphasize different aspects. In addition, the usage of the abstract process model shows that these guidelines tend to ignore the application of measurement results. We conclude that the consensus success factors for measurement programs as found in the literature are

necessary but not sufcient preconditions for the successful implementation of measurement programs. These, what we call internal success factors, need to be complemented with external success factors that are aimed at securing that measurement programs generate value for the organization. To this end, we have proposed four external success factors and suggested a number of activities that can be used to adhere to these external success factors. Investigating maturity-based improvement of IT services, and software maintenance in particular, has resulted in a new perspective on software maintenance: namely software maintenance as a service. Viewing software maintenance as a service implies several consequences for the way in which customers will judge the quality of software maintenance, and hence it has consequences for the processes that are key to delivering high quality software maintenance. We have described these consequences in a capability maturity model for IT services and we have shown some preliminary experiences with applying the IT Service CMM in the assessment of two organizations that provide IT services.

References
[1] L. C. Briand, C. M. Differding, and H. D. Rombach. Practical Guidelines for Measurement-Based Process Improvement. Software Process Improvement and Practice, 2(4):253280, Dec. 1996. [2] P. Comer and J. Chard. A measurement maturity model. Software Quality Journal, 2(4):277289, Dec. 1993. [3] M. K. Daskalantonakis, R. H. Yacobellis, and V. R. Basili. A Method for Assessing Software Measurement Technology. Quality Engineering, 3:2740, 1990-1991. [4] T. Hall and N. Fenton. Implementing Effective Software Metrics Programs. IEEE Software, 14(2):5565, March/April 1997. [5] B. t. Horst, F. Niessink, and H. van Vliet. Implementing a Quantitatively Controlled Software Process. In R. Kusters, A. Cowderoy, F. Heemstra, and E. van Veenendaal, editors, Proceedings of the combined 10th European Software Control and Metrics Conference (ESCOM) and the 2nd SCOPE conference on Software Product Evaluation, pages 167175, Herstmonceux, England, April 27-29, 1999. Shaker Publishing. [6] B. A. Kitchenham, G. H. Travassos, A. von Mayrhauser, F. Niessink, N. F. Schneidewind, J. Singer, S. Takada, R. Vehvilainen, and H. Yang. Towards an ontology of software maintenance. Journal of Software Maintenance: Research and Practice, 11(6):365389, November/December 1999. [7] F. Niessink. Perspectives on Improving Software Maintenance. PhD thesis, Division of Mathematics and Computer Science, Faculty of Sciences, Vrije Universiteit Amsterdam, the Netherlands, Mar. 2000. Available from http://www.niessink.com/. [8] F. Niessink. The IT Service CMM website, July 2001. http://www.itservicecmm.org.

[9] F. Niessink and H. van Vliet. Predicting Maintenance Effort with Function Points. In M. J. Harrold and G. Visaggio, editors, International Conference on Software Maintenance, pages 3239, Bari, Italy, October 1-3, 1997. IEEE Computer Society. [10] F. Niessink and H. van Vliet. Towards Mature IT Services. Software Process Improvement and Practice, 4(2):5571, June 1998. [11] F. Niessink and H. van Vliet. Towards Mature Measurement Programs. In P. Nesi and F. Lehner, editors, Proceedings of the Second Euromicro Conference on Software Maintenance and Reengineering, pages 8288, Florence, Italy, March 811, 1998. IEEE Computer Society. [12] F. Niessink and H. van Vliet. Two Case Studies in Measuring Software Maintenance Effort. In T. M. Khoshgoftaar and K. Bennett, editors, International Conference on Software Maintenance, pages 7685, Bethesda, Maryland, USA, November 16-20, 1998. IEEE Computer Society. [13] F. Niessink and H. van Vliet. A Pastry Cooks View on Software Measurement. In R. Dumke and A. Abran, editors, Software Measurement - Current Trends in Research and Practice, pages 109125, Wiesbaden, Germany, 1999. Deutscher Universit ts Verlag. a [14] F. Niessink and H. van Vliet. Measurements Should Generate Value, Rather Than Data. In Proceedings of the Sixth International Software Metrics Symposium, pages 31 38, Boca Raton, Florida, USA, November 4-6, 1999. IEEE Computer Society TCSE, IEEE Computer Society Press. [15] F. Niessink and H. van Vliet. The IT Service Capability Maturity Model. Technical Report IR-463, Division of Mathematics and Computer Science, faculty of Sciences, Vrije Universiteit Amsterdam, 1999. Model version L2-1.0. [16] F. Niessink and H. van Vliet. Software maintenance from a service perspective. Journal of Software Maintenance: Research and Practice, 12(2):103120, March/April 2000. [17] F. Niessink and H. van Vliet. Measurement Program Success Factors Revisited. Information and Software Technology, 43(10):617628, Aug. 2001.