IEEE TRANSACTIONS ON SOFTWARE ENGINEERING,

VOL. 31,

NO. 8,

AUGUST 2005

679

The Impact of Institutional Forces on Software Metrics Programs

Anandasivam Gopal, Tridas Mukhopadhyay, and M.S. Krishnan
AbstractSoftware metrics programs are an important part of a software organizations productivity and quality initiatives as precursors to process-based improvement programs. Like other innovative practices, the implementation of metrics programs is prone to influences from the greater institutional environment the organization exists in. In this paper, we study the influence of both external and internal institutional forces on the assimilation of metrics programs in software organizations. We use previous case-based research in software metrics programs as well as prior work in institutional theory in proposing a model of metrics implementation. The theoretical model is tested on data collected through a survey from 214 metrics managers in defense-related and commercial software organizations. Our results show that external institutions, such as customers and competitors, and internal institutions, such as managers, directly influence the extent to which organizations change their internal work-processes around metrics programs. Additionally, the adaptation of work-processes leads to increased use of metrics programs in decision-making within the organization. Our research informs managers about the importance of management support and institutions in metrics programs adaptation. In addition, managers may note that the continued use of metrics information in decision-making is contingent on adapting the organizations work-processes around the metrics program. Without these investments in metrics program adaptation, the true business value in implementing metrics and software process improvement will not be realized. Index TermsProduct metrics, process metrics, software engineering, metrics programs, software development, institutional forces, metrics adaptation, metrics acceptance.

1 INTRODUCTION
implement an information system for collecting, analyzing, and disseminating measures of software processes, products, and services [3]. A primary objective of a software metric is to quantitatively determine the extent to which a software process, product, or project possesses a certain attribute [22]. Additionally, the metrics program is responsible for analyzing the metrics data from the development activities and providing stake-holders with the appropriate feedback. The metrics provided are used by the organization to focus on key process areas that need improvement in the organization. The business value of a metrics program is considerable in firms that have implemented them successfully, such as Motorola [10] and Contel Corporation [34]. However, anecdotal evidence shows that the successful adoption and implementation of these programs is limited. Pfleeger [34] reports that two out of three metrics programs fail in the first two years. Another study reported that, though software managers believe in the value from metrics programs, fewer than 10 percent of the industry classified their attempt at implementing a metrics program as positive [10]. Prior research has also identified low incidence of successful metrics programs as being due to organizational and managerial problems rather than short-comings in the definitions of software metrics [3]. It is thus important for software managers to understand what constrains the successful assimilation of these programs into their organizations. Past research has studied the determinants of success in establishing and maintaining metrics programs in software organizations [10], [34]. However, there is a lack of empirical evidence from the software industry as to what factors lead to successful implementation of metrics
Published by the IEEE Computer Society

HE management of software development is an integral part of industry today but most software organizations face significant barriers in managing this activity. An InformationWeek survey found that 62 percent of their respondents feel that the software industry has trouble producing good quality software [17]. Losses due to inefficient development practices lead to inadequate quality that cost the US industry approximately $60 billion per year [41]. One approach that has been shown to result in improved quality and reduced costs is the use of software process improvement activities. Investments in process improvement activities lead to improved quality and reduced rework and lifecycle costs [25], [14]. One of the important determinants of success in software process improvement is the presence of metrics programs. Indeed, the availability of reliable and accurate metrics has been highlighted in prior research [35]. In this paper, we focus on this important antecedent of process improvement initiativessoftware metrics programs. A metrics program is defined as the set of on-going organizational processes required to define, design, and

. A. Gopal is with the Robert H. Smith School of Business, University of Maryland, College Park, MD 20742. E-mail: agopal@rhsmith.umd.edu. . T. Mukhopadhyay is with the Tepper School of Business, Carnegie Mellon University, Schenley Park, Pittsburgh, PA 15217. E-mail: tridas@andrew.cmu.edu. . M.S. Krishnan is with the Ross School of Business, University of Michigan, Ann Arbor, MI 48109. E-mail: mskrish@umich.edu. Manuscript received 16 July 2004; revised 25 Mar. 2005; accepted 21 June 2005; published online 12 Aug. 2005. n. Recommended for acceptance by A. Anto For information on obtaining reprints of this article, please send e-mail to: tse@computer.org, and reference IEEECS Log Number TSE-0143-0704.
0098-5589/05/$20.00 2005 IEEE

680

IEEE TRANSACTIONS ON SOFTWARE ENGINEERING,

VOL. 31,

NO. 8,

AUGUST 2005

programs in organizations. In particular, it is necessary to understand that the process of successful adoption and implementation of metrics programs is not a binary yes/no decision but represents a series of organizational moves toward the optimal use of metrics programs, as discussed by Berry and Jeffery [3]. In this paper, we provide a multidimensional view of metrics program implementation in organizations. In order to provide a clear framework to the process by which metrics programs are assimilated into organizations, we use the six-stage innovation diffusion model proposed by Cooper and Zmud [8]: Initiation, the first stage, consists of active or passive scanning of the organizational problem and the opportunities available in the external environment to address the problem. . Adoption, the second stage, involves mobilizing the organizational and financial resources to reach the decision to formally adopt the process innovation. . Adaptation, the third stage, involves the process being developed and implemented. Organizational procedures are revised to fit the innovation and personnel are trained in the new process. . Acceptance, the fourth stage, consists of actively inducing usage of the process innovation in decision-making. . Routinization is the fifth stage wherein the process innovation is part of every-day routine and is seen as normal activity. . Infusion, the final stage, involves building valueadded processes to gain the maximum leverage out of the process innovation. Past research has indicated that the mere adoption of an innovation, as seen in the six-stage model above, does not lead to its routine use within the organization. Adoption is only the second stage of the model. Further resources and commitment is required on the part of the organization to gain the maximum benefits from the innovation. Prior work in metrics programs has discussed these issues in the implementation of metrics programs. For example, Pfleeger [34] states that the mere presence of a metrics program does not add value unless software managers can be motivated to use metrics information in their decision-making. Similarly, Berry and Jeffery [3] state that the socio-technical context that metrics programs exist in need to be studied to determine how metrics programs implementations succeed. We believe that using the above-mentioned six-stage model will enable us to provide with an opportunity to study metrics programs success at a greater level of granularity. In addition, it will also enable us to study the effects of the socio-technical contexts that the metrics programs exist in and are influenced by. In our analysis, we focus on organizations that have formally adopted metrics programs and have thus crossed the second stage of the six-stage model. We are interested in analyzing the factors that influence the extent to which organizations adapt to metrics programs, i.e., the extent to which the organization has implemented work-processes around the metrics programs. Thus, one of the contributions of our work is to propose and empirically validate a measure for adaptation (Stage 3) of metrics programs .

within software organizations. The successful adaptation of a metrics program should lead to increased use of the metrics programs in decision-making within the organization, i.e., increased acceptance of the metrics program (Stage 4). Acceptance in this context is defined as the level of use of metrics information in decision-making across different tasks in the organization [38]. Therefore, we also study the link between adaptation and acceptance of metrics programs. Innovative practices and programs such as metrics programs exist in highly complex organizational environments that are driven by strong institutions existing in the industry. These institutions can be both external to the organization as well as internal. For example, the pressure applied on firms by potential customers and clients is a strong external motivating force. In other cases, organizations adopt innovative practices because they believe these are the right way to do business, thereby establishing strong internal institutions. Thus, institutions in general are instrumental in the adoption of innovative practices in organizations. We therefore study software metrics programs using institutional theory as a theoretical basis. Institutional theory describes the forces that drive organizations to adopt practices and policies in order to gain legitimacy in the market. Thus, firms that conform to prevalent institutional norms in an industry are seen as being more legitimate than firms that do not conform [26]. We therefore evaluate the effect of these institutional forces on metrics programs implementation in organizations. Our analysis is conducted on data collected through a survey of 214 metrics professionals in software organizations in the United States. The Web-based survey contained questions on existing metrics programs and demographic information from 145 organizations, including both defense contractors and commercial sector companies. To summarize our work in this paper, we study the implementation of metrics programs in software organizations. We extend past research in metrics programs by conducting an industrywide survey of metrics programs focusing on institutional and organizational factors. Past research has argued that the implementation of complex innovations such as metrics programs requires a series of adaptation moves [8]. We therefore measure adaptation by using a multidimensional view that includes the extent to which individual workprocesses are modified or instituted to support the metrics program. Subsequently, we examine the effects of adaptation on an outcome measure that reflects the extent to which the metrics program is used and accepted in decisionmaking, i.e., acceptance [38]. In the next section, we review some of the prior work in software metrics programs and institutional theory.

BACKGROUND LITERATURE

The background literature pertinent to our analysis comes from two sourcesthe anecdotal literature on metrics programs and prior research on institutional theory. In this section, these two streams of research are briefly reviewed.

GOPAL ET AL.: THE IMPACT OF INSTITUTIONAL FORCES ON SOFTWARE METRICS PROGRAMS

681

2.1 Software Metrics The bulk of prior research in software metrics and their successful implementation in organizations is anecdotal. The focus of this literature has been on examining factors that lead to successful implementation of metrics programs in organizations. Berry and Jeffery [3] call this line of research factor-based (p. 185), where the objective is to rank factors in terms of their association with success. Note that the definition of success varies from study to study [34]. The anecdotal literature falls into two categories. Case studies on metrics programs in a single organization are in the first category. The second category includes studies that compare metrics programs in two or more organizations. Based on these comparisons, the authors then present factors that are influential in determining success. We discuss these two streams of work briefly. Company-specific case studies describe metrics implementations at organizations such as Motorola and Contel Corporation. For example, Daskalantonakis [10] discusses the introduction and implementation of a metrics program at Motorola in some detail. He focuses on the need for a set of common metrics, guidelines for collection and interpretation, and tools for automating the use of metrics. Likewise, Pfleeger [34] discusses the introduction of a metrics program in Contel Corporation. She identifies the importance of simple and usable metrics and a streamlined data collection and analysis process in addition to several other factors. While some of the factors discussed are particular to Contel, other factors are generalizable to other software development companies. Related case studies on Eastman Kodak [40], Loral Labs [27], and the US Army [12] bring out the relevance of very similar sets of factors that determine metrics programs success. In contrast to the above-mentioned papers, other work in metrics programs has looked at two or more firms in analyzing success. Hall and Fenton [15] contrast two programs in two different companies, one successful while the other was not as successful. The authors show that upper management support, resource availability, communication, and the need for adequate feedback from the stakeholders in the program increase the probability of success. Gopal et al. [14] survey more than a hundred organizations that have metrics programs in-house to identify technical and organizational factors that lead to metrics success. The importance of management support, communication, and feedback loops is underscored in their work. Our work here complements the above-mentioned prior research in several ways. First, we recognize the validity of the factor-based approach to metrics programs success and use it to drive our definition of adaptation. The important factors identified therein, such as communication and feedback, are included in our definition of adaptation. Thus, our measure of adaptation is a composite measure building on prior factor-based work. Second, we add to this literature by empirically testing one aspect of success acceptance. This construct captures the often-expressed sentiment in the literature that eventual use of metrics information in decision-making is a strong indicator of successful implementation. Note that Acceptance is only

one such dimension of success in this context; further research is needed to study other dimensions of success, such as enhanced decision-making speed and improved quality of decisions. Third, much of the prior work in metrics programs has focused on factors within the organization. Our work adds to this by explicitly considering the effects of the greater institutional environment outside the firm on the metrics programs [3]. Thus, the role of institutions in this context is important and we discuss these next.

2.2 Institutional Forces The focus in the literature has been on success factors internal to the organization. However, it is important to recognize the influence of the external environment that software organizations exist in. In particular, software organizations exist in institutionalized environments that exert strong pressures on the organizations to conform to certain industry-wide norms. Like any other industry, the maturing of the global software industry has led to institutions and institutional norms emerging in the environment. Institutional theory studies the sources of these institutional norms and the manner in which they affect decision-making. Institutional theory describes the institutional forces that drive organizations to adopt practices and policies in order to gain legitimacy in the market. At its core, institutional theory tries to explain institutional isomorphism, i.e., the constraining process that forces one unit in a population to resemble other units that face the same environment [9]. Institutional pressure can be applied by rules, laws, public opinion, views of important constituents such as customers and suppliers, knowledge legitimated through education and universities, and social prestige. The institutional forces on firms typically arise from the presence of professional bodies, cadres of similarly trained people, and strong methods and processes [11]. Note that these forces are not imposed by any one body but are based on the general emergence of strong norms and accepted beliefs within the industry. Strong institutions have emerged in the software industry in the last decade and these norms have come from the growth of the worldwide software industry and a movement toward an engineering focus. Formal programs in computer science and information systems, strong academic institutions such as the ACM, growth of software-related journals such as IEEE Software, and software organizations such as IBM, Oracle, and Motorola, have added to the institutionalization of the industry. These institutions provide organizations with the incentives to adopt methods and practices that have, over time, become the norm. With respect to metrics programs in particular, the importance given to process-based improvement initiatives in software organizations has increased pressure on organizations to adopt such programs [25], [35]. This has been facilitated by organizations such as the Software Engineering Institute (SEI) and the International Standards Organization (ISO) which provide companies with risk frameworks, process frameworks, and quality management techniques [30]. The SEI has proposed the Capability

682

IEEE TRANSACTIONS ON SOFTWARE ENGINEERING,

VOL. 31,

NO. 8,

AUGUST 2005

Maturity Model (CMM), a staged model of process improvement [30]. The ISO, on the other hand, has proposed several frameworks for quality systems in production of which the ISO 9001 describes a model for quality assurance in design, development, and installation services [29]. Although some differences exist between the CMM and the ISO model, both standards have gained acceptance in the software development community as the normative way to develop software. Certification of software development organizations on the basis of the Capability Maturity Model (CMM) is often required by clients in the US and has become accepted as part of legitimate practice [2]. European software companies and clients tend to use the ISO 9001 and the newer ISO/IEC 15504 standard [31] as accepted frameworks for assessing software vendors. Software organizations are also motivated to acquire process certification by competing organizations in the software market [17]. Moreover, the evaluation and selection of software vendors by both government and commercial customers is sometimes conditional on process certification [5]. As the value of process certification and process improvement activities increases, the need for metrics programs within organizations will also increase. Thus, the value of process-based improvement and the associated process certification is a significant factor in promoting the implementation of metrics programs in organizations. Although researchers in software metrics have not explicitly addressed the institutional environment in their analysis, they have recognized the importance of these forces in determining metrics programs success. The importance of management commitment has been mentioned in several case studies on metrics programs [10], [34]. The role of management is important in setting up internal institutions within the organization, i.e., the managers establish norms within the firm. The external environment has also been discussed as influencing metrics programs success [20]. In another paper, the same authors describe the relevance of external stakeholders such as customers and potential customers on metrics programs [3]. Offen and Jeffery [28] have included the business and strategic goals in their M3P framework for establishing measurement programs. Business and strategic goals are set in conjunction with the external environment and partners such as suppliers, customers, and competitors. Thus, the use of institutional theory provides a good fit with the extant literature in metrics programs. We discuss our research model and hypotheses next.

of the innovation [6]. From adaptation, the organization then progresses to the next stage of change, acceptance, in which the innovation is accepted and used by the organization. Thus, acceptance represents a direct outcome of the investments in new work-processes and procedures surrounding metrics programs made by the organization. The empirical definition of adaptation we use is similar to that used by Westphal et al. [43] in their study of TQM adoption in hospitals. The authors measured the extent to which hospitals adopted each of 20 different TQM-related activities. Each of these activities was required to some extent for the hospital to have adopted TQM, but hospitals varied in terms of how many of these activities had been adopted fully. Therefore, the extent to which a hospital was considered to have adapted to TQM was based on an average of the extent to which each individual activity was carried out. We use a similar definition of adaptation in our context. Successful adaptation of metrics programs consists of several different but related organizational activities. Adaptation is reflected by the extent to which each of these individual organizational activities is begun and conducted in a systematic manner. In order to identify these activities that together constitute adaptation, we use the factor-based literature in software metrics discussed before. Specifically, we identify five primary factors that have been addressed in all the factor-based past research in metrics programs, as highlighted below.

3.1.1 Regularity of Metrics Collection Pfleeger [34] mentions the discipline involved in regularly collecting the appropriate metrics as being key to the success of a metrics program. Even if the set of metrics collected is small and focused, the organization needs to establish due processes by which the metrics are collected and archived at the appropriate times. Indirectly, the regularity of metrics collection also measures metrics collection across the phases of a project. Thus, although we dont explicitly capture where in the development process metrics are collected, a systematic and regular set of metrics collected implies that, on average, metrics are collected across different phases of development. Note that this assumption might not be true for metrics needed for predictive purposes and this is a potential limitation of our model. Therefore, a key work-process is the regularity with which metrics selected by the organization are collected. 3.1.2 Data Collection Methodology In addition to metrics regularity, there must be a concerted effort to introduce a seamless and well-understood data collection methodology in the organization. Prior work in metrics programs has discussed the need for a properly structured data collection methodology that is less burdensome and is well integrated into the work of an average software engineer [10], [27], [34]. 3.1.3 Sophisticated Data Analysis A key component of a metrics programs value is the ability to conduct sophisticated analysis on the metrics information collected [15]. Briand et al. [7] provide a taxonomy of analyses that are possible with software metrics, from the

RESEARCH MODEL

AND

HYPOTHESES

3.1 Adaptation and Acceptance As mentioned before, adaptation is defined as the stage in the implementation process in which the innovation is developed, installed, and maintained. During adaptation, organizational procedures around the innovation (or innovative practice) are revised within the organization or new procedures are developed. New work-practices are developed to fully leverage the innovation and organizational members are trained both in the procedures and use

GOPAL ET AL.: THE IMPACT OF INSTITUTIONAL FORCES ON SOFTWARE METRICS PROGRAMS

683

simplest effort estimates to sophisticated forecasting applications, and stress the role of analysis in ensuring metrics program success. Note that the ability to use sophisticated analysis techniques does not discount the value of simplicity in metrics analysis. However, the ability to carry out more sophisticated analyses such as forecasting, statistical quality control, and defect prevention increases the breadth of decision-making that metrics programs can support. Moreover, higher-level process areas in both the CMM and the ISO/IEC 15504 standard implicitly require the ability to perform more sophisticated analyses [31].

different ways across different organizations (based on organizational structure and strategy), acceptance of metrics programs need not be uniform. However, on an average, higher adaptation will be associated with greater acceptance and we therefore propose: Hypothesis 2. Greater levels of metrics programs adaptation in software organizations are associated with increased levels of acceptance.

3.1.4 Communication Mechanisms It is important to create processes to communicate the results of the data analysis back to the stakeholders involved in a timely manner. This could be through formal means such as phase documents and memos or through informal means such as at review meetings and project meetings. The lack of adequate communication is also a success factor in metrics programs mentioned by project managers in a survey of 13 companies carried out by Hall et al. [16]. 3.1.5 Automated Tools Most unsuccessful metrics programs involve additional work imposed on engineers through intrusive and laborintensive data collection methods [15], [27], [34]. The ideal method would be to automate the process of data collection in a manner that is entirely transparent to the user. Therefore, the level of automated support available to the program is an indication of the extent of adaptation. Note that the five factors discussed above are broad enough to be implemented across most organizations but are specific enough to characterize adaptation within any given organization. This definition of metrics adaptation is by no means exhaustive but provides a first attempt at capturing metrics program adaptation in organizations. Therefore, we propose:
Hypothesis 1. The extent of metrics programs adaptation into a software organization is reflected by the following workprocesses 1. 2. 3. 4. 5. Regularity of metrics collection, Seamless and efficient data collection, Use of sophisticated data analysis techniques, Use of suitable communication mechanisms, Presence of automated data collection tools.

Organizational work-processes that have been adapted to suit the metrics programs benefits prepare the organization for acceptance. Acceptance is an outcome of the efforts taken to induce organizational members to commit to the use of the innovation in decision-making [38]. In the case of metrics programs, adaptation is a particularly crucial precursor to acceptance. Increased usage of metrics information is contingent on the quality of metrics collected, the level of trust and confidence the users have in the analysis, the communication process, and manner in which the metrics are collected [10]. Clearly, inadequate processes for the collection and analysis of metrics information would be a case of Garbage In Garbage Out. Since metrics programs are adapted in

3.2 Institutional Forces and Adaptation Institutional forces can be differentiated into three types mimetic, coercive, and normative. Mimetic forces influence organizations operating in uncertain environments and are defined as those forces which induce an organization to mimic other organizations that are perceived to be successful. They manifest when technologies, processes, and goals are vague and ambiguous [11]. This is true of the software industry which has been associated with high uncertainty, inadequate processes, and serious cost and quality issues [17]. Thus, software organizations react to uncertainty by instituting metrics-based process initiatives that mimic successful firms with metrics programs. Successful metrics implementation in firms like Motorola [10] induces other firms to follow suit with their own metrics programs in order to appear legitimate. Coercive forces are exerted on organizations by other organizations upon which it is dependent, such as government organizations, present and potential clients, and regulatory agencies. Customers could enforce metrics programs on software organizations as a means to controlling costs and ensuring quality. For example, customers and potential clients sometimes require either CMM or ISO certification as a prerequisite for business [33]. The presence of such metrics initiatives in competing organizations also drives software organizations toward implementing metrics programs [2]. Therefore, coercive forces acting on the organization induce the organization to make the required changes to implement metrics programs. Normative forces arise from professionalization, defined as a move by members of an occupation to define the conditions and methods of their work to establish legitimacy for their occupation [11]. Professionalization arises from universities and academic institutions on one hand, and from professional and trade organizations on the other hand. The process initiatives in software engineering that started in the 1980s can be seen as steps in the professionalization of the software industry [19]. Normative forces in the software industry have manifested themselves in the form of popular software process models such as the CMM and the ISO 9001. Collectively, software organizations have adopted software processes to bring legitimacy to their development process. Thus, normative forces operating on software organizations motivate them to make adaptive changes around metrics programs. Therefore, we propose:
Hypothesis 3. Higher levels of institutional forces perceived by software organizations are associated with higher levels of adaptation of work-processes associated with metrics programs. In contrast to the external institutional forces discussed above, there are internal institutional forces acting on

684

IEEE TRANSACTIONS ON SOFTWARE ENGINEERING,

VOL. 31,

NO. 8,

AUGUST 2005

Fig. 1. Conceptual model of metrics adaptation.

metrics programs as well. The internal institutions are created by management support and commitment to the metrics program. The external institutions influence software managers who, in turn, establish the appropriate environment within the organization. The internal institutions allocate resources, set broad strategic directions for the firms activities, and establish organizational norms and culture [26]. Therefore, demonstrated management commitment is an important part of the internal institution of the organization. In the context of software metrics, management commitment is an important facilitator of metrics implementation. Upper management commitment is an enabler for greater levels of metrics implementation within the organization in two ways. First, upper management support results in greater resources being allocated to the program [14]. Second, it increases the visibility and legitimacy of the program within the organization [15], [28]. Research in software process improvement has also identified management commitment as an important determinant of success in process improvement activities [35]. We believe therefore that the level of management commitment exhibited to a metrics program in the organization will directly drive the level of investments made in the adaptation of workprocesses around the program. Thus, we propose the following: Hypothesis 4. Higher levels of management commitment in software organizations are associated with higher levels of metrics programs adaptation. The over-all research model postulated is shown in Fig. 1 with each hypothesis located on the figure.

RESEARCH METHODOLOGY

4.1 Data Collection Given that the focus of our paper is on the adaptation and acceptance of metrics programs, data was collected from organizations that were in the postadoption phase. It was important for us to locate companies that were in the process of implementing metrics programs. Moreover, since our research methodology was a survey, we needed to contact the appropriate manager within these organizations. In order to locate the appropriate software companies

and the required survey respondent within these companies, two trade organizations active in the software metrics area were contacted. The first organization contacted was a private company that ran tutorials and courses for metrics managers, published a journal relating to metrics information, and organized a major annual conference for metrics professionals. Thus, this company had access to metrics professionals from software organizations who attended their conference and meetings. Lists of attendees to the annual conference held by the organization were collected for three years and collated. A final list of 296 names was identified through this list. Software companies sent one to three persons to the conference. In certain cases, different divisions within the same organization had different metrics programs and, therefore, sent different teams to the conference. Eighty people were from military organizations and the remaining were either private sector or defense contractors. The second organization we contacted was a division of the US Department of Defense (DoD) that coordinated metrics-related activities for software divisions and contractors. This organization worked with different contractors for the DoD as well as with commercial software vendors in disseminating metrics information through meetings and conferences. The lists of attendees to their quarterly meetings were collected for two years and collated. This list yielded 219 names of which 115 were military agencies or departments. In addition to the above two sources, we were allowed access to the list of attendees to the Software Engineering Institutes (SEI) training programs in software metrics programs for two years. This list included 178 names of which 55 were in the defense and the remaining in the private sector. Putting the lists of names together from the two aforementioned sources with the list of attendees from the SEI provided us with an addressable sample of 594 names. The data collection methodology used for this paper was a survey. The data was collected on the Internet through an online survey instrument hosted on the SEIs Web-server. Before the survey was administered, it was pretested by five members of the in-house Software Measurement Group at the SEI, which is responsible for providing training to

GOPAL ET AL.: THE IMPACT OF INSTITUTIONAL FORCES ON SOFTWARE METRICS PROGRAMS

685

TABLE 1 Sample Profile

TABLE 2 Summary Statistics

practitioners on metrics initiatives. The survey was also administered to 15 practitioners attending a training session at the SEI on metrics, who provided valuable feedback. These 15 respondents were not included in the final sampling frame used. All changes and feedback were incorporated into the questionnaire before it was administered to respondents. The changes pertained to altering words and structure of questions, providing definitions for terms used, and introducing clarificatory instructions in the survey. In addition, we randomly distribute questions pertaining to a single construct within the questionnaire to avoid respondent bias. Finally, a small set of questions were dropped from the survey due to space constraints. The final questionnaire contained approximately 80 questions spanning roughly 12 pages on paper and took about 30 minutes to complete. The questionnaire also contained demographic information on the respondent and respondents organization. The 594 potential respondents were contacted by email and were also sent unique IDs and passwords to protect the integrity and confidentiality of their responses. The instructions included in the email and on the Website indicated that if multiple persons from the same metrics program were being contacted, the senior-most member was asked to respond to the questionnaire. In cases where participants were likely from the same organization, they were all copied on the introductory email. The recipients were also asked to contact the research team in cases where multiple people in the same organization were possibly contacted. In many cases, the authors received email from respondents indicating that multiple people had been contacted and that only one designated person would be responding for the organization. Given the number of such interactions, we believe that most responses can be associated with a unique metrics program. As mentioned before, in some cases, different divisions of the same company had different metrics initiatives in place and these responses were counted separately. This was particularly true of defense organizations where the scale of the organization was much larger and included a number of independent software

organizations. Of the 594 names, 90 were removed from the sample due to these interactions with the authors. Due to the time elapsed between the capture of the email addresses at our sources and data collection, some respondents on our e-mail lists were no longer available. Therefore, our initial sampling frame of 504 names was reduced to 385 names due to unavailability of persons concerned. Of these 385 persons contacted, 228 complete responses were received resulting in a 59 percent response rate.1 Of these 228, 14 were deleted due to incomplete data for the purposes of this study. The final sample size used in subsequent analysis is 214. The sample profile and summary statistics are shown in Tables 1 and 2, respectively. We describe the measures used next.

4.2 Variable Definitions The following section describes the measures used to capture the various constructs in our research model. There are four central constructsadaptation, acceptance, institutional forces, and management commitment. Each of these constructs was measured using a distinct set of questionnaire items, which are described below. The questionnaire items were measured on 5-point Likert scales and were created on the basis of prior research in both metrics programs and institutional theory. We describe the measures in our model and their psychometric tests next. 4.2.1 Metrics Adaptation We model adaptation as consisting of five reflective factors discussed in the previous section. They are described below.
1. No incentives were offered for participation in the survey. We thank an anonymous reviewer for this point.

686

IEEE TRANSACTIONS ON SOFTWARE ENGINEERING,

VOL. 31,

NO. 8,

AUGUST 2005

Metrics Regularity (Metrics): Most metrics programs are built on a foundation of basic software engineering metrics such as cost data, schedule information, and effort information. This construct was measured using four questionnaire items addressing regularity of the collection of base metrics in the organization. 2. Data Collection (Collect): This construct measured how well the organization had specified responsibility for metrics data collection and was based on prior research [10], [30]. The measures for this construct reflect the need for a regular and systematic methodology for collecting project-level metrics discussed in past work [4]. The construct was measured using three questionnaire items. 3. Quality of Analysis (Analysis): This construct captured the level of sophisticated analysis conducted using metrics information and is based on Briand et al.s [7] taxonomy of analysis options. Briefly, the taxonomy describes three broad kinds of goals that metrics enabledescriptive, evaluative, and prescriptive goals. The basic set of metrics can provide descriptions or snapshots of the current state of development. More sophisticated analysis can provide managers with the tools to evaluate the development process, thereby enabling better decisionmaking. Prescriptive models use metrics data and sophisticated techniques to enable prediction of performance and make prescriptive recommendations. For our purposes, we use four items in the survey that address the increasing order of sophistication in analysis. Since this is a more broad-based study of metrics in the industry, the more commonly used set of analyses were included in the survey. 4. Communication (Comm): This construct measured the level to which feedback on metrics information was communicated to various stakeholders in the program. It also measured the extent to which formal and informal communication methods were used based on prior research [24]. The variable was measured using four questionnaire items. 5. Automated Tools (Auto): The presence of automated tools has often been addressed in past work and we use three questionnaire items to measure this construct. The questions pertain to the presence of tools for data collection, management, and analysis. Before the constructs measured above can be used in analysis, we need to establish their reliability and validity. Reliability of the measures expresses the degree to which it is consistent and repeatable. Thus, reliability involves establishing high test-retest correlation, indicated by Cronbachs alpha coefficient of 0.70 and higher. Validity indicates the extent to which the questionnaire items truly measure what it claims to measure, i.e., the underlying construct. We need to establish construct and discriminant validity in our context. Construct validity tests whether the questionnaire items measuring a construct are intercorrelated well enough to jointly form the construct. Discriminant validity tests if the items measuring one construct are sufficiently uncorrelated from other constructs, thereby establishing the ability 1.

to discriminate between constructs. We perform these tests on our measures below. All constructs displayed good reliability as shown, along with questionnaire items, in Appendix A. In order to test construct validity, each construct was subjected to exploratory factor analysis and the results are shown in Appendix A. Each set of questionnaire items associated with an underlying construct loaded appropriately on one factor when subjected to factor analysis, indicating good construct validity. To assess discriminant validity, all the questionnaire items were factor analyzed together to examine the factor structure and loadings. The factor structure emerging fitted the hypothesized 5-factor model and is shown in Table 3. Thus, our constructs exhibit acceptable construct and discriminant validity.

4.2.2 Institutional Forces The institutional forces were measured using five items in the questionnaire. As discussed before, mimetic and coercive forces in this industry are applied by external entities such as customers and competitors. Therefore, we include two questionnaire items that capture the coercive elements and pertain to the extent to which measurement or external certifications are required by customers. The mimetic aspect is measured using one item that captures the extent to which competitors have used measurement to their advantage. The normative aspect of institutional forces is built around professionalization, which is best captured by the extent to which metrics-trained persons are employed within the organization [11]. Although this concept is difficult to assess directly, we can infer it through the extent to which the organization adapts itself to be assessed at higher maturity levels. To capture this, we included two questionnaire items that pertained to the presence of quality initiatives such TQM, CMM, or ISO in the organizations. The five questionnaire items used here show good reliability (Cronbachs alpha = 0.81) and load well on one factor in factor analysis. 4.2.3 Management Commitment To capture the level of management commitment displayed by the organization, four questionnaire items are used. We measure management commitment through demonstrated support to the metrics program and the allocation of resources to the metrics program. In terms of resources, we use two items to capture the training provided to the metrics program and the availability of required funding for the program. The four items load on one factor and show high reliability (Cronbachs alpha = 0.79). 4.2.4 Metrics Acceptance Acceptance is typically captured in one of three ways attitudes toward use of an innovative process, intention to use, and frequency of use [38]. In our context, the appropriate measure of acceptance is the frequency with which metrics information is used in decision-making. We use prior work in metrics programs that describe how metrics information can be used [7] as well as the factorbased research on metrics programs [28], [41] to create our measures. This construct is measured using four

GOPAL ET AL.: THE IMPACT OF INSTITUTIONAL FORCES ON SOFTWARE METRICS PROGRAMS

687

TABLE 3 Exploratory Factor Analysis on Individual Work-Processes (N=214, Principal Components with Varimax Rotation)

questionnaire items that pertain to the frequency with which software organizations use metrics information in decision-making. The construct shows good reliability (Cronbachs alpha = 0.76) and loads well on one factor.

4.3 Data Analysis The data analysis for our model was carried out using structural equation modeling (SEM). SEM is useful when it is important to estimate all the paths of a hypothesized model together rather than evaluate the paths individually. There are two primary analytical procedures that were conducted in evaluating our research model. First, we ran a confirmatory factor analysis procedure on the five work-processes that constitute adaptation. In this analysis, we empirically test for the presence of a higherorder latent construct, Adaptation, which is composed of the five underlying factors. Second, we estimate the research model shown in Fig. 1 to address our research hypotheses. In this analysis, each of the paths shown in the model was estimated along with standard errors. In addition, overall goodness of fit indices for the overall model were also estimated. The statistical procedures are described in detail in Appendix B. We briefly describe the statistical results below. The first objective, to perform confirmatory factor analysis of adaptation, was strongly supported by the analysis. Each of the underlying five factors loaded well on a higher-order construct and the over-all goodness of fit indices were excellent, as shown in Table 4. Thus, our analysis shows that the definition of Adaptation as being composed of five underlying work-processes receives strong empirical support. Adaptation of metrics programs in an organization can be thus driven to a large extent by instituting work-processes such as the five factors studied.

The second procedure to estimate the research model (Fig. 1) shows goodness of fit indices that are statistically significant, indicating a sound model.2 In addition, each of the paths estimated were statistically significant as shown in Table 5 and Fig. 2. Thus, the statistical procedure shows that the data fit our proposed model in a statistically significant manner. We discuss the individual results next.

RESULTS

AND

DISCUSSION

5.1 Hypotheses Tests The results of the models presented above show strong support for Hypotheses 1 and 2. The confirmatory factor analysis results on metrics adaptation show that the five work-processes hypothesized to constitute adaptation receive strong empirical validation, thus supporting Hypothesis 1. Hypothesis 2 pertained to the significant association between adaptation and acceptance and the results support this reasoning. As seen in Fig. 2, the parameter is positive (1.09) and is statistically significant (p < 0.05). Acceptance of metrics programs is higher in organizations with workprocesses that have substantially adapted to the program. Our results also show support for the assertions made in the anecdotal literature in software metrics about the need for appropriate procedures and methods to be defined. These work-processes have to be defined and put in place before metrics-based information is used in decision-making. The hypotheses pertaining to Institutional Forces and Management Commitment are also strongly supported. The institutional pressures exerted on organizations are associated with greater investments in defined work-processes
2. The actual procedures and statistics are described in detail in Appendix B.

688

IEEE TRANSACTIONS ON SOFTWARE ENGINEERING,

VOL. 31,

NO. 8,

AUGUST 2005

TABLE 4 Confirmatory Factor Analysis for Metrics Adaptation (Second Order Latent Variable) (N = 214, All Paths Significant at p < 0.05)

TABLE 5 Measurement Model for Institutional Forces on Metrics Adaptation (N = 214, All Paths Significant at p < 0.05)

within the organization, thereby possibly leading to greater adaptation. As adaptation is required for acceptance of the metrics programs, the second-order effects of the institutional forces include promoting acceptance of metrics

programs within the organization. The presence of demonstrated management commitment also influences the adaptation of software metrics programs and supports past work in software metrics [34], [15].

GOPAL ET AL.: THE IMPACT OF INSTITUTIONAL FORCES ON SOFTWARE METRICS PROGRAMS

689

Fig. 2. Results of the Structural Model (N = 214, Parameter estimates with t-stats in parenthesis).

5.2 Posthoc Analysis Our sample consists of two sectorsthe government or defense sector and the commercial sector. It is conceivable that these two sectors exhibit some fundamental differences in the organization of metrics programs. In particular, there is a possibility that institutional forces or adaptation differs systematically between the defense and commercial sectors. Therefore, a difference of means t-test was performed on all the factor variables between the two subsamples. The only significant result was in the institutional forces. The perception of institutional forces is lower in the defense sector than in the commercial sector (t = 3.43, p < 0.001). This could be due to the fact that, in the defense sector, metrics related activities are required by governmental agencies. The defense sector has led the commercial sector in the process movement since metrics and measurement have longer been an integral part of all defense-related software development [16]. The presence of case studies from the defense sector such as the US Army [12], NASA [41], and Goddard Space Flight Center [36] attests to this fact. It can be argued that larger organizations are more willing to make adaptive changes to their organization since the returns could be greater. Researchers have hypothesized that larger organizations have more slack resources and are therefore more likely to adopt innovations [8]. We test for this effect in our sample by including size as a control variable in our model. We use the total number of full-time employees within the business unit as our measure of size. Unfortunately, this data is only available for 162 responses in our sample. We estimated our research model with size as a control variable on metrics adaptation using 162 responses. The log of total fulltime employees was used to reduce the scale of the variable. The size variable was not significant and did not change other results. 5.3 Limitations A common problem with survey methodologies is common method variance. In other words, the questionnaire items could be tapping a latent, common factor not captured in the instruments. In order to test for this issue, we use Harmons one factor test [32]. According to this

test, if there was indeed one latent common factor that was driving the results, a factor analysis of all the questionnaire items together should show the presence of one significant underlying factor. Accordingly, all the questionnaire items and factor scores in the model were subjected to a single factor analysis. Four factors were extracted with three eigenvalues over 1.0 and the fifth eigenvalue was 0.92. The first factor captured only 36 percent of the variance. Since a single factor did not emerge from the factor analysis and one factor did not capture the majority of the variance in the data, common method variance might not be causing significant problems in our analysis. All of the data in our analysis is perceptual, which is a limitation of the survey research methodology. Moreover, pretesting of the survey was done with 20 persons, which might not have identified all sources of bias. Another limitation of our analysis is that we do not capture where in the development process the metrics are collected. However, our measure of Metrics Regularity addresses the frequency with which metrics are collected in the organization, which indirectly gets at collection of project-level metrics across all phases of projects. The unit of analysis in our paper is the organization and, therefore, if the organization regularly and systematically collects metrics on all projects, it seems reasonable to expect that metrics are collected on all projects across all phases. It would be hard to collect project-level data in our case where the unit of analysis is the organization. Therefore, we try to indirectly capture the phases of a project through the Metrics Regularity construct. Since our objective was to study organizations with existing metrics programs, our sampling frame included only organizations with metrics programs. Additionally, there is a possible sampling bias, i.e., the organizations that responded were probably not randomly distributed in the population. Since our response rate is relatively high, we believe that the sampling frame might not affect the robustness of our results. However, the sampling bias is a common issue with most survey research and our results have to be interpreted keeping this issue in mind.

690

IEEE TRANSACTIONS ON SOFTWARE ENGINEERING,

VOL. 31,

NO. 8,

AUGUST 2005

MANAGERIAL IMPLICATIONS

AND

CONCLUSION

Software managers are typically the primary users and beneficiaries of software metrics in their decision-making. Our work informs them at multiple levels. At the level of the metrics program, managers need to understand the importance of adaptation. The onus of setting up appropriate procedures for collecting metrics data, analyzing it, and disseminating this information is up to software managers. Thus, our research provides them with a firstlevel description of the five important factors that need to be addressed for adaptation. First-time metrics managers can benefit by addressing these five work-procedures first before venturing onto other aspects of these programs. This phased approach can provide them with some probability of success. Our research also cautions software managers that expecting people to use metrics information in decision-making without adequate investments in adapting work-procedures may not work. Mandating metrics use in organizations without underlying structural changes in the work-procedures will result in inadequate acceptance. Finally, software managers are also usually cognizant of the institutional norms active in the software industry. Therefore, it is through their initiatives that the bulk of the adaptive processes take place. Our research informs them of the beneficial and, in some cases, harmful, effects of institutional norms. Although we dont explicitly analyze these issues, software managers must assess the institutional norms in the industry and structure the metrics program accordingly. Our research also has direct implications for upper management in software development organizations. First, our analysis directly informs upper management about the importance of continued support to metrics programs. It is vital to not just formally adopt metrics programs but to enable the adaptive process so that the metrics program assimilates fully into the organization. Second, management is responsible for setting up the internal institutional environment that motivates and incentivizes software managers to make the investments required. It is also important to note that incentive-based behavior often does not last once the incentives are removed. Management is also responsible for providing resources and demonstrated support for the metrics programs. Our research shows that these sources of support are required for adaptation. Finally, upper management needs to understand the vital role of metrics in software process initiatives that lead to improved quality and reduced costs. The business value indicated in prior research from process initiatives [25] is largely contingent on the availability of good metrics. Thus, an investment in a productive metrics programs pays off through process initiatives that address larger cost and quality issues. Our work points out several avenues for future work. First, our definition of adaptation is a first attempt to capture the essential components of this construct. More work is required to refine and possibly add other factors to this definition. Second, the metrics programs success is determined by the organizations returns on investment. We measure acceptance which is one possible outcome measure of a metrics program. However, more work is

needed to quantify the benefits from the implementation of a metrics program in organizations. Third, several structural issues about metrics programs need study. For example, is it preferable to have a unified suite of metrics for the whole organization or is decentralized metrics collection appropriate? These questions are best addressed in the context of the organizational strategy. More work is also required in analyzing the institutional contexts that software organizations function in. The increasing institutionalization of the software industry makes this an interesting area for research, both from theoretical and empirical viewpoints.

APPENDIX A QUESTIONNAIRE
The sample questionnaire is presented in Tables 6a and 6b.

APPENDIX B STATISTICAL PROCEDURES

There are primarily two statistical procedures described in this appendix. The first procedure performs confirmatory factor analysis to test out the proposed five-factor model of Adaptation. The second procedure is used to evaluate the research model shown in Fig. 1. We use structural equation modeling techniques to perform both statistical procedures. Structural equation modeling is a technique that allows the estimation of all the paths in a model simultaneously instead of breaking them down into constituent paths. This analysis also provides us with overall goodness of fit indices which represent the extent to which the hypothesized model fits the data. Thus, we get estimates of paths between constructs of interest as well as indices of overall model fit. In cases where the whole model is more than just the sum of the constituent paths, structural equation modeling is useful and informative. Structural equation modeling involves the estimation of two separate modelsthe measurement model and the structural model. The measurement model addresses the degree to which the observed variables, i.e., the questionnaire items, capture the underlying constructs. This model also allows the estimation of measurement error in the estimation of the measurement model, thus allowing the researcher to test for the strength of the measures used. In this way, the measurement model also provides evidence of construct and discriminant validity [39]. The structural model describes the relationships between the constructs, i.e., the hypothesized paths between the variables of interest [1]. The structural model is therefore used to test the research hypotheses proposed in the paper. In order to perform our analysis, we use a structural equation modeling software called LISREL. LISREL is one of the most widely used software packages used to perform this kind of analysis.3 We first describe the confirmatory factor analysis on Adaptation and subsequently describe the estimation of the research model.
3. More information on LISREL is available at www.ssicentral.com.

GOPAL ET AL.: THE IMPACT OF INSTITUTIONAL FORCES ON SOFTWARE METRICS PROGRAMS

691

TABLE 6a Questionnaire

B.1 Confirmatory Factor Analysis of Metrics Adaptation Metrics adaptation was hypothesized to be a latent construct characterized by a set of underlying workprocesses. In order to establish that the five work-processes indeed were reflective of a higher-order construct (adaptation in this case), we conducted a confirmatory factor analysis of a second-order latent variable Metrics Adaptation, as shown in the box labeled Hypothesis 1 in Fig. 1. Thus, the five factors (Regularity, Data Collection, Analyses, Communication, and Automated Tools) are the first-order latent constructs, which are hypothesized to collectively

form a second-order latent construct called Adaptation. This analysis will confirm the validity of the adaptation measure well as strengthen subsequent analysis of institutional forces on adaptation. Thus, we perform confirmatory factor analysis on the factors that constitute Adaptation. The estimation of all the paths in the factor model is done using maximum likelihood (ML) estimation. Structural equation modeling is based on two assumptionsmultivariate normality and the presence of intervalscaled, continuous data [39]and we test for these assumptions. Prior research has shown categorized data with five or more categories can be treated as continuous for

692

IEEE TRANSACTIONS ON SOFTWARE ENGINEERING,

VOL. 31,

NO. 8,

AUGUST 2005

TABLE 6b Questionnaire (cont.)

the purposes of confirmatory factor analysis [21]. In a set of Monte Carlo simulations examining the effect of categorization of continuous variables, the use of a covariance matrix with maximum likelihood estimation provides estimates that are sufficiently robust even in the case of categorized variables [18]. In addition, research has shown that with five categories used to approximate underlying continuous variables, the bias or distortion in correlation coefficients is small [6]. It is therefore acceptable, in some conditions, to treat categorical data as continuous. Consistent with this approach, we treat our five-point Likert scaled data as continuous. The presence of nonnormal data, i.e., nonzero third and fourth-order moments, provides consistent but inefficient estimates of parameters [42]. Therefore, it is important to test for the presence of univariate and multivariate normality in the data. In Monte Carlo simulations, Curran et al. [9] characterize moderate nonnormality with skewness of 2 and kurtosis of 7 and their results show that maximum likelihood estimation is robust even in the presence of mild non-normality. Univariate skewness in our sample ranges from -1.35 to 1.719 and univariate kurtosis ranges from -1.381 to 2.075, indicating low nonnormality. The estimate of relative multivariate kurtosis is 1.089 for the sample. Therefore, we can safely assume multivariate normality. The results from the ML analysis are shown in Table 4. The estimated model shows strong support for the second-order latent variable of Metrics Adaptation. The goodness of fit index for the model is 0.90, indicating that the overall hypothesized factor structure fits the data well.

In addition, the ratio of the fit function chi-square to the degrees of freedom is 1.7, again indicating a good model fit [39]. The acceptable range for this statistic is under 5 [39]. The high comparative fit index (0.95) and low root mean square residual (0.05) also indicate good model fit [18]. Thus, our confirmatory factor model of metrics adaptation receives strong support.

B.2 Structural Model of Institutional Forces on Metrics Adaptation Having established the factor structure for adaptation, we next estimate the research model in Fig. 1. Since our focus is estimating the institutional effects on adaptation, we use factor scores for the individual work-processes that compose metrics adaptation instead of individual questionnaire items. Although some information is lost in using factor scores instead of raw data, there are two advantages to using factor scores. First, factor scores more closely approximate normality since they are combinations of individual questionnaire items. Second, they increase the degrees of freedom available by reducing the number of estimated parameters. Moreover, based on our confirmatory factor analysis of metrics adaptation, we believe this approach will not detract from the primary objective of the structural model. The factor scores were calculated based on exploratory factor analysis reported in Table 3. As discussed before, we treat our data as continuous and test for normality in the sample. Univariate skewness in our sample ranges from -0.689 to 0.762, and kurtosis from -1.411 to 0.678, indicating reasonable univariate normality. The

GOPAL ET AL.: THE IMPACT OF INSTITUTIONAL FORCES ON SOFTWARE METRICS PROGRAMS

693

coefficient of relative multivariate kurtosis is 1.063, indicating mild nonnormality. We use ML estimation and also report the Satorra-Bentler chi-square [39] that corrects the minimum-fit chi-square for nonnormality. The SatorraBentler chi-square is not based on the multivariate normality assumption and, therefore, provides an indirect measure of the level of nonnormality in the data. In cases of mild nonnormality, the Satorra-Bentler chi-square will be very similar in value to the overall goodness of fit chi-square. The results of the measurement and structural models are shown in Table 5 and Fig. 2, respectively. The measurement model in Table 5 shows support for the factor structure hypothesized amongst the indicator variables. The paths from the observed questionnaire items to the latent constructs are all positive and significant. The structural model fit is acceptable, with a goodness of fit index of 0.88. Other goodness of fit indices such as the comparative fit index (0.93) and the root mean square residual (0.05) are also within acceptable ranges [15]. The ratio of the chi-square to degrees of freedom is 1.98, well within the acceptable limit of 5 [32]. The root mean square residual is also low at 0.05, indicating low discordance between the hypothesized model and the data. Thus, all fit indices indicate that the measurement and structural model are significant and valid. In addition, the Satorra-Bentler chi-square is slightly lower than the normal-theory-based chi-square. The difference between the minimum-fit chisquare (246.64) and the Satorra-Bentler chi-square (228.21) is small, indicating low nonnormality. Thus, the proposed overall model of institutional forces on metrics adaptation is statistically significant.

ACKNOWLEDGMENTS
The authors are grateful to Dennis Goldenson and Ritu Agarwal for their support during the course of this research.

REFERENCES
[1] [2] [3] [4] [5] [6] [7] [8] [9] J.C. Anderson and D.W. Gerbing, Structural Equation Modeling in Practice: A Review and Recommended Two-Step Approach, Psychological Bull., vol. 103, no. 3, pp. 411-423, 1988. G.H. Anthes and J. Vijayan, Lessons from India Inc, Computerworld, vol. 35, no. 14, pp. 40-42, 2001. M. Berry and R. Jeffery, An Instrument for Assessing Software Measurement Programs, Empirical Software Eng., vol. 5, pp. 183200, 2000. M. Berry and M.F. Vandenbroek, A Targeted Assessment of the Software Measurement Process, Proc. Seventh Software Metrics Symp., pp. 222-235, 2001. A. Binstock, Outside Development Partners, InformationWeek, pp. 133-140, Oct. 1999. K.A. Bollen and B.H. Barb, Pearsons R and Coarsely Categorized Measures, Am. Sociological Rev., vol. 46, pp. 232-239, 1981. L.C. Briand, C.M. Differding, and H.D. Rombach, Practical Guidelines for Measurement-Based Process Improvement, Software ProcessImprovement and Practice, vol. 2, pp. 253-280, 1996. R.B. Cooper and R.W. Zmud, Information Technology Implementation Research: A Technological Diffusion Approach, Management Science, vol. 36, no. 2, pp. 123-139, 1990. P.J. Curran, S.G. West, and J.F. Finch, The Robustness of Test Statistics to Nonnormality and Specification Error in Confirmatory Factor Analysis, Psychological Methods, vol. 1, no. 1, pp. 16-29, 1996.

[10] M. Daskalantonakis, A Practical View of Software Measurement and Implementation Experiences Within Motorola, IEEE Trans. Software Eng., vol. 18, no. 11, pp. 998-1010, Nov. 1992. [11] P.J. DiMaggio and W.W. Powell, The Iron Cage Revisited: Institutional Isomorphism and Collective Rationality in Organizational Fields, Am. Sociological Rev., vol. 48, pp. 147-160, 1983. [12] S. Fenick, Implementing Management Metrics: An Army Program, IEEE Software, Mar. 1990. [13] N.E. Fenton and S.L. Pfleeger, Software Metrics: A Rigorous and Practical Approach. London, U.K.: Intl Thompson Publishers, 1998. [14] A. Gopal, T. Mukhopadhyay, M.S. Krishnan, and D.R. Goldenson, Measurement Programs in Software Development: Determinants of Success, IEEE Trans. Software Eng., vol. 28, no. 9, pp. 863-875, 2002. [15] T. Hall and N. Fenton, Implementing Effective Software Metrics Programs, IEEE Software, pp. 55-65, 1997. [16] T. Hall, N. Baddoo, and D. Wilson, Measurement in Software Process Improvement Programmes: An Empirical Study, Proc. Intl Workshop Software Measurement (IWSM 2000), pp. 73-82, 2000. [17] M. Hayes, Precious Connection, InformationWeek, pp. 34-50, Oct. 2003. [18] L. Hu and P.M. Bentler, Evaluating Model Fit, Structural Equation Modeling: Concepts, Issues and Applications, 1995. [19] S.W. Humphrey, Discipline for Software Engineering. AddisionWesley, 1995. [20] R. Jeffery and M. Berry, A Framework for Evaluation and Prediction of Metrics Programs Success, Proc. First Intl Software Metrics Symp., 1993. [21] D.R. Johnson and J.C. Creech, Ordinal Measures in Multiple Indicator Models: A Simulation Study of Categorization Error, Am. Sociological Rev., vol. 48, pp. 398-407, 1993. [22] B. Kitchenham, S.L. Pfleeger, and N. Fenton, Toward a Framework for Software Measurement Validation, IEEE Trans. Software Eng., vol. 21, no. 12, pp. 929-944, Dec. 1995. [23] B.A. Kitchenham, R.T. Hughes, and S. Linkman, Modeling Software Measurement Data, IEEE Trans. Software Eng., vol. 27, no. 9, pp. 788-804, Sept. 2001. [24] R.E. Kraut and L.A. Streeter, Coordination in Large Scale Software Development, Comm. ACM, vol. 38, no. 7, pp. 69-81, 1995. [25] M.S. Krishnan, C.H. Kriebel, S. Kekre, and T. Mukhopadhyay, An Empirical Analysis of Productivity and Quality of Software Products, Management Science, vol. 46, no. 6, pp. 745-759, 2000. [26] J.W. Meyer and B. Rowan, Institutionalized Organizations: Formal Structure as Myth and Ceremony, Am. J. Sociology, vol. 83, no. 2, pp. 340-363, 1983. [27] R.E. Nusenoff and D.C. Bunde, A Guidebook and A Spreadsheet Tool for a Corporate Metrics Program, J. Systems and Software, vol. 23, pp. 245-255, 1993. [28] R.J. Offen and R. Jeffery, Establishing Software Measurement Programs, IEEE Software, pp. 45-53, Mar./Apr. 1997. [29] F. OHara, European Experiences with Software Process Improvement, Proc. Intl Conf. Software Eng. (ICSE 2000), 2000. [30] M.C. Paulk, B. Curtis, M.B. Chrissis, and C.V. Weber, Capability Maturity Model for Software Version 1.1, Technical Report SEI93-TR-24, Software Eng. Inst., Pittsburgh, Penn., 1993. [31] M.C. Paulk, Analyzing the Conceptual Relationship between ISO/IEC 15504 and the Capability Maturity Model for Software, Proc. Intl Conf. Software Quality, 1999. [32] P.M. Podsakoff and D. Organ, Self-Reports in Organizational Research: Problems and Prospects, J. Management, vol. 12, pp. 531543, 1986. [33] O. Port, Will Bugs Eat Up the US Lead in Software? Business Week, Dec. 1999. [34] S.L. Pfleeger, Lessons Learned in Building a Corporate Metrics Program, IEEE Software, vol. 10, no. 3, pp. 67-74, 1993. [35] A. Rainer and T. Hall, A Quantitative and Qualitative Analysis of Factors Affecting Software Processes, J. Systems and Software, vol. 66, pp. 7-21, 2003. [36] L.H. Rosenberg and L. Hyatt, Developing a Successful Metrics Program, Proc. 1996 Software Technology Conf., 1996. [37] I. Rozman, R.V. Horvat, and J. Gyorkos, United View on ISO 9001 Model and SEI CMM, Proc. IEEE Intl Eng. Management Conf., pp. 56-63, 1994. [38] V.L. Saga and R.W. Zmud, The Nature and Determinants of IT Acceptance, Routinization and Infusion, Diffusion, Transfer and Implementation of Information Technology, 1994.

694

IEEE TRANSACTIONS ON SOFTWARE ENGINEERING,

VOL. 31,

NO. 8,

AUGUST 2005

[39] R.E. Schumacker and R.G. Lomax, A Beginners Guide to Structural Equation Modeling. New Jersey: Lawrence Erlbaum Assoc., 1996. [40] C. Seddio, Integrating Test Metrics within a Software Engineering Measurement Program at Eastman Kodak Company: A Follow-Up Case Study, J. Systems and Software, vol. 20, 1993. [41] P. Thibodeau and L. Rosencrance, Users Losing Billions Due to Bugs, Computerworld, vol. 36, no. 27, pp. 1-2, 2002. [42] S.G. West, J.F. Finch, and P.J. Curran, Structural Equation Models with Non-Normal Variables: Problems and Remedies, Structural Equation Modeling: Concepts, Issues, and Applications, 1995. [43] J.D. Westphal, R. Gulati, and S.M. Shortell, Customization or Conformity? An Institutional and Network Perspective on the Content and Consequences of TQM Adoption, Administrative Science Quarterly, vol. 42, pp. 366-394, 1997. Anandasivam Gopal received the PhD degree in information systems from Carnegie Mellon University in 2000. He is an assistant professor of information systems at the University of Maryland College Park. His research interests include software engineering economics, offshore software development, and software metrics programs. In specific, his interests focus on contracts in software development and the success factors of metrics programs. His research has been published in the Communications of the ACM, IEEE Transactions on Software Engineering, and Management Science.

Tridas Mukhopadhyay received the PhD degree in computer and information systems from the University of Michigan in 1987. He is the Deloitte Consulting Professor of e-Business at Carnegie Mellon University. He is also the director of the Institute for Electronic Commerce and MSEC (Master of Science in Electronic Commerce) program at CMU. His research interests include strategic use of IT, businessto-business commerce, Internet use at home, business value of information technology, and software development productivity. In addition, Professor Mukhopadhyay has worked with several organizations, such as General Motors, Ford, Chrysler, PPG, Port of Pittsburgh, Texas Instruments, IBM, Diamond Technology Partners, LTV Steel, and governmental agencies including the United States Post Office and the Pennsylvania Turnpike. He has published more than 60 papers. His research appears in Information Systems Research, Communication of the ACM, Journal of Manufacturing and Operations Management, MIS Quarterly, Omega, IEEE Transactions on Software Engineering, Journal of Operations Management, Accounting Review, Management Science, Journal of Management Information Systems, Decision Support Systems, Journal of Experimental and Theoretical Artificial Intelligence, Journal of Organizational Computing, International Journal of Electronic Commerce, American Psychologist, and other publications. M.S. Krishnan (Krishnan) received the PhD degree in information systems from the Graduate School of Industrial Administration, Carnegie Mellon University, in 1996. He is the Mary and Mike Hallman e-Business Fellow, Area Chairman, and Professor of Business Information Technology at the University of Michigan Business School. Dr. Krishnan is also a codirector of the Center for Global Resource Leverage: India at the Michigan Business School. He was awarded the ICIS Best Dissertation Prize for his doctoral thesis on Cost and Quality Considerations in Software Product Management. His research interests include corporate IT strategy, business value of IT investments, and management of distributed business processes, software engineering economics, metrics and measures for quality, productivity, and customer satisfaction for products in software and information technology industries. In January 2000, the American Society for Quality (ASQ) selected him as one of the 21 voices of quality for the 21st century. His research articles have appeared in several journals including Management Science, Information Systems Research Information Technology and People, Strategic Management Journal, IEEE Transactions on Software Engineering, IEEE Software, Decision Support Systems, Harvard Business Review, Information Week, Sloan Management Review, Optimize, and Communications of the ACM. His article, The Role of Team Factors in Software Cost and Quality, was awarded the 1999 ANBAR Electronic Citation of Excellence. He serves on the editorial board of reputed academic journals including Management Science and Information Systems Research. Dr. Krishnan has consulted with Ford Motor Company, NCR, HIP, IBM, Bellsouth, TVS group, and Ramco Systems.

. For more information on this or any other computing topic, please visit our Digital Library at www.computer.org/publications/dlib.