Process Improvement and Risk Management in Off-The-Shelf Component-Based Development

Process Improvement and Risk
Management in Off-The-Shelf
Component-Based Development
Jingyue Li
Doctoral Thesis
Submitted for the Partial Fulfilment of the Requirements for the Degree of
Philosophiae Doctor
Department of Computer and Information Science

Faculty of Information Technology, Mathematics and Electrical
Engineering
Norwegian University of Science and Technology
June 2006
Copyright © 2006 Jingyue Li
ISBN printed 82-471-7920-2

ISBN electronic 82-471-7919-9
ISSN 1503-8181
NTNU 2006:84
Printed in Norway by NTNU Trykk, Trondheim

TO MY PARENTS ZHONGMIN LI AND YUSHU SHEN,
MY WIFE FUFEN JIN AND MY SON LELE
iv
Abstract
Reusing software components from third-party vendors is one key technology to gain
shorter time-to-market and better quality of the software system. These components,
also known as OTS (Off-the-Shelf) components, come in two types: COTS
(Commercial-Off-The-Shelf) and OSS (Open–Source-Software) components. To use
OTS components successfully, it is necessary to know how the development processes
and methods have to be adapted. Most current studies are either theoretical proposals
without empirical assessment or case studies in similar project contexts. It is therefore
necessary to conduct more empirical studies on how process improvement and risk
management were performed and what were the results in various project contexts.
The overall research design of the work was a sequential, mixed method design that
consists of several empirical studies. The thesis contains five novel main contributions
(C1 to C5) and one minor contribution (C6):
C1. Better understanding on reusing in-house built components. A quantitative
preliminary study shows that reusing in-house built components has the same
challenges as OTS components related to requirements (re)negotiation, component
documentation, and the specification of quality attributes. The informal communication
between developers supplements component documentation. The component repository
is not a key factor to successful component reuse.
C2: Summarizing the state-of-the-practice of development and selection processes in
OTS component-based projects. A qualitative pre-study, a quantitative main study, and
a follow-up study with industrial seminar show that OTS component-based
development processes are typically variations of well-known process models, mixed
with OTS specific activities. Concerning the OTS component selection, we find that
mainly familiarity-based and Internet searches with hands-on-trials processes are
applied. We summarize the state-of-the-practice of OTS processes into seven scenarios
and propose how to customize the processes in OTS component-based projects.
C3: Empirical verification of the risk management proposals in OTS component-
based development. The results of the main study reveal that allocating more effort into
learning OTS components, performing the integration testing early, evaluating the
quality of OTS components thoroughly, and following the update of OTS components
have helped to mitigate corresponding risks. However, some problems, such as wrong
estimation of the integration efforts and inefficient debugging, still happen frequently in
practice and need further investigations.
C4: Better understanding of decision-making in OTS component-based development.
By comparing projects using COTS components with those using OSS components, we
reveal who, why, and the results of using different OTS components.
C5: Empirical validation of six newly published theses from the literature, which are
related to process and risk management in OTS component-based development. The
results of the main study support four theses and contradict the two others.
C6. Improved understanding about performing an international survey in the ICT
industry (minor contribution). The main study is probably the first software engineering
survey using a representative subset (sample) of industrial companies. Lessons learned
from performing such as study reveal that, at best, we can achieve a stratified-random
sample of ICT companies, followed by a convenient sample of relevant projects.
i
ii
Preface
This thesis is submitted to the Norwegian University of Science and Technology
(NTNU) for partial fulfilment of the requirements for the degree of Philosophiae
Doctor.
The work referred to has been performed at the Department of Computer and
Information Science, NTNU, Trondheim, under the supervision of Professor Reidar
Conradi.
The doctoral work was financed by NTNU for four years. This thesis has also been
affiliated with the INCO (INcremental and COmponent-based development) done
jointly by the University of Oslo and NTNU.
iii
iv
Acknowledgements
During the work on this thesis, I have benefited from a lot of communication with many
people, who have provided valuable input to my studies. First of all, I would like to
thank my supervisor Prof. Reidar Conradi for giving in valuable feedback and advice
during my work. His engagement and knowledge have inspired me a lot. I would also
thank Prof. Maria Letizia Jeccheri, Associate Prof. Dag Svanæs, Prof. Tor Stålhane, Dr.
Parastoo Mohagheghi, Dr. Carl Fredrik Sørensen, Odd Petter N. Slyngstad, Prof.
Kristen Ringdal, and Prof. Ola Listhaug at NTNU. They have all helped me with their
insights and advice. Furthermore, I acknowledge Assistant Prof. Marco Torchiano and
Associate Prof. Maurizio Morisio at Politecnico di Torino and Dr. Christian Bunse at
Fraunhofer IESE for their support and cooperation during the international survey.
Stewart Clark has been especially helpful by proof-reading this thesis. I also
acknowledge all the industrial partners that participated in this work.
Several master’s students at NTNU have contributed to studies in this thesis. I would
like to thank Axel Anders Kvale, Odd Are Sæhle, and Øivind Wang for their
contributions. Also, thanks to present and former colleagues at the software engineering
group at the Department of Computer and Information Science, NTNU, for providing a
good work environment.
I would like to thank Prof. Anthony Finkelstein at University College London, for his
kind hospitality and cooperation during my stay there in spring 2005.
I would also like to thank my family. A specific appreciation goes to my parents for
supporting and believing in me during all these years. Their effort in taking care of my
child provided me enough time to finish this work. I would especially express my
thanks to my wife, Fufen Jin and my son Lele for their love and support.
v
vi
Contents
Abstract........................................................................................................................ i
Preface........................................................................................................................ iii
Acknowledgements .................................................................................................... v
Contents .................................................................................................................... vii
List of Figures............................................................................................................. x
List of Tables .............................................................................................................. x
Abbreviations ............................................................................................................ xi
1 Introduction ........................................................................................................ 1
1.1 Problem Outline 1
1.2 Research Questions 2
1.3 Research Design 3
1.4 Contributions 4
1.5 Selected Papers 9
1.6 Thesis Structure 11
2 Software Reuse and Component-Based Development .................................. 13

2.1 Software Engineering Definitions and Challenges 13
2.2 Software Reuse 14
2.3 Component-Based Development 15
2.4 COTS-Based Development 18
2.5 OSS-Based Development 19
2.6 OTS Component-Based Development 21
2.7 Summary and Discussion 22
3 Process Improvement in OTS Component-Based Development.................. 25

3.1 Software Process Improvement (SPI) 25
3.2 SPI of OTS Component-Based Development 27
3.2.1 Improvement of the Whole Life Cycle............................................... 27
3.2.2 Improvement of the OTS Component Selection ................................ 28
4 Risk Management in OTS Component-Based Development........................ 31

4.1 Risk Management in Software Development 31
vii
4.2 Risk Management in OTS Component-Based Development 33
5 Research Methods............................................................................................. 37
5.1 Research Strategies in Empirical Software Engineering 37
5.2 Survey Methods 39
5.3 Research Design 40
5.3.1 Research Design of the Preliminary Study......................................... 40
5.3.2 Research Design of the Pre-study....................................................... 43
5.3.3 Research Design of the Main Study ................................................... 45
5.3.4 Research Design of the Follow-up Study ........................................... 52
5.4 Validity Issues 53
6 Results................................................................................................................ 55
6.1 Collected Samples in Each Study 55
6.1.1 Collected Samples in the Preliminary Study ...................................... 55
6.1.2 Collected Samples in the Pre-study .................................................... 55
6.1.3 Collected Samples in the Main Study................................................. 56
6.1.4 Collected Samples in the Follow-up Study ........................................ 57
6.2 Answers to Research Questions 58
6.2.1 Reusing In-House Built Components – RQ1 ..................................... 58
6.2.2 Software Process Improvement – RQ2 .............................................. 59
6.2.3 Risk Management – RQ3 ................................................................... 65
6.2.4 Decision Making – RQ4 ..................................................................... 69
6.3 Our Lessons Learned on Performing the Studies 73
7 Evaluation and Discussion ............................................................................... 75

7.1 RQ1: How to Improve the Reuse of In-house Built Components? 75
7.1.1 Component-Related Requirements (Re) negotiation.......................... 75
7.1.2 Component Repository ....................................................................... 76
7.1.3 Component Understanding ................................................................. 76
7.1.4 Quality Attributes of Components...................................................... 76
7.2 RQ2: How to Do SPI in OTS Component-Based Development? 76
7.2.1 How to Improve the Whole Life Cycle .............................................. 77
7.2.2 How to Select the OTS Components.................................................. 80
7.3 RQ3: How to Mitigate Risks in OTS Component-Based Development? 81
7.3.1 How to Mitigate the Cost Estimation Risks ....................................... 81
7.3.2 How to Mitigate the Quality Risks ..................................................... 82
7.3.3 How to Mitigate the Requirements Risks........................................... 83
7.3.4 How to Mitigate the Debugging and Deployment Risks.................... 83
7.3.5 How to Mitigate the Maintenance Risks ............................................ 84
7.3.6 How to Mitigate the Provider Support Risks...................................... 85
7.4 RQ4: How to Make the Decision between COTS or OSS Components? 85
7.4.1 Why Did Developers Decide to Use OTS Components..................... 86
viii
7.4.2 How Were the OTS Components Used in Practice? .......................... 86
7.4.3 COTS or OSS? Our Suggestions ........................................................ 87
7.5 Evaluation of Validity Threats 87
7.5.1 Possible Validity Threats in the Preliminary Study............................ 87
7.5.2 Possible Validity Threats in the Pre-study ......................................... 88
7.5.3 Possible Validity Threats in the Main Study ...................................... 89
7.5.4 Possible Validity Threats in the Follow-up Study.............................. 90
7.6 Summary 90
8 Conclusions and Directions for Future Work................................................ 91

8.1 Conclusions 91
8.1.1 Describing Different Aspects of Reusing Component ....................... 91
8.1.2 Verifying Existing Theories and Assessing Existing Methods .......... 92
8.1.3 Generating New Theories from New Perspectives............................. 93
8.2 Future Work 94
8.2.1 Need to Study the Cause-Effects of the SPI Issues ............................ 94
8.2.2 Risk Management Issues Need a Deeper Investigation...................... 94
8.2.3 Need to Better Understand OSS Projects and Products ..................... 94
References................................................................................................................. 95
Appendix A: Selected papers ................................................................................ 107

P1. Developer Attitude to Component Reuse 107
P2. Variations in COTS-Based Development Process 123
P3. Validation of Theses on OTS-Based Development 153
P4. SPI on OTS-Based Development Process 171
P5. Risk Management in OTS-Based Development 185
P6. Decision Making in OTS Component-Based Development 225
P7. Barriers to Disseminating Theory 233
P8. Reflections on Conducting a Survey 241
Appendix B: Other Papers.................................................................................... 259
Appendix C: Interview Guide for the Pre-study................................................. 265
Appendix D: Questionnaire in the Main Study................................................... 279
ix
List of Figures
Figure 1-1 Research questions and corresponding studies............................................... 3
Figure 1-2 Contributions C1 to C5 ................................................................................... 5
Figure 4-1 Risk management framework ....................................................................... 31
Figure 4-2 Risk management in OTS component-based development .......................... 35
Figure 5-1 Risk management before a project starts ...................................................... 46
Figure 5-2 Research framework for the risk management in the main study................. 46
Figure 6-1 Distribution of company size........................................................................ 56
Figure 6-2 Distribution of companies main business areas............................................ 57
Figure 6-3 Distribution of application domain of the systems ....................................... 57
Figure 6-4 The actual development process in the OTS-based project.......................... 60
Figure 6-5 What selection and evaluation actions were performed?.............................. 62
Figure 6-6 When was the OTS component selected?..................................................... 62
Figure 6-7 Result of thesis T5 ........................................................................................ 64
Figure 6-8 Occurrences of typical risks.......................................................................... 66
Figure 6-9 Risk management activities performed......................................................... 67
Figure 6-10 Project profiles of the COTS component-based system ............................. 69
Figure 6-11 Project profiles of the OSS component-based system ................................ 70
Figure 6-12 General motivations of using COTS components ...................................... 70
Figure 6-13 General motivations of using OSS components ......................................... 71
Figure 6-14 Specific motivations of using COTS components...................................... 71
Figure 6-15 Specific motivations of using OSS components......................................... 72
Figure 6-16 Occurrences of risks in COTS component-based projects ......................... 72
Figure 6-17 Occurrences of risks in OSS component-based projects ............................ 73
Figure 7-1 Scenarios 1 to 3............................................................................................. 78
Figure 7-2 Scenarios 4 to 6............................................................................................. 79
List of Tables
Table 1-1 Relations between research questions, contributions, and papers.................... 9
Table 2-1 Advantages and disadvantages of using COTS software............................... 19
Table 5-1 Qualitative vs. quantitative in empirical strategies ........................................ 38
Table 5-2 Questionnaire used in the preliminary study.................................................. 42
Table 5-3 Correspondence between questions and RQs in the preliminary study ......... 43
Table 5-4 Typical risks in OTS component-based projects ........................................... 48
Table 5-5 Typical risks management activities in OTS-based projects ......................... 49
Table 5-6 Revised research questions to investigate the newly published theses .......... 50
Table 6-1 Barriers to disseminating SPI theories to the IT industry .............................. 64
Table 6-2 Effective risk management activities ............................................................. 68
Table 6-3 Barriers to disseminating RM theories to the IT industry.............................. 68
x
Abbreviations
CBD Component-Based Development
CBSE Component-Based Software Engineering
CMM Capability Maturity Model
CRM Customer Relationship Management
COCOTS COnstructiveCOTS
CORBA Common Object Request Broker Architecture
COM Component Object Model (Microsoft)
COTS Commercial-Off-The-Shelf
DCOM Distributed Component Object Model (Microsoft)
EJB Enterprise Java Beans (Sun)
ERP Enterprise Resource Planning
GUI Graphical User Interface
HHM Hierarchical Holographic Modelling
ICT Information and Communication Technology
ISO International Standards Organization
IEC International Electrotechnical Commission
LOC Lines of Code
NTNU Norwegian University of Science and Technology
OO Object-Oriented
OTS Off-The-Shelf
OSS Open Source Software
ROI Return On Investment
RUP Rational Unified Process
SEI the Software Engineering Institute (SEI) at Carnegie Mellon University
SPI Software Process Improvement
UP Unified Process
XP eXtreme Programming
xi
xii
1 Introduction
In this chapter, the background to the research and the research context are briefly
presented. The chapter also describes research questions, research design, and the
claimed contributions. The list of papers, my specific contributions to each study, and
the thesis outline are also presented.
1.1 Problem Outline

To gain competitive advantages, software organizations are forced to develop systems
quickly and cost-efficiently. Reuse is therefore one key technology to reach their goals.
Reuse is known since the beginning of computer science (e.g., reuse of command
sequences) and has now be raised to the level of components (i.e., a reusable piece of
software that can be easily integrated with other components with relatively little
effort). The components can be built in-house, be acquired from the third-party
providers, or be ordered from the sub-contractors. The components acquired from third-
party providers, also known as OTS (Off-the-Shelf) components, come in two different
types: COTS (Commercial-Off-The-Shelf) and OSS (Open–Source-Software)
components. To integrate the OTS components successfully, one major research
question is: how development processes or methods have to be adapted concerning the
development of systems with such components?
Although researchers and practitioners have been dealing with OTS component-
based development processes for quite a time, most studies are based on military or
aerospace projects, or similar large projects. To propose and design cost-effective OTS-
based development process, it is necessary to empirically investigate how OTS-based
projects are performed in different application domains, especially in small or medium-
sized projects and companies.
Although using OTS component may help to shorten time-to-market and save
development effort, using such external components introduces many risks. Before
project managers decide to acquire an external component instead of building it in-
house, they must evaluate and compare possible risks and benefits. Although several
risks and risk management activities in OTS component-based development have been
identified from case studies, few empirical studies have subsequently verified their
1
Introduction
conclusions. As a result, software project managers have few effective and well-proven
guidelines on using specific risk management activities to mitigate corresponding risks.
Although both COTS and OSS components can be acquired from third-party
providers, they are different in their nature and appearance. COTS components are
owned by commercial vendors that often provide specific support. Users of COTS
components have limited access to the source code. On the other hand, OSS
components are provided by open source communities with freely accessible source
code, but with no promise about specific support. When planning a new software
project, project decision-makers need to decide whether they should buy a COTS
component or acquire an OSS one, if it was decided to use OTS components. To make
such a decision, it is important to investigate previous projects using such components.
Unfortunately, the relevant decision-making processes are often intuitive and only
sparsely documented. Thus, further research is warranted to examine the current state-
of-the-practice and to define guidelines for making such decisions.
1.2 Research Questions

The goal of this research is to investigate state-of-the-practice of COTS and OSS
component-based development from the process and organizational viewpoints. The
motivation is to give guidelines on how to select and integrate COTS or OSS
components based on the experiences and lessons learned from finished projects. There
are four main research questions in this study. The research questions are designed and
refined step by step.
The process and risk management issues of component-based development were
firstly investigated through a preliminary study on reusing in-house built components.
This study examined research question RQ1:
RQ1: What is the status of component-based development based on components

built in-house?
Results of the preliminary study gave valuable insights on designing research

questions for later studies. We are aware that there are many common issues between
reusing in-house built components and using OTS components. Therefore, we decided
to focus more on using OTS components, because it will cover broader issues and some
of these issues are also relevant to in-house built components.
Based on results of the preliminary study, it was decided to investigate the software
process improvement (SPI) and risk management in COTS components-based
development with a pre-study. This study investigated research questions RQ2 and
RQ3:
RQ2: What are the development processes and OTS component-selection processes
used in OTS component-based development projects?
RQ3: Which risk management activities have been performed and what are the
results of performing them?
2
Introduction
By talking with the respondents face-to-face in the pre-study, we got insights on SPI
and risk management in OTS component-based development. We also found that OSS
components should be involved in future studies, because OSS components represent an
important type of OTS components. In addition to know their commonalities, it is
interesting to know the differences in using these two different types of OTS
components. Therefore, research question RQ4 is defined as:
RQ4: How do project managers make their acquire decisions in using COTS
instead of OSS components, or vice versa?
1.3 Research Design

Figure 1-1 shows the studies performed, their date and sequence, type of studies, and
their relations with the research questions.
Figure 1-1 Research questions and corresponding studies
3
Introduction
Empirical studies may be performed quantitatively, qualitatively or in combination.

The studies reported in this thesis have been a combination of qualitative and
quantitative studies. The choices of approach are decided by the research questions.
The research question RQ1 was studied by an explorative quantitative preliminary
study. Data from 26 people in three Norwegian IT companies were collected by
interviews using a structured questionnaire.
Research questions RQ2 and RQ3 were first investigated by an explorative
qualitative pre-study. The pre-study was performed through semi-structured interviews
on COTS component-based development. In this pre-study, we interviewed 16 COTS
component-based development projects in 13 Norwegian IT companies. In addition,
RQ4 was initiated based on the results of the pre-study.
To validate the conclusions of the pre-study, a descriptive quantitative main study
was performed. Data was collected with a structured questionnaire. Results from 133
OTS component-based projects in Norway, Italy, and Germany have been collected.
Results of the main study verified the conclusions of the pre-study.
In the late phase of the main study, we performed an explorative qualitative follow-
up study with an industrial seminar to present our conclusions, to discuss with industrial
partners, and to get feedback from them. The intention was to briefly understand some
cause-effects of the phenomena we discovered from the previous studies.
1.4 Contributions
To manage the OTS component-based project successfully, project members need to
make several decisions. These decisions may not be sequential and may need several
iterations. We mainly contributed on summarizing the state-of-the-practice and giving
guidelines on making several key decisions, as shown in Figure 1-2.
4
Introduction
Figure 1-2 Contributions C1 to C5
Contribution C6 is a secondary contribution, which comes from the lessons learned

in performing the survey. It gives guidelines on how to perform an international survey
with stratified-random sample selection strategy.
Detailed descriptions of each contribution:
C1: Increased understanding on risks of reusing in-house built components
In a preliminary quantitative study, we have investigated challenges related to four key

factors for development based on in-house built components, especially in
development-with-reuse. These factors are component-related requirements
(re)negotiation, component repository, component understanding, and components’
quality attribute specification. Another contribution is that we have compared three IT
companies with different reuse levels to study the possible trend and challenges in these
factors when more and more code is encapsulated as reusable components.
For component-related requirements (re)negotiation, we conclude that requirements
(re)negotiation for in-house built components is important but not efficient. The
5
Introduction
efficiency will probably not increase with a higher reuse level (i.e., more reusable
components are available in the organization).
For a component repository, we confirm that this is not a key factor for successful
reuse. Furthermore, the potential value of a component repository will probably not
increase with higher reuse levels.
For component understanding, we reveal that most developers are not satisfied with
component documentation, and that developers’ satisfaction with component
documentation will decrease with higher reuse levels. The results also show that
informal communication channels (i.e., tacit knowledge), through which developers
may get necessary information about the components, should be given more attention.
For components’ quality attribute specification, we discover that developers still
need to spend much effort on testing the components before they start to use them, as
they cannot get relevant information from component specifications.
C2: Summarizing the state-of-the-practice of development and selection processes

in OTS component-based projects
The contributions to the OTS component-based development process focus on two

dimensions:
- The whole software development life cycle.
- OTS component selection and evaluation.
A qualitative pre-study with semi-structured interviews was first performed to

examine the state-of-the-practice of the development processes in COTS component-
based development. We found:
- The use of COTS components can be done as part of traditional development

processes (e.g., waterfall and evolutionary), i.e., there is no special “COTS-based
development process”.
- Successful use of COTS components in such processes, however, requires that
some new activities and roles are introduced. Typical new activities are the make
vs. acquire decision, COTS component selection, and COTS component
integration. A new role is that of a knowledge keeper.
- Two of the new activities, the make vs. acquire decision and the COTS
component selection, can be placed in different development phases
(requirements, design, or implementation). This depends on project context,
especially on the familiarity with possible COTS components and the flexibility
of requirements.
- A set of explanatory scenarios and guidelines have been synthesized to assist in
the customization of the traditional development processes with the new
activities and roles.
Based on the results of the pre-study, a main study with a formal questionnaire was
launched. This studied 133 projects from Norway, Italy, and Germany. It had a broader
focus than the pre-study and also included projects using OSS components. The results
confirm our findings in the pre-study. We conclude:
6
Introduction
- The actual OTS-based development process was the traditional process with
OTS-specific activities. The development process was dominated by the
company/department rules, instead of the decision of using OTS components.
- The actual OTS component selection can be done in different phases. The main
evaluation processes are familiarity-based or hands-on-trial-based. The phase to
select OTS component has relationship with the project members’ familiarity
with possible component candidates.
- The proposed software development process and OTS component selection
process need further investigation to be suited to different project contexts.
By summarizing the state-of-the-practice of OTS component-based development

processes into seven scenarios, we gave systematic proposals about how to adopt the
OTS-based development and selection processes based on the project contexts.
C3: Empirical validation of risk management proposals in OTS component-based

development
By summarizing the good practices and lessons learned in the projects investigated in
the pre-study, we got new insights on managing the possible risks in OTS component-
based development. In the main study, we further examined our findings. Our
contributions answered the following three sub-questions:
- What are the most frequent problems occurred in OTS component-based
development?
Two of the most frequent problems in OTS component-based development are
the incorrect estimation of the integration effort and inefficient debugging. The
least frequent problems are relevant to the negative effect of the OTS
components on the quality of the whole system.
- What are the most frequent risk management activities performed in OTS
component-based development?
Developers have seriously evaluated the quality of the OTS components in the
selection phase. They have considered the possible learning effort in the effort
estimation. They have done integration testing as early as possible and usually
have internal experts to follow the update of the OTS components. However,
developers have rarely involved their customers in the make vs. acquire decision
and OTS component selection.
- What are the results of the risk management activities on corresponding risks?
To increase the accuracy of effort estimation, it is important to allocate enough
effort to learn and understand the OTS components. To avoid the possible
quality problems and to increase the efficiency of locating the defects, it is
important to do black-box testing as part of OTS component selection and to do
integration testing as early as possible. To mitigate the problems due to
customers’ requirements changes, it is important to learn the OTS components
thoroughly in the selection phases and to integrate the unfamiliar OTS
components first. It is also important to be able to negotiate changed
requirements with the customers. To increase the efficiency of debugging and
7
Introduction
system deployment, it is necessary to understand the components well and to do

integration testing early. It is critical to keep and share the knowledge about OTS
components and the component providers inside the organization. It shows to
help mitigate the possible risks in system maintenance and evolution.
C4: Increased understanding of the decision-making practice in OTS component-

based development
We compared who, why, and the results of using COTS components instead of OSS
components, or vice versa. This contributed to answer the three questions:
- Who were using OTS components?
Both COTS and OSS components were used in projects with different application
domains and different non-functional requirements. There was no significant
difference between the profiles of COTS-based and OSS-based projects.
- Why was it decided to use OTS components?

The main expectations of using either COTS or OSS components are to obtain
shorter time-to-market and less development effort. COTS users have higher
expectation on COTS component quality and COTS vendor support. Possible no-
cost source code is the key motivation of OSS users. OSS users prefer to have
access to the source code, so that they can revise it when necessary.
- What are the possible problems of using each type of OTS components?
It is more difficult for COTS component users to follow requirement changes
than OSS users. It is also more difficult for COTS users to estimate the selection
effort.
C5: Empirical validation of the six newly published theses, which are related to the
process and risk management in OTS component-based development
The pre-study and main study also investigated six newly published theses in COTS
component-based development proposed in [Torchiano04]. Our results support four of
the theses and contradict the two others. The supported theses are: OSS components
were mainly used without modification in practice; custom code mainly provided
additional functionality; formal COTS selection processes were seldom used; COTS
component users managed to get the required changes from vendors. The unsupported
theses are: standard mismatches were more frequent than architecture mismatches;
COTS components were mainly selected based on architecture compliance instead of
functional completeness.
C6: Improved understanding on performing an international survey in the ICT

industry
In the main study, a stratified-random sample selection strategy was used to collect data
in three European countries. The lessons learned from performing the main study
contributed to give guidelines on performing similar empirical studies in the IT
industry. We found that:
8
Introduction
- A pre-study often clarifies both research questions and concrete questions in a

later questionnaire. For instance, we “discovered” several variants of OTS-based
development processes in our pre-study.
- A sample of national Information and Communication Technology (ICT)
companies assumes access to public data sources, like company lists from census
registers, but even such data may contain many errors. Close contact with people
in such organs are needed to obtain stratified samples and expedite delivery of
data in electronic form.
- For a variety of reasons it seems impossible to perform the same random
sampling and data collection in different countries. The national differences may
give an unknown mode effect in result interpretation between the countries.
- Random sampling and the ensuing contact process appear to be much more
expensive (5 times?) than convenient sampling. In our case, this is because we
got stuck with no previous contact person, and scant pre-knowledge and
“goodwill” from such companies, especially the large ones.
- If the final sample size is under 100 respondents, use phone interviews. This
applies particularly when phone contact is anyhow needed, e.g. due to a non-
convenient sample. A written procedure for the contact process is anyhow
needed.
- A project baseline can be useful for comparison purposes. In our case, we did
not ask the companies about non-OTS projects, so we missed this.
- Interviewing on this level requires a sound background knowledge of the
material; Company phone contact assumes at least PhD student qualifications.
1.5 Selected Papers

This thesis includes eight papers from P1 to P8 as follows. These papers show our main
contributions to the research questions and are attached in Appendix A. Abstracts of
other papers (from AP1 to AP7) related to work in this thesis are attached in Appendix
B. All these papers can be downloaded from http://www.idi.ntnu.no/grupper/su/. The
relations between the research questions, contribution, and the papers are shown in
Table 1-1.
Table 1-1 Relations between research questions, contributions, and papers

Research questions Contributions Papers
RQ1 C1 P1*,
RQ2 C2 P2*, P4*, P7*, AP6, AP7
RQ3 C3 P2*, P5*, P7*, AP1, AP2, AP3, AP4
RQ4 C4 P6*, AP5
RQ2&RQ3 C5 P3*
Method issues C6 P8*
* Papers attached in Appendix A
[P1] [Li04a] Jingyue Li, Reidar Conradi, Parastoo Mohagheghi, Odd Are Sæhle, Øivind
Wang, Erlend Naalsund, and Ole Anders Walseth: A Study of Developer Attitude to
Component Reuse in Three IT Companies. Proceedings of 5th International Conf. on
9
Introduction
Product Focused Software Process Improvement (PROFES'2004), Apr. 2004, Nara

area, Japan, Springer LNCS Vol. 3009, pp. 538-552.
Relevance to this thesis: This paper reports the result of the preliminary study,
which covers RQ1 and shows our main contribution to C1.
My contribution: I was the leading author and contributed about 70% of the total
work, including research design, questionnaire design, data analysis, and paper
writing.
[P2] [Li06a] Jingyue Li, Finn Olav Bjørnson, Reidar Conradi, and Vigdis By
Kampenes: An Empirical Study of Variations in COTS-based Software Development
Processes in Norwegian IT Industry. To appear in the Journal of Empirical Software
Engineering (Special issue from Metrics 2004).
Relevance to this thesis: This paper reports the result of the pre-study, which
covers RQ2 and RQ3 and partly shows our contribution to C2 and C3.
My contribution: I was the leading author and contributed about 80% of the total
work, including research design, interview guide design, performing most interviews,
data analysis, and paper writing.
[P3] [Li05a] Jingyue Li, Reidar Conradi, Odd Petter N. Slyngstad, Christian Bunse,
Umair Khan, Marco Torchiano and Maurizio Morisio: Validation of New Theses on
Off-The-Shelf Component Based Development. Proceedings of the 11th IEEE
International Metrics Symposium (Metrics'05), Sep. 2005, Como, Italy, IEEE Press,
pp.26 (abstract), 10p.
Relevance to this thesis: This paper reports the result of the main study, which
covers RQ2 and RQ3 and shows our contribution to C5.
My contribution: I was the leading author and contributed 70% of the total work,
including research design, questionnaire design, and data collection in Norway, data
analysis, and paper writing.
[P4] [Li06b] Jingyue Li, Marco Torchiano, Reidar Conradi, Odd Petter N. Slyngstad,
and Christian Bunse: A State-of-the-Practice Survey of the Off-the-Shelf Component-
Based Development Processes, Proceedings of the 9th International Conference on
Software Reuse, Torino, Italy, June, 2006, Springer LNCS Vol. 4039, pp. 16-28.
covers RQ2 and shows our main contribution to C2.
[P5] [Li06c] Jingyue Li, Reidar Conradi, Odd Petter N. Slyngstad, Christian Bunse,
Marco Torchiano, and Maurizio Morisio: A State-of-the-practice Study on Risk
Management in OTS Component-Based Development. Submitted to IEEE Transactions
on Software Engineering, June 2006.
10
Introduction
including research design, questionnaire design, data collection in Norway, data
[P6] [Li06d] Jingyue Li, Reidar Conradi, Odd Petter N. Slyngstad, Christian Bunse,
Marco Torchiano, and Maurizio Morisio: An Empirical Study on the Decision Making
Process in Off-The-Shelf Component Based Development. Proceedings of the 28th
International Conference on Software Engineering (ICSE 2006), 20-28 May 2006,
Shanghai, P.R. China, pp. 897-900, IEEE Press.
[P7] [Li05b] Jingyue Li, Reidar Conradi, Odd Petter N. Slyngstad, Christian Bunse,
Umair Khan, Marco Torchiano and Maurizio Morisio: Barriers to Disseminating Off-
The-Shelf Based Development Theories to IT Industry. Proceedings of the International
Workshop on Models and Processes for the Evaluation of COTS Components
(MPEC'05 Arranged in co-location with ICSE'05), May, 2005, St. Louis, Missouri,
USA, pp. 1-4. It is also published in the electronic version of ACM Software
Engineering Notes, Vol. 30, No. 4, July 2005.
Relevance to this thesis: This paper reports the result of the follow-up study,
which covers RQ2 and RQ3, and shows partly our contribution to C2 and C3.
My contribution: I was the leading author and contributed on 80% of the total
work, including research design, seminar organization, data collection, data analysis
and paper writing.
[P8] [Conradi05a] Reidar Conradi, Jingyue Li, Odd Petter N. Slyngstad, Christian
Bunse, Marco Torchiano and Maurizio Morisio: Reflections on conducting an
international CBSE survey in ICT industry. Proceedings of the 4th International
Symposium on Empirical Software Engineering (ISESE 2005), Nov. 2005, Noosa
Heads, Australia, pp. 214-223, IEEE Press.
Relevance to this thesis: This paper reports our lessons learned on the main study,
and shows our main contribution to C6.
My contribution: I contributed on 40% of the total work, including sample
selection design, performing the data collection, and major comments on the data
analysis and draft paper.
1.6 Thesis Structure

Chapters 2 to 5 give the state-of-the-art of relevant fields of this thesis and introduce
how the research questions are designed. Chapters 6 to 8 give the results and discuss our
contributions to the research questions.
Chapter 2: Software reuse and component-based development. This chapter

introduces the state-of-the-art and challenges in software reuse and component-based
11
Introduction
development. First, we discuss the motivation of systematic reuse in software

development and illustrate different kinds of reuse, such as reusing documentation,
architecture, design, and test cases. Then, we introduce the intention and challenges of
reusing software components. Finally, we focus on issues of reusing the third-party
software components, such as COTS and OSS components.
Chapter 3: Software process improvement in OTS component-based development.

This chapter introduces the state-of-the-art of SPI in OTS component-based
development. We describe not only proposed process models in COTS or OSS
component-based development, but also OTS component selection processes.
Chapter 4: Risk management in OTS component-based development. This chapter

first introduces the theory of risk management. We then list the proposed specific risks
and according risk management activities in OTS component-based development.
Chapter 5: Research methods and metrics. In this chapter, the research goals and
research questions are specified. We give our rationale for using different research
methods to answer different research questions.
Chapter 6: Empirical investigations and main results/findings. This chapter explains

how we perform the preliminary study, the pre-study, the main study, and the follow-up
study. We give detailed information on data collection methods, collected samples, and
answers to research questions in each study.
Chapter 7: Evaluation and discussion. The research questions are answered in this
chapter. We summarize the results from the empirical investigation and compare them
with the state-of-the-art. We also discuss possible threats to the validity of each study.
Chapter 8: Conclusion and future work. This chapter sums up the main findings from
the discussion ands outline possible future work.
Appendix A contains 8 selected papers (P1 to P8) that provide detailed results and
discussions of the individual studies.
Appendix B contains the abstracts of other seven relevant papers (AP1 to AP7) which
are not attached in this thesis.
Appendix C includes the interview guide for the pre-study.
Appendix D contains the questionnaire used in the main study.
In both parts of this thesis, I have generally used the term “we” to present the work. The
layout of papers in Appendix A has been changed to fit the format of the rest of the
thesis.
12
Software reuse and component-based development
2 Software Reuse and Component-Based

Development
This chapter describes the challenges in software engineering that are the motivations
behind reuse. The state-of-the-art in Component-Based Development (CBD), COTS
development, and OSS-based development are then described. Finally, the whole
chapter is summarized and the research challenges related to this thesis are discussed.
2.1 Software Engineering Definitions and Challenges

Software engineering is the branch of system engineering concerned with the
development of large and complex software intensive systems. It is concerned with the
processes, methods, and tools for the development of software intensive systems in an
economic and timely manner [Finkelstein00]. Software engineering activities or phases
include managing, estimating, planning, modelling, analysing, specifying, designing,
implementing, testing, and maintaining [Fenton97].
The term software crisis was first used in 1968 to describe the ever-increasing
burden and frustration that software development and maintenance placed on otherwise
happy and productive organizations [Griss93]. The much cited Standish report on
software projects [Standish95] shows “a staggering 31.1% of projects will be cancelled
before they ever get completed”. Further results indicate 52.7% of projects will cost
189% of their original estimates. The cost of these failures and overruns are just the tip
of the proverbial iceberg.
Philippe Kruchten [Kruchten01] discusses why software engineering differs from
structural, mechanical, and electrical engineering due to the soft, but unkind nature of
software. He suggests four key differentiating characteristics:
− Absence of fundamental theories or at least practically applicable theories makes
is difficult to reason about software without building it.
− Ease of change encourages changes in software, but it is hard to predict the
impact.
− Rapid evolution of technology does not allow proper assessment, and makes it
difficult to maintain and evolve legacy systems.
− Very low manufacturing costs combined with ease of change have led the
software industry into a fairly complex mess.
13
These characteristics have even become more extreme due to developments in

Internet-speed. Internet-speed development involves rapid requirement changes and
unpredictable product complexity [Baskerville03]. Software organizations have always
been looking for effective strategies to develop software faster, cheaper, and better.
2.2 Software Reuse

The motivations for reuse are primarily economic: the potential of saving cost, time and
effort of redundant work, increasing productivity, decreasing time to market and
improving systems quality by reusing both artefacts, and the underlying engineering
experience.
Software reuse means reusing the inputs, the processes, and the outputs of previous
software development efforts. The idea of reuse was originally due to Doug McIlroy
[McIlroy68] in his paper “Mass Produced Software Components”. Reuse is later named
as a potential “silver bullet” [Brooks87]. Many software organizations around the world
have reported successful reuse programs, such as at IBM, Hewlett-Packard, and Hitachi
[Griss93]. Jones [Jones84] identified four types of reusable artifacts:
- Architecture reuse, which consists of standardizing a set of design and

programming conventions dealing with the logical organization of software.
- (detailed) Design reuse: for some common business application.
- Data reuse, involving a standardization of data formats.
- Program reuse, which deals with reusing executable code.
The challenges facing reuse are structural, organizational, managerial, and technical
[Mili95]. Many organizations treat reuse as technology-acquisition problems instead of
a technology-transition problem. However, just buying technology usually does not lead
to extensive reuse. Morisio et al. [Morisio02b] found that:
− Top management commitment is the prerequisite for success.
− Product family practice, common architecture, and domain engineering increase
reuse capability.
− Size, development approach (object-oriented or not), rewards, repository, and
reuse measurement are not decisive factors, while training is.
− The other three factors that are considered to be success factors are reuse process
introduced, non-reuse process modified, and human factors.
− Successful cases tried to minimize change, to retain their existing development
approach, choosing reuse technology to fit that.
Griss [Griss95] summarized the experience from industrial reuse and pointed out
that:
− Reuse needs management support, since reuse involves more than one project.
− Object technologies or libraries give no improvement in reuse.
− Domain stability and experience are often more important for successful reuse
than general process maturity.
− One way to adopt reuse incrementally is to increase your investment and
experience.
14
Frakes et al. [Frakes95] report that most software engineers prefer to reuse rather
than to build from scratch. In addition, they did not find any evidence that use of certain
programming languages, CASE tools, or software repositories promote reuse. On the
other hand, reuse education and a software process that promotes reuse have a positive
impact on reuse. They also found that the telecom industry has higher levels of reuse
than some other fields.
Software reuse includes two interrelated parts: reuse as and reuse with. Reuse as
means creating the asset for future reuse. Reuse with indicates using the reusable
artefacts in the new system.
2.3 Component-Based Development

Component-based development (CBD) and Component-based Software Engineering
(CBSE) facilitate reuse by providing logical units for assembly and make systematic
reuse possible by demanding that components should adhere to a component model.
CBD focuses on software architecture as a guideline to put pieces together, viewing
components as independent units of development and deployment, and on component
models. Developing components is one kind of developing for reuse, while developing
systems of reusable components is developing with reuse [Karlsson95]. Many CBD
concepts originate from reusability and Object-Oriented (OO) approaches.
Components, the heart of CBSE, are defined in various ways depending on the
viewpoint. We introduce several theoretical definitions in this section. The component
definition used in this thesis is shown in Section 2.6.
In the SEI’s report [Bachmann00] on technical aspects of CBD, a component is
defined as:
− An opaque implementation of functionality.
− Subject to third-party composition.
− Conformant to component model.
Heineman and Council [Heineman01] define a software component as: A software
element that conforms to a component model, which can be independently deployed and
can be composed without modification according to a composition standard.
Szyperski’s [Szyperski02] definition is: A software component is an executable unit
of independent production, acquisition, and deployment that can be composed into a
functioning system. To enable composition, a software component adheres to a
particular component model and targets a particular component platform.
What these three definitions have in common are:
− Components are units of independent development and acquisition.
− Components adhere to a component model that enables composition of
components. Composition is the term used for components, instead of
integration.
None of these two aspects are found in OO approach. Some other differences
between component approach and OO approach are:
− Instantiation: components may be instantiated or not, and if instantiated there are
usually not many instances of them [Atkinson02].
15
− Components may have state or not (in order to be replaceable they should not
have state).
− Components are generally considerably larger than individual classes [Bosch00].
Bachman et al. [Bachman00] list the advantages of CBD as:

− Reduced time to market: even if component families are not available in the
application domain, the uniform component abstractions will reduce overall
development and maintenance costs.
− Independent extensions: components are units of extension and component
models prescribe how extensions are made.
− Easy to acquire: there are component markets to acquire components.
− Improved predictability: component frameworks can be designed to support the
quality attributes that are most important.
Advantages added by Bass et al. [Bass00] are:

− Improved productivity, which can lead to shorter time-to-market.
− Separation of skills: complexity is packaged into the component framework and
new roles are added, such as developer, assembler, and deployer.
− Components provide a base for reuse, since components are a convenient way to
package value. They have direct usability (may be used to build systems
directly); while other approaches such as design patterns are more abstract.
Component-based system development is often promoted in the literature as a low

risk development strategy, which provides a simple and rapid mechanism for increasing
the functionality and capability of a system. In reality, CBD carries significant risk
throughout the system life cycle. Bass et al. [Bass00] mention inhibitors in CBD as
being lack of available components, lack of standards for component technology, lack
of certified components, and lack of engineering methods. Some challenges in each
development phase and for a project as a whole are discussed here, using [Crnkovic02],
[Ghosh02], [Jacobson97] and other various sources as:
− Management: decision-making on make vs. reuse vs. acquire, initiating product
families, Return on Investment (ROI), vendor interactions for cost estimates.
Although component markets have grown in the recent years, there are few
empirical studies that can verify increased productivity or shorter time to market
due to acquiring components.
− Component based software life cycle: The life cycle of component-based
software is becoming more complex because many phases are separated in
unsynchronized activities. For example, the development of components may be
completely independent of the development of systems using those components.
The process of engineering requirements is much more complex because the
possible candidate components usually lack one of more features that the system
requires.
− Composition predictability: Even if we assume that we can specify all the
relevant attributes of components, we do not necessarily know how these
16
attributes will determine the corresponding attributes of systems of which they

may become part.
− Trusted components and component certification: One way of classifying
components is to certify them [Voas98a]. In spite of the component belief that
certification means absolute trustworthiness, it in fact merely provides the results
of tests performed and a description of the environment in which the tests were
performed. Although certification is a standard procedure in many domains, it
has not yet been established in software in general especially not for software
components.
− Component configurations: As soon as we begin to work with complex
structures, problems involving structural configurations appear.
− Tools support: The purpose of software engineering is to provide practical
solutions to practical problems. Component evaluation tools, component
repositories and tools for managing the repositories, component test tools, and
component-based design tools, run-time analysis tools, and component
configuration tools and so on.
− Dependable system and CBSE: The use of CBD in safety-critical domains,
real-time systems and different process – control systems, in which reliability
requirements are particularly rigorous, is particularly challenging.
Components can be built in-house, acquired from the third-companies, or just

downloaded from the websites of the OSS communities.
− In-house development
In-house development means that the provider and the consumer of the software
reuse artefacts is the same person or organization. So, a component is a white-
box for the person who may use it. Moreover, the component may be only used
for special application and domain. Thus, it possible that the component is not
based on uniform, standard coordination schemes.
− COTS-based development
Practitioners using COTS software equate components to COTS products.
According to the perspective of SEI [Brownsword00], “A COTS product is:
sold, leased, or licensed to the general public; offered by a vendor trying to
profit from it; supported and evolved by the vendor, who retains the intellectual
property rights; available in multiple, identical copies; and used without source
code modification.” Here, COTS components are mostly executable black-box
components.
− Open Source Software
Open Source Software (OSS) [Hissam01] makes the definition of component
more complex. OSS means that the customers can get the source code of the
software and they may change the source code and give back the modified
artefacts. Thus, the open source component can also be regarded as a white-box.
17
2.4 COTS-Based Development

COTS software is generally a software that is not developed inside the project, but
acquired from a vendor and used “as-is”, or with minor modifications.
Oberndorf [Oberndorf97] defines the term COTS product on the basis of the
“Federal Acquisition Regulations”. It is defined as something that one can buy, ready-
made, from some manufacturer’s virtual store shelf (e.g., through a catalogue or from a
price list). It carries with a sense of getting, at a reasonable cost, something that already
does the job. The main characteristics are 1) it exists a priori, 2) it is available to the
general public, or 3) it can be bought (or leased or licenced).
Vidger et al. [Vidger96][Vidger97] define COTS as pre-existing software products;
sold in many copies with minimal changes; whose customers have no control over
specification, schedule, and evolution; access to source code as well as internal
documentation is usually unavailable; complete and correct behavioral specifications
are not available.
Basili and Boehm [Basili01] specify that COTS has the following characteristics 1)
the buyer has no access to the source code, 2) the vendor controls its development, and
3) it has a nontrivial installed base.
Torchiano and Morisio [Torchiano04] give a more detailed, empirically based
definition as: A COTS product is a commercially available or open source piece of
software that other software projects can reuse and integrate into their own products.
They also specify a COTS product:
− Is not produced in or exclusively for the project.

− Can be closed source or open source. If it is open source software, it is usually
treated as if it were closed.
− Is not a commodity. It is not shipped with the operating system, provided with
the development environment, or generally included in any pre-existing
platform.
− Is integrated into the final delivered system. It is not a development tool.
− Is not controllable, in terms of provided features and their evolution.
COTS software can be classified into different categories. Morisio et al.

[Morisio02a] summarized some of the previous work and gave a proposal on how to
classify COTS. These classification attributes are supposed to be linked with software
process (e.g., cost models, selection methods, architectures, testing, and validation
techniques).
COTS usage promises faster time-to-market and increased productivity [Voas98c].
Developing software with as much COTS functionality as possible saves you from
reinventing the wheel. With COTS software, the functionality you desire can be:
− Accessed immediately.
− Obtained at significantly lower prices than otherwise.
− Developed by someone who is an expert in that functionality.
18
At the same time, COTS-based software introduces many challenges, such as

unknown quality properties of the chosen COTS components can be harmful for the
final product and the economic instability of the COTS vendor may terminate the
maintenance support of its COTS components [Voas98b]. The advantages and
disadvantages of using COTS software are summarized in [Boehm99] as shown in the
Table 2-1.
Table 2-1 Advantages and disadvantages of using COTS software

Advantages Disadvantages
Immediately available; earlier payback Licensing, intellectual property
procurement delays
Avoids expensive development Up-front licensing fees
Avoids expensive maintenance Recurring maintenance fees
Predictable, confirmable licence fees Reliability often unknown or
and performance inadequate; scale difficult to change
Rich functionality Too-rich functionality compromises
usability, performance
Broadly used, mature technologies Constraints on functionality, efficiency
Frequent upgrades often anticipate No control over upgrades and
organization’s needs maintenance
Dedicated support organization Dependence on vendor
Hardware/software independence Integration not always trivial;
incompatibilities among vendors
Tracks technology needs Synchronizing multiple-vendor
upgrades
2.5 OSS-Based Development

The basic idea behind OSS is very simple: When programmers can read, redistribute,
and modify the source code for a piece of software, the software evolves. People
improve it, people adapt it, and people fix bugs. This can happen at a speed that, if one
is used to the slow pace of conventional software development, seems astonishing.
However, open source does not just mean access to the source code. The distribution
terms of OSS must comply with the following criteria, such as free redistribution, open
source code, derived work and so on [OSS04].
Many people prefer the term free software to open source software. The term free
software dates from 1984, when the idea and arguments for it were first published by
Richard Stallman. The idea is not that software should be free “as in beer,” or available
at no charge, but that it should be free “as in speech”, so that you can review it and
change it as you need to [Freesoftware05]. The term open source dates from 1997, when
a group of people, including Eric Raymond, Tim O’Reilly, and Bruce Perens, decided
that the term free software and some of the arguments employed in support of it were
making the idea less attractive to many businesses. They decided, as a marketing
decision, to emphasize technical and practical advantages of open source software rather
than arguments from principle [OSS04].
19
OSS software has risen to great prominence. There is excellent evidence that OSS
has a significant market share in numerous markets, for example,
− Netcraft’s statistics on web servers shows that Apache is the number one web
server with over three times the market share of its next ranked competitor
[Netcraft05]
− A 2004 InformationWeek survey found that 67% of companies use OSS
products with another 16% expecting to use it in 2005, and only 17% have no
near-term plans to support OSS products [Infoweek04].
OSS software offers many choices from large suites to small components. Typical
OSS suites are Linux (operating system), Apache (web server), MySQL (database),
PostgreSQL (database), Eclipse (development platform), and OpenOffice (office suites).
Examples of software components are zlib (compression/decompression Library),
SpamAssassin (spam-checker for email servers), and log4j (logging component). As of
27th April 2006, there are more than 118,615 OSS project registered at sourceforge.net
and more than 40,590 project registered at freshmeat.net. The projects are classified
according to the application domain of the software, such as clustering, games,
networking, and security at sourceforge.net and freshmeat.net.
OSS has many proposed advantages [Fitzgerald04], [Ruffin04] [Madanmohan04]:
− OSS is usually freely available for public download. The collaborative, parallel
efforts of globally distributed developers allow OSS to be developed more
quickly than conventional software.
− Many OSS products are recognized for high reliability, efficiency, and
robustness.
− When OSS is a de facto standard component and has a large user community, it
is a lasting solution that can resist the commercial supplier uncertainties that can
often abruptly end a product’s life
− Using mainstream OSS reduces cycle time for component updates and
corrections. More and often free labour is available to localize and correct
defects. Especially for embedded systems, OSS provides fast, new drivers and
hardware-related features, even for rather exotic environments.
− Another attraction is that the marginal cost of scaling up is zero. OSS does not
require additional licences as installation numbers grow.
Despite its wide appeal, OSS software faces a number of serious challenges and
constraints [Fitzgerald04] [Ruffin04]:
− The popularity of OSS increases the risk that so-called net-negative-producing

programmers will become involved.
− Many software tasks, such as documentation, testing, and field support are
lacking in OSS projects.
20
− If developers produce or sell a product, which integrates OSS as source code,

they may need the licensor’s permission. Otherwise, the licensor might claim for
damages or force them to end the product’s further development, delivery, and
sale.
− Open source systems lack the comfort zone that a commercially acquired
solution provides because support is in the form of bulletin boards and the like.
The studies on OSS focus on three dimensions:
− Since OSS projects have some typical characters, such as parallel development,
fast release, and peer review, some studies investigated how the OSS projects
are organized and managed. The intentions of these studies are to summarize the
good practice in OSS projects in order to improve the development of
commercial software [Feller02].
− Some studies compared the OSS products with COTS products. For example,
Paulson et al. [Paulson04] validated several hypotheses about the difference
between OSS and COTS products. They concluded that there is still no empirical
evidence that OSS fosters faster system growth and there is also no strong
evidence that OSS is more modular than COTS software.
− Other studies concentrate on using OSS in the commercial software projects in
order to facilitate the development in the IT industry [Madanmohan04].
2.6 OTS Component-Based Development

The granularity of the component in practice/marketplace is generally the same or larger
than the theoretical component defined in Section 2.3. The components following COM,
CORBA, and EJB models, or software libraries like those in C++ or Java, have all been
regarded as components in industries and component marketplace
[Componentsource04].
In our study, we investigated the components with the similar granularity as in
industrial practice and component marketplace. However, we exclude platform
software, such as OS and DB. The OTS components definition used in our studies are:
− Component: Software components are program units of independent

production, acquisition, and deployment that can be composed into a functioning
system. We limit ourselves to components that have been explicitly decided
either to be built from scratch or to be acquired externally as an OTS-
component. That is, to components that are not shipped with the operating
system, not provided by the development environment, and not included in any
pre-existing platform. That is, platform (“commodity”) software is not
considered, e.g., an OS like Linux, DBMSes, various servers, or similar
software. Furthermore, components usually follow some component model, as
expressed by the COM, CORBA, EJB, .Net, or Web Service standards, or they
can be a C++/Java library.
21
− OTS (Off-The-Shelf) component is a component that:

• is provided (by a so-called provider) from:
- A commercial vendor (Commercial-Off-The-Shelf, COTS), or
- The Open Source community (Open Source Software, OSS).
The components may come with certain obligations, e.g., payment or
licensing terms. NB: we do not consider proprietary components
developed by some other organization in the same company, i.e., internal
reuse.
• Is not controllable, in terms of provided features and their evolution.
• Is mainly used as closed source, i.e., no source code is usually modified,
even it is available.
Typical COTS components are: Vector draw professional (3D modelling), Shamman
Reports for Excel (Database reporting), Dynamic Cube (OLAP). Typical OSS
components are: Xerces (XML parser), Kiwi Toolkit (Class library), OpenEJB, Python
(Interpreter).
2.7 Summary and Discussion

This chapter shows that software reuse is the systematic practice of developing software
from a stock of building blocks. The components can either be an in-house built
component, a COTS component, or an OSS component. Different kinds of components
have different advantages and challenges. Moreover, the challenges facing reuse and
CBD are structural, organizational, managerial, and technical. Focusing only on the
technological issues usually does not bring the whole benefit of reuse and CBD. In this
thesis, we focus on research challenges as follows:
− How to prepare for reusing more in-house built components?

Component reuse is generally an incremental procedure. The company will build
some reusable components in the beginning. In case of successful reuse, more and
more code will be encapsulated into reusable components. The more reusable
components are developed, the more complex the development process will be, and
more support is required from the organization [Crnkovic01]. Most studies on
software reuse focus on the final results of reusing the in-house built components
without investigating how the reuse has been developed inside an organization. The
challenge is to examine the relationship between companies’ reuse level and some
key factors in component-based development, so that company with low reuse level
can make necessary preparation when moving to a higher reuse level. This research
challenge is formulated into research question RQ1 in this thesis.
− How to manage OTS component-based projects?
The use of COTS or OSS components has both advantages and challenges. Many
researchers have summarized their experience on managing these risks through case
studies in specific domains. One of the limitations of a case study is that the
conclusion is difficult to be generalized if the studies are limited into a specific
context. Therefore, it is necessary to do further empirical studies on managing the
22
challenges in reusing the OTS components in different project contexts, so that the
project manager can efficiently customize their process, organization, and
technology according to their needs. This research challenge is investigated by
research questions RQ2 and RQ3 in this thesis.
− What are the differences between using COTS and OSS components?
Although both COTS and OSS components claim to save development effort, they
are different in their nature and appearance. When project members want to use
OTS components, they must select to either use COTS or OSS. It is therefore
important for them to be able to compare the benefits and risks of using COTS or
OSS components. This research challenge is investigated by research question RQ4
in this thesis.
23
24
Process improvement
3 Process Improvement in OTS Component-

Based Development
This chapter starts by introducing some software development models and the
challenges of process improvement in general. After that, it describes the proposed
processes in OTS component-based development and OTS component selection.
Finally, challenges facing our study are discussed.
3.1 Software Process Improvement (SPI)

A process is the glue that ties together people, technology, organizational structures,
and management in a coherent whole, focusing on the business goals and objectives
[Zahran98]. A process has three aspects. To start with, the process must be defined.
Therefore, the first step is the process definition. Secondly, the process knowledge must
pass to those who will perform it; hence the second aspect is process learning. The third
aspect is the process results as manifested by the products produced as a result of
executing the process activities. A software process is a set of activities that leads to the
production of a software product [Sommerville04]. Although there are many software
processes, some fundamental activities are common to all software processes:
- Software specification: the functionality of the software and constraints on its

operation must be defined.
- Software design and implementation: the software to meet the specification
must be produced.
- Software validation: the software must be validated to ensure that it does what
the customer wants.
- Software evolution: the software must evolve to meet changing customer needs.
A framework proposed for software process improvement comprises four parts

[Zahran98]:
- Software process infrastructure, which includes organization, management, and

technical infrastructure.
- Software process improvement roadmap, which specifies a model for
characterizing the software process and a logical step-by-step approach towards
25
Process improvement
the realization of effective software processes. It could be one of the standard

public process roadmaps, for example the software capability maturity model
(SW-CMM) [CMM06] or maturity levels of ISO/IEC 15504 [ISO06], or a
tailored version to match the organizations’ own needs.
- Software process assessment method, which specify methods and techniques for
assessing the organizations’ current software process, practice, and
infrastructure.
- Software process improvement plan, which involves transforming the
assessment findings into specific actions for software process improvement.
There are three main types of processes: management, engineering, and support.
Within a software project, key processes could be classified in three major process
classes as follows:
- Management processes
- Software engineering (life cycle) processes
- Support processes (life cycle support)
A software process model is an abstract representation of a software process life

cycle [Sommerville04]. Each process life cycle represents a process from a particular
perspective, and thus provides only partial information about that process. The
traditional process models are: Code and Fix, Waterfall, V-Model, Spiral, Evolutionary
Prototyping, and Adaptive Software Development.
- Code and Fix: Developers begin work without a set of good requirements or a
vision, writing code as long as there is time and money. This approach has no
planning and therefore yields early results. High maintenance and rework effort
are the result of this model.
- Waterfall: This model is a sequential, document-driven methodology. To
advance from the current phase to the next phase, the project team must review
and release an artefact in the form of a document [Royce87].
- V-model: This model is an extended waterfall model, adding more details on the
validation and verification side. For each phase in development, there is an
associated phase to verify and validate the result produced [V-Model05].
- Spiral: The spiral model breaks the project down into risk-oriented subprojects
[Boehm88], [Boehm98]. The advantage of this model is the breakdown of
development into small pieces, so that the areas of risk can be tackled at the
beginning of the project.
- Evolutionary Prototyping: In this model, the system starts with an initial idea or
proposal, which is then prototyped and released to the customer in incremental
releases based on feedback.
- Adaptive Software Development: The adaptive software development model is
an iterative, risk- and mission- driven, component-based, time-boxed, and
change-tolerant process [Highsmith99]. It works well for small teams (four to
eight people) when the requirements are uncertain or domain knowledge is
missing.
26
Process improvement
The composed process models, which cover several aggregations, are:
- Unified Process: The author of the Unified Modelling Language (UML) defined
a software process model called the Unified Process (UP) [Jacobson99]. The UP
is a risk- and use-driven, architecture-centric, iterative and incremental software
development model. The Rational Unified Process (RUP) [Kruchten00] is a
commercial extension of the UP.
- Extreme Programming: The eXtreme Programming (XP) model [Kent99] is a
flexible, lightweight, people- and result-oriented development process that allows
for requirement changes at any time during development. XP defines 12 core
practices that are essential to the success of an XP project, such as test-driven
development, planning game, on-site customer, pair programming, and so on.
A software development process model should not be dumped blindly onto the
project but rather should be carefully chosen according to the project, the environment,
and the goals. In addition, researchers and practitioners have realized that processes
cannot be defined and frozen once for all. Processes need to continuously undergo
changes and refinements to increase their ability to deal with the market expectations
and requirements of the company stakeholders. Hence, process need to be continuously
assessed and improved [Fuggetta00].
3.2 SPI of OTS Component-Based Development

Although an OTS component can be either COTS or OSS component, most process
improvement studies focus on COTS-based development. Typically, a COTS-based
process consists of four phases [Abts00], comprising:
- COTS component assessment and selection

- COTS component tailoring
- COTS component integration
- Maintenance of COTS and non-COTS parts of the system
3.2.1 Improvement of the Whole Life Cycle

There is consensus that the use of a COTS component implies changes in the software
process [Brownsword00]. Most studies of a COTS-based development process focus on
two dimensions: The whole software development life cycle and process of a specific
phase, especially in COTS component selection and evaluation
Boehm and Abts [Boehm99] regard both the waterfall model and evolutionary
development as unsuitable for COTS-based development because:
- In the sequential waterfall model, requirements are identified at an earlier stage

and the COTS components are chosen at a later stage. This increases the
likelihood of the COTS components not offering the required features.
- Evolutionary development assumes that additional features can be added if
required. However, COTS components cannot be upgraded for one particular
27
Process improvement
development team. The absence of source code bars the development team from
adjusting the COTS components to their needs.
Based on the above arguments, Boehm and Abts [Boehm99] proposed that
development models which explicitly take risk into account are more suitable for
COTS-based development than the traditional waterfall or evolutionary approaches.
The Software Engineering Institute has a large ongoing effort to address the
development of COTS-based systems, and they have developed the Evolutionary
Process for Integrating COTS-based Systems (EPIC) [Albert02]. The proposed process
integrates COTS-related roles and activities into a RUP process. The iterative and
evolutionary nature inherent in this process allows developers to adjust the architecture
and system design, as more knowledge is gained about the operations of the COTS
components.
The National Aeronautic and Space Administration (NASA) has been developing IT
systems with COTS for many years. Their experience has been captured by Morisio et
al. [Morisio00]. They have investigated the various processes that were used across 15
projects at NASA. Based on their insight, they have developed a COTS-based
development process that was the most representative for the processes used in the
projects.
3.2.2 Improvement of the OTS Component Selection

To perform a successful OTS component-based project, one of the key issues is to select
the right OTS component. Most previous studies focus on proposing selection processes
and methods to select COTS components. One kind of such processes is based on direct
assessment [Leung02], such as MBASE (Model Based Architecting and Software
Engineering) [Boehm00], OTSO (Off-the-Shelf-Option) [Kontio96], CISD (COTS-
based Integrated System Development) [Tran97a], PORE (Procurement-Oriented
Requirement Engineering) [Maiden98], CAP (COTS Acquisition Process) [Ochs01],
IIDA (Infrastructure Incremental Development Approach) [Fox97], CARE (COTS-
Aware Requirements Engineering) [Chung02], RCPEP [Lawlis01], and CRE (COTS-
Based Requirements Engineering) [Alves03]. In general, the direct assessment process
includes three basic steps:
- Inspect all modules of each of the available COTS components to check whether
they satisfy some or all of the functional requirements of the COTS-based system
being developed.
- Check whether a COTS component also satisfies the non-functional requirements
of the COTS-based system. Non-functional requirements may include properties
such as the interoperability of the modules of the COTS product with other
systems.
- Compare the COTS component candidates and select the most appropriate COTS
component that satisfies both the functional and non-functional requirement of
the COTS-based system.
Some of the direct assessment processes, such as OTSO [Kontio96], CAP [Ochs01],
and CISD [Tran97a], assume that the requirements are fixed and select the COTS
components by comparing how well the COTS component candidates satisfy the
28
Process improvement
requirements. A formal decision-making process is usually used to select the best COTS
component [Ncube02]. The formal decision-making process usually includes three basic
elements: selecting evaluation criteria (factors), collecting and assigning values to these
criteria, and applying formal decision-making algorithms such as MAUT
[Maccrimmon73], MCDA [Morisio97], and AHP [Saaty90]. Other direct assessment
processes, such as MBASE [Boehm00], PORE [Maiden98], IIDA [Fox97], CARE
[Chung02], RCPEP [Lawlis01], and CRE [Alves03] emphasize the trade-offs between
the requirements and COTS component functionalities. These processes do not assume
that the requirements are fixed and should not be changed. On the other hand, these
processes propose that the requirements in COTS-based development should be flexible
enough so that they can be negotiated or changed according to the available
functionalities of the COTS components.
Another kind of process is based on a domain-model [Leung02]. It includes a set-up
phase and a selection phase. In the set-up phase, the vendors need to map their COTS
modules to those modules of the domain that they find are applicable. In the selection
phase, the corresponding modules in a domain model are identified for each of the
modules of the COTS-based system in question. Then, the COTS modules that claim to
be applicable are identified by the mapping from the domain models to the COTS
modules. After that, the non-functional properties of the identified COTS modules are
assessed. In the end, the most appropriate COTS modules are selected with reference to
all of the assessment results.
After performing seven structured interviews of small and medium software
companies in Norway and Italy, Torchiano and Morisio [Torchiano04] proposed two
these T4 and T5 that reflect the state-of-the-practice of SPI in COTS-based
development.
- T4: Developers seldom use formal selection procedures. Familiarity with either
the product or the generic architecture is the leading factor in selection.
- T5: Architecture is more important than requirements for product selection.

This chapter has discussed the software development process improvement in general. It
has also briefly introduced the proposed process improvement in COTS-based
development. Finally, several COTS selection processes have been introduced. Since
both the COTS and OSS components are acquired from third-party companies, the users
usually have no control of the provided functionality and evolution of these
components. Therefore, we generally believe that most proposed process improvement
in COTS component-based development can also be used in OSS component-based
development.
The limitations of current studies are that few of them have been empirically
verified. In this thesis, we focus on answering:
− What are the state-of-the-practice of OTS component-based development
processes and OTS component selection processes?
Since most proposed COTS-based development processes are either based on
military projects or very large projects, it is difficult for small and medium-sized
29
Process improvement
projects to follow their proposals. It is therefore necessary to summarize the

relationship between the results of the project and the process improvement
activities performed. The intentions is to give guidelines about how to effectively
improve the development process to get better project outputs, such as better quality
products and shorter effort spent on development and maintenance. This research
challenge is formulated into research question RQ2 in this thesis.
30
Risk management
4 Risk Management in OTS Component-

Based Development
This chapter introduces risk management theory in software development projects.

Then, it describes the particular risks and risk management activities in OTS
component-based development. Finally, the challenges facing this thesis are discussed.
4.1 Risk Management in Software Development

Risk management is increasingly seen as one of the main jobs of project managers. It
involves anticipating risks that might affect the project schedule or the quality of the
software being developed and taking action to avoid these risks [Hall98] [Ould99].
Risk management includes two primary steps each with three subsidiary steps as
described by Boehm [Boehm91] and shown in Figure 4-1.
Figure 4-1 Risk management framework
Risk identification produces lists of project-specific risk items likely to compromise

the success of a project. Risk analysis assesses the loss probability and loss magnitude
31
Risk management
for each identified risk item. Risk prioritization ranks the risk items identified and
analysed. Risk management planning helps prepare to address each risk item. Risk
resolution produces a situation in which the risk items are eliminated or otherwise
resolved. Risk monitoring involves tracking the progress of a project towards resolving
its risk items and taking corrective activities where appropriate.
In order to identify risks, a list of possible project risks must be produced. Boehm
[Boehm89a] lists the top 10 risks by probing several large software projects and their
common risks. Barki et al. [Barki93] identified 35 variables, which are subsequently
used as a basis for creating scales in a project risk assessment instrument. The Software
Engineering Institute (SEI) [Carr93] has contributed significantly to the literature with
the SEI Taxonomy-Based risk identification instrument, which contains 194 questions.
However, the notion of building single all-encompassing risk taxonomy to be globally
used by all software developers is probably unrealistic. Thus, we may need different
risk taxonomies in different project contexts [Moynihan97].
A few other studies have classified software risk items, i.e., consider software risks
along several dimensions and have provided some empirically founded insights of
typical software risks and their variation. The risks can be classified into technical,
organizational, and environmental [Sherer95]. The possible risks can also be
summarized into a hierarchical holographic modelling (HHM) framework
[Chittister96]. Ropponen and Lyytinen [Ropponen00] empirically delineate six
components of software development risks, such as scheduling and timing risks, system
functionality risks, subcontracting risks, requirement management risks, resource usage
and performance risks, and personnel management risks. Keil et al. [Keil98] extracted
the experience of more than 40 software project managers and identified a universal set
of risk factors, such as customer mandate, scope and requirements, environment, and
execution. A following study [Wallace04] investigated the relationship between these
factors and their effect on the result of the project.
During the risk analysis process, project managers have to consider each identified
risk and make a judgment about its probability to occur and seriousness. The project
managers can tabulate the results of this analysis process using a table ordered
according to the serious of the risk [Sommerville04]. The estimation of the seriousness
and the rank of the risk should be combined with cost estimation [Madachy97],
[Kansala97]. Once the risks have been analysed and ranked, the project managers need
to assess which are the most significant risks.
The purpose of identifying and analysing the risks is to manage these possible risks.
The possible risk management activities have been proposed from processes
[Boehm88], organization [Roppoen00], [Gemmer97], and technology [Hecht04]
viewpoints. However, many researchers advocate rather than investigate. As a result,
some advocated concepts are worth less than their advocates believe and there is a
shortage of evaluative research to help determine the actual value of new tools and
techniques [Glass01]. Although risk management proposals of the in-house software
development have been verified by a survey-based research [Verner05], few studies
have been performed to empirically investigate the risk management proposals in using
the OTS components.
32
Risk management
4.2 Risk Management in OTS Component-Based

Development
In addition to the classical risks associated with developing large systems, OTS
components requires managers to modify their typical mitigation activities for some of
the classic risks and to develop new mitigation activities for risks that are particular to
the use of OTS components in a system.
Vigder et al. [Vigder96] investigated the specific risk issues in the COTS-based
development and point out the fundamental differences between COTS-based
development and building in-house are:
- COTS-based development may need infrastructure earlier to demonstrate and

validate the COTS component.
- The COTS component may dictate standards, architecture, and design.
- The COTS component may influence work flow.
- Picking the wrong COTS component may be more expensive than fixing
problems in the in-house built software.
- Issue resolution processes need to be in place earlier to resolve COTS component
issues.
- Issue resolution processes may be more complicated because of the addition of
the vendor and possible incompatibilities between the vendor’s practices and the
users.
Boehm et al. [Boehm03] summarized experience from their COTS-based university

projects. Examples of the summarized risks and corresponding risk management
activities (in brackets) are:
- Requirement changes and mismatches (prototyping, win-win negotiation among

all stakeholders).
- Overly optimistic expectation on COTS quality attributes (significant quality
features must be tested).
- Overly optimistic COTS component learning curve (a most likely COTS
component learning curve must be accessed during planning and scheduling).
- Version upgrade may result in retailoring of the COTS component (ensure the
new version still can implement the capability of the old version).
Torchiano and Morisio [Torchiano04] discovered several phenomena about risks and
risk management in COTS component-based development and proposed theses T2, T3,
and T6 as:
- T2: Integration problems result from lack of compliance with standards.

Architecture mismatches constitute a secondary issue.
- T3: Custom code mainly provides additional functionalities.
- T6: Integrators tend to influence the vendor on product evolution whenever
possible.
33
Risk management
The software engineering resource centre at the Federal Aviation Administration

(FAA) [FAA03] summarized 10 risk factors in COTS-based development and proposed
corresponding risk mitigation activities. The COTS lessons-learned in a repository made
by Basili et al. [Basili02] summarized the possible risk management activities based on
case studies. In these studies, COTS software can be components or very large
packages, such as ERP (Enterprise Resource Planning), CRM (Customer Relationship
Management), Data Bases, or Operating Systems.
A few other studies contribute to investigate risk management in OSS-based
development. For example, Ruffin and Ebert [Ruffin04] point out the possible licensing
problem of using OSS components. Fitzgerald [Fitzgerald04] mentioned that
organizations should not expect maintenance and support to be available at a lower cost
that would apply for commercial software, even though OSS may be available at little
or no cost. Other researchers mention that freely available source code addresses two
typical concerns with using COTS components: unknown implementation quality and
long-term vendor support [Madanmohan04].
Other researchers have focused only on the component-based development without
telling the differences between in-house built, COTS, or OSS components [Bass00],
[Crnkovic02], [Ghosh02], [Jacobson97]. Kotonya and Awais [Kotonya01] put risk
management in component-based development into the risk-weighting scheme of
[Moynihan97] and listed several questions in each risk item to help project managers to
quantitatively assess the risks.
Some studies focus only on COTS components. Rose [Rose03] classifies possible
risks in COTS component-based development based on the different phases. Examples
of possible risks and corresponding risk management activities (in brackets) are:
- Lack of skilled personnel (ensure that there is an appropriate number of senior

technical personnel assigned to the project).
- Architecture mismatch (design the architecture first and select COTS component
that best fits the architecture).
- Product obsolescence (maintain a continual watch on the marketplace looking for
substitute products as well as new technologies that might replace older ones).
- COTS components have asynchronous update cycles with the COTS-based
system (involve senior analysts directly involved in the analysis of the upgrades
and derive effective field upgrade procedure and schedules)
The CBSEnet project [CBSEnet04] investigated the state-of-the practice in COTS

component-based development and summarized the challenges from business,
information technology, and legal perspectives.
Our pre-study (see paper P2 [Li06a] in Appendix A) summarized the problems
encountered and presented good practices from finished COTS component-based
projects. Examples of the problems encountered are:
- Could not negotiate requirements with the customer.

- The learning effort was underestimated.
- It was difficult to locate defects.
- The COTS component could not be deployed as advocated by the vendor.
34
Risk management
Examples of good practices are:
- Integrate unfamiliar COTS component first.

- Do integration testing early and incrementally.
- Investigate COTS component market and prepare to replace obsolescent
components.
Since the OTS components involve both COTS and OSS components, risks and risk
management activities relevant to OTS components-based development are therefore
the subset of risks and risk management in component-based development, COTS-
based, and Open Source based development, as shown in Figure 4-2.
Figure 4-2 Risk management in OTS component-based development

This chapter has discussed risk management in software projects in general. Due to the
specific attributes of OTS components, the OTS component-based project is more risky
than a traditional software development project. The proposed risk identification and
risk management activities are briefly introduced. The relevant research challenges in
this thesis are:
− What are the results of the proposed risk management activities?

Although many risk management activities have been proposed to avoid the typical
risks in OTS component-based development, they are basically based either on
common sense or few case studies. It is therefore necessary to empirically verify the
effect of performing such risk management activities in projects with different
profiles. This research challenge is formulated into research question RQ3 in this
thesis
35
Risk management
− What are the differences of risks in COTS component-based development

and OSS component-based development?
As we have shown earlier, there are both similarities and differences between the
risks in COTS and OSS component-based development. It is therefore important to
compare what had happened in finished COTS and OSS component-based projects.
The intention is to help project managers to compare the possible benefits and risks
of buying the COTS components, or downloading the OSS components. This
research challenge is formulated into research question RQ4 in this thesis.
36
Research methods
5 Research Methods
This chapter provides a brief review of empirical research approaches and strategies in
the empirical software engineering field. Since the survey method has been mainly used
in several studies in this thesis, the advantages, disadvantages and guidelines for using
this method are discussed. The research designs for each study in this thesis are
described. Validity threats for all studies and in particular how to overcome these in our
studies are further discussed.
5.1 Research Strategies in Empirical Software Engineering

Empirical research is research based on the scientific paradigm of observation,
reflection and experimentation as a vehicle for the advancement of knowledge
[Endres03]. Empirical studies may have different purposes, being exploratory
(investigating parameters or doing a pre-study to decide whether all parameters of a
study are foreseen), descriptive (finding distribution of a certain characteristics), or
explanatory (investigating why certain phenomena happen).
There are three types of research paradigms that have different approaches to
empirical studies [Wohlin00], [Creswell94], [Creswell03], [Seaman99]:
- Qualitative research is concerned with studying objects in their natural setting. A

qualitative researcher attempts to interpret a phenomenon based on explanations
that people bring to them [Denzin94].
- Quantitative research is concerned with discovering causes noticed by the
subject in the study, and understanding their view of the problem at hand. A
quantitative study is mainly concerned with quantifying a relationship or to
compare two or more groups [Creswell94]. The quantitative research is often
conducted through setting up controlled experiments or collecting data through
case studies or surveys.
- The mixed-method approach is evolved to compensate for limitations and biases
of the above strategies, seeking convergence across other methods. The
combination of quantitative and qualitative methods is usually more fruitful than
either in isolation [Seaman99]. How to combine the qualitative and quantitative
method in the design is described by [Basili86] and further discussed in
[Seaman99].
37
Research methods
Depending on the purpose of the evaluation, whether it is techniques, methods, or

tools, and depending on the conditions for the empirical investigation, the empirical
research strategies can be classified into different categories. Zelkowitz and Wallace
[Zelkowitz98] summarized 12 technology validation models and grouped these models
into categories according to the data collection methods: observational, historical, and
controlled. The validation models in [Zelkowitz98] include models to examine both the
projects (e.g., case study, project monitoring, and field study) and products (e.g., static
analysis, simulation, and dynamic analysis).
In the empirical software engineering field, three major different types of
investigations are carried out frequently.
- Survey. A survey is often investigations performed in retrospect, when, for

example, a tool or technique, has been in use for a while [Pfleeger94]. The
primary means of gathering qualitative or quantitative data are interviews or
questionnaires. The results from the survey are then analysed to derive
descriptive and explanatory conclusions. They are then generalized to the
population from which the sample was taken. Surveys are discussed further in
[Robson02] [Fowler01].
- Case study. Case studies are used for monitoring projects, activities or
assignments. The case study is normally aimed at tracking a specific attribute or
establishing relationships between different attributes. Case studies are discussed
further in [Yin03].
- Experiment. Experiments are normally done in a laboratory environment, which
provides a high level of control. When experimenting, subjects are assigned to
different treatments at random. The object is to manipulate one or more variables
and control all other variables at fixed levels. The effect of the manipulation is
measured. Based on the measurement statistical analysis can be performed. A
case study is an observation study while the experiment is a controlled study
[Zelkowitz98]. Experiment design is discussed further in [Pfleeger94].
Some of the research strategies could be classified both as qualitative and

quantitative strategies depending on the design of the investigation [Wohlin00], as
shown in Table 5-1.
Table 5-1 Qualitative vs. quantitative in empirical strategies

Strategy Qualitative/Quantitative
Survey Both (mostly quantitative)
Case study Both
Experiment Quantitative
The important question in research design is when to use each strategy. If the
problem is identifying factors that influence an outcome or test the effect of some
manipulation, quantitative approaches are chosen. If the problem is to understand why
the results are as they are or to identify causes, a qualitative approach is better. The
mixed method approach uses different methods in the various phases of a study. Yin
[Yin03] answers that question of choosing an approach by listing three conditions:
38
Research methods
- The type of research question posed. How and why questions are explanatory,
and usually should be studied over time in replicated experiments or case studies.
What, who, where, how many, or how much questions asks about the frequency
or describe the incidence of a phenomenon.
- The extent of control. Only in experiments, the researcher can control treatments
or behavioural events. In a case study, the researcher cannot control treatment,
but may control the measures to be collected.
- The focus. The experiment or case study focuses on the contemporary as opposed
to the surveys, which focuses more on the historical events.
5.2 Survey Methods

The general objectives for conducting a survey are one of the following [Babbie90]:
- Explorative. Explorative surveys are used as a pre-study to a more thorough

investigation to assure that important issues are not foreseen. Creating a loosely
structured questionnaire and letting a sample from the population answer it could
do it. The information is gathered and analysed, and the results are used to
improve the full investigation.
- Descriptive. Descriptive surveys can be conducted to enable assertions about
some population. This could be determining the distribution of certain
characteristics or attributes. The concern is not about why the observed
distribution exists, but instead what the distribution is.
- Explanatory. Explanatory surveys aim at making explanatory claims about the
population. For example, when studying how developers use certain inspection
techniques, we might want to know why some developers prefer one technique
while the others prefer another.
The two most common data collection methods for survey are questionnaires and
interviews [Babbie90]. Questionnaires could both be provided in paper form or in some
electronic form, for example, email or internet. Comparing with the questionnaire, the
advantages of interviews are:
- Interview surveys typically achieve higher response rates than, for example, a
mail survey.
- An interviewer generally decreases the number of “do not know” and “no
answer”.
- It is possible for the interviewer to observe and ask questions.
The disadvantage of an interview is that it usually costs more effort than a

questionnaire. However, the interview can include several open questions to help to get
new insights on the research question.
Interviews come in three types. In [Lincoln85], a structured interview is described as
one in which “the questions are in the hands of the interviewer and the response rests
with the interviewee,” as opposed to an unstructured interview in which the interviewee
is the source of both questions and answers. In an unstructured interview, the object is
to elicit as much information as possible on a broadly defined topic. The interviewer
39
Research methods
does not know the form of this information ahead of time, so the questions asked must
be as open-ended as possible. In a structured interview, the interviewer has very specific
objectives for the type of information sought in the interview, so the questions can be
fairly specific. Another form of the interview is the semi-structured interview. These
interviews include a mixture of open-ended and specific questions, designed to elicit not
only the information foreseen, but also unexpected types of information. One useful tool
to gather the data during an interview is the interview guide [Taylor84]. It usually
consists of a list of close questions and open-ended questions, possibly with some notes
about the direction in which to steer the interview under different circumstances.
To collect the data of the survey, samples should be selected from the target
population. The basic idea of sampling is that by selecting some of the elements in a
population, we may draw conclusions about the entire population. The sample selection
methods should consider [Cooper03]:
- What is the relevant population?
- What are the parameters of interests?
- What is the sampling frame?
- What is the type of the sample?
- What size sample is needed?
- How much it will cost?
The sample can be selected on a probability basis, which is called probability

sampling. The probability sampling is a controlled procedure that assures that each
population element is given a known non-zero chance of selection. Several strategies,
such as simple random, systematic, stratified, cluster, and double, can be used in the
probability sampling [Cooper03]. In contrast, non-probability sampling is arbitrary
(non-random) and subjective. Each member does not have a known non-zero chance of
being included. Typical non-probability sampling methods include convenient
sampling, purposive sampling, and snowball sampling [Cooper03].
The result of the survey can be analysed either qualitatively or quantitatively. In the
interview with open-ended questions, the analysis method can be constant comparison
[Glaser67], as cross-case analysis [Eisenhardt89]. For the closed questions in the
interview or questionnaire, the data analysis will usually use statistics methods. In order
to present the result of the survey, the profile should be described and the details of the
data analysis methods should be specified [Kitchenham02].
5.3 Research Design
5.3.1 Research Design of the Preliminary Study

The purpose of the preliminary study (see paper P1 [Li04a] in Appendix A) is to
investigate RQ1 (What is the status of component-based development based on in-house
built components?).
To reuse in-house components successfully, developers must follow three basic steps
[Mili95]:
- Step 1: Formulate the requirements in a way that supports retrieval of potentially
useful reusable components.
40
Research methods
- Step 2: Understand the retrieved components.

- Step 3: If the retrieved components are sufficiently close to the needs at hand and
are of sufficient quality, then adapt them.
From these steps, we have selected several key factors. For step 1, we focus on the
efficiency of component-related requirements (re)negotiation and the value of
component repository. For step 2, we study how knowledge about components can be
transferred from a component provider to a component user. For step 3, we focus on
definition and reasoning of quality attributes for components.
The sub-questions and hypotheses are as following. Detailed rationale for these sub-
questions are in Section 3.1 of paper P1 [Li04a] in Appendix A.
- RQ1.1: Does requirements (re)negotiation for in-house components really work

as efficiently as people assume?
- RQ1.2: Does the efficiency of component-related requirements (re)negotiation

increase with more in-house built components available?
Concerning the RQ1.2, the null hypothesis H01 and alternative hypothesis HA1
are:
o H01. There is no relationship between the companies’ reuse level and the
efficiency of component-related requirements (re)negotiation.
o HA1. There is a positive relationship between the companies’ reuse level and
the efficiency of component-related requirements (re)negotiation.
- RQ1.3: Does the value of component repository increase with more reusable
components available?
are:
o H02. There is no relationship between the companies’ reuse level and the
value of component repository.
o HA2. There is a positive relationship between the companies’ reuse level and
the value of component repository.
- RQ1.4: How can a component user acquire sufficient information about relevant
components?
- RQ1.5: Does the difficulty of component documentation and component

knowledge management increase with increased reuse level?
are:
o H03. There is no relationship between the companies’ reuse level and
developers’ satisfaction with component documentation.
o HA3. There is a negative relationship between the companies’ reuse level
and developer’ satisfaction with component documentation.
41
Research methods
- RQ1.6: Do developers trust the quality specification of their in-house built

components? If the answer is no, how can they solve this problem?
In this study, we used a questionnaire to collect the data. The questionnaire included
five parts. The questions in the first part were used to investigate the reuse level of the
companies. The definition of reuse level in this study is the number of reused
components vs. the number of total components in the organization. The other four parts
were organized based on the four key factors. Each question in the questionnaire was
used to study one or more research questions. The details of questions are given in the
Table 5-2. The correspondences between the questions in the questionnaire and research
questions are shown in Table 5-3. To increase the reliability of our survey, the
questionnaire also included the definition of concepts used in the questionnaire, and the
questions about the respondents’ personal information.
Table 5-2 Questionnaire used in the preliminary study

Reuse level
Q1. What is the reuse level in your organization?
Q2. To what extent do you feel affected by reuse in your work?
Component-related requirements (re)negotiation
Q3. Are requirements often changed/ (re)negotiated in typical development
projects?
Q4. Are requirements usually flexible in typical projects?
Q5. Do the component-related requirements (re)negotiation processes work
efficiently in typical projects?
Value of component repository
Q6. Would the construction of a reuse repository be worthwhile?
Component understanding
Q7. Do you know the architecture of the components well?
Q8. Do you know the interface of the components well?
Q9. Do you know the design rules of the components well?
Q10a. Is the existing design/code of reusable components sufficiently
documented?
Q10b. If the answer of Q10a is ‘sometimes’ or ‘no’, is this a problem?
Q10c. If the answer of Q10a is ‘sometimes’ or ‘no’, what are the problems with
the documentation?
Q10d. If the answer of Q10a is ‘sometimes’ or ‘no’, how would you prefer the
documentation?
Q10e. What is your main source of information about reusable components
during implementation?
Q10f. How do you decide whether to reuse a component ‘as-is’, ‘reuse with
modification’ or ‘make a new one from scratch’?
Quality attributes specification of components
Q11. Are specifications the quality attributes of components well defined?
Q12. Do you test components after modification for their quality attributes
before integrating them with other components?
42
Research methods
Table 5-3 Correspondence between questions and RQs in the preliminary study
RQ1.1 RQ1.2 RQ1.3 RQ1.4 RQ1.5 RQ1.6
Q1-Q2 X X X
Q3-Q5 X X
Q6 X
Q7-Q10f X X
Q11-Q12 X
For research questions RQ1.1, RQ1.4, and RQ1.6, the results were analysed by
examining their distribution. Since the research questions RQ1.2, RQ1.3, and RQ1.5
are related to the reuse level of the companies, the results were analysed with Spearman
Rank Correlation Coefficient in SPSS version 11.0.
5.3.2 Research Design of the Pre-study

The intention of the pre-study (see paper P2 [Li06a] in Appendix A) is to do an
exploratory study on RQ2 (What were the development processes and OTS component-
selection processes used in OTS component-based development projects?) and RQ3
(Which risk management activities have been performed and the result of performing
them?)
Two sub-questions RQ2.1 and RQ2.2 examine RQ2 and the other, i.e., RQ3.1,
examine the RQ3. The background and rationale for these sub-questions are in Section
3 of Paper P2 [Li06a] in Appendix A.
- RQ2.1: What was the actual process used in the projects using COTS
components?
- RQ2.2: What are the commonalities and possible variations in COTS-based
development processes?
- RQ3.1: What are process scenarios (variations) of the projects using COTS
components successfully and not successfully? What are possible problems and
good practices in different process scenarios?
Another intention of this study is to clarify research questions to test the newly
published theses in COTS-based development. The study performed by Torchiano and
Morisio [Torchiano04] with seven interviews reveals six theses, which contradicted
widely accepted (or simply undisputed) insights in COTS-based development. Since
the sample size of that study was small, it is necessary to investigate the six theses in a
larger and randomized sample. To test the theses, we decided to collect more data using
a randomized sample. Our initial plan was to transfer the theses directly into hypotheses
and make a quantitative survey with standardized questions in a formal questionnaire.
We designed six fixed hypotheses according to the theses and a preliminary
questionnaire. After two rounds of pre-testing, we discovered that it was difficult to
reliably investigate the six theses without a pre-study to clarify our research questions.
We therefore decided to resort to structured interviews, i.e., a qualitative pre-study,
before using a larger and randomized sample.
We used an interview guide (see Appendix C) to collect data. The interview guide
includes both closed and open-ended questions. The closed questions were used to
43
Research methods
solicit information on project context. For example, one question used the application
domains of the final system (product), which was based on the North American
Industrial Classification System (NAICS) [NAICS04]. The open-ended questions were
used to gather information on actual COTS-based development processes. The
questions covered the main life cycle process, the process changes due to using COTS
components, and the experience and lessons learned in the whole COTS-based
development process.
Interviews and data collection were conducted from Nov. 2003 to Feb. 2004. The
contacted 34 IT companies were mostly involved in two Norwegian R&D Projects
[INCO00], [SPIKE002] and one EU R&D project [FAMILIES03].
A pre-process was used to select relevant COTS-based projects. Contact people (IT
managers in the most cases) in the above projects were first contacted by phone and
email, using an introductory letter with our definition of COTS component and project
(the same as the definitions in the interview guide). They were asked to check if they
have a relevant COTS-based project. During this pre-process, 17 companies (among 34)
did not match our definition of a COTS-based project, and therefore could not join the
study. Of these 17 “rejected” companies, 13 used only in-house built components and 3
used only OSS components. There was one other company that could not participate
because of confidentiality issues.
For the remaining 17 companies, 13 of these volunteered to join the study. The
contact people in these 13 companies were asked to recommend people with most
experience on the relevant projects. In all, 16 people were recommended from these 13
companies. For each recommended respondents, they were asked to select one relevant
COTS-project. If a respondent claimed that he/she had experience from several COTS-
based projects, the respondent was asked to select one that he/she was most familiar
with. The whole process to find a suitable respondent and project in a prospective
company, and later agree upon time and place for a physical interview took weeks.
The interview guide was sent to respondents a few days before the personal
interview. In this way, the respondents were well-prepared and could have time to
remember the details of the projects and find some relevant documents. Each interview
was conducted by one of the two interviewers (none with two interviewers), took from
60 to 80 minutes, and was recorded.
The first step in the data analysis was to listen to the tape, and supplement and
transcribe the interview “as-is” into a field note. After that, we used different analysis
procedures based on the purpose of the different research questions.
The purpose of research question RQ2.1 is to investigate the actual development
processes used in the project. The data was analysed by the constant comparison method
[Strauss98].
The purpose of research question RQ2.2 is to see the similarities and differences
between process changes in the studied projects. The main analysis method used is
cross-case analysis [Strauss98].
The purpose of research question RQ3.1 is to summarize the process scenarios of
the successful and unsuccessful projects. We first grouped projects into two groups
based on the positive or negative effect of the COTS components regarding time-to-
market and general quality of the system. In each group, we divided projects into sub-
groups based on their main actual processes used. For each sub-group, we performed a
similar cross-case analysis [Strauss98], as for research question RQ2.2.
44
Research methods
Detailed processes of data analysis of RQ2.1, RQ2.2 and RQ3.1 are in Section 3.6
of Paper P2 [Li06a] in Appendix A.
5.3.3 Research Design of the Main Study
The research focus of the main study (see papers P3 [Li05a], P4 [Li06b], P5 [Li06c],
and P6 [Li06d] in Appendix A) includes four parts:
5.3.3.1. Part one: Verify our conclusions about RQ2 (What were the development
processes and OTS component-selection processes used in OTS component-based
development projects?).
Based on our findings in the pre-study, we designed sub-questions RQ2.3 to RQ2.8.

The background and rationale for these sub-questions are in Sections 2.1 and 2.2 in
Paper P4 [Li06b] in Appendix A.
- RQ2.3: What were the actual development processes in OTS-based projects?

- RQ2.4: Was the actual development process decided before the make vs.
acquire decision or after the make vs. acquire decision.
- RQ2.5: Who decided the actual development process?
- RQ2.6: What was the actual selection process used to select one of the OTS
components (we asked for information about the component, named Comp. 1,
that provides the most functionalities)?
- RQ2.7: When was the Comp. 1 selected?
- RQ2.8 What was the relationship between the selection phase and the project
context, such as the familiarities with the Comp. 1 and the importance of the
Comp. 1?
As research questions RQ2.3 to RQ2.7 focus on state-of-the-practice, we just use

statistical graphs to show the current trends. Since RQ2.8 is to examine the correlations
between variables, statistical methods, such as Spearman correlation and Mann-
Whitney, are used. Detailed information of the design of the data analysis is shown in
Section 4 of Paper P4 [Li06b] in Appendix A.
5.3.3.2. Part two: Investigate our findings from RQ3 (Which risk management
activities have been performed and what are the results of performing them?).
In this study, we have used the conceptual measurement framework proposed by Bush
and Fenton [Bush90]. The study includes one entity (i.e., the project) and two attributes
(i.e., possible risks of the project and possible risk mitigation activities to avoid these
risks). There are also two relationships, i.e., risks are related to a project and risk
mitigation activities that IMPACTS risks. Before the start of a project, project managers
must upfront estimate how often the risks may happen, how much effort they need to
put in to certain risk mitigation activities, and how well the risks may be managed. The
relationships are shown in Figure 5-1.
45
Research methods
Figure 5-1 Risk management before a project starts
To help project managers predict the possibility of risks and to plan the risk
management efficiently before a project starts, it is necessary to summarize practices
and results of risk management from finished projects. The research framework of risk
management in the main study, as shown in Figure 5-2, is designed to do posteriori
measurements (after the project is finished) on the occurrence of project risks and the
impacts of performed risk management activities on these risks.
Figure 5-2 Research framework for the risk management in the main study
The research questions RQ3.2, as shown in Figure 5-2, is to investigate the actual
risk incidence, i.e., how often had problems occurred in practice? The research question
46
Research methods
RQ3.3, is to investigate the execution of the risk management activities in projects, i.e.,
how often had certain risk management activities been performed in practice? The
research question RQ3.4 is to examine the effect of performed risk management
activities, i.e., how well the performed risk mitigation activities had reduced
corresponding risks? For a specific risk, we are interested in the most effective
mitigation activities.
In our study, we selected several typical risks and possible risk management
activities proposed in [Boehm03], [FAA03], [Li06a], [Rose03], [CBSEnet04],
[Vitharana03], [Lawton02], [Fitzgerald04], [Torchiano04]. The selection criteria are:
- First, the risks and risk management activities should address the common issues
in the CBD, COTS-based development, and OSS-based development, as shown
in Figure 4-2.
- OTS component-based development encompasses three primary types of
stakeholders, namely component developers, application assemblers/integrators,
and customer. Different stakeholders have different risks and challenges
[Vitharana03]. We have selected risks and challenges that are relevant only to
the application assemblers/integrators.
- The selected risk management activities in our study focus mainly on process
and organizational issues, instead of technical methods, such as wrapping,
composition prediction, and testing.
The selected typical risks in OTS components-based cover different phases of a

project, as shown in Table 5-4. The typical risk management activities are shown in
Table 5-5. In addition, the results from process improvement studies show that one
added role in OTS component-based development, i.e., that of a knowledge keeper
(dedicated project personnel to keep track of OTS component knowledge/information),
may help to mitigate the risks in OTS component-based development [Li06a],
[Morisio00], [Rose03]. We therefore propose the existence of knowledge keeper as one
possible risk management activity.
Concerning research questions RQ3.2 and RQ3.3, we compared the occurrences of
typical risks and performed risk management activities. The results are visualized with
the boxplot graph.
To analyse the results of RQ3.4, we decided to use Nonlinear Canonical Correlation
Analysis (Nonlinear CCA) [Gifi90], [Michailidis98], because we face a nonlinear CCA
problem with a set of explanatory variable vector defined by a nominal (knowledge
keeper) variable and several ordinal (risk management activities performed) variables
and a dependent variable matrix defined by a series of ordinal (risk occurrences) scales.
The linear combination on the explanatory variable side is undefined, because we have
no metric to quantify the categories of each variable. The linear combination of the
variables on the dependent side is also undefined, because the categories of each
variable can be re-scaled by any nonlinear function the preserves monotonicity. Thus,
we need to optimally scale or quantify the variables while simultaneously solving the
traditional linear CCA problem of finding weights for each explanatory variable.
Nonlinear CCA corresponds to categorical canonical correlation analysis with optimal
scaling. The detailed introduction of Nonlinear Canonical Correlation Analysis and the
rationales for using it are reported Section 3.2 in Paper P5 [Li06c] in Appendix A.
47
Research methods
Table 5-4 Typical risks in OTS component-based projects

Category ID Name Explanation
Estimation R1 R_SelEff Effort to select OTS components was not
risks satisfactorily estimated [Boehm03].
R2 R_IntEff Effort to integrate OTS components was
not satisfactorily estimated [Rose03].
Quality risks R3 R_ComRel OTS components negatively affected
system reliability [Vitharana03],
[Ruffin04].
R4 R_ComSec OTS components negatively affected
system security [Lawton02],
[Vitharana03],
[ Ruffin04]
R5 R_ComPerf OTS components negatively affected
system performance [Lawton02],
[Vitharana03],
[ Ruffin04]
Requirements R6 R_ComAda OTS components could not be sufficiently
risks adapted to changing requirements
[Boehm03].
R7 R_ReqRen It is not possible to (re) negotiate

requirements with the customer, if OTS
components could not satisfy all
requirements [Li06a].
Debug and R8 R_DefLoc It was difficult to identify whether defects
deployment were inside or outside the OTS
risks components [Boehm03].
R9 R_ComComp OTS components were not satisfactorily
compatible with the production
environment when the system was
deployed [Vitharana03]
Maintenance R10 R_MaintPlan It was difficult to plan system maintenance,
risks e.g. because different OTS components had
asynchronous release cycles [Rose03].
R11 R_ComUpd It was difficult to update the system with
the last OTS component version [Rose03].
Vendor R12 R_ProvInf Information on the reputation and technical

support risks support ability of provider were inadequate
[Rose2003], [Fitzgerald04].
R13 R_MissSupp Provider did not provide enough technical
support/ training [Rose03], [Fitzgerald04].
48
Research methods
Table 5-5 Typical risks management activities in OTS-based projects

Category ID Name Explanation
Development Rm1 Rm_CstMkBuy Customer had been actively involved
in the “acquire” vs. “make” decision
of OTS components [FAA03],
[Li06a].
Rm2 Rm_CstSel Customer had been actively involved
in OTS component selection
[FAA03].
Rm3 Rm_SelArch OTS components were selected
mainly based on architecture and
standards compliance, instead of
expected functionality [Torchiano04].
Rm4 Rm_SelQual OTS component qualities (reliability,
security etc.) were seriously
considered in the selection process
[Boehm03], [Li06a].
Rm5 Rm_LearnEff Effort in learning the OTS component
was seriously considered in effort
estimation [Boehm03].
Rm6 Rm_BBEff Effort in black-box testing of OTS
components was seriously considered
in effort estimation [Boehm03],
[Li06a].
Rm7 Rm_UnfFirst Unfamiliar OTS components were
integrated first [Rose03].
Rm8 Rm_IncTest Did integration testing incrementally
(after each OTS component was
integrated [Li06a].
Maintenance Rm9 Rm_UpdFoll Local OTS-experts actively followed
updates of OTS components and
possible consequences [Li06a].
Rm10 Rm_MktWtch Maintained a continuous watch on the
market and looked for possible
substitute components [Li06a].
Rm11 Rm_ProvWtch Maintained a continuous watch on
provider support ability and
reputation [Rose03].
Rm_KK Knowledge Have a knowledge keeper to own the
Keeper knowledge of the OTS components
[Li06a].
5.3.3.3. Part three: Verify the six newly published theses
Based on results of the pre-study (see paper P2 [Li06a] in Appendix A), we clarified the
research questions that can be used to verify the six newly published theses
49
Research methods
[Torchiano04], as shown in Table 5-6. How we clarified the research questions is

reported in Section 3.2 of Paper P3 [Li05a] in Appendix A.
We used different analysis methods to analyse the research questions.
- Thesis T1 is relevant to OSS components. To study research questions T1.R1

and T1.R2, we compared the usage of OSS components with COTS
components.
- Because T2, T3, T4, and T6 are not relevant to OSS components, only
information from COTS components was analysed.
- Thesis T5 should be tested using data from the whole project rather than a
specific COTS component. The unit to be analysed for research question
T5.RQ1 is therefore the whole project instead of a specific OTS component.
Table 5-6 Revised research questions to investigate the newly published theses
Newly published theses Revised research questions
T1: Open source software is often T1.RQ1: Had the source code of OSS
used as closed source. components been read by integrators
during software development?
T1.RQ2: Had the source code of OSS
components been changed by integrators
T2: Integration problems result from T2.RQ1: Were the standard mismatches
lack of compliance with standards. more frequent than architecture
Architecture mismatches constitute a mismatches?
secondary issue.
T3: Custom code mainly provides T3.RQ1: Were the occurrences of building
additional functionalities. glueware more frequent than the
occurrences of building addware?
T4: Developers seldom use formal T4.RQ1: How often had a formal process
selection procedures. Familiarity with been used in practice?
either the product or the generic T4.RQ2: How often had the familiarity-
architecture is the leading factor in based selection process been used in
selection. practice?
T4.RQ3: How often had the hands-on
experimentation based selection process
been used in practice?
T5: Architecture is more important T5.RQ1: Had architectural compliance
than requirements for product been considered more important than
selection. functionality completeness in the whole
COTS-based project?
T6: Integrators tend to influence the T6.RQ1: How often the required changes
vendor on product evolution whenever (corrections or new functionalities) from
possible. integrators had been satisfied by the
vendors.
50
Research methods
5.3.3.4. Part four: Compare COTS with OSS-based projects to investigate RQ4 (How
did project managers make their make vs. acquire decision in using COTS instead of
OSS, or vice versa?).
The high level question will be: Why do I use COTS instead of OSS components, or
vice versa? To investigate this question, we asked three sub-questions RQ4.1 to RQ4.3
as follows:
- RQ4.1: What are the commonalities and differences in profiles of projects using
COTS components vs. those using OSS components?
- RQ4.2: What are the commonalities and differences in the expectation of
projects using COTS components vs. those using OSS components?
- RQ4.3: What are the commonalities and differences in possible risks (problems)
of projects using COTS components vs. those using OSS components?
The background and rationale for RQ4.1 to RQ4.3 are reported in Section 3 of Paper
P6 [Li06d] in Appendix A.
We used a questionnaire (see Appendix D) to collect data in the main study. The
questionnaire was organized into three main parts.
- Questions to collect background information on the company, project, and

respondents.
- Questions to get answers of the research questions in process improvement, risk
management, and decision making.
- Questions to collect information about OTS components actually used in the
project. In the questionnaire, we asked the respondents to give brief information
(name, COTS/OSS, version) of all OTS used and detailed information of one of
the most important COTS/OSS component (the one provides most functionality
compared to other OTS components).
The final questionnaire was first designed and pre-tested in English (internal and
external reviews). It was then translated into the native languages (Norwegian, German,
and Italian) and published on the SESE web survey tool [Sese04] at Simula Research
Lab in OSLO.
We defined the unit of this study as a completed software development project, and
its OTS-relevant properties. The projects were selected based on two criteria:
- The project should use one or more OTS components.

- The project should be a finished project, possibly with maintenance, and
possibly with several releases.
Projects were collected randomly from IT companies in Norway, Italy, and

Germany. Slightly different sampling procedures were used in each country due to
limited resources. Detailed discussions on sample selection in this study are reported in
Paper P8 [Conradi05a] in Appendix A.
51
Research methods
- In Norway, we gathered a company list from the Norwegian Census Bureau

(SSB) [SSB02]. We included mainly companies which were registered as IT
companies. Based on the number of employees, we selected the 115 largest IT
companies (100 IT companies and 15 IT departments in the largest 3 companies
in 5 other sectors), 200 medium-sized software companies (20-99 employees),
and 100 small companies (5-19 employees) as the original contact list.
- In Italy, we first got 43580 software companies from the “yellow pages”. We
then randomly selected companies from them. For these randomly selected
companies, we read their web-site to ensure they really are software companies.
196 companies were finally clarified as software companies, and were included
in the original contact list.
- In Germany, we composed a company list from a company list from an
organization similar to the Norwegian Census Bureau and in a similar manner as
for Norway. We then used the existing Fraunhofer IESE customer database to
get contact information.
To avoid bias in the data collection procedure, we made a guideline about how to
contact respondents before the survey started. All countries – Norway, Italy, and
Germany – followed the process in the guideline. Possible respondents were contacted
first by telephone. If the candidate respondents had suitable OTS-based projects and
would like to join our study, a username and password of the survey tool and an electric
version of the questionnaire were sent to them. The respondents could use either the
SESE web tool or the electronic version to fill in the questionnaire. The respondents
who did not want to answer the questionnaire were also registered. We logged the main
reasons of non-response, such as “no software development”, “no OTS-based projects”,
and “busy”. Most respondents used the Web tool to fill in the questionnaire. However,
phone interviews were used in Germany because of the confidentiality concern in some
companies [Conradi05a].
5.3.4 Research Design of the Follow-up Study

As we promised in the main study, we will hold a seminar to report our results. The
follow-up study (see Paper P7 [Li05b] in Appendix A) was performed together with this
industrial seminar. The intentions of the seminar included two parts:
- First, we would like to report our results by giving guidelines on how to improve
processes and manage risks in OTS-based development.
- As most data of this study show only state-of-the-practice, we do not know
possible cause-effect relationships of our conclusions. We therefore had one hour
discussion as the second part of the seminar to get feedback from the participants.
We sent invitations to 100 industrial respondents and 100 academic researchers.

Twenty-three industrial respondents participated in the seminar. In this study, we first
showed our survey results in one hour presentation. During our talk, participants were
able to interrupt and ask questions. In the second part, we hypothesized some questions
based on the survey result. We also listed our assumed explanations on these questions.
Industrial participants gave their opinions to support or reject our explanations. The
52
Research methods
whole discussion section has been tape recorded. The data analysis was performed by
listening to the tape afterwards.
5.4 Validity Issues

A fundamental discussion concerning results of a study is how valid they are. Empirical
research usually uses definitions of validity threats that originate from statistics and not
all the threats are relevant for all types of studies. Wohlin et al. [Wohlin00] define four
categories of validity threats:
- Conclusion validity (for statistical analysis) - “right analysis”: this validity is

concerned with the relationship between the treatment (the independent variable
in a study) and outcome (the dependent variable). We want to make sure that
there is a statistical relationship of significance. Threats are related to choice of
statistical tests, sample sizes, reliability of measures etc.
- Internal validity (for explanatory and causal studies, not for exploratory or
descriptive studies) - “right data”: we must make sure that there is a causal
relationship between treatment and outcome and that is not a result of factors that
are not measured. Threats are related to history, maturation, selection of subjects,
unpredicted events and interactions, ambiguity about the direction of causal
influence etc. Yin [Yin03] adds (experimental) reliability to this: demonstrating
that the operations in a study can be repeated with the same results such as data
collecting.
- Construct validity- “right metrics”: we must ensure that the treatment reflects the
cause and the outcome reflects the effect. Threats are mono-operation bias (a
single case may not reflect the constructs), mono-method bias (a single type of
measure may be misleading), hypothesis guessing etc.
- External validity- “right context”: this validity is concerned with generalization
of results outside the scope of a study. Three types of interactions with the
treatment may happen: people (the subjects are not representative for the
population), place (the setting is not representative), and time (the experiment is
conducted in a special time for example right after a big software crash). Yin
[Yin03] terms this establishing the domain to which the findings of a study can
be generalized.
To avoid the possible threats to validity in the studies in this thesis, we performed
different methods. For example, in the pre-study (see Paper P2 [Li06a] in Appendix A)
and the main study (see papers P3 [Li05a], P4 [Li06b], P5 [Li06c], and P6 [Li06d] in
Appendix A), we did several rounds of pre-tests of the interview guide and the
questionnaire. These pre-tests helped to decrease the possible construct validity threats.
In the main study, we used random sample selection strategy to decrease the threats to
the external validity. Detailed information on the validity issue of the studies reported in
this thesis can be found in the “possible threats to validity” sections in Papers from P1
to P8 in Appendix A.
53
Research methods

In this chapter, we first gave a brief review of empirical methods in software
engineering fields. The rationale and steps of using different research methods in our
studies are given. Finally, we discuss how to manage the four types of validity issues in
empirical software engineering in our studies.
54
Results
6 Results
This chapter summarizes and interprets the collected data for research questions RQ1 to
RQ4. Our lessons learned from performing the main study, i.e., the international survey,
are described at the end.
6.1 Collected Samples in Each Study
6.1.1 Collected Samples in the Preliminary Study

The data was collected from three IT companies, Ericsson in Grimstad, EDB Business
Consulting in Trondheim, and Mogul Technology in Trondheim. There were 200
developers at Ericsson in Grimstad, where we sent out 10 questionnaires to developers
in one development team and got 9 filled-in questionnaires back. There were 20
developers in EDB Business Consulting in Trondheim, and we gathered 10 filled-in
questionnaires back out of 10. We distributed 10 questionnaires to 22 developers at
Mogul Technology in Trondheim and got 7 back. Those developers were selected
because their work was related to component reuse, and they could assign effort to
participate in the survey. This is non-probability sampling, which is based on
convenience. Most participants in this survey have a solid IT background. Six of 26
respondents have MSc degrees in computer science and all others have a bachelor’s
degree in computer science or telecommunications. More that 80% of them have more
than 5 years of programming experience. The detailed information of each respondent is
in the Table 3 of Paper P1 [Li04a] in Appendix A.
6.1.2 Collected Samples in the Pre-study

We interviewed 16 projects in 13 companies. All the companies are Norwegian IT
companies. Nine of these 13 are stand-alone companies, with staff size between 5 and
500. The other companies are subsidiaries, with local staff size ranging from 50 to 320.
Six companies are IT consultancies, 5 are software vendors, and 2 are telecom
companies. Five of these companies are publicly traded companies, the others are
privately owned. Detailed background information about the selected companies is
given in Table 1 of Paper P2 [Li06a] in Appendix A.
The company projects were selected based on two criteria:
55
Results
- The project should use one or more COTS components

- The project should be a finished project and possibly with maintenance, and
Generally, we selected only one project from a company. However, we selected two
projects in three companies because these projects differed largely in project members,
development process, and COTS components used. Detailed background information
about the 16 projects is given in Table 2 of Paper P2 [Li06a] in Appendix A.
We had one respondent from each project. Three respondents are IT managers, 4 are
project managers, 5 are software architects, 3 are software developers, and 1 is a
software development researcher. Nine of them have more than 10 years of software
development experience, and 12 of them have more than 4 years of working experience
with COTS-based development. Eight of them have a master’s degree, and the rest have
a bachelor’s degree. Twelve of them have a principal degree in informatics or computer
science.
Some projects used one or two COTS components, others used more. Because of
time limitations, we asked the respondents to select no more than three typical COTS
components which were used in their projects. Totally, we gathered information about
30 different COTS components in 16 projects. Some typical COTS components are
listed in Table 3 of Paper P2 [Li06a] in Appendix A.
6.1.3 Collected Samples in the Main Study

We have gathered results from 133 projects (47 from Norway, 48 from Germany, and
38 from Italy) from 127 companies. In general, we selected one project from each
company. However, we selected more than one project in three Norwegian IT
companies because those companies have many OTS-based projects and would like to
share more experience with this study.
The responding companies involved small (5-19 employees), medium (20-99
employees), and large (100 or more employees) ones as shown in Figure 6-1. The main
business areas of the sampled companies also cover different types as shown in Figure
6-2.
Figure 6-1 Distribution of company size
56
Results
Figure 6-2 Distribution of companies main business areas
The final systems produced by the 133 projects cover different application domains
as shown in Figure 6-3.
Figure 6-3 Distribution of application domain of the systems
In the selected 133 projects, 83 used only COTS components, 44 used only OSS
components, and six used both COTS and OSS components. For the 44 projects using
only OSS components, each of them gave detailed information of an OSS component.
For the 83 projects using only COTS components, each of them gave detailed
information of a COTS component. For the six projects using both COTS and OSS
components, the respondents selected to give detailed information of five COTS
components and one OSS component.
Most respondents of the 133 projects have a solid IT background. More than 90% of
them are IT managers, project managers, or software architects. Most of them have
more than 2 years of experience with OTS-based development. All of them have at least
a bachelor’s degree in informatics, computer science, or telecommunications.
6.1.4 Collected Samples in the Follow-up Study

Twenty-three Norwegian respondents attended the seminar. More than 40% of them
came from large-sized IT companies (more than 100 employees). 40% came from a
57
Results
medium-sized (with 20-99 employees) and 20% came from small-sized companies (less
than 20 employees).
6.2 Answers to Research Questions
6.2.1 Reusing In-House Built Components – RQ1

The companies’ reuse levels of the three investigated companies are classified as large,
medium, and small. Most developers in Ericsson think that the reuse level in their
company is very high or high. Most developers in EDB regard the reuse level in their
company is high or medium. Most developers in Mogul think that the reuse level in
their company is medium or little. Detailed data are shown in Figure 1 and 2 in Paper P1
[Li04a] in Appendix A. Therefore, we define the reuse level in Ericsson as high, the
reuse level at EDB as medium, and the reuse level at Mogul as low.
Answers to sub-questions are summarized as following. Detailed information is in
Section 4 of Paper P1 [Li04a] in Appendix A.
- RQ1.1: Does requirements (re)negotiation for in-house components really work

as efficiently as people assume?
No respondents believe that the requirements were never changed/
(re)negotiated. Only 8% of respondents think the requirements of their typical
project are not flexible. However, only 48% of respondents think component-
related requirements (re)negotiation works well.
- RQ1.2: Does the efficiency of component related requirements (re)negotiation

increase when more in-house built components are available?
The result of correlation between the reuse level and the efficiency of
requirements (re)negotiation, using one-tailed Spearman Rank Correlation
Coefficient analysis, is .112, and the significance is .306. This shows that there
is no significant relationship between the reuse level and the efficiency of
component-related requirements (re)negotiation.
- RQ1.3: Does the value of component repository increase with more reusable
Seventy-one percent of the respondents in Mogul and EDB regard
constructing a component repository as worthwhile, against 57% in Ericsson.
The result of correlation between the reuse level and the value of the repository,
using one-tailed Spearman Rank Correlation Coefficient analysis, is -.124, and
significance is .297. It shows that there is no obvious relationship between them.
- RQ1.4: How can a component user acquire sufficient information about relevant
components?
Sixty-seven percent of the respondents think the component structure is well
understood, 61% say that the component interfaces are understood, and 63%
regard the design rules of components are also well understood. However, no
one considers that the design/code of components is well documented. Eighty-
58
Results
six percent believe that insufficient component documentation is a problem, e.g.,

documentation is not complete, not updated, and difficult to understand. The
preferable way of documentation is web pages. Some of the developers’
knowledge about how to use components comes from informal communication
sources, for example, previous experience and suggestions from local experts.
- RQ1.5: Does the difficulty of component documentation and component

knowledge management increase with increasing reuse level?
The result of correlation between the reuse level and the satisfaction of
documentation, using one-tailed Spearman Rank Correlation Coefficient
analysis, is -.469, and significance is .014, which shows a weak negative
relationship between them. It means that the higher the companies’ reuse level,
the less satisfied a developer is with the component documentation.
- RQ1.6: Do developers trust the quality specification of their in-house built

Seventy percent of the participants regard the design criteria for quality
requirements are not well defined, and 87% will test the quality attributes of
components after component modification, before integrating them into the
system.
6.2.2 Software Process Improvement – RQ2

The research question RQ2 is examined by sub-questions RQ2.1 to RQ2.8. RQ2.1 and
RQ2.2 are investigated by the pre-study and detailed answers are shown in Section 5 of
Paper P2 [Li06a] in Appendix A. Questions RQ2.3 to RQ2.8 are studied by the main
study and detailed answers are shown in Section 4 of Paper P4 [Li06b] in Appendix A.
Answers to these sub-questions are as follows:
- RQ2.1: What was the actual process used in the projects using COTS
components?
Results show that the main development processes of the studied projects can
be grouped into three categories: pure waterfall, waterfall mixed with
prototyping, and incremental mixed with prototyping. Five projects used
waterfall. One used waterfall mixed with a little prototyping (a small prototype
was used for discussing requirements of the Graphical User Interface, i.e., GUI,
part of the system). Ten projects used incremental mixed with some prototyping.
None of the respondents claimed that they selected the main development process
based on whether or not they intended to use COTS components. That is, they
decided the main process before they started to think about using COTS
components or not. They just added or changed some activities and roles in their
traditional process to reflect the use of COTS components.
Therefore, the answer to research question RQ2.1 is: The so-called COTS-
based development process is the customization of the traditional development
process due to the use of COTS components.
- RQ2.2: What are the commonalities and possible variations in COTS-based

59
Results
The commonalities are that new activities were added and possible one new
role was added. The added new activities include make vs. acquire decision,
COTS component selection, learn and understand the COTS components, and
building glueware and addware. The possible added new role is: a COTS
component knowledge keeper. One or more project members had some previous
experience with selection and integration of the actual COTS components in ten
of our studied projects. They provided suggestions on COTS component
selection and integration. Although some of them were not dedicated to work as a
COTS component knowledge keeper, their previous experience improved the
speed and quality of COTS component selection and integration.
One variation in the customization of main development processes deals with
the phase to make the make vs. acquire decision and COTS component selection.
Another variation is the COTS component selection and evaluation process. The
selection process in our studied projects can be summarized into two categories,
i.e., familiarity-based selection process and Internet search, hands-on trial-based
selection process.
Therefore, the answer to research question RQ2.2 is: There are some common
new activities and one new role is added in COTS-based development processes.
The possible variations are when and how to perform these new activities.
- RQ2.3: What were the actual development processes in OTS-based projects?

The results are summarized in Figure 6-4 and show that the waterfall,
incremental, XP, German V-model, and prototyping model are those mainly used
in practice. Between these, the incremental with prototyping model and XP was
the most popular. The German V-model was used widely in Germany as the
OTS-based development process.
Figure 6-4 The actual development process in the OTS-based project
60
Results
- RQ2.4: Was the actual development process decided before the make vs. acquire
decision or after the make vs. acquire decision.
The results show that 75% projects decided their main development processes
before they started to think about using OTS-components
- RQ2.5: Who decided the actual development process?

The answers reveal that in 29% of all projects the development process is
predefined by the global company or the department rules. In addition,
concerning 14% of the remaining projects the decision was at least affected by
company/department rules. This trend is especially obvious in Germany as the
company rules dominated the development processes in 65% of our studied
projects.
- RQ2.6: What was the actual selection process used to select one of the OTS
components (we asked for information about the component, named Comp. 1,
that provides the most functionalities)?
We listed six possible selection activities as named a) to f) in the
questionnaire:
a) Searched Internet for possible OTS component candidates.

b) Got recommendation of possible OTS component candidates from the
customer.
c) Got recommendation of possible candidates from a local colleague/OTS-
expert.
d) Used a formal decision-making method to compare possible OTS
component candidates, e.g., with weighted evaluation criteria.
e) Limited possible candidates into 1-3 components, by reading
literature/documentation.
f) Did “hands-on” try-out of 1-3 components, e.g., on a downloaded demo
version.
The results are summarized in Figure 6-5 and show that activities a, c, e, and
f were those mainly used. Thus, the familiarity-based selection process (activity
c) and the Internet search, trial-based selection process (activity a, e and f) from
the pre-study (see paper P2 [Li06a] in Appendix A) are proved to be more
popular than a formal processes (activity d) in general. However, the further
analysis with Chi-square shows that there are significant differences of the
activities b and c between countries. In Norway and Italy, very few projects used
the activity b. However, activity b was popular in Germany. In addition, almost
all German projects performed activity c, while it was used only in half of the
projects in Norway and Italy.
61
Results
Figure 6-5 What selection and evaluation actions were performed?
- RQ2.7: When was the Comp. 1 selected?

Figure 6-6 shows that OTS components were selected in the early stages,
such as pre-study, requirement, and overall design, in most projects, especially
in Germany.
Figure 6-6 When was the OTS component selected?
- RQ2.8: What was the relationship between the selection phase and the project
context, such as the familiarities with the Comp. 1 and the importance of the
Comp. 1?
The results reveal that the selection phase was weakly (with Spearman
correlation coefficient value -.356) correlated with the project members’
familiarity with it. It means that the project members preferred to decide the
familiar OTS components in the early stage of the project. However, there is no
62
Results
significant connection between the importance of an OTS component and the

phase it was selected.
In addition to answer the sub-questions RQ2.1 to RQ2.8, the pre-study (see paper P2
[Li06a] in Appendix A) and the main study (see paper P3 [Li05a] in Appendix A)
verified six newly published theses [Torchiano04] in OTS component-based
development. The results are summarized as following. Detailed information can be
found in Section 5 of Paper P3 [Li05a] in Appendix A.
- Thesis T1: Open source software is often used as closed source.
We support T1. We have asked the respondents to answer whether they have
read or changed the source code of the component. We used a five-point Likert
scale to measure the answer. Respondents were asked to answer “very little”,
“little”, “some”, “much”, “very much”, or “don’t know”. The results show that
the median value of reading OSS code is 3 (meaning “some”). However, the
source code has seldom been changed as the median value of changing the OSS
code is 2 (meaning “little”). Further analysis discovered that 46% of the
respondents claimed that they modified “very little” parts of the source code in
OSS components. In addition, 29 (out of 88) selected COTS components
provided source code to their users.
- Thesis T2: Integration problems result from lack of compliance with standards.
Architecture mismatches constitute a secondary issue.
We are against T2. For the selected 88 COTS components, only 4.5% of the
COTS components had standard mismatches, compared with 28.4% COTS
components having architecture mismatches.
- Thesis T3: Custom code mainly provides additional functionalities.
We support T3. Glueware was needed for 32.9% of the selected COTS
components and addware was needed for 65.9% of the selected COTS
components.
- Thesis T4: Developers seldom use formal selection procedures. Familiarity with
either the product or the generic architecture is the leading factor in selection.
We support T4. Formal procedures were only used in evaluating 19% of the
selected COTS components.
- Thesis T5: Architecture is more important than requirements for product

selection.
We are against T5. We asked the respondents whether COTS components

were selected mainly based on architectural and standards compliance, or
expected functionality. The answer is measured by “don’t agree at all”, “hardly
agree”, “agree somewhat”, “agree mostly”, “strongly agree”, or “don’t know”.
63
Results
We assigned an ordinal number from 1 to 5 to the above alternatives (5 meaning

strongly agree). Results are in Figure 6-7 and show that the median value of this
question is 2 (meaning “hardly agree”). It indicates that most respondents hardly
agree that COTS components were selected mainly based on architectural and
standards compliance.
0
N = 82
A RC H. VS . F U NC .
Figure 6-7 Result of thesis T5
- Thesis T6: Integrators tend to influence the vendor on product evolution

whenever possible.
We support T6. Sixty percent of the selected COTS components have been
changed by vendors according to integrators’ requirements.
In the follow-up study (see paper P5 [Li05d] in Appendix A), we investigated the
cause-effect reason of the phenomena discovered from the previous studies. The results
are summarized in Table 6-1.
Table 6-1 Barriers to disseminating SPI theories to the IT industry

Observations Industrial Our insights Barriers
from the main explanations
study
Few projects used Few empirical The industrial Without
the proposed OTS- studies have practitioners will sufficient
based development investigated the need evidence that empirical
processes. cost and benefit of the formal process evidence on OTS
the proposed is cost-effective. based
COTS-based development
development It is also necessary processes,
processes. to know how to industrial project
downsize a formal managers would
process according not like to
to different project change their de-
contexts. facto
development
process.
64
Results
Formal OTS It is not necessary Most formal Little advice on

selection processes because OTS selection processes how to
were seldom used components are not were proposed customize a
in practice. the key parts of the without evaluations formal process
system in some on their pre- according to
cases. conditions and their project context
cost-benefit. and OTS itself
Few studies prevents the use
showing that a of a formal
formal process selection
actually gave better process.
results than
informal processes.
6.2.3 Risk Management – RQ3

The research question RQ3 is investigated by sub-questions RQ3.1 to RQ3.4. RQ3.1
was examined by the pre-study and detailed answers to this question are in Section 5.3
of Paper P2 [Li06a] in Appendix A. Research questions RQ3.2 to RQ3.4 are examined
in the main-study, and detailed answers to these questions are given in Section 5 of
Paper P5 [Li06c] in Appendix A. The summaries of answers to these questions are as
follows:
- RQ3.1: What are process scenarios (variations) of the projects using COTS
components successfully and not successfully? What are possible problems and
good practices in different process scenarios?
The results of the risk management in the pre-study are the 11 problems
encountered (PEs) and the 12 good practices (GPs). One example of the
problems encountered is that the integration effort was under-estimated. An
example of the good practice is that the integration testing was done
incrementally. Details of these PEs and GPs are shown in Section 5.3 of Paper
P2 [Li06a] in Appendix A.
- RQ3.2: Occurrences of Risks in OTS Component-Based Projects

We compared typical risks in Table 5-4 (see Section 5.3.3). Results are in
Figure 6-8 and show that the most frequent risks are R2 (R_IntEff) and R8
(R_DefLoc). It means that the integration effort was usually wrongly estimated.
In addition, it was difficult to figure out whether the defects are inside or outside
the OTS components. The least frequent risks are R3 (R_ComRel), R4
(R_ComSec) and R5 (R_ComPerf), i.e., the problems relevant to the negative
effect of the OTS components on the quality of the whole system.
65
Results
R_
R_
R_
R_
R_
R_
R_
R_
R_
R_
R_
R_
R_
Pr
Co
Co
M
Co
Se
M
Co
In
Co
Co
Re
De
tE
ov
iss
ain
lE
qR
m
m
m
m
fL
ff
In
ff
Up
Se
Su
Ad
Co
Re
Pe
tP
oc
en
f
c
lan
rf
pp
l
d
a
Figure 6-8 Occurrences of typical risks
- RQ3.3: Which risk management actions have been performed in practice?
We compared typical management actions in Table 5-5 (see Section 5.3.3).

The results for this question are shown in Figure 6-9. We can see that the
activities Rm4 (Rm_SelQual), Rm5 (Rm_LearnEff), Rm7 (Rm_UnfFirst), Rm8
(Rm_IncTest) and Rm9 (Rm_UpdFoll) were used more frequently than others. It
means that developers have seriously evaluated the quality of the OTS
components in the selection phase. They have considered the possible learning
effort in the effort estimation. They have done integration testing as early as
possible and they usually have internal expert to follow the update of the OTS
components. However, the activities relevant to the users, such as Rm1
(Rm_CstMkBuy) and Rm2 (Rm_CstSel), were least frequently used.
66
Results
2
Rm
Rm
Rm
Rm
Rm
Rm
Rm
Rm
Rm
Rm
Rm
_S
_I
_C
_L
_M
_B
_C
_S
_U
_U
_P
nc
elQ
elA
ro
ea
stS
BE
stM
pd
nf
ktW
Te
vW
rn
Fi
Fo
el
ff
ua
rch
kB
st
Ef
rst
tch
ll
tch
l
f
uy
Figure 6-9 Risk management activities performed
- RQ3.4: Can performed risk mitigation actions help to mitigate the

corresponding risks?
To investigate research question RQ3.4, we used the non-linear canonical

correlation analysis (the OVERALS in SPSS [SPSS01]) to find the relationships
between the occurred problems (see Table 5-4 in Section 5.3.3) and the risk
management activities performed (see Table 5-5 in Section 5.3.3). Since we put
the data from three countries together, we also analysed whether the differences
between the three nations will affect our conclusions. In addition, we analysed
whether the differences between COTS and OSS components have effect on our
results. The results are summarized in Table 6-2. Details are shown in Section
5.3 of Paper P5 [Li06c] in Appendix A.
67
Results
Table 6-2 Effective risk management activities
In the follow-up study (see paper P7 [Li05b] in Appendix A), we investigated two
risk management issues, i.e., the customer involvement and knowledge management.
The details are shown in Table 6-3.
Table 6-3 Barriers to disseminating RM theories to the IT industry

Observations Industrial answers Our insight Barriers
from the main
study
Customers were Their customers As it is difficult to Few case studies
seldom involved in either had no discover all have observed
OTS component technical limitations of OTS successful
selection and background to components at the requirements
decision making. make a decision or very beginning, it (re)negotiation in
cared only for the is necessary for OTS-based
final product integrators to be development.
without considering able to There are
the technical (re)negotiate therefore few
details. requirements with guidelines about
the customer. when and how to
involve customers
in requirements
(re)negotiation
because of OTS
limitations.
68
Results
Proper knowledge They were What knowledge Few studies on

management and interested in how should be knowledge
sharing helped to much effort the documented and accumulation and
avoid the risks. knowledge keeper shared in OTS- transfer delayed
needs to spend on based development the dissemination
OTS component- has seldom been of the OTS-based
based development investigated. It is development
and what kind of therefore difficult experience.
knowledge needs to for industrial
be kept. practitioners to
collect and share
relevant OTS-based
development
knowledge.
6.2.4 Decision Making – RQ4

Research question RQ4 is investigated by sub-questions RQ4.1 to RQ4.3. These sub-
questions were mainly examined in the main study, and detailed results are in Section 4
of the paper P6 [Li06d] in Appendix A and are summarized as follows:
- RQ4.1: What are the commonalities and differences in profiles of projects using
COTS components vs. those using OSS components?
The results in Figure 6-10 and 6-11 illustrate that there are no significant
differences of the project profiles between COTS and OSS-based projects. The
most emphasized requirements of the final systems were time-to-market,
reliability, and performance. Security was surprisingly listed as the least
emphasized requirement of the system.
Figure 6-10 Project profiles of the COTS component-based system
69
Results
Figure 6-11 Project profiles of the OSS component-based system
- RQ4.2: What are the commonalities and differences in the expectation of

Results in Figure 6-12 and Figure 6-13 show that there are no significant
differences between the general expectation of using either COTS or OSS
components. The main expectation of using OTS components is to shorten the
time-to-market and to save development cost.
Figure 6-12 General motivations of using COTS components
70
Results
Figure 6-13 General motivations of using OSS components
For the specific expectations, results in Figure 6-14 show that COTS users
believe that paid software will give good reliability and will follow the market
trend. They also believe that the COTS vendor will provide good technical
support. On the other hand, results in Figure 6-15 show that OSS components
pay more attention to the free and available source code.
Figure 6-14 Specific motivations of using COTS components
71
Results
Figure 6-15 Specific motivations of using OSS components
- RQ4.3: What are the commonalities and differences in possible risks (problems)
of projects using COTS components vs. those using OSS components?
We compared the occurrences of typical risks in Table 5-4 (see Section 5.3.3)
in COTS vs. OSS projects. Results are shown in Figures 6-16 and 6-17, and
illustrate that COTS users had significantly more difficulties in R1 (R_SelEff),
i.e., estimating the component selection effort and R6 (R_ComAda), i.e.,
following requirement changes.
Figure 6-16 Occurrences of risks in COTS component-based projects
72
Results
Figure 6-17 Occurrences of risks in OSS component-based projects
6.3 Our Lessons Learned on Performing the Studies

In addition to the results of the research question RQ1 to RQ4, we got several insights
on methodology issues of implementing an international industrial survey. The details
are reported in Paper P8 [Conradi05a] in Appendix A.
The insights on methodology issues focus on three categories. The first is how to
sample the population. As our study object is to investigate the OTS component-based
development projects in ICT (Information and Communication Technology) sector,
there are two issues that need to be investigated. The first is how to define the ICT
sector, and the second is how to define our study objects. To get the information about
the ICT companies, we have used a complete set of registered ICT companies defined
by a public source from three countries. However, our real problem is that the actual
study object (experimental unit) – a special kind of project – is not identical to the
primary object (accessible sampling unit), being a company or a respondent. Due to
resource constraints and logistical problems, it seems totally unrealistic to perform an
explorative pre-study to gather the information of all projects from each ICT company
and randomly select samples from these projects. At best we can achieve a stratified-
random sample of ICT companies, followed by a convenient sample of relevant
projects. Other reflections on sampling and response rates are:
- For all countries, there were problems in classifying large and perhaps
international ICT corporations, with many subsidiary companies and divisions.
73
Results

sampling and data collection in different countries.
- In Norway, 50 of 380 (13%) of the companies were no longer “contactable”, as
the company lists were two years old in 2004. It represents a very dynamic
marketplace.
- In Norway, we found that 70 of 285 (24%) software companies (NACE code
72.xx) claimed to have no software development at all.
- The response rate in Norway increases with decreasing company size. The
response rate was especially low in large companies, cf. “gatekeeper [Corra02]”
problem.
The second reflection from the main study is about the questionnaire design. A pre-
study often clarifies both research questions and concrete questions in a later
questionnaire.
The third reflection is about contacting the companies. Through a list of potential
companies, we had to find a contact person in each company. For convenience samples,
there are few problems in getting hold of a potential respondent, as we often have
personal and professional ties from earlier, joint activities. But for a large corporation,
from a census or member list and with thousands of employees, it can be very
frustrating to try to locate a quality manager, development/project manager, or just an
“IT responsible”. The final struggle is to convince a possible respondent over the phone
to use his/her time and effort to fill in a questionnaire, sent by email. The results show
that random sampling and the ensuing contact process appears to be much more
expensive (5 times?) than convenience sampling. In our case, this is because we got
stuck with no previous contact person, and scant pre-knowledge and “goodwill” from
such companies, especially the large ones.

In this chapter, we have illustrated the collected data of each study and the results to the
research questions. Discussion of these results, the possible validity issues, and
limitation of each study will be given in the next chapter.
74
Evaluation and discussion
7 Evaluation and Discussion
This chapter discusses our contributions to research questions RQ1 to RQ4. We

evaluate our contributions against the state-of-the-art. The lessons learned from
performing the main study are also discussed. In addition, we explain the possible
threats to validity of each study to help the reader interpret our conclusions.
7.1 RQ1: How to Improve the Reuse of In-house Built

Components?
For research question RQ1, we investigated the state-of-the-practice of reusing in-house
built components and proposed how to prepare for more reuse through the preliminary
study (see paper P1 [Li04a] in Appendix A). The study focused on four issues as
following, and contributed to C1, i.e., increased understanding on the risks of reusing
in-house built components.
7.1.1 Component-Related Requirements (Re) negotiation

A lot of researchers focus on how to improve the efficiency of component-related
requirements (re)negotiation in COTS-based development [Morisio00], [Sai03],
[Tran99]. The main reason is that people think the challenges in requirements
(re)negotiation are due to the lack of access to source code, to timely vendor supports,
or to the lack of engineering expertise to modify the integrated components [Tran99].
The above constraints on COTS components are not considered as challenges with built
in-house components. We found that although 92% of our investigated developers think
that requirements of their typical projects are flexible, less than half think that the
component-related requirements (re)negotiation in their typical projects works well.
Since components are intended to be used ‘as-is’, it is possible that an in-house reusable
component meeting all the requirements will not be found. So, even though the
components are built in-house, requirements (re)negotiation is necessary. We do not
claim that the requirements (re)negotiation based on in-house components is more
difficult than COTS-based components. We just emphasize that requirements
(re)negotiation based on in-house components is also important but not efficient.
Moreover, we found that when IT companies move from a low reuse level to a higher
reuse level, they probably cannot expect that component-related requirements
(re)negotiation becomes easier and more efficient.
75
7.1.2 Component Repository

Some researchers have claimed that a repository is important, but not sufficient for
successful reuse [Rothenberger03], [Morisio02b]. Our data confirm that developers are
positive, but not strongly positive to the value of a component repository. So, this result
gives further support to their conclusion. In addition, we concluded that companies are
not expected to invest in a repository to increase reuse.
7.1.3 Component Understanding

Transferring component knowledge from the component developer to the component
user is critical for successful component reuse. We found that most developers got the
knowledge about components from informal channels, such as previous experience and
local experts. The most important feature of a component is the separation of its
interface from its implementation. Moreover, current component documentation
technologies cannot describe all the information developers require, such as
performance, reliability, and security. Therefore, informal knowledge transfer should be
considered to supplement the insufficiency of formal component documentation and
specification. One possible solution is to have special interest groups or mailing lists for
a component (or a group of similar components) so that component users can share
knowledge and experience of component usage. We also found that the higher a
companies’ reuse level, the less satisfied a developer is with the component
documentation. Rothenberger et al. [Rothenberger03] concluded that combining reuse
education and training provided for staff with other reuse activity can lead to success in
reuse. Our conclusion implies that when a company moves from a low reuse level to
high level, more effort should be spent on the component documentation and
component knowledge management.
7.1.4 Quality Attributes of Components

Component-based development relies on the availability of high quality components to
fill roles in a new intended system. When components are created or changed, we must
ensure that they not only fulfil the functional requirements, but also quality
requirements. We found that most developers are not satisfied with the specification of
the quality attributes of components and therefore cannot use this information.
Therefore, how can we model quality properties of both components and systems, and
reason about them, particularly in the early stage of system development is still a key
challenge in component-based development.
7.2 RQ2: How to Do SPI in OTS Component-Based

Development?
For the research question RQ2, we compared theories from the academic field and
industrial practice. We contributed to C2, i.e., summarizing what is the current state-of-
the-practice of SPI and OTS selection in OTS component-based development (see
papers P2 [Li06a], P4 [Li06b], and P7 [Li05b] in Appendix A). In addition, we
contributed to C5, i.e., validated newly published theses T4 and T5 in [Torchiano04]
that are relevant to the SPI in OTS component-based development (see paper P3 [Li05a]
76
in Appendix A). Our contributions can be used to help project managers to decide how
to improve the whole life cycle of the OTS-based projects and how to select the OTS
components based on the project contexts.
7.2.1 How to Improve the Whole Life Cycle

Concerning the conclusions drawn by [Boehm99], we agree that using OTS components
in system development involves potential danger, especially when using unfamiliar
OTS components. Although the risk driven processes proposed in [Boehm99] may give
benefits, results of RQ2.3 (see Section 6.2.2) show, however, that most projects
integrate OTS components successfully with traditional processes, such as waterfall,
incremental with prototyping, XP, or V-model. Results of RQ2.3 (see Section 6.2.2)
show that most of the actual development processes were decided before the decision of
using OTS components was made. In addition, results of RQ2.5 (see Section 6.2.2)
show that company rules pre-defined the main development processes in many OTS-
based projects. According to the results of RQ2.3 to RQ2.5 (see Section 6.2.2), we
summarize the state-of-the-practice of OTS-based development processes into seven
different scenarios. The classification of the scenarios is based on the phase of the make
vs. acquire decision and the developers’ familiarity with the OTS candidates.
Scenarios 1 to 3: Scenarios of planned OTS-based software development
In scenarios 1 to 3 (see Figure 7-1), project members prefer to evaluate the possibility of
using OTS components right from the start. Due to time-to-market, cost, or capacity
issues, project managers realize that it is difficult, impossible, or not cost-effective to
make everything from scratch. In these scenarios, the first step is to do a make vs.
acquire decision (see step A in Figure 7-1). In case project members decide not to use
OTS components, the development process will be the same as the non-COTS based
development. If project members decide to use OTS components, they then need to
choose the main development process (see step B in Figure 7-1). To decide the main
development process, main issues are the traditional risks in a project, such as the
stability of the requirements. One of the other issues is their familiarity with possible
OTS candidates. According to the project members’ familiarity with the OTS candidates,
we define three scenarios as scenarios 1 to 3:
- Scenario 1 – planned OTS-based project with unfamiliar OTS candidate: In

this scenario, the project members prefer to use OTS components to provide
certain functionalities of the system. However, they are not familiar with the OTS
candidate at all. To decide the main development process, the OTS relevant risks,
such as the quality of OTS components and the vendor’s support, should be
seriously considered. In addition, project members should pay attention to other
risks, which are not relevant to an OTS component but the overall project (e.g.,
which process is more suitable for projects with unclear or unstable requirements
- waterfall or an incremental with prototyping model?).
- Scenario 2 – planned OTS-based project with some familiar OTS candidates:
In this scenario, the project members have certain experience with some of the
possible OTS candidates. However, there are still certain candidates that they
77
have never used before. In order to decide the main development process, the
issues are similar with those of scenario 1.
- Scenario 3 – planned OTS-based project with familiar OTS candidates: In this
scenario, the project members are familiar with all possible candidate
components. When deciding the main development process the OTS relevant
risks are not as critical as in scenarios 1 or 2. Other non-OTS relevant factors,
such as company rules or requirement flexibility may be used to decide the main
development process. In this scenario, the main development process can be any,
such as waterfall, incremental, or XP. It may not need to be changed because the
project members are very familiar with the OTS component candidates.
Figure 7-1 Scenarios 1 to 3
The COTS-based development model in [Morisio00] illustrates the differences

between COTS-based development and traditional software development, such as new,
reduced, or modified activities. The underlying assumption of this model is that the rate
or degree of using OTS components is fixed in the very early phases of a project and
that the main development process has to be modified accordingly. The model can
therefore be used as a reference in scenarios 1 to 3.
Scenarios 4 to 6: Scenarios of unplanned OTS-based software development
In scenarios 4 to 6 (see Figure 7-2), there is no clear intention of using OTS components
in the early phases of such a project. Project members decide on the main development
process according to some non-OTS relevant factors, such as company rules and
requirement stability. Due to the time-to-market pressure or internal capability, project
members start to evaluate the possibility of using OTS components at certain stages of
the project. At our follow-up study (see Paper P7 [Li05b] in Appendix A), several
78
industrial participants mentioned such conditions. According to the project members’

familiarity with the OTS candidates, we classify three scenarios from scenarios 4 to 6.
- Scenario 4 – Unplanned OTS-based project with unfamiliar OTS candidates:

In this scenario, the project members decided to use OTS components in a late
phase of the project, such as detailed design or coding. However, they do not
have any previous experience with the possible OTS candidates. Therefore, the
use of OTS components may bring several problems into the whole project. The
project members need a second risk-evaluation to investigate whether the
development process should be modified concerning the use of OTS components.
For example, in projects with pure waterfall processes, the project members may
need to add a short prototyping step to evaluate the OTS component and to
negotiate the use of OTS components with their customer.
- Scenario 5 – Unplanned OTS-based project with some familiar OTS
candidates: In this scenario, the project members decide to use OTS components
in a late phase. However, they do not have previous experience with all OTS
component candidates. In order to decide the main development process, the
issues are similar to those of scenario 4.
- Scenario 6 – Unplanned OTS-based project with familiar OTS candidates: In
this scenario, although the project members decide to use OTS components in a
late development phase, they have enough experience with the OTS component
candidates or are able to hire a person with such experience. In case the project
members have used the OTS component many times before, the OTS component
can be regarded as comparable to in-house built components. Although the use of
OTS component is a little risky due to the same reasons as described in the
scenarios 4 and 5, this scenario is less risky because of using familiar OTS
components. It may not be necessary to do major changes to the current
Figure 7-2 Scenarios 4 to 6
79
Scenario 7: The scenario of semi-planned OTS-based software development
Scenario 7 is a mix of scenarios 1, 2, or 3 with scenarios 4, 5 or 6, whereby the use of

OTS components is planned for some parts of the system. The main development
process is decided according to the use of OTS components. However, in a late phase of
the project, project members may need to add some unplanned OTS component due to
the same reasons as for scenarios 4, 5, or 6.
The EPIC approach [Albert02], like RUP, consists of four phases and each phase
consists of one or more EPIC iterations. The information of four spheres of influences,
such as stakeholders’ needs, architecture design, risk, and marketplace, are gathered and
evaluated in each iteration. The project management, customer requirements, and
system architecture will be adjusted according to the trade-offs of the four spheres of
influences. Since every iteration includes a complete sub-process to make the trade-off
decision and select the OTS component, it is more flexible and suitable to be used as a
reference in scenarios 4 to 7.
7.2.2 How to Select the OTS Components

In these different scenarios above, various OTS component selection strategies and
processes can be used. However, our results support the thesis T4 (see Section 6.2.2)
proposed in [Torchiano04] and show that formal selection processes were rarely used.
In addition, our results contradict T5 (see Section 6.2.2) in [Torchiano04] and conclude
that functionality completeness was regarded more important than architectural
compliance in COTS component selection.
In scenarios 1, 2, 4, 5, and 7, some unfamiliar OTS component candidates are going
to be used. The selection process for unfamiliar OTS components can either be an
Internet search with hands-on trial-based, or a formal selection process, such as
processes shown in [Kontio96], [Tran97b], [Morisio97], [Ochs01]. In scenarios 3 and 6,
all possible OTS component candidates have been used by the project members, the
selection process could mainly be familiarity-based.
In scenarios 1 to 3, the use of OTS component is well-planned. In general, the OTS
component will be selected in the early phase of the project in these scenarios. Selecting
OTS components in the early phase can help to design the software architecture with
consideration on constraints of the OTS components. Therefore, it is easier to avoid the
mismatch between the OTS components and other components in the system.
In scenarios 4 to 7, the OTS components are usually evaluated at a phase in which
most of the other components are already integrated in the system. Since the goal is to
integrate the OTS component into the current system and to check its integration
thoroughly, integrating the demo version of the OTS component and performing a
hands-on trial is necessary. In case the OTS component is selected in the very late stage
of the project, it is better to select a component that has a loose coupling with other
components. OTS components tightly coupled with existing components may impose a
large number of changes into the system.
80
7.3 RQ3: How to Mitigate Risks in OTS Component-Based

Development?
For research question RQ3, we contributed to C3, i.e., empirical validation of risk
management proposals in OTS component-based development (see Papers P2 [Li06a],
P5 [Li06c], and P7 [Li05b] in Appendix A). In addition, we contributed to C5, i.e.,
empirically validated three newly published theses (see Paper P3 [Li05a]in Appendix
A), i.e., T2, T3, and T6 in [Torchiano04], which are related to risk management in OTS
component-based development.
Concerning the theses related to the risk management in OTS-based development,
results of thesis T2 (see Section 6.2.2) show that architecture mismatches were more
frequent than standard mismatches. With the maturity of COTS component standards,
we believe that less standard mismatches will occur in the future. Results of thesis T3
(see Section 6.2.2) show that missing or incomplete features in COTS products raise
significant problems. It is therefore important to develop effective techniques that allow
seamless extension of existing products. Results of thesis T6 (see Section 6.2.2) show
that COTS component users actually pushed vendors to change the COTS components
either because of bug fixes or new features. However, this result does not imply that
most COTS component users actually influenced COTS component evolution. In
general, we believe that the COTS component evolution is decided by the vendors’
marketing strategy rather than the requirements of a specific user, unless this user is
very powerful.
Concerning the validation on the risk management proposals, we first compared the
frequency of some typical risks. The results can be used to help project managers to
identify the most possible risks. We then investigated the correlations between the
problems occurred and the risk management activities performed. The results are
summarized in the following Section 7.3.1 to 7.3.6, and can be used to help project
managers to do a proper risk management plan based on the risks identified.
7.3.1 How to Mitigate the Cost Estimation Risks

Both the effort in selecting OTS components and the effort to integrate them sufficiently
are difficult to estimate [Boehm03], [Mitre05]. Basili and Boehm [Basili01]
hypothesized that COTS-based development is currently a high-risk activity, with effort
and schedule overruns exceeding non-CBS software overruns. The results shown in
Figure 6-8 illustrate that the wrong estimation of the integration effort happened more
frequently than other problems.
To estimate the selection effort correctly, Boehm et al. [Boehm03] propose to
identify the core “showstopper” requirements and filter all the OTS candidates that do
not meet these during the initial assessment and then proceed for a more detailed
assessment with the remaining OTS candidates. Although this method may be
applicable to limit the number of OTS components that need to be evaluated, it is still
difficult to estimate the possible effort to compare the limited OTS candidates, because
both technical and non-technical issues of the OTS components need to be evaluated
and compared [Rose03]. Our findings of RQ3.4 (see Section 6.2.3) show that in case
the effort on learning in the OTS components has been seriously considered, it will be
easier to estimate the selection effort correctly. Boehm et al. [Boehm03] found that one
81
common error in OTS-based development is the overly optimistic OTS learning curve
and propose that a most likely OTS learning curve must be accounted for during
planning and scheduling. Initially, the OTS component’s functional capability must be
well understood, which can be a daunting task because there may be a significant
amount of functionality to learn and that functionality might involve new technologies
to the team. Also, there is more to learn than just the functionality. The methods of
integration that the OTS component employs must be understood sufficiently to ensure
that the integration can be succeeded into the current architecture. Since the intention of
using OTS components is to shorten time-to-market and to save development cost, OTS
component users are eager to acquire the components and to integrate them as soon as
possible without understanding them thoroughly. Rose [Rose03] pointed out that
developers are usually working with one or more OTS components for which they most
likely have only a partial understanding. To improve the estimation on the OTS
integration effort, the proposed activity is to use OTS-sensitive analytical and budget
processes [Rose03], [FAA03], such as COCOTS [Abts00]. In our study, we did not
investigate the effect of using COCOTS because our pre-study (see Paper P2 [Li06a] in
Appendix A) did not find any companies using it. Therefore, we did not investigate this
activity because we wondered whether this activity has been popularly used in industry.
Although the results shown in Figure 6-9 reveal that the risk management activity Rm5
(i.e., effort in learning OTS component was seriously considered in effort estimation)
has been widely used, we still wonder whether there are good tools to help the project
managers to estimate the learning effort precisely.
7.3.2 How to Mitigate the Quality Risks

To get a high-quality system, it is important to ensure that every component in the
system does not bring negative effects on the whole system. The results shown in Figure
6-8 illustrate that OTS components rarely have a negative effect on system quality.
Our results of RQ3.3 and RQ3.4 (see Section 6.2.3) show that the black-box testing,
and early integration testing, which have been very popular in practice, helped to avoid
possible quality problems. Boehm et al. [Boehm03] discovered that the black-box
testing is always imposed in OTS-based development and suggested that a risk based
testing plan must be developed and accounted for in the project life cycle schedule. In
the COTS component market, the salesman may make faulty claims about the
functional and non-functional characters of their products [Boehm03], [FAA03],
[Mitre05]. Thus, the users need to do some testing themselves to ensure that the quality
of the component is the same as that claimed by the vendor. For the OSS components,
there is never enough information to illustrate the full picture of the functions and the
quality of the components. Thus, users need to test and evaluate the issues they are
interested in by either reading the source code or doing black box testing.
However, the performance of the system highly depends a lot on the interaction
between components and the environment, such as the hardware and the workload. It is
therefore difficult to predict and evaluate by just incremental integration testing. Our
investigated risk mitigation activities did not show effective reduction on the OTS
components’ possible negative effect on system performance.
Although other solutions have been proposed to mitigate the possible quality issues
of the overall system, such as selecting the OTS based on architecture instead of
functionalities [Rose03] [Torchiano04], [Li06a], or leverage the commercial
82
infrastructure [FAA03]. Our results of RQ3.4 (see Section 6.2.3) did not show a strong
effect from these activities. A possible reason is that most OTS components investigated
in our studies followed the industrial standards or infrastructures - such as COM,
CORBA, .Net, and EJB - by default. The possible quality problems caused by the
architecture mismatch itself are therefore minimized.
7.3.3 How to Mitigate the Requirements Risks

Another popular issue on OTS component integration is the requirements changes and
mismatches with OTS component functionalities [Boehm03], [FAA03], [Rose03]
[Mitre05].
Since it is difficult to change the OTS component to follow the customers’ changes
in requirements, it is important for the OTS component integrator to be able to negotiate
requirements with the customer [Boehm03], [Rose03], and involving customers into
make vs. acquire decision and OTS component selection [FAA03]. However, our results
in Figure 6-9 show that very few integrators have actually involved customers into the
make vs. acquire decision and the OTS component selection. Results of the pre-study
(see Paper P2 [Li06a] in Appendix A) illustrate that it is not easy to negotiate
requirements with the customer in OTS component-based development. One reason is
that customer only cares about the final product without willingness or capability of
knowing the technical details.
To mitigate the possible requirements risks, our results of RQ3.4 (see Section 6.2.3)
show that careful evaluation and testing of OTS components in the selection phase
helped to increase the possibility to follow the customers’ requirements changes. The
possible explanation is that people understand the OTS components better when they
did testing and quality evaluation in the selection phase. They therefore know the
capability and the possible extensions of the OTS components.
Another possible activity is to first integrate the unfamiliar, i.e., high risk or critical
OTS components [Rose03], [Li06a]. Integrating the critical components first helped to
provide the solid foundation for the architecture of the system [Rose03]. Integrating the
high risk or most complex components first can help to find problems in the early phase
of the project, which may give the integrator leeway to negotiate with their customers to
redefine the requirements [Lauesen06]. To negotiate the requirements with the
customer, educating the customer in requirements negotiation will be critical, but this
must be accomplished in a manner that is politically viable for the particular customer
[Rose03]. The possibility of negotiation depends on the willingness of the customer to
negotiate issues and to accept changes. Our results of RQ3.4 (see Section 6.2.3)
illustrate that customers in certain business culture, such as in Germany, are more
difficult to accept requirements negotiation than others, as shown in Section 5.5 in
Paper P5 [Li06c] in Appendix A.
7.3.4 How to Mitigate the Debugging and Deployment Risks

Our results shown in Figure 6-8 reveal that the difficulty in locating the cause of defects
happened more frequently than other problems. Kotonya and Rashid [Kotonya01] point
out that OTS component offering similar functionalities may have very different system
resource requirements (i.e., memory and processor requirements). When a problem is
identified, it may not be obvious where the root cause of the problem is located. The
83
problem could be in one of the components, it could be in the integration code, or it

could be a misinterpretation of the semantic operation of a component interface
[Rose03].
The results of the research question RQ3.4 (see Section 6.2.3) show that this risk can
be mitigated by giving developers enough time to learn and understand the OTS
components. Since both COTS and OSS components are acquired from a third-party
organization, it will require a lot of time for the users to understand the underlying
design and code in these components. It is especially difficult for the COTS component
users to use the components properly when source code is not available. In case there
are defects in the OTS components or the interactions between the OTS components are
not properly controlled, it may take longer time for the component users to figure out
the problems than they had expected.
For the possible deployment problem, our findings of RQ3.4 (see Section 6.2.3)
indicate that an effective way of controlling it is to start integration testing as early as
possible. In the early days of integrating OTS into a system solution, there was a false
assumption that the need for integration testing was either minimal or not needed at all.
It seems that there was a belief (actually a false hope) that once the OTS components
were installed, the system would be operational [Rose03]. The OTS providers are only
knowledgeable in their own OTS components. If there is no strong evidence that
problems in the integrated system are caused by a specific OTS component, the
component vendor may not want to help the users to debug and solve the problems. The
benefit of doing integration testing immediately after one OTS component is integrated
is that it is easy to conclude that the problems lie in the newly added component. An
additional possible mitigating activity is to do some analysis to determine the order in
which the OTS components are to be integrated [Spc01].
7.3.5 How to Mitigate the Maintenance Risks

Maintaining an OTS component-based system is difficult. The post-deployment costs
may exceed the development costs [Basili01]. The different customer-vendor evolution
cycles may result in an uncertainty about how often the OTS components in a system
may have to be replaced and the extent of the impact of such a change on the rest of the
system [Kotony01, [Rose03], [Mitre05], [FAA03].
One risk mitigation solution proposed is to build an explicit, formal strategic
relationship that includes mutual non-disclosure agreements to ensure that the provider
gives full disclosure of the products future upgrades [Rose03]. However, this strategy is
only applicable in a few situations. In case the customer is not regarded as a major one
by the COTS component provider, it is difficult to build such a relationship. In the case
of using OSS components, it is also difficult to build such a relationship, because the
OSS components are updated by volunteers and nobody knows what functions may be
added in future releases.
Another risk mitigation activity is to have senior analysts (serving as knowledge
keepers) directly involved in the analysis of the updates and deriving effective field
upgrade procedures and schedules [Rose03], [Li06a]. This is a non-trivial system
analyst task that requires appropriate level of expertise, as there is very little in the
literature to help guide the analysts to a feasible solution [Rose03]. The existence of the
knowledge keeper has been investigated by RQ3.4 (see Section 6.2.3) and show
effective results on mitigating the possible risks due to the OTS component evolution.
84
Since the knowledge keepers know the current component version being used, it is
possible for them to evaluate the new versions and to figure out the benefits and risks of
updating the current version with the newer one. The personal capability and experience
remain the dominant factors influencing OTS-based development [Basili01]. Therefore,
it is important to involve OTS- knowledgeable individuals in all analytical processes
[FAA03].
However, planning maintenance of an OTS component-based system is very difficult
because the OTS component users have no control of the release cycle of the OTS
components being used. Since the release cycle of the OTS component depends on the
marketing strategy of the component vendors, the component users usually have no idea
when the new release of the OTS component will be published. Our investigated risk
management activities focus on the activities that can be controlled by the component
users. Such risk management activities did not show effectiveness to mitigate the
maintenance plan risk.
7.3.6 How to Mitigate the Provider Support Risks

Another issue in OTS component-based development is to get enough technical support
from the provider. Although the users can have access to the source code when using
the OSS components, few of them have actually modified the code to meet their
requirements [Li05a]. It is also advised to avoid modifications of OTS components,
even when it is possible [FAA03]. COTS component users can get support from the
provider by signing a support contract. However, COTS providers may still stop to
support their products due to different reasons, or even go out of business [Rose03], or
the actual version of a COTS component is too old to be supported [Basili01]. For the
OSS components, some of them are currently well supported by the volunteers in the
OSS projects. However, it does not ensure that the users can get the same level of
support in the future. The users have no control of the technical support they want to get
from the OSS community. As our study tried to find out the common risk mitigation
activities for both COTS and OSS components, we did not investigate the contract and
licensing techniques, which is not applicable for the OSS user. We therefore looked
more on the activities that are under control among OTS component users.
For the provider support risks, our results of RQ3.4 (see Section 6.2.3) show that the
most important remedy is still to select the right providers. It is necessary to perform
continuous watch on the support reputation of the OTS component providers, as
proposed in [FAA03], [Mitre05]. In addition, it is important to trace the history of the
provider support by consulting the knowledge keeper.
7.4 RQ4: How to Make the Decision between COTS or OSS

Components?
For research question RQ4, we compared the use of COTS and OSS components in
practice. We contributed to the C4, i.e., increased understanding on the decision
making practices in OTS-based development (see paper P6 [Li06d] in Appendix A).
We summarized who, why and how these components are used. We then proposed a
brief guideline on selecting one from two possible options - COTS components and
OSS components.
85
7.4.1 Why Did Developers Decide to Use OTS Components

Using COTS and OSS components promises shorter time-to-market and increased
productivity [Voas98c]. Our results of RQ4.2 (see Section 6.2.4) confirm that the main
expectation of using OTS components is to shorten the time-to-market and to save
development cost. Results also show that COTS users believe that paid software will
give good reliability and will follow the market trend. They also believe that the COTS
vendor will provide good technical support. On the other hand, the main motivation of
using OSS components is that code could be acquired for free and that the source code
is available. It means that the OSS component users care more about the initial
investment. Another attraction of OSS components is that the marginal cost of scaling
up is zero, i.e., using OSS does not require additional licences as the number of
installations grows. This benefit is especially important when integrating OSS
components into a software product with many installation bases.
7.4.2 How Were the OTS Components Used in Practice?

If “white-box” vs. “black-box” is regarded as the main difference between COTS and
OSS components, the results of T1 (see Section 6.2.2) show that COTS and OSS
components are not as different as one might assume. COTS components were not
always “black-box”, as 33% of the COTS components selected in this study opened
their source code to users. If the source code is available, both COTS and OSS users
tend to read the code. However, neither COTS nor OSS component users did change
much of the available source code. From this result, we can generally conclude that the
OSS component was used in the same way as COTS component, i.e., integrated without
modification. Without source code, it is difficult for COTS component users to identify
whether a defect is inside or outside the component. Although the source code of OSS
components is available, OSS users might not have bothered to understand the source
code of an OSS component before starting to use it. It is therefore difficult for them to
figure out if a defect is inside an OSS component or not.
By analysing all activities normally performed when integrating OTS components,
Giacomo, P. D. [Giacomo05] concluded that the OSS component is not completely
different from the COTS component. Our results of RQ4.1 (see Section 6.2.4) show that
both COTS and OSS components were used in projects and companies with similar
profiles. The most emphasized requirements of the final systems were time-to-market,
reliability, and performance. Security was surprisingly listed as the least emphasized
requirement of the system. There are many discussions about whether OSS components
are more secure than COTS components. Raymond [Raymond99] argued that “given
enough eyeballs, all bugs are shallow.” Others argued that it is the quality, not the
number of eyes looking at code that count [Hissam02]. Our results show that OTS
component users are still very cautious of using either COTS or OSS components in the
security-critical systems. So, software security aspects may not be distinguished
between COTS and OSS components. However, trust in the security of third-party
components may become an issue in the future.
Although COTS users believe that the paid components should have good quality
and good technical support, our results of RQ4.3 (see Section 6.2.4) show that they
were not more satisfied with reliability and performance of the components than OSS
users. In addition, COTS users had significantly more difficulties in estimating the
86
component selection effort. The possible reason is that it is not easy or cost-effective for
COTS users to switch to another COTS component once they have made the decision
and paid for some selected components. Therefore, COTS users might be more cautious
in selecting the right COTS component and spend more effort than estimated on
selecting the best one. COTS users have also more difficulties on following the
requirements changes. It might be because the source code is not available and it is
therefore difficult to change the source code of the COTS component.
Although OSS users did not have a commercial support contract with OSS
component providers, results of RQ4.3 (see Section 6.2.4) show that they were not less
satisfied by the technical support than COTS users. The possible interpretation is that
OSS users would like to read and change the code themselves, instead of asking for
help, as we found in the key motivations of using OSS.
7.4.3 COTS or OSS? Our Suggestions

In a system with undefined or fast changing requirements, OSS components might be
better than COTS components. It might be easier for OSS users to change the source
code of the OSS component if their customers’ requirements do change. Therefore, OSS
components are ideal when the software solution requires high customization
[Madanmohan04]. However, COTS components users can also ask for source code
from the vendor in some situations. In this case, there will be fewer difficulties on
following the requirements changes.
Concerning the use of domain-specific OTS components, it is important to ensure
that the evolution of the selected OTS component will follow the market trend.
Although a COTS user cannot directly influence the evolution of the selected COTS
component, results of RQ4.2 (see Section 6.2.4) show that these users still trust that the
COTS vendor will update the component according to market needs.
OSS components can be changed for similar market reasons, but can also be changed
for political or social reasons – not necessarily in a direction suitable to the organization
[Hissman01]. It is therefore important for OSS users to ensure that they are planning to
use the software in the way it was intended to be used and the way most other users are
using it. If OSS users stray too far away from the beaten path, the OSS developers might
not want to help them when they run into trouble, or OSS developers will not accept the
changes contributed by them [Madanmohan04]. More and more OSS components are
supported by a commercial organization [Stewart05]. It is possible for OSS users to
select components with commercial support in order to avoid the possible maintenance
and evolution problems.
7.5 Evaluation of Validity Threats

Four groups of validity threats in empirical research are considered important and
validity threats of the individual studies are discussed in this section.
7.5.1 Possible Validity Threats in the Preliminary Study

The preliminary study (see paper P1 [Li04a] in Appendix A) is used to investigate RQ1
and contributed to C1. The study used the questionnaire to investigate the developers’
87
attitude on reuse in three IT companies. Some threats to validity of this study and how
these are handled are as following:
• Internal validity. A major threat to internal validity is that we have not assessed
the reliability of our measurement. Most variables are measured on a subjective
ordinal scale. An important issue for future studies is to ensure the reliability and
validity of all measurement. In this survey, we gave clearly specified concepts in
the questionnaire and provided support to possible misunderstanding. These
methods increased the reliability.
• Construct validity. In our case, the main construct issue applies to the variables
chosen to characterize the data set. The independent variable, i.e., reuse level, is
the most sensitive one. We asked several questions in the questionnaire to
evaluate this variable. The results of these questions give a consistent value on
this variable.
• External validity. The small sample size and lack of randomness in the choice of
companies and respondents are threats to external validity.
• Conclusion validity. Although we used some statistical methods to analyse
results of this study. This study is still an explorative study. Future studies will
be implemented to give more statistically significant results.
7.5.2 Possible Validity Threats in the Pre-study

The pre-study (see papers P2 [Li06a] in Appendix A) used semi-structured interviews to
clarify research questions RQ2, RQ3, RQ4. Some threats to validity of this study and
how these are handled are as following:
• Internal validity. The first possible threat to internal validity is possible

misunderstanding of respondents’ answers. Although two interviewers carried
out the interviews and there was only one in each interview, we taped all
interviews. Listening to the tape helped ensure correct interpretation of answers
and comments. However, having an independent (third) person to listen to the
tape might increase data quality. The second possible threat to internal validity is
that people tend to talk about examples of good practice instead of problems
they encountered. As we had a confidentiality agreement with the respondents,
the respondents were encouraged to tell the truth. However, the respondents
might give only the most salient problems they encountered and forget some
minor problems because we asked for data about past projects. The third
possible threat to internal validity is that the life cycle phases, such as
requirement, design, and implementation, may not be clearly separated in
projects using prototyping and incremental processes. We called back some
respondents to ask whether they had specific software architecture in mind
before the make vs. acquire decision and COTS component selection. Their
answers gave future clarification on whether COTS components were selected in
the requirements or design phases. The fourth possible threat to internal validity
is that some projects might not tailor the process because their processes already
support COTS usage. So, we might miss some necessary process improvements
88
that have been performed before the project start because we asked only salient
process changes within the studied projects.
• Construct validity. In this study, most variables are taken directly, or with little
modification, from the existing literature. To ensure construct validity, we did
two rounds of pre-tests on the interview guide in ten local IT companies. About
10% of the questions and alternatives in the interview guide were revised based
on the feedback from the pre-test. In addition we used three meta-questions to
evaluate whether the questions in the interview guide were understood
consistently. Answers of these meta-questions illustrate that the questions are
easy-to-understand and reliable. In the data analysis part, we divided the projects
into successful and unsuccessful projects. The criterion to define a successful
project is that the integrated COTS components should contribute positively
both to time-to-market and to the quality of the final system. However, this
criterion may not apply for all kinds of projects as some projects might emphasis
only time-to-market, and others might emphasis only quality.
• External validity. The primary threat to external validity is that the study is
based on few and possibly not typical companies in Norway. According to
Norwegian “Census Bureau” (SSB) [SSB02], the mean value of employee
number in Norwegian IT companies is 6. The first possible external validity
threat to this study is that our selected companies are mostly medium and large
sized. However, the selected projects covered several application domains. The
second possible threat to external validity is that the size (based on person-
hours) of the projects might bias the conclusions from this study. We have
divided the projects into three groups: small (less than 10,000 person-hours),
medium (between 10,000 and 100,000 person-hours), and large (more than
100,000 person-hours). We found out that the actual process used, the problems
encountered, and the good practice crossed small, medium, and large projects.
The third possible threat to external validity is that most COTS components are
well-defined and provided only secondary (support) functionality of the
delivered system. The process customizations may be different if the COTS
components are intended as the core part of the system, or the COTS
components are large packages, such as ERP and content management systems.
• Conclusion validity. This study is a qualitative study without statistical analysis.
The conclusions of this study are descriptive and may need further refinement
and verification.
7.5.3 Possible Validity Threats in the Main Study

The main study (see papers P3 [Li05a], P4 [Li06b], P5 [Li06c], and P6 [Li06d] in
Appendix A) investigate the clarified research questions RQ2, RQ3, RQ4 from the pre-
study. The data of the main study were collected by a questionnaire and analysed by
statistical methods. The results of the main study show our contributions C2, C3, C4,
and C5. Some threats to validity of this study and how these are handled are as
following:
• Internal validity. We promised respondents in this study a final report and a

seminar to share experience. The respondents were people who wanted to share
89
their experience and wanted to learn from others. In general, we think that the
respondents answered the questionnaire truthfully. However, different people in
the same project might have different opinions on the same project. Asking only
one person in each project might not be able to get the whole picture of the
project. Due to length limitation of a questionnaire, we asked the respondent to
fill in information about only one component in the project. The possible threat
is that other OTS components in the same project might give different answers
to our questions.
• Construct validity. In this study, most variables and alternatives are taken
directly, or with little modification, from existing literature. The questionnaire
was pre-tested using a paper version by 10 internal experts and 8 industrial
respondents before being published on the SESE tool [Sese04]. About 15% of
the questions have been revised based on pre-test results. One possible threat to
construct validity is that respondents from different companies may have slightly
different understanding of the same concept, such as software architecture and
architectural mismatches.
• External validity. We used different randomization methods to select samples in
different countries. However, the sample selection processes were not exactly
the same due to resource limitations. Another possible threat to external validity
is that our study focused on OTS components followed the definition in Section
2.6. Conclusions may be different in projects using complex and large OTS
packages, such as ERP, content management systems, and web services in
general.
• Conclusion validity. This study is a state-of-the-practice one. We studied what
had happened in industrial projects. The same size of this study is generally
enough for statistical methods used to analyse the result. However, we generally
treat the data from three countries the same. The cultural differences between
three countries might affect the reliability of the conclusions.
7.5.4 Possible Validity Threats in the Follow-up Study

The follow-up study (see paper P7 [Li05b] in Appendix A) was performed informally as
an industrial seminar. This study is designed to briefly investigate the cause-effects of
our results on research question R2, RQ3, and RQ4. It improves our understanding on
contributes C2 to C5.
The possible threats to the construct validity are that there are no well-defined
metrics to measure the answers of respondents. The possible threat to the external
validity is that the conclusions are based on personal opinions without verifications on a
large scale.
7.6 Summary
In this chapter, we have illustrated our contributions to each research questions. We
compared our results with the state-of-the-art and gave some suggestions on process
improvement, risk management, and decision-making in OTS component-based
development. Due to possible threats to validity of each study, the conclusions of each
study should be interpreted considering these validity issues
90
Conclusions and future work
8 Conclusions and Directions for Future

Work
This thesis has presented the results of several empirical studies performed to
investigate project management issues of OTS component-based development. The
studies included several steps with mixed-method research design to take benefit of
different empirical methods and answer the research questions reliably.
8.1 Conclusions
Empirical research can be performed to verify theories, develop new theories or extend
existing ones, and improve practice. The thesis contributes to all these aspects by
describing difference aspects of reusing components, i.e., the power of example
[Parastoo04]; verifying existing theories and assessing existing method, i.e., the power
of replication; generating new theories, hypotheses, or methods, i.e., the power of
generalization.
8.1.1 Describing Different Aspects of Reusing Component

We have investigated challenges related to four key factors for development based on
in-house components, especially in development-with-reuse. These factors are
component-related requirements (re)negotiation, component repository, component
understanding, and components’ quality attribute specification. We also compared three
IT companies with different reuse levels to study the possible trend and challenges in
these factors when more and more code will be encapsulated as reusable components
inside a company. We concluded that component-related requirements (re)negotiation
for in-house built components is important but not efficient. The efficiency will not
increase with higher reuse level. We confirmed that the component repository is not a
key factor for successful reuse and the potential value of a component repository will
not increase with higher reuse levels. For component understanding, we found that most
developers are not satisfied with the component documentation, and developers’
satisfaction with component documentation decreased with a higher reuse level. We
recommend that the informal communication channels (tacit knowledge), through which
developers can get necessary information about the components, should be given more
attention. For the quality attribute specification of components, we found that
91
developers still need to spend a lot of effort on testing, as they cannot get relevant
information from the component specifications.
8.1.2 Verifying Existing Theories and Assessing Existing Methods

We have assessed six newly published theses from the literature [Torchiano04] on OTS-
based development. The results of our study gave support to four new theses and
contradict the two others as follows:
- We support T1 (i.e., open source software is often used as closed source) and
conclude that source code of OSS components was read frequently with very
few modifications. OSS components were de facto used as COTS components,
even if the source code was available.
- We support T3 (i.e., custom code mainly provides additional functionalities.).
We conclude that more addware had been built than glueware in the COTS
components integration.
- We support T4 (i.e., developers seldom use formal selection procedures.
Familiarity with either the product or the generic architecture is the leading
factor in selection.) We conclude that formal selection procedures were seldom
used to select COTS components. Familiarity-based and hands-on experiment-
based selection process were frequently used in practice.
- We support T6 (i.e., integrators tend to influence the vendor on product
evolution whenever possible). We conclude that, in most cases, COTS users
managed to get their required changes in the COTS components from COTS
vendors.
- We are against T2 (i.e., integration problems result from lack of compliance
with standards. Architecture mismatches constitute a secondary issue). We
conclude that there were more architectural mismatches than standard
mismatches in practice.
- We are against T5 (i.e., architecture is more important than requirements for
product selection). We conclude that functionality completeness was regarded to
be more important than architectural compliance in COTS component selection.
We have investigated state-of-the-practice of the main development processes and

OTS component selection processes. The main findings are:
OTS-specific activities. The development process was dominated by existing
company/department rule instead of the decision of using OTS components.
evaluation processes are familiarity-based or hands-on-trial-based. The phase to
select OTS component depends on the project members’ familiarity.
We have also investigated the proposed risks and risk management theories in OTS
component-based development. The main findings are:
92
- The most frequent problems in OTS-based projects are wrong estimation of the
integration effort and the difficulty to locate defects. The least frequent problems
are relevant to the negative effect of the OTS components on the quality of the
whole system.
- To increase the accuracy of effort estimation, it is important to allocate enough
time to learning and understanding the OTS components.
- To avoid the possible quality problems and to increase the efficiency of locating
the defects, it is important to do black-box testing as part of OTS component
selection and to do integration testing as early as possible.
- To mitigate the problems due to customers’ requirements changes, it is
important to learn the OTS components thoroughly in the selection phases and to
integrate the unfamiliar OTS components first. It is also important to be able to
negotiate changed requirements with the customers.
- To increase the efficiency of debugging and system deployment, it is necessary
to understand the components well and to do integration testing early.
- It is critical to keep and share the knowledge about OTS components and the
component providers inside the organization. It shows to help mitigate the
possible risks in system maintenance and evolution.
We have used stratified-random sample selection strategies to select representative

samples in three European countries. The lessons we learned show that all previous
surveys have had problems in defining the study population and in selecting a valid
sample out of this, especially since the study object is a special project, not a company.
We also conclude that we can achieve a stratified-random (random based on size in
three nations) sample of ICT companies, followed by a convenient sample of relevant
projects at best.
8.1.3 Generating New Theories from New Perspectives

By summarizing our findings on SPI in OTS component-based development, we have
classified the state-of-the-practice OTS processes into seven scenarios. In addition, we
have discussed how to decide/modify OTS-based development processes and how to
select OTS components in each scenario.
By discussing practice with industrial representatives, we have briefly investigated
the possible reasons for our discovered phenomena in SPI and risk management. These
findings proposed some possible research issues in OTS component-based development
in the future.
By summarizing our findings on risk management issues for OTS component-based
development, we have recommended how to identify the possible risks and how to plan
risk management in OTS-based projects.
By comparing the expectations and results of using COTS and OSS components, we
have proposed how to select from two options, COTS and OSS components.
93
8.2 Future Work

The results of this thesis, its contributions and limitations, point to several possibilities
for future research as following, which have relevance for theory, methodology, and
practice.
8.2.1 Need to Study the Cause-Effects of the SPI Issues

The main limitation of the study on SPI is that it is a state-of-the-practice survey.
Although the discussion with industrial partners gave us some insights on the reasons of
industrial practice, we need more detailed case studies or semi-structured interviews to
propose and verify cause-effect issues. The intention is to deepen our understanding of
process improvement in OTS component-based development and verify our proposals
from this study.
Another limitation of the study on SPI is that we did not involve business critical
projects, such as safety-critical projects and security-critical projects. Comparing the
differences of SPI in these projects with projects in our study will provide more insights
on SPI issues.
Due to the fast changes in the IT industry and OTS technologies, it is also necessary
to replicate such a study to investigate the possible changes of the state-of-the-practice.
8.2.2 Risk Management Issues Need a Deeper Investigation

We did an exploratory investigation of risk management issues in OTS component-
based development. Due to the explorative nature of the study, further studies are
needed to verify our conclusions. The assumption of our study is that the a priori risk
probabilities of typical risks in the investigated projects are similar. However, it is
possible that there are minor differences in the a priori risk probabilities between
projects. It is therefore necessary to follow several OTS component-based projects from
start by case studies. In the future, we plan to examine the several projects by estimating
risk probabilities before the project start, following the project execution, and measuring
the risk occurrences after the projects are finished. The intention is to investigate the
cause-effect of the risk management activities on the corresponding risks.
Moreover, some of the conclusions of this study need further clarification. For
example, which knowledge needs to be kept and shared in the organization and how to
share it? How to estimate the possible learning effort needed when using the unfamiliar
OTS components? These research questions need to be further studied by case studies
or experiments.
8.2.3 Need to Better Understand OSS Projects and Products

In order to provide guidelines for using OTS components in practice, it is important to
understand how OSS projects are organized. Furthermore, it is important to know how
to cooperate with OSS developers, for example, how to ensure that changes from users
are accepted by the OSS community, and how to motivate developers to enhance and
adapt OSS components into a desirable direction. Another important issue is the
definition of suitable metrics to obtain reliable data about the concrete nature of OSS
projects and products. Future studies have to examine OSS projects in order to identify,
define, and validate more metrics.
94
References
References
[Abts00] Abts, C., Boehm, B. W., and Clark, E. B.: COCOTS: A COTS Software
Integration Cost Model - Model Overview and Preliminary Data Findings. Proc. of
the 11th European Software Control and Metrics Conference (ESCOM-SCOPE
2000), Apr. 2000, Munich, Germany, Shaker Publ., pp. 325-334.
[Albert02] Albert, C. and Brownsword L.: Evolutionary Process for Integrating COTS-
Based System (EPIC): An Overview. SEI, Pittsburgh, 2002, available at:
http://www.sei.cmu.edu/publications/documents/02.reports/02tr009.html.
[Alves03] Alves, C. and Fineklstein, A.: Investigating Conflicts in COTS Decision-
Making. Journal of Software Engineering and Knowledge Engineering, 13 (5): 473-
493, Oct. 2003.
[Atkinson02] Atkinson, C., Bayer, J., Bunse, C., Kamsties, E., Laitenberger, O., Laqua,
R., Muthig, D., Paech, B., Wüst, J., and Zettel, J.: Component-based Product Line
Engineering with UML. Addison-Wesley, 2002.
[Babbie90] Babbie, E.: Survey Research Methods. Wadsworth, 1990.
[Bachmann00] Bachmann, F., Bass, L., Buhman, C., Comella-Dorda, S., Long, F.,
Robert, J., Seacord, R., and Wallnau, K.: Volume II: Technical Concepts of
Component-Based Software Engineering. SEI Technical Report number CMU/SEI-
2000-TR-008, 2000, available at: http://www.sei.cmu.edu/
[Barki93] Barki, H., Rivard, S., and Talbot, J.: Toward an Assessment of Software
Development Risk. Journal of Management Information Technology, 22 (2): 359-
371, Dec. 1993.
[Basili86] Basili, V. R., Selby, R. W., and Hutchens, D. H.: Experimentation in
Software Engineering. IEEE Transaction on Software Engineering, 12 (7): 733-743,
July 1986.
[Basili01] Basili, V. R. and Boehm, B. W.: COTS-Based Systems Top 10 List. IEEE
Computer, 34 (5): 91-93, May 2001.
[Basili02] Basili V. R., Lindvall M., Rus I., Seaman C., and Boehm B.: Lessons-
Learned Repository for COTS-Based SW Development. Software Technology
Newsletter, 5 (3): 4-7, Sept. 2002, available at: http://fc-md.umd.edu/ll/index.asp
[Baskerville03] Baskerville, R., Ramesh, B., Levine, L., Pries-Heje, J., and Slaughter,
S.: Is Internet-Speed Software Development Different? IEEE Software, 20 (6): 70-
77, Nov./Dec. 2003.
[Bass00] Bass, L., Buhman, C., Comella-Dorda, S., Long, F., Robert, J., Seacord, R.,
and Wallnau, K: Volume I: Market Assessment of Component-based Software
Engineering. SEI Technical Report number CMU/SEI-2001-TN-007, available at:
http://www.sei.cmu.edu/
[Boehm88] Boehm, B. W.: A Spiral Model of Software Development and
Enhancement. IEEE Computer, 21 (5): 61-72, May 1988.
[Boehm89a] Boehm, B. W.: Software Risk Management, Tutorial, IEEE CS Press,
1989.
95
References
[Boehm89b] Boehm, B. W. and Ross, R.: Theory-W Software Project Management:

Principles and Examples. IEEE Transactions on Software Engineering, 15 (7): 902-
916, July 1989.
[Boehm91] Boehm, B. W: Software Risk management: Principles and Practices. IEEE
Software, 8 (1): 32-41, Jan.1991.
[Boehm98] Boehm, B. W., Egyed, A., Kwan, J., Port, D., Shah, A., and Madachy, R. J.,
Using the WinWin Spiral Model: A Case Study. IEEE Computer, 31 (7): 33-44,
July 1998.
[Boehm99] Boehm B. W. and Abts, C.: COTS integration: Plug and Pray? IEEE
Computer, 32 (1): 135-138, Jan. 1999.
[Boehm00] Boehm, B. W.: Requirements that Handle IKIWISI, COTS, and Rapid
Change. IEEE Computer, 33 (7): 99-102, July 2000.
[Boehm03] Boehm, B. W., Port, D., Yang, Y. and Bhuta, J.: Not All CBS Are Created
Equally COTS-intensive Project Types. Proc. of the 2nd International Conference
on COTS-Based Software Systems (ICCBSS 2003), Feb. 2003, Ottawa, Canada,
Springer LNCS Vol. 2580, pp. 36-50.
[Bosch00] Bosch, J.: Design and Use of Software Architectures: Adopting and Evolving
a Product-line Approach. Addison-Wesley, 2000.
[Brooks87] Brooks, P. F. Jr.: No Silver Bullet: Essence and Accidents of Software
Engineering. IEEE Computer, 20 (4): 10-19, Apr. 1987.
[Brownsword00] Brownsword, L., Oberndorf, T., and Sledge, C.: Developing New
Processes for COTS-Based Systems. IEEE Software, 17 (4): 48-55, July/August
2000.
[Bush90] Bush, M. E. and Fenton, N. E.: Software Measurement: A Conceptual
Framework. Journal of System and Software, 12 (3): 223-231, March 1990.
[Carr93] Carr, M., Kondra, S., Monarch, I., Ulrich, F., and Walker, C.: Taxonomy-
Based Risk Identification. SEI report SEI-93-TR-006, Pittsburgh, USA, 1993.
[CBSEnet04] The CBSEnet project, July 5, 2004, available at: http://www.cbsenet.org
[Cechich04] Cechich, A., and Piattini, M.: Quantifying COTS Component Functional
Adaptation. Proc. of the International Conference on Software Reuse, July, 2004,
Madrid, Spain, Springer LNCS Vol. 3107, pp. 195-204.
[Charette89] Charette, R. N.: Software Engineering Risk Analysis and Management.
Intertex Publications McGraw-Hill Book Col, 1989.
[Chittister96] Chittister, C. G., and Haimes, Y. Y.: System Integration via Software
Risk Management. IEEE Transactions on Systems, Man, and Cybernetics, 26 (5):
521-532, Sep. 1996.
[Chung02] Chung, L. and Cooper, K.: A Knowledge-Based COTS-Aware
Requirements Engineering Approach. Proc. of the 14th International Conference on
Software Engineering and Knowledge Engineering, July 2002, Ischia, Italy, pp.
175-182.
[CMM06] http://www.sei.cmu.edu/cmm/
96
References
[Componentsource04] http://www.componentsource.com
[Conradi05a] [P8]Conradi, R., Li, J., Slyngstad, O. P. N., Bunse, C., Torchiano, M.
and Morisio, M.: Reflections on conducting an international CBSE survey in ICT
industry. Proc. of the 4th International Symposium on Empirical Software
Engineering (ISESE 2005), Nov. 2005, Noosa Heads, Australia, IEEE Press, pp.
214-223.
[Conradi05b] [AP3] Reidar Conradi and Jingyue Li: Observations on Versioning of
Off-the-Shelf Components in Industrial Projects (short paper). Proc. of the 12th
International Workshop on Software Configuration Management (co-located with
the ESEC/FSE 2005), Sept. 2005, Lisbon, Portugal, pp. 33-42, ACM Digital
Library.
[Cooper03] Cooper, D. R. and Schindler, P. S.: Business Research Methods. 8th
Edition, McGraw-Hill Press, 2003.
[Corra02] Corra, M. and Willer, D.: The Gatekeeper. Sociological Theory, 20 (2): 180-
207, July 2002.
[Creswell94] Creswell, J.W.: Research Design, Qualitative and Quantitative
Approaches. Sage Publications, 1994.
[Creswell03] Creswell, J.W.: Research Design, Qualitative, Quantitative, and Mixed
Methods Approaches. 2nd edition, Sage Publications, 2003.
[Crnkovic01] Crnkovic, I. and Larsson, M.: A Case Study: Demands on Component-
based Development. Proc. of the 22nd International Conference on Software
Engineering, June 2000, Limerick, Ireland, pp. 21-31.
[Crnkovic02] Crnkovic, I., Larsson, M.: Building Reliable Component-Based System.
Artech House, 2002.
[Denzin94] Denzin, N. K. and Lincoln, Y. S.: Handbook of Qualitative Research, Sage
Publications, London, UK 1994.
[Eisenhardt89] Eisenhardt, K.M.: Building Theories from Case Study Research.
Academy of Management Review, 14 (4): 532-550, 1989.
[Endres03] Endres, A. and Rombach, D.: A Handbook of Software and Systems
Engineering, Empirical Observations, Laws, and Theories. Addison-Wesley
Professional, 2003.
[FAA03] COTS risk factor, available at:
http://www.faa.gov/aua/resources/cots/Guide/CRMG.htm, 2003.
[FAMILIES03] FAMILIES project, available at:
http://www.esi.es/en/Projects/Families, 2003.
[Feller02] Feller, J. and Fitzgerald, B.: Understanding Open Source Software
Development. Addison-Wesley, 2002.
[Fenton97] Fenton, N. and Pfleeger, S.L.: Software Metrics: A Rigorous and Practical
Approach. International Thomson Computer Press, 2nd edition, 1997.
[Finkelstein00] Finkelstein, A. and Kramer, J.: Software Engineering: a Road Map.
Proc. of the 22nd International Conference on Software Engineering, Future of
Software Engineering Track, June 2000, Limerick Ireland, pp. 3-22.
97
References
[Fitzgerald04] Fitzgerald, B.: A Critical Look at Open Source. IEEE Computer, 37 (7):
92-94, July 2004.
[Frakes95] Frakes, W.B. and Fox, C.J.: Sixteen Questions about Software Reuse.
Communications of the ACM, 38 (6): 75-78, June 1995.
[Freesoftware05] www.gnu.org, 2005.
[Fox97] Fox, G., Lantner, K., and Marcom, S.: A Software Development Process for
COTS-based Information System Infrastructure. Proc. of the 5th International
Symposium on Assessment of Software Tools (SAST '97), June, 1997, Pittsburgh,
PA, USA, pp. 133-142.
[Fowler01] Fowler. F. J., Jr.: Survey Research Methods (Applied Social Research
Methods). 3rd edition, Sage Publications, 2001.
[Fuggetta00] Fuggetta, A.: Software Process: A Roadmap. Proc. of the 22nd
International Conference on Software Engineering, Future of Software Engineering
Track, June 2000, Limerick Ireland, pp. 25-34.
[Gemmer97] Gemmer, A.: Risk Management: Moving Beyond Process. IEEE
Computer, 30 (5): 33-41, May, 1997.
[Ghosh02] Ghosh, S.: Improving Current Component-Based Development Techniques
for Successful Component-Based Software Development. Proc. of International
Conference on Software Reuse (ICSR7), Workshop on Component-based Software
Development Processes, 2002, available at: http://www.idt.mdh.se/CBprocesses/
[Giacomo05] Giacomo, P. D.: COTS and Open Source Software Components: Are
They Really Different on the Battlefield? Proc. of the 4th International Conference
on COTS-Based Software Systems, Feb. 2005, Bilbo, Spain, Springer, LNCS Vol.
3412, pp. 301-310.
[Gifi90] Gifi, A.: Nonlinear Multivariate Analysis, John Wiley & Sons, 1990.
[Glaser67] Glaser, B.G. and Strauss, A. L.: The Discovery of Grounded Theory:
Strategies for Qualitative Research. Aldine Publishing, 1967.
[Glass01] Glass, R. L.: Frequently Forgotten Fundamental Facts about Software
Engineering, IEEE Software, 18 (3): 110-112, May/June, 2001.
[Griss93] Griss, M.L.: Software Reuse: From Library to Factory. IBM Systems Journal,
32 (4): 548-566, Nov./Dec. 1993.
[Griss95] Griss, M.L. and Wosser, M.: Making Reuse Work in Hewlett-Packard. IEEE
Software, 12 (1): 105-107, Jan. 1995.
[Hall98] Hall, E.: Managing Risk: Methods for Software Systems Development.
Readings, MA: Addision-Wesley, 1998.
[Hecht04] Hecht, H.: Systems Reliability and Failure Prevention. Artech House
Publishers, 2003.
[Heineman01] Heineman, G.T. and Councill, W.T.: Component-Based Software
Engineering, Putting Pieces Together. Addison-Wesley, 2001.
98
References
[Highsmith99]: Highsmith III, J. A.: Adaptive Software Development: A Collaborative

Approach to Managing Complex Systems. Dorset House Publishing, New York,
USA, 1999.
[Hissman01] Hissam, S. Weinstock, C. Plakosh, D., and Asundi, J: Perspectives on
Open Source Software. Technical Report, SEI number, CMU/SEI-2001-TR-019,
2001.
[Hissam02] Hissam, S. A. Plakosh, D., and Weinstock, C.: Trust and Vulnerability in
Open Source Software. IEE Proceedings - Software, 49 (1): 47-51, Feb. 2002.
[INCO00] INCO project, available at: http://www.ifi.uio.no/~isu/INCO/, 2000,
[Infoweek04] InformationWeek Survey. Available at:
http://www.informationweek.com/story/showArticle.jhtml?articleID=51201599&ti
d=5979, 2004
[ISO06] http://www.iso.org/iso/en/ISOOnline.frontpage, 2006
[Jacobson97] Jacobson, I., Griss, M., and Jonsson, P.: Software Reuse: Architecture,
Process and Organization for Business Success. Addison-Wesley, 1997.
[Jacobson99]Jacobson, I., Booch, G., Rumbaugh, J.: The Unified Software
Development Process, 1st edition, Addison-Wesley, 1999.
[Jones84] Jones, T. C.: Reusability in Programming: A Survey of the State of the Art.
IEEE Transactions on Software Engineering, 10 (5): 488-494, Sep. 1984.
[Kansala97] Kansala, K.: Integrating Risk Assessment with Cost Estimation. IEEE
Software, 14 (3): 69-74, May/June 1997.
[Karlsson95] Karlsson, E.A. (Ed.): Software Reuse, a Holistic Approach. John Wiley &
Sons, 1995.
[Keil98] Keil, M., Cule, P.E., Lyytinen, K., and Schmidt, R.C.: A Framework for
Identifying Software Project Risks. Communications of the ACM, 4 (11): 76-83,
Nov. 1998.
[Kent99] Kent, B.: Extreme Programming Explained: Embrace Change. Addison-
Wesley, 1999.
[Kitchenham02] Kitchenham, B.A., Pfleeger, S.L., Hoaglin, D.C., and Rosenberg, J.:
Preliminary Guidelines for Empirical Research in Software Engineering. IEEE
Transactions on Software Engineering, 28 (8): 721-734, Aug. 2002.
[Kontio96] Kontio, J.: A Case Study in Applying a Systematic Method for COTS
Selection. Proc. of the 18th International Conference on Software Engineering, Mar.
1996, Berlin, Germany, pp. 201 – 209.
[Kotonya01] Kotonya, G. and Rashid, A.: A Strategy for Managing Risk in
Component-based Software Development. Proc. of the 27th EUROMICRO
Conference, Sept. 2001, Warsaw, Poland, pp. 12-21.
[Kruchten00] Kruchten, P.: The Rational Unified Process. An Introduction. Addison-
Wesley, 2000.
[Kruchten01] Kruchten, P.: The Nature of Software: What’s So Special about Software
Engineering? The Rational Edge, Oct. 2001, http://www.therationaledge.com/.
99
References
[Lauesen06] Lauesen, S.: COTS Tenders and Integration Requirements. Journal of

Requirements Engineering, 11 (2): 111-122, 2006.
[Lawlis01] Lawlis, P. K., Mark, K. E., Thomas, D.A., and Courtheyn, T.: A Formal
Process for Evaluating COTS Software Products. IEEE Computer, 34 (5):58-63,
May 2001.
[Lawton02] Lawton, G.: Open Source Security: Opportunity or Oxymoron? IEEE
Computer, 35 (3): 18-21, March 2002.
[Leung02] Leung, K. R. P. H. and Leung, H. K. N.: On the Efficiency of Domain-
Based COTS Product Selection Method. Journal of Information and Software
Technology, 44 (12): 703-715, Sept. 2002.
[Li04a] [P1] Li, J., Conradi, R., Mohagheghi, P., Sæhle, O. A., Wang, Ø., Naalsund, E.,
and Walseth, O. A.: A Study of Developer Attitude to Component Reuse in Three
IT Companies. Proc. of the 5th International Conference on Product Focused
Software Process Improvement (PROFES'2004), Apr. 2004, Nara, Japan, Springer
LNCS Vol. 3009, pp. 538-552.
[Li04b] [AP2] Li, J., Houmb, S. V., Kvale, A. A.: A Process to Combine AOM and
AOP: A Proposal Based on a Case Study. Proc. of the 5th Aspect-Oriented
Workshop (In conjunction with UML 2004), Oct. 11, 2004, Lisbon, Portugal.
Available at: http://www.cs.iit.edu/~oaldawud/AOM/AcceptedPapers/
[Li04c] [AP6] Li, J., Bjørnson, F. O., Conradi, R. and Kampenes, V. B.: An Empirical
Study of COTS Component Selection Processes in Norwegian IT companies. Proc.
of the International Workshop on Models and Processes for the Evaluation of COTS
Components (MPEC'04), May, 2004, Edinburgh, Scotland, IEE Press, pp. 27-30.
[Li04d] [AP7] Li, J., Bjørnson, F. O., and Conradi, R.: Empirical Study on COTS
Component Classification, Proc. Intl. Workshop on COTS Terminology and
Concepts, Co-located with International Conference on Component-Based Software
Systems (ICCBSS'04), 1-4 Feb., 2004, Redondo Beach, USA,
http://softeng.polito.it/iwctc/accepted.html - with list of accepted papers,
[Li05a] [P3] Li, J. Conradi, R., Slyngstad, O. P. N., Bunse, C., Khan, U., Torchiano,
M., and Morisio, M.: Validation of New Theses on Off-The-Shelf Component
Based Development. Proc. of the 11th IEEE International Metrics Symposium
(Metrics'05), Sep. 2005, Como, Italy, IEEE Press, pp.26, 10p.
[Li05b] [P7] Li, J. Conradi, R., Slyngstad, O. P. N., Bunse, C., Khan, U., Torchiano,
M., and Morisio, M.: Barriers to Disseminating Off-The-Shelf Based Development.
ACM Software Engineering Notes (elec. version), Vol. 30, No. 4, July 2005.
[Li05c] [AP4] Li, J., Conradi, R., Slyngstad, O. P. N., Torchiano, M., Morisio, M., and
Bunse, C.: Preliminary Results from a State-of-the-Practice Survey on Risk
Management in Off-The-Shelf Component-Based Development. Proc. of the 4th
International Conference on COTS-based Software System (ICCBSS 2005), Feb.
2005, Bilbao, Spain, Springer LNCS Vol. 3412, pp. 278-288.
[Li05d] [AP5] Li, J., Conradi, R., Slyngstad, O. P. N., Torchiano, M., Morisio, M., and
Bunse, C.: An Empirical Study on Off-the-Shelf Component Usage in Industrial
Projects. Proc. of the 6th International Conference on Product Focused Software
100
References
Process Improvement, Oulu, Finland (PROFES 2005), June 2005, Springer

LNCS Vol. 3547, pp. 54-68.
[Li06a] [P2] Li, J., Bjørnson, F. O., Conradi, R., and Kampenes, V. B.: An Empirical
Study of Variations in COTS-based Software Development Processes in Norwegian
IT Industry. To appear in the Journal of Empirical Software Engineering 2006
(Special issue from Metrics 2004).
[Li06b] [P4] Li, J., Torchiano, M., Conradi, R., Slyngstad, O. P. N., and Bunse, C.: A
State-of-the-Practice Survey of the Off-the-Shelf Component-Based Development
Processes, Proc. of the 9th International Conference on Software Reuse, Torino,
Italy, June, 2006, Springer LNCS Vol. 4039, pp. 16-28.
[Li06c] [P5] Li, J., Conradi, R., Slyngstad, O. P. N., Bunse, C., Torchiano, M., and
Morisio, M.: An Empirical Study on the Decision Making Process in Off-The-Shelf
Component Based Development. Proc. of the International Conference on Software
Engineering (ICSE 2006), Emerging Result Track, 20-28 May 2006, Shanghai, P.R.
China, IEEE Press, pp. 897-900.
[Li06d] [P6] Li, J., Conradi, R., Slyngstad, O. P. N., Bunse, C., Torchiano, M., and
Morisio, M.: An Exploratory Study on Risk Management in OTS Component-
Based Development. Submitted to IEEE Transactions on Software Engineering.
[Li06e] [AP1] Li, J., Kvale, A. A., and Conradi, R.: A Case Study on Improving the
Changeability of COTS-Based System Using Aspect-Oriented Programming.
Journal of Information Science and Engineering, 22 (2): 370-390, March 2006.
[Lincoln85] Lincoln, Y.S. and Guba, E.G.: Naturalistic Inquiry. Sage, 1985.
[Maccrimmon97] MacCrimmon, K. R.: An Overview of Multiple Objective Decision
Making. In Cochrane, J. L. and Zeleny, M. editors: Multiple Criteria Decision
Making, University of South Carolina Press, 1973, pp.18-44.
[Madachy97] Madachy, R. J.: Heuristic Risk Assessment Using Cost Factors. IEEE
[Madanmohan04] Madanmohan, T.R. and De’, R.: Open Source Reuse in Commercial
Firms. IEEE Software, 21 (1): 62-69, January/February, 2004.
[Maiden98] Maiden, N. A. M, and Ncube, C.: Acquiring COTS Software Selection
Requirement. IEEE Software, 15 (2): 46-56, Mar./Apr. 1998.
[McIlroy68] McIlroy, D.: Mass-produced Software Components. Proc. of the Software
Engineering Concepts and Techniques Conference, 1968 Report on NATO
Conference on Software Engineering, 1968, pp. 138-155.
[Michailidis98] Michailidis, G. and Leeuw, J. d.: The Gifi System of Descriptive
Multivariate Analysis. Journal of Statistical Science, 13 (4): 307-336, 1998.
[Mili95] Mili, H., Mili, F., and Mili, A.: Reuse Software: Issues and Research
Directions. IEEE Transactions on Software Engineering, 21 (6): 528-562, June
1995.
[Mitre05] Common Risks and Risk Mitigation Actions for a COTS-based System,
www.mitre.org/work/sepo/toolkits/risk/taxonomies/files/CommonRisksCOTS.doc,
2005.
101
References
[Morisio97] Morisio, M. and Tsoukias, A.: IusWare: a Methodology for the Evaluation
and Selection of Software Products. IEE Proceedings – Software Engineering, 144
(3): 162-174, Mar. 1997.
[Morisio00] Morisio, M., Seaman, S., Parra, A., Basili, V., Kraft, S., and Condon, S.:
Investigating and Improving a COTS-Based Software Development Process, Proc.
of the 22nd International Conference on Software Engineering, June 200, Limerick,
Ireland, pp. 31-40.
[Morisio02a] Morisio, M. and Torchiano, M.: Definition and Classification of COTS: a
Proposal. Proc. of the 1st International Conference on COTS-Based Software
Systems (ICCBSS’02), Orlando, FL, USA, Springer, LNCS VOL. 2255, pp. 165-
175.
[Morisio02b] Morisio, M., Ezran, M., and Tully, C.: Success and Failures in Software
Reuse. IEEE Transactions on Software Engineering, 28 (4): 340-357, April 2002.
[Moynihan97] Moynihan, T.: How Experienced Project Managers Assess Risk. IEEE
[NAICS04] NAICS Code, Available at:
http://www.revenue.state.ne.us/tax/current/buscodes.pdf, 2004
[Ncube02] Ncube, C. and Dean, J. C.: The Limitation of Current Decision-Making
Techniques in the Procurement of COTS Software Components. Proc. of the 1st
International Conference on COTS-Based Software Systems (ICCBSS’02), Orlando,
FL, USA, Springer, LNCS VOL. 2255, pp. 176-187.
[Netcraft05] Netcraft’s survey available at:
http://news.netcraft.com/archives/2005/01/01/january_2005_web_server_survey.ht
m
[Oberndorf97] Oberndorf, T.: COTS and Open Systems - An Overview. 1997,
available at: http://www.sei.cmu.edu/str/descriptions/cots.html#ndi
[Ochs01] Ochs, M., Pfahl, D., Diening, G. C., and Kolb, B. N.: A Method for Efficient
Measurement-based COTS Assessment and Selection - Method Description and
Evaluation Results. Proc. of the 7th IEEE International Software Metrics
Symposium, Apr. 2001, London, England, IEEE Press, pp. 285-297.
[OSS04] Open Source Initiative, 2004, available at:
http://www.opensource.org/index.php
[Ould99] Ould, M. A.: Managing Software Quality and Business Risk. John Wiley &
Sons, 1999.
[Paulson04] Paulson, J.W., Succi, G. and Eberlein, A.: An Empirical Study of Open-
Source and Closed-Source Software Products. IEEE Transactions on Software
Engineering, Vol. 30, No. 4, Apr. 2004, pp. 246-256.
[Parastoo04] Mohagheghi, P.: The Impact of Software Reuse and Incremental
Development on the Quality of Large Systems. PhD thesis, ISBN: 82-471-6408-6.
[Pfleeger94] Pfleeger, S.: Experimental Design and Analysis in Software Engineering
Part 1-5. ACM SIGSOFT Software Engineering Notes, 19 (4):16-20; 20 (1):22-26;
20 (2):14-16; 20 (3):13-15; and 20 (4):14-17, 1994-1995.
102
References
[Raymond99] Raymond, E. S. The Cathedral and the Bazaar: Musings on Linux and
Open Source by an Accidental Revolutionary. O’Reilly, 1999.
[Robson02] Robson, C.: Real World Research: A Resource for Social Scientists and
Practitioner-researchers (Regional Surveys of the World), 2nd edition, Blackwell
Publishers, 2002.
[Ropponen00] Ropponen, J. and Lyytinen, K.: Components of Software Development
Risk: How to Address Them? A Project Manager Survey. IEEE Transactions of
Software Engineering, 26 (2): 98-112, Feb. 2000.
[Rose03] Rose, L. C.: Risk Management of COTS Based System Development.
Component-based Software Quality, Springer LNCS 2693, 2003, pp. 352-373.
[Rothenberger03] Rothenberger, M. A., Dooley, K. J. and Kulkarni, U. R.: Strategies
for Software Reuse: A Principal Component Analysis of Reuse Practices. IEEE
Transactions on Software Engineering, 29 (9): 825-837, Sept. 2003.
[Royce87] Royce W. W.: Managing the Development of Large Software Systems:
Concepts and Techniques. Proc. of the 9th International Conference on Software
Engineering, Mar. 1987, Monterey, California, USA, pp. 328-338.
[Ruffin04] Ruffin, M. and Ebert, C.: Using Open Source Software in Product
Development: A Primer. IEEE Software, 21 (1): 82-86, January/February 2004.
[Saaty90] Saaty, T. L.: How to make a decision: The Analytic Hierarchy Process
(AHP). European Journal of Operational Research, 48 (1): 9-26, Jan. 1990.
[Sai03] Sai, V.: COTS Acquisition Evaluation Process: The Preacher’s Practice. Proc.
of the 2nd International Conference on COTS-Based Software Systems (ICCBSS
2003), Feb. 2003, Ottawa, Canada, Springer LNCS Vol. 2580, pp. 196-206.
[Seaman99] Seaman, C.B.: Qualitative Methods in Empirical Studies of Software
Engineering. IEEE Transactions on Software Engineering, 25 (4): 557-572, Apr.
1999.
[Sese04] SESE tool, available at: http://sese.simula.no, 2004.
[Sherer95] Sherer, S. A.: The Three dimensions of Software Risk: Technical,
Organizational, and Environmental. Proc. of the 28th Hawaii International
Conference on System Sciences, Maui, Hawaii, January, Jan. 1995, pp. 369-378.
[Sommerville04] Sommerville, I.: Software Engineering. 7th edition, Addison-Wesley,
2004.
[Spc01] Software Productivity Consortium: Phased-Integrated COTS Approach.
Technical Report SPC-2001006-CMC, available at:
http://www.software.org/pub/pic, 2001.
[SPIKE02] SPIKE project, available at:
http://www.idi.ntnu.no/grupper/su/spike.html, 2002.
[SPSS01] SPSS Categories 11.0, available at www.spss-sa.com, 2001.
[SSB02] Norwegian Census Bureau (SSB), Oslo, ICT company data 2002,
http://www.ssb.no/emner/10 /03/ikt/.
[Stanish95] The Stanish Group: http://www.stanishgroup.com/, 1995.
103
References
[Stewart05] Stewart, K. J., Ammeter, A. P., and Maruping, L. M.: Preliminary Analysis
of the Influences of Licensing and Organizational Sponsorship on Success in Open
Source Projects. Proc. of the 38th Hawaii International Conference on System
Sciences, Jan. 2005, Hawaii, USA, pp. 197c.
[Strauss98] Strauss, A. and Corbin, J. M.: Basics of Qualitative Research: Grounded
Theory Procedures and Techniques. 2nd edition, Sage Publications, 1998.
[Szyperski02] Szyperski, C., Gruntz, D., and Murer, S.: Component Software, Beyond
Object-Oriented Programming. Addison Wesley, 2nd edition, 2002.
[Taylor84] Taylor, S. J. and Bogdan, R.: Introduction to Qualitative Research Methods.
John Wiley&Sons, 1984.
[Torchiano04] Torchiano, M. and Morisio, M.: Overlooked Facts on COTS-Based
Development. IEEE Software, 21 (2): 88-93, Mar./Apr. 2004.
[Tran97a] Tran, V., Liu, D. B., and Hummel, B.: Component-Based Systems
Development: Challenges and Lessons Learned. Proc. of the 8th IEEE International
Workshop on Software Technology and Engineering Practice, Jul. 1997, London,
UK, pp. 452-462.
[Tran97b] Tran, V. Liu, D. B.: A Risk-Mitigating Model For the Development of
Reliable and Maintainable Large-Scale Commercial-Off-the-Shelf Integrated
Software Systems, Proc. of the Annual Reliability and Maintainability Symposium,
Philadelphia, PA, USA, Jan. 1997, pp. 361-387.
[Tran99] Tran, V.N. and Liu, D. B.: Application of CBSE to Projects with Evolving
Requirements- A Lesson-learned. Proc. of the 6th Asia-Pacific Software
Engineering Conference (APSEC’ 99), Dec. 1999, Takamatsu, Japan, pp. 28-37.
[Verner05] Verner, J. M. and Evanco, W. M.: In-House Software Development: What
Project Management Practices Lead to Success? IEEE Software, 22 (1): 86-93,
Jan./Feb. 2005.
[Vigder96] Vigder, M. Gentleman, M. and Dean, J.: COTS Software Integration: State
of the Art. Technical Report NRC No. 39190, 1996.
[Vigder97] Vidger, M. and Dean, J.: An Architectural Approach to Building Systems
from COTS Software Components. Proc. of the 1997 Center for Advanced Studies
Conference (CASCON 97), Toronto, Ontario, Nov. 1997, available at:
http://seg.iit.nrc.ca/English/abstracts/NRC40221abs.html
[Vitharana03] Vitharana, P.: Risks and Challenges of Component-Based Software
Development. Communications of the ACM, 46 (8): 67-72, Aug. 2003.
[V-Model05] German V-model: http://www.v-modell.iabg.de/#AU250, 2005.
[Voas98a] Voas, J.M.: Certifying Off-the-Shelf Software Components. IEEE
Computer, 31 (6): 53-59, June 1998, pp.
[Voas98b] Voas, J.M.: The Challenges of Using COTS Software in Component-Based
Development. IEEE Computer, 31 (6): 44-45, June 1998.
[Voas98c] Voas, J.: COTS Software – the Economical Choice? IEEE Software, 15 (2):
16-19, March/April 1998.
104
References
[Wallace04] Wallace, L., Keil, M.: Software Project Risks and Their Effect on
Outcomes, Communications of the ACM, 47 (4): 68-73, April 2004.
[Wohlin00] Wohlin, C., Runeseon, P., M. Höst, Ohlsson, M.C., Regnell, B., and
Wesslén, A.: Experimentation in Software Engineering. Kluwer Academic
Publications, 2000.
[Yin03] Yin, R.K.: Case Study Research, Design and Methods. Sage Publications,
2003.
[Zahran98] Zahran, S.: Software Process Improvement-Practical Guidelines for
Business Success. Addison-Wesley, 1998.
[Zelkowitz98] Zelkowitz, M. V. and Wallace, D. R: Experimental Models for
Validating Technology. IEEE Computer, 31 (5): 23-31, May 1998.
105
References
106
Appendix A
Appendix A: Selected papers
This chapter contains the papers in the order given in Section 1.5.
P1. Developer Attitude to Component Reuse

A Study of Developer Attitude to Component Reuse in Three
IT Companies
Jingyue Li1, Reidar Conradi1,3, Parastoo Mohagheghi1,2,3, Odd Are Sæhle1, Øivind
Wang1, Erlend Naalsund1, and Ole Anders Walseth1
1
Department of Computer and Information Science,
Norwegian University of Science and Technology (NTNU),
NO-7491 Trondheim, Norway
{jingyue, conradi}@idi.ntnu.no
2
Ericsson Norway-Grimstad, Postuttak, NO-4898 Grimstad, Norway
{parastoo.mohagheghi}@ericsson.com
3
Simula Research Laboratory, P.O.BOX 134, NO-1325 Lysker, Norway
Abstract. The paper describes an empirical study to investigate the state of practice and
challenges concerning some key factors in reusing of in-house built components. It also
studies the relationship between the companies’ reuse level and these factors. We have
collected research questions and hypotheses from a literature review and designed a
questionnaire. 26 developers from three Norwegian companies filled in the
questionnaire based on their experience and attitudes to component reuse and
component-based development. Most component-based software engineering articles
deal with COTS components, while components in our study are in-house built. The
results show that challenges are the same in component related requirements
(re)negotiation, component documentation and quality attributes specification. The
results also show that informal communications between developers are very helpful to
supplement the limitation of component documentation, and therefore should be given
more attention. The results confirm that component repositories are not a key factor to
successful component reuse.
107
Appendix A
1. Introduction
Systematic reuse is generally recognized as a key technology for improving software

productivity and quality [17]. With the maturity of component technologies, more and
more companies have reused their software (program) in the form of components.
Component reuse consists of two separate but related processes. The first deals with
analysis of the application domain and development of domain-related components, i.e.
development-for-reuse. The second process is concerned with assembling software
system from prefabricated components, i.e. development-with-reuse. These two
processes are tightly related, especially in reusing in-house built components. The
number of components and the ratio of reused components to total components will
determine the reuse benefits (e.g. improved productivity and quality) [11] [23].
To investigate the current state of practice and challenges for development-with-
reuse in the IT industry, and to investigate the relationship between companies’ reuse
level and some key factors in reusing of in-house components, an empirical study was
performed as part of two Norwegian R&D projects. These projects were SPIKE
(Software Process Improvement based on Knowledge and Experience) [29] and INCO
(INcremental and COmponent-based development) [30]. From the literature review, we
defined several research questions and hypotheses. A questionnaire was designed to
investigate these questions. Developers from three Norwegian IT companies filled in the
questionnaire based on their experience and attitudes to component reuse.
From the results of the survey, we found some new challenges in component reuse
and component-based development based on in-house built components. The results
support some commonly held beliefs and contradict others.
As the sample size of current research is still small, this study cannot provide
statistically significant tests on hypotheses, and is therefore a pre-study. Later studies
will be undertaken with refined hypotheses and on a larger sample.
The reminder of the paper is structured as follows: Section 2 presents some general
concepts. Section 3 describes the research approach. Section 4 presents the survey
results. Section 5 gives a detail discussion on the survey results. Conclusion and future
research are presented in section 6.
2 Component reuse and component-based development
Software reuse can take many different forms, from ad-hoc to systematic [16]. In the
broad definition of reuse, it includes reusing everything associated with software
projects, such as procedures, knowledge, documentation, architecture, design and code.
In our research, we focus on systematic reuse of software code. The code reuse
literature has identified reuse practice and success factors through several case studies
and surveys. A major reuse effort is the REBOOT (Reuse Based on Object-Oriented
Techniques) consortium [25]. This effort was one of the early reuse programs that
recognized the importance of not only the technical, but also the organizational aspects
of reuse [18]. As more experience become available from industrial studies, non-
technical factors, such as organization, processes, business drivers and human
involvement, appeared to be at least as important as technological issues [15][19].
Following the success of the structured design and OO paradigms, component -based
software development has emerged as the next revolution in software development [27].
108
Appendix A
More and more IT companies have started to reuse code by encapsulating it into
components. Whitehead defines a component as: A software component is a separable
piece of executable software, which makes sense as a unit, and can interoperate with
other components, within some supporting environment. The component is accessible
only via its interface and is capable of use ‘as-is’, after any necessary installation and
configuration procedures have been carried out [28].
Component-based development is assumed to have many advantages. These include
more effective management of complexity, reduced time to market, increased
productivity, improved quality, a greater degree of consistency and a wider range of
usability [4][13]. It also brings many challenges, because it involves various
stakeholders and roles, such as component developers, application developers, and
customers. Different stakeholders and roles have different concerns [3], and face
different issues and risks [2] [27].
Component-based development differs from traditional development, where the
usual approach is for stakeholders to agree upon a set of requirements and then build a
system that satisfies these requirements from scratch. Component-based development
builds application by reusing existing components. Available components may not be
able to satisfy all the requirements. Therefore, component-based projects must have
flexibility in requirements, and must be ready to (re)negotiate the requirements with the
customer. Moreover, components are intended to be used ‘as-is’. If some additional
functionality is required, ‘glue-code’ is needed to be built to meet the differences
between the requirement and component functionality. Another important feature of
component-based development is the strong focus on the quality attributes (such as
reliability, performance, and security etc.) and related testing. A major effort has to be
put into checking how components perform, how well they interact, and to make sure
that they are indeed compatible. Components may be developed in-house, acquired as
COTS (commercial-off-the-shelf) [3], or even as OSS (Open Source Software) [5].
Most current research on component-based software engineering focuses on COTS-
based development. Because COTS users cannot access the source code and must rely
on vendors to give technical support, COTS-based development is assumed to be more
challenging. Therefore, there is little research on the challenges based on in-house built
components.
3 Research approach
The difference between development based on in-house built components and

development based on COTS is that the former is related very tightly with development-
for-reuse. Component reuse is generally an incremental procedure. The company will
build some reusable components in the beginning. In case of successful reuse, more and
more code will be encapsulated into reusable components. The more reusable
components are developed, the more complex will the development process be, and
more support is required from the organization [8]. Our motivation is to investigate the
relationship between companies’ reuse level and some key factors in component-based
development so that company with low reuse level can make necessary software process
improvements when moving to a higher reuse level.
109
Appendix A
3.1 Research questions
To reuse in-house components successfully, developers must follow three basic steps
[19]:
- Formulate the requirements in a way that supports retrieval of potentially useful
reusable components.
- Understand the retrieved components.
- If the retrieved components are sufficiently ‘close’ to the needs at hand and are of
sufficient quality, then adapt them.
From these steps, we selected several key factors. For step 1, we focus on the
efficiency of component related requirements (re)negotiation and the value of
component repository. For step 2, we study how knowledge about components can be
transferred from a component provider to a component user. For step 3, our study
focuses on definition and reasoning of quality attributes of components.
There is little research on the need for requirements (re)negotiation when
components are built in-house. People assume that owning source code of in-house built
components allows them to do any changes to meet the customers’ requirements.
However, components are intended to be used ‘as-is’, even it is built in-house. So, our
first research question is:
RQ1. Does requirements (re)negotiation for in-house components really work as
efficiently as people assume?
Crnkovic et al. have proposed that to successfully perform the component-based

requirements (re)negotiation, a vast number of possible component candidates must be
available, as well as tools for finding them [9]. Companies with a higher reuse level
usually have more component candidates, more experience, and better experience than
companies with a lower reuse level. So, our second research question is:
RQ2. Does the efficiency of component related requirements (re)negotiation
increase with more in-house built components available?
To investigate this question, we formalized a null hypothesis H01 and an alternative
hypothesis HA1 as follows:
H01. There is no relationship between the companies’ reuse level and the efficiency
of component related requirements (re)negotiation.
HA1. There is a positive relationship between the companies’ reuse level and the
efficiency of component related requirements (re)negotiation.
Concerning a component repository, Frakes claimed that it should not be given much
attention, at least initially [12]. So, our third research question is:
RQ3. Does the value of component repository increase with more reusable
To investigate this opinion more deeply, a null hypothesis H02 and an alternative
hypothesis HA2 was proposed:
H02. There is no relationship between the companies’ reuse level and the value of
component repository.
HA2. There is a positive relationship between the companies’ reuse level and the
value of component repository.
110
Appendix A
A complete specification of a component should include its functional interface,

quality characteristics, use cases, tests, etc. While current component-based
technologies successfully manage functional interfaces, there is no satisfactory support
for expressing quality parts of a component [9]. So, our fourth research question is:
RQ4. How can a component user acquire sufficient information about relevant
components?
Berglund claimed that growing reusable software components will create a new
problem, i.e. the information-overload problem. Therefore, learning which component
to use and how to use them become the central part of software development [1].
Companies with a higher reuse level usually have more reusable components than
companies with lower reuse level. So, our fifth research question is:
RQ5. Does the difficulty of component documentation and component knowledge
management increase with increasing reuse level?
To study this question, we formalize null hypothesis H03 and alternative hypothesis
HA3:
H03. There is no relationship between the companies’ reuse level and developers’
satisfaction with component documentation.
HA3. There is a negative relationship between the companies’ reuse level and
developer’ satisfaction with component documentation.
One key issue in component-based development is trust, i.e. we want to build

trustworthy systems out of parts for which we have only partial knowledge [7]. Current
component technologies allow systems builders to plug components together, but
contribute little to ensure how well they will play together or to fulfill certain quality
properties. So, the sixth research question is:
RQ6. Do developers trust the quality specification of their in-house built
3.2 The questionnaire
The questionnaire included five parts. The questions in the first part were used to
investigate the reuse level of the companies. The definition of reuse level in this study is
the number of reused components vs. the number of total components in the
organization. The other four parts were organized based on the four key factors. Each
question in the questionnaire was used to study one or more research questions. The
details of questions are shown in the following Table 1. The correspondences between
the questions in the questionnaire and research questions are shown in Table 2. To
increase the reliability of our survey, the questionnaire also included the definition of
concepts used in the questionnaire, and the questions about the respondents’ personal
information.
3.3 Data collection
The study was performed in three Norwegian IT companies. Data collection was carried
out by NTNU PhD and MSc students. Mohagheghi, Naalsund, and Walseth performed
the first survey in Ericsson in 2002. In 2003, Li, Sæhle and Wang performed the survey
111
Appendix A
reusing the core parts of the questionnaire in two other companies (i.e. EDB Business
Consulting and Mogul Technology). We selected those three companies because they
have experience on component reuse and would like to cooperate with NTNU in this
research. The respondents are developers in these three companies. They answered the
questionnaires separately. The questionnaires were filled in either by hand or
electronically (as a Word file). The MSc students provided support with possible
problems in answering the questionnaire.
Table 1. Questions in the questionnaire
Reuse level
Q1. What is the reuse level in your organization?
Q2. To what extend do you feel affected by reuse in your work?
Component related requirements (re)negotiation
Q3. Are requirements often changed/ (re)negotiated in typical develop projects?
Q4. Are requirements usually flexible in typical projects?
Q5. Do the component related requirements (re)negotiation processes work
efficiently in typical projects?
Value of component repository
Q6. Would the construction of a reuse repository be worthwhile?
Component understanding
Q7. Do you know the architecture of the components well?
Q8. Do you know the interface of the components well?
Q9. Do you know the design rules of the components well?
Q10a. Is the existing design/code of reusable components sufficiently
documented?
Q10b. If the answer of Q10a is ‘sometimes’ or ‘no’, is this a problem?
Q10c. If the answer of Q10a is ‘sometimes’ or ‘no’, what are the problems with
the documentation?
Q10d. If the answer of Q10a is ‘sometimes’ or ‘no’, how would you prefer the
documentation?
Q10e. What is your main source of information about reusable components
during implementation?
Q10f. How do you decide whether to reuse a component ‘as-is’, ‘reuse with
modification’ or ‘make a new one from scratch’?
Quality attributes specification of components
Q11. Are specifications for components’ quality attributes well defined?
Q12. Do you test components after modification for their quality attributes
before integrating them with other components?
Table 2. Correspondence between Questions in the questionnaire and Research

Questions
RQ1 RQ2 RQ3 RQ4 RQ5 RQ6
Q1-Q2 X X X
Q3-Q5 X X
Q6 X
Q7-Q10f X X
Q11-Q12 X
112
Appendix A
Below, we briefly characterize these three companies and respondents.
3.3.1 Companies
Ericsson Norway-Grimstad started a development project five years ago and has
successfully developed two large-scale telecommunication systems based on the same
architecture and many reusable components in cooperation with other Ericsson
organization. Their two main applications share more than 60% of ca. 1M lines of code
[22].
EDB Business Consulting in Trondheim (now Fundator) is an IT-consultant firm
which helps its customers to utilize new technology. It started to build reusable
components from 2001. They have built some reusable components based on the
Microsoft .Net in their eCportal framework (i.e. a web-application framework) 1.0 &
2.0. These components have been successfully reused in their new e-commence
applications.
Mogul Technology (now Kantega) in Trondheim has large customers in the
Norwegian finance- and bank sector. The main responsibilities are development and
maintenance of the customers’ Internet bank application. The application was originally
a monolithic system. After several years in production, the customer itself took initiative
to reengineer the old system to a component-based solution based on EJB component
model in 2002. At the time of the survey, some components have been created and
reused in their new Internet bank system.
3.3.2 Respondents
There were 200 developers at Ericsson in Grimstad, where we sent out 10

questionnaires to developers in one development team and got 9 filled-in questionnaires
back. There were 20 developers in EDB Business Consulting in Trondheim, and we
gathered 10 filled-in questionnaires back out of 10. We distributed 10 questionnaires to
22 developers at Mogul Technology in Trondheim and got 7 back. Those developers
were selected because their work was related to component reuse, and they could assign
effort to participate in the survey. This is non-probability sampling, which is based on
convenience. Most participants in this survey have a solid IT background. 6 of 26
respondents have MSc degree in computer science and all others have a bachelor degree
in computer science or telecommunication. More that 80% of them have more than 5
years of programming experience. The details of their position and their experience in
the current organization are summarized in the following Table 3.
4 Survey Results
In this section, we summarize the result of the survey. All the following statistical
analyses are based on valid answers, i.e. Don’t Know answers are excluded. The
statistical analysis tool we used is SPSS Version 11.0.
4.1 Different reuse level in these companies
113
Appendix A
First, we wanted to know the reuse level in those three companies. Q1 and Q2 were
asked to get the answer based on developers’ subjective opinion on this issue. The result
of Q1 is shown in Fig. 1, and the result of Q2 is shown in Fig. 2. From Fig. 1 and Fig. 2,
we can see that most developers in Ericsson think that the reuse level in their company
is very high or high. Most developers in EDB regard the reuse level in their company is
high or medium. Most developers in Mogul think that the reuse level in their company
is medium or little.
Table 3. Background of the respondents

Company Position and working experience in the
organization
Ericsson Norway-Grimstad 2 system architects, 7 designers.
1 person has 13 years of experience
7 persons have experience from 2-5 years,
1 person has 9 months of experience,
EDB Business Consulting 1 project manager, 5 developers and 4 IT
in Trondheim consultants.
1 person has 17 years of experience
8 persons have experience from 3-8 years,
1 person has 2 years of experience.
Mogul Technology in 6 developers and 1 maintainer (previous developer).
Trondheim 1 person has 10 years of experience,
6 persons have experience from 2-5 years.
4.2 Component related requirements (re)negotiation
Questions Q3-Q5 were asked to investigate RQ1.We can see that no respondents to Q3
believe that the requirements were never changed/ (re)negotiated. Only 8% of
respondents to Q4 think the requirements of their typical project are not flexible.
However, only 48% of respondents to Q5 think component related requirements
(re)negotiation works well. To study RQ2 and test the hypothesis H01, the correlation
between the reuse level and response to Q5 is studied. We assign ordinal values to
Ericsson, EDB and Mogul to represent their different reuse levels based on the
responses to Q1 and Q2 (Ericsson = 3, EDB = 2, Mogul = 1). We also assign ordinal
value to the answer of Q5 (Yes = 3, Sometimes = 2, No =1). The result of correlation
between them using one-tailed Spearman Rank Correlation Coefficient analysis is .112,
and the significance is .306. This shows that there is no significant statistical
relationship between the reuse level and the efficiency of component related
requirements (re)negotiation.
4.3 Value of component repository
From the answer of Q6, we found that 71% of respondents in Mogul and EDB regard
constructing a component repository as worthwhile, against 57% in Ericsson. To study
RQ3 and test hypothesis H02, the relationship between the answer of Q6 and the reuse
level is studied. We use the same ordinal number mapping as previously. The result of
correlation between them using one-tailed Spearman Rank Correlation Coefficient
114
Appendix A
analysis is -.124, and significance is .297, which shows that there is no obvious
relationship between them.
What i s t he r euse l evel i n your company?
80%
70%
ers
60%
ercentage of answ
50%
ERI CSSON
40% EDB AS
MOGUL
30%
P
20%
10%
0%
Ver y hi gh Hi gh Medi um Li t t l e Don' t know
Reuse Level
Fig. 1. Result of the question “What is the reuse level in your company?”
To what ext end do you f eel af f ect ed by r euse i n your wor k
80%
70%
ers
60%
percent age of answ
50%
ERI CSSON
40% EDB AS
MOGUL
30%
20%
10%
0%
Ver y hi gh Hi gh Medi um Li t t l e Don' t know
Ef f ect Level
Fig. 2. Result of the question “To what extend do you feel affected by reuse in your
work?”
4.4 Component understanding
Questions Q7-Q10f were used to investigate RQ4. For Q7, Q8 and Q9, the results show
that 67% of the respondents think the component structure is well understood, 61% say
that the component interfaces are understood, and 63% regard the design rules of
components are also well understood. But for the responses to question Q10a, no one
thinks that the design/code of components is well documented, 73% think that they are
sometimes well defined, and 27% believe that they are not well documented.
Furthermore, the answers to questions Q10b and Q10c indicate that 86% believe that
insufficient component documentation is a problem, e.g. documentation is not complete,
not updated, and difficult to understand, etc. From responses to Q10d and Q10f, we can
see that the preferable way of documentation is web pages. Some of the developers’
knowledge of how to use components comes from informal communication sources, for
example, previous experience, suggestions from local experts, etc. To study RQ5 and
test hypothesis H03, the association between reuse level and response to Q10a is
115
Appendix A
studied. We use the same ordinal number mapping as previously. The result of
correlation between them using one-tailed Spearman Rank Correlation Coefficient
analysis is -.469, and significance is .014, which shows that there is a weak negative
relationship between them. It means that the higher the companies’ reuse level, the less
satisfied a developer is with the component documentation.
4.5 Quality attributes of components
Question Q11 and Q12 were used to investigate RQ6. From the responses to these
questions, we see that 70% of the participants regard the design criteria for quality
requirements are not well defined, and 87% will test the quality attributes of
components after component modification, before integrating them into the system.
5 Discussions
Based on the result of the survey, we discuss our research questions and hypotheses, and
discuss the limitations and threats to validity.
5.1 Component related requirements (re)negotiation
Much research focus on how to improve the efficiency of component related

requirements (re)negotiation in COTS-based development [20] [24] [26]. The main
reason is that people think the challenges in requirements (re)negotiation are due to the
lack of access to source code, to timely vendor supports, or to the lack of engineering
expertise to modify the integrated components [26]. In our case, the components are
mostly built in-house. The above constrains on COTS components are not considered as
challenges with built in-house components. From the responses to question Q3-Q5, we
found that although 92% think that requirements of their typical projects are flexible,
less than half think the component related requirements (re)negotiation in their typical
projects works well.
Since components are intended to be used ‘as-is’, it is possible that an in-house
reusable component meeting all the requirements will not be found. So, even though the
components are built in-house, requirements (re)negotiation is necessary. For research
question RQ1, we do not want to claim that the requirements (re)negotiation based on
in-house components is more difficult than COTS-based components. We just want to
emphasize that requirements (re)negotiation based on in-house components is also
important but not efficient.
From the test result on H01, we cannot find a statistically significant relationship
between the reuse level and the efficiency of component related requirements
(re)negotiation. So, we cannot reject null hypothesis H01. Our conclusion to RQ2 is that
when IT companies change from a low reuse level to a higher reuse level, they probably
cannot expect that component-based requirements (re)negotiation becomes easier and
more efficient.
5.2 Component repository
116
Appendix A
Some researchers have claimed that repository is important, but not sufficient for
successful reuse [18] [21]. Our data confirms that developers are positive, but not
strongly positive to the value of component repository. So, this result gives future
support to the previous conclusion.
From the test result on H02, we can see that there is no statistically significant
relationship between developers’ positive attitude to a component repository and reuse
level. So, we cannot reject null hypothesis H02. Our conclusion to RQ3 is that
companies are not expected to invest in a repository to increase reuse.
5.3 Component understanding
Transferring component knowledge from the component developer to the component

user is critical for successful component reuse. The answers of Q7-Q9 show that most
developers understand the components in detail. However, the answers of Q10a-Q10c
show that no one believes that the components are well documented because the
documents are either incomplete or not updated. So, our question is “How can
developers still understand the components without good documentation?” From the
answers to question Q10e and Q10f, we found that most developers got the knowledge
of components from informal channels, such as previous experience and local experts.
The most important feature of a component is the separation of its interface from its
implementation. The component implementation is only visible through its interface.
Moreover, current component documentation technologies cannot describe all the
information the developer required, such as performance, reliability, and security etc.
Therefore, informal knowledge transfer should be considered to supplement the
insufficiency of formal component documentation and specification. This point was
shown in other empirical studies as well [6] [10]. For research question RQ4, we found
that informal knowledge transfer is especially important in the component reuse. One
possible solution is to have special interest groups or mailing lists for a components (or
group of similar components) so that component users can share knowledge and
experience of component usage.
From the test result on H03, we found a weak negative relationship between reuse
level and developers’ satisfaction with the documentation. We reject the null hypothesis
H03 and accept the alternative hypothesis HA3, It means the higher the companies’
reuse level, the less satisfied a developer is with components’ documentation. Marcus et
al. concluded that combine reuse education and training provided for staff with other
reuse activity can lead to all the success of reuse [18]. Our conclusion to RQ5 implies
that when a company moves from a low reuse level to high level, more effort should be
spent on the component documentation and component knowledge management.
5.4 Quality attributes of components
Component-based development relies on the availability of high quality components to

fill roles in a new intended system. When components are created or changed, we must
ensure that they do not only fulfill the functional requirements, but also quality
requirements. For research question RQ6, we found that most developers are not
satisfied with the specification of components’ quality attributes and therefore cannot
use this information. Therefore, how can we model quality properties of both
117
Appendix A
components and systems, and reason about them, particularly in the early stage of
system development is still a key challenge in component-based development.
5.5 Threats to validity
We now discuss the possible validity threats in this study. We use the definition given
by Judd et al. [14].
Construct validity In our case, the main construct issue applies to the variables
chosen to characterize the data set. The independent variable, i.e. reuse level, is the most
sensible one. The results of questions Q1 and Q2 give a qualitative and consistent value
on this variable.
Internal validity A major threat to this validity is that we have not assessed the
reliability of our measurement. Most variables are measured on a subjective ordinal
scale. An important issue for future studies is to ensure the reliability and validity of all
measurement. In this survey, we gave clearly specified concepts in the questionnaire
and provided support to possible misunderstanding. These methods partly increased the
reliability.
External validity The small sample size and lack of randomness in the choice of
companies and respondents are threats to external validity. In general, most empirical
studies in industry suffer from non-representative participation, since companies that
voluntarily engage in systematic improvement activities must be assumed to be better-
than-average.
Conclusion validity This study is still a pre-study. Future studies will be implemented
to give more statistically significant results.
6 Conclusion and future work
This study has investigated challenges related to four key factors for development based
on in-house components, especially in development-with-reuse. These factors are
component related requirements (re)negotiation, component repository, component
understanding and components’ quality attribute specification. Another contribution is
that we compared three IT companies with different reuse levels to study the possible
trend and challenges in these factors when more and more code will be encapsulated as
reusable components inside a company.
- For component-based requirements (re)negotiation, the results of research
questions RQ1 and RQ2 show that requirements (re)negotiation for in-house built
components is important but not efficient. The efficiency will probably not
increase with higher reuse level.
- For the component repository, the results of research question RQ3 confirm that a
component repository is not a key factor for successful reuse. Furthermore, the
potential value of a component repository will probably not increase with higher
reuse levels.
- For component understanding, the results of research questions RQ4 and RQ5
show that most developers are not satisfied with the component documentation,
and developers’ satisfaction with component documentation will probably
decrease with higher reuse level. The results also show that informal
118
Appendix A
communication channels, which developers can get necessary information about

the components through, should be given more attention
- For components’ quality attribute specification, the result of research question
RQ6 shows that developers still need to spend much effort on testing, as they
cannot get relevant information from component specifications.
The main limitation of our survey is that it depends on the subjective attitudes of
developers, and with few companies and participants involved. Later studies are
planned to be undertaken with more precise quantitative methods and on more
companies with more distinct reuse levels. Case studies will also be undertaken to
follow the change of companies from lower reuse level to higher reuse level to future
investigate our research questions.
7 Acknowledgements
This study is supported by the SPIKE and INCO projects. We thank the colleagues in
these projects, and all the participants in the survey.
References
1. Erik Berglund: Writing for Adaptable Documentation. Proceedings of IEEE

Professional
Communication Society International Professional Communication Conference and
Proceedings of the 18th ACM International Conference on Computer Documentation:
Technology & Teamwork, Cambridge, Massachusetts, September (2000) 497– 508
2. Pearl Brereton: Component-Based System: A Classification of Issues. IEEE
Computer, November (2000), 33(11): 54– 62
3. Alan W. Brown: The Current State of CBSE. IEEE Software, September/October
(1998) 37– 46
4. Alan W. Brown: Large-Scale Component-Based Development. Prentice Hall, (2000)
5. Alan. W. Brown and Grady. Booch: Reusing Open-source Software and Practices:
The Impact of Open-source on Commercial Vendors. Proceedings: Seventh
International Conference on Software Reuse, Lecture Notes in Computer Science,
Vol. 2319. Springer, (2002) 123– 136.
6. Reidar Conradi, Tore Dybå: An Empirical Study on the Utility of Formal Routines to
Transfer Knowledge and Experience. Proceedings of European Software Engineering
Conference, Vienna, September (2001) 268– 276
7. Bill Councill and George T. Heineman: Component-Base Software Engineering and
the Issue of Trust. Proceedings of the 22nd International Conference on Software
Engineering, Limerick, Ireland, June (2000) 661– 664
8. Ivica Crnkovic and Magnus Larsson: A Case Study: Demands on Component-based
Development. Proceedings of the 22nd International Conference on Software
Engineering, Limerick, Ireland, June (2000) 21– 31 .
9. Ivica Crnkovic: Component-based Software Engineering - New Challenges in
Software Development. Proceedings of 25th International Conference on Information
Technology Interfaces, Cavtat, Croatia, June (2003) 9– 18
119
Appendix A
10. Torgeir Dingsøyr, Emil Røyrvik: An Empirical Study of an Informal Knowledge

Repository in a Medium-Sized Software Consulting Company. Proceedings of 25th
International Conference on Software Engineering, Portland, Oregon, USA, May
(2003) 84– 92
11. W. B. Frakes: An Empirical Framework for Software Reuse Research. Proceedings
of the Third Annual Reuse Workshop, Syracuse University, Syracuse, N.Y. (1990)
12. W. B. Frakes, C.J. Fox: Sixteen Questions about Software Reuse. Communication
of the ACM, June (1995), 38(6): 75– 87
13. Ivar, Jacobson, Martin Griss, Patrick Jonsson: Software Reuse-Architecture, Process
and Organization for Business Success. Addison Wesley Professional, (1997)
14. C.M. Judd, E.R. Smith, L.H. Kidder: Research Methods in Social Relations. Sixth
edition, Holt Rinehart and Winston, (1991)
15. Y. Kim and E.A. Stohr: Software Reuse: Survey and Research Directions. Journal of
Management Information System, (1998), 14(4): 113– 147
16. C. Kruger: Software Reuse. ACM Computing Surveys, (1992), 24(2): 131– 183
17. N. Y. Lee, C. R. Litecky: An Empirical Study on Software Reuse with Special
Attention to Ada. IEEE Transactions on Software Engineering, September (1997),
23(9): 537– 549
18. Marcus A. Rothenberger, Kevin J. Dooley and Uday R. Kulkarni: Strategies for
Software Reuse: A Principal Component Analysis of Reuse Practices. IEEE
Transactions on Software Engineering, September (2003), 29(9): 825– 837
19. H. Mili, F. Mili, A. Mili: Reusing Software: Issues and Research Directions. IEEE
Transactions on Software Engineering, June (1995), 21(6): 528– 561
20. M. Morisio, C.B. Seaman, A. T. Parra, V.R. Basili, S.E. Kraft, S.E. Condon:
Investigating and Improving a COTS-Based Software Development Process.
Proceeding of 22nd International Conference on Software Engineering, Limerick,
Ireland, June (2000) 31– 40
21. Maurizio Morisio, Michel Ezran, Colin Tully: Success and Failure Factors in
Software Reuse. IEEE Transactions on Software Engineering, April (2002), 28(4):
340– 357
22. Parastoo Mohagheghi and Reidar Conradi: Experiences with Certification of
Reusable Components in the GSN Project in Ericsson, Norway. Proceedings of the
4th ICSE Workshop on Component-Based Software Engineering: Component
Certification and System Prediction. Toronto, May (2001) 27– 31
23. Jeffrey S. Poulin: Measuring Software Reuse-Principles, Practices, and Economic
Models. Addison-Wesley, (1997)
24. Vijay Sai: COTS Acquisition Evaluation Process: The Preacher’s Practice.
Proceedings of 2nd International Conference on COTS-based software systems,
Lecture Notes in Computer Science, Vol. 2580. Springer, 2003, Ottawa, Canada,
February (2003) 196– 206
25. Guttorm Sindre, Reidar Conradi, and Even-Andre Karlsson: The REBOOT
Approach to Software Reuse. Journal of System Software, (1995), 30(3): 201– 212
26. Vu N. Tran, Dar-Biau Liu: Application of CBSE to Projects with Evolving
Requirements- A Lesson-learned. Proceeding of the 6th Asia-Pacific Software
Engineering Conference (APSEC’ 99) Takamatsu, Japan, December (1999) 28– 37
120
Appendix A
27. Padmal Vitharana: Risks and Challenges of Component-based Software

Development. Communications of the ACM, August (2003), 46(8): 67– 72
28. Katharine Whitehead: Component-Based Development: Principles and Planning for
Business Systems. Addison-Wesley, (2002)
29. http://www.idi.ntnu.no/grupper/su/spike.html
30. http://www.ifi.uio.no/~isu/INCO/
121
Appendix A
122
Appendix A
P2. Variations in COTS-Based Development Process

An Empirical Study of Variations in COTS-based Software
Development Processes in the Norwegian IT Industry
Jingyue Li 1, Finn Olav Bjørnson 1, Reidar Conradi 1,2, Vigdis By Kampenes 2

1
Dept. of Computer and Information Science
Norwegian Univ. of Science and Technology
{jingyue,bjornson,conradi}@idi.ntnu.no
2
Simula Research Laboratory
P.O.BOX 134, NO-1325 Lysaker, Norway
vigdis@simula.no
Abstract. More and more software projects use Commercial-Off-The-Shelf (COTS)

components. Although previous studies have proposed specific COTS-based
development processes, there are few empirical studies that investigate how to use and
customize COTS-based development processes for different project contexts. This paper
describes an exploratory study of state-of-the-practice of COTS-based development
processes. Sixteen software projects in the Norwegian IT companies have been studied
by structured interviews. The results are that COTS-specific activities can be
successfully incorporated in most traditional development processes (such as waterfall
or prototyping), given proper guidelines to reduce risks and provide specific assistance.
We have identified four COTS-specific activities – the make vs. buy decision, COTS
component selection, learning and understanding COTS components, and COTS
component integration – and one new role, that of a knowledge keeper. We have also
found a special COTS component selection activity for unfamiliar components,
combining Internet searches with hands-on trials. The process guidelines are expressed
as scenarios, problems encountered, and examples of good practice. They can be used to
customize the actual development processes, such as in which lifecycle phase to put the
new activities into. Such customization crucially depends on the project context, such as
previous familiarity with possible COTS components and flexibility of requirements.
1. Introduction
COTS-based development has become more and more important in software and system
development. Using COTS components promises faster time-to-market and increased
productivity (Voas, 1998a). At the same time, COTS-based software introduces many
risks. This means that unknown quality properties of the chosen COTS components can
be harmful for the final product, and a COTS vendor may terminate the maintenance
support of its COTS components (Voas, 1998b).
The use of COTS components introduces new system issues, which require revised
software development processes. Although researchers and practitioners have been
grappling with these new processes, most studies have been based on military or
aerospace projects (Morisio et al., 2000; SEI, 2004), or similar large projects. In order to
propose and design cost-effective COTS-based development processes, it is necessary
123
Appendix A
to empirically investigate how COTS-based projects were performed in different

application domains, particularly in small or medium-sized companies.
This study has investigated the commonalities and differences between development
processes in 16 finished COTS-based software projects in Norway. It also summarized
process scenarios in successful vs. unsuccessful COTS-based projects. Eleven salient
problems encountered and 12 examples of good practice are extracted. A process
customization guideline is proposed to help software practitioners to integrate relevant
COTS components activities successfully in different project contexts.
The remainder of the paper is structured as follows: Section 2 presents previous
studies on COTS-based development processes. Section 3 describes the research
approach. Contexts of selected projects are presented in Section 4. The results are
presented in Section 5 and discussed in Section 6. Our conclusions and proposals for
future research are presented in Section 7.
2 COTS-based software development processes
2.1 COTS component definition
A pertinent question is “What do you mean by a COTS component?” There are many
different definitions of COTS components (Basili and Boehm, 2001; Carney and Long,
2001). We have used the definition by Torchiano and Morisio (2004), where a COTS
component should satisfy all the following requirements:
- It is either provided by some other organizations in the same company, or

provided by external companies as a commercial product.
- It is integrated into the final delivered system.
- It is not a commodity. It is not shipped with an operating system, not provided
with the development environment, and not generally included in any pre-
existing platforms.
- It is not controllable by the user, in terms of provided features and their
evolution.
- It is “black box” software. The source code is not available.
The granularity of a COTS component can be different (Torchiano and Morisio,

2004). Some consider that COTS components could or should include very large
software packages such as Microsoft Office. Others limit COTS components to GUI
libraries. In this study, we focus on COTS components as software components. Such a
component is a unit of composition, and must be specified so that it can be composed
with other components and integrated into a system (product) in a predictable way
(Crnkovic et al., 2002). That is, a component is an “Executable unit of independent
production, acquisition, and deployment that can be composed into a functioning
system.” This definition implies that we include not only components following COM,
CORBA, and EJB standards, but also software libraries like those in C++ or Java. This
definition is consistent with the scope used in the component marketplace
(Componentsource, 2004).
124
Appendix A
2.2 COTS-based development processes
Typically, a COTS-based process consists of four phases, comprising (Abts et al.,

2000):
- COTS component assessment and selection
- COTS component tailoring
- COTS component integration
- Maintenance of COTS and non-COTS parts of the system
There is consensus that the use of a COTS component implies changes in the
software process (Brownsword et al., 2000). Most studies of a COTS-based
development process focus on two dimensions: Process of the whole software
development lifecycle and process of the specific phase, especially in COTS component
selection and evaluation.
2.2.1 Process of the whole software development lifecycle. Boehm and Abts (1999)
regard both the waterfall model and evolutionary development as unsuitable for COTS-
based development because:
- In the sequential waterfall model, requirements are identified at an earlier stage

and the COTS components are chosen at a later stage. This increases the
likelihood of the COTS components not offering the required features.
- Evolutionary development assumes that additional features can be added if
required. However, COTS components cannot be upgraded for one particular
development team. The absence of source code bars the development team from
adjusting the COTS components to their needs.
Based on the above arguments, Boehm and Abts (1999) proposed that development
models which explicitly take risk into account are more suitable for COTS-based
development than the traditional waterfall or evolutionary approaches.
The Software Engineering Institute has a large ongoing effort to address the
development of COTS-based systems, and they have developed the Evolutionary
Process for Integrating COTS-based Systems (EPIC) (Albert and Brownsword, 2002).
The proposed process integrates COTS-related roles and activities into a RUP process.
The iterative and evolutionary nature inherent in this process allows developers to adjust
the architecture and system design, as more knowledge is gained about the operations of
the COTS components.
The National Aeronautic and Space Administration (NASA) has been developing IT
systems with COTS components for many years. Their experience has been captured by
Morisio et al. (2000). They have investigated the various processes that were used
across 15 projects at NASA. Based on their insight, they have developed a COTS-based
development process that was the most representative for the processes used in the
projects.
2.2.2 COTS component selection and evaluation process. Based on case studies,
researchers have proposed several COTS component selection processes and methods.
One kind of process is based on direct assessment (Leung and Leung, 2002), such as
125
Appendix A
MBASE (Model Based Architecting and Software Engineering) (Boehm, 2000), OTSO
(Off-the-Shelf-Option) (Kontio, 1996), CISD (COTS-based Integrated System
Development) (Tran et al., 1997), PORE (Procurement-Oriented Requirement
Engineering) (Maiden and Ncube, 1998), CAP (COTS Acquisition Process) (Ochs et
al., 2001), IIDA (Infrastructure Incremental Development Approach)(Fox et al., 1997),
CARE (COTS-Aware Requirements Engineering) (Chung and Cooper, 2002), RCPEP
(Lawlis et al., 2001), and CRE (COTS-Based Requirements Engineering) (Alves and
Finkelstein, 2003). In general, the direct assessment process includes three basic steps:
- Inspect all modules of each of the available COTS components to check whether
they satisfy some or all of the functional requirements of the COTS-based system
being developed.
- Check whether a COTS component also satisfies the non-functional requirements
of the COTS-based system. Non-functional requirements may include properties
such as the interoperability of the modules of the COTS product with other
systems.
- Compare the COTS component candidates and select the most appropriate COTS
component that satisfies both the functional and non-functional requirement of
the COTS-based system.
Some of the direct assessment processes, such as OTSO (Kontio, 1996), CAP (Ochs
et al., 2001), and CISD (Tran et al., 1997), assume that the requirements are fixed and
select the COTS components by comparing how well the COTS component candidates
satisfy the requirements. A formal decision-making process is usually used to select the
best COTS component (Ncube and Dean, 2002). The formal decision-making process
usually includes three basic elements: selecting evaluation criteria (factors), collecting
and assigning values to these criteria, and applying formal decision-making algorithms
such as MAUT (MacCrimmon, 1973), MCDA (Morisio and Tsoukias, 1997), and AHP
(Saaty, 1990). Other direct assessment processes, such as MBASE (Boehm, 2000),
PORE (Maiden and Ncube, 1998), IIDA (Fox et al., 1997), CARE (Chung and Cooper,
2002), RCPEP (Lawlis et al., 2001), and CRE (Alves and Finkelstein, 2003) emphasize
the trade-offs between the requirements and COTS component functionalities. These
processes do not assume that the requirements are fixed and should not be changed. On
the other hand, these processes propose that the requirements in COTS-based
development should be flexible enough so that they can be negotiated or changed
according to the available functionalities of the COTS components.
Another kind of process is based on a domain-model (Leung and Leung, 2002). It
includes a set-up phase and a selection phase. In the set-up phase, the vendors need to
map their COTS modules to those modules of the domain that they find are applicable.
In the selection phase, the corresponding modules in a domain model are identified for
each of the modules of the COTS-based system in question. Then, the COTS modules
that claim to be applicable are identified by the mapping from the domain models to the
COTS modules. After that, the non-functional properties of the identified COTS
modules are assessed. In the end, the most appropriate COTS modules are selected with
reference to all of the assessment results.
126
Appendix A
3 Research approach
Most studies on COTS-based development processes are limited to proposals of new

technology (e.g. revised processes) without empirical evidence, or to discovery of
existing practice (e.g. component selection) from case studies. Empirical studies are few
and mostly based on a small project sample or on large projects with rather similar
contexts, such as from aerospace or defense. It is therefore difficult to identify the
precise relationship between a project context and COTS-based development processes.
It is consequently hard for project managers to customize their COTS-based processes
according to their project context.
Our research was designed as an exploratory study, i.e. mostly a qualitative study.
The focus is to investigate the variations in COTS-based development processes and
summarize these variations by scenarios. The intent is to find out how to customize a
COTS-based development process based on its project context. Our research focuses on
the development phases, not on software maintenance and evolution.
3.1 Research questions
3.1.1 Research question RQ1. Boehm and Abts (1999) regard both the waterfall and
evolutionary lifecycle process as unsuitable for COTS-based development. Most COTS-
based projects must therefore adjust their development process to a risk-driven spiral
process, instead of a waterfall and evolutionary one. However, there are still no
conclusive empirical studies on this issue. So, our first research question is:
RQ1: What was the actual process used in the projects using COTS components?
3.1.2 Research question RQ2. Morisio et al. (2000) have proposed that COTS-based
software processes differ considerably from “traditional” lifecycle processes, such as
waterfall, or prototyping. Not modifying the traditional process can be a failure factor.
They also pointed out that their proposed process has several variations for
customization purposes. They also indicated that it is important to investigate under
what conditions each variation is most suitable. However, there have been no further
studies on this so far. Our second research question is then:
RQ2: What are the commonalities and possible variations in COTS-based

3.1.3 Research question RQ3. All previous studies on COTS-based development

processes have mentioned the possible problems (or risks) in using COTS components
(Boehm and Abts, 1999; Morisio et al., 2000; Albert and Brownsword, 2002). To
successfully customize COTS-based processes, it is important to summarize and
understand the experience from successful COTS-based projects vs. those that were not
so successful. It is especially valuable to generalize the relationship between possible
risks and the process adjustments (variations). It may help project managers to
successfully adopt a COTS-based development process without taking unnecessary
risks. So, our third research question is:
127
Appendix A
RQ3. What are process scenarios (variations) of the projects using COTS
components successfully and not successfully? What are possible problems and good
practices in different process scenarios?
3.2 Data collection method
The initial plan for this study was to make a quantitative survey using a pre-stated
hypothesis with standardized questions in a formal questionnaire. After two rounds of
pre-tests on the questionnaire, we found that the questionnaire requires substantial work
to give consistent and reliable results. We decided, as a first step, to go for an
exploratory pre-study using semi-structured interview.
Face-to-face interviews offer the possibility of modifying one’s line of enquiry,
following up interesting responses and investigating underlying motives in a way that
questionnaire cannot (Robson, 2002). The choice of the semi-structured interview
approach enabled more open research questions, not fixed hypotheses, but still relying
on a rather detailed interview guide with both closed and open-ended questions. The
advantages of open ended questions are that they allow us to go into depth and clear up
misunderstandings, enable testing of limits of a respondent’s knowledge, allow us to
make a truer assessment of what the respondents really believe, and opening for
unexpected or unanticipated answers.
3.3 Interview guide
We used a modified version of the original questionnaire to serve as an interview guide.

The interview guide includes both closed and open-ended questions.
The closed questions were used to solicit information on project context. For
example, one question used the application domains of the final system (product), which
was based on the North American Industrial Classification System (NAICS) (NAICS,
2004).
The open-ended questions were used to gather information on actual COTS-based
development processes. The questions covered the main lifecycle process, the process
changes due to using COTS components, and the experience and lessons learned in the
whole COTS-based development process. Some of the questions were arranged in an
“if-then” structure that leads the interviewer along one of several paths depending on
the answers to previous questions. For example, one question was to ask about whether
they had changed or added some activities in their process because of using a COTS
component. We then asked when and how did they make these changes if they added or
changed some activities.
The interview guide was written in English. The reason was that we planned to
extend this study to other countries later. On the first page, we gave the definitions of
most concepts that were used in later questions, such as COTS component, project, and
system (consisting of application, components, and addware/glueware). The rest of the
interview guide had questions organized in three sections:
- Questions to characterize companies, projects, and COTS components.

- Questions to characterize the individual respondents.
128
Appendix A
- Questions on COTS-based development processes, and covering mainly COTS

component selection and integration.
To evaluate the construct validity of the interview guide, we added three meta-
questions following each question. These meta-questions asked the respondents, if the
question was clear, if respondents needed clarification, and if respondents could give
specific comments if the question was confusing. All this is a standard procedure from
social science (Fowler, 2001).
Lastly, to ensure that a respondent had understood the COTS-relevant definitions
correctly and could select a proper company project, we asked the respondent to give
further examples of COTS components based on their understanding.
3.4 Sampling
Interviews and data collection were conducted by Li and Bjørnson from Nov. 2003 to
Jan. 2004. They contacted 34 IT companies, where most were involved in two
Norwegian R&D Projects (INCO, 2000; SPIKE, 2002) and one EU R&D project
(FAMILIES, 2003).
A pre-process was used to select relevant COTS-based projects. Contact persons (IT
manager in most cases) in the above projects were first contacted by phone and email,
using an introductory letter with our definition on COTS component and project (the
same as definitions in the interview guide). They were asked to check if they have a
relevant COTS-based project. During this pre-process, 17 companies (among 34) did
not match our definition of a COTS-based project, and therefore could not join the
study. Of these 17 “rejected” companies, 13 used only in-house built components and 3
used only open source components. There was one other company that could not
participate because of confidentiality issues.
For the remaining 17 companies, 13 of these volunteered to join in the study. The
contact persons in these 13 companies were asked to recommend persons with most
experience on the relevant projects. In all, 16 persons were recommended from these 13
companies. For each recommended respondents, they were asked to select one relevant
COTS-project. If a respondent claimed that he/she had experience from several COTS-
based projects, the respondent was asked to select one that he/she was most familiar
with. The whole process to find a suitable respondent and project in a prospective
company, and later agree upon place and time for a physical interview took weeks.
Although the total sample was based on convenience, we ensured that we had
included respondents from large, medium and small companies.
3.5 Data collection procedure
The interview guide was sent to respondents a few days before the personal interview.
In this way, the respondents were well-prepared and could have time to remember the
details of the projects and find some relevant documents.
We offered the company respondent a compensation of 100 euro and/or a copy of a
status report. Most companies did not want money, rather a status report. We
emphasized that all answers and other information would be treated strictly as
confidential.
129
Appendix A
Most of the answers and comments requested by the interview guide were filled-in
on paper during the interview by the interviewer. Each interview was conducted by one
of the two interviewers (none with two interviewers), took from 60 to 80 minutes, and
was recorded.
3.6 Data analysis
The first step in data analysis was to listen to the tape, and supplement and transcribe
the interview “as-is” into a field note. After that, we used different analysis procedures
according to the purpose of different research questions.
The purpose of the first research question RQ1 is to investigate the actual
development processes used in the project. The data was analysed by the constant
comparison method (Strauss and Corbin, 1998). As all respondents had a clear
definition on the main process used in their project. These processes were early grouped
into traditional process (waterfall, prototyping, or incremental) or a totally new process.
The purpose of the second research question RQ2 is to see the similarities and
differences between process changes in the studied projects. The main analysis method
used is cross-case analysis (Strauss and Corbin, 1998), i.e. treating each project as a
separate “case”. The field notes for the first two projects were first reviewed. For each
of these two projects, a list was compiled with short phases that describe the process
changes in each project (i.e. new or changed activities, when and how these changes
were performed). Then these two lists were compared to determine the similarities and
differences. The next step was to list, in the form of propositions, conclusions one
would draw if these two projects were the only two in the data set. Each proposition had
associated process changes that supported it. After analysing the first two projects in
this way, the third project was examined, and a list of its process changes was compiled.
Then it was determined whether this third project supported or refuted any of the
propositions formulated from the first two. If a proposition was supported, this third
project was added to its list of supporting evidence. If it contradicted a proposition, then
either the proposition was modified or the project was noted as refuting that proposition.
Any additional propositions suggested by the third inspection were added to the list.
This process was repeated for each project. The end result was a list of propositions,
each with a set of supporting and refuting evidence (projects).
The purpose of the third research question RQ3 is to summarize the process
scenarios of the successful and unsuccessful projects. We first grouped projects into two
groups based on the positive or negative effect of the COTS components regarding
time-to-market and general quality of the system. In each group, we divided projects
into sub-groups based on their main actual processes used. For each sub-group, we
performed a similar cross-case analysis as for research question RQ2. The difference is
that we included more information for analysis. For each project, a list of short phases
was compiled that describe not only the process changes in each project, i.e. new or
changed activities, but also when and how these changes were performed. We also
recorded project contexts, i.e. the developers’ experience with COTS components. To
summarize the lesson learned in different process scenarios, we selected the main
problems met and examples of good practice performed in each process scenario. The
problems and examples of good practice that occurred in more than one project were
grouped into one item. Some problems and examples of good practice, which occurred
130
Appendix A
in only one project, were also listed because they were regarded as the main factor that
decided the success and failure of the project.
4 Collected samples
4.1 Companies
As mentioned, we interviewed 16 projects in 13 companies. All the companies are

Norwegian IT companies. Nine of these 13 are stand-alone companies, with staff size
between 5 and 500. The other companies are subsidiaries, with local staff size ranging
from 50 to 320. Six companies are IT consultancies, 5 are software vendors, and 2 are
telecom companies. Five of these companies are publicly traded companies, the others
are privately held. Some company background information is listed in Table 1.
Table 1. Background information on the 13 IT companies (called C1 to C13)

Comp Staff Size Type Ownership Main business
any ID area
C1 42 Stand-alone Privately held IT Consulting
C2 60 Subsidiary Privately traded IT Consulting
C5 500 Stand-alone Publicly traded IT Consulting
C6 320 Subsidiary Publicly traded Telecom Industry
C7 120 Stand-alone Publicly traded Software Vendor
C8 100 Stand-alone Privately held Software Vendor
C10 275 Subsidiary Publicly traded Telecom Industry
C11 50 Subsidiary Publicly traded Software Vendor
C12 5 Stand-alone Privately held Software Vendor
C13 80 Stand-alone Privately traded Software Vendor
4.2 Respondents
We had one respondent from each project. Three respondents are IT managers, 4 are
project managers, 5 are software architects, 3 are software developers, and 1 is a
software development researcher. Nine of them have more than 10 years of software
development experience, and 12 of them have more than 4 years of working experience
with COTS-based development. Eight of them have a master degree, and the rest have a
bachelor’s degree. Twelve of them have a principal degree in informatics or computer
science.
4.3 Projects
The company projects were selected based on two criteria:

- The project should use one or more COTS components
- The project should be a finished project and possibly with maintenance, and
131
Appendix A
Generally, we selected only one project from a company. However, we selected two
projects in three companies because these projects differed largely in project members,
development process, and COTS components used.
Background information includes effort spent, development language, and
application domain. Table 2 lists the basic information on the 16 projects.
Table 2. Background information on the 16 projects (called P1 to P16)

Company Project Effort Main Application
ID ID (person-hours) development domain
language
C1 P1 7,000 Java Finance
C1 P2 30,000 Power Builder Finance
C2 P3 700 C, Java Scientific
C2 P4 42,000 C Scientific
C3 P5 16,000 C++ Internet Service
provider
C3 P6 23,000 C++ Telecom.
C4 P7 2,400 Java Agriculture
C5 P8 3,000 Visual Basic El. Power
C6 P9 58,000 C++, Java Telecom
C7 P10 63,000 C++, VB Whole trade
C8 P11 8,000 C++ El. Power
C9 P12 10,000 C++ Construction
C10 P13 128,000 C++ Telecom
C11 P14 120,000 C++ Transport
C12 P15 92,000 Ada, Java Finance
C13 P16 15,000 C++ Scientific
4.4 COTS components
Some projects used one or two COTS components, others used more. Because of time
limitations, we asked the respondents to select no more than three typical COTS
components which were used in their projects. Totally, we gathered information about
30 different COTS components in 16 projects. Some typical COTS components are
listed in Table 3.
Table 3. Background information on some COTS components
COTS component name Functionality
CTI Computer Telephony Integration
Sheridan GUI
Intelligent Report Report generator
Active Reports Report preview and printing
Open MP Open parallel processing
IMAPP Image processing
SearchEngine Searching the context in the website
ComWork Workflow management
XML lite XML parser
132
Appendix A
The list is not exhaustive, and is only used to indicate the kinds of selected COTS
components. Some of the COTS components are based on standards like COM,
CORBA, and EJB, and some of them are C++ or Java libraries. All the COTS
components follow our previous definition, i.e., they were integrated into the final
product and were delivered together with the final system to the customer.
5 Results
5.1 Overall process used in COTS-based software development
Research question RQ1 is to identify the actual processes used in projects with COTS
components. To study this issue, we asked two open-ended questions: we first ask the
respondents to describe the development processes they used. We also asked if they
selected the actual development processes because of using COTS components.
Results of the first question show that the main development processes of the studied
projects can be grouped into three categories: pure waterfall, waterfall mixed with
prototyping, and incremental mixed with prototyping. Five projects used waterfall. One
used waterfall mixed with little prototyping (a small prototype was used for discussing
requirements of the GUI part of the system). Ten projects used incremental mixed with
some prototyping.
Results of the second question show that none of the respondent claimed that they
selected the main development process based on whether or not they intended to use
COTS components. That is, they decided the main process before they started to think
about using COTS components or not. They just added or changed some activities and
roles in their traditional process to reflect the use of COTS component.
Therefore, the answer to research question RQ1: The so-called COTS-based
development process is the customization of the traditional development process due to
the use of COTS components.
5.2 Commonalities and variances in COTS-based software development process
Research question RQ2 investigates the commonalities and possible variations in the
development processes. To study the common changes in the develop processes, we
asked if there are any new or changed activities, role and responsibilities in the project
due to use of COTS components. The detailed answers are listed in Appendix A. The
commonalities are that new activities were added and possible one new role was added.
The added new activities include:
- Make vs. buy decision: In COTS-based development, the first extra activity in
all the studied projects was to make the make vs. buy decision. Non-technical
issues, such as costs, licensing, and market trends, as well as technical issues
may dominate this stage.
- COTS component selection: After the respondents decided to use COTS
component; the following extra activity in all the 16 projects was COTS
component selection. However, different projects used different selection
processes and criteria in their COTS component selection.
133
Appendix A
- Learn and understand the COTS components: Another new activity is to

learn and understand the COTS components. Developers needed to spend time
to study several possible candidates briefly before the final selection. After they
have selected the COTS components, they needed to spend time to learn how to
use the selected COTS components. Because the source code of a COTS
component is often not available, it is difficult for developers to profoundly
understand a COTS component.
- Build glueware and addware: A future step after COTS component selection is
integration. All the projects built more or less glueware (the code to integrate the
COTS components) and/or addware (the code to complete the required
functionality of COTS components).
The possible added new role is: a COTS component knowledge keeper. As
mentioned above, it was difficult to really understand the COTS components. In 10
projects, one or more project members had some previous experience with selection and
integration of the actual COTS components. They provided suggestions on COTS
component selection and integration. Although some of them were not dedicated to
work as a COTS component knowledge keeper, their previous experience improved the
speed and quality of COTS component selection and integration.
To investigate the variations in the processes, we transcribed respondents’
description on the above new activities and compared when, how, and why these
activities are performed. The detailed information on when and how the COTS
components were selected is listed in Appendix B. The results show that the main
variations in the customization of main development processes deal with the following
activities:
- The phase to make the make vs. buy decision and the COTS component
selection: Morisio et al. (2000) proposed a two-phase make vs. buy decision.
The first decision is in the requirement phase and the second in the design phase.
Our results indicate that only 2 of the 5 waterfall projects made the make vs. buy
decision and COTS component selection in the requirement phase. If the project
members had no relevant experience with specific COTS component, it was very
hard to make that choice in the requirement phase. Two waterfall process
projects managed to make the make vs. buy decision and COTS component
selection in the requirement phase, because they were already familiar with the
possible COTS components.
- The COTS component selection and evaluation process: The selection
process in our studied projects can be summarized in two categories:
• Familiarity-based selection process (SP_fam): The previous familiarity
came either from the project members in the projects, i.e. the COTS
component knowledge keeper, or from a colleague outside the project. If the
project members had enough previous experience with the candidate COTS
components, this experience will be the key factor in the COTS component
selection. In all, 16 of the 30 given COTS components were selected based
on the suggestion of the COTS component knowledge keeper inside the
projects. Four COTS components were selected based on recommendations
from colleagues outside the project or organization.
134
Appendix A
• Internet search, trial-based selection process (SP_unfam): This kind of

selection was used when there was no previous experience available. This
selection was more complex than SP_fam, and involved three steps:
Step 1: First, developers used a search engine to browse the Internet, using
keywords to express decided functionality. This usually gave them a handful
of candidates.
Step 2: If there were more than 2 to 3 possible candidates, they selected 2 to
3 of them very quickly based on some key issues, such as licence issues,
cost, and vendor reputation.
Step 3: After a small number of candidate COTS components had been
decided, they downloaded a demo version of these from the web and tested
the COTS components.
This kind of selection is similar to some proposed processes, such as OTSO (Kontio,
1996), CAP (Ochs et al., 2001), and CISD (Tran et al., 1997). The main difference is
that the actual process was much faster and no decision-making algorithms have been
used. In addition, developers knew that it was impossible to test several COTS
components completely due to limited time and cost. They generally tested only some
of the key functionalities, and depended on comments from a newsgroup to evaluate the
quality of the COTS components. Therefore, COTS components with more and better
comments in the newsgroup or marketplace bulletin had a good chance to be selected,
because they were assumed to be better tested and in general of higher quality.
The answer to research question RQ2: There are some common new activities and
one new role is added in COTS-based development processes. The possible variations
are when and how to perform these new activities.
5.3 Scenarios of projects that used COTS successfully and unsuccessfully
Research question RQ3 is about summarizing the process scenarios of projects using
COTS components successfully and unsuccessfully. The focus is to investigate the
relationship between the possible risks and the development processes. We asked
respondents to summarize their problems encountered (PEs) and their examples of good
practices (GP) in the studied projects. First, we extracted process-relevant parts, i.e.
what were the main development processes, and when and how were the new activities
performed. We then summarized some other salient problems and examples of good
practice, which may give guidance to COTS-based development.
In our definition, a project that used COTS components successfully means that the
COTS components contributed positively to time-to-market and system quality.
Otherwise, it is regarded as being used unsuccessfully.
5.3.1 Scenarios of projects that used COTS components unsuccessfully. Four of the
16 studied projects were regarded as using COTS components unsuccessfully. The
scenarios are summarized in scenarios Sc1 to Sc3 in Table 4.
135
Appendix A
Table 4. The scenarios of the 4 projects that used COTS components unsuccessfully
Scenari Characterizatio Problems Good Practice (GP)
o ID ns Encountered (PEs)
(Project
ID)
Sc1 (P1 Two projects used PE1: Selected the GP1: Used external
and waterfall processes. COTS component too COTS expert.
P14) Made the make vs. late.
buy decision and PE2: Used only
the COTS demonstration,
component without pilot.
selection in the PE3: Could not (re)
implementation negotiate
phase. requirements with
Selected unfamiliar customer.
COTS components. PE4: It was difficult
to formally verify the
COTS component’s
reliability.
Sc2 One project used PE5: Selection was GP2: Used external
(P10) waterfall process. based only on COTS expert with
Selected COTS functionality. Ease of internal developers.
components in the integration was not
design phase, but considered.
mainly on their
functionality
Selected unfamiliar
COTS components.
Sc3 One project used PE5: Selection was
(P2) incremental & based only on
prototyping functionality. Ease of
process. integration was not
Selected the COTS considered.
components in the
design phase, but PE6: COTS
mainly because of component’s delay
their functionality. caused the delay of
Selected unfamiliar the project.
COTS components.
The detailed backgrounds of each problem encountered are following:

PE1: The two waterfall projects (P1 and P14) started to select COTS components in
the implementation phase. It was very difficult for them to select the suitable COTS
components, when the customer requirements and design have been fixed. They
built a lot of glueware to integrate the selected COTS components, and this brought
many integration problems at the very end.
136
Appendix A
PE2: In project P1, the COTS component was selected only based on the vendor’s
demonstration. They did not do any hands-on trial to investigate how much effort is
needed to integrate the COTS component into the current system. As mentioned in
PE1, they had to spend a lot of effort on integrating the COTS components into the
system, which caused delay in the project.
PE3: Although some previous studies proposed to (re)negotiate requirements with
the customer when developers found later limitations of COTS components,
developers in project P14 could not (re)negotiate the requirements with customers
even if they wanted to. This is because the developer wanted to use COTS
components, not the customer. The customer cared only about the final result. If
developers were not allowed to use COTS components, (re)negotiation of
requirements due to COTS limitations is difficult.
PE4: The final product of project P14 is a highly safety-critical system. To ensure
reliability of the product, each line of code was required not only to be tested
strictly, but also to be formally proved. As the source code of the COTS component
was not available, they could not formally prove the correctness of the COTS
component even the COTS component passed all existing testing. They were
therefore always worried about it until they bought the source code.
PE5: In project P10, the project members did not know exactly how to integrate the
COTS components into the system and how to integrate them with other
components before COTS component selection. Their customer intervened in the
COTS component selection and recommended a certain COTS component that
could provide the needed functionality. This COTS component caused a lot of
quality problems.
Although these projects were regarded as being unsuccessful, there were still some
examples of good practice that helped to improve the project results. The background of
each example of good practices is the following:
- GP1: Project P1 hired one person from the vendor as a part-time consultant to
configure and integrate the COTS. Although a lot of glueware was needed to
integrate the COTS component because of the wrong selection, this external
expert helped to shorten the integration time to some degree.
- GP2: In project P10, the whole component part was outsourced to a consulting
company. The software company signed a fix-cost bid with the consulting
company. The consulting company did all the work to make the integration
work. As a result, the software company met the budget for the project. One of
their architecture and project managers worked closely with consulting
company. Therefore, these internal persons learned the COTS components in
this project. They believed that they could save a lot of money in the following
project, which will use the same COTS component.
5.3.2 Scenarios of projects that used COTS component successfully. Twelve projects
successfully integrated the COTS components. The scenarios are summarized into
scenarios Sc4 to Sc6 in Table 5.
137
Appendix A
Table 5.The scenarios of 12 projects that used COTS component successfully

Scenairo Characterization Problems Good Practice
ID Encountered(PEs) (GP)
(Proj.ID)
Sc4 (P4 Two projects used a PE7: Under-estimated GP3: COTS
and P13) waterfall process.
learning effort. component
Made the make vs. buy PE8: Under-estimated selection was based
decision and selected the integration effort. on architecture.
COTS components in PE9: It was difficult GP4: Used COTS
the requirement phase. to isolate and prove component
newsgroup.
Selected familiar COTS bugs.
components. GP5: Prepared for
replacement.
Sc5 (P12) One project used GP5: Prepared for
waterfall with some replacement.
prototyping process.
Made the make vs. buy
decision and selected
COTS components in
the design phase.
Selected familiar COTS
components.
Sc6 (P3, Nine projects used PE10: Met standard GP3: COTS
P5, P6, incremental with mismatch. component
P7, P8, prototyping processes. PE11: Met selection was based
P9, P11, deployment mismatch. on architecture.
P15, and Some projects selected
P16) COTS components in GP5: Prepared for
the requirements phase replacement.
with familiar COTS GP6: Using COTS
components. components in the
Some projects selected incremental &
COTS components in prototyping process
the implementation helped the success
phase with familiar of the project.
COTS components GP7: Integrated
Some projects selected unfamiliar COTS
COTS component in component first.
the design phase with GP8: Limited the
unfamiliar COTS amount of
components. glueware.
GP9: Did not
change COTS code.
138
Appendix A
GP10: Did
integration testing
incrementally.
GP11: Paid vendor
incrementally.
GP12: Investigated
COTS component
market.
The detailed explanations of each problem encountered are following:

- PE7: In project P4, they used a COTS component which includes more than 500
different classes. It took longer time to read the document and to understand how
to use these classes than they originally estimated. As the COTS component
provided most functionality, they still believed that it shortened the time-to-
market of the system.
- PE8: In project P13, the project manager made a too optimistic estimation on
the effort for the COTS component integration. Although they have used these
COTS components before, it still took unexpectedly more time to integrate it
into the new project. However, they still thought the COTS component provided
positive effect on the time-to-market of the system.
- PE9: In project P13, they found some defects in the system during testing.
However, it was difficult to convince the vendor that these defects were inside
rather than outside the COTS components. The vendor took a long time to
rectify these defects.
- PE10: In projects P3 and P9, developers found that the COTS components did
not support the standard (such as EJB, DCOM) the vendor claimed. The COTS
supported some old version standard instead of the new one as they claimed.
- PE11: In project P8, the COTS components could not work in the first round of
integration testing. The COTS components failed because it could not work on
the deployed platform. The reason is that developers used different platforms in
the development and test.
To ensure the success of these projects, members of these projects performed some
other activities to reduce some possible negative effects of COTS components. Their
examples of good practices are summarized in the following:
- GP3: As the developers had experience with the relevant COTS components,
they knew how to integrate them and their limitations. In the COTS components
selection, they focused mainly on the ease of integration. If the COTS
components cannot provide all the required functionality, they built some
addware. In project P4, selecting COTS components based on the ease of
integration was highly recommended.
- GP4: In project P4, they not only consulted an internal expert, but also followed
the newsgroups of a COTS component on the Internet. Comments in the
newsgroups helped them to make the right selection.
- GP5: In project P13, they investigated the architecture of the system and the
COTS component to ensure the substitutability of the COTS component. This
139
Appendix A
means that they can easily change to other COTS components, if they are not
satisfied with the current COTS component. The same experience was also
emphasized by respondents in projects P11 and P12.
- GP6: Several projects started to select COTS components in the requirements
phase. This makes it simpler to match and possibly negotiate customers’
(volatile) requirements with the available functionality of the COTS
components, because the customer will accept the functionality of the COTS
components shown in the prototype. At the start of the prototyping process, the
main benefit of using COTS components is to build a prototype rapidly. They
could also learn and understand how to use the COTS components in building
such a prototype.
- GP7: In project P3, integrating unfamiliar COTS components was ranked as a
high risk task. These unfamiliar components had been integrated and tested
before other components.
- GP8: In project P3, one experience was that they tried to limit the amount of
glueware. They believed that the less glueware they need to produce, the easier
it would be to update in case a vendor provides new versions.
- GP9: In project P13, they had the source code of one COTS component.
However, they decided not to change the code because if the vendor released a
new version, it will be hard for the customer to maintain possible changes
himself.
- GP10: In some projects that used more than one COTS component, the
experience is that the COTS components should be integrated and tested
incrementally. It means that the integration test should be performed
immediately after one COTS component was integrated, instead of doing the
final integration testing. Without incremental integration testing, it may be
difficult to locate defects after all the COTS components are integrated.
- GP11: In project P7, the project paid part of the agreed amount after the
evaluation. As the project members discovered some limitations of the COTS
component, they asked the vendor to perform changes. After they got the
component as required, they paid the rest of the money to the vendor. They
believe that this allowed them to persuade the vendor to change the component.
- GP12: In project P15, they investigated the COTS component market share
before they selected it. They believed the vendor can survive for a long time if
their COTS products were used sufficiently in the market.
The studied projects showed that the COTS-based development could succeed under
different process scenarios. The answer to research question RQ3: COTS-based
development could be performed successfully using waterfall and evolutionary
processes. However, different process scenarios may face various risks.
6 Discussion
6.1 Comparison with related work
Results of this study confirmed some conclusions from previous studies and
contradicted others.
140
Appendix A
Boehm and Abts (1999) regard both the waterfall model and evolutionary
development as unsuitable for COTS-based development. The results from research
question RQ3 showed however that some projects could integrate COTS components
successfully, using waterfall or evolutionary processes. However, the results of RQ3
revealed that different process scenarios (Sc1 to Sc6) may have different risks. It is
therefore necessary to perform risk management according to customized process
scenarios, as proposed by Boehm and Abts (1999). In addition, results of RQ1 showed
that the main development processes were decided before the decision was made about
whether to use COTS components. That is, if the process is risk-driven as Boehm and
Abts (1999) proposed, developers should use non-COTS related risks to decide the main
development processes first. They should (only) start to consider COTS related risks,
after they decide to use COTS components.
Both EPIC (Albert and Brownsword, 2002) and the process proposed by Morisio et
al. (2000) indicated that some new activities and roles should be added to traditional
development processes. Examples are the make vs. buy decision, COTS components
selection, COTS components integration, and COTS component knowledge
management. The results of RQ2 gave further support to their conclusions. This study
shows that there are several possible variations in COTS-based development process,
for example, the COTS components were selected not only in the requirement phase,
but also in the design or coding phases.
Although the formal decision-making algorithms, such as MAUT (MacCrimmon,
1973), MCDA (Morisio and Tsoukias, 1997), or AHP (Saaty, 1990), was used in some
previous studies, such as OTSO (Kontio, 1996), CAP (Ochs et al., 2001), and CISD
(Tran et al., 1997). The results of RQ2 showed that none of the studied projects used
these formal decision-making algorithms. The results support findings (named thesis
T4) of Torchiano and Morisio (2004). The possible reason is that these techniques may
not be applicable in selecting COTS (Ncube and Dean, 2002; Maiden et al., 2002). In
practice, if developers had enough previous experience or tacit knowledge with specific
components, COTS component selection was based on their experience (see SP_fam in
Section 5.2). If they had no previous experience with the selected COTS components,
the selection was based on hands-on trials (see SP_unfam in Section 5.2). A handful of
candidate COTS components were compared very quickly with some key factors, for
example price, platform, and vendor reputations. Two or three COTS components were
selected for further hands-on trials.
Although methods to change or negotiate the requirements, such as MBASE
(Boehm, 2000), PORE (Maiden and Ncube, 1998), IIDA (Fox et al., 1997), CARE
(Chung and Cooper, 2002), RCPEP (Lawlis et al., 2001), and CRE (Alves and
Finkelstein, 2003), have been proposed to select COTS products. The results of RQ2
showed that these methods were not used in our studied projects. The possible reason is
that most COTS components being used in our studied projects are fine-grained COTS
components with limited functionalities (see Table 3). In case of using these fine-
grained COTS components, it is difficult for developers to negotiate the requirements
with the customer. As shown in PE3 (see Section 5.3.1), the customers can easily ask
the developers to build the required functionalities in-house instead of using a COTS
component, if the candidate COTS component cannot satisfy the customers’
requirements.
141
Appendix A
COTS-based software introduces many risks (Voas, 1998b). Some previous studies
summarized lessons learned and experiences by case studies (Boehm, et al., 2003;
COTS lessons learned repository, 2004; Tran et al., 1997; Fox et al., 1997; Rose, 2003;
Morris et al., 2003; FAA Guide, 2003; Kotonya and Rashid, 2001). The results of this
study gave further support to some of their conclusions. For example, faulty vendor
claims may result in feature loss and/or significant delays (Boehm, et al., 2003), and it is
not recommend changing the source code of the COTS component even if such a code
is available (COTS lessons learned repository, 2004; Morris et al., 2003). However, we
discovered some new salient problems encountered and examples of good practice, such
as incremental payment and preparation for replacement. In addition, previous studies
did not investigate the relationship between the risks and corresponding process
scenarios. Their lessons learned are proposed as general guidelines. It is therefore
difficult for practitioners to assess and evaluate the possible risks according to their
project contexts. The results of our study illustrated that some risks occur more
frequently in some development processes, such as waterfall process, than in other
development processes, such as incremental process.
6.2 Customizing the COTS-based development process
Our results show that the COTS-based development process is not a totally new process.
It needs a customization (change or add new activities) on the traditional development
processes. Based on the results of this study, we propose how to design and customize a
COTS-based development process. As our studied projects mainly used waterfall and
evolutionary processes, our proposal focuses on these. The process design could include
two elements:
- First, decide the main development processes based on the project profile. The
main development process could be decided based on risk considerations such as
proposed by Boehm and Abts (1999), and based on the typical problems
discovered in this study.
- If the main development process is waterfall or evolutionary, the process could
be customized based on the actual main development process and project
members’ familiarity with relevant COTS components as following Figure 1. For
each customization, some possible risks must be identified and managed as we
discovered from this study.
6.2.1 Waterfall with unfamiliar COTS components. Scenarios Sc1 and Sc2 showed
that it is difficult to use an unfamiliar COTS component successfully if the general
process is waterfall. It will be difficult to find COTS components to fully satisfy the
requirements after the requirements and design have been decided upon. Therefore, our
recommendation is that the make vs. buy decision and the COTS components selection
should be performed before the implementation phase. The COTS components selection
process could be SP_unfam, and the ease of integration should be given much
consideration in the make vs. buy decision. If the candidate COTS components do not
appear to be easy to integrate, it is better to develop them in-house. Another
recommendation is to inform the customer; if possible, about using COTS components
after the make vs. buy decision. It may give the developers leeway to (re)negotiate the
requirements if later limitations of COTS components are found.
142
Appendix A
6.2.2 Waterfall with familiar COTS components. If the project members are quite
familiar with relevant COTS components, the make vs. buy decision and COTS
component selection could be performed in the requirement phase, because the
integrators know how to integrate them, and which functionality could be provided by
the COTS components. Our results showed that COTS components selection drives the
requirements to some extent.
6.2.3 Incremental & prototyping with unfamiliar COTS components. Here, the risk
of using unfamiliar COTS components is relatively low, compared to waterfall projects.
The integrator can select the COTS components based on the completeness of required
functionalities and build a prototype in a short time. The customers’ requirements can
be agreed upon by evaluating the prototype. For this customization, the
recommendation is to integrate the unfamiliar COTS components first, if there are
different phases to implement the system incrementally.
6.2.4 Incremental & prototyping with familiar COTS components. If the project
members have enough experience with relevant COTS components; they can easily
build a prototype of the system with the COTS component. In this case, our results
showed that buying COTS components had more advantages than building them in-
house. Here, the make vs. buy decision can be performed in the requirements and/or
design phase. If there are different phases to implement the system incrementally, they
do not need to integrate all COTS components upfront. In this case, COTS components
could be considered as in-house built components. The process could be the same as a
non-COTS one.
Waterfall with unfamiliar COTS component
Requirement Design Implementation

SP_unfam
Waterfall
Waterfall with familiar COTS component

SP_fam SP_fam
Incremental & prototyping with unfamiliar COTS component

SP_unfam SP_unfam
Incremental
& prototyping Incremental & prototyping with familiar COTS component

SP_fam SP_fam SP_fam
Fig. 1. Possible COTS-based development process customizations

SP_fam: Familiarity-based selection process
SP_unfam: Internet search, trial-based selection process
6.3 Threats to validity
6.3.1 Construct validity. In this study, most variables are taken directly, or with little
modification, from the existing literatures. To ensure construct validity, we did two
rounds of pre-tests on the interview guide in ten local IT companies. These companies
143
Appendix A
were selected based on convenience. For this, we wrote an introductory letter, sent by
email. This was followed up by a phone call to see if the company had a willing contact
person and a suitable COTS-based project. The interview guide (a MS-Word document)
was then sent to the respondents by email. The respondents filled in the questions in the
interview guide electronically, and gave free-text comments to each question and the
following alternatives. 10% of the questions and alternatives in the interview guide were
revised based on such comments.
The selected COTS component and further COTS component examples proposed
from respondents show that the definition of a COTS component was understood
correctly. Furthermore, the responses to the three meta-questions show that the
questions in the interview guide were understood consistently.
However, one possible threat to construct validity is our chosen success criteria for a
project successfully using COTS components, implying that the integrated COTS
components should contribute positively both to time-to-market and to the quality of the
final system. This may not apply for all kinds of projects as some projects might
emphasis only time-to-market, and others might emphasis only quality.
6.3.2 Internal validity. The first possible threat to internal validity is our
misunderstanding of respondents’ answers. Although two interviewers carried out the
interviews and there was only one in each interview, we taped all interviews. Listening
to the tape helped ensure correct interpretation of answers and comments. However,
having an independent (third) person to listen to the tape might increase data quality.
The second possible threat to internal validity is that people tend to talk about
examples of good practices instead of problems they encountered. As we had a
confidentiality agreement with the respondents, the respondents were encouraged to tell
the truth. The data in Tables 4 and 5 showed that 11 (out of 16) respondents stated at
least one problem encountered in their projects. However, the respondents might give
only the most salient problems they encountered and forget some minor problems
because we asked for data about past projects.
The third possible threat to the internal validity is that the lifecycle phases, such as
requirement, design, and implementation, may not be clearly separated in projects using
prototyping and incremental processes. We called back some respondents to ask
whether they had specific software architecture in mind before the make vs. buy
decision and COTS component selection. Their answers gave future clarification on
whether COTS components were selected in the requirements or design phases.
The fourth possible threat to the internal validity is that some projects might not
tailor the process because their processes already support COTS usage. So, we might
miss some necessary process improvements that have been performed before the project
start because we asked only salient process changes within the studied projects.
We selected 10 projects in 10 companies. However, we selected 2 projects in each of
the other 3 companies. The fifth possible threat is that they are the same instead of
different. However, data in Appendix A showed that all projects in the same company
are different:
- In the first company, one project used waterfall and another used incremental
process with prototyping. The project members were totally different, and the
COTS selection and strategy to integrate COTS were also different.
144
Appendix A
- In the second company, one project used waterfall and another used incremental
process with prototyping. The project members were totally different. The
architectures of the final systems were also different.
- In the third company, the first project was finished two years before the second
one started. The project members were totally different. The COTS selection
method and strategy were also different.
6.3.3 External validity. The primary threat to external validity is that the study is based
on few and possibly not typical companies in Norway. According to Norwegian
“Census Bureau” (SSB) (SSB, 2004), the mean value of employee number in
Norwegian IT companies is 6. The first possible external validity to this study is that our
selected companies are mostly medium and large sized. However, the selected projects
covered several application domains.
The second possible threat to external validity is that the size (based on person-
hours) of the projects might bias the conclusion of this study. We have divided the
projects into three groups: small (less than 10,000 person-hours), medium (between
10,000 and 100,000 person-hours), and large (more than 100,000 person-hours). We
found out that the actual process used, the problems encountered, and the good practice
crossed small, medium, and large projects.
The third possible threat to external validity is that most COTS components are well-
defined and provided only secondary (support) functionality of the delivered system.
The process customizations may be different if the COTS components are intended as
the core part of the system, or the COTS components are large packages, such as ERP
and content management systems.
7 Conclusions and future work
This paper has presented an exploratory study of COTS-based development processes in

16 software projects in 13 Norwegian IT companies. We conclude that using COTS
components can be done as part of traditional development processes (e.g. waterfall and
evolutionary) – there is no special “COTS-based development process”. However,
successful use of COTS components in such processes requires that some new activities
and roles are introduced in order to reduce risks. Typical new activities are the make vs.
buy decision, COTS component selection, and COTS component integration. A new
role is that of a knowledge keeper. Two of the new activities, the make vs. buy decision
and the COTS component selection, can be placed in different development phases
(requirements, design, or implementation). This depends on project context, especially
on the tacit familiarity with possible COTS components and the flexibility of
requirements.
Although there were several different COTS component selection processes
proposed from academia, we discovered that two COTS component selection processes
were popular in practice. The first one is familiarity-based selection process. If the
project members or external experts had enough previous experience with the candidate
COTS component, their preference will decide which COTS component will be used in
the new project. The second selection process is a process combining Internet searches
with hands-on trials. Although several formal and complex COTS selection processes
have been proposed, they were not used in this study. To reduce time-to-market,
145
Appendix A
industrial practitioners performed only the most necessary steps in COTS component
selection and made the final decision based on the results of the hands-on trial.
A set of explanatory scenarios, encountered problems, and examples of good
practices have been synthesized to assist in the customization of the traditional
development processes with the new activities and roles. Although some of these
problems and examples of good practice have been mentioned in other previous studies,
we discovered several new ones, which decided the success or failure of the studied
projects and deserve to be investigated further.
The main limitation of this research is that it is based only on software development
projects in Norway, and that the sample size is small and perhaps not representative in
profile for a typical COTS-based project. In the future, we will investigate further the
actual COTS-based development processes. Although we proposed some possible
customization of COTS-based development processes based on the results of this study,
more samples are needed to validate them. The validation will focus on when, how, and
why the new activities (make vs. buy decision, and COTS component selection) are
performed.
We will also do a further study on systematic risk management in COTS-based
development. The different process scenarios may meet different risks and need
different risk mitigation methods. To manage risks in COTS-based development, it is
necessary to identify risks, evaluate them and design risk mitigation strategies.
Although several risk management methods have been proposed by case studies, there
are few studies to verify these proposals and investigate how to use them in different
process scenarios.
We are now formulating more explicit research hypotheses based on the findings of
this study and the lessons learned. We plan to design a web-based questionnaire and use
this to perform a quantitative survey in a large sample in Norwegian IT companies, in
parallel with replicating the study in Germany and Italy.
8 Acknowledgements
This study is supported by the INCO, SPIKE, and FAMILIES projects. Comments from
our colleague Tore Dybå gave valuable input to this study. We thank the colleagues in
these projects, and all the participants in this study.
Appendix A: Detailed information on process customization
Com Proj. Succ. or Main Other activities New role

p. ID ID failed process Added added
C1 P1 Failed Waterfall Buy vs. build. External
Selection. expert:
Build glueware & Hired one
addware. person from
the vendor
as
consultant.
C1 P2 Failed Incremental Customer decided
& COTS.
prototyping Learning.
146
Appendix A
Build glueware.
C2 P3 Succ. Incremental Buy vs. build. Internal
& Selection. expert.
Build glueware &
addware.
C2 P4 Succ. Waterfall Buy vs. build. Internal
Selection. expert.
Learning.
Build glueware &
addware.
C3 P5 Succ. Incremental Buy vs. build.
& Selection.
Build glueware.
& No selection as it
prototyping was the only
alternative.
Learning.
Build glueware.
& Selection.
Build glueware &
addware.
prototyping Build addware.
C6 P9 Succ. Incremental Without make vs. Internal
& buy decision expert.
prototyping because using
COTS is a
company strategy.
Selection.
Learning.
Build glueware &
addware.
C7 P10 Failed Waterfall Buy vs. build. External
Selection. expert: The
Build glueware integration
Learning. was
outsourced
to a
consulting
company.
147
Appendix A
& Selection. expert to

prototyping Learning. follow
Build glueware. COTS
evolution.
C9 P12 Succ. Waterfall & Buy vs. build.
prototyping Selection.
Learning.
Build glueware.
C10 P13 Succ. Waterfall Buy vs. build Internal
Selection. expert.
Learning.
Build glueware.
C11 P14 Failed Waterfall Buy vs. build.
Selection.
Learning.
Build glueware.
Build glueware.
Build glueware.
Appendix B: Detailed information on COTS component selection
Proj. COTS Selection process In which phase was

ID ID COTS selected
P1 1 Demo (recommended from Implementation
external expert).
P1 2 Demo (recommended from Implementation
external expert).
P2 3 Decided by customer Design
P3 4 Experience Design
P3 5 Internet search with pilot Requirements
P4 7 Internet search with pilot Requirements
P4 8 Experience Requirements
P5 10 Experience Implementation
P5 11 Internet search with pilot Implementation
P7 13 Internet search with pilot Design
148
Appendix A
P10 19 Demo. Design
P10 20 Demo. Design
P10 21 Demo. Design
P14 26 Demo. Implementation
P15 28 Recommended from external Requirements
expert
References
Abts, C., Boehm, B. W., and Clark, E. B. 2000. COCOTS: A COTS Software
Integration Cost Model - Model Overview and Preliminary Data Findings. Proc. of
the 11th ESCOM Conf, Munich, Germany, 325 -- 333.
Albert, C. and Brownsword L. 2002. Evolutionary Process for Integrating COTS-Based
System (EPIC): An Overview. SEI, Pittsburgh, Available at:
Alves, C. and Fineklstein, A. 2003. Investigating Conflicts in COTS Decision-Making.
Journal of Software Engineering and Knowledge Engineering 13(5):473 – 493.
Basili, V.R. and Boehm, B. W. 2001. COTS-Based System Top 10 Lists. IEEE
Computer, 34(5):91 -- 93.
Boehm, B.W. and Abts, C. 1999. COTS integration: Plug and Pray? IEEE Computer,
32(1):135 -- 138.
Boehm, B. W., Port, D., Yang, Y., and Bhuta, J. 2003. Not All CBS Are Created
Equally COTS-intensive Project Types. Proc. of the 2nd Int. Conf. on COTS-Based
Software Systems (ICCBSS’03), Ottawa, Canada, Springer, LNCS Vol. 2580, 36 --
50.
Boehm, B. 2000. Requirements that Handle IKIWISI, COTS, and Rapid Change. IEEE
Computer 33(7):99 --102.
Brownsword, L., Oberndorf, T. and Sledge, C. 2000. Developing New Processes for
COTS-Based Systems. IEEE Software, 17(4):48 -- 55.
Carney, D. and Long, F. 2001. What do You Mean by COTS? Finally, a Useful Answer.
IEEE Software, 7(2):83 -- 86.
Chung, L. and Cooper, K. 2002. A Knowledge-Based COTS-Aware Requirements
Engineering Approach. Proc. of the 14th Int. Conf. on Software Engineering and
Knowledge Engineering, Ischia, Italy, 175 – 182.
ComponentSource. 2004. Available at: http://www.componentsource.com/.
COTS Lessons Learned Repository. 2004. Available at: http://fc-
md.umd.edu/ll/index.asp
149
Appendix A
Crnkovic, I., Hnich, B., Jonsson, T., and Kiziltan, Z. 2002. Specification,
Implementation, and Deployment of Components. Communication of the ACM,
45(19):35 -- 40.
FAA Guide. 2003. Federal Aviation Administration, Software Engineering Resource
Center. FAA COTS Risk Mitigation Guide: Practical Methods for Effective COTS
Acquisition and Life Cycle Support. Available at:
http://www.faa.gov/aua/resources/cots/Guide/CRMG.htm
FAMILIES project. 2003. Available at: http://www.esi.es/en/Projects/Families.
Fowler. F. J., Jr 2001. Survey Research Methods (Applied Social Research Methods).
3rd edition, Sage Publications.
Fox, G., Lantner, K., and Marcom, S. 1997. A Software Development Process for
COTS-based Information System Infrastructure. Proc. of the 5th Int. Symposium on
Assessment of Software Tools (SAST '97), Pittsburgh, PA, USA, 133 – 142.
INCO project. 2000. Available at: http://www.ifi.uio.no/~isu/INCO/.
Kontio, J. 1996. A Case Study in Applying a Systematic Method for COTS Selection.
Proc. of the 18th Int. Conf. on Software Engineering, Berlin, Germany, 201 – 209.
Kotonya, G. and Rashid, A. 2001. A Strategy for Managing Risk in Component-based
Software Development. Proc. of the 27th EUROMICRO Conference, Warsaw,
Poland, 12 -- 21.
Lawlis, P. K., Mark, K. E., Thomas, D.A., and Courtheyn, T. 2001. A Formal Process
for Evaluating COTS Software Products. IEEE Computer, 34(5):58 -- 63.
Leung, K. R. P. H. and Leung, H. K. N. 2002. On the Efficiency of Domain-Based
COTS product Selection Method. Journal of Information and Software Technology
44(12):703 --715.
MacCrimmon, K. R. 1973. An Overview of Multiple Objective Decision Making. Proc.
of the Multiple Criteria Decision Making, University of South Carolina Press, 18 --
44.
Maiden, N. A. M., Kim, H., and Ncube, C. 2002. Rethinking Process Guidance for
Selecting Software Components. Proc. of the 1st Int. Conf. on COTS-Based Software
Systems (ICCBSS’02), Orlando, FL, USA, Springer, LNCS VOL. 2255, 151 -- 164.
Maiden, N. A. M, and Ncube, C. 1998. Acquiring COTS Software Selection
Requirement. IEEE Software 15 (2):46-56.
Morisio, M. and Tsoukias, A. 1997. IusWare: a Methodology for the Evaluation and
Selection of Software Products. IEE Proceedings – Software Engineering, 144(3):162
-- 174.
Morisio, M., Seaman, C.B., Parra, A. T., Basili, V. R., Kraft, S. E., and Condon, S. E.
2000. Investigating and Improving a COTS-Based Software Development Process.
Proc. of the 22nd Int. Conf. on Software Engineering, Limerick, Ireland, 31 -- 40.
Morris, E., Albert, C., and Brownsword, L. 2003. COTS-Based Development: Taking
the Pulse of a Project. Proc. of the 2nd Int. Conf. on COTS-Based Software Systems
(ICCBSS 2003), Ottawa, Canada, Springer LNCS, VOL. 2580,168 – 177.
NAICS.2004. Available at: http://www.revenue.state.ne.us/tax/current/buscodes.pdf.
Ncube, C. and Dean, J. C. 2002. The Limitation of Current Decision-Making
Techniques in the Procurement of COTS Software Components. Proc. of the 1st Int.
Conf. on COTS-Based Software Systems (ICCBSS’02), Orlando, FL, USA, Springer,
LNCS VOL. 2255, 176 -- 187.
150
Appendix A
Ochs, M., Pfahl, D., Diening, G. C., and Kolb, B. N. 2001. A Method for Efficient
Evaluation Results. Proc. of the 7th Int. Software Metrics Symposium, London,
England, 285 – 297.
Robson, C. 2002. Real World Research: A Resource for Social Scientists and
Practitioner-researchers (Regional Surveys of the World), 2nd edition, Blackwell
Publishers.
Rose, L. C. 2003. Risk Management of COTS Based Systems Development. In:
Component-Based Software Quality, Springer LNCS, VOL. 2693, 352 -- 373.
Saaty, T. L. 1990. How to make a decision: The Analytic Hierarchy Process (AHP).
European Journal of Operational Research, 48(1): 9 -- 26.
SPIKE project. 2002. Available at: http://www.idi.ntnu.no/grupper/su/spike.html.
SEI (Software Engineering Institute). 2004. COTS-Based Initiative Description.
Available at http://www.sei.cmu.edu/cbs/cbs_description.html.
SSB: Norwegian Census Bureau. 2004. Available at: http://www.ssb.no
Strauss, A. and Corbin, J. M. 1998. Basics of Qualitative Research: Grounded Theory
Procedures and Techniques. 2nd edition, Sage Publications.
Torchiano, M. and Morisio. M. 2004. Overlooked Aspects on COTS-based
Development. IEEE Software, 21(2):88 -- 93.
Tran, V., Liu, D. B., and Hummel, B. 1997. Component-Based Systems Development:
Challenges and Lessons Learned. Proc. of the 8th IEEE Int. Workshop on Software
Technology and Engineering Practice, London, UK, 452 -- 462.
Voas, J. 1998a. COTS Software – the Economical Choice? IEEE Software, 15(2):16 --
19.
Voas, J. 1998b. The challenges of Using COTS Software in Component-Based
Development. IEEE Computer, 31(6):44 -- 45.
151
Appendix A
152
Appendix A
P3. Validation of Theses on OTS-Based Development

Validation of New Theses on Off-The-Shelf Component Based
Development
Jingyue Li1, Reidar Conradi1,2, Odd Petter N. Slyngstad1, Christian Bunse3, Umair
Khan3,
Marco Torchiano4 and Maurizio Morisio4
1
{jingyue, conradi, oslyngst}@idi.ntnu.no
2
Simula Research Laboratory, P.O.BOX 134, NO-1325 Lysaker, Norway
3
Fraunhofer IESE, Sauerwiesen 6,
D- 67661 Kaiserslautern, Germany
{Christian.Bunse, khan}@iese.fraunhofer.de
4
Dip. Automatica e Informatica, Politecnico di Torino
Corso Duca degli Abruzzi, 24, I-10129 Torino, Italy
{maurizio.morisio, marco.torchiano}@polito.it
Abstract: Using OTS (Off-The-Shelf) components in software development has

become increasingly popular in the IT industry. OTS components can be either COTS
(Commercial-Off-The-Shelf), or OSS (Open-Source-Software) components. A recent
study with seven structured interviews concluded with six theses, which contradicted
widely accepted (or simply undisputed) insight. Since the sample size of that study was
very small, it is necessary to investigate these theses in a larger and randomized sample.
A state-of-the-practice survey in three countries – Norway, Italy, and Germany – has
been performed to validate these new theses. Data from 133 OTS component-based
projects has been collected. Results of this survey support four and contradict two of the
initial theses. The supported theses are: OSS components were mainly used without
modification in practice; custom code mainly provided additional functionality; formal
OTS selection processes were seldom used; OTS component users managed to get
required changes from vendors. The unsupported theses are: standard mismatches were
more frequent than architecture mismatches; OTS components were mainly selected
based on architecture compliance instead of function completeness.
1. Introduction
OTS components (Off-The-Shelf) includes COTS (Commercial-Off-The-Shelf) and

OSS (Open Source Software) components. Developing with the OTS components is
gaining more attention from both the research and industrial community. A newly
performed study, in the form of seven structured interviews, proposed six new theses in
OTS based development [12]. Since the sample size of that study was small, it is
necessary to investigate the six theses in a larger and randomized sample. We therefore
designed a state-of-the-practice study on OTS component-based development to
validate these theses. The survey was performed in three European countries (Norway,
153
Appendix A
Italy, and Germany). We have gathered answer from 133 projects using either COTS or
OSS components from a sample of 1500 companies.
Data of this survey support four of the six new theses and contradict the other two.
How to interpret the survey result is also discussed in detail in order to give
recommendations and suggestions on OTS based development.
The rest of this paper is organized as follows: Section 2 introduces the six theses we
intend to investigate. Section 3 presents the research questions and survey design.
Section 4 presents the actual sample. Section 5 lists results of each research question.
Detailed discussion on the survey results is given in Section 6. Conclusions and future
work are presented in Section 7.
2. Six new theses on OTS-based development
After performing seven structured interviews of small and medium software companies
in Norway and Italy, Torchiano and Morisio summarized six new theses which
contradicted statements or implicit assumptions from previous literatures [12].
2.1 New Thesis T1: Open source software is often used as closed source.
In previous literatures, OSS components are usually considered to be completely

different from closed-source components [6]. OSS products are provided by open
source communities with source code available. COTS components are provided by a
commercial vendor. The source code of COTS component is not available in most
cases. Torchiano and Morisio discovered that developers didn’t look and change the
source code of OSS components. The assumed explanation is that the users didn’t need
to see and modify, or they lacked the knowledge, skill, or resources to do so [12].
2.2 New thesis T2: Integration problems result from lack of compliance with
standards. Architecture mismatches constitute a secondary issue.
In previous literature, architectural mismatches were cited as one of the primary sources
of integration problems [5, 13]. However, Torchiano and Morisio did not find any
instances of this. The integration problem in their interviewed was that the selected
components did not fully or correctly support the claimed standards [12].
2.3 New thesis T3: Custom code mainly provides additional functionalities.
The previous literature greatly emphasized the integration aspects of custom code
(named glueware or gluecode) [3, 9]. However, Torchiano and Morisio concluded that
one of the most common remedial solutions among interviewees was to add code
(called addware) to provide missing functionalities [12].
2.4 New Thesis T4: Developers seldom use formal selection procedures. Familiarity
with either the product or the generic architecture is the leading factor in selection.
Previous studies have proposed several structured, formal, or semi-formal selection

procedures [1, 7]. However, no project interviewed by Torchiano and Morisio used any
of them.
154
Appendix A
2.5 New thesis T5: Architecture is more important than requirements for product
selection.
For COTS component selection, previous literature proposes selection processes mainly
based on requirements and their (re)negotiation [2, 7]. However, Torchiano and Morisio
discovered that the architecture drove the selection process, and was the fulcrum of all
trade-off activities [12].
2.6 New thesis T6: Integrators tend to influence the vendor on product evolution
whenever possible.
Literature generally describes the lack of control over the features and evolution of
COTS component as an immutable condition that integrators must tolerate [3].
However, Torchiano and Morisio found out that their interviewees were acting to
modify this condition [12].
3. Research design
To test the six new theses, we decided to collect more data using a randomized sample.
Our initial plan was to transfer the theses directly into hypotheses and make a
quantitative survey with standardized questions in a formal questionnaire. We designed
six fixed hypotheses according to the new theses and a preliminary questionnaire. The
process was as following:
- Jan.-June 2003: Read and discuss about the topic.

- June-Aug. 2003: Make initial research questions and draft questionnaire (in
English).
- Late Aug. 2003: first industrial pre-test in five local IT companies and with three
internal colleagues.
- Sep.-Oct. 2003: Revision of the draft questionnaire.
- Late Oct. 2003: Second pre-test in five companies.
After these two rounds of pre-testing, we discovered that it was difficult to reliably
investigate the six theses without a pre-study to clarify our research questions. For
example, we asked respondents whether they had influenced the vendors on product
evolution. The question was intent to investigate T6. After the pre-tests, we concluded
that it was not possible to get reliable results on this question. Some respondents had
pushed the vendor to change a COTS component concerning bugs fixing or new
functionalities. Although these respondents could tell whether they had got the required
changes or not, they were not sure whether they had influenced the evolution of the
COTS component.
3.1. A pre-study to clarify research questions
To clarify the research questions, we decided to resort to structured interviews, i.e. a

qualitative pre-study, before using a larger and randomized sample. The pre-study was
designed to employee more open research questions, not fixed hypotheses, and with a
155
Appendix A
rather detailed interview guide. We used a convenience sample, i.e. a list of large,
medium and small IT companies from existing project contacts. We ended up with 16
personal interviews in 13 Norwegian IT companies with an initial selection of 34
companies. The results of the pre-study were reported in [8].
3.2. Clarified research questions
After the pre-study, we were able to make testable research questions to investigate each
new thesis. Although the research questions might not give complete answers to each
thesis, they were designed to investigate each thesis in a reliable way.
3.2.1. Research questions for thesis T1. T1 claims that OSS components are being
used as COTS components in practice. From the pre-study, we discovered that we had
to interpret the word “used” correctly. The word “used” in [12] means that the source
code of OSS component was seldom read and changed. OSS components were therefore
“used” as “black-box”. As the meaning of “used” in T1 includes two activities, read and
change, we designed two research questions T1.RQ1 and T1.RQ2 to study T1.
T1.RQ1: Had the source code of OSS components been read by integrators
T1.RQ2: Had the source code of OSS components been changed by integrators
3.2.2. Research questions for thesis T2. T2 indicates that standard mismatches were
the most frequent problem in COTS based development, and the architecture
mismatches were a secondary issue. From the pre-study, we concluded that it was
difficult to conclude that the architecture mismatches constituted a second issue.
Although we can answer whether standard mismatches were more frequent than
architecture mismatches, some other mismatches might be more frequent than these
two. As a result, we limited our research question to compare the frequency of standard
mismatches and architecture mismatches, as shown in T2.RQ1.
T2.RQ1: Were the standard mismatches more frequent than architecture

mismatches?
3.2.3. Research questions for thesis T3. T3 argues that most custom code was used to
provide additional functionalities rather than solving mismatches between components,
or between components and application. In one word, T3 concludes that more addware
(code to add missing functionalities) was built than glueware (code to solve various
mismatches) in COTS-based development. In the pre-study, we asked the respondents
to give the exact LOC (Line-Of-Code) of glueware and addware they developed in the
project. However, most respondents could not give us the exact LOC of
glueware/addware. Some respondents claimed that this information was confidential.
Others complained that it was time-consuming to calculate this number. To avoid
getting wrong or incomplete data, we limited the research question to investigate the
156
Appendix A
occurrences of building glueware and addware instead of comparing the exact LOC. So,
the research question T3.RQ1 is:
T3.RQ1: Was the occurrences of building glueware more frequent than the
occurrences of building addware?
3.2.4. Research question for thesis T4. T4 concludes that formal selection procedures,
which generally come from the academia, were seldom used in practice. T4 also
concludes that familiarity with either the product or the generic architecture was used as
the leading factor in COTS component selection.
From the pre-study, we discovered that the definition of “formal selection procedure”
in [12] was too general, because there are several different kinds of formal selection
methods. After detailed analysis, we concluded the formal process defined in [12] is a
kind of process includes three basic elements: selecting evaluation criteria (factors);
collecting and assigning values to these criteria; making decisions using formal decision
making methods. In the pre-study, we discovered two other selection processes were in
frequent use [8]:
- Familiarity-based selection process: If the project members had enough previous

experience on the candidate COTS components, this experience will be the key
factor in the COTS component selection.
- Hands-on experimentation based selection process: This kind of selection was
used when there was no previous experience available. This selection process
was more complex than familiarity-based selection process, and had three steps:
Step 1: First, developers used a search engine to browse the Internet, and used
keywords to express decided functionality. This usually gave them a handful of
candidates.
Step 2: If there were more than 2 to 3 possible candidates, they selected 2 to 3
of them very quickly based on some key issues, such as licence issues, cost, or
vendor reputation.
Step 3: After a small number of candidate COTS components had been
decided, they downloaded a demo version of these from the web and tested the
COTS components locally.
To test T4, we decided to compare how often these three kinds of selection processes
had been used in practice. So, the corresponding three research questions are:
T4.RQ1: How often had a formal process been used in practice?
T4.RQ2: How often had the familiarity-based selection process been used in
practice?
T4.RQ3: How often had the hands-on experimentation based selection process
been used in practice?
3.2.5. Research questions for thesis T5. T5 claims that architecture was more
important than requirements for product selection. From the pre-study, we discovered
that the architecture decision affected most components in the system and there were
157
Appendix A
trade-offs between architecture and COTS component selection. Some COTS

components were selected to satisfy the required functionalities. Other COTS
components in the same project were selected because they could easily be integrated,
although they could not provide all required functionalities. Since different COTS
components in the same project might be selected according to different criteria, our
research question T5.RQ1 investigate which criteria were more important for the whole
project rather than just for a specific COTS component.
T5.RQ1: Had architectural compliance been considered more important than

functionality completeness in the whole COTS-based project?
3.2.6. Research questions for thesis T6. T6 argues that COTS users were not
completely passive in the COTS market and they managed to influence the vendor on
product evolution whenever possible. As mentioned in above, it was difficult to know
whether the COTS users actually influenced COTS component evolutions or not, even
if it was possible to know whether the required changes (corrections or new
functionalities) from the COTS users had been satisfied. To get a reliable conclusion on
T6, we limited our research question T6.RQ1 to evaluate how often the required
changes had been implemented by the vendors.
T6.RQ1: How often the required changes (corrections or new functionalities)

from integrators had been satisfied by the vendors.
3.3. Questionnaire design
After we clarified the research questions based on results from the pre-study, we
designed a new questionnaire. To validate thesis T1, several questions related to OSS
components were also included.
The questions in the questionnaire were organized into three main parts:
- Questions to collect background information of the company, project, and

respondents.
- Questions to get answers of the research questions mentioned in Section 3.2
(Detailed information of each question is introduced in the Section 5). Some
other questions to investigate process improvement and risk management were
also included. However, answers to these questions are out of the scope of this
paper.
- Questions to collect information about OTS components actually used in the
project. In the questionnaire, we asked the respondents to give brief information
(name, COTS/OSS, version) of all OTS used and detailed information of one of
the most important COTS/OSS component (the one provides most functionality
compared to other OTS components). For the selected most important
COTS/OSS component, we gathered detailed background information, such as:
• Name, version, and functionality

• How large a percentage of the application functionality in the whole system
is provided by the selected OTS component?
• Whether the source code is available or not?
158
Appendix A
• What was the technical base, such as Library, COM; CORBA, EJB etc.
Concepts used in this survey are listed in the first page of the questionnaire. In
general, the concepts are consistent with definitions proposed by Torchiano and Morisio
[12]. The difference is that we extended the definition to involve OSS component. The
key concepts are:
- Component: Software components are program units of independent

production, acquisition, and deployment and which can be composed into a
functioning system. We limit ourselves to components that have been
explicitly decided either to be built from scratch or to be acquired externally
as an OTS-component. That is, to components that are not shipped with the
operating system, not provided by the development environment, and not
included in any pre-existing platform.
- An OTS component is a component provided (by a so-called provider) from
a commercial vendor or the Open Source community. An OTS component
may come with certain obligations, e.g. payment or licensing terms. An OTS
component is not controllable, in terms of provided features and their
evolution.
The final questionnaire was first designed and pre-tested in English (internal and
external reviews). It was then translated into the native languages and published on the
SESE web survey tool [11] at Simula Research Lab in OSLO.
3.4. Sample definition and selection
We define the unit of this study as a completed software development project, and its
OTS-relevant properties. The projects were selected based on two criteria:
Projects were collected randomly from IT companies in Norway, Italy and Germany.
Slightly different sampling procedures were used in each country due to limited
resources. Detailed discussions on sample selection in this study are reported in [4].

(SSB) [10]. We included mainly companies which were registered as IT
companies. Based on the number of employees, we selected the 115 largest IT
companies (100 IT companies and 15 IT departments in the largest 3 companies
in 5 other sectors), 200 medium-sized software companies (20-99 employees),
and 100 small companies (5-19 employees) as the original contacting list.
then randomly selected companies from them. For these randomly selected
159
Appendix A

3.5. Data collection procedure.
To avoid bias in the data collection procedure, we made a guideline on how to contact
respondents before the survey started. All countries – Norway, Italy, and Germany –
followed the process in the guideline.
Possible respondents were contacted first by telephone. If the candidate respondents
had suitable OTS-based projects and would like to join our study, a username and
password of the survey tool and an electric version of the questionnaire were sent to
them. The respondents could use either the SESE web tool or the electronic version to
fill in the questionnaire. The respondents who didn’t want to answer the questionnaire
were also registered. We logged the main reasons of non-response, such as “no software
development”, “no OTS-based projects”, and “busy”. Most respondents used the Web
tool to fill in the questionnaire. However, phone interview were used in Germany
because of the confidential concerns in some companies [4].
3.6. Data analysis method
We used different analysis methods to analyse different research questions.
- Thesis T1 is relevant to OSS components. To study research questions T1.R1

and T1.R2, we compared the usage of OSS components with COTS
components.
- Because T2, T3, T4, and T6 are not relevant to OSS components, only
information from COTS components was analysed.
- As we mentioned in section 3.2.5, thesis T5 should be tested using data from the
whole project rather than a specific COTS component. The unit to be analysed
for research question T5.RQ1 is therefore the whole project instead of the
selected OTS component.
4. Selected sample
4.1. Selected companies
38 from Italy) from 127 companies. In general, we selected one project from each
company. However, we selected more than one projects in three Norwegian IT
share more experience to this study.
employees) and large (100 or more employees) ones as shown in Figure 1. The main
business areas of the sampled companies also cover different types as shown in Figure
2.
160
Appendix A
Figure 1. Distribution of the company size
Figure 2. Distribution of the company’s main business area
4.2. Selected projects
The final systems produced by the 133 projects cover different application domains as
shown in the Figure 3.
Figure 3. Distribution of application domain of the systems produced by these

projects
161
Appendix A
components, and six used both COTS and OSS components. For the 44 projects using
only OSS components, each of them gave the detailed information of an OSS
component. For the 83 projects using only COTS components, 83 different COTS
components were selected. For the six projects used both COTS and OSS components,
the respondents selected to give detailed information of five COTS components and one
OSS component. In total, we got detailed information of 88 COTS components and 45
OSS components.
4.3. Respondents
Most respondents of the 133 projects have a solid IT background. More than 90% of
them are IT managers, project managers, or software architects. Most of them have
more than 2 year experiences with OTS-based development. All of them have at least a
bachelor degree in informatics, computer science, or telecommunication.
5. Answers to research questions
5.1. Answers to T1.RQ1 and T1.RQ2
T1.RQ1 and T1.RQ2 is to test thesis T1, i.e. whether the OSS components had been
used in the same way as COTS components. To investigate T1.RQ1, the corresponding
question in the questionnaire is:
Have you read parts of the source code of OTS component?
We used a five-point Likert scale to measure the answer. Respondents were asked to
answer “very little”, “little”, “some”, “much”, “very much”, or “don’t know”.
For question T2.RQ2, we used the same measurement scale as for T1.RQ1, and the
corresponding question is:
Have you modified parts of the source code of OTS component?
We assigned an ordinal number from 1 to 5 to the above alternatives (5 meaning

“very much”). Results of these two research questions are summarized in Figure 4. The
results show that the median value of reading OSS code is 3 (meaning “some”).
However, the source code had seldom been changed as the median value of changing
the OSS code is 2 (meaning “little”). Our future analysis discovered that 46% of the
respondents claimed that they modified “very little” parts of the source code in OSS
components.
Another question in the questionnaire asked if the source code of OTS components is
available or not. Results show that not all COTS components in this study were “black-
box”. 29 (out of 88) selected COTS components provided source code to their users. To
compare with OSS components, we investigated whether the source code in “white-
box” COTS components have been read or modified. Results are shown in Figure 5.
162
Appendix A
0
N = 44 44
RE A D C O DE C HA NGE C O DE
Figure 4. The source code usage of the OSS components

6
0
N = 26 27
RE A D C O DE C HA NGE C O DE
Figure 5. The source code usage of the “white-box” COTS components
From Figure 4 and Figure 5, we can see that the source code of COTS or OSS
components had been read to some degree, if it was available. However, the source code
in both COTS and OSS components had seldom been changed.
5.2. Answers to T2.RQ1
T2.RQ1 asks whether the architecture mismatches were more frequent than standard
mismatches. To investigate T2.RQ1, the corresponding question is:
Did you encounter some of the following aspects (risks) with the selected OTS
component?
Standard mismatch and architecture mismatch were listed as option a) and b) of this
question:
a) The OTS component did not follow industrial standards (COM, CORBA etc.) as
the provider claimed (i.e. glueware was needed)
b) The OTS component had mismatching architectural assumptions towards other
parts of the system (i.e. glueware was needed)
Respondents were asked to answer “yes”, “no”, or “don’t know”. For the selected 88
COTS components, only 4.5% of the COTS components had standard mismatches,
compared with 28.4% COTS components having architecture mismatches.
163
Appendix A
T3.RQ1 is to test thesis T3, i.e. to compare glueware vs. addware. To investigate
T3.RQ1, we asked respondents the same questions as in T2.RQ1. We listed another
option c):
c) The OTS component could not provide all required functionality (i.e. addware
needed)
Respondents were also expected to answer “yes”, “no”, or “don’t know”.

In data analysis, we added all the “yes” answers for a) and b) options of this question
to count the occurrences of building glueware. The rationale is that glueware will be
needed in case of either standard, or architecture mismatch. To count the occurrences of
building addware, we added all the “yes” answers for option c).
Results show that glueware was needed for 32.9% of the selected COTS components
and addware was needed for 65.9% of the selected COTS components.
5.4. Answers to T4.RQ1 to T4.RQ3
To answer T4.RQ1 to T4.RQ3, the corresponding question is:
Have you performed some of the following actions for the selected OTS component?
The listed options are a) to f) as follows:
- Concerning the identification of OTS components:

b) Got recommendation of possible OTS component candidates from the customer
c) Got recommendation of possible OTS component candidates from a local
colleague/OTS-expert
- Concerning the evaluation of components:

d) Used a formal decision-making method to compare possible OTS component
candidates, e.g. with weighted evaluation criteria
e) Limited possible candidates into 1-3 components, by reading literature or other
documentation
f) Did “hands-on” try-out of 1-3 components, e.g. on a down-loaded demo version.
Respondents were asked to answer “yes”, “no”, or “don’t know” for each option.
Results of the five options are listed in Table 1.
Data in Table 1 shows that one common criterion for identifying OTS components is
familiarity-based. 58% COTS components were selected according to local expert
(option c). Formal procedures (option d) were only used in evaluating 19% of the
selected COTS components. The most common evaluation method was hands-on
experimentation (option f). It was used to select 65% of the selected COTS components
in this study.
164
Appendix A
Table 1. Results of T4.RQ1 – T4.RQ3

Option Yes No Don’t
know
a) 59% 36% 5%
b) 24% 73% 3%
c) 58% 36% 6%
d) 19% 74% 7%
e) 48% 45% 7%
f) 65% 27% 8%
To answer T5.RQ1, the corresponding question is:
What actions were performed during development or maintenance of the project?
We listed several possible actions in the project. The action relevant to T5.RQ1 is:
OTS components were selected mainly based on architectural and standards
compliance, instead of expected functionality.
Respondents were asked to answer “don’t agree at all”, “hardly agree”, “agree
somewhat”, “agree mostly”, “strongly agree”, or “don’t know”. We assigned an ordinal
number from 1 to 5 to the above alternatives (5 meaning strongly agree). Results of the
83 projects using only COTS components are shown in Figure 6.
0
N = 82
A RC H. VS. F U NC .
Figure 6. Result of T5.RQ1
From Figure 6, we can see that the median value of this question is 2. It indicates that
most respondents hardly agree that COTS components were selected mainly based on
architectural and standards compliance. So, we concluded that expected functionality
still constituted the main concern in COTS components selection.
To investigate T6.RQ1, we asked the same question as T2.RQ1 with another option
e):
165
Appendix A
e) The provider did not want to or could not change OTS component functionality as
requested.
Results show that 60% of the selected COTS components have been changed by
vendors according to integrators’ requirements.
6. Discussion
6.1. Differences between COTS and OSS components
Our results support thesis T1. If “white-box” vs. “black-box” is regarded as the main
difference between COTS and OSS components, results of T1.RQ1 and T1.RQ2 show
that COTS and OSS component are not as different as people might assume. COTS
components were not always “black-box”, as 33% of the COTS components selected in
this study opened their source code to users.
If the source code is available, both COTS and OSS users tend to read the code.
However, both COTS and OSS component users did not change much of the available
source code. From this result, we can generally conclude that the OSS component was
used the same as COTS component, i.e. integrated without modification.
6.2. COTS component selection and integration
Our results support thesis T4. For COTS component selection, results of T4.RQ1 to
T4.RQ3 show that developers often choose a sub-optimal product only because they
already know it. Formal selection procedures try to find out the most optimal product.
However, these procedures might not be cost-effective in some industrial projects. As a
result, selection techniques should also include familiarity as a main criterion [12].
Results of T5.RQ1 do not support thesis T5. COTS components users believe that
functionality completeness is more important than architectural compliance in COTS
component selection. However, we need to interpret this result carefully. The result
does not claim that architectural and standard compliance are not important in COTS
component selection. One lesson learned from our pre-study was that easiness of
integration should be seriously considered before you decide to use a COTS component
[8]. The possible explanation for result of T5 is that all candidate COTS components
support required standards and architectures. So, the only comparable features are their
functionalities. Then, COTS components were appeared to be selected based on their
functionalities instead of architectural compliance.
For COTS component integration, results of T2.RQ1 contradict thesis T2, and show
that architecture mismatches were more frequent than standard mismatches. With the
maturity of COTS component standard, we believe that less standard mismatches will
occur in the future.
Results of T3.RQ1 support thesis T3. It means that missing or incomplete features in
COTS products raise significant problems. It is therefore important to develop effective
techniques that allow seamless extension of existing products [12].
166
Appendix A
6.3. Vendor relationship in COTS based development
Results of T6.RQ1 support thesis T6. It shows that COTS component users actually
pushed vendors to change the COTS components either because of bug fixes or new
features. However, this result does not imply that most COTS component users actually
influenced COTS component evolution. In general, we believe that the COTS
component evolution is decided by the vendors’ marketing strategy rather than the
requirements of a specific user, unless this user is very powerful.
6.4. Possible treats to validity
6.4.1. Construct validity. In this study, most variables and alternatives are taken
directly, or with little modification, from existing literature. The questionnaire was pre-
tested using a paper version by 10 internal experts and 8 industrial respondents before
being published on the SESE tool. About 15% questions have been revised based on
pre-test results. One possible threat to construct validity is that respondents from
different companies may have slight different understanding on the same concept, such
as software architecture and architectural mismatches.
6.4.2. Internal validity. We have promised respondents in this study a final report and a
seminar to share experience. The respondents were persons who wanted to share their
experience and wanted to learn from others. In general, we think that the respondents
answered the questionnaire truthfully. However, different persons in the same project
might have different opinions on the same project. Asking only one person in each
project might not be able to get the whole picture of the project. Due to length limitation
of a questionnaire, we asked the respondent to fill in information of only one component
in the project. The possible threat is that other OTS components in the same project
might give different answers to our questions.
6.4.3. Conclusion validity. This study is a state-of-the-practice study. We studied what

had happened in industrial projects. However, we did not investigate the cause-effect
relation of the phenomena discovered in this study.
6.4.4. External validity. We used different randomization to select samples in different

countries. However, the sample selection processes were not exactly the same due to
resource limitations [4]. Another possible threat to external validity is that our study
focused on fine-grained OTS components. Conclusions may be different in projects
using complex and large OTS packages, such as ERP, content management systems,
and web services in general.
7. Conclusion and future work
This paper has presented results of a state-of-the practice survey on OTS-based

development in industrial projects. The study has investigated six new theses in OTS
component based development. Results of this study gave support on four (T1, T3, T4,
and T6) new theses and concluded that:
167
Appendix A
- T1: Source code of OSS components was read frequently with very few
modifications. OSS components were de facto used as COTS components, even
if the source code was available.
- T3: In the COTS components integration process, more addware had been built
than glueware.
- T4: Formal selection procedures were seldom used to select COTS components.
Familiarity-based and hands-on experiment based selection process were
frequently used in practice.
- T6: In most cases, COTS users managed to get their required changes in the
COTS components from COTS vendors.
Results of this study contradict the two (T2 and T5) other theses and concluded that:
- T2: There were more architecture mismatches than standard mismatches in
practice.
- T5: Functionality completeness was regarded more important than architectural
compliance in COTS component selection.
As this study is a state-of-the-practice study, we could not give cause-effect

conclusions based on results of this survey. The next step is to do a further qualitative
study with structured interviews to find out the possible explanations for the conclusions
of this study.
8. Acknowledgements
This study was partially funded by the INCO (INcremental COmponent based
development) project [14]. We thank the colleagues in these projects, and all the
participants in the survey
9. References
[1] Albert, C. and Brownsword, L. “Evolutionary Process for Integrating COTS-Based

System (EPIC): An Overview,” SEI, Pittsburgh, 2002. Available at:
[2] Boehm, B., “Requirements That Handle IKIWISI, COTS, and Rapid Change,” IEEE
Computer, Vol. 33, No. 7, 2000, pp. 99–102.
[3] Boehm, B. and Abts, C., “COTS Integration: Plug and Pray?” IEEE Computer, Vol.
32, No. 1, 1999, pp. 135–138.
[4] Conradi, R., Li, J., Slyngstad, O. P. N., Bunse, C., Kampenes, V.B., Torchiano, M.,
and Morisio, M. “Reflections on conducting an international CBSE survey in ICT
industry,” submitted to 4th International Symposium on Empirical Software
Engineering, Nov. 2005, Noosa Heads, Australia, 11 pages.
[5] Egyed, A., Medvidovic, N., and Gacek, C., “Component-Based Perspective on
Software Mismatch Detection,” IEE Proc.-Software, Vol. 147, No. 6, 2000, pp. 225–
236.
[6] Lawton, G., “Open Source Security: Opportunity or Oxymoron?” IEEE Computer,
Vol. 35, No. 3, 2002, pp. 18–21.
168
Appendix A
[7] Lawlis, P., Mark, K. E., Thomas, D.A., Courtheyn, T. “A Formal Process for
Evaluating COTS Software Products,” IEEE Computer, Vol. 34, No. 5, 2001, pp. 58–
63.
[8] Li, J., Bjørnson, F. O., Conradi, R., and Kampenes, V. B., “An Empirical Study of
Variations in COTS-based Software Development Processes in Norwegian IT
Industry,” Proc. of the 10th IEEE International Metrics Symposium, Chicago, USA,
September, 2004, IEEE CS Press (2004), pp. 72-83.
[9] Morisio, M. et al., “Investigating and Improving a COTS-Based Software
Development Process,” Proc. of 22nd International Conference on Software
Engineering, Limerick, Ireland, June, 2000, ACM Press, pp. 32–41.
[10] Norwegian Census Bureau (2004): Available at: http://www.ssb.no
[11] SESE tool (2004), Available at: http://sese.simula.no
[12] Torchiano, M. and Morisio, M., “Overlooked Aspects on COTS-based
Development,” IEEE Software, Vol. 21, No. 2, 2004, pp. 88-93.
[13] Yakimovich, D., Bieman, J., and Basili, V., “Software Architecture Classification
for Estimating the Cost of COTS Integration,” Proc. 21st International Conference on
Software Engineering (ICSE 99), ACM Press, 1999, pp. 296–302.
[14] INCO Project (2000): INCO project description, Available at:
http://www.ifi.uio.no/~isu/INCO.
169
Appendix A
170
Appendix A
P4. SPI on OTS-Based Development Process

A State-of-the-Practice Survey of Off-the-Shelf Component-
Based Development Processes
Jingyue Li1, Marco Torchiano2, Reidar Conradi1,

Odd Petter N. Slyngstad1, and Christian Bunse3
1
2
marco.torchiano@polito.it
3
Fraunhofer IESE, Fraunhoferplatz 1,
D-67663 Kaiserslautern, Germany
Christian.Bunse@iese.fraunhofer.de
Abstract. To gain competitive advantages software organizations are forced to develop

systems quickly and cost-efficiently. Reusing components from third-party providers is
one key technology to reach these goals. These components, also known as OTS (Off-
the-Shelf) components, come in two different types: COTS (Commercial-Off-The-
Shelf) and OSS (Open–Source-Software) components. However, the reuse of pre-
fabricated components bears one major question: How to adapt development
processes/methods with refer to system development using OTS components. To
examine the state-of-the-practice in OTS component-based development a survey on
133 software projects in Norway, Italy and Germany was performed. The results show
that OTS-based development processes are typically variations of well-known process
models, such as the waterfall- or prototyping model, mixed with OTS-specific activities.
One reason might be that often the process is selected before the use of OTS
components is considered. Furthermore, the survey shows that the selection of OTS
components is based on two processes: “Familiarity-based” and “Internet search-based”.
Moreover, it appears that the lifecycle phase to select OTS components is significantly
correlated with a project members’ previous familiarity with possible OTS candidates.
Within this paper, we characterize the state-of-the-practice concerning OTS processes,
using seven scenarios, and discuss how to decide or modify such processes and how to
select OTS components.
1 Introduction
Software development with OTS components is becoming increasingly popular in

research and industrial communities. The use of OTS components introduces new
requirements, which again require revised development processes. Although researchers
and practitioners have been dealing with such processes quite a time, most studies are
based on military or aerospace projects [10], [15], or other large projects. To propose
171
Appendix A
and design cost-effective OTS-based development processes, it is necessary to

investigate how such projects are performed in different domains and project contexts.
Within a first, exploratory study, we investigated the commonalities and differences
between development processes in 16 COTS-based software projects in Norway [6].
The study summarized several variations in COTS-based development processes and
concluded that the customization of such development processes crucially depends on
the project context, such as familiarity with possible COTS components and flexibility
of requirements. Due to the small sample size it is necessary to verify these conclusions
with a larger and more representative sample.
The study presented in this paper investigated several conclusions about variations in
development processes, based on the exploratory study. The results show that the actual
OTS component-based development processes are typically variations of well-known
process models. In addition, two OTS component selection processes, such as
familiarity-based and Internet search-based, are widely used in practice. By
summarizing the state-of-the-practice of OTS component-based development processes
using seven scenarios, we give systematic proposals on how to adopt the OTS-based
development and OTS selection processes based on the project context.
The remainder of the paper is structured as follows: Section 2 presents previous
studies and their research questions. Section 3 presents selected samples and Section 4
presents the empirical results. Discussions on customizing OTS-based development
processes are given in Section 5 and Section 6 separately. Possible threats to validity are
discussed in Section 7. Conclusion and future research are in Section 8.
2 Related work and research questions
There is a consensus that the use of COTS components implies changes in the software
process [2]. Some studies focused on the whole software development lifecycle [1], [3],
and [10]. Others investigated the specific phase, especially in COTS component
selection and evaluation [5], [8], [9], [11], [12], [13], [14].
2.1 Process of the whole software development lifecycle
Boehm et al. [3] regard both the waterfall model and evolutionary development as
unsuitable for COTS-based development. In the waterfall model, requirements are
identified at an earlier stage and COTS components chosen at a later stage. This
increases the likelihood of COTS components not offering the required features.
Evolutionary development assumes that additional features can be added if required.
However, COTS components cannot be upgraded by one particular development team.
The frequent lack of code availability hinders developers to adapt them to their needs.
Therefore, Boehm et al. proposed that development models, which explicitly take risk
into account, are more suitable for COTS-based development than the traditional
waterfall or evolutionary approaches.
The National Aeronautic and Space Administration (NASA) has been developing
systems using COTS components for many years (see [10] for the summary of
experience made). Various processes, used across 15 projects, were examined and used
as a basis for a common COTS-based development process.
172
Appendix A
The Software Engineering Institute developed the Evolutionary Process for

Integrating COTS-based Systems (EPIC) [1]. EPIC integrates COTS component related
roles and activities into a RUP process. The iterative and evolutionary nature inherent in
EPIC allows developers to adjust the architecture and system design, as more
knowledge is gained about the operations of the COTS components.
Our investigation on COTS-based development process in Norwegian IT companies,
however, revealed that the main COTS-based development process is to customize the
traditional development process to account for use of COTS components [6]. In
addition, in all our investigated projects, the project members decided the main process
before they started to think about using COTS components. To verify our findings, we
needed more representative samples. Therefore, our first three research questions RQ1
to RQ3 are designed to examine the state-of-the-practice of the actual development
process in OTS-based projects.
- RQ1: What were the actual development processes in OTS-based projects?

- RQ2: Was the actual development process decided before the make vs. acquire
decision or after the make vs. acquire decision.
- RQ3: Who decided the actual development process?
2.2 COTS component selection and evaluation process
Based on case studies, researchers have proposed several COTS component selection
processes and methods. Some of the direct assessment processes, such as OTSO [5],
CAP [13], and CISD [8], assume that the requirements are fixed and select the COTS
components by comparing how well the COTS component candidates satisfy the
requirements. A formal decision-making process is usually used to select the “best”
COTS component [12]. The formal decision-making process usually includes three
basic elements: selecting evaluation criteria (factors), collecting and assigning values to
these criteria, and applying formal decision-making algorithms such as MAUT [9],
MCDA [11], and AHP [14]. However, both the study of Torchiano and Morisio [16]
and our exploratory study [6] showed that these formal selection methods were seldom
used. In fact, our exploratory study discovered two other popularly used selection
processes. One is familiarity-based selection process and the other is Internet search
with trial-based selection process. In addition, our exploratory study concluded that
there are common new activities and that the possible variations are when and how to
perform them, especially when and how to select COTS components. Therefore,
research questions RQ4 to RQ6 are designed to investigate how OTS components were
selected.
- RQ4: What was the actual selection process used?
- RQ5: When was the OTS component selected?
- RQ6: What was the relationship between the selection phase and the project
context, such as the familiarities with the OTS component candidate and the
importance of the OTS component candidate?
173
Appendix A
3. Questionnaire design and sample selection
The general questionnaire design, sample definition and selection, and data collection
procedures are reported in [4], [7]. This study extended the exploratory study in two
dimensions. First, it included OSS components because they represent an alternative to
COTS components. Second, this study included much larger samples from three
countries - Norway, Italy, and Germany. In addition, the sample was selected randomly
instead of by convenience as in the exploratory study.
We have gathered results from 133 projects (47 from Norway, 48 from Germany,
and 38 from Italy) from 127 companies. In general, we selected one project from each
share more experience to this study. In the selected 133 projects, 83 used only COTS
components, 44 used only OSS components, and six used both COTS and OSS
components. Profiles of collected companies and projects are reported in [7].
4. Empirical results
4.1. RQ1: what were the OTS component-based development processes?
The first research question RQ1 is to investigate the actual development processes used
in an OTS-based project. In the questionnaire, we asked the respondents to describe
their development process in detail, whereby the answers were then summarized by
different categories. The results are summarized in Figure 1 and show that the waterfall,
incremental, XP [18], German V-model [19], and prototyping model are those mainly
used in practice. Between these, the ‘incremental with prototyping’ model and XP was
the most popular. The German V-model was also used widely in Germany as the OTS-
based development process.
4.2. RQ2: when was the process decision made?
The second research question RQ2 is to examine whether the actual development
process was changed, considering the use of OTS-components. In the questionnaire, we
asked the respondents to select whether the actual development process was decided
before or after the make vs. acquire decision or after that. The results show that most
(75%) projects decided their main development processes before they started to think
about using OTS-components.
174
Appendix A
Fig. 1. The actual development process in the OTS-based project
4.3. RQ3: who was responsible for the process selection?
The third research question RQ3 is to identify the decision maker concerning the actual
development process. Within the questionnaire, we listed five options, such as
company/department rules, project manager, software architect, software developer, and
customer. The respondents were asked to select one or more from the options. The
answers reveal that in 29% of all projects the development process is predefined by
global company or department rules. In addition, concerning 14% of the remaining
projects the decision was at least affected by company/department rules. This trend is
especially obvious in Germany as the company rules dominated the development
processes in 65% of our studied projects.
4.4. RQ4: what was the OTS component selection processes?
The research question RQ4 is aimed at summarizing the OTS selection and evaluation
processes used in practice. Due to the length limitation of the questionnaire, we could
not ask the respondents to fill in the details of every OTS component in their projects.
Instead, they were asked to select one of the most important OTS components and fill in
the details for this component, named Comp.1. Information provided for Comp. 1 was
then used to investigate RQ5 and RQ6. To answer RQ4, we listed six possible activities
as named a) to f) in the questionnaire:

b) Got recommendation of possible OTS component candidates from the
customer.
c) Got recommendation of possible candidates from a local colleague/OTS-
expert.
d) Used a formal decision-making method to compare possible OTS component
candidates, e.g., with weighted evaluation criteria.
e) Limited possible candidates into 1-3 components, by reading
literature/documentation.
175
Appendix A
f) Did “hands-on” try-out of 1-3 components, e.g., on a downloaded demo

version.
The respondents were asked to fill in “yes”, “no”, or “do not know” for these
options. The results are summarized in Figure 2 and show that activities a, c, e, and f
were those mainly used. Thus, the familiarity-based selection process (activity c) and
the Internet search, trial-based selection process (activity a, e and f) from the
exploratory study are proved to be more popular than a formal processes (activity d) in
general. However, the further analysis with Chi-square shows that there are significant
differences of the activities b and c between countries. In Norway and Italy, very few
projects used the activity b. However, activity b was popularly used in Germany. In
addition, almost all German projects performed activity c, while it was used only in half
projects in Norway and Italy.
4.5. RQ5: when were OTS components selected?
From the exploratory study, we concluded that OTS components are selected in
different development phases. Thus, research question RQ5 was postulated to
investigate the selection phase of OTS component. The respondents were asked to select
one of five phases, ranging from pre-study to coding, as selection time. Figure 3 shows
that OTS components were selected in the early stages, such as pre-study, requirement,
and overall design, in most projects, especially in Germany.
4.6. RQ6: what was the influence of the project context?
RQ6 aims at investigating the correlation between the project context and the OTS
component selection process. We investigated two project context variables. The first is
the project members’ familiarity with the Comp.1, measured by a five point Likert scale
(i.e. very little, little, some, much, very much). The answers were coded into 1 to 5 (1
means very little and 5 means very much). The second variable is the importance of
Comp.1 in the system, measured by the contribution (in %) of ‘Comp.1’ to the overall
system functionality.
First, we investigated the correlation between the two context variables and the phase
in which Comp.1 was selected. The selecting phases are treated as an ordinal variable
and coded into the values 1 to 5 (1 means the pre-study phase and 5 means the coding
phase). The importance of Comp.1 was also treated as an ordinal variable, although it is
measured on a continuous scale. We then used the Spearman correlation test in SPSS
14.0 to investigate the correlation. The results are shown in Table 1 and reveal that the
selection phase of Comp.1 was weakly (with correlation coefficient value -.356)
correlated with the project members’ familiarity with it. It means that the project
members preferred to decide the familiar OTS components in the early stage of the
project. However, there is no significant connection between the importance of an OTS
component and the phase it was selected.
176
Appendix A
Fig. 2. What selection and evaluation actions were performed?
Fig. 3. When was the OTS component selected?
Second, we investigated the relationship between the two context variables with the
selection and evaluation activities performed. Since possible answers on the selection
and evaluation activity variable are “yes”, “no”, or “do not know”, we treated the
projects with “yes” and “no” answers as two independent samples and then compared
the mean differences of the two context variables. For the project members’ familiarity
variable, we used the Mann-Whitney test in SPSS 14.0 to compare their rank
differences. Results, presented in Table 2, show that project members will search the
internet for a possible candidate, if there is no proper or familiar candidate at hand.
177
Appendix A
Table 1. The correlation between the selection phase and the context variables
Context variables What phase was Comp.1 selected
Spearman correlation
Project members’ experience Correlation Coefficient -.356
with the Comp.1 Sig (2-tailed) .000*
The importance of the Comp.1 Correlation Coefficient -.132
Sig (2-tailed) .150
* P < .05
For the importance of Comp.1 variable, we used the independent T-test in SPSS 14.0
to compare mean differences, whereby the results are shown in Table 3. The overall
result is that there is no significant relationship between the selection and evaluation
activities performed with the importance of the OTS component.
Table 2. The relationship between selection activities and familiarity with the OTS
component
Selection activities Familiarity with Comp.1
Mann-Whitney Z Asymp. Sig
(2-tailed)
a) 1263.000 -2.407 .016*
b) 1072.500 -1.622 .105
c) 1626.500 -1.043 .297
d) 964.000 -.533 .594
e) 1724.500 -.870 .385
f) 1481.500 -.913 .361
(Selection activities a) to f) are explained in Section 4.4) * P < .05
Table 3. The relationship between the selection activities with the importance the
OTS component
Selection activities Importance of the Comp.1
t-test (equal variance not assumed) Sig (2-tailed)
a) .379 .706
b) -.971 .337
c) .804 .424
d) -.436 .666
e) 1.543 .126
f) 1.096 .278
(Selection activities a) to f) are explained in Section 4.4) * P < .05
5. How to customize the OTS-based development process
Concerning the conclusions drawn in another study [3], we agree that using OTS
components in system development is bearing potential danger, especially when using
unfamiliar OTS components. Although the risk driven processes proposed in [3] may
give benefits, results of RQ1 show, however, that most projects integrate OTS
components successfully with traditional processes, such as waterfall, incremental with
prototyping, XP, or V-model. Results of RQ2 show that most actual development
178
Appendix A
processes were decided before the decision of using OTS components was made. In
addition, results of RQ3 show that company rules pre-defined the main development
processes in many OTS-based projects. According to the results of RQ1 to RQ3, we
summarize the state-of-the-practice of OTS-based development processes into seven
different scenarios. The classification of the scenarios is based on the phase of the make
vs. acquire decision and the developers’ familiarity with the OTS candidates.
5.1. Scenarios of planned OTS-based software development
In these scenarios (see Figure 4), project members prefer to evaluate the possibility of
using OTS components right from the start. Due to the time-to-market, cost, or capacity
issues, project managers realize that it is difficult, impossible, or not cost-effective to
make everything from scratch. In these scenarios, the first step is to do a make vs.
acquire decision (see step A in Figure 4). In case project members decide not to use
OTS components, the development process will be the same as the non-COTS based
development. If project members decide to use OTS components, they then need to
decide the main development process (see step B in Figure 4). To decide the main
development process, main issues are the traditional risks in a project, such as the
stability of the requirements. One of the other issues is their familiarity with possible
OTS candidates. According to the project members’ familiarity with the OTS
candidates, we define three scenarios from scenario 1 to 3.
- Scenario 1 – planned OTS-based project with unfamiliar OTS candidates: In

this scenario, the project members prefer to use OTS component to provide
certain functionalities of the system. However, they are not familiar with the
OTS candidate at all. To decide the main development process, the OTS relevant
risks, such as the quality of OTS components and the vendor’s support, should
be seriously considered. In addition, project members should pay attention to
other risks, which are not relevant to an OTS component but the overall project
(e.g., which process is more suitable for projects with unclear or unstable
requirements? Waterfall or an incremental with prototyping’ model?).
- Scenario 2 – planned OTS-based project with some familiar OTS candidates: In
this scenario, the project members have a certain experience on some of the
possible OTS candidates. However, there are still certain candidates that they
have never used before. In order to decide the main development process, the
issues are similar with those of the scenario 1.
- Scenario 3 – planned OTS-based project with familiar OTS candidates: In this
scenario, the project members are familiar with all possible candidate
components. When deciding the main development process the OTS relevant
risks are not as critical as in scenario 1 or 2. Other non-OTS relevant factors,
such as company rules or requirement flexibility may be used to decide the main
development process. In this scenario, the main development process can be any,
such as waterfall, incremental, or XP. It may not need to be changed because the
project members are very familiar with the OTS component candidates.
In our investigated projects, only 25% can be classified into scenario 1 to 3, because
they decided their main development processes after they decided to use OTS
179
Appendix A
components. In these projects, 67% can be put into scenario 3, because they used
familiar OTS components.
The COTS-based development model in [10] illustrates the differences between
COTS-based development and traditional software development, such as new, reduced,
or modified activities. The underlying assumption of this model is that the rate or degree
of using OTS components is fixed in the very early phases of a project and that the main
development process has to be modified accordingly. The model can therefore be used
as a reference in scenarios 1 to 3.
Fig. 4. Scenario 1 to 3
5.2. Scenarios of unplanned OTS-based software development
In these scenarios (see Figure 5), the decision of using OTS component was not planned
or made in advance, i.e. there is no clear intention of using OTS components in the early
phases of such a project. Project members decide on the main development process
according to some non-OTS relevant factors, such as company rules and requirement
stability. Due to the time-to-market pressure or internal capability, project members
start to evaluate the possibility of using OTS components at certain stages of the project.
At an industrial seminar [17], several industrial participants mentioned such conditions.
According to the project members’ familiarity with the OTS candidates, we classify
three scenarios from scenario 4 to 6.
- Scenario 4 – Unplanned OTS-based project with unfamiliar OTS candidates: In

this scenario, the project members decided to use OTS components in a late
phase of the project, such as detailed design or coding. However, they do not
have any previous experience with the possible OTS candidates. Therefore, the
use of OTS components may bring several problems for the whole project. They
need a second risk-evaluation to investigate whether the development process
should be modified concerning the use of OTS components. For example, in
180
Appendix A
projects with pure waterfall processes, the project members may need to add a
short prototyping step to evaluate the OTS component and to negotiate the use
of OTS component with their customer.
- Scenario 5 – Unplanned OTS-based project with some familiar OTS candidates:
In this scenario, the project members decided to use OTS components in a late
phase. However, they do not have previous experience with all OTS component
candidates. In order to decide the main development process, the issues are
similar with those of scenario 4.
- Scenario 6 – Unplanned OTS-based project with familiar OTS candidates: In
this scenario, although the project members decided to use OTS components in a
late development phase, they have enough experience on the OTS component
candidates or are able to hire a person with such experience. In case the project
members have used the OTS component many times before, the OTS component
can be regarded as comparable to in-house built components. Although the use
of OTS component is a little risky due to the same reasons as described in
scenarios 4 and 5, this scenario is less risky because of using familiar OTS
components. It may not be necessary to do major changes to the current
Fig. 5. Scenario 4 to 6
In our studied project, about 75% of them can be classified into scenario 4 to 6,
because they decided their main development processes before they started to think
about using OTS components. In these projects, 65% used familiar OTS components
and therefore can be classified into scenario 6.
5.3. The scenario of semi-planned OTS-based software development
This scenario is a mix of scenarios 1, 2, or 3 with scenarios 4, 5 or 6, whereby the use of

OTS components is planned for some parts of the system. The main development
181
Appendix A
process is decided according to the use of OTS components. However, in a late phase of
the project, the project member may need to add some unplanned OTS component due
to the same reasons as scenario 4, 5, or 6.
The EPIC approach [1], like the RUP, consists of four phases and each phase
consists of one or more EPIC iterations. The information of four spheres of influences,
such as stakeholders’ needs, architecture design, risk, and marketplace, was gathered
and evaluated in each iteration. The project management, customer requirements, and
system architecture will be adjusted according to the trade-offs of the four spheres of
influences. Since every iteration includes a complete sub-process to make the trade-off
decision and select the OTS component, it is more flexible and suitable to be used as a
reference in scenarios 4 to 7.
6. How to select the OTS components
The results of this study support our conclusion in the exploratory study [6], and shows
that the familiarity-based and internet search-based processes are frequently used in
practice. The results also confirm that OTS component selection can be performed in
different development phases (requirements, design, or implementation), depending on
the project context, especially on the familiarity with possible OTS components, and/or
the flexibility of requirements [6]. In addition, results show project members aim at
reusing familiar OTS components and selecting them in early phases. In our study, we
measured the importance of an OTS component by their contribution on the
functionality of the system. Although the results show no correlation between the OTS
component’s importance with the phase it was selected, we still suggest project
members to select OTS in the earlier phases, especially if the OTS component has a
tight coupling with other components.
In different scenarios shown in Section 5, different OTS component selection
strategies and processes can be used. In scenarios 1, 2, 4, 5, and 7, some unfamiliar OTS
component candidates are going to be used. The selection process for unfamiliar OTS
components can either be an internet search with hand-on trial-based, or a formal
selection process, such as processes shown in [5], [8], [9], [11], [13], and [14]. In
scenarios 3 and 6, all possible OTS component candidates have been used by the project
members, the selection process could mainly be familiarity-based.
In scenarios 1 to 3, the use of OTS component is well-planned. In general, the OTS
component will be selected in the early phase of the project in these scenarios. Selecting
OTS component in the early phase can help to design the software architecture with
consideration on constrains of the OTS components. Therefore, it is easier to avoid the
mismatch between the OTS components and other components in the system.
In scenarios 4 to7, the OTS components are usually be evaluated at a phase which
most of the other components are already integrated in the system. Since the goal is to
integrate the OTS component into the current system and to check its integration
thoroughly, integrate the demo version of the OTS component and do a hand-on trial is
necessary. In case the OTS component was selected in the very late stage of the project,
it is better to select a component that has a loose coupling with the other components.
OTS components tightly coupled with existing components may impose a high number
of changes onto the system.
182
Appendix A
7. Threats to validity
To avoid possible threats to the construct validity of this study, we performed a pre-test
to verify the questionnaire. In addition, we use a lot of effort to ensure the representative
of the sample. Detailed discussion on other validities issue of this study are reported at
[4], [7].
8. Conclusion and future work
This paper has presented a state-of-the-practice survey on the process issues in OTS-
based development. We have studied 133 projects from Norway, Italy, and Germany.
The main findings are:
OTS-specific activities. The development process was dominated by the
company/department rule instead of the decision of using OTS components.
evaluation processes are familiarity-based or Internet search with hands-on-trial-
based. The phase to select OTS component has relationship with the project
members’ familiarity.
The main limitation of this study is that it is a state-of-the-practice survey. We are

going to perform more detailed case studies to get detailed information in projects with
different contexts. The intention is to deepen our understanding the process
improvement in OTS component-based development and to verify our proposals.
7. References
1. Albert, C. and Brownsword, L.: Evolutionary Process for Integrating COTS-Based

System (EPIC): An Overview. Software Engineering Institute, Pittsburgh, (2002),
2. Brownsword, L., Oberndorf, T., and Sledge, C.: Developing New Processes for
COTS-Based Systems. IEEE Software, July/August (2000), 17(4):48-55.
3. Boehm, B. W. and Abts, C.: COTS integration: Plug and Pray? IEEE Computer, Jan.
(1999), 32(1):135-138.
4. Conradi, R., Li, J., Slyngstad, O. P. N., Bunse, C., Torchiano, M., and Morisio, M.:
Reflections on Conducting an International CBSE Survey in ICT Industry. Proc. of
the 4th Int. Symposium on Empirical Software Engineering, Noosa Heads, Australia,
Nov. (2005) 207-216.
5. Kontio, J.: A Case Study in Applying a Systematic Method for COTS Selection.
Proc. of the 18th Int. Conf. on Software Engineering, Berlin, Germany, March (1996)
201-209.
6. Li, J., Bjørnson, F. O., Conradi, R. and Kampenes, V. B.: An Empirical Study of
183
Appendix A
Industry. Proc. of the 10th IEEE Int. Metrics Symposium (Metrics'04), Chicago,
USA, Sept. (2004) 72-83.
7. Li, J., Conradi, R., Slyngstad, O. P. N., Bunse, C., Khan, U., Torchiano, M., and
Morisio, M.: Validation of New Theses on Off-The-Shelf Component Based
Development. Proc. of the 11th IEEE Int. Metrics Symposium (Metrics’05), Como,
Italy, Sept. (2005) 26.
8. Tran, V., Liu, D. B., and Hummel, B.: Component Based Systems Development:
Challenges and Lessons Learned. Proc. of the 8th IEEE Int. Workshop on Software
Technology and Engineering Practice, London, UK, (1997) 452-462.
9. MacCrimmon, K. R.: An Overview of Multiple Objective Decision Making. Proc. of
the Multiple Criteria Decision Making, University of South Carolina Press, (1973)
18-44.
10. Morisio, M., Seaman, C.B., Parra, A. T., Basili, V. R., Kraft, S. E., and Condon, S.
E.: Investigating and Improving a COTS-Based Software Development Process. Proc.
of the 22nd Int. Conf. on Software Engineering, Limerick, Ireland, June (2000) 31-40.
11. Morisio, M. and Tsoukias, A.: IusWare: a Methodology for the Evaluation and
Selection of Software Products. IEE Proceedings-Software Engineering, June (1997),
144(3):162-174.
12. Ncube, C. and Dean, J. C.: The Limitation of Current Decision-Making Techniques
in the Procurement of COTS Software Components. Proc. of the 1st Int. Conf. on
COTS-Based Software Systems (ICCBSS’02), Orlando, FL, USA, Feb. (2002),
LNCS Vol. 2255, Springer-Verlag Berlin Heidelberg New York 176-187.
13. Ochs, M., Pfahl, D., Diening, G. C., and Kolb, B. N.: A Method for Efficient
Evaluation Results. Proc. of the 7th IEEE Int. Software Metrics Symposium, London,
England, Apr. (2001) 285-297.
14. Saaty, T. L.: How to Make a Decision: The Analytic Hierarchy Process (AHP).
European Journal of Operational Research, (1990), 48(1): 9-26.
15. SEI COTS-Based Initiative Description. Software Engineering Institute, Pittsburgh,
(2004), http://www.sei.cmu.edu/cbs/cbs_description.html
16. Torchiano, M. and Morisio, M.: Overlooked Facts on COTS-based Development.
IEEE Software, March/April (2004), 21(2):88-93.
17. Li, J. Conradi, R. Slyngstad, O. P. N., Bunse, C., Khan, U., Torchiano, M., and
Morisio, M.: Barriers to Disseminating Off-The-Shelf Based Development Theories
to IT Industry. Proc. of the ICSE2005/MPEC workshop, St. Louis, Missouri, USA,
May (2005), 1-4.
18. Kent, B.: Extreme Programming Explained: Embrace Change. (1999) Addison-
Wesley.
19. German V-model: http://www.v-modell.iabg.de/#AU250
184
Appendix A
P5. Risk Management in OTS-Based Development

A State-of-the-Practice Study on Risk Management in Off-
the-Shelf Component-Based Development
Jingyue Li1, Marco Torchiano2, Reidar Conradi1,
Odd Petter N. Slyngstad1, and Christian Bunse3
1
2
marco.torchiano@polito.it
3
Fraunhofer IESE, Fraunhoferplatz 1,
D-67663 Kaiserslautern, Germany
INDEX TERMS
D.2.13. Software Engineering/Reusable Software, D.2.9. Software

Engineering/Management, D.2.18. Software Engineering/Software Engineering Process
ABSTRACT: This paper reports and discusses an international survey on risk

management in OTS (off-the-shelf) component-based software development. Most
studies on project risk management in this area are still limited to theoretical proposals
or case studies in specific project contexts. Without experience derived from large scale
verifications in different project environments, it is difficult for project managers to use
available risk management guidelines properly and efficiently. This study investigates
performed risk management activities and their results in OTS component-based
development. Data from 133 projects in Norway, Italy, and Germany have been
collected using the stratified-random sample selection strategy. Results reveal that
putting more effort into learning OTS components, performing the integration testing
early, evaluating the quality of OTS components thoroughly, and following the update
of OTS components have helped to mitigate corresponding risks. However, some
problems, such as wrong estimation of the integration efforts and inefficient debugging,
still happen frequently in practice and need further investigations.
1 Introduction
Long term success in commercial software development is becoming increasingly

challenging. As customers expect better, earlier, and cheaper software, software
companies are finding it increasingly difficult to protect their technology investments
and maintain productivity levels. The software industry is therefore increasingly turning
to approaches which offer a larger than normal return on software development effort.
Component-based software development (CBSD) [Szyperski02] is one of these
approaches. In addition to reusing in-house built components, software companies are
185
Appendix A
using an increasing amount of COTS (Commercial-Off-The-Shelf) and OSS (Open

Source Software) components. However, using such external components introduces a
number of additional risks [Rose03], [Boehm03], [Voas98a], [Voas98b]. Thus, project
managers have to identify and evaluate possible risks before deciding to acquire an
external component, instead of developing an in-house component. Although several
risks and risk management activities in OTS-based development have been identified
from case studies [Rose03], [Boehm03], [Abts00], [Voas98a], [Voas98b], [Kotonya01],
[Faa03], [Fitzgerald04], [Lawton02], [Vitharana03], [Ruffin04], few subsequent
empirical studies have verified their conclusions. As a result, software project managers
have few effective and well-proven guidelines to identify the relative effects of the
various risks, and to select the effective activities to reduce such risks.
In this study, we did postmortem investigations on problems occurred and the risk
management activities performed in 133 finished OTS component-based projects. The
first research question was to find out which problems occurred more frequently than
others. The second research question was to investigate which risk management
activities have been performed frequently, and the third research question was to
investigate the correlation between the (lower) occurrence of the problems and the
application of risk mitigation activities. It was intended to find out which performed risk
management activities were most effective on avoiding corresponding risks.
Results of this study show that the most frequent problems in OTS-based
development are cost estimation and defect localization. It is notable that problems
related to the quality of the OTS components do not appear as frequent as commonly
assumed. To manage the risks in OTS-based development, it is critical to study and
understand the OTS components thoroughly, to evaluate their quality by black-box
testing, to keep track with the update of these OTS components, and to manage and
share the knowledge on these OTS components.
The remainder of the paper is structured as follows: Section 2 presents work related
to this study, while Section 3 discusses the research design and related research
questions. Profiles of the collected samples are presented in Section 4 and the empirical
results concerning the research questions are given by Section 5. Section 6 presents
discussions on the study results, and Section 7 provides conclusions and open issues.
2 Related Work
2.1 Risk Management in Software Development
Risks are factors that may adversely affect a project, unless project managers take
appropriate countermeasures. Risk management includes two primary steps each with
three subsidiary steps as described by Boehm [Boehm91], and shown in Figure 1. Risk
identification produces lists of project-specific risk items likely to compromise the
success of a project. Risk analysis assesses the loss probability and loss magnitude for
each identified risk item. Risk prioritization ranks the risk items identified and analysed.
Risk management planning helps prepare to address each risk item. Risk resolution
produces a situation in which the risk items are eliminated or otherwise resolved. Risk
monitoring involves tracking the progress of a project towards resolving its risk items
and taking corrective activities where appropriate.
186
Appendix A
Fig.1. Risk Management Framework
In order to identify risks, a list of possible project risks must be produced. Boehm
[Boehm89] lists the top 10 risks by probing several large software projects and their
common risks. Barki et al. [Barki93] identified 35 variables, which are subsequently
used as a basis for creating scales in a project risk assessment instrument. The Software
Engineering Institute (SEI) [Carr93] has contributed significantly to the literature with
the SEI Taxonomy-Based risk identification instrument, which contains 194 questions.
However, the notion of building single all-encompassing risk taxonomy to be globally
used by all software developers is probably unrealistic. Thus, we may need different
risk taxonomies in different project contexts [Moynihan97].
A few other studies have classified software risk items, i.e., consider software risks
along several dimensions and have provided some empirically founded insights of
typical software risks and their variation. The risks can be classified into technical,
organizational, and environmental [Sherer95]. The possible risks can also be
summarized into a hierarchical holographic modelling (HHM) framework
[Chittister96]. Ropponen and Lyytinen [Ropponen00] empirically delineate six
components of software development risks, such as scheduling and timing risks, system
functionality risks, subcontracting risks, requirement management risks, resource usage
and performance risks, and personnel management risks. Keil et al. [Keil98] extracted
the experience of more than 40 software project managers and identified a universal set
of risk factors, such as customer mandate, scope and requirements, environment, and
execution. A following study investigated the relationship between these factors and
their effect on the result of the project [Wallace04].
During the risk analysis process, project managers have to consider each identified
risk and make a judgment about its probability to occur and seriousness. The project
managers can tabulate the results of this analysis process using a table ordered
according to the serious of the risk [Sommerville04]. The estimation of the seriousness
and the rank of the risk should be combined with cost estimation [Madachy97],
187
Appendix A
[Kansala97]. Once the risks have been analysed and ranked, the project managers need
to assess which are the most significant risks.
The purpose of identifying and analysing the risks is to manage these possible risks.
The possible risk management activities have been proposed from processes
[Boehm88], organization [Roppoen00], [Gemmer97], and technology [Hecht04]
viewpoints. However, many researchers advocate rather than investigate. As a result,
some advocated concepts are worth less than their advocates believe and there is a
shortage of evaluative research to help determine the actual value of new tools and
techniques [Glass01]. Although risk management proposals of the in-house software
development have been verified by a survey-based research [Verner05], few studies
have been performed to empirically investigate the risk management proposals in using
the OTS components.
2.2 Specific Risk Management in OTS Component-Based Development
In addition to the classical risks associated with developing large systems, OTS
components requires managers to modify their typical mitigation activities for some of
the classic risks and to develop new mitigation activities for risks that are particular to
the use of OTS components in a system.
Vigder et al. [Vigder96] investigated the specific risk issues in the COTS-based
development and point out the fundamental differences between COTS-based
development and building in-house are:
- COTS-based development may need infrastructure earlier to demonstrate and

validate the COTS component.
- The COTS component may dictate standards, architecture, and design.
- The COTS component may influence work flow.
- Picking the wrong COTS component may be more expensive than fixing
problems in the in-house built software.
- Issue resolution processes need to be in place earlier to resolve COTS
component issues.
- Issue resolution processes may be more complicated because of the addition of
the vendor and possible incompatibilities between the vendor’s practices and
yours.
Boehm et al. [Boehm03] summarized experience from their COTS-based university

projects. Examples of the summarized risks and corresponding risk management
activities (In brackets) are:
- Requirement changes and mismatches (Prototyping, win-win negotiation among

all stakeholders).
- Overly optimistic expectation on COTS quality attributes (Significant quality
features must be tested).
- Overly optimistic COTS component learning curve (A most likely COTS
component learning curve must be accessed during planning and scheduling).
- Version upgrade may result in retailoring of the COTS component (Ensure the
new version still can implement the capability of the old version).
188
Appendix A
The software engineering resource centre at the Federal Aviation Administration

(FAA) [Faa03] summarized 10 risk factors in COTS-based development and proposed
corresponding risk mitigation activities. The COTS lessons-learned in a repository made
by Basili et al. [Basili02] summarized the possible risk management activities based on
case studies. In these studies, COTS software can be either components or very large
packages, such as ERP, CRM, Data Bases, or Operating Systems.
A few other studies contribute to investigate risk management in OSS-based
development. For example, Ruffin and Ebert [Ruffin04] point out the possible licensing
problem of using OSS as follows:
- If you reproduce or sell the product without the licensor’s permission, the
licensor might claim damages or force you to end the product’s production,
delivery and sale.
- Another exposure is infringement of third party’s patents or other intellectual-
property rights.
- A third risk area is the use of OSS in development activities when OSS is a tool.
The developed product may need to follow the licences of the OSS tool, i.e., the
code should also be open.
Fitzgerald [Fitzgerald04] mentioned that organizations should not expect

maintenance and support to be available at a lower cost that would apply for
commercial software, even though OSS may be available at little or no cost. Other
researchers mention that freely available source code addresses two typical concerns
with using COTS components: unknown implementation quality and long-term vendor
support [Madanmohan04].
Other researchers have focused only on the component-based development without
telling the differences between in-house built COTS, or OSS components. Bass et al.
[Bass00] mention inhibitors in component-based development as being: lack of
available components, lack of standards for component technology, lack of certified
components, and lack of engineering methods. Some challenges in each development
phase and for a project as a whole, such as management, component-based software life
cycle, composition predictability, component certification, component configuration,
and tool support, are discussed in detail in [Crnkovic02], [Ghosh02], [Jacobson97].
Kotonya and Awais [Kotonya01] put risk management in component-based
development into the risk-weighting scheme of [Moynihan97] and listed several
questions in each risk item to help project managers to quantitatively assess the risks.
Some studies focus only on COTS components. Rose [Rose03] classifies possible
risks in COTS component-based development based on the different phases. Examples
of possible risks and corresponding risk management activities (In brackets) are:
- Lack of skilled personnel (Ensure that there is an appropriate number of senior

technical personnel assigned to the project).
- Architecture mismatch (Design the architecture first and select COTS
component that best fits the architecture).
- Product obsolescence (Maintain a continual watch on the marketplace looking
for substitute products as well as new technologies that might replace older
ones).
189
Appendix A
- COTS components have asynchronous update cycles with the COTS-based

system (Involve senior analysts directly involved in the analysis of the upgrades
and derive effective field upgrade procedure and schedules)
The CBSEnet project [CBSEnet04] investigated the state-of-the practice in COTS

component-based development and summarized the challenges from business,
information technology, and legal perspectives. The study performed by Li et al. [Li04]
summarized the problems encountered and presented good practices from finished
COTS component-based projects. Examples of the problems encountered are:
- Could not negotiate requirements with the customer.

- The learning effort was underestimated.
- It was difficult to locate defects.
- The COTS component could not be deployed as advocated by the vendor.
- Examples of good practices are:
- Integrate unfamiliar COTS component first.
- Do integration testing early and incrementally.
- Investigate COTS component market and prepare to replace obsolescent
components.
Since the OTS components involve both COTS and OSS components, risks and risk
management activities relevant to OTS components-based development are therefore
the subset of risks and risk management in component-based development, COTS-
based, and Open Source based development, as shown in Figure 2.
Fig. 2 OTS Component-based Development Risks
Although the mentioned risk management activities have been proposed to avoid the
corresponding risks in OTS component-based development, they are generally based
either on common sense or on small and non-representative case studies. Few empirical
studies have subsequently validated their conclusions. In addition, it is important to
compare the results of the performed risk management activities and to find out the
most efficient ones that mitigated a specific risk. The results can help project managers
put their (limited) resources on the most efficient risk mitigation activities to avoid
identified risks.
190
Appendix A
3 Research Design
In this study, we have used the conceptual measurement framework proposed by Bush
and Fenton [Bush90]. The study includes one entity (i.e., the project) and two attributes
(i.e., possible risks of the project and possible risk mitigation activities to avoid these
risks). There are also two relationships, i.e., risks are related to a project and risk
mitigation activities that IMPACTS risks. Before the start of a project, project managers
must upfront estimate how often the risks may happen, how much effort they need to
put in to certain risk mitigation activities, and how well the risks may be managed. The
relationships are shown in Figure 3.
Fig. 3 Risk management before a project starts
To help project managers predict the possibility of risks and to plan the risk
management efficiently before a project starts, it is necessary to summarize practices
and results of risk management from finished projects. The research framework, as
shown in Figure 4, of this study is designed to do posteriori measurements (after the
project is finished) on the occurrence of project risks and the impacts of performed
risk management activities on these risks.
The first research questions RQ1, as shown in Figure 4, is to investigate the actual
risk incidence, i.e., how often had problems occurred in practice? The second research
question RQ2, is to investigate the execution of the risk management activities in
projects, i.e., how often had certain risk management activities been performed in
practice? The third research question RQ3 is to examine the effect of performed risk
management activities, i.e., how well the performed risk mitigation activities had
reduced corresponding risks? For a specific risk, we are interested in the most effective
mitigation activities.
191
Appendix A
Fig. 4 Research framework
In our study, we selected several typical risks and possible risk management
activities proposed in [Boehm03], [Faa03], [Li04], [Rose03], [CBSEnet04],
[Vitharana03], [Lawton02], [Fitzgerald04] [Torchiano04]. The selection criteria are:
- First, the risks and risk management activities should address the common issues
in the CBD, COTS-based development, and OSS-based development, as shown
in Figure 2.
- OTS component-based development encompasses three primary types of
stakeholders, namely component developers, application assemblers/integrators,
and customer. Different stakeholders have different risks and challenges
[Vitharana03]. We have selected risks and challenges that are relevant only to
the application assemblers/integrators.
- The selected risk management activities in our study focus mainly on process
and organizational issues, instead of technical methods, such as wrapping,
composition prediction, and testing.
The selected typical risks in OTS components-based cover different phases of a

project, as shown in Table 1. The typical risk management activities are shown in Table
2. In addition, the results from process improvement studies show that one added role in
OTS component-based development, i.e., that of a knowledge keeper (dedicated project
personnel to keep track of OTS component knowledge/information), may help to
mitigate the risks in OTS component-based development [Li04], [Morisio00], [Rose03].
We therefore propose the existence of knowledge keeper as one possible risk
management activity.
192
Appendix A
Table 1 Typical Risks in OTS Component-Based Development
Table 2 Typical Risk Management Activities in OTS Component-Based

Development
193
Appendix A
The data was collected with a web-based questionnaire using the SESE tool [Sese04]
at Simula Research Laboratory. To answer research questions RQ1 to RQ3, we listed
the risks and risk management activities shown in Table 1 and 2. Respondents are asked
if these risks and risk management activities had occurred. The alternatives are “don’t
agree at all”, “hardly agree”, “agree somewhat”, “agree mostly”, “strongly agree”, or
“don’t know”. We assign an ordinal number from 1 to 5 to the above alternatives (5
meaning strongly agree). In addition, we asked if there are any dedicated personnel (or
knowledge keeper) in the actual project to keep track of OTS component
knowledge/information. The answer is “yes” or “no”.
3.1 Data Collection

3.1.1 Key definitions Questionnaire Design
The components investigate in our study have larger granularity than the component in
“OO” theory [Crnkovic02]. The main definitions used in the questionnaire are:
- Component: Program units of independent production, acquisition, and

deployment and which can be composed into a functioning system. We limit
ourselves to components that have been explicitly decided either to be built from
scratch or to be acquired externally as an OTS-component. That is, components
that are not shipped with the operating system, not provided by the development
environment, and not included in any pre-existing platform. That is, platform
(“commodity”) software is not considered, e.g., an OS like Linux, DBMSes,
various servers, or similar software. Furthermore, components usually follow
some component model, as expressed by the COM, CORBA, EJB, .Net, or Web
Service standards, or they can be a C++/Java library.
- OTS component: A component provided (by a so-called provider) from a
commercial vendor or the Open Source community. An OTS component may
come with certain obligations, e.g., payment or licensing terms. An OTS
component is not controllable in terms of provided features and their evolution.
An OTS component is mainly used as closed source, i.e., no source code is
usually modified, even if it may be available.
3.1.2 Questionnaire structure
The questionnaire includes six sections. The three sections relevant to this study are:
- Background questions to collect information on the company, project, and

respondents.
- Main questions about risk and risk management.
- Questions to collect information about concrete OTS components used in the
project.
3.1.3 Data collection procedure
We define the unit of this study as a completed software development project using
OTS components. The projects were selected based on two criteria:
194
Appendix A

We used random sample selection strategy to collect representative companies and

convenient strategy to select projects in a company in Norway, Italy and Germany:

(SSB) [Ssb02] because they have a pool of large, medium, and small companies.
Based on the number of employees, we selected the 115 largest IT companies
(100 IT companies and 15 IT departments in the largest 3 companies in 5 other
sectors), 200 medium-sized software companies (20-99 employees), and 100
small companies (5-19 employees) as the original contacting list.
then randomly selected companies from this list. For these randomly selected
To avoid bias in the data collection procedure, we made a guideline on how to

contact respondents before the survey started. All countries – Norway, Italy, and
Germany – followed the process from the guidelines explained below.
Possible respondents were contacted first by telephone. If the candidate respondents
had suitable OTS-based projects and would like to join our study, a username and
password of the survey tool and an electronic version of the questionnaire were sent to
them. The respondents could use either the SESE web tool [Sese04] or the electronic
version (.doc file) to fill in the questionnaire. The respondents who didn’t want to
answer the questionnaire were also registered. We logged the main reasons of non-
response, such as “no software development”, “no OTS-based projects”, and “busy”.
Most respondents used the Web tool to fill in the questionnaire. However, phone
interview were used in Germany because of the confidential concerns in some
companies. In total, we contacted 1087 companies in three countries. In these
companies, 425 of them have experience on OTS component-based development. From
these relevant companies, we managed to collect data from 127 companies. This study
is probably the first software engineering survey using census type data, and has
revealed that the entire sampling and contact process can be unexpectedly expensive.
The study reveals that, at best, we can achieve a stratified-random sample of ICT
companies, followed by a convenience sample of relevant projects. Detailed discussions
on sample selection, the contact process and the response rate are reported in
[Conradi05].
195
Appendix A
3.2 Data Analysis
Concerning research questions RQ1 and RQ2, we compared the occurrences of typical
risks and performed risk management activities. The results are visualized with the
boxplot graph.
The intention of research question RQ3 is to find out the correlations between
occurrences of risks and risk management activities performed. If there is a negative
correlation between the risk management activity and the occurrence of possible
problems, we assume that the performed risk management activity actually helped to
mitigate the possible problems. Since several performed risk management activities may
have helped to mitigate the same risk, we are only interested in finding out the most
effective risk mitigation activities.
For research question RQ3, we decided to use Nonlinear Canonical Correlation
Analysis (Nonlinear CCA) [Gifi90], [Michailidis98], because we face a nonlinear CCA
problem with a set of explanatory variable vector defined by a nominal (knowledge
keeper) variable and several ordinal (risk management activities performed) variables
and a dependent variable matrix defined by a series of ordinal (risk occurrences) scales.
The linear combination on the explanatory variable side is undefined, because we have
no metric to quantify the categories of each variable. The linear combination of the
variables on the dependent side is also undefined, because the categories of each
variable can be re-scaled by any nonlinear function the preserves monotonicity. Thus,
we need to optimally scale or quantify the variables while simultaneously solving the
traditional linear CCA problem of finding weights for each explanatory variable.
Nonlinear CCA corresponds to categorical canonical correlation analysis with optimal
scaling. The detailed introductions and discussions on using the Nonlinear CCA instead
of other analysis methods in this study are given in the Appendix A.
The nonlinear CCA used in this study is the OVERALS in SPSS version 14.0
[Spss06]. The outputs of the OVERALS in SPSS include several parts. The most
important ones are:
- The fit and loss values tell how well the nonlinear CCA solution fits the
optimally quantified data with respect to the association between the sets.
- The plot of component loading for the data. When there is no missing data, the
component loading is equivalent to the Pearson correlation between the
quantified variables and the canonical variate. The distance from the origin to
each variable point approximates the importance of that variable.
- The object score plots and centroids plots provide the remainder of the
information we need to interpret a nonlinear CCA solution.
4 Collected Data
38 from Italy) in 127 companies. In general, we selected one project from each
companies because those companies have many OTS-based projects wanted to
contribute more experience to this study.
196
Appendix A

employees) and large (100 or more employees) ones. The main business areas of the
sampled companies also cover different types and the final systems produced by the 133
projects cover various application domains. Detailed profiles of the selected companies
and projects are shown in [Li05].
components, and six used both COTS and OSS components. Most respondents of the
133 projects have a solid IT background. More than 90% of them are IT managers,
project managers, or software architects. Most of them have more than 2 year
experience with OTS-based development. All of them have at least a bachelor degree in
informatics, computer science, or telecom.
The OTS components used in the investigated projects include components that
follow the COM, DCOM, and EJB models. There are also some components in the
format of C++ or Java libraries. Examples of the selected components are listed in the
Table 1 in Appendix B.
5 Results of Research Questions
5.1 RQ1: Occurrences of Risks in OTS Component-Based Projects
As the occurrences of possible risks are measured with an ordinal scale, we used
boxplot to illustrate the distribution of the risk occurrences. The results are in Figure 5
and show that the most frequent risks are R2 (R_IntEff) and R8 (R_DefLoc). It means
that the integration effort was usually wrongly estimated. In addition, it was difficult to
figure out whether the defects are inside or outside the OTS components. The least
frequent risks are R3 (R_ComRel), R4 (R_ComSec) and R5 (R_ComPerf), i.e., the
problems relevant to the negative effect of the OTS components on the quality of the
whole system.
2
R
R
R
R
R
R
R
R
R
R
_C
_C f
_C
_I
_M
_C
_C an
_C
_P d
_M
_S
_R a
_D
nt
ro
el
om
om
om
om
om
om
eq
ef
ai
is
E
E
vI
L
sS
nt
R
ff
ff
Se
C
R
U
Pe
oc
nf
en
Pl
up
el
om
d
p
c
p
p
Fig. 5 Occurrences of Risks
197
Appendix A
5.2 RQ2: Performed risk management Activities
The occurrence of performed risk management activities are measured by an ordinal

scale too. Figure 6 presents a boxplot illustrating the distribution of the risk
management activities.
R
R
R
R
R
R
R
R
R
m
m
m
m
m
m
m
m
m
_I
_M
_S
_L
_C
_B
_U
_S
_U
_P
_C
nc
el
el
ea
ro
B
st
st
nf
pd
kt
Te
Q
A
Se
Ef
M
vW
rn
Fi
W
Fo
ua
rc
kB
st
Ef
l
rs
tc
ll
h
tc
l
t
f
h
uy
h
Fig. 6 Occurrences of Risk Management Activities
From Figure 6, we can see that the activities Rm4 (Rm_SelQual), Rm5
(Rm_LearnEff), Rm7 (Rm_UnfFirst), Rm8 (Rm_IncTest) and Rm9 (Rm_UpdFoll)
were used more frequently than others. It means that developers have seriously
evaluated the quality of the OTS components in the selection phase. They have
considered the possible learning effort in the effort estimation. They have done
integration testing as early as possible and they usually have internal expert to follow
the update of the OTS components. However, the activities relevant to the users, such as
Rm1 (Rm_CstMkBuy) and Rm2 (Rm_CstSel), were least frequently used.
5.3 RQ3: Correlations between the Risks and Risk Management Activities
The analysis is organized in six parts (See section 5.3.1 to 5.3.6). In the first section, we
analyze the correlation between the effort estimation risks, i.e., R1 and R2, with the risk
management activities Rm1 to Rm8, i.e., for development. Although Rm1 to Rm8 have
been proposed as effective activities to mitigate possible risks in OTS component-based
development, there are few empirical studies to verify and compare which risks they can
most effectively mitigate. The target of our analysis is to find out which of the activities
Rm1 to Rm8 are most effective to avoid overly optimistic effort estimation. Due to the
same rationale, we analyzed the correlation between the quality risks, i.e., R3 to R5, and
the risk management activities Rm1 to Rm8 in the second section. In the third section,
we analyzed the correlation between the requirements relevant risks, i.e., R6 and R7,
198
Appendix A
with the same Rm1 to Rm8. In the forth section, we analyzed the debugging and
deployment risks, i.e., R8 and R9, also with Rm1 to Rm8.
In the fifth section, we analyze the maintenance risks, i.e., R10 and R11, with Rm9 to
Rm13, i.e., for maintenance. In addition, we want to investigate whether the projects
involving a knowledge keeper had less maintenance problems than projects without this
role. Therefore, the existence of a knowledge keeper was also added as a risk
management variable. In the sixth section, we analyzed the correlation between the
vendor support risks, i.e., R12 and R13, with Rm9 to Rm13 and the existence of the
knowledge keeper.
5.3.1 Effort estimation risks – R1 and R2
The result of the correlations between the effort estimation risks, i.e., R1 (R_SelEff) and
R2 (R_IntEff) and risk management activities (Rm1 to Rm8) are analyzed with
OVERALS (two dimensions, some variables not multiple nominal, multiple steps,
random initial configuration) in SPSS. We selected to use two dimensions because the
upper bound of the number of the dimensions, i.e., the number of the canonical variates,
is the minimum of the number of the variables in each set [Gifi90]. In our study, the
minimum number of the variable in our analysis is two. Another reason for selecting the
two-dimension analysis is due to the interpretative convenience of two-dimension plots
[Gifi90]. The risk management activities (i.e., Rm1 to Rm8) are given to the first set
and each is defined as an ordinal variable with scale from 1 to 5. The effort estimation
risks (i.e., R1 and R2) are given as the second set and each is also defined as ordinal
variable with scale from 1 to 5.
The purpose of the analysis is to find out the most negative correlation between the
risk management activities and the occurrence of possible problems. The analysis
includes two steps:
- Step 1: For each possible problem, for example R1 (R_SelEff), we first check
the dimension (i.e., dimension 1 or 2) that R1 has been mostly loaded. If R1 is
loaded more in one dimension, for example dimension 1, it means that the
correlation between R1 and the dimension 1 is stronger than the correlation
between the R1 and dimension 2.
- Step 2: In the most loaded dimension of a risk, for example R1 (R_SelEff), we
then check the component loading to find out the risk management actions (i.e.,
Rms), which are in the peak positions of the opposite domain of R1.
The summary of OVERALS analysis show that 53% of the actual fit is accounted for
by the first dimension, and the 47% of the actual fit is accounted for by the second
dimension. The component loadings plots are shown in Figure 7 and detailed
component loading data are shown in Table 2 in Appendix B.
The results reveal that R1 (R_SelEff) is much more loaded (-.844) in the first
dimension (i.e., the horizontal axis) than (-.496) in the second dimension (i.e., the
vertical axis). R1 (R_SelEff) is in the negative domain of the first dimension and the
Rm5 (Rm_LearnEff) is in the peak of its positive domain.
199
Appendix A
R2 (R_IntEff) is also much more loaded (-.859) in the first dimension than (-.424) in
the second dimension. R2 (R_IntEff) is also in the negative domain of the first
dimension and the Rm5 (Rm_LearnEff) is in the peak of its positive domain.
The results illustrate that seriously taking the learning effort in consideration and
giving enough leeway on the learning curve helped to mitigate the effort estimation
risks.
Fig. 7 Component loading plots of the cost estimation risks
5.3.2 Quality risks – R3 to R5
For the quality risks, the results of the OVERALS analysis show that 51% of the actual
fit is accounted for by the first dimension and 49% by the second dimension. The
component loading plots are shown in Figure 8 and detailed component loading is
shown in Table 3 in Appendix B.
Results show that R3 (R_ComRel) is much more loaded (-.530) in the first
dimension (i.e., the horizontal axis) than (.264) in the second dimension (i.e., the
vertical axis). R4 (R_ComSec) is also much more loaded (-.643) in the first dimension
than (-.258) in the second dimension. However, R5 (R_ComPerf) is loaded weakly both
in the first dimension (.087) and the second dimension (.072).
Both R3 (R_ComRel) and R4 (R_ComSec) are in the negative domain of the first
dimension and the Rm8 (Rm_IncTest) is in the peak (with loading .456) of its positive
domain. In addition, Rm6 (Rm_BBEff) is very close (i.e., with loading .435, which is
4.6% less than Rm8) to Rm8 (Rm_IncTest) in the peak of the positive domain of the
first dimension.
The results illustrate that early incremental testing and black-box testing of the OTS
components in the selection phases helped to increase the reliability and security of the
OTS component-based system. However, none of our investigated risk management
activities helped to avoid the OTS components’ negative effect on the system
performance.
200
Appendix A
Fig. 8 Component loading plots of the quality risk
5.3.3 Requirement risks – R6 and R7
For the requirements risks, the results of the OVERALS show that 52% of the actual fit
is accounted for by the first dimension and the 48% by the second dimension. The
component loading is shown in Figure 9 and detailed component loading data is shown
in Table 4 in Appendix B.
Results show that R6 (R_ComAda) is loaded strongly both in the first dimension
(.647) and in the second dimension (-.620). R7 (R_ReqRen) is also loaded strongly both
in the first dimension (-.553) and the second dimension (-.761).
In the first dimension, R6 (R_ComAda) is in the positive domain and Rm6
(Rm_BBEff) is loaded (-.428) in the peak of the negative domain. In addition, Rm5
(Rm_LearnEff) is very close (i.e., with loading -.408, which is 4.6% less than Rm6) to
Rm6 (Rm_BBEff) in the peak of the negative domain of the first dimension. In the
second dimension, R6 (R_ComAda) is in the negative domain and Rm4 (Rm_SelQual)
is in the peak of the positive domain.
R7 (R_ReqRen) is in the negative domain of the first dimension and Rm7
(Rm_UnfFirst) is in the peak of the positive domain. In the second dimension, R7
(R_ReqRen) is in the negative domain and the peak in the positive domain is Rm4
(Rm_SelQual). Moreover, the results show that R6 (R_ComAda) and R7 (R_ReqRen)
are in different domains of the first dimension.
The results illustrate that evaluating the quality of OTS components carefully and
doing a thorough black-box testing in the selection phase may help to reduce the effect
of requirements changes. Integrating the unfamiliar OTS components first helped to
facilitate requirements negotiation with the customer. If the developers have
possibilities to (re) negotiate the requirements with their customer, it will also avoid the
occurrences of OTS components being unable to satisfy requirements changes.
201
Appendix A
Fig. 9. Component loading plots of the requirements risks
5.3.4 Debugging and deployment risks – R8 and R9
For the debugging and deployment risks, the results of the OVERALS show that 52% of
the actual fit is accounted for by the first dimension and the 48% by the second
dimension. The component loading plots are shown in Figure 10 and detailed
component loading data is shown in Table 5 in Appendix B.
Results show that R8 (R_DefLoc) is much more loaded in the second dimension
(.961) than in the first dimension (.223). R9 (R_ComComp) is much more loaded in the
first dimension (-.829) than in the second dimension (.303).
R8 (R_DefLoc) is in the positive domain of the second dimension and Rm5
(Rm_LearnEff) is in the peak of the negative domain. R9 (R_ComComp) is in the
negative domain of the first dimension and Rm8 (Rm_IncTest) is in the peak of the
positive domain.
The results illustrate that more effort on learning the OTS components helped to
solve the most frequent risk, i.e., identifying whether the bugs were inside or outside the
OTS components. Starting the integration testing early helped to mitigate the possible
deployment problems.
202
Appendix A
Fig. 10 Component loading plots of the deployment and debugging risks
5.3.5 Maintenance risks – R10 and R11
For the maintenance risks, we used the same methods as others. However, the
knowledge keeper variable is added as a single nominal with the value “Yes” (coded as
1) and “No” (coded as 2). The knowledge keeper variable is added in the same set as
Rm9 to Rm11. The maintenance risks, i.e., R10 (R_MaintPlan) and R11 (R_CompUpd)
are added in the second set. The results of the OVERALS show that 53% of the actual
fit is accounted for by the first dimension and the 47% by the second dimension. The
component loading plots are shown in Figure 11 and detailed component loading is
shown in Table 6 in Appendix B.
Results show that R10 (R_MaintPlan) is loaded weakly both in the first dimension
(.286) and in the second dimension (.072). R11 (R_CompUpd) is much more loaded in
the first dimension (-.622) than the second dimension (.158).
The R11 (R_CompUpd) is loaded in the negative domain of the first dimension and
Rm9 (Rm_UpdFoll) is in the peak of the corresponding positive domain. It illustrates
that local experts following the update of the OTS component helped to make the
system up-to-date with the newest OTS components.
203
Appendix A
Fig. 11. Component loading plots of the maintenance risk
5.3.6 Provider support risks – R12 and R13
For the provider support risks, we used the same methods as the maintenance risks. The
knowledge keeper variable is also added in the same set as Rm9 to Rm11. The provider
support risks, i.e., R12 (R_ProvInf) and R13 (R_MissSupp) are added in the second set.
The results of the OVERALS show that 52% of the actual fit is accounted for by the
first dimension and the 48% by the second dimension. The component loading plots are
shown in Figure 12. The detailed component loading data is shown in Table 7 in
Appendix B.
It shows that the risk R12 (R_ProvInf) is much more loaded in the first dimension (-
.839) than in the second dimension (-.012). The R13 (R_MissSupp) is much more
loaded in the second dimension (-.624) than the first dimension (-.117).
R12 (R_ProvInf) is in the negative domain of the first dimension. The knowledge
keeper is also in the peak of the negative domain. The analysis on the centroids of the
variable “knowledge keeper” shows that the projects without a knowledge keeper (code
2 in the Figure 1 in the Appendix B) are in the lower left domain as the project with
high occurrences of risk R12 (coded 5 or 4 in the Figure 2 in Appendix B). It means that
projects without the knowledge keeper have more problems on getting the information
of providers than the project with this role.
R13 (R_MissSupp) is in the negative domain of the second dimension and Rm11
(Rm_ProvWtch) is in the peak of the corresponding positive domain.
The results reveal that the experience from the knowledge keeper and maintaining a
continuous watch on the providers’ support ability and reputation is an effective way to
ensure the providers’ support.
204
Appendix A
Fig. 12 Component loading plots of the provider support risks
5.4 Sensitivity analysis
Nonlinear CCA is one kind of the Alternating Least Squares methods [Gifi90],
[Burg94]. All least squares techniques are prone to be over sensitive to outliers, and the
outlier problem is manifested in nonlinear CCA in a terms of variable categories with
small number of observations (small usually being defined as less than about five
observations). To avoid the outlier problem, samples with missing data are simply
deleted. In addition, we checked the sensitivity of the results in two steps:
- The first step is to identify abnormal/unusual data points by using the plot of
object scores (see Figures 3 to 8 in Appendix B). Results show that there are no
outliers that might tend to dominate the solution.
- The second step to verify our conclusion on the random subsets of the data. We
deleted one data point every 10th (i.e., 10th, 20th, 30th etc.) and run the same
analysis on the remaining data, which accounts for about 90% of the original
data. The conclusions based on the remaining data are the same as the
conclusions shown in sections 5.3.1 to 5.3.6.
5.5 Analysis on possible moderator variables
Since we put data from three countries together in our data analysis, we wonder whether
the national differences will affect the direction and/or strength of the relationship
between the risk management activities and risks occurrences we discovered. It is
therefore necessary to verify the moderator effect of the country differences on our
conclusions. As we also put the data from projects using COTS component and projects
using OSS components into one analysis, the same validity analysis is done to figure out
whether the differences between COTS and OSS will have moderator effect on our
conclusions.
205
Appendix A
To analyze the effect of the national and OTS differences, we added the category
variable country and OTS types as the third set in each correlation between the risks and
risk management activities. The results show that our conclusions on the requirements
risks indeed are affected by the country variable. The component loading plots are
shown in Figure 13. It shows that the correlation between R7 (R_ReqRen) and Rm7
(Rm_UnfFirst) in Figure 9 is not valid anymore. Therefore, we need to interpret these
conclusions cautiously in different countries. The further analysis on the centroids of the
variable country and the centroids of R7 shows that Italy (coded 3 in Figure 9 in
Appendix B) is in the positive domain of the first dimension. It means that it has fewer
occurrences of R7 (R_ReqRen) (coded 1 in Figure 10 in Appendix B). On the other
hand, Germany (coded 1 in the Figure 9 in Appendix B) is in the negative of the first
dimension, which means that it has more occurrences of R7 (R_ReqRen) (coded 5 or 4
in Figure 10 in Appendix B). The occurrences of R7 (R_ReqRen) in Norway are
between Germany and Italy. The possible explanation is from the business etiquette in
different countries. The Geert Hofstede study [Geert06] indicates that “Germans do not
like surprises. Sudden changes in business transactions, even if they may improve the
outcome, are unwelcome.” On the other hand, a known Italian tacit is to dramatically
change demands at the eleventh hour to unsettle or test the flexibility of their
counterparts [kwintessential06].
Fig. 13 Component Loading Plots of Effect of Country on the Requirements Risks
5.6 Risks and effective risk management activities
The result of the correlations between the risks and risk management activities are
summarized in the Table 3.
206
Appendix A
Table 3 Summary of the Risks and Most effective Risk Management Activities
Results in Table 3 show that some risk management actives are more effective on
mitigating possible risks than the other activities. However, the effectiveness of the
activities may not be different valid in different countries.
6 Discussion
6.1 Mitigation of the cost estimation risks
Both the effort in selecting OTS components and the effort to integrate them sufficiently
are difficult to estimate [Boehm03], [Mitre05]. Basili and Boehm [Basili01]
hypothesized that COTS-based development is currently a high-risk activity, with effort
and schedule overruns exceeding non-CBS software overruns. The results of the first
research question show that the wrong estimation of the integration effort happened
more frequently than other problems.
To estimate the selection effort correctly, Boehm et al. [Boehm03] propose to
identify the core “showstopper” requirements and filter all the OTS candidates that do
not meet these during the initial assessment and then proceed for a more detailed
assessment with the remaining OTS candidates. Although this method may be
applicable to limit the number of OTS components that need to be evaluated, it is still
difficult to estimate the possible effort to compare the limited OTS candidates, because
both technical and non-technical issues of the OTS components need to be evaluated
and compared [Rose03]. Our findings show that in case the effort on learning in the
OTS components has been seriously considered, it will be easier to estimate the
selection effort correctly. Boehm et al. [Boehm03] found that one common error in
OTS-based development is the overly optimistic OTS learning curve and propose that a
most likely OTS learning curve must be accounted for during planning and scheduling.
207
Appendix A
Initially, the OTS component’s functional capability must be well understood, which
can be a daunting task because there may be a significant amount of functionality to
learn and that functionality might involve new technologies to the team. Also, there is
more to learn than just the functionality. The methods of integration that the OTS
component employs must be understood sufficiently to ensure that the integration can
be succeeded into the current architecture. Since the intention of using OTS components
is to shorten time-to-market and to save development cost, OTS component users are
eager to acquire the components and to integrate them as soon as possible without
understanding them thoroughly. Rose [Rose03] pointed out that developers are usually
working with one or more OTS components for which they most likely have only a
partial understanding. To improve the estimation on the OTS integration effort, the
proposed activity is to use OTS-sensitive analytical and budget processes [Rose03],
[Faa03], such as COCOTS [Abts00]. In our study, we did not investigate the effect of
using COCOTS because our pre-study [Li04] did not find any companies using it.
Therefore, we did not investigate this activity because we wondered whether this
activity has been popularly used in industry. Although the results shown in Figure 6
reveal that the risk management activity Rm5 (i.e., effort in learning OTS component
was seriously considered in effort estimation) has been widely used, we still wonder
whether there are good tools to help the project managers to estimate the learning effort
precisely.
6.2 Mitigation of the quality risks
To get a high-quality system, it is important to ensure that every component in the

system does not bring negative effects on the whole system. The results shown in Figure
4 illustrate that OTS components rarely have a negative effect on system quality.
Our results on the second research question show that the black-box testing, and
early integration testing, which have been very popular in practice, helped to avoid
possible quality problems. Boehm et al. [Boehm03] discovered that the black-box
testing is always imposed in OTS-based development and suggested that a risk based
testing plan must be developed and accounted for in the project life cycle schedule. In
the COTS component market, the salesman may make faulty claims about the
functional and non-functional characters of their products [Boehm03], [Faa03],
[Mitre05]. Thus, the users need to do some testing themselves to ensure that the quality
of the component is the same as that claimed by the vendor. For the OSS components,
there is never enough information to illustrate the full picture of the functions and the
quality of the components. Thus, users need to test and evaluate the issues they are
interested in by either reading the source code or doing black box testing.
However, the performance of the system highly depends a lot on the interaction
between components and the environment, such as the hardware and the workload. It is
therefore difficult to predict and evaluate by just incremental integration testing. Our
investigated risk mitigation activities did not show effective reduction on the OTS
components’ possible negative effect on system performance.
Although other solutions have been proposed to mitigate the possible quality issues
of the overall system, such as selecting the OTS based on architecture instead of
functionalities [Rose03], [Torchiano04], [Li04], or leverage the commercial
infrastructure [Faa03]. Our results did not show a strong effect from these activities. A
208
Appendix A
possible reason is that most OTS components investigated in our studies followed the
industrial standards or infrastructures - such as COM, CORBA, .Net, and EJB - by
default. The possible quality problems caused by the architecture mismatch itself are
therefore minimized.
6.3 Mitigation of the requirements risks
Another popular issue on OTS component integration is the requirements changes and
mismatches with OTS component functionalities [Boehm03], [Faa03], [Rose03]
[Mitre05].
Since it is difficult to change the OTS component to follow the customers’ changes
in requirements, it is important for the OTS component integrator to be able to negotiate
requirements with the customer [Boehm03], [Rose03], and involving customers into
make vs. acquire decision and OTS component selection [Faa03]. However, our results
show that very few integrators have actually involved customers into the make vs.
acquire decision and the OTS component selection. Results of the study [Li04] illustrate
that it is not easy to negotiate requirements with the customer in OTS component-based
development. One reason is that customer only cares about the final product without
willingness or capability of knowing the technical details.
To mitigate the possible requirements risks, our results show that careful evaluation
and testing of OTS components in the selection phase helped to increase the possibility
to follow the customers’ requirements changes. The possible explanation is that people
understand the OTS components better when they did testing and quality evaluation in
the selection phase. They therefore know the capability and the possible extensions of
the OTS components.
Another possible activity is to first integrate the unfamiliar, i.e., high risk or critical
OTS components [Rose03], [Li04]. Integrating the critical components first helped to
provide the solid foundation for the architecture of the system [Rose03]. Integrating the
high risk or most complex components first can help to find problems in the early phase
of the project, which may give the integrator leeway to negotiate with their customers to
redefine the requirements [Lauesen06]. To negotiate the requirements with the
customer, educating the customer in requirements negotiation will be critical, but this
must be accomplished in a manner that is politically viable for the particular customer
[Rose03]. The possibility of negotiation depends on the willingness of the customer to
negotiate issues and to accept changes. Our results illustrate that customers in certain
business culture, such as in Germany, are more difficult to accept requirements
negotiation than others.
6.4 Mitigation of the debugging and deployment risks
Our results show that the difficulty in locating the cause of defects happened more
frequently than other problems. Kotonya and Rashid [Kotonya01] point out that OTS
component offering similar functionalities may have very different system resource
requirements (i.e., memory and processor requirements). When a problem is identified,
it may not be obvious where the root cause of the problem is located. The problem
could be in one of the components, it could be in the integration code, or it could be a
misinterpretation of the semantic operation of a component interface [Rose03].
209
Appendix A
The results of the research question RQ3 show that this risk can be mitigated by
giving developers enough time to learn and understand the OTS components. Since both
COTS and OSS components are acquired from a third-party organization, it will require
a lot of time for the users to understand the underlying design and code in these
components. It is especially difficult for the COTS component users to use the
components properly when source code is not available. In case there are defects in the
OTS components or the interactions between the OTS components are not properly
controlled, it may take longer time for the component users to figure out the problems
than they had expected.
For the possible deployment problem, our findings indicate that an effective way of
controlling it is to start integration testing as early as possible. In the early days of
integrating OTS into a system solution, there was a false assumption that the need for
integration testing was either minimal or not needed at all. It seems that there was a
belief (actually a false hope) that once the OTS components were installed, the system
would be operational [Rose03]. The OTS providers are only knowledgeable in their
own OTS components. If there is no strong evidence that problems in the integrated
system are caused by a specific OTS component, the component vendor may not want
to help the users to debug and solve the problems. The benefit of doing integration
testing after one OTS component is integrated is that it is easy to conclude that the
problems lie in a specific component. An additional possible mitigating activity is to do
some analysis to determine the order in which the OTS components are to be integrated
[Spc01].
6.5 Mitigation of the maintenance risks
Maintaining an OTS component-based system is difficult. The post-deployment costs

may exceed the development costs [Basili01]. The different customer-vendor evolution
cycles may result in an uncertainty about how often the OTS components in a system
may have to be replaced and the extent of the impact of such a change on the rest of the
system [Kotony01, [Rose03], [Mitre05], [Faa03].
One risk mitigation solution proposed is to build an explicit, formal strategic
relationship that includes mutual non-disclosure agreements to ensure that the provider
gives full disclosure of the products future upgrades [Rose03]. However, this strategy is
only applicable in a few situations. In case the customer is not regarded as a major one
by the COTS component provider, it is difficult to build such a relationship. In the case
of using OSS components, it is also difficult to build such a relationship, because the
OSS components are updated by volunteers and nobody knows what functions may be
added in future releases.
Another risk mitigation activity is to have senior analysts (serving as knowledge
keepers) directly involved in the analysis of the updates and deriving effective field
upgrade procedures and schedules [Rose03], [Li04]. This is a non-trivial system analyst
task that requires appropriate level of expertise, as there is very little in the literature to
help guide the analysts to a feasible solution [Rose03]. The existence of the knowledge
keeper has been investigated and show effective results on mitigating the possible risks
due to the OTS component evolution. Since the knowledge keepers know the current
component version being used, it is possible for them to evaluate the new versions and
to figure out the benefits and risks of updating the current version with the newer one.
210
Appendix A
The personal capability and experience remain the dominant factors influencing OTS-
based development [Basili01]. Therefore, it is important to involve OTS-
knowledgeable individuals in all analytical processes [Faa03].
However, planning maintenance of an OTS component-based system is very difficult
because the OTS component users have no control of the release cycle of the OTS
components being used. Since the release cycle of the OTS component depends on the
marketing strategy of the component vendors, the component users usually have no idea
when the new release of the OTS component will be published. Our investigated risk
management activities focus on the activities that can be controlled by the component
users. Such risk management activities were not effective in mitigating the maintenance
plan risk.
6.6 Mitigation of the provider support risks
Another issue in OTS component-based development is to get enough technical support

from the provider. Although the users can have access to the source code when using
the OSS components, few of them have actually modified the code to meet their
requirements [Li05]. It is also advised to avoid modifications of OTS components, even
when it is possible [Faa03]. COTS component users can get support from the provider
by signing a support contract. However, COTS providers may still stop to support their
products due to different reasons, or even go out of business [Rose03], or the actual
version of a COTS component is too old to be supported [Basili01]. For the OSS
components, some of them are currently well supported by the volunteers in the OSS
projects. However, it does not ensure that the users can get the same level of support in
the future. The users have no control of the technical support they want to get from the
OSS community. As our study tried to find out the common risk mitigation activities for
both COTS and OSS components, we did not investigate the contract and licensing
techniques, which is not applicable for the OSS user. We therefore looked more on the
activities that are under control among OTS component users.
For the provider support risks, our results show that the most important remedy is
still to select the right providers. It is necessary to perform continuous watch on the
support reputation of the OTS component providers, as proposed in [Faa03], [Mitre05].
In addition, it is important to trace the history of the provider support by consulting the
knowledge keeper.
6.7 Possible Threats to Validity
6.7.1 International Validity Threats
We have promised respondents in this study a final report and a seminar (in Norway) to
share experience. The respondents were persons who wanted to share their experience
and wanted to learn from others. In general, we think that the respondents answered the
questionnaire truthfully. However, different persons in the same project might have
different opinions on the same project. Asking only one person in each project might
not be able to get the whole picture of the project. Although all questionnaires were
filled in through the web tool in Norway, questionnaires were mostly filled in by
211
Appendix A
telephone interviews in Germany. The different fill-in methods may bring biases to the
results of this study (a so-called “mode problem” in surveys).
6.7.2 External Validity Threats
Although we used different techniques to select samples in different countries, this

study is the first survey on component-based software engineering and indeed software
engineering, using stratified-random sampling of the IT companies in several countries.
However, this study focused OTS components that satisfy our component definitions in
Section 3.3.1. Conclusions may be different in projects using complex and large OTS
packages, such as ERP, CRM, or content management systems solutions.
6.7.3 Construct Validity Threats
In this study, most variables and alternatives are taken directly, or with little
modification, from existing literatures. However, we may omit some possible risks and
risk management activities because they were not mentioned in previous literatures. The
questionnaire was pre-tested using a paper version by 10 internal experts and 8
industrial respondents before being published on the SESE tool [Sese04]. About 15%
questions have been revised based on pre-test results.
6.7.4 Conclusion Validity Threats
This study is a state-of-the-practice study. We studied what had happened in industrial

projects. However, we did not investigate the cause-effect relation of the phenomena
discovered in this study. In our study, we assume that the apriori probabilities of typical
risks are similar in most OTS component-based projects and examined only the
relations between the performed risk management activities and the actual risk
incidence after projects are finished. Since different project might have slightly different
a priori probability of typical risks, our assumption might bring biases to our
conclusions.
7 Conclusion and future work
OTS component-based development brings both benefits and risks. Although several
studies have investigated and proposed risk management activities in OTS component-
based development, few studies have verified and compared the effects of the risk
management activities on the corresponding risks. This study is an exploratory study on
examining the occurrences of typical risks in OTS-based projects and relations between
the occurrence of these risks and the risk mitigation activities performed. The intention
is to find out the most effective activities that helped to mitigate corresponding risks in
practice. The data was collected from 133 OTS component-based projects in Norway,
Italy and Germany. The analysis was performed with nonlinear canonical correlation
analysis, i.e., OVERALS in SPSS.
The results of this study show that two of the most frequent problems in OTS-based
projects are the incorrect estimation of the integration effort and inefficient debugging.
212
Appendix A
The least frequent problems are relevant to the negative effect of the OTS components
on the quality of the whole system. We also conclude that:
- To increase the accuracy of effort estimation, it is important to allocate enough

effort to learn and understand the OTS components.
- To avoid the possible quality problems and to increase the efficiency of locating
the defects, it is important to do black-box testing as part of OTS component
selection and to do integration testing as early as possible.
- To mitigate the problems due to customers’ requirements changes, it is
important to learn the OTS components thoroughly in the selection phases and to
integrate the unfamiliar OTS components first. It is also important to be able to
negotiate changed requirements with the customers.
- To increase the efficiency of debugging and system deployment, it is necessary
to understand the components well and to do integration testing early.
- It is critical to keep and share the knowledge about OTS components and the
component providers inside the organization. It shows to help mitigate the
possible risks in system maintenance and evolution.
As this is an exploratory study, further studies are needed to confirm our conclusions.
The assumption of this study is that the a priori risk probabilities of typical risks in the
investigated projects are similar. However, it is possible that there are minor differences
in the a priori risk probabilities between projects. It is therefore necessary to follow
several OTS component-based projects from start by case studies. In the future, we plan
to examine the several projects by estimating risk probabilities before the project start,
following the project execution, and measuring the risk occurrences after the projects
are finished. The intention is to investigate the cause-effect of the risk management
activities on the corresponding risks.
Moreover, some of the conclusions of this study need further clarification. For
example, which knowledge needs to be kept and shared in the organization and how to
share it? How to estimate the possible learning effort needed when using the unfamiliar
OTS components? These research questions need to be further studied by case studies
or experiments.
Acknowledgment
The authors wish to thank Barbara Kitchenham for commenting on the data analysis
methods.
References
[Abts00] C. Abts, B. W. Boehm, and E. B. Clark, “COCOTS: A COTS Software

Integration Lifecycle Cost Model - Model Overview and Preliminary Data Collection
Findings,” Technical report USC-CSE-2000-501, USC Center for Software
Engineering, http://sunset.usc.edu/publications/TECHRPTS/2000/usccse2000-
501/usccse2000-501.pdf, 2000.
213
Appendix A
[Barki93] H. Barki, S.Rivard, and J. Talbot, “Toward an Assessment of Software

Development Risk,” J. Management Information Technology, vol. 22, no. 2, pp. 359-
371, Dec. 1993.
[Basili01] V. R. Basili, and B. W. Boehm, “COTS-Based Systems Top 10 List,” IEEE
Computer, vol. 34, no. 5, pp. 91-93, May 2001.
[Basili02] V. R. Basili, M. Lindvall, I. Rus, C. Seaman, and B. W. Boehm, “Lessons-
Learned Repository for COTS-Based SW Development,” Software Technology
Newsletter, vol. 5, no. 3, pp. 4-7, available at: http://fc-md.umd.edu/ll/index.asp,
2002.
[Bass00] L. Bass, C. Buhman, S. Comella-Dorda, F. Long, J. Robert, R. Seacord, and
K. Wallnau, “Volume I: Market Assessment of Component-based Software
Engineering,” SEI Technical Report number CMU/SEI-2001-TN-007, available at:
http://www.sei.cmu.edu/, 2001.
[Boehm88] B. W. Boehm, “A Spiral Model of Software Development and
Enhancement,” IEEE Computer, vol. 21, no. 5, pp. 61-72, May 1988.
[Boehm89] B. W. Boehm, Software Risk Management, Tutorial, IEEE CS Press, 1989.
[Boehm91] B. W. Boehm, “Software Risk management: Principles and Practices,”
IEEE Software, vol. 8, no. 1, pp. 32-41, Jan.1991.
[Boehm03] B. W. Boehm, D. Port, Y. Yang, and J. Bhuta, “Not All CBS Are Created
Equally COTS-intensive Project Types,” Proc. Second Intl. Conf. on COTS-Based
Software Systems, pp. 36-50, 2003.
[Brakk90] C.J.F. Ter Brakk, “Interpreting canonical correlation analysis through biplots
of structure correlations and weights,” Psychometrika, vol. 55, pp. 519-531, 1990.
[Burg94] E.V.D. Burg, J. D. Leeuw, and G. Dijksterhuis, “OVERALS – Nonlinear
Canonical Correlation with k sets of Variables”, J. Computational Statistics & Data
Analysis, vol. 18, pp. 141-163, 1994.
[Bush90] M. E. Bush, N. E. Fenton, “Software Measurement: A Conceptual
Framework”, J. System and Software, vol. 12, no. 3, pp. 223-231, 1990.
[Carr93] M. Carr, S. Kondra, I. Monarch, F. Ulrich, and C. Walker, “Taxonomy-Based
Risk Identification,” Technical Report SEI-93-TR-006, SEI, Pittsburgh, USA, 1993.
[CBSEnet04] Component-based Software Engineering Network. CBSEnet, available at
http://www.cbsenet.org, 2004.
[Chittister96] C. G. Chittister and Y. Y. Haimes, “System Integration via Software Risk
Management,” IEEE Trans Systems, Man, and Cybernetics, vol. 26, no. 5, pp. 521-
532, Sep. 1996.
[Conradi05] R. Conradi, J. Li, O. P. N. Slyngstad, C. Bunse, M. Torchiano, and M.
Morisio, “Reflections on Conducting an International CBSE Survey in ICT Industry,”
Proc. of the forth Intl. Symp. on Empirical Software Engineering , pp. 214-223, 2005.
[Crnkovic02] I. Crnkovic and M. Larsson, “Building Reliable Component-Based
System,” Artech House, 2002.
[Faa03] COTS risk factor, available at
http://www.faa.gov/aua/resources/cots/Guide/CRMG.htm, 2003.
[Fitzgerald04] B. Fitzgerald, “A Critical Look at Open Source,” IEEE Computer, vol.
37, no. 7. pp. 92-94, July 2004.
[Geert06] Geert Hofstede™ Cultural Dimensions, available at http://geert-
hofstede.international-business-center.com/, 2006.
214
Appendix A
[Gemmer97] A. Gemmer, “Risk Management: Moving Beyond Process,” IEEE

Computer, vol. 30, no. 5, pp. 33-41, May, 1997.
[Ghosh02] S. Ghosh, “Improving Current Component-Based Development Techniques
for Successful Component-Based Software Development,” Proc. Intl. Conf. Software
Reuse, Workshop on Component-based Software Development Processes, 2002,
available at: http://www.idt.mdh.se/CBprocesses/
[Gifi90] A. Gifi, Nonlinear Multivariate Analysis, John Wiley & Sons, 1990.
[Glass01] R. L. Glass, “Frequently Forgotten Fundamental Facts about Software
Engineering”, IEEE Software, vol. 18, no. 3, pp. 110-112, May/June, 2001.
[Hecht04] H. Hecht, Systems Reliability and Failure Prevention. Artech House
Publishers, 2003.
[INCO00] INCO project description, http://www.ifi.uio.no/~isu/INCO, 2000.
[Jacobson97] I. Jacobson, M. Griss, and P. Jonsson, “Software Reuse: Architecture,
Process and Organization for Business Success,” Addison-Wesley, 1997.
[Kansala97] K. Kansala, “Integrating Risk Assessment with Cost Estimation,” IEEE
Software, vol. 14, no. 3, pp. 69-74, May/June 1997.
[Keil98] M. Keil, P.E. Cule, K. Lyytinen, and R.C. Schmidt, “A Framework for
Identifying Software Project Risks,” Communications of the ACM, vol. 4, no. 11, pp.
76-83, Nov. 1998.
[Kitchenham03] B. Kitchenham, S. L. Pfleeger, “Principles of Survey Research Part 6:
Data Analysis,” Software Engineering Notes, vol. 28, no. 2, pp. 24-27, March 2003.
[Kotonya01] G. Kotonya and A. Rashid, “A Strategy for Managing Risk in Component-
based Software Development,” Proc. 27th EUROMICRO Conf, pp. 12-21, 2001.
[Kwintessential06] http://www.kwintessential.co.uk/etiquette/doing-business-italy.html,
2006.
[Lawton02] G. Lawton, “Open Source Security: Opportunity or Oxymoron?” IEEE
Computer, vol. 35, no. 3, pp. 18-21, March 2002.
[Li04] J. Li, F. O. Bjørnson, R. Conradi, and V. B. Kampenes, “An Empirical Study of
Industry, “ Proc. 10th IEEE Intl. Metrics Symp. pp. 72-83, 2004.
[Li05] J. Li, R. Conradi, O. P. N. Slyngstad, C. Bunse, U. Khan, M. Torchiano, and M.
Morisio, “Validation of New Theses on Off-The-Shelf Component Based
Development,” Proc. 11th IEEE Intl. Metrics Symp., pp. 26, Sept. 2005.
[Lauesen06] S. Lauesen, “COTS Tenders and Integration Requirements,” J.
Requirements Engineering, vol. 11, no. 2, pp. 111-122, 2006.
[Madachy97] R. J. Madachy, “Heuristic Risk Assessment Using Cost Factors,” IEEE
Software, vol. 14, no. 3, pp. 69-74, May/June 1997.
[Madanmohan04] T.R. Madanmohan and R. De’, “Open Source Reuse in Commercial
Firms,” IEEE Software, vol. 21, no. 1, pp. 62-69, January/February, 2004.
[Manly04] B. F. J. Manly, “Multivariate Statistical Methods: A Primer,” 3rd edition.
Chapman & Hall/CRC, 2004.
[Michailidis98] G. Michailidis and J. d. Leeuw, “The Gifi System of Descriptive
Multivariate Analysis,” J. Statistical Science, vol. 13, no. 4, pp. 307-336, 1998.
[Mitre05] Common Risks and Risk Mitigation Actions for a COTS-based System,
www.mitre.org/work/sepo/toolkits/risk/taxonomies/files/CommonRisksCOTS.doc,
2005.
215
Appendix A
[Morisio00] M. Morisio, S. Seaman, A. Parra, V. Basili, S. Kraft, and S. Condon,

“Investigating and Improving a COTS-Based Software Development Process,” Proc.
22nd Intl. Conf. on Software Engineering, pp. 31-40, 2000.
[Moynihan97] T. Moynihan, “How Experienced Project Managers Assess Risk,” IEEE
Software, vol. 14, no. 3, May/June 1997.
[Ropponen00] J. Ropponen and K. Lyytinen, “Components of Software Development
Risk: How to Address Them? A Project Manager Survey,” IEEE Trans. Software
Engineering, vol. 26, no. 2, pp. 98-112, February 2000.
[Rose03] L. C. Rose, “Risk Management of COTS Based System Development,”
Component-Based Software Quality - Methods and Techniques, Springer – Verlag
LNCS vol. 2693, pp. 352-373, 2003.
[Ruffin04] M. Ruffin and C. Ebert, “Using Open Source Software in Product
Development: A Primer,” IEEE Software, vol. 21, no. 1, pp. 82-86. January/February
2004.
[Sese04] Sese tool, available at http://www. Simula.no, 2004.
[Sherer95] S. A. Sherer, “The Three dimensions of Software Risk: Technical,
Organizational, and Environmental,” Proc. 28th Hawaii International Conference on
System Sciences, pp. 369-378, 1995.
[Ssb02] Norwegian Census Bureau (SSB), Oslo, ICT company data, available at
http://www.ssb.no/emner/10 /03/ikt/, 2002.
[Sommerville04] I. Sommerville, Software Engineering, 7th edition, Addison-Wesley,
2004.
[Spc01] Software Productivity Consortium, “Phased-Integrated COTS Approach,”
Technical Report SPC-2001006-CMC, http://www.software.org/pub/pic, 2001.
[Spearman1904] C. Spearman, “The Proof and Measurement of Association between
Two Things”, American Journal of Psychology, vol. 15, pp. 72-101, 1904.
[Spss02] SPSS Categories 11.0, available at www.spss-sa.com, 2001.
[Spss06] SPSS Categories 14.0, available at http://www.spss.com/categories/, 2006.
[Szyperski02] C. Szyperski, D. Gruntz, and S. Murer, Component Software – Beyond
Object-Oriented Programming. Addison-Wesley, 2002.
[Torchiano04] M. Torchiano and M. Morisio, “Overlooked Facts on COTS-based
Development,” IEEE Software, vol. 21, no. 2, pp. 88-93, March/April 2004.
[Verner05] J. M. Verner and W. M. Evanco, “In-House Software Development: What
Project Management Practices Lead to Success?” IEEE Software, vol. 22, no. 1, pp.
86-93, Jan./Feb. 2005.
[Vigder96] M. Vigder, M. Gentleman, and J. Dean, “COTS Software Integration: State
of the Art”, Technical Report NRC No. 39190, 1996.
[Vitharana03] P. Vitharana, “Risks and Challengs of Component-Based Software
Development,” Communications of the ACM, vol. 46, no. 8, pp. 67-72, August 2003.
[Voas98a] J. Voas, “COTS Software – the Economical Choice?” IEEE Software, vol.
15, no. 2, pp. 16-19, March/April 1998.
[Voas98b] J. Voas, “The challenges of Using COTS Software in Component-Based
Development,” IEEE Computer, vol. 31, no. 6, pp. 44-45, June 1998.
[Wallace04] L. Wallace and M. Keil, “Software Project Risks and Their Effect on
Outcomes,” Communications of the ACM, vol. 47, no. 4, pp. 68-73, April 2004.
216
Appendix A
Appendix A: The rationale of using Nonlinear CCA
Standard canonical correlation analysis (CCA) [Manly04] is an extension of multiple

regressions, where the second set does not contain a single dependent variable but
instead contain multiple dependent variables. The Nonlinear CCA approach expands the
canonical correlation analysis in three crucial ways. First, Nonlinear CCA allows more
than two sets of variables. Second, variables can be scaled as nominal, ordinal, or
numerical. As a result, nonlinear relationships between variables can be analyzed.
Finally, instead of maximizing correlations between the variable sets, the sets are
compared to an unknown compromise set that is defined by the canonical variate
[Spss02].
An efficient solution to the nonlinear CCA problem was first proposed by Gifi
[Gifi90]. It was later extended to generalized canonical analysis with more than two sets
of variables. The generalized nonlinear CCA program, called OVERALS is available in
the SPSS CATEGORIES [Spss06] program suites. The OVERALS simultaneously
determines both (1) optimal re-scalings of the nominal and ordinal variables and (2)
explanatory variable weights, such that the linear combination of the weighted re-scaled
variables in one set has the maximum possible correlation with the linear combination
of weighted re-scaled variables in the second set. Both the variable weights and optimal
category scores are determined by minimizing a loss function derived from the concept
of “meet” in lattice theory. Graphical representations are very important in interpreting
this plethora of results. In fact, several authors have argued that graphical
representations are crucial in understanding the results of linear multivariate methods,
particularly CCA, because patterns in the data can best be detected by the eyes
[Brakk90].
In our study, a possible data analysis is to find out the correlations between the risk
management activities performed and the occurrences of risks, using Spearman
correlation analysis [Spearman1904]. To use this method, we must assume that the risks
are independent and the effects of risk management activities are independent.
However, we generally believe that each risk management activities performed may
help to mitigate several risks. Therefore, we cannot ensure that the effects of risk
management activities performed are independent.
Another possible data analysis method is to do a multiple regression analysis with the
risk management activities as the independent variables and one specific risk as the
dependent variables. The weights of each risk management activity in the regression
equation can be used to compare their effectiveness. However, this method also assumes
that the occurrences of different risks are independent. Since no previous studies have
investigated the relations between these risks, we cannot ensure that they are
independent of each other. Thus, we just divided them into different groups according to
the phases in the project as in the study [Rose03].
The third possible data analysis method is to do a two- step analysis. The first step is
to do a principal component analysis with the risks and risks management activities to
figure out several risk factors and risk management factors. The second step is to find
out the correlations between the risk factors and the risk activity factors. The same
method has been used by [Ropponen00] to analyze risk management issues. However,
Kitchenham and Pfleeger have commented this method on [Kitchenham03]: Principal
component analysis can be used either to test a hypothesis or to generate hypotheses.
217
Appendix A
However, it cannot do both things at once. Thus, the risk factors identified using
principal component analysis represent hypotheses/theories that should have been
confirmed by an independent study before investigating risk strategies. The underlying
problem in this study [Ropponen00] (and many others) was trying to do too much in
one investigation.
Appendix B: Supplement Tables and Figures
Table 1 Examples of OTS components

Name of the OTS OSS / Function Technique
components COTS
Log4j OSS Logging and tracing Library
Snoopy OSS RSS Parser PHP class
Crystal Report COTS Report design and development J2EE, .NET, COM
MapObjects COTS Collection of embeddable Library
mapping and GIS components
Table 2 Component Loading of Effort Estimation Risks with RM Activities

Set Dimension
1 2
1 Rm1 ,035 -,242
Rm2 ,349 ,095
Rm3 ,214 ,330
Rm4 ,400 ,024
Rm5 ,562 -,021
Rm6 ,164 ,402
Rm7 ,444 -,060
Rm8 -,087 ,139
2 R1 -,844 -,496
R2 -,859 ,424
Table 3 Component Loading of Quality Risks with RM Activities

Set Dimension
1 2
1 Rm1 ,286 ,037
Rm2 -,217 -,356
Rm3 ,071 -,391
Rm4 ,384 -,838
Rm5 ,055 ,128
Rm6 ,435 -,037
Rm7 -,202 ,096
Rm8 ,456 ,059
2 R3 -,530 ,264
R4 -,643 -,258
R5 ,087 ,072
Table 4 Component Loading of Requirements Risks with RM Activities
218
Appendix A
Set Dimension
1 2
1 Rm1 -,271 ,042
Rm2 -,039 -,149
Rm3 ,226 -,414
Rm4 -,168 ,596
Rm5 -,408 ,165
Rm6 -,428 ,301
Rm7 ,538 ,214
Rm8 ,309 ,281
2 R6 ,647 -,620
R7 -,553 -,761
Table 5 Component Loading of Debugging and Deployment Risks with RM

Activities
Set Dimension
1 2
1 Rm1 -,029 -,170
Rm2 -,576 ,073
Rm3 -,111 -,194
Rm4 -,033 -,083
Rm5 ,003 -,675
Rm6 ,140 -,230
Rm7 -,382 -,159
Rm8 ,464 -,053
2 R8 ,223 ,961
R9 -,829 ,303
Table 6 Component Loading of Maintenance Risks with RM Activities

Set Dimension
1 2
1 Knowledge keeper -,524 ,074
Rm9 ,796 ,378
Rm10 ,203 -,818
Rm11 ,240 ,144
2 R10 ,286 ,072
R11 -,622 ,158
Table 7 Component Loading of Provider Support Risks with RM Activities

Set Dimension
1 2
1 Knowledge keeper -,734 -,092
Rm9 ,199 -,139
Rm10 -,362 ,509
Rm11 ,275 ,806
2 R12 -,839 -,012
R13 -,117 -,624
219
Appendix A
Fig.1. Centroids of the Knowledge keeper in the analysis of provider support risks
Fig.2. Centroids of the R12 in the analysis of provider support risks
Fig. 3 Object Scores of the Cost Estimation Risks
220
Appendix A
Fig. 4 Object Scores of the Quality Risks
Fig. 5 Object Scores of the Requirements Risks
Fig. 6 Object Scores of the Debugging and Deployment Risks
221
Appendix A
Fig. 7 Object Scores of the Maintenance Risks
Fig. 8 Object Scores of the Provider support risks
1: Germany 2: Norway 3: Italy
Fig.9. Centroids of the countries in the analysis of requirements risks
222
Appendix A
Fig.10. Centroids of the R7 in the analysis of requirements risks
223
Appendix A
224
Appendix A
P6. Decision Making in OTS Component-Based Development

An Empirical Study on Decision Making in Off-The-Shelf
Component-Based Development
Jingyue Li1, Reidar Conradi1,2, Odd Petter N. Slyngstad1, Christian Bunse3, Maurizio
Morisio4, and Marco Torchiano4
1
2
3
4
Dip.Automatica e Informatica, Politecnico di Torino
{morisio, marco.torchiano}@polito.it
ABSTRACT: Component-based software development (CBSD) is becoming more and

more important since it promotes reuse to higher levels of abstraction. As a
consequence, many components are available being either open-source software (OSS)
or commercial-off-the-shelf (COTS). However, it is still unclear how the decision for
acquiring OSS or COTS components is made in practice. This paper describes an
empirical study on why project decision-makers selected COTS instead of OSS
components, or vice versa. The study was performed as an international survey in
Norway, Italy and Germany. It focused on decision making on using off-the-shelf
(OTS) components. We have gathered answers from 83 projects using only COTS
components and 44 projects using only OSS components. Results of this study show
significant differences and commonalities of integrating OSS or COTS components.
Moreover, the study illustrates several research questions that warrant future research.
Categories and Subject Descriptors

D.2.13 [Software Engineering]: Reusable Software – Reuse model
General Terms
Management
Keywords
Commercial-Off-the-Shelf Software, Open Source Software, Empirical Study,
Component-based Development
225
Appendix A
1. INTRODUCTION
Long-term success in commercial software development is becoming increasingly

challenging. The software industry is increasingly turning to approaches which offer a
larger return on software development effort. Component-based software development
(CBSD) [19] is one of these approaches. Besides reusing in-house built components,
software companies are using an increasing amount of COTS and OSS components.
Although both COTS and OSS component claim to save development effort, they are
different in their nature and appearance. COTS components are owned by commercial
vendors with agreed specific support. The users of COTS component normally have no
access to the source code. On the other hand, OSS components are provided by open
source communities with freely accessible source code but promise no specific support
[14].
When planning a new software project, decision makers must decide whether they
should buy a COTS component or acquire an OSS component, if it was decided to use
OTS components. To make such a decision, it is important to investigate previous
projects using such components. Unfortunately, the relevant decision-making processes
are often intuitive, and if at all, only sparsely documented. Thus, further research is
needed to examine the current state-of-the-practice and to define guidelines for OTS-
based development.
This paper investigates the state-of-the-practice in OTS-based development on use of
COTS and OSS components, by means of an international representative survey in
Norway, Italy and Germany. We have gathered answers from 83 projects using only
COTS components and 44 projects using only OSS components. Results of this study
illustrated who are using OTS components, why was it decided to use such components,
and what were the problems. Results also illustrated the commonalities and differences
in integrating COTS or OSS components.
The remainder of this paper is structured as follows: Section two presents related
research. Section three introduces research design. Section four presents obtained results
and discussion. A short summary and some future work are given in Section five.
2. RELATED WORK
COTS components promise faster time-to-market and increased productivity. At the

same time, COTS software introduces many risks, such as the unknown quality of
COTS components, or possible instability of the COTS vendor support [20]. Although
some research stated that OSS makes only sense for operating systems [5], the use of
OSS in industrial products is growing rapidly [11, 13, 16, 18]. OSS offer many
advantages: it is usually freely available for public download. The collaborative, parallel
efforts of globally distributed developers allow much OSS to be developed more
quickly than conventional software. Many OSS products are recognized for high
reliability, efficiency, and robustness [4]. Despite its wide appeal, OSS software faces
several challenges and constraints: the documentation, testing, and field support may
not be available [4], and long-term support is not guaranteed [11]. The OSS licence may
restrict the user of changing and integrating the OSS components [18]. Previous studies
have looked at using COTS and OSS components in software development [2, 13].
226
Appendix A
However, few studies have investigated a higher level question: Why do I use COTS
instead of OSS components, or vice versa?
Some studies compared the differences between COTS and OSS products per se and
concluded that there is still no empirical evidence that OSS fosters faster system growth
and that OSS is more modular than closed source software [15]. However, there were
few systematic empirical studies of the commonalities and differences in integrating
COTS vs. OSS components in a software product. In order to guide the project decision
maker in deciding between an OSS component and a COTS component, it is necessary
to empirically investigate how the decision was made and what was the result of the
decision in finished projects.
3. RESEARCH DESIGN
The empirical study on decision making OTS based development followed a three step
approach. We started from a pre-study of state-of-the-practice and the challenges in
integrating COTS components. The pre-study was performed by structured interviews
of 16 COTS projects in 13 Norwegian IT companies selected by convenience [8]. The
study reported in this paper represents the second step, and investigated decision
making in OTS component-based development by a questionnaire-based survey. This
study extended the pre-study in two dimensions. First, it included OSS. Second, the
study included randomly selected IT companies from Norway, Italy and Germany. The
third future step will focus on investigating the underlining reasons of the conclusions in
the second step and more studies on OSS projects. It will be performed with structured
interviews and case studies.
3.1 Research Questions
Based on the design of the study a couple of research questions were defined. First, we
wanted to know the profiles of those using OTS components. Therefore, the first
research question is:
RQ1: What are the commonalities and differences in profiles of projects using COTS
components vs. those using OSS components?
We then wanted to know their expectation of using OTS components. The second
research question therefore is:
RQ2: What are the commonalities and differences in the expectation of projects
using COTS components vs. those using OSS components?
After we know who and why, we need to know the actual risks/problems of using
such components. We investigated whether the motivation and expectation of using
OTS components are proper or not. So, the third research question is:
RQ3: What are the commonalities and differences in possible risks (problems) of
227
Appendix A
3.2 Research Method
In this study, we used a questionnaire to collect data. The detailed information of the
questionnaire and sample selection are reported in [3, 9, 10]
4. RESULTS AND DISCUSSION
In [10], we reported the preliminary results of this study with 71 projects using only
COTS components and 39 projects using only OSS components. Now, the data
collection is finished. In total, we gathered results from 83 projects using only COTS
components, 44 projects using only OSS components, and six projects using both COTS
and OSS components.
4.1 RQ1: What was the Profile of Projects Using OTS Components?
To answer RQ1, we investigated the commonalities and differences of some

requirements of the actual system, such as time-to-market, effort (cost), reliability,
security, performance, maintainability, new functionality (first launch in the market),
and improved functionality (over competitors).
We compared each requirement in systems using COTS vs. systems using OSS
components. The results show that there are no such significant differences. The most
emphasized requirements of the final systems were time-to-market, reliability, and
performance. Security was surprising listed as the least emphasized requirement of the
system. There are many discussions on whether OSS components are more secure than
COTS components. Raymond [17] argued that “given enough eyeballs, all bugs are
shallow.” Other argued that it is the quality, not the number, of the eyes looking at code
that count [7]. Our results show that OTS component users are still very cautious of
using either COTS or OSS components in the security-critical systems. So, software
security aspect may not be distinguished between COTS and OSS components.
However, trust in the security of third-party components may become an issue in the
future.
4.2 RQ2: What were the Expectations of Using OTS Components?
To answer RQ2, we investigated the respondent’s general expectations using OTS

components, and specific expectations of using COTS and OSS components.
The general expectations of using OTS components are extracted from our pre-study
[8] and literature reviews [4, 20]. Details of these expectations are shown in [10].
Results show again that there are no significant differences. The main expectation of
using OTS components is to shorten the time-to-market and to save development cost.
Pizka [16] reported experience on building the same system with three different
strategies, such as wrapping an OSS component, adapting/changing the source code of
the OSS component, or building the same component from scratch. Their results gave
strong support on reusing OSS components and concluded that re-implementing
existing OSS components should be avoided if there are no serious reasons for doing so.
The specific expectations of using COTS components listed in the questionnaire
were shown in [10]. Results show that COTS users believe that paid software will give
228
Appendix A
good reliability and will follow the market trend. They also believe that the COTS
vendor will provide good technical support. We also listed specific expectations of
using OSS components listed in the questionnaire. Results show that the main
motivation of using OSS components is that code could be acquired for free and that
the source code is available.
In summary, results show that business considerations, such as reducing time and
cost, are the main motivation for using OTS components. OSS component users care
more about the initial investment. Another attraction of OSS components is that the
marginal cost of scaling up is zero, i.e. using OSS does not require additional licences as
the number of installations grows [18]. This benefit is especially important when
integrating OSS components into a software product with many installation bases.
Concerning the use of domain-specific OTS components, it is important to ensure
that the evolution of the selected OTS component will follow the market trend.
Although a COTS user cannot directly influence the evolution of the selected COTS
component, results of this study show that these users still trust that the COTS vendor
will update the component according to market needs. OSS components can be changed
out of similar market reasons, but can also be changed out of political or social reasons
– not necessarily in a direction suitable to the organization [6]. It is therefore important
for OSS users to ensure that they are planning to use the software in the way it was
intended to be used and the way most other users are using it. If OSS users stray too far
away from the beaten path, the OSS developers might not want to help them when they
run into trouble, or OSS developers will not accept the changes contributed by them
[11].
4.3 RQ3: What were the Risks of Using OTS Components?
To answer RQ3, we formulated 15 possible problems (see Table 3 in [10]) on OTS-

based development, and asked the respondents if these problems had happened in their
projects. Results showed that common problems both in projects using COTS and in
those using OSS are: the effort to integrate OTS components was not satisfactorily
estimated and it was difficult to identify whether the defects were inside or outside the
OTS component.
The estimation of the component-integration effort in COTS-based development
projects has been investigated by several studies [1], whereby the estimation of
integration effort in OSS- based development has been neglected so far. It is therefore
interesting to know the commonalities and differences in cost models for COTS-based
vs. OSS-based projects.
Without source code, it is difficult for COTS component users to identify whether
the defects is inside or outside the component. Although the source code of OSS
components is available to their users, many OSS users did not change or even read its
source code. As Hissam and Weinstock proposed in [6], OSS is the other commercial
software, i.e. they were used as COTS in practice [9, 11]. OSS user might not have
bothered to understand the source code of an OSS component before starting to use it. It
is therefore difficult for them to figure out if the defects are inside an OSS component or
not.
Results of RQ3 also show that COTS users had significantly more difficulties in
estimating the component selection effort and following requirement changes. It is not
229
Appendix A
easy or cost-effective for COTS users to switch to another COTS component once they
have made the decision and paid for some selected components. Therefore, COTS users
might be more cautious in selecting the right COTS component and spend more effort
than estimated on selecting the best one. Without paying for the OSS components, OSS
users might be less cautious and might spend less effort on selecting the best OSS
component from several candidates. In addition, it might be easier for OSS users to
change the source code of the OSS component if their customers’ requirements do
change. Therefore, OSS components are ideal when the software solution requires high
customization [11].
OSS users had more problems on getting the information on a providers’ support
reputation. Although the preliminary data [10] showed the differences between the OSS
users and COTS users were significant, the current data (six months after the analysis of
the preliminary data) does not show such significant differences any longer. This might
be because more and more commercial organizations provide support to OSS
components. In fact, OSS components with support by a commercial organization
usually become more popular than components without such support [12]. In addition,
more and more OSS portals, such as sourceforge.net and freshmeat.net, have started to
measure the vitality and popularity of an OSS project. These metrics might have helped
OSS users to estimate the quality and duration of support they can expect.
5. CONCLUSION AND FUTURE WORK
This paper has presented the results of an international empirical study to investigate the
decision making of integration COTS or OSS components in software products. In
general, it appears that the motivation for using either COTS or OSS components and
the risks involved are important factors to be concerned. Future research has to pay
specific attention to these facts before defining systematic guidelines and methods.
Results of this study have answered the following three questions:
RQ1: Who were using OTS components?

Both COTS and OSS components are used in projects with different application
domains and different non-functional requirements. There was no significant difference
between the profiles of COTS-based and OSS-based projects.
RQ2: Why was it decided to use OTS components?

The main expectation of using either COTS or OSS components is to obtain shorter
time-to-market and less development effort. COTS users have higher expectation on
COTS component quality and COTS vendor support. Possible no-cost source code is
the key motivation of OSS users. OSS users prefer to have access the source code, so
that they can revise it when necessary.
RQ3: What are the possible problems of using OTS components?

It is more difficult for COTS component users to follow requirement changes than
OSS users. It is also more difficult for COTS users to estimate the selection effort.
Based on these results we identified several emerging research questions that should
be investigated in the near future:
230
Appendix A
- Understand the reasons behind the current results
This study is a state-of-the-practice survey that provides evidence of some

phenomena in OTS-based development. These phenomena have to be studied in
more detail to better understand the nature of OTS projects and to provide systematic
support. Examples of interesting issues are: How is the support capability of an OSS
project estimated, how support for OSS components is realized, and how to avoid the
licensing restriction.
- Understand the OSS projects per se
To provide guidelines on using OTS component in practice, it is important to

understand how OSS projects are organized. Furthermore, it is important to know
how to cooperate with OSS developers, for example, how to ensure that changes of
users are accepted by the OSS community, and how to motivate developers to
enhance and adapt OSS component into a desirable direction. Another important
issue is the definition of metrics to obtain reliable data about the concrete nature of
OSS projects and products. For example, the organization support proved to give a
positive effect on the popularity of an OSS component [12]. Future studies have to
examine OSS projects in order to identify, define, and validate more such measures.
A good candidate seems the question: whether the vitality and popularity of data
provided by an OSS portal gives insight on the longevity of an OSS projects?
REFERENCES
[1]Abts, C. et al. COCOTS: A COTS Software Integration Lifecycle CostModel -

Model Overview and Preliminary Data Collection Findings. TR: USC-CSE-2000-
501, Mar. 2000.
[2]Brownsword, L. et al. Developing New Processes for COTS-Based Systems. IEEE
Software. 17, 4 (Jul./Aug. 2000), 48-55.
[3]Conradi, R. et al. Reflections on conducting an international CBSE survey in ICT
industry. Proc. of the 4th Intl. Symposium on Empirical Software Engineering
(Noosa Heads, Australia, Nov. 2005), 214-223.
[4] Fitzgerald, B. A Critical Look at Open Source. IEEE Computer. 37, 7 (Jul. 2004),
92-94.
[5] Glass, R. L. A Look at the Economics of Open Source. Communication of the ACM,
47, 2 (Feb. 2004), 25-27.
[6] Hissam, S. A. and Weinstock, C. B. Open Source Software: The Other Commercial
Software. Proc. of the 1st workshop on open source software engineering. (Toronto,
Canada, May, 2001).
[7] Hissam, S. A. et al. Trust and Vulnerability in Open Source Software. IEE
Proceedings - Software, 49, 1 (Feb. 2002) 47-51.
[8] Li, J. et al. An Empirical Study of Variations in COTS-based Software Development
Processes in Norwegian IT Industry. Proc. of the 10th IEEE Intl. Metrics Symposium
(Chicago, USA, Sep. 2004) 72-83.
231
Appendix A
[9] Li, J. et al. Validation of New Theses on Off-The-Shelf Component Based

Development. Proc. of the 11th IEEE Intl. Metrics Symposium (Como, Italy, Sep.
2005), 26.
[10] Li, J. et al. An Empirical Study on Off-the-Shelf Component Usage in Industrial
Projects. Proc. of the 6th Intl. Conf. on Product Focused Software Process
Improvement (Oulu, Finland, Jun. 2005), Springer Verlag LNCS Vol. 3547, 54-68.
[11] Madanmohan, T. R. and De’, R. Open Source Reuse in Commercial Firms. IEEE
Software, 21, 1 (Jan./Feb. 2004), 62-69.
[12] Stewart, K. J. et al. Preliminary Analysis of the Influences of Licensing and
Organizational Sponsorship on Success in Open Source Projects. Proc. of the 38th
Hawaii Intl. Conf. on System Sciences (Hawaii, USA; Jan. 2005), 197c.
[13] Norris, J. S. Mission-Critical Development with Open Source Software: Lessons
Learned. IEEE Software. 21, 1 (Jan./Feb. 2004), 42-49.
[14] OSS Initiative: http://www.opensource.org/index.php
[15] Paulson, J. W. et al. An Empirical Study of Open-Source and Closed-Source
Software Products. IEEE Transactions on Software Engineering. 30, 4 (Apr. 2004),
246-256.
[16] Pizka, M. Adaptation of Large-Scale Open Source Software – An Experience
Report, Proc. of the 8th Euro. Conf. on Software Maintenance and Reengineering
(Tampere, Finland, Mar. 2004) 147-153.
[17] Raymond, E. S. The Cathedral and the Bazaar: Musings on Linux and Open Source
by an Accidental Revolutionary. O’Reilly, 1999.
[18] Ruffin, M. and Ebert, C. Using Open Source Software in Product Development: A
Primer. IEEE Software, 21, 1 (Jan./Feb. 2004), 82-86.
[19] Szyperski, C. et al. Component Software – Beyond Object-Oriented Programming.
Addison-Wesley, 2002.
[20] Voas, J. The challenges of Using COTS Software in Component-Based
Development. IEEE Computer, 31, 6 (Jun. 1998), 44-45.
232
Appendix A
P7. Barriers to Disseminating Theory

Barriers to Disseminating Off-The-Shelf Based Development
Theories to IT Industry
Jingyue Li1, Reidar Conradi1,2, Odd Petter N. Slyngstad1, Christian Bunse3, Umair
Khan3,
Marco Torchiano4 and Maurizio Morisio4
1
2
3
{Christian.Bunse, khan}@iese.fraunhofer.de
4
{maurizio.morisio, marco.torchiano}@polito.it
ABSTRACT: In this position paper, we have reported results of an industrial seminar.

The seminar was intended to show our findings in an international survey, conducted in
Norway, Italy and Germany, on off-the-shelf component-based development.
Discussion in the second section of the seminar revealed several obstacles of
popularizing the OTS based development theories into IT industry.
Keywords: COTS component, OSS component, Empirical Study
1. INTRODUCTION
Software developers are using an increasing amount of COTS (Commercial-Off-The-

Shelf) and OSS (Open Source Software) components in their projects. However, using
such external components introduces many risks [2, 5, 10]. Before project managers
decide to acquire an external component, instead of building it in-house, they must
identify and mitigate possible risks. Researchers and practitioners have proposed many
new COTS selection and development processes [1, 14,19].
As most theories were based on local case studies, it is necessary to evaluate their
conclusions on a larger sample. We have performed an international survey on process
improvement and risk management in COTS and OSS component based development in
Norway, Germany and Italy [12, 13]. After we analysed data, we hold an industrial
seminar to report our preliminary results to the participants of this study and some
interested project managers. An hour discussion revealed many new concerns from an
industrial viewpoint.
233
Appendix A
2. BACKGROUND
Our survey is the second phase of a systematic study on process improvement and risk
management in OTS based development. We started from a pre-study, focusing on
software process improvement in COTS based development, and being performed with
structured interviews of 16 COTS projects in Norwegian IT companies [11]. The
following main survey extended the pre-study in two dimensions. First, it included OSS
components because they represent an alternative to COTS components. Second, this
study included samples from Norway, Italy and Germany. In addition, the sample was
selected randomly instead of on convenience as in the pre-study.
The survey was designed to investigate six main research questions:
- RQ1: What are the main motivations of using OTS components instead of
building them in-house?
- RQ2: What are the real common OTS-based process models, and what are the
possible variances?
- RQ3: What is the OTS component selection process used in practice?
- RQ4: What risks have been identified and what are their relationships with the
project contexts?
- RQ5: Which risk management methods have been performed and with what the
results?
- RQ6: What are the differences between COTS and OSS components?
The unit of study is a completed software development project. The projects were
selected based on two criteria:
- The project should use one or more OTS components
- The project should be a finished project, possibly with maintenance, and possibly
with several releases.
In the data collection process, we promised our participants that we would report the
result of the study in the form of a seminar. After we have finished the data collection in
Norway (47 projects) and gathered some data from Germany and Italy (23 from
Germany and 20 from Italy), we held a seminar at Oslo in Feb. 2005 as we promised to
the Norwegian participants.
The intentions of the seminar include two parts:
- First, we would like to report our results by giving guidelines on how to improve
processes and manage risks in OTS based development.
- As most data of this study show only the state-of-the-practice, we do not know
possible cause-effect relationships of our conclusions. We therefore had one hour
discussion as the second part of the seminar to get feedback from the participants.
Twenty-three Norwegian industrial persons showed up in the seminar. More than

40% of them came from large-sized IT companies (more than 100 employees). 40%
came from a medium-sized (with 20-99 employees) and 20% came from small-sized
companies (less than 20 employees). We first showed our survey results in one hour
presentation. On-going our talk, participants were able to interrupt and ask questions. In
234
Appendix A
the second part, we hypothesized some questions based on the survey result. We also
listed our assumed explanations on these questions. Industrial participants gave their
opinions to support or reject our explanations.
3. RESULTS
Concerns and comments from industrial colleagues gave us valuable ideas on what we
need to do in order to disseminate OTS based development theories into industry. We
have summarized industrial concerns and comments into five barriers.
3.1 The barrier to disseminating the overall formal OTS-based development

process
The pre-study showed that most projects decided their main development process before
they started to think about using OTS components [11]. The result from the survey
follows the same trend. In the discussion section, we asked why they did not use the
proposed OTS-based development processes as in [3,14]. Our assumed explanations
were:
- Not necessary, as OTS is a minor part of the system
- It is difficult to change (company level)
Industrial answers: Industrial colleagues agreed with our second explanation. They
said people would not like to change the way they work. They think it is either too
expensive or too difficult. They argued that there were few empirical studies to
investigate the cost and benefit of the proposed COTS-based development processes. It
is therefore difficult to convince industrial developers to use them. They do not want to
change their development processes dramatically, being a painful procedure, without
enough confidence.
Relevant studies: Some previous studies have analysed the barriers of using COTS
information technology based on lessons learned in small and medium sized companies
(SMEs) [18]. The conclusions are:
- SMEs do not formally define their processes and tend to use tribal knowledge.
- SMEs use flexible processes as a competitive advantage and take pride in that
flexibility. Changing this approach is difficult, because many SMEs equate
process flexibility with having undocumented processes.
- The culture within SMEs is one of informal teamwork and therefore is resistant
to formalism or change.
Our insight: In our study, we excluded companies with less than 5 employees
because most such companies do not have a formally defined process. Most companies
in our study have one de-facto process, such as RUP or CMMI. The reason for not using
a formal COTS process is not because they do not have any formal development
process.
However, before adopting their process into a formal process, such as EPIC [1], the
industrial practitioners will need evidences that the formal process is more cost-
effective. It is also necessary for them to know how to downsize a formal process
235
Appendix A
according to different project contexts, such as time-to-market pressure, quality

requirements and so on.
Barrier one: Without sufficient empirical evidence on OTS based development

processes, industrial project managers would not like to change their de-facto
3.2 The barrier to disseminating the formal OTS selection process
Although many formal OTS selection processes have been proposed [4, 9], results of
this study and some other studies [20] showed that they were seldom used in practice. In
the discussion section, we asked why they did not use these formal selection processes.
Our assumed explanations were:
- Don’t know the formal selection process

- Not necessary
- It was not easy to use
Industrial answers: Industrial participants agreed with our second explanation.

They argued that OTS component are not the key parts of the system in some cases.
They just want to select one as fast as they can. These formal selection processes are not
cost-effective for them. In addition, they argued that there were few empirical studies on
when they should use the formal process instead of downloading a demonstration and
trying it out, as the process discovered in study [11]. They were few studies showing
that a formal process actually gave better results than informal processes.
Relevant studies: The balance between COTS uncertainty risks and risks resulting
from project delay has been investigated by an empirical study [16]. They concluded
COTS assessment appears qualitatively intuitive for any given aspect. However there
are often many factors that must be considered simultaneously. The impact of all
considerations may be quite complex and counter intuitive.
Our insights: Most formal selection processes were proposed without evaluations on
their pre-conditions and their cost-benefit. For example, the selection process for fine-
grained small OTS component (the object of this study) would be different with
selection process for large OTS suites. The large OTS suites usually have more features
than required from the customer. How to evaluate these extra features have not been
investigated in current formal selection processes [15]. In addition, most selection
processes assumed that the general software architecture was ready before OTS
selection. However, our studies discovered that the OTS component can be selected in
the pre-study phase, requirement phase, general design phase, detailed phase, and
coding phase [11]. That is, how to customize the formal selection process according to
little information needs future studies.
Barrier two: Little advice on how to customize a formal process according to

project context and OTS itself prevents the use of a formal selection process.
236
Appendix A
3.3 The barrier to sharing OTS based development knowledge inside company
Our pre-study and some other studies [14] showed that a new role, an OTS knowledge
keeper, is necessary in COTS-based development, Result of this survey showed that
projects with a dedicate person keeping fresh of the knowledge of OTS component had
fewer problems (see problems we listed in [12]) than projects without it. The results
also show that the more project members had general experience with OTS-based
development, the fewer problems they had in their projects. In the discussion section,
we asked what other knowledge is necessary, in addition to the technical insights in
concrete OTS components.
Industrial answers: Industrial participants argued that general risk management and
cost estimation experience must be shared. However, they had no good idea on how to
do it. Moreover, they were interested in how much effort the knowledge keeper needs to
spend on OTS component based development and what kind of knowledge needs to be
kept.
Relevant studies: In study [6], the authors concluded that a pre-fabricated, reusable
set of generic requirements, based upon the practical analysis in business environments
of application and COTS middleware components characteristics helped to ensure the
success of COTS middleware selection.
Our insight: Knowledge accumulation and transfer inside a software company and
software projects have been investigated in other software development fields.
However, what knowledge should be documented and shared in OTS-based
development has seldom been investigated. It is therefore difficult for industrial
practitioners to collect and share relevant OTS based development knowledge.
Barrier three: Few studies on knowledge accumulation and transfer delayed the
dissemination of the OTS based development experience.
3.4 The barrier to using OSS components
Our study compared the differences between projects using only COTS component with
project using only OSS components. Results show that OSS users have serious
problems in getting the support reputation information on OSS component providers
[13]. We asked the question: How to evaluate the support reputation of OSS
community?
Industrial answers: The conclusion from industrial participants is that it is not

possible to get this information and ensure the technical support from OSS providers
currently. They argued that most available information of OSS component support come
from the previous experience, i.e. the previous response time, the bug fixing speed etc.
Without a commercial contract, as with a COTS vendor, it is not possible to ensure the
future response time and the quality of technical support because most technical support
work is based on volunteers.
237
Appendix A
Relevant studies: OSS component have been used more popular in commercial
projects. As discussed in [7], how to establish credible and mutually acceptable
combinations of OSS delivery and procurement models therefore needs future research.
Our insight: Industrial practitioners inclined to use OSS components in their

projects. However, concerns on incredible technical support from OSS blocked the
process.
Barrier four: Without credible and mutually acceptable combinations of OSS

components’ technical support, it is difficult for an industrial developer to use OSS in
practice.
3.5 The barrier to doing requirements (re)negotiation in OTS-based development
Our pre-study showed that customers’ involvement on OTS component selection and
decision making could be a good risk mitigation strategy. It gives integrators the leeway
to (re) negotiate requirements when the limitations of an OTS component are found at
the very end [11]. However, data of this study shows that this strategy has been seldom
used in practice. We asked the question: why customer had seldom been involved in
COTS component decision making?
Industrial answers: Industrial colleagues explained that their customers either had
no technical background to make such decisions or cared only the final product without
considering the technical details. So, the integrators had to take all the responsibilities
on OTS component decision making. As a result, the requirements (re)negotiation
because of OTS component’s limitations is impossible in practice.
Relevant studies: Methods of requirements (re)negotiation have been proposed as

an important part of requirements engineering. However, both our pre-study [11] and
another study [17] found out the same phenomena. If the COTS candidates could not
feasibly satisfy pre-defined system project objectives, constraints, and priorities, these
project objectives, constraints, and priorities could not be changed or negotiated to
accommodate the constraints imposed by the available COTS. Therefore, building the
required functionalities in-house became the only viable option.
Our insight: As it is difficult to discover all OTS components’ limitations at the

very beginning, it is necessary for integrators to be able to (re) negotiate the
requirements with the customer. How to involve customers in OTS component decision
making therefore needs future studies.
Barrier five: Few case studies have observed successful requirements

(re)negotiation in OTS based development. There are therefore few guidelines on when
and how to involve customers in requirements (re)negotiation because of OTS
limitations.
238
Appendix A
4. CONCLUSION AND FUTURE WORK
This paper has reported some industrial obstacles to conduct OTS based theory into
practice. We believe that answers to these barriers will help to improve OTS-based
development practice in industry. In the future, we plan to study on these barriers by
case studies and personal interviews.
5. ACKNOWLEDGMENTS
This study was partially funded by the INCO (INcremental COmponent based
development) project [8]. We thank the colleagues in these projects, and all the
participants in the survey and seminar.
REFERENCES
Albert, C. and Brownsword L. Evolutionary Process for Integrating COTS-Based

System (EPIC): An Overview. SEI, Pittsburgh.
http://www.sei.cmu.edu/publications/documents/02.reports/02tr009.html. 2002.
Boehm, B. W., Port, D., Yang, Y., and Bhuta, J. Not All CBS Are Created Equally
COTS-intensive Project Types. In Proc. of the 2PndP International Conference on
COTS-Based Software Systems (ICCBSS’03) (Ottawa, Canada, February 2003),
LNCS Vol. 2580, Springer-Verlag, 2003, 36-50.
Brownsword, L., Oberndorf, T. and Sledge, C. Developing New Processes for COTS-
Based Systems. IEEE Software, 71, 4, (July/August, 2000), 48-55.
Dorda, S. C., Dean, J. C., Morris, E., and Oberndorf, P. A Process for COTS Software
Product Evaluation. In Proc. of the 1PstP Int. Conf. on COTS-Based Software
Systems (ICCBSS’02), (Orlando, FL, USA, Feb. 2002), LNCS Vol. 2255, Springer-
Verlag, 2002, 176-187.
Fitzgerald, B. A Critical Look at Open Source. IEEE Computer, 37, 7, (July 2004), 92-
94.
Gorton, I. and Liu, A. Streamlining the Acquisition Process for Large-Scale COTS
Middleware Components. In Proc. of the 1PstP Int. Conf. on COTS-Based Software
Systems (ICCBSS’02), (Orlando, FL, USA, Feb. 2002), LNCS Vol. 2255, Springer-
Verlag, 2002, 122-131.
Holck, J., Larsen, M. H., and Pedersen, M. K. Managerial and Technical Barriers to the
Adoption of Open Source Software. In Proc. of the 4PthP Int. Conf. on COTS-Based
Software Systems (ICCBSS’05), (Bibao, Spain Feb. 2005), LNCS Vol. 3412.
Springer-Verlag, 2005, 289-300.
INCO project description, http://www.ifi.uio.no/~isu/INCO, 2000
Lawlis, P. K., Mark, K. E., Thomas, D. A., and Courtheyn, T. A Formal Processes for
Evaluating COTS Software Products. IEEE Computer, 34, 5, (May 2001), 58-63.
Lawton, G. Open Source Security: Opportunity or Oxymoron? IEEE Computer, 35, 3,
(March 2002), 18-21.
Li, J., Bjørnson, F. O., Conradi, R., and Kampenes, V. B. An Empirical Study of
Industry. In Proc. of the 10PthP IEEE Int. Metrics Symposium (Metrics'04),
(Chicago, USA, September 2004), IEEE Press, 2004, 72-83.
239
Appendix A
Li, J., Conradi, R., Slyngstad, O. P. N., Torchiano, M., Morisio, M., and Bunse, C. A
State-of-Practice Survey on Risk Management in Off-the-Shelf Component-Based
Development. In Proc. of the 4PthP Int. Conf. on COTS-based Software System
(ICCBSS’05), (Bibao, Spain Feb. 2005), LNCS Vol. 3412 Springer-Verlag, 2005,
278-288.
Li, J., Conradi, R., Slyngstad, O. P. N., Bunse, C., Khan, M., Torchiano, M., and
Morisio, M. An Empirical Study on Off-the-Shelf Component Usage in Industrial
Projects. Forthcoming in the 6PthP Int. Conf. on Product Focused Software Process
Improvement(PROFES 2005) (Oulu, Finland, June, 2005)
Morisio, M., Seaman, C.B., Parra, A. T., Basili, V. R., Kraft, S. E., and Condon, S. E.
Investigating and Improving a COTS-Based Software Development Process. In Proc.
of the 22PndP Int. Conf. on Software Engineering (ICSE 2000). (Limerick, Ireland,
June, 2000). ACM Press, New York, NY, 2000, 31-40.
Perrone, V. A Wish List for Requirements Engineering for COTS-Based Information
Systems. In Proc. of the 3PrdP Int. Conf. on COTS-Based Software Systems
(ICCBSS’04) (Redondo Beach, CA, USA, Feb. 2004), LNCS Vol. 2959, Springer-
Verlage, 2004, 146-158.
Port, D. and Chen, S. Assessing COTS Assessment: How Much Is Enough? In Proc. of
the 3PrdP Int. Conf. on COTS-Based Software Systems (ICCBSS’04) (Redondo
Beach, CA, USA, Feb. 2004), LNCS Vol. 2959, Springer-Verlage, Berlin Heidelberg
New York (2004), 183-198.
Port, D. and Chen, S.,: Empirical Analysis of COTS Activity Effort Sequences. Proc. of
the 3PrdP Int. Conf. on COTS-Based Software Systems (ICCBSS’04) (Redondo
Beach, CA, USA, Feb. 2004), LNCS Vol. 2959, Springer-Verlage, 2004, 169-182.
Robert, J., Buhman, C., Garcia, S., Allinder, D., Bringing COTS Information
Technology into Small Manufacturing Enterprises. In Proc. of the 2PndP Int. Conf.
on COTS-Based Software Systems (ICCBSS’03), (Ottawa Canada, Feb. 2003),
LNCS Vol. 2580. Springer-Verlag, 2003, 187-195.
SEI (Software Engineering Institute). COTS-Based Initiative Description.
http://www.sei.cmu.edu/cbs/cbs_description.html. 2004
Torchiano M. and Morisio M., “Overlooked Facts on COTS-based Development”, IEEE
Software, 21, 2, (March/April 2004), 88-93.
240
Appendix A
P8. Reflections on Conducting a Survey

Reflections on Conducting an International Survey of
Software Engineering
Reidar Conradi 1, 2, Jingyue Li 1, Odd Petter N. Slyngstad 1, Vigdis By Kampenes 2,
Christian Bunse 3, Maurizio Morisio 4, and Marco Torchiano 4
1
Department of Computer and Information Science
Norwegian University of Science and Technology
{conradi, jingyue, oslyngst}@idi.ntnu.no
2
Simula Research Lab
P.O. Box 134, NO-1325, Lysaker, Norway
vigdis@simula.no
3
Fraunhofer IESE
Sauerwiesen 6, D-67661 Kaiserslautern, Germany
bunse@iese.fraunhofer.de
4
Dip. Automatica e Informatica Politecnico di Torino
Corso Duca degli Abruzzi 24, I-10129 Torino, Italy
{morisio, torchiano}@polito.it
Abstract: Component-based software engineering (CBSE) with Commercial-Off-The-

Shelf (COTS) or Open Source Software (OSS) components are more and more
frequently being used in industrial software development. We therefore need to issue
experience-based guidelines for the evaluation, selection and integration of such
components. We have performed a survey on industrial COTS/OSS development in
three countries – Norway, Italy and Germany. Concrete survey results, e.g. on risk
management policies and process tailoring, are not being described here, but in other
papers. This is a method paper, reporting on the challenges, approaches and experiences
gained by conducting the main survey. The main contributions are as follows: At best
we can achieve a stratified-random sample of ICT companies, followed by a
convenience sample of relevant projects. This is probably the first software engineering
survey using census type data, and has revealed that the entire sampling and contact
process can be unexpectedly expensive. It is also hard to avoid national variations in
the total process, possibly leading to uncontrollable method biases.
Keywords: Surveys, component-based software engineering, COTS/OSS components,

open source software, ICT industry.
1. Introduction
Component-based software engineering (CBSE) is considered one of the most important

technologies to reduce time-to-market and cost and to improve software quality.
Although there are many proposed CBSE methods in research papers and some books,
few representative industrial studies have been done to study such methods in practice.
241
Appendix A
Although the survey topic is CBSE, the focus of this paper is to convey some
reflections and lessons learned in performing a large-scale, multinational survey of
software engineering in the ICT industry. For a definition of “component”, see [30]. Let
us summarize the overall survey process:
We scanned the literature to formulate research hypotheses and questions. Sampling
was planned.
NTNU and the Simula Research Lab in Oslo performed a pre-study on COTS
(Commercial-Off-The-Shelf) development through on-site, structured interviews of 16
projects in 13 companies in autumn 2003 [18].
Based on the pre-study, we started preparing the main study in late 2003. This
included questions on OSS components, and extended the survey to Germany and Italy.
We designed a more formal questionnaire for this study in spring 2004, translated it into
Norwegian, German, and Italian and installed it on the SESE web server at the Simula
Research Lab [28].
The main study was performed in May 2004 – March 2005. We got name lists of ca.
500 ICT companies in each country. For each company, we got hold of a possible
respondent over the phone. This person was asked to identify a suitable OTS-based
project (Off-The-Shelf, being either COTS or OSS), and encouraged to answer the
questionnaire. Most respondents filled in the web-questionnaire him/herself, but some
were filled in by phone interviews. Per 16 August 2005, 133 company responses have
been recorded.
We are now analysing and publishing the results [19]. The survey has taken much
more effort than anticipated, so we need to reflect on the choices, e.g. on sampling,
approaching the companies, filling in the questionnaire etc.
The three research partners (NTNU/Simula, IESE, Torino group) have been working
on software reuse, CBSE, process improvement, and industrial studies for over 15 years.
We have planned and conducted the entire survey with no physical meetings, just email
contact and two-weekly phone meetings – even the web server has been accessed
remotely.
The main conclusions of this paper are that we at best can obtain a stratified-random
sample of ICT companies, and thereafter a convenience sample of relevant projects. The
sampling and contact process can be exceedingly expensive, and national process
variations are difficult to avoid. This in turn can lead to uncontrollable method biases.
This paper is a method paper to reflect on the main study. The remainder of the paper
is structured as follows. Section 2 presents some related work. Section 3 discusses
definition of population, and Section 4 how to make a sample from this population.
Section 5 discusses questionnaire design, and Section 6 discusses the company contact
process. Section 7 has lessons learned, and Section 8 concludes the paper.
2. Related work on surveys of software technologies
2.1. Empirical study types for CBSE
There are many different types of empirical studies for software engineering [31].
Several guidelines have been published for how to conduct such studies [15] [4]. Aside
from action research and case studies, the following two study types seem most relevant
for CBSE, but are not discussed in [31]:
242
Appendix A
- Structured interviews (qualitative surveys) with an interview guide, to

investigate rather open and qualitative research questions with some
generalization potential. The Torino group has investigated the success of 32
reuse projects under the PIE program in EU [23]. In 2002, NTNU and the Torino
group did structured interviews of seven small software companies in Norway
and Italy. The results were synthesized as six theses [32] that contradicted many
claims from the literature. This was followed up with a qualitative pre-study [18]
in Norway by NTNU and the Simula Research Lab. The aim was to confirm
these new findings and perhaps discover more revealing “truths”.
- Quantitative surveys with a questionnaire containing mostly closed questions.
This is suited to investigate formalized hypotheses upon a sample from a larger
population, where the results can be generalized (see 2.2). For instance, the
above pre-study encouraged us to perform a main survey of industrial practice
regarding OTS- (COTS- and OSS) based development [19].
Sometimes there is a sequence, even an iteration, in the studies. That is, we start with
local case studies and through interviews end up with a survey, reflecting increased
theory building and generalization. As an example, the research partners have
previously carried out case studies of CBSE at NASA [22], at Ericsson [20] and in
German companies [2].
2.2. How to perform sound surveys in software engineering
There is a massive literature from the social sciences on how to perform valid and
reliable surveys - e.g. regarding questionnaire design, sampling, and how to contact the
respondents [10]. A challenge in international surveys is to stay methodologically
similar across countries to avoid national bias, e.g. with sampling, contacting, and
questionnaire fill-in.
Typical samples are based on convenience (reusing previous contacts), snowball
(from a few seeds like a pyramid game), randomly selected subsets (from some
population list), or stratified subsets (considering e.g. ICT sub-sector, company size, or
country). Typical ways to fill in a questionnaire are by paper copy via post or possibly
fax, by phone or site interviews, and recently by email or web. The questionnaire can be
filled in either by the respondent her/himself or by an interviewer. Interviews, or phone
contact in general, may result in a higher response rate, and may also contribute to
quality-assure the returned data. However, they cost more to perform and some answers
may be influenced by the interviewer, possibly resulting in dishonest answers (a “mode”
problem, see later discussion in Section 5).
When asking about past events, in our case finished software projects, a respondent
may have to think a bit, retrieve old documents, or speak to colleagues. This may take
time and effort. We must therefore offer a print-out version, in addition to a web-
questionnaire - read on the bus, or similar.
There have been published some more specific advice for surveys on software
engineering [25], and for how to use web-based questionnaires in surveys [26].
However, even these more ICT-specific papers have not been of much help in our
sampling and contact process.
243
Appendix A
2.3. A review of previous surveys in software engineering
In light of the issues discussed in Sections 2.1 and 2.2, we will briefly present some
published software engineering studies. The external validity or generalizability of all
these surveys is speculatively or not at all discussed, because the population and
associated sample are not clearly specified. First some non-CBSE surveys, summarized
in Table 1, then an annotated list of CBSE surveys as shown in Table 2.
Table 1: Some non-CBSE surveys.

Study topic Study object Sample selection Sample # responses
size
1.Software Portfolio of Random sample of 800 500 53 valid of
maintenance maintained company members of 97 returned
processes [13] software systems Norwegian IFIP responses
in ICT user- subsidiary (Mean: 656
organizations for all
empl.,
11 in IS-
dept.)
2. SPI success Software Complete member lists 154 120 (Mean:
criteria engineering of two Norwegian ICT 196 empl.,
[6] processes of ICT industrial organizations median: 55)
companies
3. Estimation Finished software From 100 largest 37 18
accuracy [21] projects in ICT Norwegian ICT companies
companies companies, listed in a with 52
business magazine projects
(Mean: 141
empl.,
median: 75)
4. Effort Incremental Convenience sample of 100 20
estimation, COTS projects aerospace and defense
COTS contractors
development [1]
5. Risk Finished software Random sample of 248 83
management projects in IT managers in IT
policies [27] departments/ companies, from a
companies Finnish IT Assoc.
6. Efficiency of State of the Convenience sample of 865 121
inspection proc’s practice of contact persons in
[26] inspections German industry and
academia
244
Appendix A
Table 2: Some CBSE studies, mostly surveys.

Study topic Study object Sample selection Sample #
size responses
1. Software reuse Software Convenience sample of 113
success factors organizations 113 persons in 29 NA (Median:
[11] organizations 25,000
empl.,
range 35 to
350,000)
2. Reuse proc’s Software Subscriber lists of three NA 75
with Ada [16] professionals Ada magazines
mostly in IT
industry
3. Software reuse Finished EU From official list of 288 32 24 proj.
success factors projects on PIE projects in EU projects interviews
[23] software reuse in 19 comp.
(9 small, 9
medium, 6
large)
4. COTS dev. COTS-based Convenience sample of 7 7 7 (Mean
proc’s [32] projects small sw companies and
median: 7
empl.)
5. Company Companies Snowball sample of ICT 146 (Mean:
attitudes towards using OSS companies using OSS 275 17 empl.,
OSS [3] range 1-
320)
6. Company Companies Convenience sample 1300 109
experience and practicing (Median:
motivation on CBSE ca. 100)
CBSE [7]
7. COTS/OSS Finished sw From e.g. stratified lists 108 127 (Mean:
dev. proc’s [19] projs with OTS of ICT companies from 3 2240,
(this study) dev. Census Bureau median: 34)
As with these studies, most published surveys in our field fail to report on how the
population is defined, what ICT sub-sectors are covered, what sampling procedure has
been applied, how representative the final sample is, what the mean and median
company sizes are etc. Even most controlled experiments in our field, where the
demand for rigor is high, have similar problems. Furthermore, generalizability raises
more fundamental concerns than valid statistical sampling [17].
245
Appendix A
3. Defining the relevant population
This has two parts: how to define the ICT sector (Section 3.1), and how to define our
study objects inside this (3.2). We will use the terminology at
(www.socialresearchmethods.net/kb/sampterm.htm):
- Theoretical population: all OTS-based software projects globally,

- Study or target population: all OTS-based projects in the entire ICT industry in
the three countries.
- Sampling frame: list of OTS-based projects in stratified subset of above (by
NACE codes and company size).
- Sample: complete or random selection from the above sample frame.
- Subsample: final or responding subset of sample.
3.1. Defining the population of ICT companies
We generally have a problem in defining our study population of ICT companies, or the
commercial ICT sector. In our study, this should be defined for three countries, and is
expected to be representative for the ICT sector world-wide, the theoretical population.
A quick glance at public statistics, however – using Standard Industry Codes from the
US Census Bureau or their EU/OECD adaptation as four digit NACE codes [24] – gives
little clue of what is ICT or not. For instance, the software business is bafflingly
categorized under sector K for “Real estate, renting and business activities”. Thus,
OECD, SSB (the Norwegian Census Bureau), and similar national agencies have jointly
defined the following four sub-sectors to constitute the ICT sector in a country:
- 3x.xx ICT industry (Motorola, Nokia, etc.)

- 5x.xx ICT trade (Computerland etc.)
- 64.2x ICT telecom (AT&T, BT, Telenor etc.)
- 72.xx ICT software, both vendors (Oracle etc.) and consultants (Accenture, Cap
Gemini etc.)
We will use ICT sector as the general name. In Norway, this sector comprises 4-5%
of the economy, excluding public services, i.e. it is the biggest industrial sector after
petroleum. Of course, the trade sub-section (5x.xx) is not very R&D-intensive, and
hence outside the scope of our study.
In addition comes the ICT contents sector (database and web services, media
industry etc.), expected to be finally classified by OECD et al. in 2007. There is also
internal software development outside the ICT sector – as part of finance, engineering,
public services etc. This is considered by the EU to be larger than the ICT sector itself.
We will not deal with the contents sector or the non-ICT sector (including the public
one) here, other than including some large non-ICT companies in our survey.
As a start, let us look at the structure of the ICT sector in Norway [29]. By employee
is meant both owners and hired staff. By company (the Norwegian SSB uses the term
enterprise) is meant a “legal” economic unit (typically a public shareholding company),
not just an internal development department or a branch office. Some comments are:
246
Appendix A
- For software, ICT as well as the total, over 2/3 of the companies are small (1-4
employees) and with a median of 1 employee. The company average is 4.5
employees for software, 7 for all ICT, and 5 for the total. Only 0.5% (46) of the
software companies, 1.5% of the ICT companies, and 0.5% of all companies
have 100 or more employees.
- 8,228 (about 2/3) of the 12,861 ICT companies are software companies (NACE
code 72.xx), having over 40% of the employees. Of the 7,171 software
companies with 0-4 employees, 4,113 have one employee and 1610 have no
employees!
- Ca. 30% of both software and ICT employees work in companies with less than
20 employees, 30% in companies with 20-99 employees, while 40% work in
companies with 100 or more employees.
- There are regrettably some deviations in the totals, since they are drawn from
different SSB sources.
- The ICT company distribution in Germany [8] and Italy [14] in 2002 shows that,
as in Norway, most ICT companies are small. In Germany, the average software
company has 7 employees and the ICT average is 13. 1% of German software
companies have 100 employees or more. In Italy, the average software company
has 4.5 employees, the median is 1 employee, and the average ICT company has
7 employees.
The software company distribution in the US in 1997 can be found in [9]. The US
pattern is similar to that described in Europe, with many small companies and a median
of 1 (!) employee. However, the average company size is 150 employees in the US (4.5
in Norway, 7 in Germany, 7 in Italy), and 3% of the software companies had 100 or
more employees.
The documented structure of the ICT sector in all these countries has clear
consequences for defining our population and later samples of this (Section 4). We have
four main sources for concrete names of ICT companies in our study population, called
P1-P4:
- P1) A convenience subset of company contacts among the research partners, as in

the pre-study. Relevant contact persons are often available.
- P2) The 100 largest ICT companies and similar, often summarized in annual lists
in business magazines. No contact person is usually given.
- P3) Members of one or several industrial organizations. Norway has three such
organizations, Germany two, and Italy two. However, we do not know how
comprehensive or accurate (“ICT-related”) such member bodies might be. In
Norway, one of these organizations covers 5% of the total (600 of 13,000
companies). The listed contact is often an administrative person.
- P4) Complete set of registered ICT companies defined by public sources, e.g. by
NACE-based lists from census bureaus or as sector lists in the yellow pages in
phone books. Such sources should give a trustworthy coverage of the ICT sector.
In all such lists, there will be a substantial share of outdated information: ceased
companies, mergers or split-ups, or companies with e.g. new names, contact persons,
locations, or phone/email addresses. All this reflects a very dynamic marketplace.
247
Appendix A
3.2. Defining the population of finished OTS-based projects
For many ICT surveys, including this one, we do not only have a problem in defining
the population of all ICT companies, and later samples with respondents of these. Our
real problem is that the actual study object (experimental unit) – a special kind of
project – is not identical to the primary object (accessible sampling unit), being a
company or a respondent.
To be precise, our study object is “finished software project using OTS components”.
The corresponding study topic is “OTS-based development processes”. In other studies,
the study object could be “maintained software system” (to study software evolution
processes), “finished software project” (to study estimation accuracy), or “applications
of inspection methods” (to study related cost-efficiency).
Thus, we should first ask “every” ICT company (see 3.1) in the relevant countries to
comprehensively document all recent instances of OTS-based projects, e.g. by an
explorative pre-study. We might even conduct such a pre-study in two steps: first
identify “typical” software projects, and then identify the “OTS” subset of this. But
which project attributes should be collected to serve as later sampling criteria, and how
to validate such criteria? We could consider company size, company profile, project size
and duration, application domain, lifecycle process, developer skills, development
technology, actual OTS-components being used etc. etc.
The above circumstances mean that many companies may have to give us project
data, that later will be unused (“wasted”). Due to resource constraints and logistical
problems, it seems totally unrealistic to perform such an explorative pre-study of a
potential population of relevant study objects, both from the research and industry side.
Furthermore, only 1/3 of the sampled ICT companies in our survey had OTS-based
development. This share would be close to 100%, if the study topic was effort
estimation or software maintenance – i.e. activities that all companies routinely perform.
In surveys in medicine and social sciences, the population is often well-defined and
coming from reliable public sources, although use of volunteers leads to an open-ended
population. There, however, a responding person often matches the actual study object.
4. Sampling from the population
A valid sample is a representative subset of the study population. Sections 3.1 and 3.2
have indicated some problems in defining the study population in software engineering
surveys in the ICT sector. Then, how to pick a valid sample from a poorly or even un-
documented population of study objects? Even if we had a complete list of all OTS-
based projects, what would be the selection method – complete, stratified, random, or
combinations?
At best, we can establish a hopefully representative population of primary objects
(ICT companies) from which we can draw a valid sample. The study population of ICT
companies may be defined as described in 3.1. We could then draw either a complete or
a random-subset sample of primary objects as defined by P1-P3, or a stratified-random-
subset sample from P4, as mostly done here.
The final sample will be a convenience sample of study objects (OTS-based
projects), that the actual contact person (respondent) in the selected ICT company is
248
Appendix A
willing to provide. We can expect one or two such projects per company, but mostly
none.
In our case, we needed about 50 study objects from each country to achieve
statistical significance, given our research questions and related questionnaire questions.
We therefore planned to ask about 250 companies in each country. Assuming an overall
response rate of 1/3 or around 80 answers in each country, that would give us a
satisfactory yield with some margin. (In reality, we needed 400-500 companies to get 50
responses per country). However, we should not draw a random sample from “all” ICT
companies. Based on the documented structure of the ICT sector, our sample frame
should ideally be stratified according to NACE code and company size.
For the Norwegian sample, our sample frame consists of three groups of ICT
companies: large (all ICT sub-sectors including some non-ICT ones), medium (only
software), and small (also software). Each group has roughly the same amount of
software developers. The sample frame is then as follows:
- All large ICT companies (with 100 or more employees) with any ICT-relevant
NACE code; totally 114 companies and with average size of 330 employees. We
sampled the topmost 100 of these 114 ICT companies, and 26 turned out to be
software companies.
- Additionally, simply the 3 largest companies in 5 non-IT sectors (finance,
engineering, energy, food processing, public/other). We sampled all these 15
companies.
- All medium-sized software companies, NACE codes 72.xx (20-99 employees),
totally 228 companies with average size of 40 employees. From this we randomly
sampled 100 companies (increased to 200, due to poor response rate).
- All small software companies, NACE codes 72.xx (5-19 employees), totally 783
companies and with average size of 9 employees. From these 783 companies we
randomly sampled 100. We removed those with 0-4 employees, as they represent
87% (7171) of the companies, but only 19% (7514) of the employees. They are
not expected to have very structured work processes, and are assumed irrelevant
for our study.
We got company names for year 2002 for category 1, 3, and 4 of ICT-companies.
Randomization for category 3 and 4 was performed beforehand, and the data sent to us
as spreadsheets via email. The 15 large non-ICT companies in category 2 are from a
business magazine for year 2002.
Some observations: The Norwegian SSB struggles with incomplete or erroneous
company lists. SSB has one part-time person on ICT and twelve on agricultural
statistics! SSB has cooperated with one of the Norwegian industrial ICT organizations,
IKT-Norge, since 1998 to help establish a reliable database of Norwegian ICT
companies. IKT-Norge has recently set up the Infosector subsidiary [33] to sell services
around this database. This subsidiary turned out to be the only practical way to get a
stratified sample of the official database on electronic form, based on NACE category
and company size. We hence got the requested company list very fast at a low price
(1250 euro) – on the condition that we reported back anomalies in the data.
For the German sample, we got a list of company names for the years 2001 and
2002 from the German census bureau (Statistisches Bundesamt) for the four sub-sectors
249
Appendix A
of the ICT industry, enhanced by major non-ICT companies whose revenues are largely
based on software (e.g., Siemens, DaimlerChrysler, Allianz). These lists were then
validated using our customer database (e.g., removing falsely classified organizations or
companies which clearly do not develop software, such as hardware stores). From the
resulting sample frame of 430 companies, we selected a random sample for each
category. Each sample was then cross-checked with the IESE customer database in
order to obtain a contact point.
For the Italian sample, there is no national directory of ICT companies available to
us. We therefore started with a study population of 43,580 assumed software companies
from the yellow pages in 2001 (the same database as in the ELISS project on OSS [3]).
From this population, we made a sample frame of 6,000 possible company contacts.
From this sample frame, we randomly picked a sample of 747 companies, which we
cross-checked (by web etc.) that their addresses were still valid and that they really did
software development. We were left with 248 software companies, whereof 125 were
using OTS components. 118 of these accepted to fill in the web-questionnaire.
However, many potential respondents didn’t know what an OTS component was.
Tables 3 and 4 show the response rates in the three countries. In “Likely” we have
pruned duplicates or irrelevant items such as holding companies. “Contactable” means
the company coordinates were still valid.
Reflections on sampling and response rates:
- For all countries, there were problems in classifying large and perhaps
international ICT corporations, with many daughter companies and divisions.
That is, what is the relevant company unit? A company (legal economic unit)
may have 10,000s of employees, but a local business unit (e.g. R&D department)
less than 100. E.g., we twice “adjusted” a too high company size in Norway.
- The relatively high mean size for companies in Germany and especially in Italy is
caused by a few large “outlier” companies.
- The local business unit size is considered the most representative measure. This
does not vary much between the three countries, with the larger business units in
Italy and Germany, as expected.
- In the pre-study in Norway, 17 of 34 (50%) of the ICT companies in our
convenience sample had COTS-based development, and most of the remaining
said they also used internal components. 13 of these 17 companies (almost 80%)
agreed to participate and all responded.
- In the main study in Norway, 94 of 334 (28%) of the “contactable” companies
had OTS-based development, and only 41 of the 94 (44%) eventually responded
with a total of 47 projects. That gives an overall response rate of 12% (41 of
334).
- In Norway, 50 of 380 (13%) of the companies were no longer “contactable”, as
the company lists were two years old in 2004. This is not surprising, and
represents a very dynamic marketplace, as mentioned earlier.
- In Norway, we found that 70 of 285 (24%) software companies (NACE code
72.xx) claimed to have no software development at all. Many of these companies
were sales subsidiaries of international ICT corporations, perhaps belonging to
NACE code 51.84? However, no “penalty” for a company to be registered with a
wrong NACE code exists!
250
Appendix A
- The response rate in Norway increases with decreasing company size. The
response rate was especially low in large companies, cf. “gatekeeper” problem in
Section 6.
Table 3: Response rate for Norwegian companies.

Init. Likely Contacta Do SW Use OTS Decl. Final
sample (pruned ble dev. comp. willing sample N
s.)
Large 100 82 80 51 28 8 3
ICT (26) (23) (23) (21) (13) (6) (2)
comp.
(where-of
SW)
Large 15 15 15 9 1 0 0
non-ICT
comp.
Medium 200 183 152 110 41 30 22
SW
comp.
Small 100 100 87 61 24 18 16
SW
comp.
Sum 415 380 334 231 94 56 41
main
study
Sum pre- 34 34 34 34 17 13 13
study
Table 4: Company responses in the three countries.

Initial Likely Contac Do SW Use Declare Final Mean/ Mean/
sample (prune t-able dev. OTS d sample median media
d s.) comp. willing “N” in final n in
sample final
:compa sample
ny :
busine
ss unit
Norwa 415 380 334 231 94 56 41 279 / 30 20 / 11
y
Germa 430 430 430 314 206 60 48 1933 / 74 / 40
ny 200
Italy 747 747 323 248 125 118 38 3091/ 76 / 9
25
Total 1592 1557 1087 793 425 234 127 3371 / 56 / 20
44
251
Appendix A
5. Questionnaire design
The (English) interview guide for the pre-study included standard meta-questions to test
understandability, was held in two runs in fall 2003 on ten industrial colleagues in
Norway, and lead to 10% of the questions being revised.
The main study questionnaire was designed in cooperation with the partners in Dec.
2003 – April 2004, relying on email correspondence and phone meetings. Most of the
questions from the pre-study were reused, and we used much effort to reliably map the
research questions and hypotheses into the questionnaire. The design went through 16
major revisions and this lead to changes in hypotheses, measure scale, and answer
categories. Also, we created an analysis plan and conducted a questionnaire pre-test to
increase survey quality.
The final questionnaire had an introductory letter, a set of concept definitions
(component, OTS etc.) and some control questions to the reader. Thereafter 10 pages
with 80 questions divided in 6 parts. Questions on personal or company matters were
put at the end, emphasizing the confidentiality of the responses.
The pre-test had 18 respondents (industrial and academic) in the three countries. 15%
of the questions were altered based on this. All work was done in English using an ms-
word (.doc) file.
The questionnaire was finally translated into the native language of the three partners
to ensure equal comprehension across cultures, and this revealed only few unclarities.
Filling in took ca. 20-30 minutes.
Reflections on the questionnaire:
- The questionnaire showed robust, with very few misunderstandings. Only four
incompletely filled-in forms (of 47) came in for Norway.
- In Germany the fill-in procedure was improvised to be phone interviewing of
respondents. This introduced a mode problem that could affect “sensitive”
answers, the response rate, or cause misunderstandings [12]. However, most
questions are factual and technical, and the few national differences in German
answer are not likely caused by mode problems, yet this mode difference
represents an unplanned method bias.
- In Germany, we also had to offer extra anonymity for respondents and
companies, in spite of an explicit confidentiality clause.
6. Contacting the companies
Through a list of potential companies, we have to find a contact person in each

company. The initial contact person (not the switchboard operator!) in a survey is called
a gatekeeper [5]. Ideally, this person should connect us with a respondent for the actual
study object. Note again, that the respondent is not our study object, neither is the
company itself.
For convenience samples, there are few problems in getting hold of a potential
respondent, as we often have personal and professional ties from earlier, joint activities.
But for a large corporation, from a census or member list and with thousands of
employees, it can be very frustrating to try to locate a quality manager,
development/project manager, or just an “IT responsible”. The final struggle is to
252
Appendix A
convince a possible respondent over the phone to use his/her time and effort to fill in a
questionnaire, sent by email.
Phone or site-interviews were discussed thoroughly in March 2004, but, despite
recommendation [21] by researchers at Simula Research Lab, rejected, as we at that
time estimated having 80-100 respondents per country and therefore did not have the
capacity to do so (especially based on German arguments!).
Problems (low response rate etc.) were anticipated, so written procedural guidelines
were made early:
1. Establish a representative sample of companies (Section 4). Generally, this meant

no pre-known gatekeeper, respondent, or phone number.
2. Rewards for survey participation were the final report, invitation to an industrial
sum-up seminar (www.idi.ntnu.no/grupper/su/cbse-survey/), and participation in
a holiday lottery worth 1250 euro.
3. Call up the company, ask to be connected to a probable gatekeeper (first try the
“IT manager”), and eventually to a prospective respondent. Explain the survey
context including reward incentives, and ask if the company has software
development and also OTS-based development. A positive return should yield an
operative email address and a direct phone number.
4. The respondent is then emailed a “paper copy” of the questionnaire, telling
her/him to please fill in the questionnaire online by the web (preferred option), on
paper, or as an email attachment.
5. A few days later, re-phone and ask if the respondent is willing to participate in
the survey. If yes, issue a userid/password to the web server.
6. Then wait for a final response, re-phone if necessary etc. This process could take
months, even if the requested work took only half an hour.
7. In case of an incompletely or wrongly filled in questionnaire, we would try to call
back.
8. Though this may sound smoothly, we have some reflections on the company
contact process:
- All this turned out to be very costly and with a rather low response rate.
Given a person-year of total effort just in Norway, the cost of a filled-in
questionnaire is ca. one person-week (2000 euro).
- As mentioned, the German fill-in procedure was performed through
interviews, i.e. different from that used in Norway and Italy.
- Italy lost their research assistant (a summer student) in Oct. 2004, not
replaced before March 2005. Likewise, IESE had 5 months delay (May-Sept.
2004) in PhD student upstart. Also, Norway hired an extra research assistant
(upstarting PhD-student Slyngstad in March-June 2004). All this lead too
much improvisation.
- There was marginal interest for any reward system - maybe we were not
inventive enough?
253
Appendix A
7. Lessons learned
7.1. Method Aspects
- A pre-study often clarifies both research questions and concrete questions in a

later questionnaire. For instance, we “discovered” several variants of OTS-based
development processes in our pre-study [18].
- A representative sample of national ICT companies assumes access to public
data sources, like NACE-based company lists from census registers, but even
such data may contain many errors. Close contact persons in such organs are
needed to obtain stratified samples and expedite delivery of data in electronic
form.
- Better criteria for selecting projects are needed. Since the study population (of
finished OTS-based projects) was undefined, we had to do a two-phase
approach: first pick a representative sample of companies, then a convenience
sample of projects.
sampling and data collection in different countries. The mentioned national
differences now give an unknown mode effect in result interpretation between
the countries. However, the observed national differences for Germany are
unlikely to be linked to mode effects.
- Random sampling and the ensuing contact process appears to be much more
expensive (5 times?) than convenience sampling. In our case, this is because we
got stuck with no previous contact person, and scant pre-knowledge and
“goodwill” from such companies, especially the large ones.
- If the final sample size is under 100 respondents, use phone interviews. This
applies particularly when phone contact is anyhow needed, e.g. due to a non-
convenience sample. A written procedure for the contact process is also needed.
- A baseline can be useful for comparison purposes. In our case, we did not ask
the companies about non-OTS projects, so we missed this.
- Interviewing at this level requires solid background knowledge of the material;
Company phone contact assumes at least PhD student qualifications.
7.2. Practical experiences
The industrial ICT environment seems extremely busy in recent years. It is very hard to
ask for any extra commitment of effort. Also, of an audience of 25 at our final seminar,
only 5 were respondents.
The SESE web server in Oslo has generally worked fine. One of the researchers had
earlier experience as a webmaster, and spent two days in Oslo to learn about the SESE
tool. The tool has already been used by over 20 studies, mostly controlled experiments,
including a shared one by NTNU and the Simula Research Laboratory on software
maintenance.
254
Appendix A
7.3. Research cooperation
The researcher cooperation has gone very well, due to previous contacts, shared
research interests, and good person chemistry. No physical meetings were held, and all
data are sharable among the partners.
Extra resources through an external project would have helped to minimize national
process differences, and having a social scientist on the team is strongly advised, as we
found it highly useful.
Finally, the senior researchers must be thoroughly acquainted with details around
sampling etc., i.e. getting their hands “dirty” by serving as research assistants.
8. Conclusion
This study is the first survey on CBSE and indeed software engineering, which uses
stratified-random sampling of ICT companies in several countries. All previous surveys
have had problems in defining the study population and in selecting a valid sample of
this, especially since the study object is a special project, not a company.
We had few real problems with questionnaire design and implementation, and some
problems with randomized company sampling. On all these points we anticipated
problems, and put in a lot of effort. However, unexpected much work went into
contacting the companies and having the questionnaires filled in. National differences in
sampling procedures and questionnaire fill-in may have introduced method biases,
although not serious it seems.
We have learned a lot in running a representative, multinational survey, and also on
OTS-based development. We plan to follow up the survey in other countries (China and
US), and to supplement with more qualitative studies.
9. Acknowledgements
This study was funded in part by INCO (Incremental COmponent based development,
2001-2004,http://www.ifi.uio.no/~isu/INCO) and SPIKE (SPI by better Knowledge and
Experience), 2003-2005, http://www.idi.ntnu.no/grupper/su/spike.html). We thank our
colleagues in these projects, social scientist Ola Listhaug at NTNU, and all participants
in the survey.
10. References
[1] C. Abts, B.W. Boehm, and E.B. Clark, “COCOTS: A COTS Software Integration
Lifecycle Cost Model – Model Overview and Preliminary Data Collection Findings”,
Proc. of the ESCOM-SCOPE 2000, 18-20 April 2000, Munich, Germany, Shaker
Publ., pp. 325-333.
[2] C. Atkinson, C. Bunse, and J. Wüst, “Driving Component-Based Software
Development through Quality Modelling”, In A. Cechich, M. Piattini, and A.
Vallecillo (Eds.), “Component-Based Software Quality: Methods and Techniques,
State-of-the-Art Survey”, Springer Verlag LNCS 2693, 2003, pp. 207-224.
[3] Andrea Bonaccorsi and Cristina Rossi, "Why Open Source software may succeed”,
Research Policy, 32(7):1243–1258, 2003.
255
Appendix A
[4] Reidar Conradi and Alf Inge Wang (Eds.), Empirical Methods and Studies in
Software Engineering -- Experiences from ESERNET, Springer Verlag LNCS 2765,
ISBN 3-540-40672-7, Aug. 2003, 278 pages.
[5] M. Corra and D. Willer, “The Gatekeeper”, Sociological Theory, 20(2):180-207,
July 2002.
[6] Tore Dybå, "Factors of SPI Success in Small and Large Organizations: An
Empirical Study in the Scandinavian Context", Proc. Joint ESEC'03/FSE-11 conf.,
Helsinki, Finland, 1-5 Sept., ACM Press, pp.148-157.
[7] Marisa Escalante and Pedro Gutierrez, “CBSE: State of the Practice”, Proc. 16th
Int’l Conf. on Software & Systems Engineering and their Applications
(ICSSEA'2003), 2-4 Dec. 2003, Paris, Vol. 2, Session 9: Reuse & Components, ISSN
1637-5033, 16 p.
[8] EuroStat office, EU, Luxembourg, http://europa.eu.int.comm/eurostat, 2002.
[9] M. E. Fayad, M. Laitinen, R. P. Ward, “Software Engineering in the Small”,
CACM, 43(3):115-118, March 2000.
[10] Floyd D. Fowler, Survey Research Methods, 3rd edition, Thousand Oaks, Sage,
1988.
[11] W. B. Frakes and C. J. Fox, “Sixteen Questions About Software Reuse”, CACM,
38(6):75-87, June 1995.
[12] Robert M. Groves et al., Survey Methodology, Hoboken, NJ, John Wiley, 2004.
[13] Knut K. Holgeid, John Krogstie, Dag I. K. Sjøberg, “A study of development and
maintenance in Norway: assessing the efficiency of information systems support
using functional maintenance”, Information & Software Technology, 42(10):687-700,
2000.
[14] Italian Census Bureau, www.istat.it, 2002.
[15] Barbara A. Kitchenham et al., "Preliminary guidelines for empirical research in
software engineering", IEEE Trans. on Software Engr., 28(8):721-734, Aug. 2002.
[16] N. Y. Lee and C. R. Litecky, “An Empirical Study of Software Reuse with Special
Attention to Ada”, IEEE Trans. on Sw. Engineering, 23(9):537-549, Sept. 1997.
[17] A. S. Lee and R. L. Baskerville, “Generalizing Generalizability in Information
Systems Research”, Information Systems Research, 14(3):221-243, Sept. 2003.
[18] J. Li, F. O. Bjørnson, R. Conradi, and V. B. Kampenes, “An Empirical Study of
Industry", Proc. 10th IEEE Int’l Metrics Symposium (Metrics'04), Sept. 14-16 2004,
Chicago,p.72-83 (pre-study).
[19] J. Li, R. Conradi, O. P. N. Slyngstad, C. Bunse, M. Torchiano, and M. Morisio,
"Preliminary Results of a State-of-the-Practice Survey on Motivations of Using Off-
The-Shelf Components", in Proc. 6th International Conference on Product Focused
Software Process Improvement (PROFES'2005), 13-16 June, 2005, Oulu, Finland, 17
p. (on main study).
[20] P. Mohagheghi et al., "An Empirical Study of Software Reuse vs. Defect Density
and Stability", Proc. 26th Int’l Conf. on Software Engineering (ICSE'2004), 23-28
May 2004, Edinburgh, Scotland, IEEE-CS Press, pp. 282-292.
[21] K. J. Moløkken-Østvold et al., “A Survey on Software Estimation in the
Norwegian Industry”, Proc. 10th IEEE Int’l Metrics Symposium (Metrics'04),
Chicago, USA, 14-16 Sept. 2004, p. 208-219.
256
Appendix A
[22] M. Morisio et al., “Investigating and Improving a COTS-Based Software

Development Process”, Proc. 22nd Int’l Conference on Sw. Engineering (ICSE’00),
Limerick, Ireland, IEEE-CS Press, 4-11 June 2000, pp.31-40.
[23] M. Morisio, M. Ezran, and C. Tully, “Success and Failure Factors in Software
Reuse”, IEEE Trans. on Software Engineering, 28(4):340-357, April 2002.
[24] Nomenclature Generale des Activites Economiques dans I`Union Europeenne,
European Union, Brussels, 2005,
http://europa.eu.int/comm/competition/mer gers/cases/index/nace_all.html.
[25] S. L. Pfleeger and B. A. Kitchenham, “Principles of Survey Research”, 6 parts in
ACM SIGSOFT Engineering Notes, Nov. 2001 – March 2003.
[26] T. Punter, M. Ciolkowski, B. Freimut, and I. John, ”Conducting On-line Surveys in
Software Engineering”, Proc. 2nd Symp. of Emp. Sw. Eng. (ISESE’03), Rome, 30
Sept.–1 Oct. 2003, IEEE-CS Press, pp. 80-88.
[27] J. Ropponen and K. Lyytinen, “Components of Software Development Risk: How
to Address”, IEEE Trans. on Software Engineering, 26(2):98-111, Feb. 2000.
[28] Dag I. K. Sjøberg et al., “Conducting Realistic Experiments in Software
Engineering”, Proc. 1st Int’l Symp. on Empirical Sw. Eng. (ISESE’2002), 3-4 Oct.
2002, Nara, Japan, IEEE-CS Press, pp. 17-26
[29] Norwegian Census Bureau (SSB), Oslo, ICT company data 2002,
http://www.ssb.no/emner/10 /03/ikt/.
[30] J. Li et al.: "An Empirical Study of COTS Component Selection Processes in
Norwegian IT companies", MPEC'04 at ICSE ‘04, May 25, 2004.
[31] M. V. Zelkowitz and D. R. Wallace, "Experimental Models for Validating
Technology", IEEE Computer, 31(5):23-31, May 1998.
[32] M. Torchiano, M. Morisio, “Overlooked facts on COTS-based Development”,
IEEE Software, 21(2):88-93, March/April 2004.
[33] Infosector company, www.infosector.net
257
Appendix A
258
Appendix B
Appendix B: Other Papers

[AP1] [Li06e] Jingyue Li, Axel Anders Kvale, and Reidar Conradi: A Case Study on
Improving the Changeability of COTS-Based System Using Aspect-Oriented
Programming. Journal of Information Science and Engineering, Vol. 22, No. 2, March
2006, pp. 370-390.
Abstract: More and more software projects are using COTS (commercial-off-
the-shelf) components. One of the most challenging problems in COTS-based
development is to evolve a system to follow changes in the customer
requirements. It is therefore important to increase the changeability of the
COTS-based system, so that new component can easily replace the problematic
COTS components. Aspect-Oriented Programming (AOP) claims to make it
easier to develop and maintain certain kinds of application. We performed a case
study to investigate whether AOP can help to build an easy-to-change COTS-
based system. This compared the changes when adding and replacing
components between a COTS system implemented using Object-Oriented
Programming (OOP) and the same system implemented using AOP. The results
show that integrating COTS components using AOP may help to increase the
changeability of the COTS-based system if the cross-cutting concerns in the
code used to glue the COTS component are homogeneous (i.e., have a consistent
application of the same or very similar policy in multiple places). Extracting
heterogeneous or partially homogeneous cross-cutting concerns in glue-code as
aspects does not give any benefits. This study also discovered some limitations
in the AOP tool that makes it difficult or even impossible to integrate COTS
components with AOP.
Relevance to this thesis: This paper is relevant to the risk management
section of this thesis. It reports a case study on mitigating the one typical risk,
difficult to replace the problematic components, in COTS-based development.
My contribution: I contributed on 50% of the total work, including research
design, data analysis and paper writing. Axel Anders Kvale worked on
implementing the system, which includes 40% of the total work.
[AP2] [Li04b] Jingyue Li, Siv Hilde Houmb, Axel Anders Kvale: A Process to
Combine AOM and AOP: A Proposal Based on a Case Study. Proceedings of the 5th
Aspect-Oriented Workshop (In conjunction with UML 2004), Oct. 11, 2004, Lisbon,
Portugal, http://www.cs.iit.edu/~oaldawud/AOM/AcceptedPapers/
Abstract: Traditional object-oriented programming (OOP) paradigm focused

on structuring systems into distinguished objects that work together to realize a
system. However, when dealing with non-functional or quality requirements,
such as security and fault tolerance, these are not easily structured into separate
objects, but do rather crosscut a set of objects. Aspect-oriented programming
(AOP) separate crosscutting concerns into single units called aspects. Aspect-
oriented modelling (AOM) techniques allow system developers to design and
verify an aspect-oriented system on the modelling level. During a case study of
re-engineering an object-oriented system using aspect-oriented programming,
259
Appendix B
we learned that well-designed aspect-oriented modelling (AOM) is essential to

the success of aspect-oriented system. We also learned that current aspect-
oriented programming tools (AOP) pose limitations on the design of an ideal
aspect-oriented model. Based on lessons learned from this study we propose a
process that handles aspects at two levels, both at the modelling level (AOM)
and the programming language level (AOP). At the AOM level, aspects are
identified and weaved together by AOM weaving to verify and do trade-off
between different mechanisms. However, models are not woven together for the
purpose of code generation based on a combined model. The actual weaving is
done by the AOP compiler.
section of this thesis. It reports a proposal on avoiding the risks in COTS-based
projects from the requirement and design phase.
design, theory building and paper writing.
[AP3][Conradi05b] Reidar Conradi and Jingyue Li: Observations on Versioning of Off-

the-Shelf Components in Industrial Projects (short paper). Proceeding of the 12th
International Workshop on Software Configuration Management (co-located with the
ESEC/FSE 2005), Sept. 2005, Lisbon, Portugal, pp.33-42, ACM Digital Library.
Abstract: Using OTS (Off-The-Shelf) components in software projects has

become increasing popular in the IT industry. After project managers opt for
OTS components, they can decide to use COTS (Commercial-Off-The-Shelf)
components or OSS (Open Source Software) components. We have done a
series of interviews and surveys to document and understand industrial practice
with OTS-based development in Norwegian, German, and Italian IT industry.
The perspective is that of a software architect or system integrator, not a
developer or maintainer of such components. The study object is a completed
development project using one or several OTS components. This paper reports
on the versioning aspects of OTS components in such projects. We found that
one third of the COTS components actually provided source code, in addition to
all OSS components. However, OTS components were seldom modified (i.e.
reused “as-is”), even if source code was available. Although backward
compatibility of new releases did not cause noticeable problems for a single
OTS component, using several different OTS components in a project caused
difficulties in maintenance planning of asynchronous releases and system
integration of new releases. Several new research questions have been
formulated based on the results of this study.
section of this thesis. It reports on the versioning risks in OTS component-based
development.
My contribution: I contributed 40% percent of the total work, including
research design, data collection and data analysis.
[AP4] [Li05c] Jingyue Li, Reidar Conradi, Odd Petter N. Slyngstad, Marco
Torchiano, Maurizio Morisio, and Christian Bunse: Preliminary Results from a State-
260
Appendix B
of-the-Practice Survey on Risk Management in Off-The-Shelf Component-Based

Development. Proceedings of the 4th International Conference on COTS-based
Software System (ICCBSS 2005), Feb. 2005, Bilbao, Spain, Springer LNCS Vol. 3412,
pp. 278-288.
Abstract: Software components, both Commercial-Off-The-Shelf and Open

Source, are being increasingly used in software development. Previous studies
have identified typical risks and related risk management strategies for what we
will call OTS-based (Off-the-Shelf) development. However, there are few
effective and well-proven guidelines to help project managers to identify and
manage these risks. We are performing an international state-of-the-practice
survey in three countries - Norway, Italy, and Germany - to investigate the
relative frequency of typical risks, and the effect of the corresponding risk
management methods. Preliminary results show that risks concerning changing
requirements and effort estimation are the most frequent risks. Risks concerning
traditional quality attributes such as reliability and security of OTS component
seem less frequent. Incremental testing and strict quality evaluation have been
used to manage the possible negative impact of poor component quality.
Realistic effort estimation on OTS quality evaluation helped to mitigate the
possible effort estimation biases in OTS component selection and integration.
section of this thesis. It reports risk management issues based on our preliminary
data.
design, questionnaire design, data collection in Norway, data analysis and paper
writing.
[AP5] [Li05d] Jingyue Li, Reidar Conradi, Odd Petter N. Slyngstad, Marco
Torchiano, Maurizio Morisio, and Christian Bunse: An Empirical Study on Off-the-
Shelf Component Usage in Industrial Projects. Proceeding of the 6th International
Conference on Product Focused Software Process Improvement, Oulu, Finland
(PROFES 2005), June 2005, Springer LNCS Vol. 3547, pp. 54-68.
Abstract: Using OTS (Off-The-Shelf) components in software projects has

become increasing popular in the IT industry. After project managers opt for
OTS components, they can decide to use COTS (Commercial-Off-The-Shelf)
components or OSS (Open Source Software) components instead of building
these themselves. This paper describes an empirical study on why project
decision-makers use COTS components instead of OSS components, or vice
versa. The study was performed in form of an international survey on motivation
and risks of using OTS components, conducted in Norway, Italy and Germany.
We have currently gathered data on 71 projects using only COTS components
and 39 projects using only OSS components, and 5 using both COTS and OSS
components. Results show that both COTS and OSS components were used in
small, medium and large software houses and IT consulting companies. The
overall software system also covers several application domains. Both COTS
and OSS were expected to contribute to shorter time-to-market, less
261
Appendix B
development effort and the application of newest technology. However, COTS

users believe that COTS component should have good quality, technical support,
and will follow the market trend. OSS users care more about the free ownership
and openness of the source code. Projects using COTS components had more
difficulties in estimating selection effort, following customer requirement
changes, and controlling the component’s negative effect on system security.
Relevance to this thesis: This paper is relevant to the decision making
section of the thesis. It reports our preliminary findings based on 115 projects.
design, questionnaire design, data collection in Norway, data analysis and paper
writing.
[AP6] [Li04c] Jingyue Li, Finn Olav Bjørnson, Reidar Conradi, and Vigdis By
Kampenes: An Empirical Study of COTS Component Selection Processes in Norwegian
IT companies. Proceeding of the Int'l Workshop on Models and Processes for the
Evaluation of COTS Components (MPEC'04 Arranged in co-location with ICSE'04),
May, 2004, Edinburgh, Scotland, IEE Press, pp. 27-30.
Abstract: The use of commercial-off-the-shelf (COTS) software has become

more and more important in software development. In COTS-based
development, COTS component selection is the most crucial phase. Although
some selection processes have been proposed, empirical studies are necessary to
assess these processes. We describe an exploratory study by structured
interviews of 16 COTS-based development projects in Norwegian IT companies.
The results indicate that successful COTS component selection can be
implemented without using formal processes, and projects with different
contexts may use different selection processes. If members in new project has
enough practical experience with actual COTS components, such experience can
be the dominant factor in selection. In the case of using a new COTS component
in the project, hands-on experimentation is needed as an effective way of
evaluating the component.
Relevance to this thesis: This paper is relevant to the SPI section of the
thesis. It reports the results related to COTS selection in the pre-study.
My contribution: I contributed to 80% of the total work, including research
design, interview guide design, performing most interviews, data analysis and
paper writing.
[AP7] [Li04d] Jingyue Li, Finn Olav Bjørnson, and Reidar Conradi: Empirical Study on
COTS Component Classification, Proc. International Workshop on COTS Terminology
and Concepts, Co-located with International Conference on Component-Based Software
Systems (ICCBSS'04), 1-4 Feb., 2004, Redondo Beach, USA,
http://softeng.polito.it/iwctc/accepted.html
Abstract: COTS-based development is gaining more and more attention.

Effective COTS component classification will help integrators to successfully
control the development process, such as selection and integration. In this paper,
we present an empirical study to investigate the characterized classification
262
Appendix B
proposed by previous research. From the result of this study, we conclude that
some attributes are not good because they are either not measurable or
unnecessary. We also propose one other attribute that will affect the
development process dramatically. Our future study will focus on investigating
these attributes in a larger sample.
Relevance to this thesis: This paper is relevant to the SPI section of the
thesis. It reports the results related to COTS classification.
My contribution: I contributed to 80% of the total work, including research
design, interview guide design, performing most interviews, data analysis and
paper writing.
263
Appendix B
264
Appendix C
Appendix C: Interview Guide for the Pre-study

Our definition on concepts used in this questionnaire:
Project: Software development project, in which the respondent has been involved. We
aim for finished projects and possibly with maintenance, and possibly with several
releases.
Component: Software components are executable units of independent production,
acquisition, and deployment that can be composed into a functioning system. To enable
composition, a software component adheres to a particular component model and targets
a particular component platform.
COTS (Commercial Of The Shelf) component: A component that:
• is either provided by some other organizations in the same company, or
provided by external companies as a commercial product.
• is integrated into the final delivered system.
• is not a commodity, i.e. not shipped with operating system, not provided with
the development environment, not generally included in any pre-existing
platform
• is not controllable, in terms of provided features and their evolution
System: Product delivered by the actual project, consisting of application plus

components, but again excluding platform software (commodities like OS, DBMS etc.).
Application: Code built in-house to provide certain functionalities of the system.
Glueware: Code to solve possible mismatch between components, or between
components and application/platform.
Addware: Code to add the functionalities which are required but not provided by the
components.
The relationship between the four last concepts is shown in the following figure.
System
Glueware/
Addware
Application
COTS
Components
Platform (OS, DBMS)
Questions for Comments:

1. Are the definitions easy to read as worded? □ Yes □ No
2. Do you need clarification for these definitions? □ Yes □ No
3. If you have comments on these definitions, please fill in here.
4.Can you give a example of the COTS component based on your understanding?
265
Appendix C
Section 1: Questions on the context of the project and system
1.1 Information about the project
The projects we want to investigate in this survey are finished software development
project using one or more COTS components. If you have experience with several
such projects, please select one that you are most familiar with and base your answers
on this project.
1.1.1 What was the total staff size of the project? (full-time and part-time persons)
Full time: _________________
Part time: _________________
1.1.2 What was the starting time of the project (mm/yy) / ?
1.1.3 What was the time of final delivery of the project (mm/yy) ___/____?
1.1.4 What was the development environment of the project?
• Programming Languages: ____________________
• CASE tools: ______________________________
• Overall develop process (e.g. waterfall, prototyping, incremental, etc.):
_________
1.2 Information about the system
1.2.1 What were the main functionalities of the system?
1.2.2 What were the major types of industry / services (application domain) of the
customers of the system?
[Mark one or more alternatives]
o Agriculture and forestry, fishing and o Professional, scientific, and
hunting technical service
o Mining and quarrying o Management of companies and
o Power, electricity, gas, water supply enterprises
o Construction o Administration and support
o Manufacturing services
o Whole trade o Waste management services
o Retail trade o Educational services
o Transport, storage, and communication o Health care and social assistance
o Information(informatics/telecommunica o Arts, entertainment, and
tion ) recreation
o Finance, Insurance o Accommodation and food
o Real estate and rental and leasing services
o Public administration
o Other sevices
Reference: North American Industrial Classification System (NAICS)
Link: http://www.revenue.state.ne.us/tax/current/buscodes.pdf
1.2.3 What is the executing environment of the system?

z OS : ____________________________________________
z DBMS: __________________________________________
z Other Supporting Software: __________________________
266
Appendix C
1.2.4 What is the architecture of the system?

If the architecture is essentially uniform, mark the one style which best describes
that architecture. If the architecture is a mix of multiple architectural styles, mark as
many styles as needed. [Mark one or more alternatives]
o Pipe & Filter o Event Based
o Real Time o Layered
o Distributed (Client/Server) o Multithreaded
o Rule Based o Blackboard/ Repository
o Main/Subroutine o Closed Loop
o Transactional Database Centric o Feedback Control
o Other:____________________
1.2.5 What is the size of the delivered system?

Application : _____________ KB or __________________ (KLOC)
Glueware : _____________ KB or __________________ (KLOC)
Addware: _______________KB or __________________ (KLOC)
COTS :__________________ KB or __________________ (KLOC)
1.2.6 What is the current version/release of the system (Version/Release) __ / __?

1.2.7 What is the total age of the current system after initial delivery (Years /
Months)?
1.2.8 What was the emphasis on the Time-to-market of the system development?
• High: There was very restricted deadline. The time for system development was
limited.
• Nominal: There was not very restricted deadline. The time for system
development was fair.
• Low: There was no specific deadline. The time for system development was
enough.
High Nominal Low Don’t know
1.2.9 What was the emphasis on the Reliability (failure rate) of the system?
• High: If a failure occurs, the losses are very high and are very hard to be
recovered.
• Nominal: If a failure occurs, losses are fairly easily recoverable
• Low: If a failure occurs, losses are easily recoverable (e.g., document
publishing)
1.2.10 What was the emphasis on the Security of the system?

• High: There were very specific requirements on preventing unauthorized access
to, or modification of computer programs or data.
• Nominal: There were some specific requirements on preventing unauthorized
access to, or modification of computer programs or data.
• Low: There were little specific requirements on preventing unauthorized access
267
Appendix C
to, or modification of computer programs or data.

1.2.11 What was the emphasis on the Performance of the system?

• High: There were very strict requirements on the system response time.
• Nominal: There were some requirements on the system response time.
• Low: There were little requirements on system response time.
Section 2. Questions on COTS components
2.1 What is the type of your COTS-based system?

[Mark one alternative]
□ COTS-Solution system System relies on one (or more) core COTS
component(s). The system is built around these
components. COTS components provide the main
functionality of the system.
□ COTS-Integrated System System is built by integrating one or more COTS
components around the central application. COTS
components provide some functionality for the
system.
2.2 Which COTS components were used in the project? And what kind of
functionality did they provide? (If there were more than three COTS
components used in the project, please select the COTS components most
relevant to your job in the project.)
[Use text to answer the name and functionalities of COTS component]
COTS ID COTS COTS Component Standard Main Functionality
Component (COM, CORBA, EJB etc.)
Name
1
2
3
2.3 What was the highest level of stated management commitment to these COTS
component in the project? (Use the COTS ID given to your components in question
2.2)
• Business top manager: The highest level of stated commitment to use this
component is business top manager
• IT top-level manager: The highest level of stated commitment to use this
component is IT-top manager
• IT project managers: The highest level of stated commitment to use this
component is IT project manager
• No commitment: There is no management commitment to use this
components, it is decided by software architects or
268
Appendix C
developers
• Don’t know
[Mark one alternative per row below]
COTS Business IT top-level IT project No Don’t know
ID Top manager manager commitment
manager
1
2
3
2.4 What was the origin of these COTS components?

(Use the COTS ID given to your components in question 2.2)
• Component produced by contract: The COTS was acquired by asking a
component vendor to create a component for an agreed-upon fee.
• Special version of commercial item: The COTS was produced by a vendor who
produces a customized version of a commercially available component.
• Independent commercial item: The COTS was bought from market, the vendor
is selling a product to the world and has no knowledge of the individual
purchaser’s requirements or constraints.
COTS ID Component Special version of Independent commercial
produced by commercial item item
contract
1
2
3
2.5 What was your project members’ experience with these COTS components?
• Very much experience: All project members have the experience of using the
same COTS components.
• Much experience: Roughly ¾ project members have the experience of using
the same COTS components.
• Some experience: Roughly ½ project members have the experience of using the
• Little experience: Roughly ¼ project members have the experience of using the
• No experience: No project member has the experience of using the same COTS
components.
COTS ID Very much Much Some Little No
experience experience experience experience experience
1
2
3
269
Appendix C
2.6 What was the role of the COTS components at the architecture level of the
system?
• Core: The COTS component provided the main functionality of the system.
The system was built around this COTS component.
• Support: The COTS component provided some functionality to support the
central application of the system.
COTS ID Core Support
1
2
3
2.7 What was the most influential information/method to support selection of COTS
components? (Use the COTS ID given to your components in question 2.2)
• Experience: Based on own or other COTS users’ previous success or failure
experience.
• Demonstration: Based on the result of the vendors’ demonstration.
• Pilot: Based on the result of a short-term trial use or test of COTS.
• Formal method: Based on the result of formal decision making process, such as
multi-attribute utility theory, Multi-criteria decision aid, weighted sum method,
etc.
• Others: There are other information/methods except the alternatives listed
above. Please specify
COTS ID Experience Demonstration Pilot Formal Others
method (Please
Specify)
1
2
3
2.7.1 If you didn’t use Formal Selection Method (see question 2.7), why?
• I never heard about it
• It isn’t cost-effective
• It is not necessary
• It is not easy/accurate to use
• Others______________________
2.8 Please rank-order your top three non-technical factors based on their influence in
COTS components selection. Mark three factors with rating 1, 2, and 3 (1 is the most
influential factor, 3 is the least influential one)
Licensing arrangements
Cost in general
Market trend
COTS reputation (maturity, stability)
270
Appendix C
Vendor reputation
Vendor support capability
Others (please specify): _________________________
2.9 Please specify the importance of the following issues for the initial selection of
COTS components.
• Integrations easiness: Whether the COTS components can be plugged into the
system without extra integration code.
• Possible integrations effort: How much extra code may need to be produced to
correct the possible mismatch between the COTS components and other parts
of the system.
• Functionalities completeness: Whether the components have all the required
functionalities.
• Possible functionalities supplement effort: How much effort may be used to
add the required functionalities.
Scales:
• Very Much: It is considered as the most important issue
• Much: It is considered as a very important issue
• Some: It is considered as an important issue
• Little: It is not considered as an important issue
• No: I didn’t consider this issue.
Very Much Some Little No
much
Integrations easiness
Possible integrations effort
Functionalities completeness
Possible functionalities supplement
effort
Other issues:
2.10 Characterize the causes of integration problems between COTS components, or

integration problems between COTS components and other parts of the system.
• Do not support architecture style: The COTS components did not support the
requirement of architecture style of the system.
• Support different architecture styles: The COTS component supported different
architectural style with other components in the system.
• Do not support committed standard: The COTS components was claimed to
support specific standard (COM, CORBA, EJB, etc.), but they didn’t
• Support different standards: The COTS components supported different standard
(COM, CORBA, EJB, etc.)
• Support different standard versions: The COTS components supported different
version of standard (COM, CORBA, EJB, etc.)
• Other problems: There are other integration problems, please specify;
Scales:
271
Appendix C
• Very much: Almost all the integration effort was spent on solving this problem
• Much: Roughly ¾ integration effort was spent on solving this problem
• Nominal: Roughly ½ integration effort was spent on solving this problem
• Some: Roughly ¼ integration effort was spent on solving this problem
• Little: Very few effort was spent on solving this problem
• No: No such problem, no integration effort was spent on solving this problem
[Mark either No problem or one or several other alternatives per row below]
Problem Very Much Nominal Som Little No
much e
Do not support architecture
style
Support different architecture
style
Do not support committed
standard
Support different standards
Support different standard
versions
Other problems (Specify):
2.11 How the COTS components were integrated into the system (possibly to solve
the problems mentioned in 2.10)?
(Use the same COTS components and COTS ID as in question 2.4)
• By Plug: Just plug in the COTS components to assemble an executable system
without any extra work.
• By Configuration: Setting or defining shell parameters or configuration options
available for a COTS component, but which do not require modification of
COTS source code.
• By Glueware: integrate COTS components mostly by using glueware (i.e. code
to solve possible mismatch between components, or between components and
application/platform.)
• By Addware : integrate COTS components mostly by using addware (i.e. code
to add the functionalities which are required but not provided by the
components.)
• By Source Modification: change the source code of COTS components so that
it can be integrated
[Mark either By plug or one or several other four alternatives per row ]
COTS By Plug By By By Addware By Source
ID Configuration Glueware Modification
1
2
3
2.12 What level of training and/or documentation for the COTS components was
available AND obtained for the developers, either directly from the component vendors
or through third parties?
272
Appendix C
• Very High: As much training and/or documentation obtained as needed

• High: Roughly ¾ of the needed training and/or documentation obtained
• Nominal: Roughly ½ of the needed training and/or documentation obtained
• Low: Roughly ¼ of the needed training and/or documentation obtained
• Very Low: No training and/or very little documentation obtained
COTS ID Very High High Nominal Low Very Low
1
2
3
2.13 How did you try to influence a COTS components vendor to change components
to your own needs? [Mark one alternative per row below]
• No control: Cannot drive vendor to make the change and have to follow the
market
• Pay vendor: Pay the vendors to change it to meet my requirements
• Press vendor: Put pressure on the COTS vendor in order to make them change
the OTS wthout pay them
• Merge with or buy vendor: Merge with or buy the COTS vendor firm and make
them change the component
• Other methods:___________________
COTS ID No control Pay vendor Press Merge or Others

vendor buy vendor (specify)
1
2
3
2.14 What was the effect on certain properties by using COTS components in your
system?
• Strong positive effect: Using COTS contributed much in achieving the planned
result
• Some positive effect: Using COTS contributed some in achieving the planned
result.
• No effect: Using COTS had no contribution in achieving the planned result
• Some Negative effect: Using COTS hindered some in achieving the planned
result
• Strong Negative effect: Using COTS hindered much in achieving the planned
result
Perceived effect The effect on certain properties
on certain
properties Strong Some No Some Strong Don’t
positive positive effect Negative Negativ know
effect effect effect e
effect
273
Appendix C
Time-to-market
System
Reliability
System security
System
performance
Development
cost
Maintenance
cost
2.15 Would you like to use these COTS components in the future projects? [Mark
one alternative]
□ Yes, I would like to use these COTS components in the future projects.
□ No, I don’t want to use these COTS components in the future projects.
□ Don’t know.
Section 3. Questions on the context of respondent and business unit
3.1 Personal Information about the respondent

3.1.1 What is your position in the business unit? [Mark one or more alternatives]
□ IT manager
□ IT project manager
□ Software architect
□ Software developer
□ Other level (please specify): _______________
3.1.2 How long have you been working in the current business unit?
Years: __________________________
3.1.3 How long have you been working with software development?
Years: _________________________
3.1.4 How long have you been working with COTS components?
Years: __________________________
3.1.5 What is your principal education degree [Mark one alternative]?
□ Bachelor
□ Master
□ Ph.D
□ Other education: ___________________
3.1.6 Is your principal degree in software (informatics/computer science/telematics)?
□ Yes
□ No
3.2 Information about your Company
3.2.1 Type of your Company
□ Stand-alone: The highest reporting entity with no parent organization above it.
□ Subsidiary: An independent entity with majority interest held by a parent.
3.2.2 Ownership of your Company
274
Appendix C
□ Publicly traded (A entity whose stock is publicly traded on stock exchange)

□ Privately held company (A for-profit entity that is privately owned)
□ Government, education, or nonprofit organization
3.2.3 Staff size of your Company (full-time and part-time persons)?
Full time:____________________________________________
Part time:____________________________________________
3.2.4 What is the main business areas of your Company [Mark one alternative]
□ IT/Telecommunication industry (e.g. Ericsson, Telenor, etc.)
□ Software House / Software Vendor (e.g. Opera)
□ Software / IT consulting company (e.g. Accenture)
□ Others software intensive company (Please specify):
Section 4. Extra questions (will only be used in the personal interview, the
alternatives of each question are different viewpoints that you can talk with.)
4.1 Motivation of using COTS components

Why did you use COTS components instead of building components yourself?
□ Business issue (time-to-market, cost effective, partner adaptability… )
□ Technical issue (catch up with new technology, reliability, security,
maintainability…)
□ Organization (key technical person leave…)
□ Others:_______________________________
4.2 Process issues
4.2.1 Please describe briefly the overall development process used in the project?
(Elaborate on question 1.1.4)
4.2.2 Did your business unit have any specific processes concerning the selection and
use of COTS components before the project started?
If yes, please explain briefly what kind of processes:
(e.g. activities, roles, responsibilities)
4.2.3 Did you have to introduce or make any process changes during the project
related to the selection or use of COTS components?
If yes, please explain briefly the situation and the changes introduced:
(e.g. new or changed activities, roles and responsibilities)
4.2.4 What impact on the development process did the use of COTS components
have for future project?
o ease of change of software o redistribution issues (both legal and

o ease of integration of software commercial)
o availability of certification o adaptability
o language/middleware lock-in o learnability
o control on product customization o setup effort
o architectural constraints and o reusability across projects
mismatches o availability of required functionality
o marketplace competition o different integration problems
275
Appendix C
o porting and adaptation effort o the choice of the product can be

o acquisition and maintenance cost dictated by external factors
o configuration management o customization cost
o platform constraints integration
effort
4.3 Component selection

4.3.1 Did you (re) negotiate the requirements with the customer? What was the
reason?
o The requirements had to be (re) negotiated, mainly because the software
architecture style has been decided separately and could only partially satisfy
the requirements.
o The requirements had to be (re) negotiated, mainly because the available COTS
components could not fully satisfy them.
o Other reasons :___________________________________________
o No requirements (re) negotiation was performed.
4.4 System maintenance and evolution issues

4.4.1 What part of the system had the largest amount of defects (trouble reports)
during the first 6 month of system maintenance? [Mark one alternative]
□ The application part of the system
□ The COTS components of the system
□ The glueware/addware of the system
□ Don’t know
4.4.2 What part of the system had the largest amount of change requests (not
covering trouble reports, e.g. upgrade) during the first 6 months of system
maintenance? [Mark one alternative]
□ The application part of the system
□ The COTS components of the system
□ The glueware/addware of the system
□ Don’t know
4.4.3 How could you ensure the quality (reliability, security, etc.) of your COTS-
based system in the process of system evolution (system update, COTS components
update, etc.)?
o Technical (testing, configuration management, version control, ...)
o Vendor relationship (technical support, contract, …)
o Others:______________________________
4.5 General lessons learned and experience of using COTS components in your
system
o Business
o Technical
o Process
o Organization
o Others: ______________________________
276
Appendix C
In the questionnaire we used, there is a meta-question after some question (1.2.4, 1.2.5,
1.2.8 to 1.2.11, 2.1, 2.4 to 2.14, and 4.4) to evaluate the quality of the questions as
following. Due to the space reason here, they have been removed.
Questions for Comments:

1. Is this question easy to read as worded?
□ Very easy to read □ Easy to read □ Hard to read
2. Do you need clarification for this question?
□ I am totally confused with this question
□ I need some clarifications for this question
□ I don’t need clarifications for this question
3. If you have comments on this question, please fill in here.
277
Appendix C
278
Appendix D
Appendix D: Questionnaire in the Main Study

Our definition of concepts used in this questionnaire:
Project: A software development project, in which the respondent has been involved.
We aim for completed projects, possibly with maintenance, and possibly with several
releases.
Customer: Representing either an end-user, or a contractor to which your organization
is a subcontractor.
Component: Software components are program units of independent production,
acquisition, and deployment that can be composed into a functioning system. We limit
ourselves to components that have been explicitly decided either to be built from
scratch or to be acquired externally as an OTS-component. That is, to components that
are not shipped with the operating system, not provided by the development
environment, and not included in any pre-existing platform. That is, platform
(“commodity”) software are not considered, e.g. an OS like Linux, DBMSes, various
servers, or similar software. Furthermore, components usually follow some component
model, as expressed by the COM, CORBA, EJB, .Net, or Web Service standards, or
they can be a C++/Java library.
OTS (Off-The-Shelf) component: A component that:
• is provided (by a so-called provider) from:
- a commercial vendor (Commercial-Off-The-Shelf, COTS), or
- the Open Source community (Open Source Software, OSS).
The components may come with certain obligations, e.g. payment or licensing
terms. NB: we do not consider proprietary components developed by some
other organization in the same company, i.e. internal reuse.
• is not controllable, in terms of provided features and their evolution.
• is mainly used as closed source, i.e. no source code is usually modified, even it
is available.
Typical COTS components are: Vector draw professional (3D modelling),
Shamman Reports for Excel (Database reporting), Dynamic Cube (OLAP) etc.
Typical OSS components are: Xerces (XML parser), Kiwi Toolkit (Class library),
OpenEJB, Python (Interpreter) etc.
System: Product delivered by the actual project, consisting of application plus
components, but again excluding platform software.
Application: Code built in-house to provide certain functionalities of the system.
Glueware: Code to solve possible mismatch between components, or between
components and application/platform.
Addware: Code to add the functionalities which are expected but not provided by the
components.
The relationship between the four last concepts is shown in the following figure:
279
Appendix D
Question for comments. Can you mention some OTS components based on your
understanding?
Section 1: Questions on the context of the project and system
1.1 Information about the project

The projects we want to investigate in this survey are completed software
development projects using one or more OTS components. If you have experience
with several such projects, please select the one that you are most familiar with, and
base your answers on that project.
1.1.1 What was the total staff size of the project (full-time and part-time persons)?
Full time: _________________
Part time: _________________
1.1.2 How large a percentage of the project members have previous experience with
general
OTS-based development __ (%)?
1.1.3 What was the starting time of the project (mm/yy) / ?
1.1.4 What was the time of first complete delivery from the project (mm/yy)
___/____?
1.1.5 What was the development environment of the project (describe textually)?
• Programming Languages: ____________________
• CASE (Computer Assistant Software Engineering) tools: _______________
• Main development method (RUP etc.): _______________
1.2 Information about the system
1.2.1 What were the names and the main functionalities of the system (describe
textually)?
1.2.2 What was the emphasis on the following characteristics of the system?
[Mark one alternative per row]
Very Low Medium High Very Don’
low high t
know
a) Time-to-market
b) Effort (cost)
c) Reliability
d) Security
e) Performance
f) Maintainability
280
Appendix D
g) New functionality (first

launch in the market)
h) Improved functionality
(over competitors)
i) Other (please specify):
1.2.3 What were the major types of industry / services (application domain) of the
customers of the system? [Mark one or more alternatives]
□ Bank/ Finance/ Insurance
□ Other private services (consulting, wholesale, retail, entertainment, …)
□ Public sector
□ ICT sector
□ Traditional industry/engineering/construction
□ Other (please specify): _______________
1.2.4 What is the executing environment of the system (describe textually)?
z OS (Operating System):_____________________________________
z DBMS (Database Management System): _______________________
z Other Supporting Software: __________________________________
1.2.5 What is the current version/release of the system (Version/Release) ___/___?
Section 2: Questions on the motivation of using OTS components

2.1 To what extent did you expect the chosen OTS component(s) to contribute to the
following project/system targets? [Mark one alternative per row]
Project/system targets Don’t Hardl Agree Agree Strong Don’

agree y some mostly ly t
at all agree what agree know
a) Shorter time-to-market
b) Less development
effort/cost
c) Less maintenance
effort/cost
d) Larger market share
e) Compliance with industrial
standards
f) Keeping up with the newest
technology
g) Better system reliability
h) Better system security
i) Better system performance
j) Other (please specify):
2.2 What were the specific motivations for using OSS Component(s) in your project?
[NB: If you did not use an OSS Component in the project, please skip this
question]
281
Appendix D
Motivations Don’t Hardl Agree Agree Stron Don’t

agree y some mostl gly know
at all agree what y agree
a) Reduce risk of provider going
out of business
b) Reduce risk of provider
changing market strategies
c) Reduce risk of selected
component evolving into an
unwanted direction
d) Component can be acquired
for free
e) Source code is available and
can easily be changed
f) It was decided by the
customer
g) Political reasons (company
policy, licensing conditions)
h) Other (please specify):
2.3 What were the specific motivations of using COTS Component(s) in your project?
[NB: If you did not use a COTS Component in the project, please skip this
question]
Motivations Don’t Hardl Agree Agree Stron Don’t

a) Reduce risk of bad quality,
because components were paid
for
b) Reduce risk of poor security,
because source code was closed/
unavailable
c) Provider could give adequate
technical support
d) Provider may have
investigated the market trends,
and will give a component
following them
e) It was decided by customer
f) Political reasons (company
policy, licensing conditions)
g) Other (please specify):
282
Appendix D
Section 3: Questions on the general development processes

3.1 Please describe the main development process (e.g. waterfall, prototyping,
incremental, agile etc. – describe textually):
3.2 Who decided the main (i.e. lifecycle) process? [Mark one or more alternatives]
□ Company/department rules
□ Project manager
□ Customer
□ Other (please specify): _______________
3.3 When was the main process decided? [Mark one alternative]
□ Before the “acquire vs. build” decision, i.e. before the decision whether to
acquire
OTS components instead of building them in-house
□ After such an “acquire vs. build” decision
3.4 Were there any dedicated personnel in your project to keep track of OTS
component knowledge/information? [Mark one alternative]
□ Yes
□ No
If Yes, what is the role and OTS component-relevant experience of this person?
Section 4: Questions on the project level risk management

4.1 What is your opinion on the following aspects (risks) of your OTS-based project?
Don’t Hardl Agree Agree Stron Don’t
a) The project was delivered long
after schedule
b) Effort to select OTS
components was not satisfactorily
estimated
c) Effort to integrate OTS
components was not satisfactorily
estimated
d) OTS components negatively
affected system reliability
e) OTS components negatively
affected system security
f) OTS components negatively
affected system performance
283
Appendix D
g) Requirements were changed a

lot
h) OTS components could not be
sufficiently adapted to changing
requirements
i) Project could not (re)negotiate
requirements with the customer,
if OTS components could not
satisfy all requirements
j) It was difficult to identify
whether defects were inside or
outside the OTS components
k) It was difficult to plan system
maintenance, e.g. because
different OTS components had
asynchronous release cycles
l) It was difficult to update the
system with the last OTS
component version
m) OTS components were not
satisfactorily compatible with the
production environment when the
system was deployed
n) Information on the reputation
and technical support ability of
provider were inadequate
o) Provider did not provide
enough technical support/
training
4.2 What actions were performed during development or maintenance of the project?

a) Customer had been actively
involved in the “acquire” vs.
“build” decision of OTS
components
b) Customer had been actively
involved in OTS component
selection
c) OTS components were
selected mainly based on
architecture and standards
compliance, instead of expected
284
Appendix D
functionality
d) OTS components qualities
(reliability, security etc.) were
seriously considered in the
selection process
e) Effort in learning OTS
component was seriously
considered in effort estimation
f) Effort in black-box testing of
OTS components was seriously
considered in effort estimation
g) Unfamiliar OTS components
were integrated first
h) Did integration testing
incrementally (after each OTS
component was integrated)
i) Local OTS-experts actively
followed updates of OTS
components and possible
consequences
j) Maintained a continual watch
on the market and looked for
possible substitute components
k) Maintained a continual watch
on provider support ability and
reputation.
Section 5: Questions on the OTS component level risk management

5.1 How many different OTS components were used in the project (Please fill in a
number)?
5.2 Which were the name, version and functionality of OTS components used in the
project?
[NB: If there were more than five OTS components used in the project, please
choose five you are most familiar with. Also select what you think is the most
important OTS component and fill it into component Comp.1 for later use.]
OTS Full name and Origin (write Main functionality
component version (descriptive COTS or OSS) (descriptive free text)
ID free text)
Comp.1
Comp.2
Comp.3
Comp.4
Comp.5
285
Appendix D
5.3 How large a percentage of the application functionality in the whole system was
provided by the selected OTS component Comp.1 ___________________(%)?
5.4 What was the source code status of the selected OTS component Comp.1?
[Make one alternative per row]
OSS, i.e. with source COTS, but with source COTS, without
code available code available source code
Comp.1
5.5 Have you read parts of the source code of OTS component Comp.1?
[NB: If the source code of Comp.1 is not available, please skip this question]
Very Little Some Much Very Don’t
little much know
Comp.1
5.6 Have you modified parts of the source code of OTS component Comp. 1?
[NB: If the source code of Comp.1 is not available, please skip this question]
Very Little Some Much Very Don’t
little much know
Comp.1
5.7 What was the technical base of the selected OTS component Comp.1? [Mark one
or more alternatives]
o .Net o DLL
o ActiveX o VBX/ VB Class Library
o Java Class/ Library o COM/DCOM
o Java Applet/ Servlet o CORBA
o Java Bean/Enterprise Java Bean o Windows Static Link Library
o C/C++ class/Library o Windows Foundation Class (WFC)
o MFC o Others (specify):____________
5.8 What was your project members’ experience with the selected OTS component
Comp.1? [Make one alternative per row]
Very little Little Some Much Very Don’
much t
know
Comp.1
5.9 When were the selected OTS components Comp.1 selected? [Make one alternative
per row]
Pre-study Requirem Overall Detailed Coding Don’t
phase ents/Analy design design phase know
sis phase phase phase
Comp.1
5.10 Did you encounter some of the following aspects (risks) with the selected OTS
component Comp.1? [Make one alternative per row]
286
Appendix D
Yes No Don’t
know
a) The OTS component did not follow the industry standard
(COM, CORBA etc.) as the provider claimed (i.e. glueware
needed)
b) The OTS component had mismatching architectural
assumptions towards other parts of the system (i.e.
glueware needed)
c) The OTS component could not provide all required
functionality (i.e. addware needed)
d) The recent OTS component versions were not
backwards-compatible with the first version
e) The provider did not want to or could not change OTS
component functionality as requested
5.11 Have you performed some of the following actions for the selected OTS
component Comp.1?[Make one alternative per row]
Yes No Don’
t
know
a) Searched Internet for possible OTS component
candidates.
b) Got recommendation of possible OTS component
candidates from the customer
c) Got recommendation of possible OTS component
candidates from a local colleague/OTS-expert
d) Used a formal decision-making method to compare
possible OTS component candidates, e.g. with weighted
evaluation criteria
e) Limited possible candidates into 1-3 components, by
reading literature or other documentation
f) Did “hands-on” try-out of 1-3 components, e.g. on a
down-loaded demo version.
5.12 What is your opinion on the following aspects of the selected OTS component
Comp.1? [Make one alternative per row]
a) Comp. 1 was integrated
successfully
b) Comp. 1 was maintained
successfully so far
287
Appendix D
Section 6: Questions regarding the context of respondent and business

unit
[NB: The supplied information in 6.1 and 6.2 will be treated confidentially.]
6.1 Personal Information about the respondent

6.1.1 What was your position in this project? [Mark one or more alternatives]
□ Project manager
6.1.2 What is your current position in your business unit? [Mark one or more
alternatives]
□ IT manager
□ Project manager
6.1.3 How long have you been working in the current business unit?
Years: __________________________
6.1.4 How long have you been working with software development?
Years: _________________________
6.1.5 How long have you been working with OTS components?
Years: __________________________
6.1.6 How many OTS component based projects have you been involved in?
Projects: __________________________
6.1.7 What is your principal education degree? [Mark one alternative]
□ Bachelor
□ Master
□ Ph.D.
□ Other education: ___________________
6.1.8 Is your principal degree in software (informatics/computer science/telematics)?
□ Yes
□ No
6.2 Information about your company
6.2.1 What is the type of company? [Mark one alternative]
□ Stand-alone: The highest reporting entity with no parent organization above it.
□ Subsidiary: An independent entity with majority interest held by a parent.
6.2.2 What is the ownership of your company? [Mark one alternative]
□ Publicly traded on a stock exchange
□ Privately held company
□ Government, education, or nonprofit organization
6.2.3 What is the staff size of your embedding company (full- & part-time persons)
____?
What is the staff size of your local business unit (full- & part-time persons) ___?
[This staff size may be the same for small and medium sized companies]
288
Appendix D
6.2.4 What is the main business areas of your company?[Mark one alternative]
□ IT/ Telecom industry (Ericsson, Nokia, NERA, Nordic VLSI etc.)
□ Telecom service provider (Telenor, NetCom etc.)
□ Software House / Software Vendor (Opera, MicroSoft etc.)
□ Software / IT consulting company (Accenture, Cap Gemini E&Y, TietoEnator
etc.)
□ Other software-intensive company (please specify): ___________________
Note: In the main study, the questionnaire was translated into local language
(Norwegian, German, and Italian) and published in the web for filling in.
289

Process Improvement and Risk Management in Off-The-Shelf Component-Based Development

Uploaded by

Document Information

Original Title

Copyright

Available Formats

Share this document

Share or Embed Document

Sharing Options

Did you find this document useful?

Is this content inappropriate?

Copyright:

Available Formats

Process Improvement and Risk Management in Off-The-Shelf Component-Based Development

Uploaded by

Copyright:

Available Formats

Process Improvement and Risk

Department of Computer and Information Science

ISBN printed 82-471-7920-2

Printed in Norway by NTNU Trykk, Trondheim

Contents .................................................................................................................... vii

List of Tables .............................................................................................................. x

2 Software Reuse and Component-Based Development .................................. 13

3 Process Improvement in OTS Component-Based Development.................. 25

4 Risk Management in OTS Component-Based Development........................ 31

7 Evaluation and Discussion ............................................................................... 75

8 Conclusions and Directions for Future Work................................................ 91

Appendix A: Selected papers ................................................................................ 107

Appendix B: Other Papers.................................................................................... 259

Appendix C: Interview Guide for the Pre-study................................................. 265

Appendix D: Questionnaire in the Main Study................................................... 279

1.1 Problem Outline

1.2 Research Questions

RQ1: What is the status of component-based development based on components

Results of the preliminary study gave valuable insights on designing research

1.3 Research Design

Figure 1-1 Research questions and corresponding studies

Empirical studies may be performed quantitatively, qualitatively or in combination.

Figure 1-2 Contributions C1 to C5

Contribution C6 is a secondary contribution, which comes from the lessons learned

Detailed descriptions of each contribution:

C1: Increased understanding on risks of reusing in-house built components

In a preliminary quantitative study, we have investigated challenges related to four key

C2: Summarizing the state-of-the-practice of development and selection processes

The contributions to the OTS component-based development process focus on two

A qualitative pre-study with semi-structured interviews was first performed to

- The use of COTS components can be done as part of traditional development

By summarizing the state-of-the-practice of OTS component-based development

C3: Empirical validation of risk management proposals in OTS component-based

system deployment, it is necessary to understand the components well and to do

C4: Increased understanding of the decision-making practice in OTS component-

- Why was it decided to use OTS components?

C6: Improved understanding on performing an international survey in the ICT

- A pre-study often clarifies both research questions and concrete questions in a

1.5 Selected Papers

Table 1-1 Relations between research questions, contributions, and papers

Product Focused Software Process Improvement (PROFES'2004), Apr. 2004, Nara

1.6 Thesis Structure

Chapter 2: Software reuse and component-based development. This chapter

development. First, we discuss the motivation of systematic reuse in software

Chapter 3: Software process improvement in OTS component-based development.

Chapter 4: Risk management in OTS component-based development. This chapter

Chapter 6: Empirical investigations and main results/findings. This chapter explains

Appendix C includes the interview guide for the pre-study.

Appendix D contains the questionnaire used in the main study.

2 Software Reuse and Component-Based

2.1 Software Engineering Definitions and Challenges

These characteristics have even become more extreme due to developments in

2.2 Software Reuse

- Architecture reuse, which consists of standardizing a set of design and

2.3 Component-Based Development

Bachman et al. [Bachman00] list the advantages of CBD as:

Advantages added by Bass et al. [Bass00] are:

Component-based system development is often promoted in the literature as a low

attributes will determine the corresponding attributes of systems of which they

Components can be built in-house, acquired from the third-companies, or just

2.4 COTS-Based Development

− Is not produced in or exclusively for the project.

COTS software can be classified into different categories. Morisio et al.