You are on page 1of 191

Understanding Risks in Off-The-Shelf-Based Custom

Software Projects

Dana Sulistiyo Kusumo

A thesis submitted in fulfilment of the requirements for the degree of


Doctor of Philosophy

School of Computer Science and Engineering


The University of New South Wales
Sydney, Australia

April 2013
PLEASE TYPE
THE UNIVERSITY OF NEW SOUTH WALES
Thesis/Dissertation Sheet

Surname or Family name: Kusumo

First name: Dana Sulistiyo Other name/s:

Abbreviation for degree as given in the University


calendar: PhD

School: Computer Science and Engineering Faculty: Engineering

Title:
Understanding Risks in Off-The-Shelf-Based Custom
Software Projects

Abstract 350 words maximum: (PLEASE TYPE)

Different software development project stakeholders (developers and acquirers) can have different perceptions of risks and how
they should be mitigated. But these differences are not well understood or managed. This general issue occurs in off-the-shelf
(OTS)-based custom software projects, which use and integrate OTS software in developing specialized software for an individual
customer.

This research provides a better understanding of risks to developers and acquirers in OTS-based custom software projects. This
research consisted of five phases:
1. A systematic mapping study to identify OTS-based software acquisition processes.
2. A further mapping study to identify and classify risks related to OTS-based custom development and acquisition.
3. A survey of developers and acquirers in Indonesia to investigate in practice the characteristics of OTS-based risks that
are shared by developers and acquirers.
4. Analysis of the survey data using a risk assessment framework based on stakeholder analysis. This focused on
identifying differences in risk control, impact and responsibility between both stakeholders.
5. A multi-case study (four cases) to provide a deeper and more contextual analysis of these differences.

Through these investigations we were able to:


1. Identify six OTS-specific acquisition processes not previously recognized.
2. Identify and classify risks of OTS-based custom projects into seventeen categories.
3. Use the survey to investigate eleven risks of the seventeen relevant risk categories. This revealed that:
(a) A greater number of risks occurred more frequently in acquisition rather than in development.
(b) In general the stakeholders agreed on who can best control the risks (developer).
(c) There were different perceptions on who is most impacted by risks.
(d) Developers were considered to bear most responsibility for risk.
4. The case study identified seventeen detailed findings related to risk control and impact for the eleven risks studied.

In general most acquisition risks derive from the same concerns as development risks. However technically related risks are found
less often in acquisition and project management related risks less often in development.

Declaration relating to disposition of project thesis/dissertation

I hereby grant to the University of New South Wales or its agents the right to archive and to make available my thesis or
dissertation in whole or in part in the University libraries in all forms of media, now or here after known, subject to the provisions
of the Copyright Act 1968. I retain all property rights, such as patent rights. I also retain the right to use in future works (such as
articles or books) all or part of this thesis or dissertation.

I also authorise University Microfilms to use the 350 word abstract of my thesis in Dissertation Abstracts International (this is
applicable to doctoral theses only).

……………………………………..……………… …….………………
Signature Witness Date

The University recognises that there may be exceptional circumstances requiring restrictions on copying or conditions on use.
Requests for restriction for a period of up to 2 years must be made in writing. Requests for a longer period of restriction may be
considered in exceptional circumstances and require the approval of the Dean of Graduate Research.

FOR OFFICE USE ONLY Date of completion of requirements for Award:

THIS SHEET IS TO BE GLUED TO THE INSIDE FRONT COVER OF THE THESIS

ii
Declaration of Originality

‘I hereby declare that this submission is my own work and to the best of my knowledge
it contains no materials previously published or written by another person, or substantial
proportions of material which have been accepted for the award of any other degree or
diploma at UNSW or any other educational institution, except where due
acknowledgement is made in the thesis. Any contribution made to the research by
others, with whom I have worked at UNSW or elsewhere, is explicitly acknowledged in
the thesis. I also declare that the intellectual content of this thesis is the product of my
own work, except to the extent that assistance from others in the project's design and
conception or in style, presentation and linguistic expression is acknowledged.’

iii
List of Publications

The following papers were written and published during the candidature. In each case,
the candidate’s contribution was to design and also conduct the study, perform the
analysis, and draft the original manuscript. Co-authoring supervisors and a NICTA
researcher contributed guidance on content and editorial changes on content, linguistic
expression and presentation.

D. S. Kusumo, L. Zhu, M. Staples, and H. Zhang, “A Systematic Mapping Study on


Off-The-Shelf-based Software Acquisition,” in ACIS 2011 Proceedings, Sydney, 2011.
(Chapter 3 is based on this paper)

D. S. Kusumo, M. Staples, L. Zhu, H. Zhang, and R. Jeffery, “Risks of Off-The-Shelf-


based Software Acquisition and Development: A Systematic Mapping Study and A
Survey,” in 16th International Conference on Evaluation & Assessment in Software
Engineering, Ciudad Real, 2012. (Chapter 4 is based on this paper)

D. S. Kusumo, M. Staples, L. Zhu, and R. Jeffery, “Analyzing Differences in Risk


Perceptions between Developers and Acquirers in OTS-based Custom Software Projects
Using Stakeholder Analysis,” in 6th International Symposium on Empirical Software
Engineering and Measurement (ESEM 2012), Lund, 2012. (Chapter 5 is based on this
paper)

iv
Abstract

Different software development project stakeholders (developers and acquirers) can


have different perceptions of risks and how they should be mitigated. But these
differences are not well understood or managed. This general issue occurs in off-the-
shelf (OTS)-based custom software projects, which use and integrate OTS software in
developing specialized software for an individual customer.

This research provides a better understanding of risks to developers and acquirers in


OTS-based custom software projects. This research consisted of five phases:
1. A systematic mapping study to identify OTS-based software acquisition
processes.
2. A further mapping study to identify and classify risks related to OTS-based
custom development and acquisition.
3. A survey of developers and acquirers in Indonesia to investigate in practice the
characteristics of OTS-based risks that are shared by developers and acquirers.
4. Analysis of the survey data using a risk assessment framework based on
stakeholder analysis. This focused on identifying differences in risk control,
impact and responsibility between both stakeholders.
5. A multi-case study (four cases) to provide a deeper and more contextual analysis
of these differences.

Through these investigations we were able to:


1. Identify six OTS-specific acquisition processes not previously recognized.
2. Identify and classify risks of OTS-based custom projects into seventeen
categories.
3. Use the survey to investigate eleven risks of the seventeen relevant risk
categories. This revealed that:
(a) A greater number of risks occurred more frequently in acquisition rather
than in development
(b) In general the stakeholders agreed on who can best control the risks
(developer).
(c) There were different perceptions on who is most impacted by risks.
(d) Developers were considered to bear most responsibility for risk.
v
2. The case study identified seventeen detailed findings related to risk control and
impact for the eleven risks studied.

In general most acquisition risks derive from the same concerns as development risks.
However technically related risks are found less often in acquisition and project
management related risks less often in development.

vi
Acknowledgements

I would like to thank the following people for their support, without whose help this
thesis would never have been possible. Before all, I want to thank Almighty God for His
love and guidance through this journey.

I owe my deepest gratitude to my supervisors Dr. Liming Zhu, Dr. Mark Staples and
Professor Ross Jeffery for their continuous support, and valuable suggestions and
discussions. I also thank He Zhang for his contributions to ACIS and EASE papers.
Similar thanks goes to my colleagues at the Software Systems Research Group of
NICTA, Paul R., Shukor, Qinghua Lu, Sherry, Liang Zhao, Emam, Yin Kia, Bassem
and Suronapee, for their fellowship and support.

I am particularly grateful to those survey respondents and case study participants who
made this research possible.

Support was given by Higher Directorate General of Higher Education - Ministry of


Education and Culture of the Republic of Indonesia, UNSW, NICTA and IT Telkom.

Finally, I am indebted to my beloved wife Palupi, and my wonderful daughters Kanaya


and Kiara, for their continuous support, understanding and love that encouraged me to
complete my study.

vii
Contents

List of Figures .................................................................................................................. xi


List of Tables .................................................................................................................. xii
1 Introduction ........................................................................................................... 1
1.1 Background ........................................................................................................... 1
1.2 Research Problem ................................................................................................. 2
1.3 Research Questions ............................................................................................... 3
1.4 Research Methodology ......................................................................................... 4
1.5 Research Limitations............................................................................................. 7
1.6 Research Contributions ......................................................................................... 7
1.7 Thesis Outline ....................................................................................................... 9
2 Literature review ................................................................................................. 11
2.1 Off-The-Shelf-Based Custom Software Project Processes ................................. 11
2.2 Risk Management in OTS-Based Custom Software Projects ............................. 15
2.3 Stakeholder Analysis........................................................................................... 18
2.4 Research Methods in Software Engineering Research ....................................... 21
2.5 Summary ............................................................................................................. 22
3 Off-The-Shelf-based Software Acquisition Processes ........................................ 24
3.1 Introduction ......................................................................................................... 24
3.2 Method ................................................................................................................ 25
3.3 Definition of Research Questions ....................................................................... 25
3.4 Conduct Search for Primary Studies ................................................................... 25
3.5 Screening Papers for Inclusion and Exclusion (Relevant Papers) ...................... 26
3.6 Data Extraction and Mapping of Study............................................................... 27
3.7 Discussion ........................................................................................................... 31
3.8 Threats to Validity .............................................................................................. 33
3.9 Summary ............................................................................................................. 34
4 Risks of OTS-based and Custom Software Development and Acquisition ........ 35
4.1 Introduction ......................................................................................................... 35
4.2 Research Design .................................................................................................. 36
4.3 Systematic Mapping Study on Risks of Off-The-Shelf-Based Custom Software
Development and Acquisition ..................................................................................... 37

viii
4.4 Survey on Risks of Off-The-Shelf-Based Software Development and
Acquisition .................................................................................................................. 48
4.5 Discussion ........................................................................................................... 53
4.6 Threats to Validity .............................................................................................. 56
4.7 Summary ............................................................................................................. 57
5 Differences in Risk Perceptions .......................................................................... 60
5.1 Introduction ......................................................................................................... 60
5.2 Research Design .................................................................................................. 61
5.3 Collected Data ..................................................................................................... 67
5.4 Results ................................................................................................................. 67
5.5 Discussion ........................................................................................................... 72
5.6 Threats to Validity .............................................................................................. 73
5.7 Summary ............................................................................................................. 74
6 Comparison of Process-related Risk Perceptions between Developers and
Acquirers ......................................................................................................................... 76
6.1 Introduction ......................................................................................................... 76
6.2 Proposed Framework .......................................................................................... 77
6.3 Case Study Design .............................................................................................. 80
6.4 Result .................................................................................................................. 93
6.5 Discussion ......................................................................................................... 109
6.6 Threats to Validity ............................................................................................ 127
6.7 Summary ........................................................................................................... 128
7 Discussion and Conclusion ............................................................................... 132
7.1 Research Contributions ..................................................................................... 133
7.2 Comparison with Related Work ........................................................................ 139
7.3 Implications for Research ................................................................................. 144
7.4 Implications for Practice ................................................................................... 145
7.5 Limitations ........................................................................................................ 145
Bibliography.................................................................................................................. 147
Appendix A: Glossary................................................................................................... 157
Appendix B: Mapped papers for OTS-based software acquisition mapping study ...... 158
Appendix C: Mapped papers for risks of OTS-based and custom software development
and acquisition .............................................................................................................. 165
Appendix D: The framework inputs ............................................................................. 173
ix
Appendix E: Cross-case analysis for data collection 1 and 2, excluded inputs to the
framework and evaluation criteria for risk assessment ................................................. 175

x
List of Figures

Figure 1.1 OTS-based custom software developer and acquirer relationship .................. 1
Figure 1.2 Overview of the research methodology ........................................................... 6
Figure 2.1 Stakeholder power/interest matrix with stakeholder involvement ................ 19
Figure 5.1 A method for analysing differences in off-the-shelf risks between the
developer and acquirer used in the survey ....................................................................... 63
Figure 5.2 Relationship model of respondent, risk, risk control and impact to stakeholder
......................................................................................................................................... 63
Figure 5.3 Risk control/impact matrix, adapted from power/interest matrix [79][33][80]
......................................................................................................................................... 65
Figure 5.4 The developer respondents’ responses mapped to step 1 and 2 of the template
(35 respondents) .............................................................................................................. 68
Figure 5.5 The acquirer respondents’ responses mapped to step 1 and 2 of the template
(34 respondents) .............................................................................................................. 69
Figure 6.1 The proposed framework ............................................................................... 78
Figure 6.2 Risk control/impact matrix with stakeholder involvement ............................ 79

xi
List of Tables

Table 1.1 A summary of five steps of research methodology........................................... 5


Table 2.1 Software acquisition and OTS-based software acquisition process models
found in the literature ...................................................................................................... 14
Table 2.2 Risk checklists in the literature extended from Keil et al. [66]....................... 16
Table 3.1 Search results using the search string ............................................................. 26
Table 3.2 Refined results after paper screening .............................................................. 27
Table 3.3 Frequency of identified processes according to software acquisition
classification .................................................................................................................... 29
Table 3.4 Research paper classification summarised from the paper classification by
Wieringa and Heerkens [95] ........................................................................................... 30
Table 4.1 Search strings used in this study .................................................................... 38
Table 4.2 Inclusion and exclusion criteria used in this study ......................................... 39
Table 4.3 Data extraction form description..................................................................... 40
Table 4.4 Search and selection results ............................................................................ 41
Table 4.5 Mapping study classification based on type of paper ..................................... 43
Table 4.6 OTS software development risks mapped in this study .................................. 44
Table 4.7 OTS software acquisition risks mapped in this study ..................................... 46
Table 4.8 Custom software development risks mapped in this study ............................. 47
Table 4.9 Custom software acquisition risks mapped in this study ................................ 48
Table 4.10 OTS-specific project risks relevant to the developer and acquirer ............... 50
Table 4.11 Risk occurrences of OTS-based software development and acquisition ...... 52
Table 5.1 11 OTS-specific project risks relevant to the developer and acquirer
respondents ...................................................................................................................... 61
Table 5.2 A template to compare risk perceptions of OTS-based custom software
project .............................................................................................................................. 64
Table 5.3 Analysing of different risk perceptions ............................................................ 70
Table 5.4 Summary of comparison of risks mapped into the risk control/impact matrix
between the developer and acquirer respondents (summarized from Table 5.3) .............. 70
Table 6.1 Differences between the case study framework and the method used in
Chapter 5 ......................................................................................................................... 77

xii
Table 6.2 Mapping between case study questions, research questions and related
propositions ..................................................................................................................... 84
Table 6.3 Case study participants.................................................................................... 86
Table 6.4 Summary of case study participants’ project and participant information .... 90
Table 6.5 Framework outputs (stakeholder involvement and risk priorities) and
agreement of risk responsibilities.................................................................................... 95
Table 6.6. Comparison of evaluation criteria for risk assessment for four participants 105
Table 6.7 Example of cross-case analysis for data collection 1 and 2, excluding
framework inputs and evaluation criteria for risk assessment ...................................... 107
Table 6.8 Classification of key results regarding risk control and impact .................... 113
Table 6.9 Summary of case study evidence and related propositions/literature ........... 125

xiii
1 Introduction

1.1 Background
Software development projects increasingly use off-the-shelf (OTS) products,
integrating them into the systems under development. Software-developing
organisations can avoid building every part of their product software ‘from scratch’ by
reusing technologies available from third parties [1]. Acquiring ‘OTS-based software’,
i.e. software that itself uses OTS software platforms or components, can be less
expensive than acquiring fully custom-developed software. Custom software
development is either performed in-house or contracted software development with
specific requirements for an individual customer [2][3]. OTS-based custom software is
developed using and integrating OTS software in developing specialised software for an
individual customer [4].

This research focuses on OTS-based custom software that is developed by being


contracted or outsourced to an external software developing organisation. OTS products
are described in this research as “a commercially available or open source piece of
software that other software projects can reuse and integrate into their own products”
[5:91]. We also classify open source software (OSS) as OTS. For the purpose of this
research, an OTS-based custom software project consists of OTS-based software
development and acquisition processes. OTS-based custom software project processes
are detailed in Chapter 2. The relationship between OTS-based custom software
acquirer and developer is depicted in Figure 1.1.

acquire
OTS
Developer Acquirer
software
develop

Figure 1.1 OTS-based custom software developer and acquirer relationship

Throughout this thesis, we use software acquisition term definitions from , the IEEE
Standard 1062-1998 Edition: Recommended Practice for Software Acquisition [6] to
define customer/software acquiring organization and software development
1
organisation. An acquirer is defined to be “A person or organisation that acquires or
procures a system or software product (which may be part of a system) from a supplier”
[6:3] and a developer is defined to be “A person or organisation that performs
development activities (including requirements analysis, design, testing through
acceptance) during the software life cycle process” [6:4].

1.2 Research Problem


In a software project, risks affect all stakeholders [7][8][9][10]. A risk is defined as “the
effect of uncertainty on objectives” [11:1]. Stakeholders are defined as anyone who are
affected by or can influence the systems under development [10][12][13][14]. Software
project risks arise from the beginning of the software acquisition process on the
software acquirer’s side [8][7]. Most literature focuses on the risks from the software
developer organisation’s perspective and little attention has been given to software
acquirers [7][15]. However, to manage risks effectively, it is important for all software
project stakeholders to understand the risks involved [16]. In addition, there is a need to
better understand OTS specific risks related to technical development processes [17] as
a consequence of using OTS products. In this thesis, risks are investigated for both
acquirers and developers of OTS-based custom software projects.

There are differences in risk perceptions between software developers and acquirers,
since they are from different organisations and have different backgrounds, experiences,
needs and expectations [18]. These differences in perception could also occur because
of differences between each party’s goals and structures [19]. Perspective mismatch
could lead to projects not meeting their expected value, quality and schedule [20].
Differences in risk perspectives could increase areas of conflict and obstruct a shared
understanding of risks [16].

Most existing studies and guidance on software acquisition do not explicitly deal with
the acquisition of OTS-based software. For example, the IEEE Standard 1062-1998
Edition: Recommended Practice for Software Acquisition [6] can be applied to software
acquisition process regardless of the size and complexity of the software [6]. However,
this recommended practice is more applicable for fully developed software and must be
tailored to other types of software acquisition [6]. In this thesis we first investigate the
processes of OTS-based software acquisition before investigating the differences in

2
process-related risk perceptions. Process-related risks are defined as “risks that are
related to the overall process, process elements and their interaction” [21:14].

Different stakeholders perceive risk differently [16]; therefore, there are different
perceptions of stakeholder’s risk responsibility. To manage risks effectively, it is
important to identify risks [16], to involve stakeholders [22][8] aiming to take account
of differences in risk perceptions and to identify stakeholders’ risk responsibility [22].
Even though Keil et al. [16] noted the importance of reconciling user and project
manager perceptions of IT project risks and compared the two stakeholders’
perceptions, they did not provide detailed steps for reconciling risks. A recent study has
highlighted the importance of reconciling perspective mismatch in managing processes
of software projects [20]. In this study, Adolph et al. [20] introduced two stages of
reconciling perspectives: converging and validating. However, they [20] offered no
scheme for achieving perspective agreement. Therefore, it is necessary to reconcile
stakeholder perceptions for managing processes and risks in software projects [16][20].
To manage OTS-based custom software projects successfully, the overall goal of this
thesis is to understand how developers and acquirers manage differences in process-
related risk perceptions in OTS-based custom software projects.

To obtain a better understanding of differences in risk perceptions between


stakeholders, one common solution is to compare and discuss risk perceptions [22][23].
This approach was used by Svahnberg and Wohlin [24] to compare and discuss quality
attributes and structures of software architecture between different software engineers.
One approach that takes account of different stakeholders in a project is stakeholder
analysis [25][12][26][14][13][10]. Therefore, this research has compared the results of
stakeholder analysis for developers and acquirers in OTS-based custom software projects.

1.3 Research Questions


The goal of this research is to investigate differences in process-related risks of OTS-
based custom software projects from software acquirers’ and developers’ perspectives.
The motivation is to give guidelines to analyse differences in perception of these risks.
The thesis has three main research questions.

To understand differences in process-related risk perspectives between developers and


acquirers in OTS-based custom software projects, OTS-based software acquisition

3
processes were first identified and better understood. Therefore, the research question
RQ1 is defined as follows.

RQ1: What are the processes of OTS-based software acquisition?

It is important to identify and classify acquisition and development risks in OTS-based


custom software projects before investigating differences in perceptions of these risks.
We have investigated the occurrences of 11 OTS process-related risks that are relevant
to both developers and acquirers. The research question is as follows.

RQ2: What are the risks in OTS-based custom software projects?

The answers to RQ2 were expected to give us an initial understanding of differences in


risk perception between stakeholders. We then investigated differences in the
perceptions of these risks using stakeholder analysis [25][12][26][14][13][10]. This
thesis then empirically investigated perceptions of process-related risks between
developers and acquirers in OTS-based custom software projects. Therefore, research
question RQ3 is defined as:

RQ3: How should software developers and acquirers analyse differences in


perceptions of process-related risks in OTS-based custom software projects?

1.4 Research Methodology


The research methodology is described in five steps below, and shown in Figure 1.2.
The mapping among the steps, research questions and related chapters is given in Table
1.1.

Step 1: Planning the research


First we identified problematic areas and the importance of the research questions. This
included reviewing previous research in related areas, and identifying gaps. We then
formulated research problems and research objectives.

Step 2: Investigating processes in OTS-based software acquisition


A systematic mapping study of evidence-based software engineering was used to
identify and classify OTS-based software acquisition processes.

4
Step 3: Investigating risks in OTS-based custom software projects
Then we investigated risks in OTS-based custom software projects. This began by
identifying, classifying and comparing risks of OTS-based software development and
acquisition using a second mapping study. Then a survey was performed to empirically
investigate the occurrences of the 11 OTS-specific risks in practice. The survey partially
replicated a study by Li et al. [27] on risk management for OTS-based software
development.

Step 4: Investigating differences in process-related risk perceptions between


developers and acquirers using a risk framework
Next we compared perceptions of the 11 risks. A framework was developed to analyse
the survey results about differences between these perceived risks (see Chapter 5). The
framework extends a prior stakeholder analysis framework [25][12][26][14][13][10].

Step 5: Comparing differences in process-related risk perceptions between


developers and acquirers
In the case study (Chapter 6), the framework was modified to compare, analyse and
discuss levels of risk control and impact between developers and acquirers.

As shown in Table 1.1, this research consists of three research methods: systematic
mapping study, survey and case study. There are two different systematic mapping
studies, detailed in Chapter 3 and 4. Further, the survey results were then used in two
studies (Chapter 4 and 5). Two research methods, the survey and the multi-case study
were used to answer RQ2 and RQ3.

Table 1.1 A summary of five steps of research methodology

Step Activity Method (study) Research Chapter


Question
1 Search the literature Literature Review RQ1 to 1-7
Identify problematic areas and Literature Review RQ3
research importance
Formulate research objectives and Literature Review
research questions
2 Identify and classify OTS-based Systematic RQ1 3
software acquisition Mapping Study
3 Identify, classify and compare risks Systematic RQ2 4
of OTS-based software development Mapping Study
and acquisition
5
Compare the occurrences of risks Survey RQ2 4
relevant to developers and acquirers
4 Investigate differences in process- Survey RQ3 5
related risk perceptions between
developers and acquirers using a risk
framework
5 Compare differences in process- Multi-Case Study RQ3 6
related risk perceptions between
developers and acquirers

Step 1: Planning the research

Research gap

Research problems 
Research importance 
Research objectives

Step 2: Investigating processes in OTS-based software


acquisition

Literature
review Step 3: Investigating risks in OTS-based custom
software projects

Step 4: Investigating differences in process-related


risk perceptions between developers and acquirers
using a risk framework

Step 5: Comparing differences in process-related risk


perceptions between developers and acquirers

Figure 1.2 Overview of the research methodology

6
1.5 Research Limitations
To answer RQ2 and RQ3, this research investigated differences in the perception of 11
process-related risks in OTS-based custom software projects relevant to developers and
acquirers (described in Table 4.10 and in Section 4.4.1). Although focusing on these 11
risks could limit the study’s generalisability, we believe that it can increase our
understanding of how to assess risk by analysing differences in risk perceptions between
developers and acquirers. The context of OTS-based software is not narrow - it is now
very common in software development projects1 due to its advantages. These include
rich functionality, dedicated support from OTS software vendors, and less expensive
development and maintenance [28].

This research investigated risk control and impact for developers and acquirers in OTS-
based custom software projects. However, it did not investigate other risk attributes
(such as the probability of loss and the brief description of risks [29]). This research
also only focused on risk assessment, not risk-management planning, risk resolution, or
risk monitoring [30].

In our survey, we asked the survey respondents which stakeholders were affected by
risks and who could influence risks. Even though we did not investigate actual risks from
developers’ and acquirers’ perspectives (without discussing their responses to their
stakeholders), a study of risk perception is an important approach to understand
differences of actual risks between developers and acquirers. The case study investigated
actual risk perceptions.

1.6 Research Contributions


In relation to the research questions, we have made the following contributions.

RQ1: What are the processes of OTS-based software acquisition?

We identified six OTS-based software acquisition processes from the first mapping
study: decision-making to make or buy software, architectural decision-making, OTS
selection, managing relationships between developers and acquirers, managing
relationships between OTS adoption and acquirers’ organisations, and managing
relationships between OTS vendors and developers. These processes should be

http://www.gartner.com/it/page.jsp?id=2131115 describes the prediction of using OTS software (such as


1

cloud, mobility and open source component) in 2012-2015


7
considered by an acquirer when acquiring OTS-based software, in addition to the
process in the existing software acquisition process standard [6].

RQ2: What are the risks in OTS-based custom software projects?

The second mapping study provides a list of 17 risk categories of OTS-based and
custom-based software, not only from developers’ perspective but also from acquirers’
perspective: planning, requirements, design, integration and testing, system lifecycle,
maintenance, project closure, software, OTS products, cost, environment, people,
systems engineering, vendor relationships, project management, contract and legal.

RQ3: How should software developers and acquirers analyse differences in


perceptions of process-related risks in OTS-based custom software projects?

For RQ2 and RQ3 contributions, the survey and case study show that comparing and
discussing risk perceptions about risk control and impact of 11 process-related risks
relevant to developers and acquirers enhance our understanding of risks in OTS-based
custom software projects (described in Table 4.10 and in Section 4.4.1). Our survey
found that: 1) there are similar and different perceptions about risk occurrences in OTS-
based custom software projects between the developer and acquirer perspectives (RQ2
contribution); and 2) there are similarities and differences in respondents’ (developers
and acquirers) perceptions about which stakeholder is affected by and can control risks
(RQ3 contribution). For RQ3 contribution, the case study resulted in 17 detailed
findings related to levels of risk control and impact, and proposed risk priorities (are
detailed in Chapter 6, and summarised in Chapter 7).

This thesis has proposed a framework to analyse differences in risk perceptions and to
assign risk responsibilities between a developer and acquirer (RQ3 contribution). The
framework extends a stakeholder analysis framework [25][12][26][14][13][10] and can
be used to compare, analyse and discuss level of risk control and impact between a
developer and acquirer. The case study results show that all participants and their
stakeholders agreed on proposed risk responsibilities and risk priorities. Furthermore,
our case study finds that a responsibility should also be assigned to the stakeholder who
has the competence and capacity needed to discharge the responsibility [34][35].

8
From a methodological perspective, another contribution of this research is to
demonstrate that stakeholder analysis can be used for each stakeholder to compare,
analyse and discuss their risk perceptions.

1.7 Thesis Outline


This thesis consists of seven chapters. This chapter introduces the background, research
problem, research questions, research methodology and research contributions. The
remaining chapters are as follows.

Chapter 2 describes a literature review of processes for OTS-based custom software


projects, risk management of OTS-based custom software projects, stakeholder analysis
and research methods in software engineering research.

Chapter 3 describes a systematic mapping study on OTS-based software acquisition.

Chapter 4 describes a systematic mapping study and a survey on risks of OTS-based


custom software projects.

Chapter 5 reports a survey on risk perceptions of developers and acquirers in OTS-


based custom software projects.

Chapter 6 discusses a multi-case study to analyse differences in the perception of


process-related risks between developers and acquirers in OTS-based custom software
projects.

Chapter 7 concludes this thesis by listing research contributions, comparing with


related work and discussing future work.

Appendix A lists key terminologies used in this thesis.

Appendix B details mapped papers for software acquisition and OTS-based software
acquisition processes based on type of paper (empirical, experience, conceptual
framework and standard).

Appendix C details mapped papers for risk categories of OTS-based and custom
software development and acquisition based on type of paper (empirical, experience and
conceptual framework).

9
Appendix D presents the framework inputs from the case study participants and their
stakeholder (Chapter 6)

Appendix E presents a cross-case analysis for data collection 1 and 2, excluded inputs
to the framework and evaluation criteria for risk assessment, from the case study
participants (Chapter 6).

10
2 Literature review

This chapter introduces the literature on processes for OTS-based custom software
projects. Then it reviews literature on risk management of OTS-based custom software
projects, stakeholder analysis and research methods in software engineering research.

2.1 Off-The-Shelf-Based Custom Software Project Processes


Off-the-shelf (OTS)-based custom software project processes consist of OTS-based
software development and acquisition processes.

2.1.1 OTS-Based Software Development


OTS-based software development refers to software development that integrates OTS
software to develop a system. The use of off-the-shelf products in software development
changes the traditional development approach [39]. While a variety of definitions of the
term off-the-shelf (OTS) have been suggested, this study uses the definition by
Torchiano and Morisio [5] because of its broad coverage. Off-the-shelf (OTS) is defined
as “a commercially available or open source piece of software that other software
projects can reuse and integrate into their own products” [5:91]. In the traditional
approach, software development starts with a system requirements definition, then
defines a system architecture, and concludes with implementation. In OTS-based system
development, there is simultaneous definition and tradeoff among OTS selection, user
requirements and design [40]. The use of OTS products in systems development
modifies not only software development [39][40][41] but also software acquisition
[42][43][44].

There are additional OTS-specific processes and roles that must be included in custom
software development [39][40][41]. These processes are: make vs. buy decisions, OTS
selection, OTS product familiarization, learning and understanding OTS products, OTS
integration and vendor interaction [39][40][41]. Li et al. [39] point out that OTS-based
software development can be a part of traditional and custom-development process
lifecycles (e.g. waterfall or evolutionary), combined with some new activities to manage
risks. There are also new roles found in OTS-based software development literature:
OTS team, a team member responsible for interactions with an OTS vendor [40] and
OTS software knowledge keeper [39].

11
The main stages of OTS-based software development are: requirements, design, coding
and integration [40]. For each stage there are OTS-specific and non-OTS activities.
Each stage has activities that are concurrent or iterative not only within an activity but
also across activities.

i. Requirements: Requirements analysis and OTS selection are performed together.


Added activities are listed in a logical order: make vs. buy decisions, requirements
definition and analysis, OTS identification and selection, OTS familiarization,
feasibility study, definition of high level architecture.

ii. Design: Activities of design include following aspects: decision about


architecture, OTS integration and glue code, and non-OTS design.

iii. Coding: Coding deals with writing glue code and interface, and non-OTS coding.

iv. Integration: Integration consists of integration and test, and target system
installation and acceptance test.

Coding and integration are implementations of important decisions made in the


requirements and design phases [40]. In the Constructive COTS-based application cost
model (COCOTS), one of the suites of COCOMO models, there are two COCOTS sub
models related to the OTS integration process [45][46]. The first model concerns OTS
tailoring, which configures OTS software for specific use, for example, setting and
initialising parameters [45][46]. The next model concerns OTS glue code development,
which develops code to integrate OTS software into larger systems [45][46].

For OTS-based custom software, development is performed to cover remaining


requirements when OTS products cannot by themselves meet the requirements [46][40].
Based on his experience in integrating OTS software into systems at The Boeing
Company, Baker [47] describes four levels of coupling between the OTS software and
the systems: configuration (for example, setting initialization files); integration –
developing external code with minimal internal changes of OTS software; customization
– vendor-supported enhancements; and development – non vendor-supported
enhancement. These levels of coupling could be considered as the levels of
customization in OTS-based software development. A OTS characterization framework
[48] described how the degree of OTS customization can be divided into three kinds.
The first is required modification [49], which has five possible levels: minimal,

12
parameterization, customization, internal revision, and extensive rework. The second
kind is possible modification, which includes five possible levels: none or minimal,
parameterization, customization, programming, and source code. The last kind of
customisation is interface, and which also has five possible values: none,
documentation, API, OO interface, and contract with protocol.

2.1.2 OTS-Based Software Acquisition


One of the challenges in software acquisition when acquiring OTS-based systems is a
simultaneous process for OTS selection and system requirements [41][40]. An OTS-
based custom software project uses OTS software that may not fit all defined system
requirements. Therefore not only must system requirements be defined, but also the
system architecture must be simultaneously based on the OTS selection. However, no
comprehensive OTS-based software acquisition processes exists or is standardised.

There exists a generic practice of software acquisition, IEEE Standard 1062-1998


Edition: Recommended Practice for Software Acquisition [6]. This recommended
practice can be applied to software acquisition process regardless of the size and
complexity of the software [6]. However, this recommended practice is more applicable
for fully developed software and must be tailored to other types of software acquisition
[6].

In Table 2.1, we summarize several software acquisition and OTS-based software


acquisition process models found in the literature. There are four software acquisition
process models: IEEE Standard 1062-1998 Edition: Recommended Practice for
Software Acquisition [6], ISO/IEC/IEEE 12207:2008 Standard for Systems and
Software Engineering - Software Life Cycle Processes [50], GARP (Generic Acquisition
Reference Process) [51][52], and MPS.BR Model-based software acquisition [53][54].
Three OTS-based software acquisition process models are found in the literature:
Commercial Off-The-Shelf (COTS) Acquisition Process (CAP) [55], COTS Software
Acquisition Meta-Model (SAMM) [42], and COTS Software Component Acquisition
process framework (CSCA) [43]. From Table 2.1, it can be seen that the only OTS-
specific process found in existing OTS-based software acquisition process models is the
OTS selection process. This has motivated us to identify OTS-based software
acquisition processes using a systematic mapping study.

13
Table 2.1 Software acquisition and OTS-based software acquisition process models found in the literature

Model/framework Software acquisition processes Generic software OTS-based OTS specific processes
acquisition software
acquisition
IEEE 1062 Recommended Planning organisational strategy, √ - -
Practice for Software implementing organisation’s
Acquisition [6] process, determining the software
requirements, identifying potential
suppliers, preparing contract
requirements, evaluating proposals
and selecting the supplier, managing
supplier performance, accepting the
software and using the software
ISO/IEC/IEEE 12207 [50] Acquisition preparation, acquisition √ Must be adjusted -
advertisement, supplier selection,
contract agreement, agreement
monitoring, acquirer acceptance,
closure
GARP [51][52] Refer to IEEE 1062 [6] √ - -
MPS.BR Software Acquisition refers to IEEE 1062 [6] and √ - -
[53][54] ISO/IEC/IEEE 12207 [50]
CAP [55] Initialization Component, Execution - √ OTS selection and
Component and Reuse Component evaluation as the basis for
a make-or-buy decision
SAMM [42] Choice phase and implementation - √ Purchase/iterate approach
phase to select OTS products
CSCA [43] Planning, analysing and evaluating, - √ Use make vs. buy
negotiating, managing and reusing decisions implicitly to
select OTS products
14
2.2 Risk Management in OTS-Based Custom Software Projects
This thesis uses the definition of risk proposed by AS/NZS ISO 31000:2009 as “the
effect of uncertainty on objectives” [11:1]. In software projects, we can classify project
risks into generic and project-specific risks [56]. The first are common to all projects
[57][30][58][59] and the latter depend on specific aspects of the project. From the
perspective of technical development processes [17], risks specific to the use of OTS
products may arise from OTS-based custom software project processes (as mentioned in
the previous section) in OTS-based software development [39][40][41] and acquisition
[60]. One of the reasons that make the processes different is a simultaneous process for
OTS selection, system requirements and system design [40][41].

Boehm provides a foundation of software risk management consisting of two major


steps: risk assessment and risk control [30]. Risk assessment involves three activities:
risk identification, risk analysis and risk prioritization, and risk control involves three
activities: risk-management planning, risk resolution and risk monitoring.

As described in Chapter 1, this study focuses on risk assessment in OTS-based custom


software projects. The first part of risk assessment, risk identification aims to produce
all possible risks for a project [61]. There are various techniques for risk identification:
checklist, decision-driver analysis, assumption analysis, decomposition [30] and
brainstorming. Risk analysis aims to provide decision support for other risk
management activities [62]. For the second part of risk assessment, Boehm provides
typical techniques for risk analysis: performance models, cost models, network analysis,
decision analysis and quality-factor analysis [30]. Based on a literature review,
Georgieva et al. [63] classify risk assessment methods into three categories: risk
assessment techniques, which mainly use neural network; type of input information,
which is either questionnaire or software metrics; and software lifecycle phase. For the
last part of risk assessment, risk prioritization produces a ranked ordering of risk items
previously identified [30]. Risk prioritisation activities are include risk exposure, risk
leverage, compound-risk reduction [30] and risk matrix [38][64].

A risk checklist provides a set of risk items that initially helps identify possible sources
of risk and can be used to quickly assess risks associated with a project [65][66]. Boehm
identifies 10 software risk items from project managers and user representatives [30].
Based on Boehm’s [30] risk list, Ropponen and Lyytinen [58] develop six components
15
of software development risk from project manager perspectives. Other risk checklists
are based on taxonomies. For example, the Software Engineering Institute has proposed
a Taxonomy of Software Development Risk [67]. This taxonomy is organised into three
major classes: product engineering, development environment and program constraints.
This risk taxonomy has been implemented in governmental projects [68] and industry
[69] and empirically investigated in Korean projects [70]. The risk taxonomy has also
been used as a baseline to develop a risk taxonomy for software maintenance [61].
Using a literature review, Barki et al. [57] surveyed project managers and identified 23
risk items grouped by five categories. Based on literature, previous experiences, local
circumstances and usage of language of the organisation, Heemstra and Kusters [22]
identified 36 risk items in nine categories to interview each participant of 5 projects
within a Dutch governmental organisation. Based on interviews of 14 Irish software
project managers, Moynihan identified 21 personal constructs relating to risks [71].
Based on three expert group panels from Hong Kong, Finland and the United States,
Schmidt et al. [59] identified 53 risk items grouped by 14 risk categories for the source
of risk. From the above example, there are many risk items that should be considered as
relevant risk items in a software project [30][65]. Furthermore, a risk checklist must
take into account all key stakeholders in a software project [65].

Table 2.2 Risk checklists in the literature extended from Keil et al. [66]

Checklist Risk items Sources


Boehm [30] 10 risk categories Survey of experienced
project managers and user
representatives
Ropponen and Lyytinen Six risk items Survey (self reported data)
[58] of 83 project managers
covering nearly 1100
projects
The Software Engineering Three categories Literature combined with
Institute [67] experiences
Webster et al. [61] 91 risk items, three Systematic literature
categories review
Barki et al. [57] 23 risk items; five Systematic literature
categories review
Heemstra and Kusters [22] 36 risk items; nine Literature, previous
categories experiences, local
circumstances and usage of
language of the
organisation
Moynihan [71] 21 personal constructs Interviews with 14 Irish
16
relating to risks software project managers
Schmidt et al. [59] 53 risk items; 14 risk International Delphi study
categories with experienced project
managers

To effectively implement risk management, Schmidt et al. argue that all stakeholders
must cooperate in and be open about risk management [8]. Other studies propose
collaborative risk identification and analysis, by confronting each stakeholder’s
perception of potential risks, discussing risk probabilities and effects, and finally
agreeing upon the most relevant risks and their mitigation strategies [22][23]. The Riskit
method offers a broad and fully documented approach for risk management including
all relevant stakeholders [38][64]. In summary, even though the aforementioned risk
management methods are similar to many other risk management [38][64], these risk
management methods involve all relevant stakeholders in a software project. However,
these risk management methods do not provide detail on how to analyse and compare
differences in risk perceptions between stakeholders.

Different stakeholders have different perceptions regarding project risks [16].


Therefore, in risk management, it is important to involve all key stakeholders
[22][8][23] when aiming to take account of differences in risk perceptions. It is
expected that a better understanding of software risks, gained by integrating software
developer and acquirer perspectives, will help to manage risk more effectively and
avoid the conflict [16]. All of these differences may occur because of differences in
backgrounds, experiences, needs, and expectations [18][23]; and each party’s goals and
structures [19]. Perceptions are also influenced by the position, role and task of a
stakeholder [23]. This study investigates differences in process-related risk perceptions
in OTS-based custom software projects between the two main project stakeholders:
developers and acquirers. Process-related risks are defined as “risks that are related to
the overall process, process elements and their interaction” [21:14].

In OTS-based software development, some studies [72][73] map OTS-based software


development issues, challenges and risks to their related stakeholders. However, these
studies [72][73] have not analysed differences in risk perceptions among stakeholders.
Rose [74] uses his experience to discuss the risks specific to OTS-based software
development and maintenance, and describe strategies to mitigate these risks. Boehm et
17
al. [75] reports 16 risks and mitigation strategies for software engineering course
projects using OTS software. The work of Li et al. [27] validated occurrences of OTS-
based software development risks taken from experience and lessons learned, and is the
most related to our research. In this thesis, we partially replicated Li et al. [27] by using
Li’s 7 risks, and adding 4 additional risks for OTS-based custom software projects
relevant to software developers and acquirers.

Besides identifying risks and mitigation strategies, risk management of OTS-based


software development also includes risk analysis and prioritization. One approach for
risk assessment for OTS integration process risk is the COCOTS Risk Analyser [76].
This uses expert Delphi Analysis and was initially validated in nine small university-
based projects. Another approach for Kotonya and Rashid [77] uses a risk assessment
of OTS-based software development modifying a form of the risk-weighting scheme
described by Moynihan [71]. This uses questions to assess the extent to which the OTS-
based software development processes support a defined risk management strategy.
However, none of these approaches take account of different project stakeholders.

2.3 Stakeholder Analysis


In a software project, risks affect all stakeholders [8]. One approach that accounts for
different stakeholder involvement in a project is stakeholder analysis [12][13][10].
Stakeholders are defined as anyone who are affected by or can influence the system under
development [10][12][13][14]. Stakeholder analysis considers activities and issues such
as stakeholder identification, identification of area of interest, stakeholder contribution
and expectation, stakeholder influence, strategies to involve stakeholders and stakeholder
responsibility [25][26]. Responsibility is defined as “a duty, held by some agent, to
achieve, maintain or avoid some given state, subject to conformance with organisational,
social and cultural norms” [78:519].

In software-related projects, stakeholder analysis has been used to identify stakeholders,


and to identify their roles, their level of involvement [36][37][13] and their risks
[9][38][10]. With regard to software project-related risks, Gotterbarn and Rogerson
have developed a software development impact statement associating a potential risk
that impacts particular stakeholders with each task [9]. The tasks are derived from a
work breakdown structure (WBS) of the software project, and might be activities in the
WBS or list of requirement specifications [9]. Woolridge et al. [10] have proposed
18
stakeholder risk assessment during the requirements engineering process, comprising
stakeholder identification, analysing stakeholder influence on functional requirements,
assessing the impact of functional requirements on stakeholders, and assessing and
prioritizing stakeholder risks. The Riskit method [38][64] also links risks, project goals
and stakeholder in order to rank risks.

To manage risks effectively, it is important to identify risks [16], to involve


stakeholders [22][8][23] and to identify stakeholder responsibility for risks [22]. A
power/interest matrix, which combines the power to influence a project and the level of
interest that a stakeholder has in the project, can be used to understand the result of
stakeholder analysis [79][33][80]. This general idea is adapted in this thesis to analyse
differences in risk perceptions between project stakeholders. There are 4 quadrants of
the matrix: inform, involve, consult and collaborate (shown in Figure 2.1).

High Keep satisfied Key player


Involve Collaborate
Power Low Minimal effort Keep informed
Inform Consult
Low High
Interest
Figure 2.1 Stakeholder power/interest matrix with stakeholder involvement
‘Inform’ describes the minimal effort of stakeholder involvement in OTS-based custom
software project due to low interest and power of decision-making [33]. As this
stakeholder has limited power and high interest, they should be informed and
‘consulted’ about key decisions that may affect them [33]. A stakeholder that has low
interest and high power must be ‘involved’ in related project activities because they
have the power to make decisions that may affect the project [33]. A stakeholder that
has high interest and high power will be actively involved in and make decisions on
related project activities (‘collaborate’) [33].

In the context of development of a software system, stakeholder responsibility can be


modelled using a responsibility model [81][78][34][35]. A responsibility model describes
responsibilities within a system under development, agents assigned to these
responsibilities and resources used to discharge these responsibilities [78]. A

19
responsibility should also be assigned to the stakeholder who has the competence and
capacity needed to discharge it [34][35].

Control in a project involves a controller evaluating and influencing a controllee [82][83].


In software projects, control mechanisms can be classified as: 1) formal control, “which
rely on adherence to performance standards or prescribed processes”; and 2) informal
controls, “which rely on social or norm-emphasizing strategies” [83:13]. There are two
types of formal control: 1) outcome control, which specifies what the controllee should
accomplish; and 2) behavior control, which prescribes how the controllee should achieve
that outcome [82][83]. There are similarly two types of informal control: 1) clan control,
“which refers to mechanisms that minimize differences between the controller and
controllee’s goals by promulgating common values, beliefs, and acceptable behaviours”
[83:13]; and 2) self control, where, the controllee determines their own goals and actions
[82][83]. In practice, software projects combine the four control mechanisms [82][83]. In
the outsourcing context, there are several factors used to determine a set of controls:
stakeholders’ knowledge, consequent role expectations, perceptions of difficulty for
controllee monitoring, project size, and controllee performance in the early part of a
project [84][82]. In this thesis, we explored the kinds of controls used by the participants
of our case study.

Even though Keil et al. [16] point out the importance of reconciling user and project
manager perceptions of IT project risks, and comparing stakeholders’ perceptions, their
study does not provide detailed steps for reconciling risks. A recent study has also
highlighted the importance of reconciling perspective mismatch in managing processes
of software project [20]. In this study, Adolph et al. [20] introduce two stages of
reconciling perspectives: converging and validating. However, the authors offer no
detailed explanation for achieving perspective agreement. This thesis addresses this gap
directly.

To reconcile differences in stakeholder perspectives, we must first compare stakeholder


perspectives. Svahnberg and Wohlin use Analytic Hierarchy Process (AHP) to compare
and discuss quality attributes and structures of alternative software architecture
decisions [24]. The comparison and discussion aims to create joint and further
understanding of the different architecture candidates, and reasons for their differences.
In risk management of software development, Heemstra et al. [22][23] confront each
20
team member's perception of potential risks, and discuss risk probabilities and effects to
reach the most uniform team decision about risks, probabilities and effects . Their study
[22][23] does not provide detailed steps to discuss different risks perceptions and to
reach decisions (will be further discussed in Section 7.2). Therefore, this thesis uses this
general approach and stakeholder analysis to reconcile differences in process-related
risks perceptions between developers and acquirers.

2.4 Research Methods in Software Engineering Research


Easterbrook et al. [85] discuss how the selection of research methods in software
engineering research depends on the availability of resources, access to subjects [86],
control over variables [86] and skills of the researcher. Another important factor that
influences the selection of an appropriate research method is the research question [85].
It can be important to use multiple methods to address the weakness of any one method
[85].

There are different research classifications in the literature. Based on the purpose of a
piece of research, Robson defines four categories: exploratory (identifying new insights
and generating ideas and hypotheses), descriptive (portraying a phenomenon),
explanatory (investigating an explanation of a phenomenon) and improving (trying to
improve a certain aspect of the studied phenomenon). Another classification is based on
research paradigm. There are three research paradigms that are commonly used in
software engineering research: qualitative, quantitative and mixed [87][88][89][90]. A
qualitative-based study aims to discover trends, patterns and generalization from
explanatory and context information of study participants [88]. A quantitative-based
study aims to find relationships among dependent and independent variables. A mixed
study aims to complement the limitation of each strategy by combining them [87].

This research has used three research methods: systematic mapping study, survey, and
case study. We have performed two different systematic mapping studies, detailed in
Chapter 3 and 4. The survey method was used in two studies (detailed in Chapter 4 and
5). The case study is detailed in Chapter 6. The nature of this research is exploratory and
does not test hypotheses. A qualitative approach was used in this research to address the
complexity of software system development and human behavior as the central role in

21
the software system development [87]. The three research methods were used to
complement each other and strengthen this research.

2.5 Summary
In this chapter, the literature relevant to OTS-based custom software processes and
risks, and stakeholder analysis has been described and analysed. The key points are
highlighted below.

 From the acquirer perspective, there is a need to identify OTS-based software


acquisition processes. Even though there is guidance available on software
acquisition [6], this recommended practice is more applicable for fully developed
software and must be tailored to other types of software acquisition [6], such as
OTS-based software acquisition.

 Most of the literature focuses on risks from the software developer organisation’s
perspective, and little attention has been given to software acquirers [7][15].
However, to manage risks effectively, it is important for all software project
stakeholders to understand the risks [16].

 Existing methods for stakeholder analysis focus on identifying stakeholders, and


analysing roles of, level of involvement of, and risks related to stakeholders.
However, these methods do not accommodate differences in stakeholder
perceptions.

An ideal method for risk assessment of OTS-based custom software projects should
identify, analyse and prioritise risks by involving key stakeholders [22][8][23]. It should
also facilitate reconciling differences in risk perception between the developer and
acquirer.

The overall aims of this research are:

 To understand the processes of OTS-based software acquisition and the risks of


OTS-based custom software project for developers and acquirers.

 To identify risks [16], to involve stakeholders [22][8][23] when assessing


differences in risk perceptions and to identify stakeholder responsibility for risks
[22].

22
Three research methods were selected based on available resources, access to and
control of subjects, and consideration of the research questions [86][85]. A systematic
mapping study was used to answer a type of research question of description and
classification [85]. A survey was used to better understand the normal patterns of
occurrence of the phenomena by asking frequency and distribution questions [85]. A
case study was used to further understand about the phenomena of differences in risk
perception in the context of OTS-based custom software projects. The multiple research
methods complement each other and are appropriate for their purpose.

23
3 Off-The-Shelf-based Software Acquisition Processes

This chapter describes a systematic mapping study on OTS-based software acquisition.


The motivation is to compare and contrast OTS-based software acquisition and non-
OTS-based software acquisition. This chapter addresses RQ1 of this thesis. This chapter
is organized as follows. The first section introduces the background of this study. The
following section describes the mapping study protocol including the results. In next
section, we discuss the results and analyse the results based on the research question.
Then, threats to validity were defined and discussed. Last section concludes this
chapter.

3.1 Introduction
Software development projects increasingly use off-the-shelf (OTS) products,
integrating them into the systems under development. Software-developing
organisations can avoid building every part of their product software ‘from scratch’ by
reusing technologies available from third parties [1]. Acquiring ‘OTS-based software’,
i.e. software that itself uses OTS software platforms or components, can be less
expensive than acquiring fully custom-developed software. However, there is a need to
better understand the OTS-based software acquisition processes.

Most existing studies and guidance on software acquisition do not explicitly deal with
the acquisition of OTS-based software. For example, the IEEE Standard 1062-1998
Edition: Recommended Practice for Software Acquisition can be applied to software
acquisition process regardless of the size and complexity of the software [6]. However,
this recommended practice is more applicable for fully developed software and must be
tailored to other types of software acquisition [6]. Therefore, it motivated us to
investigate the processes of OTS-based software acquisition using a systematic mapping
study.

In the context of empirically-based software engineering, our study has used a


systematic mapping study or scoping study to map evidence about this topic [91][92]. A
systematic mapping study is “a broad review of primary studies in a specific topic area
that aims to identify what evidence is available on the topic” [92:vii]. The main goal of
a systematic mapping study is to provide an overview of a research area and to identify

24
the nature and quantity of evidence in a research area [92]. This chapter presents the
process and results of a mapping study to identify, compare and classify a set of primary
studies of software acquisition and OTS-based software acquisition.

3.2 Method
Our mapping study protocol was created using guidance by Petersen et al [93], adapted
by combining the last two steps of the protocol. There are five steps in the protocol:
definition of research questions, conduct search, screening of papers, keywording using
abstracts, and data extraction and mapping process.

3.3 Definition of Research Questions


The research question in a mapping study is a part of the mapping study protocol. Our
goal is to better understand how OTS-based software acquisition processes compare to
generic software acquisition processes. Our research question is:

RQ. “What are the similarities and differences between software acquisition and OTS-
based software acquisition from the process perspective?”

3.4 Conduct Search for Primary Studies


A mapping study is based on a systematic search of the literature using a search string.
A search string can be structured using the population, intervention or outcome [92]:
1. Population: published articles including empirical studies, industry and
government experiences in software acquisition domain
2. Intervention: related processes, practices and techniques in software acquisition
3. Outcomes: quantity and type of software acquisition and OTS-based software
acquisition processes, practices and techniques.

The search string defined in this mapping study is based on keywords from the research
question. The keywords ‘software procurement’ and ‘software purchase’ are used as
synonyms for ‘software acquisition’. To extend the search, we also used ‘COTS’, for
commercial-off-the-shelf and ‘OTS’ for off-the-shelf combined with one of the
following strings: ‘acquisition’, ‘procurement’ or ‘purchase’. The mapping study
focuses on processes for OTS-based software acquisition, not processes specific to
acquisition of packaged software. Nevertheless in Appendix B, we show some literature
found for acquisition of packaged software and ERP systems ([24], [71], [78] and [79])
and for OSS acquisition ([41], [53], [93] and [95]). As we focused on OTS-based
25
software acquisition, the term 'CBSE' was not used because it usually refers to OTS-
based software development. We also did not use the keyword “OSS” in the search
strings, because the definition and classification of OTS software also includes OSS
software and not vice versa [5][48].

All of the strings are combined using Boolean ORs and AND to construct the search
string used in this mapping study. The search string is: ‘software acquisition’ OR
‘software procurement’ OR ‘software purchase’ OR ((cots OR ots) AND
(acquisition OR procurement OR purchase)). The search results using the search
string are shown in Table 3.1 describing publication resources, years of publication,
advanced search methods for each of the publication and results. We used Zotero [94], a
bibliography management tool, to manage literature search results.

Table 3.1 Search results using the search string

Publication Year Advanced search Number


(search of result
results)
ACM Portal 1985-2010 Title, abstract, keywords 16
IEEE Xplore 1998-2010 Title, abstract, indexing 172
terms
Springerlink 1998-2010 Title, abstract 42
Elsevier 1984-2010 Title, abstract, keywords 36
Wiley InterScience 1990-2010 All Fields, all subjects and 51
Journals
CiteSeerX 1998-2008 Title, abstract, keywords 86
Manual using Google 1997-2009 - (manual searches) 9
Scholar

3.5 Screening Papers for Inclusion and Exclusion (Relevant Papers)


Inclusion and exclusion criteria are used to ensure search query results are relevant to
answer the research question.
1. Inclusion: books, papers, technical reports, reference models and standards that
relate to software acquisition process. For papers, which reported the same
study, the one published in a peer reviewed publication was included, or else the
most recent was included. Where a paper reported several studies, each relevant
study was treated separately. We focused only on primary studies.
2. Exclusion: hardware acquisition, acquisition risks, OTS-related papers that are
not related to software acquisition process.
26
Table 3.2 provides the results of the search after inclusion and exclusion criteria were
applied. The results are classified as relating to (generic) software acquisition or to only
OTS-based software acquisition. We designed a data extraction form to collect
information from each primary study. We collected extracted data in Zotero [94]. The
data extraction process, which will be explained in the next section, was performed
simultaneously with the final selection process. The form consisted of the following
information: publisher, published year, paper classification. The paper classification is
described in the end of the next section.

Table 3.2 Refined results after paper screening

Publication Software OTS-based Total


acquisition software
acquisition
ACM 1 3 4
IEEE 13 19 32
Springer 8 16 24
Elsevier 5 5 10
Wiley InterScience 4 6 10
CiteSeerX 1 5 6
Manual using Google 7 2 9
Scholar
Total 39 56 95

3.6 Data Extraction and Mapping of Study


The data extraction process in a mapping study uses a classification scheme [92]. In this
chapter we used keywording of abstracts [93] as a technique to extract data. The
keywording was conducted by reading abstracts and identifying keywords reflecting
topics under investigation. In the case of insufficient information provided by the
abstracts and keywords, we also read the introduction and conclusion of the paper.

In order to identify direct evidence from primary studies, we defined in the study
protocol a classification of software acquisition processes based on IEEE 1062
Recommended Practice for Software Acquisition. These processes are: Planning
organisational strategy, implementing organisation’s process, determining the software
requirements, identifying potential suppliers, preparing contract requirements,
evaluating proposals and selecting the supplier, managing supplier performance,

27
accepting the software and using the software. This topic classification was used to map
data extracted from the publications.

During the keywording process, new sub-categories were identified and added into the
topic classification based on screening results that could not be classified into the topic
classification sub-categories but suited the population, intervention and inclusion
criteria. Because the purpose of this mapping study was to identify process similarities
and differences between software acquisition and OTS-based software acquisition, the
mapping study separated publications into two different classes: (generic) software
acquisition and OTS-based software acquisition. After finishing the keywording
process, new topics were added as sub-categories, as shown in Table 3.3. Appendix B
lists the mapping study results with their mapped papers. For the (generic) software
acquisition classification, six new topics were added: decision-making: make vs. buy,
modelling and simulation, software acquisition improvement, process life cycle,
architectural decision-making and managing relationships between developers and
acquirers. Seven new topics were added to the OTS-based software acquisition
classification: decision-making: make vs. buy OTS products vs. use OSS, process life
cycle, architectural decision-making, OTS selection, managing relationships between
OTS adoption and acquirers’ organisations, managing relationships between OTS
vendors and developers, and managing relationships between developers and acquirers.
A process life cycle topic was also added to both of the software acquisition
classifications.

Three initially-proposed software acquisition topic sub-categories [6] had no matching


results from the primary studies. These topics were: managing supplier performance,
accepting the software and using the software.

After finishing the keywording and classifying the primary studies based on software
acquisition and OTS-based software acquisition topics, the frequencies of primary
studies were determined, as shown in Table 3.3. Our discussion as follows is based on
this table and on a thorough reading of the identified publications.

To ensure broader coverage [92], we did not perform specific quality assessment on the
selected studies. However, to define quality of the primary studies, we classified the
mapping study results (listed in Table 3.3) as follows: empirical research, experience,

28
conceptual framework and standard. Empirical research papers are combination of
evaluation and validation research papers [95]. Experience papers, in this mapping
study, comprised experience and solution research papers [95]. In addition, we used a
category of conceptual framework instead of opinion and philosophical paper [95]. The
details of this classification can be found in Table 3.4. We added one paper
classification, standard, because we found two standards for software acquisition
[6][50] and one best practice for software acquisition improvement that can be
considered to be a standard [96].

Table 3.3 Frequency of identified processes according to software acquisition


classification

Process Software acquisition OTS-based software


acquisition
Em Ex Cf Std Tot Em Ex Cf Tot
Planning organisational strategy 1 1 1 1
Implementing organisation’s 1 1
process
Determining the software 1 13 14
requirements
Identifying potential suppliers 1 1 1 1
Preparing contract requirements 1 1
Evaluating proposals and 5 5 2 2
selecting the supplier
Decision-making: make vs. buy 3 2 1 6 5 2 7
(also vs. buy OTS products vs.
use OSS for OTS-based software
acquisition classification) (*)(+)
Process life cycle (*)(+) 10 2 2 14 3 5 8
Architectural decision-making 1 1 1 5 6
(*)(+)
Modeling and simulation (*) 3 2 5
OTS selection (+) 6 25 31
Managing relationships between 2 1 3 1 1
developers and acquirers (*)(+)
Software acquisition 5 1 1 7
improvement (*)
Managing relationships between 1 1
OTS adoption and acquirers’
29
organisations (+)
Managing relationships between 1 1
OTS vendors and developers (+)
Total 9 27 4 3 43 18 57 75
Legend: (*): new topics added to software acquisition classification; (+): new topics
added to OTS-based software acquisition classification; Em: empirical research paper;
Ex: experience paper; Cf: conceptual framework paper; Std: standard paper; Tot: total
number of papers

Table 3.4 Research paper classification summarised from the paper classification by
Wieringa and Heerkens [95]

Paper Paper classification proposed by Wieringa and Heerkens [95]


classification
used in this Paper type Description
study
Empirical Evaluation The paper describes the investigation or the
research research implementation of a problem in practice. The
paper uses a sound research method (either
empirical or conceptual) to evaluate the
investigation or the implementation.
Validation The paper describes the investigation of the
research properties of a solution proposal using a sound
research method. The solution is novel and has
not yet been implemented. The research methods
may be experiments, simulation, prototyping,
mathematical analysis etc.
Experience Experience The paper consist a list of lessons learned from
the author's experience without a discussion of
sound research methods.
Solution proposal The paper proposes a novel solution or a
significant improvement of an existing technique
without a rigorous research method.
Conceptual Opinion The paper comprises the author's opinion about
framework what is good or bad and how we should do
something.
Philosophical The paper describes a new conceptual
framework.
Standard Standard These paper types are not included in the paper
classification by Wieringa and Heerkens [95]. In
Best practice
this mapping study, the category of standard was
used to classify standard and best practice papers
found in generic software acquisition.

30
3.7 Discussion
This section provides a discussion on the RQ “What are the similarities and differences
between software acquisition and OTS-based software acquisition from process
perspective?”

OTS-based software acquisition is the acquisition of software that itself uses OTS
software platforms or components. We identified OTS-based software acquisition
processes from the literature, and compared them with a process standard for software
acquisition [6]. The differences between these processes concern the acquisition of OTS
products [42][55][43] and also relate to the influence of the use of OTS products on
software development approaches [41][39][40]. Traditionally, software development
starts with system requirements definition, then defines the system architecture, and
continues with implementation. In OTS-based systems development, there is
simultaneous definition and tradeoff among the OTS marketplace, system requirements,
and system architecture and design [41][39][40].

Even though not all the standard software acquisition processes exist among the
software acquisition processes identified from the literature (Table 3.3), both cover the
life cycle [6]. The standard identifies processes for managing supplier performance,
accepting the software and using the software [6], which were not found in the primary
studies. However, the primary studies include implementing the organisation’s process,
determining the software requirements and preparing contract requirements topics,
which are not found in the software acquisition standard. Elgazzar et al. [97] discuss the
planning and contracting phase of OTS-based software acquisition stressing the impact
of OTS on requirements and contract structure.

There are some commonalities between (generic) software acquisition and OTS-based
software acquisition. One common process involves decision-making to make or buy
software, but a particular condition of OTS-based software acquisition is the
consideration of use of third party Commercial Off-the-Shelf (COTS) products [98][98],
Enterprise Resource Planning (ERP) systems [3], and open source software (OSS)
[99][100]. Another commonality concerns making architectural decisions during
software acquisition [101]. These should be suited to the organisation’s needs [101],
corporate governance [102], and system architecture [103]. Another common concern is

31
the nature of the working relationships between developers and acquirers, through
cooperation, integration and establishing familiarity [104][105][106][102].

Two processes were found for generic software acquisition during the literature search.
These processes are not referenced in the software acquisition standards: modelling and
simulation, and software acquisition improvement. However, there was no explicit
mention of these processes within the OTS-based software acquisition literature. These
can be viewed as gaps in the literature.

The main difference from generic software acquisition introduced by OTS-based


software acquisition is the relation between OTS selection and determining the software
requirements. As shown in Table 3.3, 31 of the total 75 publications on OTS-based
software acquisition concern OTS selection. This indicates that in OTS-based software
acquisition classification, OTS selection is a key process. As can be inferred from Table
3.3, OTS selection not only influences user requirements, but also architectural
decision-making [103]. In regard to software requirements, OTS selection is
intertwined with software requirements definition [103] to avoid risk in OTS selection
[107]. Along with determining software requirements and performing OTS selection,
software architectural is also defined and adjusted iteratively to build an OTS-based
system solution [103]. In OTS-based software acquisition, these three processes are
intertwined because OTS product selection cannot be conducted after architectural
design. This is because an architecture designed without awareness of available OTS
components is unlikely to find appropriate OTS products to meet its needs [103].

There are relationships and organisational issues that must be addressed in OTS-based
software acquisition. Two specific issues in OTS-based software acquisition that do not
occur in generic software acquisition concern managing the relationships between
(third-party) OTS vendors and the acquirers’ organisations, and managing relationships
between OTS vendors and developers. In regard to organisational issues, OTS-based
software acquisition must consider several characteristics of the organisation and its
personnel [108]. Finally, a long-lasting and deep partnership relationships between OTS
vendors and developers can provide benefits in commercial negotiations with acquirers
[109].

32
In sum, OTS product selection is a significant process in OTS-based software
acquisition that distinguishes it from generic software acquisition [6]. Existing software
acquisition standards and processes [6] must be adjusted to accommodate the impact of
third-party OTS components in software acquisition.

3.8 Threats to Validity


This section discusses construct, internal and external validity for the mapping study by
following a previous mapping study [110] to analyse threats to validity.

3.8.1 Construct Validity


Construct validity is about the use of adequate definitions and measures of variables [90].
We focused on three aspects to validate constructs used in the mapping study. The first
aspect was keywords used in the search strings. In the mapping study, the synonym of
‘software acquisition’, OTS and acquisition were used as the part of search strings to
expand the results of the query. Secondly, the mapping study searched the primary
studies using: five major digital library databases: IEEE, ACM, Springer, Science Direct
and Wiley Online; additional two manual searches: CiteSeerX and Google scholar. All
of these major databases covered variety and important related conferences and journals
in software engineering. Therefore, it was expected to find sufficient literature to be
analysed. The last aspect, we attempted to define our initial classification to be robust
enough for analysis. In order to do this, the mapping study extended keywording of
abstract [93] as a categorization scheme. Keywording began by reading abstracts and
identifying keywords reflecting topics under investigation. When insufficient
information was provided by the abstracts and keywords, the researchers also read the
introduction and conclusion of the paper.

3.8.2 Internal Validity


Internal validity is concerned with whether research procedures, treatments or experiences
of the research participants influence the researcher's ability to draw correct inferences
from the data [90]. There were minimal threats to internal validity on the mapping study
because it only used descriptive statistics to analyse mapped papers.

33
3.8.3 External Validity
External validity refers to the generalisability from this study [90]. Because the
conclusions in the mapping study were only specific for OTS-based software
acquisition processes, external validity threats are not applicable to the mapping study.

3.9 Summary
OTS-based software acquisition is the acquisition of software that itself uses OTS
components or products. We have presented the findings of a systematic mapping study
on OTS-based software acquisition. We have suggested that for OTS-based software
acquisition, changes should be considered for the existing software acquisition process
standard [6].

Both generic and OTS-based software acquisition have the same overall process
lifecycle. The main difference in OTS-based software acquisition is that there is a
relationship between determining the software requirements and OTS selection. Almost
half of publications covering OTS-based software acquisition concern OTS selection.
OTS selection is also related to architectural design making [105]. Together,
architecture and OTS selection criteria are defined and adjusted iteratively to build an
OTS-based system solution [105]. Two additional impacts in OTS-based software
acquisition concern managing relationships between OTS vendors and the acquirers’
organisations, and managing relationships between OTS vendors and developers.

34
4 Risks of OTS-based and Custom Software Development and
Acquisition

This chapter describes two studies of risks of OTS-based custom software projects: a
systematic mapping study and a survey. This chapter addresses RQ2 of this thesis. The
remainder of this chapter is organized as follows. Section 4.1 introduces the general
background. The next section discusses the research design. Then we present systematic
mapping study processes and results, followed by the survey results. We discuss
answers to the research question and threats to validity before concluding.

4.1 Introduction
In a software project, risks affect all stakeholders [8]. The risks of the software project
arise from the beginning of software acquisition process on software acquirer’s side
[8][7]. Most of the literature focuses on the risks from software developer organisation
perspective and little attention has been given to software acquirers [7][15]. However, to
manage risks effectively, it is important for all software project stakeholders to
understand the risks [16]. In addition, there is a need to better understand OTS specific
risks related to technical development processes [17] as a consequence of using OTS
products. In this study we addressed risks to both developers and acquirers of OTS-
based custom software.

This chapter reports the process and results of a mapping study of published papers on
risks in off-the-shelf and custom-based software acquisition and development. The
objective of this mapping study was to identify, classify and compare risks that exist in
OTS and custom-based software projects not only from the software development
viewpoint but also the software acquisition viewpoint. Schmidt et al. argue that software
project risks can best be managed cooperatively between software developers and
acquirers [8]. Consequently, identifying risk components of OTS-based custom
software is expected to give more understanding of risks of OTS-based custom software
projects. In order to investigate risks of OTS-based and custom software development
and acquisition in real world settings, we used the mapping study results to design a
survey of OTS-based custom software project risks that may be relevant to developers
and acquirers. The respondents of the survey were software developers and software
acquirers of Indonesian background.
35
4.2 Research Design
This study used a systematic mapping study to identify, classify and compare risks of
OTS-based and custom software development and acquisition, and then empirically
investigated OTS-specific risks that may be relevant to the developer and acquirer using
a structured online questionnaire of Indonesian developers and acquirers. The
systematic mapping study provides broad evidence of risks of OTS-based custom
software projects. In addition, the mapping study was able to provide a structure to map
primary studies of risks of OTS-based and custom software acquisition, which have
previously been given less attention than risks of OTS-based and custom software
development [7][15]. Our survey provides evidence of the existence of these risks in
practice, and complements a previous study [27] by adding information about the
acquirer perspective. This survey is exploratory and does not test hypotheses.

4.2.1 Research Questions


This chapter addresses RQ2 of this thesis, and has two sub-research questions (RQ2.1
and RQ2.2). The first sub-research question aims to identify, classify and compare
risks that exist in OTS-based and custom development not only from the development
viewpoint but also the software acquisition viewpoint. This is broken down into four
development context in the mapping study: OTS-based software development, OTS-
based software acquisition, custom software development and custom software
acquisition.
RQ2.1 What risks are related to OTS-based custom software projects?
RQ2.1.1 What risks are related to OTS-based software development?
RQ2.1.2 What risks are related to OTS-based software acquisition?
RQ2.1.3 What risks are related to custom software development?
RQ2.1.4 What risks are related to custom software acquisition?
The second research question aims to empirically validate the occurrence in practice of
risks of OTS-based custom software projects relevant to developers and acquirers.
RQ2.2 Which OTS-specific risks relevant to developers and acquirers occur
frequently?

36
4.2.2 Research Methods
There are two research methods used in this study: a systematic mapping study and a
survey. A systematic mapping study was used for answering RQ2.1. We then compared
and selected the mapping study results to find risks that may be relevant to the
developers and acquirers. A structured online questionnaire was then used to collect
data about the occurrence of these risks. The survey answered RQ2.2 of this chapter. In
the following sections, each method will be detailed.

4.3 Systematic Mapping Study on Risks of Off-The-Shelf-Based Custom


Software Development and Acquisition
We followed a template for a Mapping Study Protocol from the Evidence-Based
Software Engineering research group at Durham University [111]. Based on the
template, we developed a mapping study protocol that guided this study. There are five
steps of a mapping study described by the template: research questions (mentioned as
background in the template), search strategy, selection criteria, data extraction and
synthesis. This section describes four steps of our mapping study protocol, excluding
research questions.

4.3.1 Search Strategy


A mapping study is based on a systematic search of the literature using defined search
strings. The search strings can be structured using population, intervention and outcome
[92]. Our search strings target the following:
1. Population: published articles including empirical studies, industry and
government experiences in risks of OTS-based software development and
acquisition domain
2. Intervention: risks related to process, OTS product, people, project management
3. Outcomes: quantity and type of risks in OTS-based software development and
acquisition
The process used automated searches on five major software-related digital libraries:
IEEE Xplore, ACM Digital Library, Springer, Science Direct and Wiley Online. In this
mapping study, the use of automated searching aimed to get broad scope of peer-
reviewed papers [112] related to the investigated topics. We used Zotero [94], a
bibliography management tool, to manage literature search results.

37
We defined four groups of search strings in this mapping study derived from the
research question keywords, as shown in Table 4.1. ‘COTS’, ‘OTS’, ‘component-based’
and ‘off-the-shelf’ keywords represented the type of systems under study. These
keywords were combined with either ‘system’ or ‘software’ using the AND operator.
These keywords covered broader software-based system development. The keyword
‘development’ summarized activities related to system development life cycle. The
keyword ‘risk’ was used to focus on risk itself and not on factors constituting risk (e.g.
possibility, loss, and hazard). For OTS-based software acquisition risk, the search
strings also included the keywords ‘procurement’ and ‘purchase’ as synonyms of
‘acquisition’. This study is limited to risks in common to COTS, OTS, component-
based software and OSS [27]. Therefore we did not use OSS keyword in search strings
to answer RQ2.1.1 and RQ2.1.2, because the definition and classification of OTS
software also includes OSS software and not vice versa [5][48]. For custom software,
the keywords ‘custom’, ‘contracted’ and ‘bespoke’ were used.

Table 4.1 Search strings used in this study

RQ Development Search strings


context
RQ2.1.1 OTS-based (COTS OR OTS OR ‘component-based’ ‘OR off-the-
software shelf’) AND (system OR software) AND development
development AND risk
risks
RQ2.1.2 OTS-based (COTS OR OTS OR ‘component-based’ OR ‘off-the-
software shelf’) AND (system OR software) AND (acquisition OR
acquisition procurement OR purchase) AND risk
risks
RQ2.1.3 Custom (custom OR contracted OR bespoke) AND (software OR
software system) AND development AND risk
development
risks
RQ2.1.4 Custom  (custom OR contracted OR bespoke) AND (software
software and system) AND (acquisition OR procurement OR
acquisition purchase) AND risk
risks  (‘software acquisition’ OR ‘software procurement’ OR
‘software purchase’) AND risk
We used the above search strings in IEEE, ACM, Science Direct, Wiley Online, and
Springer databases. In IEEE, ACM, Science Direct, and Wiley Online database, we
used the advanced search feature, which searched for the search strings on title, abstract
and keywords of the papers, but in Springer database, we could only search based on
title and abstract in its advanced search feature. In Wiley Online we added a search
38
string for publication titles, ‘(computer OR computing OR computation OR information
OR software OR system OR informatics OR component)’, to limit searching only to
software-related publications.

4.3.2 Selection Criteria


After the search, the results were manually vetted. Inclusion and exclusion criteria were
used to ensure that search query results are relevant to answer the research questions,
and are shown in Table 4.2. Using these criteria, there were two steps in selecting
primary studies. The initial selection was conducted by reading abstracts and keywords
reflecting the development context under investigation. The classifications of the
development context for this study were: OTS-based software development risks, OTS-
based software acquisition risks, custom software development risks and custom
software acquisition risks. In the case of insufficient information provided by the
abstracts and keywords, in the final step we read the candidate primary studies
thoroughly to select the relevant papers. In this final step we also conducted data
extraction (discussed in the next sub-section).

Table 4.2 Inclusion and exclusion criteria used in this study

Inclusion criteria Exclusion criteria


Peer reviewed paper (proceedings and Poster, tutorial, workshop paper, panel
journals) discussion paper, guest editors,
introduction, abstract paper, book and
technical report
Risks that were related to either software Risks that were not related to either
development or acquisition software acquisition or development
Risks that were addressed and discussed in Risks that were only mentioned in the
a paper introduction or not discussed in the paper
Paper published from 1991 until 2011 In-house software development and
acquisition
For papers, which reported the same study, In ACM Digital library, resulted papers
the one published in a peer reviewed that were not published by ACM were
publication was included, or else the most excluded to omit the same resulted papers
recent was included. Where a paper from IEEE, Elsevier Science Inc.,
reported several studies, each relevant Springer-Verlag and Wiley Online.
study was treated separately
If a paper contains risks that could be
mapped to more than one investigated
topic, each identified risk was mapped to
all corresponding classifications of
development context.
If the same papers were resulted from

39
different query strings (see Table 4.1) then
only one paper was used.

4.3.3 Data Extraction


We designed a data extraction form to collect information from each selected primary
study. We collected extracted data in Zotero [94]. The data extraction process was
performed simultaneously with the final selection process by reading each candidate
primary study thoroughly. The form is described in Table 4.3.

Table 4.3 Data extraction form description

Description
Publisher
Published year
Paper classification: classified as either empirical research or experience or conceptual
framework [95]
Risks: risks found in the primary study
Risk category: defined categories to group risks in this study

4.3.4 Synthesis
We extracted keywords from abstracts [93] using the categorization scheme. Using the
data extraction results, we formed keywords to create map of the primary studies. We
initially classified risks into seven categories: four categories of OTS-based software
development processes [40] comprising requirement, design, coding, integration and
testing, and 3 non-process categories: OTS product, people, and project management.
Keywords were predefined but were extended with new keywords that were relevant to
the population, intervention and inclusion criteria. During the keywording process, the
category classifications were updated based on screening results that did not match the
proposed category classifications but did match the inclusion criteria.

4.3.5 Mapping Study Result


Following systematic search, there were 712 papers found. The search was performed
on 5 and 6 April 2011. There were 142 candidate primary studies identified in the
initial selection from the search results. In the final selection, 69 papers were
determined as primary studies. The search and selection results are presented in Table
4.4. One paper was removed from the category of custom software development to
custom software acquisition after being read thoroughly, thus making the final selection
for IEEE (5 papers) more than its initial selection (4 papers).

40
Table 4.4 Search and selection results

Digital OTS-based OTS-based Custom software Custom software


library software software development acquisition paper
development acquisition paper paper
paper
QR Fin QR Init Fin QR Init Fin QR Init Fin
(year) Init (year) (year) (year)
IEEE 142 52 23 33 4 4 242 23 6 19 4 5
(1986 - (1995 (1970 (1991-
2011) -2010) - 2011)
2011)
ACM 105 10 4 17 1 1 38 7 0 18 2 0
(1994 - (1997 (1983- (1993-
2010) -2010) 2010) 2009)
Springer 30 14 10 3 3 3 12 2 1 3 3 1
(1997 - (2002 (2003- (2003-
2010) -2005) 2010) 2010)
Science 12 3 3 1 0 0 17 0 1 2 2 2
Direct (1985 - (2003) (1979- (1993
2011) 2009) &
1999)
Wiley 10 6 3 2 2 1 1 1 0 5 3 1
Online (1996 - (2006 (2011) (2003-
2009) & 2010)
2010)
Total 310 47
QR 299 56
Total 33 14
Init 85 10
Total 8 9
Fin 43 9
Note: QR: query result; Init: initial selection; Fin: final selection

After updating the initial categorization scheme (see previous sub-section), we


identified 17 risk categories. These categories are presented in Table 4.5: planning,
requirements, design, integration and testing, system lifecycle, maintenance, project
closure, software, OTS products, cost, environment, people, systems engineering,
vendor relationships, project management, contract and legal. Of the initial risk
categories, coding was not included in the final risk categories because no papers were
mapped to this category. The 133 risks of OTS-based software development are mapped
into 13 risk categories (detailed in Table 4.6). The 36 risks of OTS-based software
acquisition are grouped into 14 risk categories (presented in Table 4.7). Risks of custom
software development are mapped into 11 risk categories consisting of 28 risks (see

41
Table 4.8). Finally, there are 20 risks mapped into 11 risk categories for custom
software acquisition. These results are broadly in line with previous studies [7][15]
pointing out that risks related to software acquisition are less studied than software
development. Appendix C lists the mapping study results with their mapped papers. The
mapping results are described in the following sub-sections.

To ensure broader coverage [92], we did not perform specific quality assessment on the
selected studies. However, to define quality of the primary studies, we classified the
mapping study results (listed in Table 4.5) into a modified paper classification scheme
[95] as follows: empirical research, experience and conceptual framework (as
discussed in Section 3.6).

42
Table 4.5 Mapping study classification based on type of paper

Risk category OTS Development OTS Acquisition Custom Development Custom Acquisition
Em Ex CF Total Em Ex CF Total Em Ex CF Total Em Ex CF Total
Planning 2 2 4 3 3 1 1 2 1 3
Requirement 24 6 5 35 2 2 3 7 5 1 1 7 1 1 2
Design 2 4 6 1 1 1 1
Integration and 19 3 5 27 4 4 1 1
testing
System lifecycle 1 3 4 1 1
Maintenance 7 1 5 13 3 3
Project closure 1 1 1 1
Software 2 2 1 1 1 1
OTS products 7 5 10 22 2 2
Cost 1 1 1 1 2 2
Environment 2 2 4 1 1 2 1 1
People 2 2 4 1 1 1 1 2 1 1
Systems 3 3 1 1
engineering
Vendor 6 2 1 9 2 3 5 6 2 8 3 2 1 6
relationships
Project 2 1 3 4 4
management
Contract 1 1 1 1 1 1
Legal 1 1 1 1 1 1
Total 70 24 39 133 10 2 24 36 21 4 3 28 12 3 5 20
Note: Em: Empirical research paper; Ex: Experience paper; CF: Conceptual framework paper

43
4.3.6 OTS-Based Software Development Risks
Table 4.6 lists 13 risk categories of OTS-based software development. In this mapping
study, there are no risks found related to: project closure, software, project management
and contract.

Table 4.6 OTS software development risks mapped in this study

Risk category Total OTS software development risks


mapped
papers
Planning 4 Difficulty in prioritizing features to be implemented;
Introducing new OTS candidates is likely and requires re-
planning;
Multi-OTS products from different vendors may result in
complicated licensing arrangements
Requirements 35 Poor requirement definition and analysis;
Re-negotiate requirements with the customer, if OTS
components could not satisfy all requirements;
Lack of OTS-driven requirements engineering process;
Unknown security of a new component;
Security concern in OTS selection;
Requirement changes;
OTS components could not be sufficiently adapted to
changing requirements;
Requirements mismatches with OTS selection;
Vendor’s financial stability and technology capability;
Wrong component may be selected;
Insufficient OTS selection estimation effort;
Ability or willingness of the organisation to accept the
impact of OTS requirements;
Too much time spent in OTS selection;
Added complexity of unused OTS features;
Overly optimistic OTS package learning curve
Design 6 Lack of early analysis of system quality;
Architectural mismatches
Integration 27 Underestimation of integration effort;
and testing Poor component documentation;
Too much effort needed to solve mismatch between OTS
components;
Difficulty when integrate with other systems;
Imposed black box testing of OTS components;
Late OTS component integration;
Lack of component integration standard;
Difficulty for sequencing OTS integration activities;
Negative impact on security;
Negative impact on reliability;
Negative impact on performance;
Difficulty to identify whether defects were inside or outside
44
the OTS components
System 4 Difficulty to integrate OTS security aspects into system
lifecycle lifecycle development;
Many new non technical activities are introduced in the
software engineering process
Maintenance 13 Difficulty to upgrade with the latest version of OTS
products;
OTS impact on upgrade cycles (installation, impact on
system operations, and impact on integration code);
OTS version upgrade during development;
Negative impact of component update on systems
operability;
The different customer-vendor evolution cycles may result
in an uncertainty about how often OTS components in a
system may have to be replaced;
Maintenance planning unfeasible;
Faulty vendor claims;
Higher long-term maintenance cost;
Reduced control of future evolution of the system
OTS product 22 Feature limitation;
OTS product defect;
Long-term product dependency;
High expectation from client;
Lack of vendor support;
OTS product aging, OTS product obsolescence;
Unknown product features;
Unknown quality;
Overly optimistic expectations of OTS quality attributes;
Multitude of licenses;
Lack of liabilities and information;
Attempting and subsequently failing in the certification
process;
The vulnerability risks associated with hidden design
assumptions in black box components;
Difficulty in mapping critical quality attributes to
component architectures
Cost 1 Increasing in the overall development effort and cost

Environment 4 Lack of infrastructure support;


Undocumented system and business processes;
Domain inadequacies
People 4 Lack of skilled resources;
Inexperienced OTS integrator;
Cultural barriers
Systems 3 Probability of success of assembled components according
Engineering to planned schedule;
Probability of achieving business value by using the new
system;
Accomplishing the development within time and budget

45
constraints
Vendor 9 Vendor did not provide enough technical support/ training;
relationships Dependency on external parties (vendor lock-in);
Information on the reputation and technical support ability
of vendor were inadequate;
Difficulty in coordinating meetings with key personnel
Legal 1 Dispute and litigation
Total 133

4.3.7 OTS-Based Software Acquisition Risks


Table 4.7 reveals 14 risk categories of OTS-based software acquisition. In this mapping
study, there are no risks found related to: legal, system lifecycle and design.

Table 4.7 OTS software acquisition risks mapped in this study

Risk category Total OTS software acquisition risks


mapped
papers
Planning 3 Underestimated cost estimation;
Multi-OTS products from different vendors may result in
complicated licensing arrangements
Requirements 7 Poor requirement definition and analysis;
Lack of OTS-driven requirements engineering process;
Requirements mismatches with OTS selection;
Vendor’s financial stability and technology capability;
Wrong component may be selected
Integration 4 Underestimation of integration effort;
and testing Poor component documentation;
Too much effort needed to solve mismatch between OTS
components;
Poor interoperability with legacy systems
Maintenance 3 Maintenance planning unfeasible;
Reduced control of future evolution of the system
Project 1 The project was delivered long after schedule
closure
Software 2 System is not developed according to the user requirements
OTS product 2 Unknown quality
Cost 1 Increasing in the overall development effort and cost
Environment 2 Changes in business and organisational environment that
affected the project
People 1 Lack of skilled resources
Systems Eng. 1 Not addressing changing in system operation
Vendor 5 Dependency on external parties (vendor lock-in);
relationships Loss of bargaining power in vendor relationships;
Parties shirking their agreed upon responsibilities;
Lack of user commitment;
Unwillingness to inform management of problems
Project Mgmt. 3 Poor systems of authority;
46
Unrealistic schedules and budgets;
High level of system complexity to be built
Contract 1 Unexpected extension of time-and-materials based contract
Total 36

4.3.8 Custom Software Development Risks


Table 4.8 describes 11 risk categories of custom software development. In this mapping
study, there are no risks found related to: system lifecycle, maintenance, project closure,
OTS products, cost and systems engineering.

Table 4.8 Custom software development risks mapped in this study

Risk category Total Custom software development risks


mapped
papers
Planning 1 Underestimated cost estimation
Requirements 7 Poor requirement definition and analysis;
Requirement changes;
Specification uncertainty
Design 1 Minimal performance analysis
Integration 1 Difficulty when integrate with other systems
and testing
Software 1 Reliability of technology
Environment 1 Changes in business and organisational environment that
affected the project
People 2 Personnel shortfalls and personnel changes;
Leave of main technician
Vendor 8 Being involved with an intractable partner;
relationships Investing significant resources in a proposal that will not be
accepted;
Lack of communication;
Political and legal environment differences;
Exchange rate fluctuations;
Lack of customer/user’s capability in the project;
Lack of developer’s capability
Project Mgmt. 4 Poor systems of authority;
Lack of top management support;
High level of system complexity to be built;
Unrealistic schedules and budgets
Contract 1 Contract-related risks
Legal 1 Dispute and litigation
Total 28

47
4.3.9 Custom Software Acquisition Risks
Table 4.9 lists 11 risk categories of custom software acquisition. In this mapping study,
there are no risks found related to: integration and testing, maintenance, OTS products,
environment, systems engineering, and project management.

Table 4.9 Custom software acquisition risks mapped in this study

Risk category Total Custom software acquisition risks


mapped
papers
Planning 3 Lack of risk assessment at a very early stage of the
project;
Selecting the lowest price from candidate developer in
bidding process;
Insufficient feasibility assessment
Requirements 2 Poor requirement definition and analysis;
Requirement creep
Design 1 Architectural missmatch
System 1 Implement obstacle and stock
lifecycle
Project 1 Insufficient software project documentation for knowledge
closure management
Software 1 System is not developed according to the user requirements
Cost 2 Unexpected transition and management costs;
Cost escalation
People 1 Personnel shortfalls and personnel changes
Vendor 6 Dependency on external parties (vendor lock-in);
relationships Being involved with an intractable partner;
Credibility of developer’s capability’s evaluation result;
Service debasement;
Loss of organisational competencies
Contract 1 Unexpected costly contractual amendments
Legal 1 Dispute and litigation
Total 20

4.4 Survey on Risks of Off-The-Shelf-Based Software Development and


Acquisition
The mapping study identified risks from the literature. We complemented that with an
industry survey to investigate risks in practice. We performed a structured online
questionnaire survey to investigate risk occurrences. The survey focused on 11 process-
related risks of OTS-based custom software projects that may be relevant to both
software developers and acquirers (as listed in Table 4.10).

48
4.4.1 Survey Design
The questionnaire targeted developers and acquirers of completed OTS-based custom
software projects, using convenience sampling to identify potential respondents.
Convenience sampling is reasonable to use in an exploratory study [113]. The sample
population of the survey was 111 respondents which had a prior academic-industry
relationship with the thesis author. The population consisted of software acquirers
contracting OTS-based software to external software developers and software
organisations developing OTS-based software for their acquirers of different
organisations. Of the 111 respondents invited by e-mail, 69 (62%) completed the survey.
We posted the questionnaire online using Google Docs. The respondents comprised 35
software developers and 34 software acquirers of Indonesian background. Based on their
project descriptions, there were no respondents working on the same project. We
excluded OTS software vendors, who build OTS software, because we wanted to focus on
the software developers and acquirers. They are the two key stakeholders in software
development projects using and integrating OTS software.

We selected 11 process-related risks of OTS-based custom software projects from the


mapping study results (Table 4.6 and 4.7) that may be relevant to developers and
acquirers. The risks are shown in Table 4.10. Process-related risks are defined as “risks
that are related to the overall process, process elements and their interaction” [21:14].
We grouped the 11 risks into two categories of process-related risk: ‘selection and
integration’, and ‘maintenance’ risks, as in the previous study [27]. We grouped risks
related to planning, requirements, and integration and testing as ‘selection and
integration’ risks. OTS selection was selected as a process category because in OTS-
based software development, the definition of requirements and the planning process
take account of OTS selection results [41][40]. Maintenance and vendor relationships
were grouped as ‘maintenance’ risks. We decided to investigate OTS-based software
maintenance-related risks because the cost of OTS-based software maintenance equals
or exceeds that of developing custom software [114]. Also, OTS-based software
maintenance is considered to be more difficult than maintaining custom software [115].

The selected risks are derived from studies that can be classified as conceptual
framework [73][116][77][118][119], experience [120] and empirical research
[27][121][122][123][124]. Even though the risks included those derived from

49
conceptual framework and experience papers, we believe these risks may occur
frequently in real world.

As can be seen in Table 4.10, there are no studies justifying the following risks for
acquirers: not adaptable to requirement changes, requirements not negotiable, upgrade
unfeasible, lack of information on vendor and lack of support. Nonetheless, these risks
were selected because the acquirers may be affected by and have influence on these
risks [10][12][13][14].

Table 4.10 OTS-specific project risks relevant to the developer and acquirer

Process Risk ID Risk Developer Acquirer


R1 Selection effort ill-estimated [27][121] [116]
[122] [123]
R2 Not adaptable to requirement [27]
changes
R3 Requirements not negotiable [27][120]
Selection and [123]
integration R4 Complicated multi-OTS [73][77] [73]
components arrangement [116]
R5 Insufficient OTS component [118][124] [116]
documents
R6 Lack of OTS-driven requirements [73][77] [73]
engineering process [116]
R7 Maintenance planning unfeasible [27][123] [116]
R8 Upgrade unfeasible [27][77]
[123]
R9 Lack of information on vendor [27][120]
Maintenance [123]
R10 Lack of support [27][120]
[122][123]
R11 Reduced control of future [119] [73]
evolution of the system [116]
As can be seen in Table 4.10, this study includes 7 out of the 13 risks of OTS-based
software development seen in the study by Li et al. [27]: selection effort ill-estimated,
not adaptable to requirement changes, requirements not negotiable, maintenance
planning unfeasible, upgrade unfeasible, lack of information on vendor and lack of
support. The other six risks in their study [27] are specific to developers, for example,
difficulty to identify defect location.

Respondents were asked to answer which risks in Table 4.10 occurred frequently. By
‘frequently’, we meant to investigate the level of presence of risks in real word settings

50
based on 69 completed OTS-based custom software projects. The respondents could
choose one out of 6 alternative answers provided: ‘do not agree at all’, ‘hardly agree’,
‘agree somewhat’, ‘agree mostly’, ‘strongly agree’ and ‘do not know’. To analyse the
results, we coded ‘do not agree at all’ to 1 and ‘strongly agree’ to 5 (see the results in
Table 4.11).

4.4.2 Data Collection


The questionnaire collected data about risks of completed OTS-based custom software
projects. The developers’ completed projects represented various domains: IT sector
(15), banking or finance (8), public sector (8), e-commerce (3) and ERP (1). 6 different
projects originated from 3 companies. All the developer respondents came from well-
established companies, i.e.: 8 multi-national software developing companies, 1 service
provider, and the others are from medium and large software developing companies.
From 35 respondents, only one respondent came from a small company. The mean
number of permanent software developers in the projects is 7 and median is 4. The
mean number of part-time software developers involved in the projects is 4 and the
median is 3. The developer respondents had positions in the completed OTS-based
custom software projects as project manager (13), as developer (12), as project manager
and software architect (3), as software architect (3), as project manager, software
architect and developer (2), as project manager and developer (1), and as software
architect and developer (1). Only 1 developer respondent did not complete his/her
position. Almost all of the developer respondents held at least a bachelor’s degree in
computer science.

The software acquirer respondents came from telecommunications (11), government


(8), banking (3), automotive (3), university (3), plantation (1), insurance (1), private
investment (1), engineering consultant (1), oil and gas (1), and energy (1). From the
telecommunications, automotive, and university domains, we gathered more than 1
respondent working in different completed projects. The respondents representing the
acquirers had positions in the completed OTS-based custom software projects as project
manager (8), system analyst (5), user representation (5), IT architect (3), developer (3),
IT staff (2), domain expert (2), client team leader (1) and project steering committee (1).
Only 4 acquirer respondents did not mention their positions in the projects. Almost all
of the acquirer respondents held at least bachelor’s degree in computer science.
51
The average of percentage of OTS software used by the developer respondents in their
developed software was 43.36% and the acquirer respondents was 67.19% (in both
developers and acquirers, there were two respondents who did not answer the question
of percentage of OTS software usage). Each developer respondent was asked to
breakdown their OTS software usage based on the degree of OTS customization [47].
The average of the percentage breakdown of OTS software usage was: 32.60%
(configuration), 24.85% (integration), 23.79% (customisation) and 18.76%
(development).

4.4.3 Survey Results


Table 4.11 presents the median and mode of frequency of 11 risk occurrences relevant
to the developers and acquirers in the survey. Table 4.11 shows that 9 out of the 11 risks
occur frequently in software acquisition, but only 5 out of 11 risks occur frequently in
software development (R4, R5, R6, R10 and R11). Three risks, complicated multi-OTS
components arrangement (R4), lack of OTS-driven requirements engineering process
(R6) and reduced control of future evolution of the system (R11), occur frequently in
both software development and acquisition. Insufficient OTS component documents
(R5) and lack of support (R10) are two risks that frequently occurred in software
development but are infrequent in software acquisition.

Table 4.11 Risk occurrences of OTS-based software development and acquisition

Risk Development Acquisition Development Acquisition


ID Valid Missing Valid Missing Median Mode Median Mode
R1 34 1 33 1 2 1 3 3
R2 34 1 24 10 2.5 2 3 3
R3 32 3 32 2 2 2 3 3
R4 34 1 34 0 3 3 and 4 3 3
R5 34 1 34 0 3 3 2 2
R6 34 1 34 0 3 3 3 4
R7 34 1 24 10 2 2 3 3
R8 34 1 24 10 2.5 2 3 4
R9 33 2 34 0 2 2 3 3
R10 34 1 34 0 3 4 2 2
R11 33 2 34 0 3 2 and 3 3 4
Note: Valid: number of respondents answered the survey question on each risk;
Missing: number of respondents did not answer the survey question on each risk.

52
4.5 Discussion
In this section, we discuss answers to our research questions arising from the mapping
study and industry survey.

4.5.1 RQ2.1.1 What risks are related to OTS-based software development?


Of 17 final risk categories (listed in Table 4.5), OTS-based software development risks
consist of 13 risk categories (described in Table 4.6). Most risks identified in OTS-
based software development are OTS-specific risks. Design, legal, poor requirements
definition and analysis, underestimation of integration effort, undocumented system and
business processes, domain inadequacies, lack of skilled resources, cultural barriers and
accomplishing the development within time and budget constraints are risks found in
OTS-based software development that may occur either in custom or in-house software
development [30][57][58][57]. The generic risks, which are common to all software
projects, can be found in 13 risk categories of OTS-based software development, except
in OTS products.

4.5.2 RQ2.1.2 What risks are related to OTS-based software acquisition?


OTS-based software acquisition risks consist of 14 out of 17 final risk categories
(described in Table 4.7). Most risks found in OTS-based software acquisition have the
same concern as OTS-based software development. The same risks as OTS-based
software development risks are found in requirements, maintenance, OTS product, cost,
people, and integration and testing. There are three risks specific only for OTS-based
software acquisition: poor interoperability with legacy systems [73] (in integration and
testing category); and loss of bargaining power in vendor relationships and parties
shirking their agreed upon responsibilities (in vendor relationships category).

OTS-based software acquisition risks also have the same risks as custom software
development (unrealistic schedules and budgets, changes in business and
organisational environment that affected the project, and underestimated cost
estimation) and custom software acquisition risks (system is not developed according to
the user requirements). These findings indicate that OTS-based software acquisition
risks also have generic risks of software development project, as reported in these
studies [30][57][58][57]. The generic risks can be found in 14 risk categories of OTS-
based software acquisition, except in OTS products category.

53
4.5.3 RQ2.1.3 What risks are related to custom software development?
Custom software development risks include 11 risk categories (listed in Table 4.8).
There is no risk specific only for custom software development. Most risks found in
vendor relationships, project management, contract and legal are associated with either
contracted or outsourced software development. Contracted software development
needs to address these risks that do not exist in in-house software development.

4.5.4 RQ2.1.4 What risks are related to custom software acquisition?


Table 4.9 provides risks of custom software acquisition consisting of 11 risk categories
(out of 17 risk categories). All risks mapped to requirements, software, cost and people
are generic risks in software development projects [57][30][58][59]. The other risks
identified in custom software acquisition are specific to contracted/outsourced custom
software acquisition.

4.5.5 RQ2.1 What risks are related to OTS-based custom software projects?
Taken together, the results of RQ2.1.1 and RQ2.1.2 provide a checklist for identifying
OTS-specific risks, which similar as micro-processes [125][17]. A micro-process
describes the internal details and working of processes [125]. In addition, these results
corroborate the previous studies [72][73] by providing a checklist for identifying OTS-
specific risks for their frameworks of component-based systems. The mapping study
results align with simultaneous processes for OTS selection, system requirements and
system design [41][40] that may increase risks of OTS-based custom software projects.

In four development contexts under study, there are four common risk categories:
planning, requirement, people and vendor relationships. However, poor requirement
definition and analysis is the only generic risk found in all development contexts under
study. OTS-specific risks can be found in all risk categories. The results show that most
risks found in OTS-based software acquisition have the same concern as OTS-based
software development. In both custom software acquisition and development, specific
risks are related to contracted or outsourced software development projects. These
software project risks should be managed cooperatively between the software
acquisition and development [8].

54
These findings provide a better understanding of risk components of OTS-based custom
software projects. The results can serve as a checklist for risk identification [126] to
help to identify more risks [66], and to assess relevancy of risks [126][66].

4.5.6 RQ2.2 Which OTS-specific risks relevant to developers and acquirers occur
frequently?
There are three points of comparison between our survey results and a previous study of
OTS-based software development risk management [27]. Firstly, in this survey, six
infrequently occurring risks are consistent with risks found in the previous study (these
six risks are infrequent) [27]. Secondly, lack of support, which occurred frequently in
this study, is inconsistent with the previous study (this risk is infrequent) [27]. We
added four risks that were not investigated in the previous study [27], as follows:
complicated multi-OTS components arrangement [73][77], insufficient OTS component
documents [118][124], lack of OTS-driven requirements engineering process [73][77]
and reduced control of future evolution of the system [119]. Thus the previous study
[27] did not identify all potential risks; so there is a need to identify and analyse
additional risks [127].

As can be seen in Table 4.11, among the 11 risks under study, more risks occurred more
frequently in software acquisition compared to software development (nine vs. five
risks out of the 11 risks). We observe three kinds of difference between developers and
acquirers. Firstly, there are three risks that occurred frequently in both software
development and acquisition: complicated multi-OTS components arrangement (R4),
lack of OTS-driven requirements engineering process (R6) and reduced control of
future evolution of the system (R11). Secondly some risks occurred frequently in
software acquisition but not in software development: selection effort ill-estimated (R1),
not adaptable to requirement changes (R2), requirements not negotiable (R3),
maintenance planning unfeasible (R7), upgrade unfeasible (R8), and lack of
information on vendor (R9). Thirdly, insufficient OTS component documents (R5) and
lack of vendor technical support and training (R10) are two risks that frequently
occurred in software development but interestingly occurred infrequently in software
acquisition. This probably indicates that developers need more OTS software
documentations, and vendor support and training for integrating OTS software,
compared to acquirers.

55
4.6 Threats to Validity
In this section we discuss construct, internal and external validity for the mapping study
and survey. We followed a previous mapping study [110] to analyse threats to validity
of this mapping study.

4.6.1 Construct Validity


Construct validity is about the use of adequate definitions and measures of variables [90].
To validate constructs used in this mapping study, we focused on three aspects. The first
aspect is the set of keywords used in the search strings. We used the keyword ‘risk’ as
the part of our search strings because this keyword is specific and well-established. In
the mapping study, the synonym of OTS, acquisition and custom were used as the part
of search strings to expand the results of the query. Another aspect of the construct
validity is we searched the primary studies using five major digital library databases:
IEEE, ACM, Springer, Science Direct and Wiley Online. All of these major databases
cover variety and important related conferences and journals in software engineering.
Therefore, we expected to find sufficient literature to be analysed. The last aspect is we
attempted to define our initial classification to be robust enough for analysis. In order to
do this, we extended keywording of abstract [93] as a categorization scheme.
Keywording began by reading abstracts and identifying keywords reflecting topics
under investigation. When insufficient information was provided by the abstracts and
keywords, we also read the introduction and conclusion of the paper.

To further reduce construct validity problems we designed the survey based on a


previous study [27] with little modification (specific only to this study). We used seven
risks from the previous empirical study [27] and four additional risks from the literature,
as shown in Table 4.10. The questionnaire used in this study was reviewed by 3
internal experts and pre-tested using a paper version by 6 industrial respondents.

4.6.2 Internal Validity


Internal validity is concerned with whether research procedures, treatments or experiences
of the research participants influence the researcher's ability to draw correct inferences
from the data [90]. There are minimal threats of internal validity on this mapping study
because it only used descriptive statistics to analyse OTS-based risks.

56
Providing related information at the beginning of a survey is expected to give
background and context information for the respondents. In addition, this information
may act as an initial filter to ensure that the respondents have needed knowledge and
want to share his/her experience. There were fewer than 10 respondent inquiries before
and after completing the questionnaire. Furthermore, the computer science educational
backgrounds of almost all of the respondents increased confidence of this study for the
respondents in understanding the survey questions.

4.6.3 External Validity


External validity refers to the generalisability from this study [90]. The conclusions in
this mapping study are specific to OTS-based and custom software development and
acquisition risks, the external validity threats are not applicable to this study beyond the
scope of OTS-based custom software projects.

This survey study has a moderate sample size and was only conducted in Indonesia;
therefore it may not represent risks of OTS-based software development and acquisition
in general. However, there were a total of 69 respondents: 35 represent software
developers and 34 represent software acquirers. The respondents in the sample vary in
organisation sizes and acquirer/customer domains, which may reduce threats to external
validity.

4.7 Summary
In this chapter, we have presented the design and results of two studies of the risks of
OTS-based and custom software development and acquisition: a mapping study and a
survey. The main result of the mapping study is the identification of risks of OTS-based
and custom software development and acquisition. The secondary result of the mapping
study is the classification of risks of OTS-based and custom software development and
acquisition into 17 categories: planning, requirements, design, integration and testing,
system lifecycle, maintenance, project closure, software, OTS products, cost,
environment, people, systems engineering, vendor relationships, project management,
contract and legal. Consistent with previous studies [7][15], our results show that both
software acquisition is less studied than the software development.

Our mapping study identifies generic and specific risks in OTS-based and custom
software acquisition and development. In all four development contexts under study,

57
there are four common risk categories: planning, requirement, people and vendor
relationships. However, poor requirements definition and analysis is the only generic
risk found in all development contexts under study. In both custom software acquisition
and development, specific risks are related to contracted or outsourced software
development projects. The results also show that most risks found in OTS-based
software acquisition have the same concern as OTS-based software development.
Technical-related risks are found less often in acquisition and project management
related risks are found less often in development. Identifying risk components of OTS-
based custom software is a first step in risk management, not only for software
developers but also for software acquirers. Software project risks should be managed
cooperatively between the software acquirers and developers [8].

Drawing on the mapping study results, we empirically surveyed the occurrence of 11


OTS-specific risks that may be relevant to both the developers and acquirers using a
structured online questionnaire. The survey is a partial replication of a study of risks of
OTS-based software development [27], complemented with an investigation of the
acquirer perspective. This study includes 7 out of the 13 risks of OTS-based software
development seen in the study by Li et al. [27], and excludes the other six risks because
they are specific to developers, for example, difficulty to identify defect location. The
questionnaire collected data on risks of completed OTS-based custom software projects
from 35 software developers and 34 acquirers of Indonesian background. This study
partially confirms the previous study [27] because the risk of lack of support is
inconsistent with the previous study [27]. This study complements the previous study
[27] with new findings about the acquirer perspective.

The results show that more risks occurred more frequently in software acquisition than
the software development. A possible explanation is that acquirers may be more
affected by or have more ability to mitigate these risks. Insufficient OTS component
documents and lack of vendor technical support and training are two risks that
frequently occurred in software development (but not in software acquisition). A
possible explanation for this might be that the limited ability of developers to control
these risks [16][59][128]. These findings provide a better understanding of risks in
OTS-based custom software projects from the developer and acquirer perspectives.

58
Some limitations are worth noting. Although we used five major software-related digital
libraries, relevant primary studies may be present in other digital libraries. The survey
population is Indonesian respondents only, which may limit its external validity.

59
5 Differences in Risk Perceptions

This chapter reports on a study of risk perceptions of developers and acquirers in OTS-
based custom software projects. The study used an online questionnaire-based survey
and compared stakeholders’ perceptions about risk control over and exposure to 11
risks. This chapter addresses RQ3 of this thesis and has two research questions as
follows, within the context of OTS-based custom software projects.
RQ3.1: How are OTS-specific risks perceived by developers and acquirers?
RQ3.2: How should risk responsibilities be defined between developers and acquirers?
This chapter analyses the results of the survey, used in the previous chapter.

The structure of this chapter is as follows. Firstly, we briefly introduce the background to
the study, and describe the overall research design. Then, we describe a method to analyse
the survey data about which stakeholder is affected by and can control each risk. Finally,
we discuss the results and threats to validity, and presents conclusions.

5.1 Introduction
The risks of OTS-based software development and acquisition have been introduced in
the previous chapter. This chapter further analyses the differences in risk perceptions of
developers and acquirer using stakeholder analysis.

Risks associated with a software project affect all stakeholders [9][7][8][10]. Previous
studies have reported that stakeholders tend to perceive the importance of certain risks as
higher than others if they cannot control the risks, and also that different stakeholders tend
to identify risk being caused by other stakeholders [128][16][59]. As different
stakeholders perceive risk differently [16], therefore there are different perceptions of
stakeholder’s responsibility for risks. Stakeholder perceptions vary based on the
individual's or organisation's background and structure, experience, needs and
expectations [18][19]. To manage risks effectively, it is important to involve stakeholders
[22][8], to take account of differences in risk perceptions, and to identify stakeholder
responsibility for risks [22].

This chapter focuses on investigating risks relevant to developers and acquirers, and in
particular differences in the perception of risks by the developers and acquirers. Another
objective of this chapter is to propose risk responsibility based on stakeholder analysis.
60
One approach that accounts for different stakeholder involvement in a project is
stakeholder analysis [25][12][26][14][13][10]. Stakeholder analysis is typically used for
issues such as stakeholder identification, identification of area of interest, stakeholder
contribution and expectation, stakeholder influence, strategy to involve stakeholder and
stakeholder responsibility [25][26].

5.2 Research Design


We performed a structured online questionnaire survey to investigate different
perceptions of risks of OTS-based custom software projects (listed in Table 5.1) from
developers and acquirers. The results of the survey of the previous chapter were used in
this study. We focused on 11 process-related risks relevant to developers and acquirers
in OTS-based custom software projects. These are the same risks as the survey in the
previous chapter (detailed in Section 4.4.1 and in Table 4.10). Furthermore, we
developed a method to analyse the survey data to clarify the nature of the differences
between perceived risks.

Table 5.1 11 OTS-specific project risks relevant to the developer and acquirer
respondents

Process Risk
ID Item
Selection and R1 Selection effort ill-estimated
integration R2 Not adaptable to requirement changes
R3 Requirements not negotiable
R4 Complicated multi-OTS components arrangement
R5 Insufficient OTS component documents
R6 Lack of OTS-driven requirements engineering process
Maintenance R7 Maintenance planning unfeasible
R8 Upgrade unfeasible
R9 Lack of information on vendor
R10 Lack of support
R11 Reduced control of future evolution of the system

5.2.1 Survey Design


This chapter used the same survey as in the previous chapter, and so more details of the
survey design are provided in Section 4.4.1. We performed a structured online
questionnaire survey based on a definition of stakeholders [12][14][13][10] and
developed a method based on stakeholder analysis [25][26] to analyse the survey data of
differences in OTS-specific risks perceived by software developer and acquirer
61
respondents. This method is illustrated in Figure 5.1. The sample population of the
survey was 111 respondents which had a prior academic-industry relationship with the
thesis author. The respondents comprised 35 software developers and 34 software
acquirers of Indonesian background. The software developers and acquirers are in
different organisations whose backgrounds, experiences, needs and expectations are
different [18].

In our survey we asked about 11 process-related risks that should be relevant not only to
developers but also to acquirers in OTS-based custom software projects. These are
described in Table 5.1. Respondents were asked who was affected by and who could
influence these risks. The respondents could choose one out of four options:
‘developer’, ‘acquirer’, ‘both’ or ‘don’t know’.

5.2.2 Method for Analysing Process-Related Risk Perspectives


The analysis method compares perceptions of the respondents about which stakeholder
is affected by and can control risks. The method extends the stakeholder analysis
approach [25][26], and is illustrated in Figure 5.1. The method uses a template to
compare risk perceptions, as summarized in Table 5.2. As illustrated in Figure 5.1, the
respondents (software developers and acquirers) completed the survey by choosing the
kind of stakeholder (developer, acquirer, both or don’t know) impacted by and can
control each risk. All respondents' responses were mapped to step 1 and 2 in a template,
shown in Table 5.2. Step 3 will map step 1 and 2 into risk control/impact matrix. Step 4
will compare risks using the risk control/impact matrix. The detailed steps in using the
template will be described in this section. Figure 5.2 models the constructs used in the
method to analyse the survey.

62
Figure 5.1 A method for analysing differences in off-the-shelf risks between the
developer and acquirer used in the survey

Figure 5.2 Relationship model of respondent, risk, risk control and impact to stakeholder

63
The method consists of four steps as in the template and one additional step to propose
risk responsibility as follows.
 Step 1/‘Risks impacting stakeholders’ counts the number of each kind of
stakeholder affected by each risk.
 Step 2/‘Risks stakeholders can control’ counts the number of each kind of
stakeholder who can control each risk.
 Step 3/‘Mapping stakeholders from step 1 and 2 into risk control/impact matrix’,
maps the number of each kind of stakeholders from step 1 and 2 into a risk
control/impact matrix (Figure 5.3). This matrix is adapted from a power/interest
matrix [79][33][80] (described in Section 2.3 and depicted in Figure 2.1). As this
survey focused to investigate different risk perceptions between developers and
acquirers, the mapping process was only conducted for two kinds of stakeholders
from step 1 and 2, developer and acquirer. Therefore if a respondent answering
‘both’ to the survey question then they are included as both a developer and an
acquirer. The mapping process is performed separately for developer and acquirer
respondents.

Table 5.2 A template to compare risk perceptions of OTS-based custom software


project

Risks
Stakeholder
R1 R2 ... R11
Step 1 Developer
Risks impacting Acquirer
stakeholders Both
Don’t know
Step 2 Developer
Risks
Acquirer
stakeholder can
Both
control
Don’t know
Step 3 Mapping
Developer
stakeholders
from step 1 and
2 into risk
control/ impact Acquirer
matrix (see
Figure 5.3)
Step 4 Comparing risks using the risk
control/impact matrix

64
High Keep Key player
Risk control satisfied (B) (D)
(Risk stakeholder can control)
Low Minimal Keep
effort (A) informed (C)
Low High
Risk impact
(Risks impacting stakeholder)
Figure 5.3 Risk control/impact matrix, adapted from power/interest matrix [79][33][80]

In order to map the number of each kind of stakeholder (developer and acquirer) from
step 1 and 2 into the risk control/impact matrix, a center point coordinate of the matrix
has first to be defined for each risk under investigation. The horizontal center is half
of the number of respondents answering step 1 (risk impact). The vertical center is
half of the number of respondents answering step 2 (risk control). The following
example demonstrates the mapping process of the developer respondents’ risk
perception about themselves. For example, for the risk of not being adaptable to
requirement changes (R2) in Figure 5.4, the total number of developer respondents
answering step 1 is 34 and step 2 is 35. Hence, the center point coordinate of the risk
control/ impact matrix is (17, 17.5). The number of the developers responding that
they are impacted by (step 1) and can control (step 2) the risk is 13 and 21
respectively. The number of the developers saying that both stakeholders are impacted
by (step 1) and can control (step 2) the risk is 20 and 11 respectively. Therefore, the
total number of the developer respondents that perceive themselves as impacted by
the risk is 33 and that can control risk is 32. So for the risk of not being adaptable to
requirement changes (R2), developers’ perceptions about themselves can be mapped
into high risk control and impact (Key player (D)), because they are greater than their
center points.

 Step 4/‘Comparing risks using the risk control/impact matrix’ compares mapped risk
control/impact matrix result between developers and acquirers. This comparison is
performed separately for the developer and acquirer respondents. For example, as can
be seen in Table 5.3, the risk of reduced control of future evolution of the system
(R11) of the developer respondents have developers mapped into Key player (D) and
acquirers mapped into Keep informed (C). These are compared (D:C). The

65
comparison shows that developer respondents perceive themselves to have more
control over the risk than their acquirers.

 Step 5/‘Identify stakeholder agreement on and reconcile differences in risk


perceptions between the inputs of developer and acquirer respondents’. This step aims
to propose a risk responsibility to mitigate a risk based on a stakeholder agreement
about which stakeholder has high risk control. A stakeholder who has high risk
control can be considered to be responsible for a risk because to mitigate a risk, high
influence and power are needed to control key decisions and task implementation
[31][32][33]. In addition, a responsibility should be assigned to a stakeholder who
has the competence and capacity needed to discharge the responsibility [34][35].
Competence and knowledge affect the choice of control [82], while resource and
capacity were needed to perform control over risks. Moreover, the consideration
becomes stronger if both stakeholders also agree on high risk impact.

After the responses of developer and acquirer respondents are mapped to and
compared using the template (as illustrated in Figure 5.1), the completed templates are
then compared to identify stakeholder agreement about risk perceptions. If there are
any differences in risk perceptions between the developer and acquirer respondents,
stakeholder agreement will be decided from partial agreement between both
stakeholder perceptions. The following examples demonstrate partial agreement and
reconciliations of different risk perceptions. The acquirer respondents in Table 5.3
perceive the risk of reduced control of future evolution of the system (R11) for
developers as Keep satisfied (B) and for acquirers as Keep informed (D). To identify
stakeholder agreement about which stakeholder has high risk control, the risk
perceptions of the developer and acquirer respondents to themselves and their
stakeholders are compared (D:C for the developer respondents and B:D for the
acquirer respondents). This comparison reveals that the developer respondents
perceived themselves to have higher risk control compared to their acquirers; while
the acquirer respondents perceived both stakeholders to have high risk control. There
is a partial agreement that the developers are perceived to best control the risk (R11)
because the acquirers perceived themselves and also the developers to have high risk
control. To reconcile this difference, it can be decided that the developers (as
indicated in Part 2 of Table 5.3) can best control the risk of reduced control of future
evolution of the system (R11). Agreement on this might be used as a rationale to
66
assign risk responsibility to the developer. The same procedure is used to reconcile
different risk impact perceptions between developers (both stakeholders are most
impacted by risk R11) and acquirers (the acquirer is most impacted by risk R11). Both
respondents agree that the acquirer has a higher risk impact.

All of the previous steps has been applied and completed for all risks in Table 5.1.

5.3 Collected Data


Data collection is described in the previous chapter (Section 4.4.2) because the survey
was used in this chapter and also the previous chapter.

5.4 Results
This section presents results of the survey organized by the comparison method. Figure
5.4 and 5.5 present the questionnaire results mapped to step 1 and 2 of the method
template (see Table 5.2). The results are organized as comparisons of the risk
control/impact matrix between kinds of stakeholder (developers and acquirers)
separately for each survey respondents (the developer and acquirer respondents) in Part
1 of Table 5.3.

67
Step 1: Risks impacting stakeholder
30

25 24

20
20 19 19
18
17 17
16 16
15 15
15 14
13 13
12 12
11
10 10
10 9 9
8 8
7 7
6
5
5 4 4 4 4
3 3
2 2 2 2
1 1 1 1 1
0 0
0
R1 R2 R3 R4 R5 R6 R7 R8 R9 R10 R11

Developer Acquirer Both Unknown

Step 2: Stakeholder can control risks


35
30
30 27
25 24
25 23
21 21 20
20 17 18

15 13 13
11 10 10
88 9
10 7 7
5 6 54
3 3 4 4 3 2 343 43
5 2 2
0 0 0 1 1 1 0
0
R1 R2 R3 R4 R5 R6 R7 R8 R9 R10 R11

Developer Acquirer Both Unknown

Figure 5.4 The developer respondents’ responses mapped to step 1 and 2 of the template
(35 respondents)

68
Step 1: Risks impacting stakeholder
25

20
20
16
15 14 14 14 14 14
13
12 12 12 12
11 11 11
10 10 10 10
10 9
8 8 8 8 888 8
7
6 6 6 6
5 5
5 4
3
2 2 2 2 2 2
1
0
R1 R2 R3 R4 R5 R6 R7 R8 R9 R10 R11

Developer Acquirer Both Unknown

Step 2: Stakeholder can control risks


18 17
16 16
16
14 13 13 13 13 13
12 12 12 12 12 1212
12 11 11 11 11 11
10
10 9 9
8 8 8
8
6 6 6 6 6
6 5 5
4 4
4 3 3 3
2 2 2 2 2 2
2
0
R1 R2 R3 R4 R5 R6 R7 R8 R9 R10 R11

Developer Acquirer Both Unknown

Figure 5.5 The acquirer respondents’ responses mapped to step 1 and 2 of the template
(34 respondents)
Part 1 of Table 5.3 shows the results of step 4 in the method template (Table 5.2). Part 2
of Table 5.3 shows stakeholder agreement on risk control and impact, and risk
responsibility following step 5 in the developed method. Table 5.4 summarizes patterns of
the comparisons from Part 1 of Table 5.3.

It can be seen from Table 5.3 and 5.4, almost all the developer respondents perceive
themselves to have higher risk control compared to their acquirers. There are 3 kinds of
differences. The first is D:D, which in Table 5.3 is the risk of requirements not
69
negotiable (R3), where the developer respondents perceive themselves and their
acquirers to have the same high risk control and impact. For the second kind, developers
perceive themselves to have higher risk control compared to their acquirers, but
perceive themselves to have equal risk impact as their acquirers (D:C in Table 5.3 and
5.4). Risks in this group are selection effort ill-estimated (R1), not adaptable to
requirement changes (R2), complicated multi-OTS components arrangement (R4), and
maintenance planning unfeasible (R7) and reduced control of future evolution of the
system (R11). For the last kind, the developer respondents perceive themselves to have
higher risk control and impact compared to their acquirers (D:A in Table 5.3 and 5.4),
for the risks: insufficient OTS component documents (R5), lack of OTS-driven
requirements engineering process (R6), upgrade unfeasible (R8), lack of information on
vendor (R9) and lack of support (R10).

Table 5.3 Analysing of different risk perceptions


Part 1
Comparison of risk mapped into the risk control/impact matrix (developer:acquirer)
from respondent perspectives
Risk (developer:acquirer)
Respondents R1 R2 R3 R4 R5 R6 R7 R8 R9 R10 R11
Developer D:C D:C D:D D:C D:A D:A D:C D:A D:A D:A D:C
Acquirer D:D D:C D:D D:D D:C D:D D:C D:C D:C B:C B:D
Part 2
Agreement of responses of stakeholders on risk control and impact, and risk responsibility
R1 R2 R3 R4 R5 R6 R7 R8 R9 R10 R11
Stakeholder
agreement on
risk control Dev Dev Both Dev Dev Dev Dev Dev Dev Dev Dev
Stakeholder
agreement on
risk impact Both Both Both Both Dev Dev Both Dev Dev - Acq
Risk
responsibility Dev Dev Both Dev Dev Dev Dev Dev Dev Dev Dev

Table 5.4 Summary of comparison of risks mapped into the risk control/impact matrix
between the developer and acquirer respondents (summarized from Table 5.3)
Patterns of comparison of risk mapped into the
risk control/impact matrix (developer:acquirer)
from respondent perspectives
D:D D:C D:A B:C B:D
Respondent Developer 1 5 5
perspective Acquirer 4 5 1 1

70
For acquirers, there are 4 kinds of differences. For the first kind, acquirer respondents
perceive the following risks to be under high control and have high impact for
themselves and their developers (D:D): selection effort ill-estimated (R1), requirements
not negotiable (R3), complicated multi-OTS components arrangement (R4) and lack of
OTS-driven requirements (R6). For the second kind, acquirers perceive their developers
to have higher risk control, and perceive themselves and their developer to have higher
risk impact (D:C) for the following risks: not adaptable to requirement changes (R2),
insufficient OTS component documents (R5), maintenance planning unfeasible (R7),
upgrade unfeasible (R8) and lack of information on vendor (R9). For the third kind,
acquirers perceive themselves to have higher risk impact but lower risk control
compared to their developers (B:C), for the risk of lack of support (R11). Finally,
acquirers perceive themselves to have higher risk impact than, but equal risk control to
their developer (B:D) for the risk of reduced control of future evolution of the system
(R11).

Part 2 of Table 5.3 shows agreements of responses of stakeholders on risk control and
impact, including reconciliation results of different risk perceptions between the
developer and acquirer respondents on the risks of R1, R4, R5, R6, R8, R9, and R11.
With regard to risks R1, R4, R6 and R11, the developer respondents perceived
themselves to have higher risk control than their acquirers, and the acquirer respondents
perceived both stakeholders to have high risk control and impact (for R1, R4 and R6) or
to have high risk control (for R11). The differences between the stakeholders can be
reconciled by agreeing that the developer has a high level of risk control. For the risks
R5, R6, R8 and R9, the developer respondents perceived only themselves as most
impacted by these risks (D:A in developer respondents row in Part 1 of Table 5.3), and
the acquirer respondents perceived both stakeholders as most impacted by these risks
(developer and acquirer of the mapped risk control/impact matrix is either C or D in
Part 1 Table 5.3). To reconcile these differences, it could be decided that the developers
are highly impacted by these risks (presented in Part 2 of Table 5.3). For risk R11, the
responses of both stakeholders agree that the acquirer is most impacted by the risk
(reduced control of future evolution of the system). This agreement is decided by
reconciling different risk perceptions between the developer (both stakeholders are most
impacted by risk R11) and acquirer (the acquirer is most impacted by risk R11). For the
risk of lack of support (R10), there was disagreement about who is most impacted by
71
the risk (developers perceive themselves to have higher risk control and impact
compared to their acquirer (D:A), while acquirers perceive themselves to have higher
risk impact compared to their developers (B:C).

5.5 Discussion

5.5.1 RQ3.1: How are OTS-specific risks perceived by the developer and
acquirer?
Although a previous study investigated differences in perceptions of IT project risks
between users and project managers [16], these risks were not specific to OTS-based
custom software projects. Our study focused on OTS-specific project risks relevant to
the developer and acquirer, as listed in Table 5.1. In this section, we compared risk
perceptions in a risk control/impact matrix. This comparison focused on perceptions of
risk control and impact by developers and acquirers.

As indicated in Part 2 of Table 5.3, almost all stakeholders agree that the developer can
best control risks. There is only one risk, R3 (requirement not negotiable), where both
stakeholders each claim to best control this risk.

The developer respondents have two different kinds of perceptions about which
stakeholder is most impacted by risks. The first, developers perceive that both
stakeholders are most impacted by risks (as shown in Part 1 of Table 5.3, for the risks
R1, R2, R3, R4, R7 and R11). The second kind, developers perceive themselves to be
most impacted by risks (as shown in Part 1 of Table 5.3, for the risks R5, R6, R8, R9
and R10).

The acquirer respondents perceive both stakeholders to be most impacted by risks,


except for their developers in R10 and R11. In regard to risk of lack of support (R10)
and reduced control of future evolution of the system (R11), acquirers perceive
themselves to have higher risk impact than their developers.

Comparing the developer and acquirer respondents’ perspectives about which


stakeholders are most impacted by risks, it could be concluded that for the risks of R1,
R2, R4 and R7, both stakeholders agree that they are both highly impacted by the risks.
For the risks of R5, R6, R8 and R9, the developers are perceived to be most impacted
by these risks (see Part 2 of Table 5.3). For the risk of lack of support (R10), there was
72
disagreement about who is most impacted by the risk. With regard to the risk of reduced
control of future evolution of the system (R11), both stakeholders agree that the acquirer
is most impacted. The comparison method used in this chapter provided a systematic
and practical approach to analyse differences in the perceptions of risks between
developers and acquirers.

5.5.2 RQ3.2: How to define risk responsibility between the developers and
acquirers?
As can be seen in Table 5.3, for only 3 out of 11 risks investigated, (R2, R3, R7) do
both the developers and acquirers have the same perceptions. For the other risks, it is
interesting to analyse the differences.

Understanding these differences in perceptions about which stakeholder has high risk
control is helpful to propose risk responsibility. High influence and power is needed to
control key decisions and task implementation [31][32][33] to mitigate risks. This
study provided a structured method to compare risk control and impact to identify
stakeholder agreement on and reconcile differences in risk perceptions between
developer and acquirer respondents. Ultimately, the outputs of the method are proposals
for the assignment of risk responsibilities, based on stakeholder agreement on which
stakeholder has high risk control. This is consistent with responsibility models
[81][78][34][35], which propose responsibility belongs to stakeholders who have the
competence and capacity needed to discharge the responsibility [34][35]. Knowing the
responsibility could subsequently lead to the identification of roles [81][78] [34][35] to
mitigate risks.

As shown in Part 2 of Table 5.3, for all risks except requirements not negotiable (R3),
the developer would be considered to be responsible for the risks. The developer should
inform, consult and involve the acquirer because the acquirer is also impacted by the
risks [33]. For the risk of requirements not negotiable (R3), both stakeholders should
collaborate to discharge the responsibility [34] of risk mitigation.

5.6 Threats to Validity


As this study used the same survey as the previous chapter, validity threats of this study
are largely the same as in the previous chapter. However, instead of using questions
adapted from Li et al. study [27], to prevent construct validity problems, we derived the

73
questionnaire questions from a stakeholder definition, which is anyone who are affected
by or can influence the system under development [12][14][13][10]. The use of
stakeholder definition as the questionnaire questions to ensure adequate definitions and
measures of variables [90].

5.7 Summary
This study attempted to analyse differences in shared risks [8] of OTS-based custom
software projects by comparing perceptions of risk control and impact of two main
stakeholders, the developers and acquirers. We performed an online questionnaire-based
survey about OTS-based custom software project risks on Indonesian software
developers and acquirers. In the survey, we asked who was affected by and who could
influence 11 risks of OTS-based custom software projects relevant to both developers
and acquirers. To analyse the survey results, we developed and applied a method for
analysing differences in perceptions of OTS-specific risks based on stakeholder analysis
[25][12] [26][14] [13][10].

The findings show some differences in risk perception. Both stakeholders can control,
and are most impacted by, risks about requirements negotiation. Developer respondents
perceived themselves to best control risks, but perceived either themselves or both
stakeholders to be most impacted by risks. Acquirer respondents agreed that their
developers can best control risks, but perceived both stakeholders as most impacted by
risks, except for risks of R10 and R11. For the risk of lack of vendor technical support
and training (R10), there was disagreement about who is most impacted by the risk
(usually each stakeholder reported themselves). With regard to the risk of reduced
control of future evolution of the system (R11), both stakeholders agreed that the
acquirer is most impacted.

The comparison method was developed to analyse the survey results. We used the
method to propose a default position about which stakeholder should be responsible for
risks based on stakeholders’ agreement about which stakeholders have high control and
power over each risk [31][32][33]. The results show that for all risks (except
requirements not negotiable) the responsibility can be proposed to be the developers’.
With regard to requirements not negotiable, we proposed both stakeholders had
responsibility of risk.

74
Even though we did not investigate the developer and acquirer’s actual perspectives on
risks, a study of risk perception is an important approach to understand differences of
actual risks between the developers and acquirers. The comparison method analysed
respondents’ responses about their risk control and impact, without discussing their
responses to their stakeholders. Overall, the comparison method in this study added
substantially to our understanding to provide a method-based consideration to analyse
different risk perceptions [16].

75
6 Comparison of Process-related Risk Perceptions between
Developers and Acquirers

This chapter discusses a multi-case study to analyse differences in the perception of


process-related risks between developers and acquirers in OTS-based custom software
projects. The aim of this chapter is to develop further understanding about stakeholders’
perceptions of their level of risk control and impact. This chapter complements Chapter
5 in addressing RQ3 of this thesis. This study has two main research questions as
follows.
RQ3.3: How are OTS-specific risks perceived by developers and acquirers?
RQ3.4: How should risk responsibilities be assigned between developers and
acquirers for these risks?

The first section introduces this chapter. The next section describes a proposed
framework, which was modified from a method in the previous chapter, to collect,
compare and analyse the case study data. Then, the case study design is detailed.
Section 6.4 presents results and findings of this study. Section 6.5 discusses answers to
the research questions. Finally we assess threats to validity and conclude.

6.1 Introduction
To manage risks effectively, it is important to identify the risks involved [16], to
involve stakeholders [22][8] aiming to take account of differences in risk perceptions
and to identify stakeholder responsibility for risks [22]. Even though Keil et al. [16]
pointed out the importance of reconciling user and project manager perceptions of IT
project risks and compared two stakeholders’ perceptions, their study did not provide
detailed steps for reconciling risks. A recent study has highlighted the importance of
reconciling perspective mismatch in managing software processes [20]. In their study,
Adolph et al. [20] introduced two stages for reconciling perspectives: converging and
validating. In this thesis, we analyse differences in the perception of process-related
risks between the developers and acquirers in OTS-based custom software projects
using a multi-case study.

We investigated differences in the perception of 11 process-related risks. Six of these 11


risks relate to the process of OTS selection and integration: selection effort ill-estimated,

76
not adaptable to requirement changes, requirements not negotiable, complicated multi-
OTS components arrangement, insufficient OTS component documents and lack of OTS-
driven requirements engineering process. Five are maintenance-related risks:
maintenance planning unfeasible, upgrade unfeasible, lack of information on vendor,
lack of support and reduced control of future evolution of the system. Section 4.4.1
provides a more detailed explanation of these 11 risks. Although focusing on the 11
risks could limit the study’s generalisability, we believe that it can help increase our
understanding to better assess risks through detailed analysing of differences in risk
perceptions between developers and acquirers.

Based on the above research questions and access to resources [85][86], which in this
study were case study participants, a case study was selected as the research method.
Using a case study was expected to provide a deeper understanding of the phenomena
under study [86][85][129] by involving stakeholders in risk management and analysing
differences in risk perceptions occurring in real contexts.

6.2 Proposed Framework


This study used a framework, based on the method developed in the previous chapter, to
collect, analyse, compare and discuss case study data about perceived risks. In the
previous chapter, we analysed survey data of process-related risk perceptions about
which stakeholder is affected by and can control risks. This study extended that method
to make a proposed framework that can be used by developers and acquires to discuss
disagreement regarding risk responsibilities. Table 6.1 describe the three changes made
to the method used in Chapter 5 to create the framework used in this chapter.

Table 6.1 Differences between the case study framework and the method used in
Chapter 5

Difference The Chapter 5 method The Chapter 6 framework


Input Who was most impacted by How was your level of risk control
these risks and who could and risk impact? (low/high)
best control these risks?
(developer/acquirer/both/
don’t know)
Decision on risk Risk responsibility is Risk responsibility is defined based
responsibility defined based on agreement on agreement of developer and
of respondents’ inputs on acquirer on which stakeholder has
which stakeholder has high higher risk control
risk control
77
Decision on The previous method does Dialogue, deliberation and
disagreement about not provide a solution communication in the framework if
risk responsibility developer and acquirer disagree
with a proposed risk responsibility

This framework analyses differences in perceived risks that should be relevant to both
developers and acquirers of OTS-based custom software. The risks are described in
Section 4.4.1 and in Table 4.10. The framework extends the stakeholder analysis
approach [10][12][13][25][26], and is illustrated in Figure 6.1. The framework
compares level of stakeholder perceptions about risk control and impact (low/high),
which are both framework inputs. Framework outputs are stakeholder involvement and
risk priority (for each stakeholder). Responsibility for a risk is proposed to belong to a
stakeholder who has higher risk control. These outputs and risk responsibilities are then
raised with the developer and acquirer for their agreement. The framework is not a risk
identification method; stakeholders must provide a list of risks to be analysed using the
framework.

Figure 6.1 The proposed framework

The framework consists of five steps:


1. Each stakeholder provides inputs to the framework, which are levels of risk control
and impact (low/high) for each risk being analysed.

78
2. The framework inputs are then mapped into a risk impact/control matrix (Figure
6.2), as in the previous chapter [79][33][80] (see Figure 5.3). The mapping process
output is a value for each risk: inform (A), involve (B), consult (C) and collaborate
(D). Each value indicates the kind of stakeholder involvement. The matrix uses an
ordinal scale. Instead of using risk-exposure quantity [30] or multiplication on
ordinal scale [38], the framework prioritises risk using available information [38] by
comparing the values of two quadrants. Cells above or right have higher risk
priority. Quadrant A has low risk priority. Quadrant B and C have medium risk
priority, and quadrant D has high risk priority. However, risks in quadrant B and C
cannot be compared [69][130].

High Involve (B) Collaborate (D)


Risk control
(Risk stakeholder can control) Low Inform (A) Consult (C)
Low High
Risk impact (risks impacting stakeholder)
Figure 6.2 Risk control/impact matrix with stakeholder involvement
3. The next step is to compare the outputs of the mapping process above between a
developer and acquirer. The comparison outputs indicate the kind of stakeholder
involvement for each stakeholder and a risk priority. Responsibility for each risk is
proposed to belong to the stakeholder who has higher risk control. This is because
high influence and power over risk is needed to control key decisions and task
implementation [31][32][33]. If both stakeholders have same level of risk control,
the responsibility is proposed to belong to both stakeholders. All of the framework’s
outputs and proposed risk responsibilities are raised for discussion between both
stakeholders for agreement.
4. Stakeholders jointly determine if they agree on the proposed risk responsibilities.
The comparison in the framework facilitates discussion between stakeholders by
obtaining their opinions and further understanding of their perceptions [24]. If both
stakeholders agree then the risk responsibilities are assigned to a stakeholder who
has higher risk control and the framework terminates. Otherwise it continues to the
next step.
5. This last optional step is dialogue, deliberation and communication to manage risk.
If both stakeholders do not agree with the previous step, they can discuss the

79
proposed risk responsibility by considering previous results. The discussion process
should skip areas of agreement, which are stakeholder-owned perspectives (inputs
and outputs), and focus on points of difference, which are proposed risk
responsibilities.
Although the framework was designed to be used by a developer and acquirer, to avoid
the effects of learning to use the framework [131], case study participants were only
asked to complete the framework inputs and to provide agreement on the framework
outputs and proposed risk responsibilities (these are detailed in following sections).

To increase validity, a new proposed framework should be evaluated [131] and then
improved [129] using retrospective data [132]. The framework was initially evaluated
using a set of evaluation criteria for risk assessment [133][134] (the results in Section
6.4.2 and discussed in Section 6.5.3), and using the results and findings of this case
study. Finally, the framework was improved [129] by comparing [135] existing risk
management practices with the framework, and making use suggestions from all
participants.

6.3 Case Study Design


The selection of case studies as a research method can depend on the research questions
and also access to resources [86][85]. Yin defines case study as “an empirical inquiry
that investigates a contemporary phenomenon within its real-life context, especially
when the boundaries between phenomenon and context are not clearly evident ”
[86:18]. Case studies can also be suitable when researchers have little control over
events [85][86]. We selected case studies as a research method for all of these reasons.

The case study followed existing guidelines [86][131][132][129]. The structure of the
case study is outlined below.
1. Case study design
The case study design comprises:
a. Objectives of the case study
b. Case study questions to guide data collection
c. Theoretical prepositions as term of reference to conduct and review the
study
d. The case and unit of analysis selection
e. Methods to collect data
80
2. Preparation for data collection:
Procedures and protocols for data collection must be defined.
3. Collecting evidence
4. Analysis of collected data
5. Reporting
The following section describes the case study according to this structure.

6.3.1 Objective of the Case Study


The objective of the case study was to develop a deeper understanding of the
phenomena under study [86][85][129]: stakeholder perspectives in risk management in
a real world context.

6.3.2 Questions of the Case Study


There were two stages of data collection in this study, as described in Section 6.3.6. The
first stage was a questionnaire-based survey to identify existing stakeholder practices in
risk management and to analyse differences in risk perceptions. This provided input
data to the framework. The second stage was a structured interview to evaluate and
analyse differences in risk perceptions. The questions for each stage are presented
below.
1. First stage
There were three parts of the questionnaire.
Q1.1 (Part 1 of the questionnaire)
The questions collected project information about project members, OTS software used
in the developed system, type of OTS integration method, type of developed system,
and type of acquirer domain. Other important questions were about risk perceptions,
collected from stakeholders in this study.

Q1.2 (Part 2 of the questionnaire)


Q1.2.1 How important to you is it to involve stakeholders in risk management, and
why?
Q1.2.2 How do you understand the differences in risk perceptions between
stakeholders, and how important is it to analyse these differences?
Q1.2.3 Who has risk responsibility for this project?
Q1.2.4 What are your roles in the project risk management?

81
Q1.2.5 How was your level of risk control and risk impact for each of these risks?
(low/high)
Q1.3 (Part 3 of the questionnaire)
The questions collected participant information: participant position in the project (unit
of analysis), participant position in their organisation, experience in software
development projects (in years), experience of using OTS in projects (in years), number
of OTS-based software development projects involved with, educational background
(degree), software engineering/computer science/information system/information
technology/telematics related education, business ownership, industry domain (IT
industry, telecommunication service provider, software vendor/producer, IT
consultant/software house, software-related company, other).

2. Second stage
There were two main questions for the interview: an initial evaluation of the framework,
and analysing differences in risk perceptions, as follows.
Q2.1. Evaluation of the framework
Participants would be asked to initially evaluate the proposed framework using
previously proposed evaluation criteria for risk assessment [133][134]. Four criteria
were used in this study:
 Adaptability:
Q2.1.1 How difficult is it to use the proposed framework in your different
projects? (Portability) (easy/medium/difficult)
Q2.1.2 How difficult is it to integrate the proposed framework with your
existing risk management? (Modifiability) (easy/medium/difficult)
 Feasibility (of the input data):
Q2.1.3 How difficult is it to get sufficient input data to use the proposed
framework? (Availability) (easy/medium/difficult)
Q2.1.4 How varying are groups of risks as the input data for the proposed
framework? (Scope of the input data) (easy/medium/difficult)
 Completeness:
Q2.1.5 Does the proposed framework provide enough level of detail for risks?
(Scope of completeness of the proposed framework) (easy/medium/difficult)
Q2.1.6 Does the proposed framework cover comprehensive risks needed to be
addressed? (Element (of risk))
82
 Validity:
Q2.1.7 How do the framework outputs represent the real phenomenon?
(Relevance)
Q2.1.8 Could you implement the framework outputs? (Practicability)
The above questions used related attributes and metrics to measure the proposed
framework [133][134]. The metric was extended into 3-points scale (easy-medium-
difficult) to investigate the degree of existence of an attribute [133].

Q2.2 Analysing differences in risk perceptions


Questions in this section aimed to identify existing risk management practices of the
case study participants (including collective risk management), to get recommendations
and comments, and to ask opinions about prediction of other stakeholder's risk
perception and comparison between the proposed framework and previous study.
Q2.2.1 How do you manage risk collectively with your stakeholder?
Q2.2.2 Please describe the process, method and tools used in your current risk
management practices.
a. Risk identification
b. Risk analysis
c. Risk prioritization
Q2.2.3 How do you prioritise risks?
Q2.2.4 What factors influence risk responsibility?
Q2.2.5 What are your recommendations to improve risk responsibility rationalisation?
Q2.2.6 Overall, what are your comments on the proposed framework?
Q2.2.7 How can you integrate the proposed framework to your software process? How
important is integrating the proposed framework to your software process?
Q2.2.8.a. Can you predict your stakeholder’s level of risk control and impact?
Q2.2.8.b.What is your opinion about the proposed framework compared to the
framework used in the previous survey?

6.3.3 Theoretical Propositions


The use of propositions in a case study is essential [86]. Theory development prior to
case study data collection guides what is being studied, what data to collect and what

83
strategies should be used to analyse data [86][85]. This case study used five
propositions (P):
P1 Risks can be managed by involving all stakeholders [8].
P2 Understanding differences of perceptions of OTS-specific risks between developers
and acquirers can help to avoid conflict and thus help to ensure successful software
projects [16].
P3 Perspective mismatch occurs because of differences between software developers
and acquirers as they have different backgrounds, experiences, needs, expectations
[18], and goals and structures [19].
P4 Risk responsibilities are assigned to a stakeholder who has the competence and
capacity needed to discharge the responsibility [34][35][78].
P5 Consensus building can be facilitated by comparing and discussing perspectives
between stakeholders [22][23][24].
Table 6.2 shows mapping between questions of data collection 1 and 2, the case study
questions and the related propositions.

Table 6.2 Mapping between case study questions, research questions and related
propositions

Question Topics of question Research Related


ID questions propositions
Data collection 1
Q1.1 Project information - -
Q1.2.1 Importance to involve stakeholder in risk RQ3.3 P1
management
Q1.2.2 Analysing differences in risk perspectives RQ3.3 and P2
between stakeholders RQ3.4
Q1.2.3 Risk responsibility RQ3.4 P1
Q1.2.4 Role in risk management RQ3.4
Q1.2.5 Framework input RQ3.3 and -
RQ3.4
Q1.3 Participant information - -
Data collection 2
Q2.1 Evaluation criteria for risk assessment RQ3.3 -
[133][134]
Q2.2.1 Collective risk management RQ3.4 P1, P2
Q2.2.2 Existing risk management practices RQ3.4
(identification, analysis, prioritisation)
Q2.2.3 Detailed risk prioritisation RQ3.4
Q2.2.4 Factors influencing risk responsibility RQ3.4 P4
Q2.2.5 Recommendation to improve risk responsibility RQ3.4 P1, P2, P3
Q2.2.6 Comments to the proposed framework RQ3.4 -
84
Q2.2.7 Integration of the proposed framework to RQ3.4 P1, P2, P3
software processes
Q2.2.8 Comparison to a method used in the previous RQ3.3 and P5
study (see Chapter 5) RQ3.4

6.3.4 The Case and Unit of Analysis Selection


The unit of analysis of each case is a process-related risk. The focus is on analysing
differences in risk perspectives between developers and acquirers in completed OTS-
based custom software projects. This can be classified as a retrospective case study
[132]. The case study consisted of four cases and was conducted in Indonesia. Each case
had 11 units of analysis. This section introduces the case study participants, covering
project information, participant information, and existing risk management practices.

The selection of cases was derived from research questions and access to resources
[86][85]. This case study used purposive sampling from the survey respondents (see
Chapters 4 and 5). One of the benefits having sampling from our survey’s respondents
was their familiarity with the 11 process-related risks. The choice of study participants
was based on a prior academic-industry relationship with the thesis author.

Four case study participants (shown in Table 6.3) were also respondents in our previous
survey. There were 69 total survey respondents. Forty-two potential survey respondents,
who were selected based on their strong relationship with the thesis author, were invited
to participate in this study and eight invited respondents agreed to participate. For the
purpose of data collection in this study, described in Section 6.3.6, the participants were
asked to involve their stakeholder. Of the eight respondents, only four could involve
their stakeholder to provide inputs for the framework. Therefore, four participants were
finally selected for this study. The case study participants and their stakeholders varied
in terms of their organisation sizes and background. Based on participatory
arrangements, the cases are referred to as 1Dev, 2AcqBank, 3AcqTelco and 4AcqGovt.

The participants were the main contact persons for the case study. The participants were
involved in all stages of the data collections. Each of their stakeholders came from
different organisations, and was only involved in providing inputs to the framework (see
question Q1.2.5) and participating in the agreement process for framework outputs after
the interview. Three stakeholders (not the one for 3AcqTelco) also gave detailed risk
perceptions along with the inputs to the framework. In the case of 1Dev and 2AcqBank,
85
detailed risk perceptions were collected using different interviews. The stakeholder of
the 2AcqBank participant was also involved in the second data collection interview,
where the agreement of the proposed risk responsibilities occurred. The other three
stakeholders agreed to the proposed risk responsibilities after the interview.

The project stakeholders included one acquirer and three developers, and were known
from the survey to have met the requirements to participate in this research.

Table 6.3 Case study participants

Case Participant Participant stakeholder


name Position and Involvement
background
1Dev Developer – Acquirer – Describing detailed risk
a domestic software a perceptions (interviewed by the
developer company telecommunication researcher) and agreement process
operators in of framework outputs after the
Indonesia interview
2AcqBank Acquirer – Developer – Describing detailed risk
one of the biggest an international- perceptions (interviewed by the
bank in Indonesia based IT consultant researcher), interview and
agreement process of framework
outputs in the interview
3AcqTelco Acquirer – Developer – Not describing detailed risk
a telecommunication a domestic perceptions but only completing
operator in software developer levels of risk control and impact,
Indonesia companies and agreement process of
framework outputs after the
interview
4AcqGovt Acquirer – Developer – Describing detailed risk
an Indonesian a domestic perceptions and agreement process
governmental software developer of framework outputs after the
agency company interview

6.3.5 The Case Description


The following descriptions of the four cases summarise the first data collection
excluding inputs to the framework information. Based on participatory agreements for
the case study, the real name of the participants, their stakeholder and OTS software that
can reveal information related to both stakeholders were coded and anonymous. Table
6.4 summarise the project, participant information and existing risk management
practices.

86
1. Case 1: 1Dev
The participant of case 1Dev is a domestic software developer company. This software
developer organisation has experience in customer relationship management, contract
management, dashboard application, workflow management system, interactive
multimedia and system integration. Their clients include government, banks, publishers,
postal service, university and telecommunication companies. The participant’s
stakeholder organisation is an acquirer. It is an information technology division of an
Indonesian telecommunication operator. It functions to support products and services
for internal usage and for external clients. The division was established in 1977 to
support and automate billing processes.

In this case, the 11 OTS-specific risks were investigated on a single sign on application.
The application was built for one a telecommunication operator in Indonesia (acquirer).
The application was used to increase interoperability and extensibility among existing
customer care systems by integrating them into one interface and single sign on
application. This application was a simplification and unification of pre-sales, sales and
after-sales functions in customer relationship management. Therefore, this application is
a critical application serving all contact points of the acquirer's service centers for all
customer segments and all types of services by minimising delay and transfer of
services. The application was built using a commercial PHP server, Oracle database,
JavaScript and commercial active directory. The use of the PHP server and JavaScript
were proposed by the developer, and the use of Oracle and the commercial active
directory were requirements from the acquirer. There were no fundamental changes to
system functionality. The possibility for requirements changes to the system were
accommodated in the software contract.

In this project, the acquirer involvement in managing risks was expected to avoid
blaming the developer, and to provide support and resources to mitigate risks. In risk
identification and analysis, the developer discussed differences in risk perceptions with
the acquirer. The aim of the discussion was to reconcile perceptions and support
development processes. However, both stakeholders did not provide the detail of the
approach used to reconcile differences in perceptions. The developer also expected that
the acquirer could understand proposed solutions and then could cooperate to manage
risks. Therefore, the responsibility of risks was on both stakeholders.

87
2. Case 2: 2AcqBank
The participant of case 2AcqBank is a Division of Information Technology of one of
the biggest banks in Indonesia, playing the role of an acquirer. The division has
experience contracting its system development needs to external parties. The
participant’s stakeholder organisation acts as a developer. It is an Indonesian division of
an American multinational technology and consulting corporation. The company offers
a range of integrated solutions, products and services. The company has operated in
Indonesia since 1937.

In this case, the 11 OTS-specific risks were investigated on customer data integration
and master data management systems. These systems need to comply with central bank
regulation, and to support core banking services and to understand customer profiles.
OTS software was proposed by an IT consultant and agreed by the acquirer. The system
was built using COBOL, Oracle, .net and commercial SOA middleware. The SOA
middleware vendor, who is an international-based IT consultant, developed the systems.
Requirements changes in the systems were accommodated through discussion and
negotiation with the developer.

There were different approaches and perceptions between both stakeholders to manage
risks. According to the acquirer, without risk management, there would be differences
in expected results between both stakeholders because the developer might not
understand the acquirer’s needs. In risk identification, brainstorming and discussion
were used; in risk analysis and prioritization, planning and review meeting were used.
Project managers of the acquirer and the developer were responsible to understand
differences in risk perceptions. User acceptance testing (UAT) and proof of concept
(POC) were used before implementation.

3. Case 3: 3AcqTelco
The participant of case 3AcqTelco was a telecommunication operator in Indonesia,
acting as an acquirer. The participant’s stakeholder organisation is a developer. It is a
contact center service and outsourcing company, supported by a large Indonesian
telecommunication operator. The company was established in 1975. The company has
734 employees and contact center operations in eight cities in Indonesia.

88
In this case, the 11 OTS-specific risks were investigated on a customer billing
recommender system. This system is critical to remind and support other units that
distribute and collect invoices. As there are customers that do not pay invoices if they
do not receive them, this system is needed to ensure high levels of invoice collection.
The developed system was added to an existing customer billing recommender system.
To anticipate changes to the acquirer's organisational structure and customer segments,
the system was developed using a parameterized design. The system was built by a
domestic software developer company. The developer also has responsibility to
maintain the system for 3 years. The system was built using Oracle, PHP, DHTMLX,
PHPMailer, FPDF, Asterisk, and jQuery. This OTS software was proposed by the
developer.

The acquirer stated the importance to involve its developer in risk management. The
acquirer required risk analysis for structuring requirements, project planning and
scheduling. To meet the requirements, development processes were monitored. Overall,
both stakeholders were responsible for risk management.

4. Case 4: 4AcqGovt
The participant of case 4AcqGovt was an Indonesian governmental agency, acting as an
acquirer. The participant’s stakeholder organisation is a developer. It is an engineering-
based university spin-off company with 15 years of experience in information
technology-based projects. The project portfolio includes telecommunication systems,
avionics systems, multimedia and online marketing solutions, academics and library,
logistics and operational, e-Government, IT consultancies, and IT empowerment. The
developer's clients are government agents, corporations and organisations. The
developer has 50 university-educated employees.

In this case, the 11 OTS-specific risks were investigated on an integrated system of e-


Planning and e-Monev (e-Monitoring and Evaluation). The main functionalities of this
system include data warehouse, searching, analytical tools, document collaboration and
knowledge sharing. The system was built by a domestic software developer company.

The system was built using commercial business analytics and business intelligence
(BI) software, a commercial document management system, Oracle, MySQL Enterprise
database and PHP. This OTS software was the result of an independent study to select

89
OTS software using analytic hierarchy process (AHP), to select OTS software.
Requirements changes were anticipated by the acquirer because of the dynamic nature
of the acquirer’s organisation.

The acquirer pointed out the importance of anticipating and managing risks of
development, and focusing and prioritising tasks. Even though there was no formal
analysis of differences in risk perceptions, both stakeholders agreed on outputs of
project tasks, which were the results of scheduled meeting to reconcile perceptions. In
case 4AcqGovt, the acquirer was responsible for risk management.

Table 6.4 Summary of case study participants’ project and participant information

Participant in case
1Dev 2AcqBank 3AcqTelco 4AcqGovt
Project information
Number of teams (full time) 4 15 4 20
Number of teams (part time) 2 3 1
Percentage of a project member 50% 80% 40% 50%
who had the most previous
experience with general OTS-
based development
Percentage of the application 40-60 % 30-40% 45% 50%
functionality in the whole
system was provided by selected
OTS components
Percentage breakdown of couple level between OTS software and system [47]
Configuration 50% 10% - 10%
Integration 30% 40% 30% 20%
Customization 15% 40% 70% 20%
Development 5% 10% - 50%
Participant information
Position in the project Project Software Software Project
manager developer architect coordinator
and system
analyst
Position in the current Chief Internal Officer Chief
organisation Operational software Information
Officer developer Officer
Experience in software 7 5 2 20
development (year)
Experience in using OTS 5 4 1 22
components (year)
Experience using OTS 5-7 5 2 >30
components (in projects)
Educational background Bachelor Bachelor Bachelor Master
(degree) (continued
90
to Master)
Educational background (major) Computer Computer Computer Computer
Science Science Science Science
(Bachelor)

6.3.6 Data Collection


There were two stages of data collection in this study. The first used a questionnaire-
based survey. The aims of the survey were to identify existing practices of involving
stakeholders in risk management and analysing differences in risk perceptions, and to
get inputs to the framework. The second stage collected data from semi-structured
interviews and document inspections. The list of interview questions is shown in
Section 6.3.2. The aims of this were to initially evaluate and also to analyse differences
in risk perceptions.

In the first data collection, the four participants were sent a questionnaire. The questions
are shown in Section 6.3.2. The participants of this study and their stakeholder then
provided inputs to the framework (these are shown in Appendix D).

The participants completed all parts of the questionnaire. Two participant’s


stakeholders, in the 1Dev and 2AcqBank cases, also completed framework inputs in an
interview. Then, the researcher followed the framework’s steps to produce outputs.
Prior to the second data collection, the framework outputs (including proposed
responsibilities) were sent to all participants to be reviewed, hence can increase
construct validity [86].

In the second data collection, semi-structured interviews were used to collect data. The
interviews were conducted in each participant’s office in Indonesia. In the interviews,
all participants were asked questions by the researcher. The researcher took notes and
recorded using audio digital recorder. In the case 2AcqBank, the stakeholder of the
participant was also interviewed.

The interviews began by explaining the framework. All participants including


participant’s stakeholder in case 2AcqBank were asked whether they agrees on the
framework outputs (stakeholder involvements, risk priorities and risk responsibilities).
All of them also provided information about their risk perceptions. Three participants
changed their framework’s inputs (level of risk control and/or impact), before the

91
agreement process in the interview. However, all participants, including the
participant’s stakeholder in case 2AcqBank, finally agreed on the framework’s outputs.

After the interview, the framework outputs were sent to the stakeholder of case 1Dev,
3AcqTelco and 4AcqGovt for their agreement (the stakeholder of case 2AcqBank gave
its agreement in the interview). The stakeholder of case 1Dev, 3AcqTelco and
4AcqGovt also agreed on the framework outputs. In conclusion, the participants and
their stakeholders did not meet as a group during this study. Rather, their individual
responses independently agreed with each other.

The participants were asked to evaluate the framework using evaluation criteria for risk
assessment [133][134]. The participants were also asked about existing risks
management practices involving their stakeholders, and about suggestions and
improvements to the framework. Finally, the researcher also asked about comparison
between the framework and the method used in previous survey (in Chapter 5).

After the interview, the validation processes were performed by sending transcripts of
the interview and the questionnaire to all participants (not to the stakeholders of the
participants) to get feedback and confirmation. The transcripts included results of this
case study (will be detailed in Section 6.4). All participants confirmed the accuracy of
the transcripts.

Another form of data collection in the case study was inspection of documents and the
participants’ websites. Three participants sent their software contract-related documents.
Additional information was also accessed from the stakeholder’s websites. The
document provided information about contract and project management. The websites
provided information about stakeholder organisation and project description. These data
collection support data triangulation, which can increase construct validity [86], and
provide evidence of case study context.

This study asked the participant to only provide inputs to the framework and answer
questions in the first and second stages of data collection. This study did not require the
participants to learn and to use the framework [131].

92
6.3.7 Data Analysis
Non-inputs of the framework’s data (participant and project information, existing risk
management practices and analysing differences in risk perceptions) were analysed by
linking to theoretical propositions and by developing a case description [86].

Differences in risk perceptions and framework outputs agreements between all


participants and their stakeholders were analysed by developing a case description, and
by using cross case analysis [86]. Cross case analysis was also used to compare and
aggregate findings from the responses to the questionnaires and interviews [86]. An
example is given in Table 6.7. This analysis is presented in tabular form grouped by
case study questions.

An initial evaluation of the framework was measured using evaluation criteria for risk
assessment [133][134]. This study only used four criteria for evaluating the framework
(adaptability, feasibility, completeness and validity) [133][134], as described in Section
6.3.2. Existing risk management practices and suggestions from all participants were
then used to improve the framework [129] (provided in Section 6.5.4).

6.4 Result
In this section, the results are organised into three parts: the results of risk perspectives
from each case study participant and their stakeholders, the risk assessment evaluation
results, and the questionnaire and interviews results (data collection 1 and 2).

6.4.1 Results of Case Study Participants Analysed Using the Framework


This section describes risk perceptions of the participants and their stakeholders,
analysed using the framework. The results are summarised in Table 6.5. Even though
there were initially nine changes to the participants’ inputs (either on risk control or
impact), all participants and their stakeholders finally agreed on proposed framework
outputs (stakeholder involvement, risk priority and risk responsibility). The changes of
the participants’ inputs happened before the agreement of the proposed risk
responsibilities during the interview in the following cases: 1Dev (4 changes);
3AcqTelco (1 change); 4AcqGovt (4 changes). Based on the risk control/impact matrix,
risk priorities can be mapped into the following: collaborate as high priority risk;
involve and consult as medium priority risk; inform as low priority risk. The inputs to

93
the framework are provided in the Appendix D. The next sub-sections details the risk
perceptions of each participant and their stakeholder.

94
Table 6.5 Framework outputs (stakeholder involvement and risk priorities) and agreement of risk responsibilities
Number of risk
Case
responsibility
1Dev 2AcqBank 3AcqTelco 4AcqGovt Dev Acq Both
Risk Stakeholder Risk Stakeholder Risk Stakeholder Risk
Stakeholder
resp. involvement resp. involvement resp. involvement resp.
involvement
Risk
Selection effort Dev: Collaborate Dev Dev: Both Dev: Both Dev: Both 1 3
ill-estimated Acq: Inform Collaborate Collaborate Collaborate
Acq: Acq: Acq:
Collaborate Collaborate Collaborate
Not adaptable Dev: Consult (*) Acq Dev: Both Dev: Dev Dev: Both 1 1 2
to requirement Acq: Collaborate Collaborate Collaborate Collaborate
changes Acq: Acq: Inform Acq:
Collaborate Collaborate
Requirements Dev: Collaborate Both Dev: Inform Acq Dev: Both Dev: Both 1 3
not negotiable Acq: Collaborate Acq: Collaborate Collaborate
Collaborate Acq: Acq: Involve
Collaborate (*)
Complicated Dev: Consult Acq Dev: Consult Both Dev: Inform Acq Dev: Involve Dev 1 2 1
multi-OTS Acq: Collaborate Acq: Consult Acq: Acq: Inform
components Collaborate
arrangement
Insufficient Dev: Collaborate Both Dev: Dev Dev: Dev Dev: Both 2 2
OTS (*) Collaborate Collaborate Collaborate
component Acq: Collaborate Acq: Consult Acq: Inform Acq: Involve
documents (*)
Lack of OTS- Dev: Collaborate Both Dev: Both Dev: Both Dev: Involve Both 4
driven Acq: Collaborate Collaborate Collaborate Acq: Involve

95
requirements Acq: Acq:
engineering Collaborate Collaborate
process
Maintenance Dev: Involve (*) Dev Dev: Inform Both Dev: Dev Dev: Involve Both 2 2
planning Acq: Consult Acq: Inform Collaborate Acq: Involve
unfeasible Acq: Inform
(*)
Upgrade Dev: Inform Acq Dev: Inform Both Dev: Inform Both Dev: Both 1 3
unfeasible Acq: Involve Acq: Inform Acq: Inform Collaborate
Acq: Involve
(*)
Lack of Dev: Collaborate Both Dev: Both Dev: Both Dev: Dev 1 3
information on Acq: Collaborate Collaborate Collaborate Collaborate
vendor Acq: Acq: Acq: Inform
Collaborate Collaborate
Lack of support Dev: Consult (*) Acq Dev: Dev Dev: Both Dev: Both 1 1 2
Acq: Involve Collaborate Collaborate Collaborate
Acq: Inform Acq: Acq:
Collaborate Collaborate
Reduced Dev: Consult Acq Dev: Inform Both Dev: Both Dev: Involve Dev 1 1 2
control of Acq: Collaborate Acq: Inform Collaborate Acq:
future Acq: Consult (*)
evolution of the Collaborate
system
Legend: Acq: acquirer; Dev: developer; Risk resp.: Risk responsibility; (*): either level of risk control or impact or both was changed before the
agreement of the proposed risk responsibilities during the interview

96
6.4.1.1 Risk Perceptions of Case 1Dev
In case 1Dev, the participant was a software developer and the stakeholder of the
participant was a software acquirer (see Table 6.3).

For the risk of selection effort ill-estimated, the developer was assigned to choose two
OTS software: a commercial PHP server and JavaScript. The other two pieces of OTS
software were defined by the acquirer: Oracle and commercial active directory. The
developer perceived itself to have high risk control and be most impacted by this risk.
The acquirer decided to use the PHP server, which complied with the acquirer’s IT
organisation procedure, and bought the license of PHP server. The acquirer already had
the license of oracle and active directory. The acquirer perceived itself to had low risk
control and impact (of the four pieces of OTS software used).

Consistent with the previous risk, the developer had high potential impact from the risk
of not adaptable to requirement changes, because the developer proposed the PHP
server. The developer had low control over the use of commercial OTS software. The
acquirer was perceived to be most impacted by this risk because the developed system
was used to service customers (critical application). Based on the high potential impact
to acquirer’s business, the acquirer perceived itself to have high risk control by
contracting the developer to build the system.

Both the developer and acquirer had high control and impact over the risk of
requirements not negotiable and lack of information on vendors (commercial OTS
software vendors).

The developer had limited control over the commercial OTS software because the
license of the commercial OTS software was owned by the acquirer. The acquirer was
perceived to be most impacted by the risk of complicated multi-OTS components
arrangement. There were two reasons for high risk impact. Firstly, multi-OTS software
is difficult to maintain. Secondly, from an organisation-wide audit process perspective,
multi-OTS software is difficult to audit. Therefore, the acquirer used higher control to
reduce the impact of this risk.

For the risk of insufficient OTS component documentation, the developer argued that the
risk responsibility should be shared by both parties because the PHP server was
97
proposed by the developer and then agreed to be used by the acquirer. To control for the
lack of the PHP server documentation, the developer customised the PHP server. The
acquirer argued that the developer should submit user manual documentation. Both
stakeholders perceived themselves to have high risk control and impact.

The developer used an estimation-based approach to select the PHP server. Therefore,
the developer had high risk control for the risk of lack of OTS-driven requirements
engineering process. In addition, the developer perceived itself to have high risk impact.
The acquirer also perceived itself to have high risk control and impact.

Even though there was a new release of any OTS software, the decision to upgrade the
OTS software in the system was the developers’. The new release of OTS software
might not have been compatible with other used OTS software. Therefore, risk impact
was low for the risk of maintenance planning unfeasible because the developer could
control the risk (high risk control) by deciding to use existing stable OTS software.
However, the acquirer perceived itself to have high risk impact because it was difficult
for it to maintain the developed system. On the other hand, the acquirer perceived itself
to have low control on OTS software from the commercial OTS software vendors.

For the risk of upgrade unfeasible, the developer perceived itself to have low risk
control and impact. On the other hand, the acquirer had control over the developer to
maintain the developed system but perceived itself to have a low risk impact.

Because the license of commercial OTS software belonged to the acquirer, the
developer perceived itself to have low risk control for the risk of lack of support. The
developer perceived itself to have high impact on this risk due to lack of technical
support and user training. However, having a license made the acquirer able to
negotiate with commercial OTS software vendors, therefore the acquirer perceived itself
to have high risk control. The acquirer perceived itself to have low risk impact because
the acquirer believed that a better understanding of OTS software features/capabilities
could substitute lack of support from vendor.

For the risk of reduced control of future evolution of the system, the developer claimed
that evolving system was hard to control (low control) and perceived this to have a high

98
risk impact. The acquirer argued that the risk was important to control by contracting
the developer (high control) and also perceived it to have a high risk impact.

6.4.1.2 Risk Perceptions of Case 2AcqBank


In the case 2AcqBank, the participant was a software acquirer and the stakeholder of the
participant was a software developer (Table 6.3). The system was built using COBOL,
Oracle, .net and commercial SOA middleware.

Both the acquirer and developer perceived themselves to have high control and impact
for the risk of selection effort ill-estimated, lack of information on vendor (commercial
OTS software vendors) and not adaptable to requirement changes. For the last risk,
based on the software contract, both stakeholders agreed that requirement changes must
be accommodated.

According to the acquirer, the control and impact of the risk of requirements not
negotiable was high. The acquirer argued that the developed system must meet business
process requirements (high impact) and budget. To control the risk, because of limited
budget, the acquirer had an option to replace the developer if it could not meet the
requirements. However, the developer perceived this risk as to be under low control and
to have low impact.

Both the acquirer and developer perceived themselves to have high impact for the risk
of complicated multi-OTS components arrangement due to high cost for maintenance.
However, both perceived themselves to have low control for this risk.

For the risk of insufficient OTS component documentation, the acquirer believed that the
documents were needed (high impact). However, the acquirer found it impossible to
completely examine the documents (low control). For this risk, the developer had high
risk control and impact because the documents were needed and able to be accessed by
the developer.

Both the acquirer and developer perceived themselves to have high control and impact
for the risk of lack of OTS-driven requirements engineering process. The acquirer
maintained high control of requirement engineering process by executing a rigorous
proof of concept (POC) process. This started by tender, and then used a product demo
and proof of concept.
99
Both the acquirer and developer perceived themselves to have low control and impact
for the risk of maintenance planning unfeasible and upgrade unfeasible. For the second
risk, based on the acquirer and developer responses, there would be fewer problems to
upgrade the developed system because the upgraded system would be tested.

For the risk of lack of support, the acquirer perceived this as low risk control and impact
but the developer perceived it as high risk control and impact. The acquirer had low
impact of this risk because all possible problems would be handed to commercial OTS
software vendors. The developer also believed that the support should be delivered by
the commercial OTS software vendors or its official representatives.

For the risk of reduced control of future evolution of the system, both the acquirer and
developer perceived themselves to have low risk control and impact. The acquirer did
not require the latest release of OTS software as long as the system functioned.
However, if the latest version of OTS software could fix important bugs then the OTS
software would be updated.

6.4.1.3 Risk Perceptions of Case 3AcqTelco


In case 3AcqTelco, the participant was a software acquirer and the stakeholder of the
participant was a software developer. However, the developer in case 3AcqTelco did
not give detailed risk perceptions. Thus, all description here is based on the acquirer’s
perspective. The system was built using Oracle, PHP, DHTMLX, PHPMailer, FPDF,
Asterisk and jQuery.

For the risk of selection effort ill-estimated, both stakeholders perceived themselves
high risk control and impact. The control of this risk was revealed by the following OTS
selection processes. Firstly, the acquirer described a list of requirements. Secondly, the
developer offered OTS software to be used. Then, the OTS software was tested by the
acquirer using benefit cost analysis of the OTS software’s features. The acquirer also
compared the use of the OTS software by existing developer’s customers, which were
in auto finance, bank, airline and automotive industries.

For the risk of not adaptable to requirement changes, the acquirer perceived itself to
have low risk control and impact. The specification of requirements was defined by the
acquirer and was stable (detailed in the risk of maintenance planning unfeasible). Based

100
on the contract, the acquirer had the low risk impact because the developer had a
responsibility to meet the requirements. However, the developer perceived itself to have
high risk control and impact.

Both the acquirer and developer had high control and impact for the risk of
requirements not negotiable. The acquirer used a software contract to control this risk.
Requirement specification and OTS software features were documented in the software
contract. The contract included shared and individual roles. The contract was
negotiable, if needed changes were not set previously.

The acquirer controlled the risk of complicated multi-OTS components arrangement by


asking in the contract for the developer to manage multi-OTS components-related
problems. The developer gave information regarding OTS software, such as the price of
OTS software license. Then, the acquirer bought licenses for the OTS software.
Problems occurring during the contract period must be managed by the developer,
including commercial OTS software vendor-related risks. From the acquirer
perspective, the high impact of this risk related to the nature of the developed system,
which was a critical system. On the other hand, the developer perceived this risk as low
risk control and impact for itself.

The acquirer perceived the risk of insufficient OTS component documentation as having
low risk control and impact but the developer perceived this risk as having high control
and impact. The acquirer expected knowledge transfer, e.g. user training and user
manual.

Both the acquirer and developer had high control and impact for the risk of lack of
OTS-driven requirements engineering process. Candidates for OTS software were
derived from an IT architecture from the acquirer’s organisation (predefined operating
systems, web servers and programming languages). Therefore, the list of requirements
described proposed OTS software, including the use of a trusted engine (see the risk of
selection effort ill-estimated).

For the risk of maintenance planning unfeasible, the acquirer perceived the risk as
having low control and impact because the contract, which lasted 3 years, did not
prioritise the use of the latest release of OTS software. To anticipate changes to the

101
acquirer's organisation structure and customer segments, the system was developed
using a parameterised design (for example, setting configuration files). The developer
perceived this risk as having high control and impact, which may relate to the long term
of the maintenance.

Both the acquirer and developer had low control and impact for the risk of upgrade
unfeasible. From the acquirer’s perspective, this risk related to the risk of maintenance
planning unfeasible. The developed system was tested using data from customer growth
over the last 5 years. As this risk related to the previous risk, the acquirer expected the
developer to have high risk control and impact.

Both the acquirer and developer had high control and impact for the risk of lack of
information on vendor (commercial OTS software vendors). The acquirer perceived a
high impact for this risk because the developed system was critical to support national-
wide operation. The acquirer had high control through benchmarking the use of OTS
software in the OTS selection. The quality of OTS software was a major factor in its
selection.

For the risk of lack of support and reduced control of future evolution of the system,
both the acquirer and developer perceived themselves to have high control and impact.
For the first risk, the acquirer perceived itself to have high risk impact because the
acquirer not only needed support but also related documentation. The acquirer asked the
developer to provide a new document for the changed system creating high risk control.
For the second risk, the high risk control for the acquirer was shown by the use of a
parameterised design to accommodate future changes.

6.4.1.4 Risk Perceptions of Case 4AcqGovt


In case 3AcqTelco, the participant was a software acquirer and the stakeholder of the
participant was a software developer. The system was built using a commercial business
analytics and business intelligence (BI) software, commercial document management
system, Oracle, MySQL Enterprise and PHP.

According to the acquirer and developer, the control and impact of the risk of selection
effort ill-estimated was high. The acquirer employed an independent consultant to
conduct a study of OTS software selection using analytic hierarchy process (AHP). The

102
results of this study were then used to define features and functionalities of the
developed system. Thus, the acquirer perceived OTS software selection-related risk as
having high impact.

Both the acquirer and developer though that the risk of not adaptable to requirement
changes had high control and impact. The acquirer expected that all OTS software used
would still give optimal benefits if requirements changed.

The acquirer had low impact and high control for the risk of requirements not
negotiable. Based on the contract, requirements and development were the developer’s
responsibility (low risk impact for the acquirer). The acquirer perceived itself to have
high risk control because the acquirer regarded the terms of reference (TOR) as a
control. The acquirer also assumed that if the developer agreed with TOR then the
developer should understand system requirements. The developer perceived itself to
have high risk control and impact for this risk.

The acquirer perceived risk impact as low for the risk of complicated multi-OTS
components arrangement because the license arrangements of commercial OTS
software were not complex. In addition, the risk control for this risk was low because
the TOR managed multi-OTS components arrangement implicitly. According to the
acquirer, the developer could ask for additional OTS software licenses as long as stated
in the contract between the acquirer and commercial OTS software vendors. However,
the developer perceived itself to have low risk impact and high risk control for this risk.
The developer expected that the acquirer should have high risk control because the
acquirer had agreed on the proposed analysis and design of the developed system
(including all related risks). Therefore, both stakeholders were responsible.

For the risk of insufficient OTS component documentation, the acquirer believed itself to
have low impact and high control and the developer believed itself to have high control
and impact. The acquirer had high risk control because there was control over output
documentation for the developed system.

Both the acquirer and developer had low impact but high control for the risk of lack of
OTS-driven requirements engineering process. The acquirer thought if had low risk
impact because the development was contracted to an external party (the developer)

103
based on the TOR. The acquirer used the results of a technical study, informing the need
of integration of multi-OTS software, because a single platform was not sufficient (high
risk control). Therefore, system development needed modification to integrate multi-
OTS software to meet requirements.

Both stakeholders also perceived the risk of maintenance planning unfeasible to have
low impact but high control for themselves. The acquirer thought the risk had low
impact because the development was contracted to an external party (the developer),
based on the TOR. Future changes were anticipated by the acquirer because of the
dynamic nature of the acquirer’s organisation (high risk control).

For the risk of upgrade unfeasible, the acquirer perceived itself to have low impact and
high control, but the developer perceived itself to have high control and impact. The
acquirer regarded the terms of reference (TOR) as a control. Furthermore, if the
developed system did not meet requirements then, based on TOR, the acquirer would
not accept the developed system. However, the developer expected that the acquirer
should also be responsible for this risk because the acquirer was the party that ordered
the update to the developed system.

The acquirer indicated having low control and impact for the risk of lack of information
on vendor (commercial OTS software vendors). As system development was contracted
to the developer, the acquirer perceived this risk as having low impact. According to the
acquirer, the developer was very concerned about information available from
commercial OTS software vendors. Based on the TOR, the developer was responsible
to manage not only open source but also commercial OTS software used in the system.
On the other hand, the developer perceived this risk to have high control and impact.

Both stakeholders had high control and impact for the risk of lack of support. The
acquirer argued that lack of support would affect internal capacity building to maintain
and enhance the developed system. Internal capacity building was needed to reduce
vendor lock-in. In addition, the acquirer required that user training must be performed
by commercial OTS software representatives or certified person from commercial OTS
software vendors.

104
For the risk of reduced control of future evolution of the system, the acquirer perceived
itself to have high impact and low control. This risk had high potential impact because
of significant changes of policies. This risk had low control because the maintenance
was performed by the developer. On the other hand, the developer perceived the risk to
have low impact and high control.

6.4.2 Risks Assessment Evaluation Results


Table 6.6 presents the results of evaluating the framework using evaluation criteria
[133][134].

Table 6.6. Comparison of evaluation criteria for risk assessment for four participants

Attribute Case name (based on the case study participant’s opinions)


1Dev 2AcqBank 3AcqTelco 4AcqGovt
Portability Easy Medium Easy Easy
Modifiability Medium Easy Medium Medium
Availability Medium Easy Easy Medium
Scope of the Easy Easy Easy Medium
input data
Scope of Medium Medium Easy Easy
completeness
of the
proposed
framework
Element (of Cover various Cover various Cover various Depend on the
risk) and different risks (in the risks and the framework’s
risks beginning) list of risk outputs.
could be
adjusted
Relevance Represent real Relevant with Represent real Map
risks real risks (to risks stakeholders
define (completed by concerns and
responsibility on the acquirer responsibilities
risks in the and developer) of tasks.
project contract)
Practicability 1. Manage risks Define risks of Used to Are applicative
(in term: who acquirer and discuss about
should be developer in and take
responsible, software action on risks
when and how development.
manage risks)
2. Define and
detail Scope of
Work (SoW)

105
6.4.3 Results of Data Collection 1 and 2
Table 6.7 summarises the questionnaire and interview answers, excluding answers to
questions about the framework inputs and evaluation. Complete results can be found in
Appendix E.

106
Table 6.7 Example of cross-case analysis for data collection 1 and 2, excluding framework inputs and evaluation criteria for risk assessment

Question Case name (based on participant’s opinions)


1Dev 2AcqBank 3AcqTelco 4AcqGovt
Factors influencing The one who has Stakeholder having Defined in the software Technical and non-technical
risk responsibility responsibility on risks. knowledge (on a risk). contract. understanding of risk.

Defined in Scope of Work. Defined in the project Acquirer organisation's Communication to achieve shared
contract. policy about understanding.
Risk outside responsibility stakeholder role and
of acquirer and developer responsibility on system Resource and capacity.
must be discussed with development.
respective stakeholder. Terms of reference.
Negotiation.
Recommendation Detailing Scope of Work Defined in software Describing level of Defining stakeholder responsibility
to improve risk (after using the contract. each stakeholder and clearly.
responsibility framework) shared responsibility.
Shared understanding of the definition of
Adding cost/benefit control.
analysis and
comparison. Detailed communication instruments,
e.g. meeting and Terms of reference
(ToR).

Clarification the understanding of each


stakeholder’s roles and responsibilities
on (ToR).
Prediction of other Cannot predict Cannot predict because Cannot predict Cannot predict unless both stakeholders
stakeholder‘s input of different interests. have a shared understanding
on the framework
107
Comparison to a This framework is better This framework is This framework is This framework is better because the
method used in better because this better because this framework provided structured,
Chapter 5 Using this framework, framework compared framework compared methodical and measured approach to
stakeholder should be able actual risk perceptions. actual risk perceptions analyse risk control and impact.
to determine risk control. and use simple level of
risk control and impact.

108
6.5 Discussion
This section discusses RQ3.3 and RQ3.4.

6.5.1 RQ3.3: How are OTS-specific risks perceived by developers and acquirers?
As our study focused on the perceptions of risk control and impact, we aimed to identify
key findings emerging from the cross-case analysis. We analysed stakeholder
involvement in Table 6.5 to identify key findings about risk priority. Finally, we
investigated commonalities of risk priorities between both stakeholders.

6.5.1.1 Classification of Key Results about Risk Control and Impact


Before performing the cross-case analysis, we classified key results about risk control
and impact into 12 important results based on categories from the Software
Engineering Institute’s Taxonomy of Software Development Risk [67], as described in
Table 6.8. Since not all results could be classified into specific attributes of the risk
taxonomy, elements (which consist of attributes) were also used to classify the results.
We initially used four elements (requirements, design, integration and test, and
contract) and two attributes (budget and vendors). However, the risk taxonomy is a
generic risk categorisation scheme; therefore, as suggested in [67], we added two
elements as risk categories: OTS software selection, which is specific to OTS-based
software custom projects, and maintenance. Maintenance was added because the risk
taxonomy was specific to software development. However, to avoid redundant
information caused by important results being classified into more than one category,
we combined categories to simplify the presentation of the risk classification. We had
12 categories: requirements and contract; OTS software selection; design; maintenance;
maintenance, design and contract; maintenance and contract; contract; contract and
OTS software selection; vendor; vendor, OTS software selection and integration; vendor
and maintenance; and budget.

Three results related to requirements and contracts. Firstly, acquirers considered that
developers had a responsibility to meet the requirements. Secondly, the acquirer
regarded the contract as a risk control mechanism. In the 4AcqGovt case, for the risk of
requirements not negotiable, it was found that requirements and development are the
developer’s responsibility, and that the acquirer would not accept a developed system
that did not meet requirements (the risk of upgrade unfeasible). Finally, the contract

109
was used to change and negotiate requirements. In the 2AcqBank case for the risk of not
adaptable to requirement changes, the contract was used to control for requirement
changes by accommodating those requirements changes. Both stakeholders agreed that
the developed system must meet system requirements. In addition, in the 3AcqTelco
case, for the risk of requirements not negotiable, the contract was negotiated to
accommodate changes needed and not previously set.

Two results related to OTS software selection. Firstly, from the acquirer perspective, an
independent study of OTS selection using AHP could define provided features and
functionalities of the system requirements (the risk of selection effort ill-estimated in
case 4AcqGovt). Secondly, from the developer perspective in case 1Dev, because the
acquirer had agreed to the proposed OTS software from the developer, the responsibility
was shared.

There is one result related to design. In the 4AcqGovt case, for the risk of complicated
multi-OTS components arrangement, the developer expected that after the acquirer had
agreed to the proposed analysis and design of the developed system (including all
related risks), the responsibility should be shared.

Regarding maintenance-related risks, acquirers had two different kinds of perspective


and developers had one. According to the first acquirer perspective, multi-OTS software
was difficult to maintain and audit; therefore, more control was needed. According to
the second acquirer perspective, to build internal capacity building and decrease vendor
lock-in, the acquirer required that user training must be performed by OTS software
vendor representatives or certified persons from the OTS software vendor. The
developer decided to use existing stable OTS versions to lower the risk impact of
maintenance planning unfeasible.

Maintenance could also be related to design and contract from the perspective of the
acquirer. Future changes, which might be caused by the dynamic nature of the
acquirer’s organisation, were anticipated by the acquirers (in the 3AcqTelco and
4AcqGovt cases). One of the strategies used to reduce control of future evolution of the
system was a parameterised design approach (case 3AcqTelco).

110
Three results are related to maintenance and contract. Firstly, from the perspective of
the acquirer in the 4AcqGovt case, to control the risk of reduced control of future
evolution of the system, the developer performed system maintenance. Secondly,
according to the acquirer, based on the contract, the responsibility of vendor-related
information (such as vendor support and customer service) was on the developer (the
risk of lack of information on vendor). Finally, because the license of the OTS software
belonged to the acquirer, the developer perceived themselves to have high impact due to
lack of technical support and user training (in case 1Dev for the risk of lack of support).

When acquirers acquire a critical system, they tried to reduce risks through their
contract with developers. From the acquirers’ perspective, there were two risks
associated with contracting the developer: complicated multi-OTS components
arrangement in the 3AcqTelco case and not adaptable to requirement changes in the
1Dev case. As stated in the contract in the 3AcqTelco case, problems occurring during
the contract period must be managed by the developer, including OTS software vendor-
related risks. By contracting the developer, the acquirer in the 1Dev case wanted to
reduce risk impact of requirement changes in a critical system.

Contract-related risks were also found in other categories. For example, based on
contract, the developer chose OTS software; therefore the developer had high impact as
a consequence of proposing and choosing OTS software (in case 1Dev for the risk of
selection effort ill-estimated). Furthermore, as stated by the acquirer in the 4AcqGovt
case, the responsibility for understanding OTS-driven requirements and maintenance
planning was on the developer’s.

As found in the 1Dev case, the developer and acquirer had different perception of
limited control over OTS software. The developer perceived themselves to have low
control over OTS software in the integration and selection phases and the acquirer
perceived themselves to have low control over OTS software in the maintenance phase.
As both stakeholders experienced high risk impact, a possible explanation for the
difference in limited control over OTS software could be attributed to the importance of
OTS software during each phase for each stakeholder. In the risk classification, the
limited control over OTS software was classified as vendor, as shown in Table 6.8.

111
Two acquires had the same perception about OTS software vendors regarding the risk
of lack of support. In the 2AcqBank case, the acquirer perceived themselves to have low
risk impact because all support-related problems would be handed to the OTS software
vendor. The acquirer in the 1Dev case argued that having a license of OTS software
made them able to negotiate with the OTS software vendor.

Regarding budget, two acquirers perceived themselves to have high risk control. For the
risk of requirements not negotiable, the acquirer in the 2AcqBank case would have
chosen another developer to suit the existing budget and business process requirements.
In the 4AcqGovt case, the acquirer expected the OTS software to meet their budget for
the optimal benefit using OTS software if the requirements changed.

112
Table 6.8 Classification of key results regarding risk control and impact

Category Case
1Dev 2AcqBank 3AcqTelco 4AcqGovt
Requirements and The contract was used Based on the contract, Requirement and
contract to control for the developer had a development were the
requirement changes responsibility to meet developer’s responsibility
(acquirer and the requirements (acquirer)
developer) (acquirer)
The acquirer would not
The contract might be accept the developed
negotiated to system that did not meet
accommodate needed requirements (acquirer)
changes not set
previously (acquirer) The responsibility for
understanding OTS-driven
requirements was on the
developer (acquirer)
OTS software selection Because the acquirer Using a method in OTS
agreed on proposed OTS software selection (i.e.
software then the AHP) could define
responsible should be provided features and
shared (developer) functionalities of the
system requirements
(acquirer)
Design Because the acquirer
agreed on proposed
analysis and design of the
developed system then the
responsible should be
113
shared (developer)

Maintenance Multi-OTS software were The support of OTS


difficult to maintenance software vendor was
and audit (acquirer) required to build internal
capacity building and
The developer decided to decrease vendor lock in
use existing stable version (acquirer)
(developer)
Maintenance, design and Changes were Changes were anticipated
contract anticipated by the by the acquirer (acquirer)
acquirer by using a
parameterised design
approach (acquirer)
Maintenance and contract The developer did not have The acquirer perceived to
the license of OTS have low control because
software; therefore the the system maintenance
developer perceived to was performed by the
have high impact due to developer (acquirer)
lack of technical support
and user training The responsibility for
(developer) maintenance planning was
on the developer (acquirer)

The responsibility of
information on vendor was
on the developer (acquirer)
Contract The impact of critical The impact of critical
system made the acquirer system made the
contracting the developer acquirer contracting the
114
(acquirer) developer (acquirer)
Contract and OTS The developer chose OTS
software selection software; therefore the
developer had high impact
as a consequence of
proposing and choosing
OTS software (developer)
Vendor Having the license of OTS The acquirer perceived
software made the acquirer to have low risk impact
to be able to negotiate with because all support-
OTS software vendor related problems would
(acquirer) be handled to OTS
software vendor
(acquirer)
Vendor, OTS software The developer perceived to
selection and integration have low control over OTS
software in integration and
selection phase (developer)
Vendor and maintenance The acquirer perceived to
have low control over OTS
software in maintenance
phase (acquirer)
Budget The acquirer would The acquirer expected OTS
choose other developer software to meet budget for
to suit with existing optimal benefit of the use
budget and business of OTS software if
process requirements requirements changed
(acquirer) (acquirer)

115
6.5.1.2 Cross-Case Analysis
Here we identify key findings emerging not only from each individual case but also
across cases [136]. As the focus of this study is to investigate risk perceptions using the
stakeholder analysis approach, the findings were based on the relationship between
control and impact of investigated risks. Firstly, similar relationships were generalised
to identify the findings not only within each case but also across cases [136]. Secondly,
the findings were selected from each category by comparing evidence either within each
case or across cases [136]. Sometimes several categories in Table 6.8 were merged or
synthesised to generalise a finding. There are 11 key findings on risk control and
impact.

Finding 1 (high risk impact and control): A high risk impact made acquirers apply
higher control over the risk. Finding 1 is synthesised from three categories:
maintenance, contract and budget (see Table 6.8). In the maintenance category, the type
of impact are the difficulty of maintaining and auditing multi-OTS software (in the
1Dev case), and the need to build internal capacity and decrease vendor lock-in (in the
4AcqGovt case). For the second impact (in the maintenance category), user training
must be performed by OTS software vendor representatives or certified persons from
the OTS software vendor. In the contract category, because of the impact of critical
application, the acquirer in the 1Dev and 3AcqTelco cases contracted the developer to
build the application. Thirdly, in the budget category, the type of impact is not meeting
the budget for the optimal use of OTS software if requirements change (in the
4AcqGovt case), and not meeting business process requirements and having a limited
budget (in the 2AcqBank case). For the second impact in the budget category, acquirers
could choose another developer to suit the existing budget and business process
requirements.

Finding 2 (OTS selection): An acquirer can use OTS selection results to gain higher
control when defining system requirements, which has high impact. In the 4AcqGovt
case, the OTS selection was performed by an independent study of OTS selection using
AHP.

Finding 3 (maintenance): A developer can decide to use an existing stable OTS


version to control the impact of system maintenance (as in the 1Dev case), providing
higher control over this risk.
116
Finding 4 (maintenance): Acquirers can anticipate future changes by asking their
developer to use a parameterised design (for example, setting configuration files).
Evidence of this can be found under the maintenance, design and contract category in
the 3AcqTelco and 4AcqGovt cases.

Finding 5 (contract): To lower risk impact, acquirers may contract an external party.
The relationship between an acquirer and its external party must be governed by a
software contract. Support for this finding can be classified into two groups: low risk
impact and low risk control; and low risk impact and high risk control. There were three
instances of the first group (low risk control and low risk impact). All support-related
problems were handled to OTS software vendor in the 2AcqBank case. Secondly, based
on the contract, in the 3AcqTelco case, the acquirer had the low risk impact because the
developer had a responsibility to meet the requirements. In the 4AcqGovt case, the
responsibility of vendor-related information was the developer’s.

There were four instances of the second group (low risk impact but high risk control).
Firstly, as shown in the 4AcqGovt case, the developer should have responsibility on
requirement changes and development, just as with understanding requirements (in the
second instance), and maintenance (in the third instance). Finally, in the 4AcqGovt case,
the acquirer used the contract to define system acceptance criteria. The first group, low
risk control and low risk impact, shows how the acquirer can lower risk impact by
contracting external parties. For the second group, the acquirer had higher control
because the contract was used to control the developer to have responsibility on
requirement changes, development and maintenance.

Finding 6 (contract): An acquirer might have lower control when it contracts a


developer (as in the 4AcqGovt case) and uses OTS software (as in the 1Dev case). In
accordance with finding 5, finding 6 shows that risk impact cannot totally be transferred
to an external party. From a risk control perspective, finding 5 and 6 illustrate that even
though a software contract can be used to control an external party, an acquirer will
reduce its control once it contracts an external party.

Finding 7 (contract): Using a software contract to control risks leads to consequences


regarding what must be agreed in the contract. In the 1Dev case, based on the contract,
the developer chose OTS software. Therefore the developer had high impact as a

117
consequence of proposing and choosing OTS software. Furthermore, using the contract
to control for requirement changes required both stakeholders to accommodate the
changes (in case 2AcqBank and 3AcqTelco).

Finding 8 (licensing OTS software): A developer without an OTS software license


has lower control because they lack support from the OTS software vendor. This can
have high risk impact, as seen in the 1Dev case.

Finding 9 (licensing OTS software): An acquirer with an OTS software license can
better negotiate with an OTS software vendor, as seen in the 1Dev case.

Finding 10 (limited control over OTS software): Stakeholders have different concerns
about their limited control over OTS software. Developers tend to have limited control
over OTS software in the OTS selection process and OTS integration, and acquirers
tend to have limited control over OTS software in maintenance, after the end of a
contract.

Finding 11 (shared responsibility): Both stakeholders share risk responsibility if


acquirers agree on the OTS software selection (as in the 1Dev case), and on the
developer’s proposed system design decision (as in the 4AcqGovt case).

The findings above emerged from cross-case analysis, and indicate that the analysis
framework helps to compare risk perceptions, to better understand stakeholders’ risk
perceptions. The comparison facilitates the agreement of proposed framework outputs
and risk responsibilities between both stakeholders. Furthermore, the comparison of risk
perceptions also support a discussion between stakeholders by providing opinions,
further understanding [24] and expectations of each stakeholder. If both stakeholders
disagree with proposed framework outputs, including proposed risk responsibilities, or
other stakeholder’s perceptions, the discussion is useful to elicit the reason of this
disagreement [24].

6.5.1.3 Key Results about Risk Priority


Ten interesting results based on similarities found in developers’ and acquirers’ priority
of each risk are detailed in Table 6.5. The first six results concern developers. There are
three risks for which the developers have high priority: selection effort ill-estimated,
insufficient OTS component documents and lack of information on vendor. The
118
developers perceived OTS software selection-related risk as a high priority. A possible
explanation is that they might take into account their experience and familiarity with
candidate OTS software when selecting OTS software [39]. Consistent with previous
explanations in this study, documentation is more needed by the developers to integrate
OTS components. Finally, developers need vendor-related information during the
software project, such as vendor support and customer service. For the risks of not
adaptable to requirements changes and lack of support, developers do not have low risk
priority, three high and one medium. For these two risks, the medium risk priority
comes from the developer perspective in case 1Dev regarding the limited control over
OTS software [5][40][41][137]. The sixth result is that the developers do not have high
risk priority (one low and three medium risk priorities) for the risk of complicated
multi-OTS components arrangement. This is because, as discussed earlier, the developer
did not own an OTS software license.

For acquirers, there are three results. There are two risks for which the acquirers do not
have high risk priority: maintenance planning unfeasible (for which they have two low
and two medium) and upgrade unfeasible (for which they have two low and two
medium). There are different responses regarding the risk of maintenance planning
unfeasible. For example, in the 1Dev case, the acquirer had a high risk impact due to the
difficulty of maintaining OTS software, but had a low control over the OTS software
vendor for maintaining OTS software. In the 3AcqTelco case, the acquirer perceived
themselves to have low risk control and impact because of not prioritising the latest
version of OTS software, and because of using a parameterised design. The acquirer in
the 3AcqTelco case for the risk of upgrade unfeasible had the same (low) level of risk
control and impact, and also the same reason as the risk of maintenance planning
unfeasible. The reason was the developed system did not prioritise the use of the latest
release of OTS software. However, the acquirer in the 2AcqBank case for the risk of
upgrade unfeasible had the same (low) level of risk control and impact, but for a
different reason, which was that upgrade would be feasible due to implementing testing
and bug fixing. The last result from the acquirers’ perspective is that they do not have
low risk priority (instead, they have three high and one medium) for the risk of
requirements not negotiable. In the 4AcqGovt case, the acquirer believed that software
requirements and development are the developer’s responsibility, as the acquirer

119
contracted the developer. The acquirer perceived this to be a high level of control. Thus,
the acquirer assumed that requirements-related risks did not give high impact to them.

Both stakeholders did not have a low risk priority for the risk of lack of OTS-driven
requirements engineering process. There are two examples showing that both the
developer (in the 1Dev case) and the acquirer (in the 4AcqGovt case) applied the OTS-
selection process in requirements engineering.

After summarising the above ten key results, there are six additional findings identified
from this study:

Finding 12: Developers have high priority for the risk of selection effort ill-estimated,
insufficient OTS component documents and lack of information on vendor.

Finding 13: Developers do not have low risk priority for the risks of not adaptable to
requirements changes and lack of support.

Finding 14: Developers do not have high risk priority for the risk of complicated multi-
OTS components arrangement.

Finding 15: Acquirers do not have high risk priority for the risk of maintenance
planning and upgrade unfeasible.

Finding 16: Acquirers do not have low risk priority for the risk of requirements not
negotiable.

Finding 17: Both stakeholders do not have low risk priority for the risk of lack of OTS-
driven requirements engineering process.

6.5.2 RQ3.4: How should risk responsibilities be assigned between developers


and acquirers for these risks?
All participants pointed out the importance of involving stakeholders and analysing
differences in risk perspectives [8]. However, the study identified various challenges to
this:
 No specific method was in use to analyse differences in risk perceptions.
 Different approaches were used to define risk responsibility.
 Different roles were defined for risk management.

120
 No formal risk management approach was shared by stakeholders.
 Different techniques were in use to identify and analyse risk (common risk
identification techniques included brainstorming and discussion).
 Different techniques were used to prioritise risks.
 Stakeholders were not always able to predict other stakeholders’ perceptions.

This study proposes a framework to collect, analyse, compare and discuss differences in
risk control and impact perceptions between the participants and their stakeholders. The
framework outputs are stakeholder involvements and risk priorities for each
stakeholder. Risk responsibility is also proposed to belong to the stakeholder who has
higher risk control. This is because high influence and power over risk is needed to
control key decisions and task implementation [31][32][33], to mitigate risk. The
responsibility should be assigned to a stakeholder who has the competence and capacity
needed to discharge the responsibility [34][35]. However, if both stakeholders have the
same level of risk control, the risk responsibility is assigned to both stakeholders. These
outputs and risk responsibilities are then proposed to a developer and acquirer for
agreement. If both stakeholders can agree on the outputs and risk responsibilities, the
risk responsibilities are assigned as agreed; otherwise, both stakeholders can discuss the
proposed risk responsibilities by considering the analysis. However, although the
framework was designed to be used by a developer and acquirer, to avoid the effects of
learning to use [131] the framework, the case study participants were only asked to
complete the framework inputs and to agree on the framework outputs and proposed
risk responsibilities.

Table 6.5 shows that all participants and their stakeholders agreed on proposed risk
responsibilities and risk priorities. In discussing the agreement of proposed framework
outputs and risk responsibilities, there was initially one participant stakeholder (in the
4AcqGovt case) who did not agree with the five proposed risk responsibilities, but this
participant stakeholder finally agreed after discussing the process of determining risk
responsibility in the framework with the researcher.

This study found that the sources of risk responsibility are software project contract,
knowledge of risk, negotiation, communication, organisation policy, and resource and
capacity. Project contracts, knowledge on risk, and organisation policy can be classified

121
as formal controls, while negotiation and communication can be classified as informal
controls [84][82]. Knowledge can complement control [84], and competence and
knowledge affect the choice of controls [82], while resource and capacity were needed to
control risks. Therefore, the framework supports responsibility model [81][78][34][35],
which assigns responsibilities to the stakeholder who has competence and capacity
needed to discharge the responsibility. Defining responsibilities could help identify roles
[81][78] [34][35] to mitigate risks.

The results show that the responsibilities of the 11 risks vary as summarised in Table
6.5. The developers were assigned to have risk responsibilities in eight of the risks, the
acquirers in six of the risks, and both stakeholders in all 11 risks. See Table 6.5 under
the heading of Number of Risk Responsibilities for the risk of lack of OTS-driven
requirements engineering process, all participants and their stakeholders agreed on
proposed risk responsibility, which is ‘both stakeholders’. However, they disagreed for
other risks. One of the possible explanations why all participants and their stakeholders
did not agree is because stakeholder perceptions vary based on individual's or
organisation's background, experience, need and expectation [18][19]. This may also be
why participants were not able to predict other stakeholders’ perceptions.

Two of the eight risks (insufficient OTS component documents and maintenance
planning unfeasible) for which the developers had responsibility, the risk
responsibilities are in the developers and both stakeholders. Documentation is more
needed by the developers to integrate OTS components and also to maintain the system.
However, the acquirers have more responsibility for the risk of complicated multi-OTS
components arrangement than the developers. A possible explanation is because the
acquirers owned the license for OTS software.

Three participants changed their input of risk perception (provided in Section 6.4.1). As
mentioned in the data collection section, the changes of the framework’s inputs, risk
control and/or impact, happened before the agreement process in the interview. There
are two possible explanations for the changes. Firstly, the participants in case 1Dev,
3AcqTelco and 4AcqGovt got more understanding of the changed inputs during the
agreement process because the participants could confirm their understanding about
risks under study with the researcher during the interview. Secondly, the participants in
case 1Dev, 3AcqTelco and 4AcqGovt 4 were affected by their stakeholder answers of
122
the framework’s inputs. The first type of changes were data-driven and the second type
of changes either data-driven or expectation-driven [138].

If a stakeholder does not agree with other stakeholders’ perception on the proposed
framework outputs, the disagreement should be discussed (step 5 of the framework).
This discussion builds consensus and provides shared understanding, and is in line with
a study comparing software architectures [24] and risk management [22][23]. In sum,
the framework provides structured and methodological steps to define a risk and role
responsibility by comparing risk perceptions of both stakeholders.

6.5.3 Evaluation Criteria for Risk Assessment Findings


Based on the evaluation criteria for risk assessment [133][134] listed in Section 6.3.2,
all participants indicated that the framework could be able to be applied to different
projects. The participants also thought that the framework could be modified by adding
other risk attributes (e.g. cost/benefit) and by quantifying risk. The participants reported
that the framework had a sufficient feasibility of input data. There was a need to better
define and justify the levels of risk control and impact (according to two out of four
participants). The participants also noted that scope of completeness of the framework
would need to be adjusted for a more detailed risk assessment. However, two
participants argued that the framework was simple to use.

According to the participants, the framework could also cover various and different
risks because the framework provided an input template and did not pre-define risks to
be analysed. The list of risks in the questionnaire could be adjusted. All participants
indicated that the framework could represent real risks; half of the participants reported
that the framework could be used to compare stakeholder perspective and be used to
propose risk responsibilities. Overall, all participants agreed that the framework outputs
could be used to manage risks, and help to inform scope of work (SoW) about
responsibilities of process-related risks.

123
6.5.4 Improving the Proposed Framework
Participants proposed improvements by comparing existing risk management practices
(found in this case study) with the framework, and making use of suggestions from all
participants. Three main improvements were proposed as follows.
1. Stakeholders should use the framework.
All stakeholders should be involved in assigning risk responsibilities (three cases).
The other case’s participant pointed out that the framework should be initiated by
the software acquirer, but that the framework would be more effective if it was
initiated by someone with the personal strength to reconcile differences in risk
perceptions [20].
2. Stakeholders should compare risks using the framework. The outputs proposed were
stakeholder involvement and risk priority (including risk responsibilities).
Three framework improvements were suggested:
 Risks should be analysed at the beginning of a project
 There is a need for a shared understanding of the definition of control and
impact. Therefore, the framework needs an introduction, a glossary of used
terms (such as control and impact) and related context, verification of
stakeholder’s answers, and a session to discuss differences of understanding of
definition of control and impact. The introduction and verification are already in
the steps of the framework.
 The framework could include other risk attributes (such as cost/benefit) and risk
quantification for more detailed risk assessment.
3. The framework could be used to detail the level of stakeholder responsibility in
software contract and the scope of work (SoW).
The framework could also be used to clarify understanding of each stakeholder’s
roles and responsibilities in the ToR. Based on the participants’ opinions given
above, the framework has potential use for reconciling different process
perspectives by defining roles for processes in OTS-based custom software projects.

This study validates the potential uses and benefits of the framework. The aims of
analysing risks are to develop shared understanding, shared responsibility, stakeholder
commitment and shared perspectives; to avoid conflict; and to focus and to prioritise
tasks. Furthermore, the framework provides early understanding of risk perceptions,

124
facilitating the planning of a software project and formulating the software contract.
Overall, all participants agree that the framework can be used to initiate and support the
discussion to reconcile different risk and process perspectives between stakeholders.

Based on our case study results, compared with standard clauses on a software contract
[139] or existing organisational policies, stakeholders were expected to clarify
understanding of each stakeholder’s roles and responsibilities in the terms of reference
(ToR) (see Recommendation to improve risk responsibility from participant
4AcqGovet in Table 6.7). It was assumed that the tender winner had understood the
ToR. However, the tender winner tended to be overoptimistic and the winner did not
discuss who would control and be impacted by risks. Thus, the framework provides a
bi-directional approach to discuss and negotiate risk responsibility.

A summary of the case study evidence and related propositions and literature related to
evaluation, and improvement of the framework is presented in Table 6.9

Table 6.9 Summary of case study evidence and related propositions/literature

Evidence of related case study question (question Related


ID) propositions/literature
Project information is described in Section 6.3.5 -
(Q1.1)
Importance to involve stakeholder in risk management Support proposition P1
(Q1.2.1)
Importance to analyse differences in risk perspectives Support proposition P2.
between stakeholders. No specific method used to Challenges for proposition P2.
analyse differences in risk perceptions (Q1.2.2)
Different approaches to define risk responsibility Challenge for proposition P1
(Q1.2.3)
Different roles in risk management (Q1.2.4) Challenge for proposition P1
Framework outputs are presented in Table 6.5 (Q1.2.5) -
Participant information is described in Section 6.3.5 -
(Q1.3)
Results of evaluation criteria of risk assessment are -
presented in Section 6.4.2 (Q2.1)
No formal risk management shared by stakeholders Challenge for proposition P1,
(Q2.2.1) P2
Different techniques used to identify and analyse. Challenge for proposition P1,
Shared risk identification techniques: brainstorming P2
and discussion (Q2.2.2)
Different technique used to prioritise risks (Q2.2.3)Challenge for proposition P1,
P2
The sources of risk responsibility: software project Support proposition P4 and

125
contract, knowledge on risk, negotiation, [84][82]
communication, organisation policy, and resource and
capacity (Q2.2.4)
The use of the framework to detail scope of work and Improvement for proposition
software contract, detail of stakeholder and shared P1, P2, and solution for P3
responsibility (two participant), shared understanding
of the definition of control, adding required attribute to
analyse and clarification the understanding of each
stakeholder’s roles and responsibilities on (ToR)
(Q2.2.5)
Risk expectation driven by risk perception (two This finding supports
participants) (Q2.2.6) [18][138]
Understanding risk control, impact and responsibility. -
Adding required attribute to analyse (Q2.2.6)
Integration of the framework in the beginning of the Improvement for proposition
software project (three participants) (Q2.2.7) P1, P2, and solution for P3
Inability to predict other stakeholder perceptions Stakeholder perceptions vary
(Q2.2.8.a) [18]
Comparison of actual risk perceptions better than Objectively reviewed decision
perception-based risk (two participants). Structured, [140]
methodical and measured approach (Q2.2.8.b)

6.5.5 Case Study Findings Related to OTS-based Custom Software Project


Stakeholders
There was one risk related to responsibility among the acquirer, developer and OTS
software vendor found in this study. The participant in case 4AcqGovt illustrated a risk
related to OTS software vendors when a system was infected by a computer virus and
need to be re-installed. This risk could only be solved by the OTS software vendor. The
question was who would be responsible to mitigate this risk, the acquirer or developer?
According to the participant in case 4AcqGovt, the framework could solve this case by
showing that both stakeholders might not prepare a mitigation plan or have the
resources to mitigate this risk. The framework could inform the acquirer that its
developer did not have the capability to re-install the software.

6.5.6 Case Study Findings Related to Expectations of Level of Risk


(Control/Impact)
Two participants discussed expectations of level of risk control and impact. The
participant of case 1Dev pointed out that its acquirer expected the participant to have
high risk control/impact. Another participant, in case 3AcqTelco, expected its developer
to have high risk control/impact after discovering its developer’s answer to another risk

126
related to the first risk. The participant in case 3AcqTelco asked the researcher, “Why
does the risk of maintenance planning being unfeasible have high risk control and
impact, but the risk of upgrade being unfeasible have low risk control and impact?”).
The participant believed that there was a relation between these two risks. These
findings support propositions of relationship between perception and expectation
[18][138] (see Table 6.9). These findings also indicate that identifying stakeholders’
risk expectations can help to ensure risks are addressed [38].

6.5.7 Case Study Findings Related to Processes (Role and Responsibility)


Investigating process-related risks in this study could lead us to identify and define roles
or responsibilities for software processes. As can be seen in Table 6.7 (Factors
influencing risk responsibility and Recommendation to improve risk responsibility), the
framework has a potential use to inform the software contract and scope of work. The
findings showed that the framework could also be used to clarify understandings about
roles and responsibilities. The potential use of the framework to analyse differences in
the process perspective is consistent with one of the uses of stakeholder analysis in
software-related projects, which is to identify process roles [36][37][13].

6.6 Threats to Validity

6.6.1 Construct Validity


To minimise threats to construct validity, related and relevant theories were used prior
to case study design. Interview transcripts were reviewed by the participants [86]. Data
triangulation was also used to increase construct validity [86]. The questions used in this
study were reviewed by two internal experts.

6.6.2 Internal Validity


To control confounding factors [131], this case study involved developers and acquirers
as the participant and their stakeholder in the case study. Moreover, the case study
participants were respondents of our survey (detailed in Chapters 4 and 5). To avoid the
effects of learning to use the framework [131], the participants and their stakeholders
were only asked to complete the framework inputs. Therefore, internal validity of this
study was expected to increase as the case study focused on evaluating and improving
of the framework, and not learning how to use the framework [131].

127
In all cases, even though there were possibilities that inaccurate answers were provided
to meet the expectations of others, these answers were minimised in this research. First,
one of the benefits using a completed project in each case is to encourage the
participants and their stakeholder to give inputs of the framework based on facts. Our
case study has found that there were no specific methods used to analyse differences in
risk perceptions. Therefore the use of retrospective risk perceptions in the participants'
completed project was expected not significantly influencing opinions given by the
respondents. Second, the participants and their stakeholder were only asked to provide
their risk control and impact (as inputs of the framework) and not others’ expected risk
control and impact. In addition, when each party provided its answers, it would not
know other's answers.

6.6.3 External Validity


The use of a case study limits the generalisability of its results to theoretical
propositions and not to populations [86]. However, all participants agreed that this
study could be used in other OTS-based custom projects and other project types.

6.6.4 Reliability
To increase reliability [86], this study followed existing case study guidelines [86]
[131][132][129]. This study also recorded and maintained a case study database to
increase reliability [86]. To simplify the presentation of the thesis, we decided to not
include quotes of the participants (there is one quote from the participant (Section
6.5.6).) and to describe and summarise the case study data and outcomes. To support
audit trail, we provide cross-case analysis from data collection 1 and 2 in Appendix E.

6.7 Summary
This chapter has investigated differences in perception of 11 risks of OTS-based custom
software projects between developers and acquirers by comparing their perceptions of
risk control and impact. It used a multi-case study method. Based on the method used to
analyse the survey data in Chapter 5, a risk framework was developed to collect,
compare and analyse these differences. The framework outputs are stakeholder
involvements and risk priorities for each stakeholder. There were four participants in the

128
case study: these were among 42 invited respondents of our survey (see Chapters 4 and
5), who agreed to participate and could also involve their stakeholder in this study.

This study yielded 17 detailed findings related to levels of risk control and impact, and
proposed risk priorities. As the participants were not able to predict other stakeholders’
perceptions, due to differences in individuals, their organisations’ background,
experience, need and expectation [18][19], these findings can serve as an initial
checklist for understanding other stakeholders’ risk perspectives.

Five findings about risk perceptions are specific to developers. Firstly, developers often
decide to use existing stable OTS versions to control the impact of system maintenance.
Secondly, developers without an OTS software license often lack support from OTS
software vendors.

There are three different findings regarding risk priority. Firstly, developers have high
priority for the risk of selection effort ill-estimated, insufficient OTS component
documents and lack of information on vendor. Secondly, developers do not have low
risk priority for the risks of not adaptable to requirements changes and lack of support.
Lastly, developers do not have high risk priority for the risk of complicated multi-OTS
components arrangement.

Eight findings about risk perceptions are specific to acquirers. The first is that risks with
high impact make acquirers apply higher control over these risks. Secondly, acquirers
can use OTS selection results to define system requirements. Thirdly, in maintenance,
acquirers can anticipate future changes by asking developers to use a parameterised
design (for example, setting configuration files). In addition, there are two findings
related to the contract: to lower risk impact, acquirers can contract an external party;
however acquirers will have lower control when they contracts developers and use OTS
software. With an OTS software license, acquirers can help negotiate with an OTS
software vendor. There are two findings related to risk priority. Acquirers have high risk
priority for the risk of requirements not negotiable. Lastly, acquirers have low risk
priority for the risk of maintenance planning and upgrade unfeasible.

The other four findings about risk perceptions relate to both stakeholders. Firstly, using
a software contract to control risk leads to consequences regarding what must be agreed

129
in the contract. Secondly, stakeholders have different concerns about their limited
control over OTS software: developers tend to have limited control over OTS software
in the OTS selection process and OTS integration, while acquirers tend to have limited
control over OTS software in maintenance, after the contract ends. Thirdly, both
stakeholders have risk responsibility if an acquirer agrees on OTS software selection,
and a developer’s proposed system design decision. Lastly, both stakeholders do not
have low risk priority for the risk of lack of OTS-driven requirements engineering
process.

The framework proposed a default position about which stakeholder should be


responsible for a risk, based on which stakeholder has higher control and power over each
risk [31][32][33]. Higher risk control indicates that a stakeholder has the competence and
capacity needed to discharge the responsibility [34][35]. This study found that the sources
of risk responsibility are software project contract, knowledge of risk, negotiation,
communication, organisation policy, and resource and capacity. Knowledge can
complement control [84], and competence and knowledge affect the choice of control
[82], while resource and capacity were needed to perform control over risks. The
framework was used to facilitate a discussion between both stakeholders to assign risk
responsibility based on the proposed risk responsibility. When stakeholders disagreed
with the proposed framework outputs and risk responsibilities, or another stakeholder’s
perspective, the discussion was useful to elicit the reason for this disagreement [24]. All
participants and their stakeholders agreed on the proposed risk responsibilities and risk
priorities.

In the context of a specific project, the framework provides an explicit recognition of


different risk perceptions, to inform risk management negotiation through dialogue,
deliberation and communication [16]. The framework may help stakeholders reduce ‘gut
feeling’ judgments and ensure decisions on the assignment of risk responsibilities are
more objectively reviewed [140].

Although these findings might not be very surprising and are not generalised beyond
OTS-based custom software projects and the 11 risks under study, the findings do
provide insight into developers’ and acquirers’ perceptions of risk control and impact.
The framework in this study added substantially to our understanding of how to provide
a methodical approach to analyse different process-related risk perceptions [16]. The
130
framework is expected to increase understanding of risk responsibility for both
stakeholders by discussing, negotiating [24] and clarifying risk responsibility in two
directions instead of deriving it from standard contract clauses [139] or existing
organisational policies. Moreover, the framework has a potential use for analysing
differences in process perspectives, by identifying stakeholders’ roles and their level of
involvement [36][37][13]. We also believe that the method used in this study could be
applied to non-OTS-based custom software projects, to analyse differences in process-
related risk perceptions.

From a methodological perspective, another contribution of this study is to extend the


use of stakeholder analysis. Previously, stakeholder analysis has only been used to
identify stakeholders, and to identify their roles, their level of involvement [36][37][13]
and their risks [9][38][10], without comparing their different perceptions.

Our study proposed a default position about which stakeholders should be responsible for
a risk based on which stakeholder has higher control and power of each risk [31][32][33].
Further investigation on other related factors influencing risk responsibility is strongly
recommended. To analyse differences in risk perception involving stakeholders other
than developers and acquirers, the framework should be adjusted, evaluated and
validated. There are another two limitations. Firstly, there was possibility of subjective
understanding of the concepts of risk control and impact by all participants and their
stakeholders. Secondly, one of four participants’ stakeholders did not give detailed
opinions about risk perspectives.

131
7 Discussion and Conclusion

In a software project, risks affect all stakeholders [7][8][9][10]. Software developers


and acquirers perceive risk differently [16] because they have different backgrounds,
experiences, needs, expectations [18], and goals and structures [19]. It is necessary to
reconcile stakeholder perceptions when managing processes and risks in software
projects [16][20]. Prior studies have noted the importance of reconciling user and
project manager perceptions of IT project risks [16], and reconciling perspective
mismatch in managing software project processes [20]. However, these prior studies do
not provide detailed steps on how to reconcile differences in stakeholder perspectives.

This research has investigated OTS-based custom software project, and has identified
risks involved [16], and analysed differences in risk perceptions and proposed
stakeholder responsibility for risks [22]. This has been done, not only from developers’
perspective, but also from acquirers’ perspective. Firstly, we investigated the processes
involved in OTS-based software acquisition using a systematic mapping study. Then,
we used a second systematic mapping study to identify risks of OTS-based custom
software projects from developers’ and acquirers’ perspectives. Finally, we extended an
existing stakeholder analysis framework [25][12][26][14][13][10] to compare, analyse
and discuss [22][23][24] risk perceptions between developers and acquirers of OTS-based
custom software projects in a survey and a multi-case study.

This chapter has five sections. Firstly, it reviews the research contributions. Then this
research is compared with related work. The next section discusses implications for
research, including future work. The fourth section discusses implications for practice
and the last section identifies limitations. In the comparison with related work and
research contribution sections, we refer to the research questions:

RQ1: What are the processes of OTS-based software acquisition?

RQ2: What are the risks in OTS-based custom software projects?

RQ3: How should software developers and acquirers analyse differences in


perceptions of process-related risks in OTS-based custom software projects?

132
7.1 Research Contributions
This thesis makes several findings and contributions to empirical software engineering
knowledge, as follows.

RQ1 contributions: We identified six OTS-based software acquisition processes.


From the first mapping study, we identified six OTS-based software acquisition
processes: decision-making to make or buy software, architectural decision-making,
OTS selection, managing relationships between developers and acquirers, managing
relationships between OTS adoption and acquirers’ organisations, and managing
relationships between OTS vendors and developers. Our case study findings validate the
existence of these processes. These processes should be considered by an acquirer when
acquiring OTS-based software, in addition to the existing software acquisition process
standard [6].

Although our mapping study results have shown that both the software acquisition
process standard and OTS-based software acquisition have the same overall process
lifecycle, there are process differences between them. The first difference is that in
OTS-based software acquisition, architecture and OTS selection criteria are defined and
adjusted iteratively in conjunction with software requirements specifications. Secondly,
there are some OTS adoption characteristics that must be considered when acquiring
OTS software (for example, the credibility of the decision maker and previous exposure
to acquisition decisions) [108]. Thirdly, managing relationships between OTS vendors
and developers can provide benefits in commercial negotiations with acquirers [109].

Based on the mapping study results, there are three processes that could be added into
(generic) software acquisition and OTS-based software acquisition: decision-making to
make or buy software, architectural decision-making during software acquisition, and
managing relationships between developers and acquirers. Based on the literature
review, there are three processes in common between acquirers and developers of OTS-
based software: make vs. buy decisions, OTS selection and vendor interaction. A better
understanding of OTS-based software acquisition and development processes, and of
differences in backgrounds, experiences, needs, expectations [18], goals, and structures
[19] should improve understanding of differences in process-related risk perceptions
between both stakeholders.

133
RQ2 contributions: We identified risks of OTS-based and custom software
development and acquisition.
We have identified and classified risks of OTS-based and custom-based software, not
only from developers’ perspective but also from acquirers’ perspective. These risks
have been classified into 17 categories: planning, requirements, design, integration and
testing, system lifecycle, maintenance, project closure, software, OTS products, cost,
environment, people, systems engineering, vendor relationships, project management,
contract and legal.

The results consist of specific and generic risks in OTS-based and custom software
acquisition and development. OTS-specific risks can be found in all risk categories. The
results show that most risks found in OTS-based software acquisition have the same
concerns as risks in OTS-based software development. In both custom software
acquisition and development, specific risks are related to contracted or outsourced
software development projects. Technical-related risks are found less often in
acquisition and project management-related risks are found less often in development.
There are generic risks, which are common to all software projects, in 13 risk categories
of OTS-based software development and 14 risk categories of OTS-based software
acquisition except in the risk category of OTS products. In this study, poor requirement
definition and analysis, found in the requirements category, is the only generic risk
found that was shared between OTS-based and custom-based software development and
acquisition.

RQ2 and RQ3 contributions: The survey and case study show that in OTS-based
custom software projects comparing and discussing risk perceptions about risk
control and impact between both stakeholders enhance our understanding of their
risk perceptions and differences in their risk perceptions.
In our survey and case study, we focused on 11 process-related risks that should be
relevant not only to developers but also to acquirers in OTS-based custom software
projects. There are four detailed findings about comparing and discussing risk
perceptions between stakeholders.

134
 RQ2 contributions: There are different perceptions about risk occurrences in
OTS-based custom software projects between the developer and acquirer
perspectives (survey-based findings).
The survey partially confirms a previous study [27], and complements it with findings
about the acquirer perspective. The results of the survey focus on 11 process-related
risks, and show that more of these risks occurred more frequently in software
acquisition than in software development (nine vs. five risks out of these 11 risks; as
shown in Table 4.11). Six risks occur more frequently in software acquisition than in
software development: selection effort ill-estimated, not adaptable to requirement
changes, requirements not negotiable, maintenance planning unfeasible, upgrade
unfeasible and lack of information on vendor. Two risks occur more frequently in
software development than in software acquisition: insufficient OTS component
documents and a lack of vendor technical support and training. There are three risks
that occurred equally frequently in both software development and acquisition:
complicated multi-OTS component arrangement, lack of OTS-driven requirements
engineering process, and reduced control of future evolution of the system.

 RQ3 contributions: There are similarities and differences in respondents’


(developers and acquirers) perceptions about which stakeholder is affected by
and can control risks (survey-based findings)
Comparing the results of respondents’ perceptions about which stakeholder is affected
by and can control risks have shown that both stakeholders can control, and are most
affected by, risks about requirements negotiations. Developer respondents perceived
themselves to best control risks, but perceived either themselves or both stakeholders to
be most impacted by risks. Acquirer respondents agreed that their developers can best
control risks, but perceived both stakeholders as most impacted by risks, except for risks
of lack of support and reduced control of future evolution of the system. For the risk of
lack of vendor technical support and training, there was disagreement about who is
most impacted by the risk (usually each stakeholder reported themselves). With regard
to the risk of reduced control of future evolution of the system both stakeholders agreed
that the acquirer is most impacted.

A comparison method was developed to analyse the survey results. We proposed a


default position about which stakeholder should be responsible for risks, based on

135
respondents’ agreement about which stakeholders have high control of each risk.
Responsibilities for all risks (except requirements not negotiable) can be proposed to
belong to the developer. With regard to requirements not negotiable, we proposed that
both stakeholders share responsibility for the risk. The proposed risk responsibility does
not consider all potentially-related factors to allocate risk responsibility, but the method
is consistent with responsibility models [81][78][34][35]. A responsibility model
describes responsibilities within a system under development, agents assigned to these
responsibilities and resources used to discharge these responsibilities [78]. In addition, a
responsibility should also be assigned to the stakeholder who has the competence and
capacity needed to discharge the responsibility [34][35]. Competence and knowledge
affect the choice of control [82], while resource and capacity were needed to perform
control over risks.

 RQ3 contributions: We made 17 findings about level of risk control and impact,
and proposed risk priority (case study-based findings)
Based on the evidence from the cross-case analysis, it can be argued that each
stakeholder has their own and shared perceptions regarding the risks of OTS-based
custom software projects. The case study revealed 17 detailed findings related to levels
of risk control and impact, and proposed risk priorities.

There are five findings about risk perceptions specific to developers. Developers tend to
perceive risks related to OTS vendors (such as a lack of technical support and training,
a lack of vendor information, and OTS software documentation) as more important than
the use of multi-OTS components. Developers also perceive risks related to
requirements and OTS selection to be important. To control the impact of system
maintenance, developers can often decide to use an existing stable version of OTS
software.

Eight findings about risk perceptions are specific to acquirers. Generally, acquirers tend
to target higher control over a risk having high impact. Acquirers perceive risks related
to requirements as more important than risks related to maintenance and upgrade.
Acquirer can use OTS selection results to define system requirements. An acquirer can
lower a risk’s impact by contracting an external party. For example, all support-related
problems will be handled to an OTS software vendor. However, contracting a developer
and using OTS software will give acquirers less control over the risks. An acquirer can
136
negotiate with an OTS vendor to manage the use of and support for OTS software. To
anticipate future change, an acquirer can ask a developer to use a parameterised design
(for example, setting configuration files).

Four findings about risk perceptions relate to both stakeholders. Firstly, using a
software contract to control risks leads to consequences regarding what must be agreed
in the contract. As mentioned by a case study participant, its developer might not fully
understand a contract because they might only use it to win a project. Next, developers
and acquirers have different kinds of control over OTS software: developers tend to
have limited control over OTS software in the OTS selection process and OTS
integration because of not having the license of OTS software, while acquirers tend to
have limited control over OTS software in maintenance because of limited access and
knowledge over OTS vendor and software. Thirdly, there is a shared risk responsibility
for agreement on OTS software selection, and the developer’s proposed system design
decisions. Fourthly, both stakeholders have a high risk priority for the risk of lack of
OTS-driven requirements engineering process, which is related to how requirements
formulation is affected by the availability of OTS software and how requirements are
mapped to OTS software [77].

The above findings show that developers and acquirers have different concerns about
risk controls and responsibilities. Developers tend to focus on risks related to OTS
vendors, OTS software integration, OTS selection and requirements. Acquirers tend to
focus on risks related to requirements and OTS selection. A better understanding of
these different concerns can help stakeholders to analyse related risks.

 RQ3 contributions: The case study findings enhance our understanding of the
importance of risks from developers’ and acquirers’ perspectives by taking
account of the level of risk control and impact.
The case study findings show that: the higher a risk control, the more important the risk;
for risks with lower control, the higher the impact, the more important the risk is (this is
further discussed in the next section).

137
RQ3 contributions: A proposed framework to analyse differences in risk
perceptions and to assign risk responsibilities between developers and acquirers.
We proposed a framework to analyse differences in risk perceptions and to assign risk
responsibilities between a developer and acquirer. The framework was initially
developed in our survey to analyse the survey data of differences in perceived risks
between developers and acquirers (see Chapter 5). The framework extends stakeholder
analysis [25][12][26][14][13][10] and can be used to compare, analyse and discuss level
of risk control and impact between a developer and acquirer. The framework proposes a
default position about which stakeholders should be responsible for a risk, based on which
stakeholder has higher control over the risk. In our case study, the framework was used to
facilitate a discussion between both stakeholders to assign a risk responsibility based on
the proposed risk responsibility.

Our findings show that all participants and their stakeholders agreed on proposed risk
responsibilities and risk priorities. As confirmed by the case study findings, a
responsibility should also be assigned to the stakeholder who has the competence and
capacity needed to discharge the responsibility [34][35] and should be agreed between
both stakeholders. In the case that stakeholders disagree with the proposed framework
outputs and risk responsibilities, or the other stakeholder’s perspective, the discussion is
useful to elicit the reason for this disagreement [24]. In the context of a specific project,
the framework provides an explicit recognition of different risk perceptions, to inform risk
management negotiation through dialogue, deliberation and communication [16]. The
framework could help stakeholders reduce ‘gut feeling’ judgments and to ensure that
decisions regarding the assignment of risk responsibilities are more objectively reviewed
[140].

From a methodological perspective, another contribution of this research is to


demonstrate that stakeholder analysis can be used to reconcile differences in risk
perceptions between software project stakeholders. Previously, stakeholder analysis has
only been used to identify stakeholders, to identify their roles, their level of involvement
[36][37][13] and their risks [9][38][10], without comparing their different perceptions.

Overall, the framework provides methodical and practical guidelines to reconcile


differences in process and risks perspectives between developers and acquirers in OTS-
based custom software projects.
138
7.2 Comparison with Related Work
The existing software acquisition standard, IEEE Standard 1062-1998 Edition:
Recommended Practice for Software Acquisition [6], can be applied to software
acquisition regardless of the size and complexity of the software [6]. However, this
recommended practice is more applicable for fully developed software and must be
tailored to other types of software acquisition [6]. From the literature, there are
additional OTS-specific processes that must be included in OTS-based software
development: make vs. buy decisions, OTS selection, OTS product familiarization,
learning and understanding OTS products, OTS integration and vendor interaction
[39][40][41]. As software projects involve and interact with acquirers
[2][141][142][143][144], it is interesting to also investigate the role of acquirers in
OTS-specific processes when acquiring OTS software.

The findings for RQ1 identify six OTS-based software acquisition processes: decision-
making to make or buy software, architectural decision-making, OTS selection,
managing relationships between developers and acquirers, managing relationships
between OTS adoption and acquirers’ organisations, and managing relationships
between OTS vendors and developers. Our case study results and findings provide
empirical evidence that these processes exist in practice. For example, the acquirer in
the 1Dev case performed OTS software selection. There are three processes in common
between OTS-based software acquisition and development processes: make vs. buy
decisions, OTS selection and vendor interaction. The responsibilities of these common
OTS-based software processes will vary depending on stakeholders’ goals and
structures [19].

This research focuses on OTS-based process-related risks, as proposed in [77][74].


However, this research does not study risk mitigation. Nonetheless, our proposed
framework needs acquirer involvement, which one of risk mitigations previously
proposed for risk management of OTS-based custom software projects [74][27].

Instead of using a purely technical approach [145][76], we focus on human and social
factors in risk management of OTS-based custom software projects. This research
investigates risks from human and social perspectives [146][147] by taking into
account developer and acquirer perspectives. To manage risks effectively, it is
important for all software project stakeholders to understand the risks [16]. To initially
139
answer RQ2, our study identified and classified risks of custom and OTS-based
software, from both perspectives. These results complement existing lists of software
project risks, which are from the perspectives of project managers [30][58][57][71][59]
and risk team members [22]. Our work is also in line with value-based software
engineering because it deals with different stakeholders’ perspectives to overcome the
limitations of a value-neutral approach [148][149]. An example of the value-neutral
approach is when every requirements, use case, object and defect is treated as equally
important [148][149]. In this study, perspectives of both stakeholders are investigated
because risks can affect both stakeholders [8].

This research supports previous works [8][22][23] arguing that, to manage risks, key
stakeholders must be involved from the beginning of software projects. However, there
are several differences between this research and these prior works. Firstly, compared
with Schmidt et al. [8], who argued to cooperatively manage risks, our research gives
detailed procedures to asses differences in risk perceptions between both stakeholders.
Secondly, prior research on control requires a ‘risk advisor’, whose tasks are to mediate
between both stakeholders and to manage risk [22][23]. In contrast, our risk framework
only includes developers and acquirers as the framework users, because a study by Keil
et al. [66] noted that an ‘outside consultant’ did not identify more risks than insiders nor
make more risk-averse decisions.

To analyse differences in risk perceptions, we proposed a risk framework for answering


RQ3. In our framework, different perceptions are compared and discussed to facilitate
negotiation. This is a widely used approach in software engineering to reconcile
conflicting perceptions [149]; it is used in risk management [22][23], software
architecture [24] and requirement elicitation [150][149]. Risk assessment as a group
task is more reasonable than as a task for an individual project manager because
considering all stakeholders' perceptions increases stakeholder commitment and
acceptance [22][23]. Heemstra et al. [22][23] proposed a collaborative risk assessment
approach by first confronting the risk perceptions of each risk team member, discussing
risk probabilities and effects, and finally agreeing on the most relevant risks and risk
reduction actions, including risk responsibilities. In contrast with Heemstra et al.
[22][23], our risk framework supports a structured discussion [24] using the framework
outputs (stakeholder involvements, risk priorities, and proposed risk responsibilities) for

140
making risk responsibility decisions. The benefits of using structured discussion are that
stakeholders can better understand and learn about other stakeholders’ perceptions,
enabling them to make a more informed decision [24]. In requirements management, the
EasyWinWin approach is a requirement elicitation and negotiation methodology, which
supports expectation management, adopts prioritisation techniques and is supported
with groupware tools [150][149]. There are two tactics from EasyWinWin relevant for
our framework: structured procedures to handle conflicted stakeholder goals [150][149];
and the use of groupware to increase productivity of collaborative work [151]. Below
we discuss how our framework compares and analyses differences in risk perception of
OTS-based custom software projects between developers and acquirers for answering
RQ3.

In the case study (Table 6.5), four developers identified ‘collaborate’/high risk control and
impact for three risks (selection effort ill-estimated, insufficient OTS-component
documents and lack of information on vendor). It is interesting to compare these risks to
the related risks in the survey of risk perceptions in the previous chapter. These case study
results corroborate our survey results (detailed in Chapter 5), which showed that
developer respondents perceived that they could best control and were most impacted by
these risks. However, in the survey, the developer respondents perceived that for the risk
of selection effort ill-estimated, both stakeholders were most impacted. This result is
confirmed by our case study finding that when acquirers agree with developers on
proposed OTS software, then the responsibility should be shared (detailed in Section
6.5.1.1). Regarding the risk of there being insufficient OTS component documents, our
survey and case study findings are consistent with Mahmood and Khan’s [124] study
reporting that component documentation is needed when integrating OTS software. In
addition, from the developers’ point of view, our survey and case study findings support
the previous studies [74][27] that pointed out the importance of monitoring information
from OTS vendors regarding product upgrades and technical support, not only during
development but also during maintenance.

In our framework, the priority of a risk is based on the result of mapping it into a 2x2
control/impact matrix. Risk priority shows which risks are most important to address
[30]. Instead of using risk-exposure quantity [30] or multiplication on ordinal scale [38],
the framework prioritises risk using available information [38] by comparing values of

141
the mapping results from a developer and acquirer. In contrast with previous studies
[16][59][128] reporting that stakeholders tend to perceive the importance of certain risks
as higher than others if they cannot control the risks, our case study found that the higher
control over a risk, the more important the risk. However, our framework corroborates
previous studies [16][59][128] showing that for risks with lower control, the impact of the
risks directly affects the importance of the risks. In short, the higher the impact, the more
important the risk is.

To analyse differences in risk perception of OTS-based custom software projects


between developers and acquirers (RQ3), our framework extended an existing
stakeholder analysis approach [25][12][26][14][13][10]. Even though using stakeholder
analysis is considered important in requirements engineering for identifying stakeholder
[37][36][152], our framework does not support stakeholder identification. We assume
that the relevant stakeholders, a developer and an acquirer, have been identified. Instead
of prioritising stakeholders (as in requirements elicitation) [37][153][154][155], our
framework prioritises risks for each stakeholder. In addition, our framework maps levels
of risk control and impact into kinds of stakeholder involvement using the
control/impact matrix. Our framework can be used to identify stakeholders’ roles, and
their level of involvement [36][37][13] and to understand the importance of risks.

Our framework assigns a risk responsibility, based on stakeholder agreement, to the


stakeholder who has higher risk control (RQ3). This is because higher control and
power over risks is needed to control key decisions and act [31][32][33] to mitigate the
risks. The case study results show that all participants and their stakeholders agreed on
proposed risk responsibilities. Furthermore, our case study finds that the risk
responsibility should also be assigned to the stakeholder who has knowledge, resources,
and capacity to deal with risks. This confirms the responsibility model [34][35] which
assigns a responsibility to the stakeholder who has the competences and capacity needed
to discharge the responsibility. Knowledge can complement control [84], and competence
and knowledge affect the choice of control [82], while resource and capacity are needed
to perform a control.

Our findings show that analysing project stakeholders can be integrated into software
project risk management. Two examples are similar to our approach. Firstly, a Software
Development Impact Statement (SoDIS) associates every software development task
142
with involved stakeholders and then uses structured questions to assess the specific
project risks generated by that particular association [9]. Therefore, our framework
could provide better justification in cases when SoDIS might assign more than one
stakeholder to be responsible for a task-related risk. Secondly, the Outcome-Based
Stakeholder Risk Assessment Model (OBSRAM) provides guidance in identifying and
managing project risks arising from project stakeholders [10]. OBSRAM identifies
stakeholder influence and project's impacts on stakeholders, and then assesses and
prioritises stakeholder risks using risk scope, domain scope, risk severity and risk
probability. Compared to OBSRAM, in which risk prioritisation does not consider
stakeholder influence and project’s impacts on stakeholders, our framework prioritises
risks based on stakeholders’ risk control and impact.

There are similarities between our framework and two studies of role assignment in
software development: role-based software development (RBSD) [156] and the
Capabilities-Oriented Software Process Model [157][158]. The common steps between
our framework and RBSD are negotiation and assignment of responsibility. However,
RBSD uses more factors to assign roles (in our framework called responsibilities):
rights, responsibilities, accessibilities, and collaboration methods. The Capabilities-
Oriented Software Process Model assigns people to roles according to their capabilities,
and capabilities demanded by the roles [157][158]. In The Capabilities-Oriented
Software Process Model, the first step to assign a role starts by comparing a personal
and role profile. The next step is identifying the closest match between the personal and
role profiles. Another capability-based role assignment method uses Little-JIL to assign
roles based on available and matching capabilities to perform a task [159]. A
mathematically-based formal model integrating the Delphi method could help project
leaders to take into account as many factors as possible in assigning individuals to
project roles [160]. It is also interesting to consider the use of network theory in the
problem of task allocation for coordinating software development [161] in our future
work.

One of the aims for eliciting and reconciling differences in stakeholders’ perceptions is
to support expectation management [150][149]. Our study on RQ3 found that
participants had invalid expectations about each others’ risk perceptions. Therefore in

143
our future work, we would like to integrate our framework with stakeholders’
expectations, as in [150][149][38][64].

7.3 Implications for Research


Differences in risk perceptions between software project stakeholders can be reconciled
[16][20] by comparing and discussing [22][23] these differences, but prior research had
not provided an explanation of how to involve stakeholders in this process. One
approach that accounts for different stakeholder involvement in a project is stakeholder
analysis [25][12][26][14][13][10]. In this thesis, we extended stakeholder analysis to
compare, analyse and discuss stakeholders’ process-related risk perceptions. The
findings provide a better understanding of risks in OTS-based custom software projects.
 The literature review indicated a lack of research on risks from software acquirer
perspectives [7][15]. However, we have found that acquirers could have better
control of, and be most impacted by risks, and so there could be more research into
acquisition risks.
 Further studies are needed to strengthen the risk mapping study results, for example
by categorising risks using categories from the Software Engineering Institute’s
Taxonomy of Software Development Risk [67].
 This research compared perceptions of process-related risks between developers
and acquirers. Research comparing other risk attributes (such as the probability of
loss and the brief description of risks [29]) should also be conducted, because
stakeholders might also differ in their perceptions of these risk attributes.
Differences in risk attributes between stakeholders should be considered in
collaborative risk management and risk negotiation.
 This research shows that there are risks and processes related to OTS software
vendors. Further study should further investigate these stakeholders, and the role
they play in OTS-based software acquisition.
 The proposed risk framework focuses assessing risks, but this is only one part of
risk management. Research is also needed to address risk-management planning,
risk resolution and risk monitoring.
 The framework has shown potential use for assessing differences in perceptions of
process-related risk. However, the case study participants also pointed out a
potential use for the framework at the beginning of a project to assign roles for
software processes. This purpose should be investigated in future research.
144
 Further should validate that the framework can be generalised to other risks than
the 11 risks examined in this thesis.
 The framework assigns risk responsibility according to which stakeholder who has
higher control over the risk. The framework does not consider all other potentially-
related factors for allocating risk responsibility. Further investigation on other
related factors influencing risk responsibility is strongly recommended.

7.4 Implications for Practice


The findings of this research have a number of important implications for future
practice.
 Our first mapping study identified six OTS-based acquisition processes. We
suggest that for OTS-based software acquisition, changes should be considered for
the existing software acquisition process standard [6] to include these processes,
and for acquirers to consider these processes when acquiring OTS software.
 Practical outcome of the second mapping study is a checklist of risks for risk
identification in OTS-based custom software projects [126][66]. The checklist can
be used to help identify more risks [66] and to assess the relevancy of risks
[126][66].
 The framework can help stakeholders to reduce reliance on ‘gut feeling’ judgments
and to ensure decisions on the assignment of risk responsibilities are more objectively
reviewed [140].
 The framework used in this research may be able to be applied to other kinds of
projects, to analyse differences in risk perceptions.

7.5 Limitations
Finally, a number of important limitations should be considered.
 Even though open source software (OSS) may present different risks, the case study
found that the process-related risks were relevant to OSS. Therefore, we believe
that process-related risks in this research are common to COTS, OTS, component-
based software and OSS.
 Although the framework was designed to be generically applicable to collect,
compare and analyse differences in risk perceptions, this capability has yet to be
implemented outside OTS-based custom software projects.

145
 In comparing levels of risk control and impact (low and high), there was a
possibility of stakeholders’ subjective understanding of risk control and impact. To
avoid this in our case study, we clarified and asked for stakeholders’ agreement on
levels of risk control and impact (see Table 6.5).
 This research analysed OTS-based custom software projects and differences in the
perception of 11 process-related risks relevant to the developers and acquirers. This
could limit generalisability of this research, because there are more risks in this
context. However, the framework used to analyse these risks is not a risk
identification technique, so that the framework can be used to analyse different
risks other than the 11 risks.
 To analyse differences in risk perceptions involving stakeholders other than
developer and acquirer, the framework should be modified by adding the dimension
of risk control/impact matrix, and re-validated.
 Having numerous differences in the four cases (Table 6.4), the case study
represented different contexts, which were expected to vary the sample of the
population under study. However, case studies have a limited ability to generalise
results to populations [86]; therefore, the case study should be replicated in
different cases.

146
Bibliography

[1] C. L. Braun, ‘A lifecycle process for the effective reuse of commercial off-the-
shelf (COTS) software’, in Proceedings of the 1999 Symposium on Software
reusability, Los Angeles, California, United States, 1999, pp. 29-36.
[2] J. Grudin, ‘Interactive systems: bridging the gaps between developers and users’,
Computer, vol. 24, no. 4, pp. 59-69, 1991.
[3] M. Keil and A. Tiwana, ‘Relative importance of evaluation criteria for enterprise
systems: a conjoint study’, Information Systems Journal, vol. 16, no. 3, pp. 237-
262, 2006.
[4] D. Carney, Assembling Large Systems from COTS Components: Opportunities,
Cautions, and Complexities. SEI Monographs on Use of Commercial Software in
Government Systems. Software Engineering Institute, Pittsburgh, USA, 1997.
[5] M. Torchiano and M. Morisio, ‘Overlooked aspects of COTS-based
development’, IEEE Software, vol. 21, no. 2, pp. 88-93, 2004.
[6] IEEE, ‘IEEE recommended practice for software acquisition’, IEEE Std 1062,
1998 Edition, 1998. .
[7] E. Rosendahl and T. Vullinghs, ‘Performing Initial Risk Assessments in
Software Acquisition Projects’, Software Quality — ECSQ 2002, 2002.
[8] C. Schmidt, P. Dart, L. Johnston, L. Sterling, and P. Thorne, ‘Disincentives for
communicating risk: a risk paradox’, Information and Software Technology, vol.
41, no. 7, pp. 403-411, May 1999.
[9] D. W. Gotterbarn and S. Rogerson, ‘Responsible risk analysis for software
development: creating the software development impact statement.’,
Communications of AIS, vol. 2005, no. 15, pp. 730-750, 2005.
[10] R. W. Woolridge, D. J. McManus, and J. E. Hale, ‘Stakeholder Risk Assessment:
An Outcome-Based Approach’, IEEE Software, vol. 24, no. 2, pp. 36-45, 2007.
[11] AS/NZS ISO 31000:2009 : Risk management - Principles and guidelines.
Standards Australia, 2009.
[12] R. E. Freeman, Strategic management: A stakeholder approach, vol. 1. Pitman,
1984.
[13] E. A. Whitley and A. Pouloudi, ‘Stakeholder identification in inter-
organizational systems: gaining insights for drug use management systems’,
European Journal of Information Systems, vol. 6, no. 1, pp. 1-14.
[14] J. Kaler, ‘Differentiating Stakeholder Theories’, Journal of Business Ethics, vol.
46, pp. 71-83, 2003.
[15] M. Jorgensen, ‘How to Avoid Selecting Bids Based on Overoptimistic Cost
Estimates’, IEEE Software, vol. 26, no. 3, pp. 79-84, 2009.
[16] M. Keil, A. Tiwana, and A. Bush, ‘Reconciling user and project manager
perceptions of IT project risk: a Delphi study1’, Information Systems Journal,
vol. 12, no. 2, pp. 103-119, 2002.
[17] L. Zhu, R. Jeffery, M. Staples, M. Huo, and T. Tran, ‘Effects of Architecture and
Technical Development Process on Micro-process’, in Software Process
Dynamics and Agility, vol. 4470, Q. Wang, D. Pfahl, and D. Raffo, Eds. Springer
Berlin / Heidelberg, 2007, pp. 49-60.
[18] S. Hornik, Houn-Gee Chen, G. Klein, and J. J. Jiang, ‘Communication skills of
IS providers: an expectation gap analysis from three stakeholder perspectives’,

147
IEEE Transactions on Professional Communication, vol. 46, no. 1, pp. 17- 34,
2003.
[19] R. Sabherwal, ‘The evolution of coordination in outsourced software
development projects: a comparison of client and vendor perspectives’,
Information and Organization, vol. 13, no. 3, pp. 153-202, Jul. 2003.
[20] S. Adolph, P. Kruchten, and W. Hall, ‘Reconciling perspectives: A grounded
theory of how people manage the process of software development’, Journal of
Systems and Software, vol. 85, no. 6, pp. 1269-1286, Jun. 2012.
[21] A. Bröckers, ‘Process-based software risk assessment’, in Software Process
Technology, W. Schäfer, Ed. Springer Berlin Heidelberg, 1995, pp. 9-29.
[22] F. J. Heemstra and R. J. Kusters, ‘Dealing with risk: a practical approach’,
Journal of Information Technology, vol. 11, no. 4, pp. 333-346, 1996.
[23] F. J. Heemstra, R. J. Kusters, and H. de Man, ‘Guidelines for managing bias in
project risk management’, in 2003 Proceedings International Symposium on
Empirical Software Engineering, 2003, pp. 272 - 280.
[24] M. Svahnberg and C. Wohlin, ‘Consensus Building when Comparing Software
Architectures’, in Product Focused Software Process Improvement, M. Oivo and
S. Komi-Sirviö, Eds. Springer Berlin Heidelberg, 2002, pp. 436-452.
[25] E. S. Andersen, K. V. Grude, and T. Haug, Goal Directed Project Management:
Effective Techniques and Strategies, 4th ed. London: Kogan Page Business Book,
2009.
[26] A. L. Jepsen and P. Eskerod, ‘Stakeholder analysis in projects: Challenges in
using current guidelines in the real world’, International Journal of Project
Management, vol. 27, no. 4, pp. 335-343, May 2009.
[27] J. Li, O. P. N. Slyngstad, M. Torchiano, M. Morisio, and C. Bunse, ‘A State-of-
the-Practice Survey of Risk Management in Development with Off-the-Shelf
Software Components’, IEEE Transactions on Software Engineering, vol. 34, no.
2, pp. 271-286, 2008.
[28] B. Boehm and C. Abts, ‘COTS integration: plug and pray?’, Computer, vol. 32,
no. 1, pp. 135-138, 1999.
[29] F. D. Patterson and K. Neailey, ‘A Risk Register Database System to aid the
management of project risk’, International Journal of Project Management, vol.
20, no. 5, pp. 365-374, Jul. 2002.
[30] B. W. Boehm, ‘Software risk management: principles and practices’, IEEE
Software, vol. 8, no. 1, pp. 32-41, 1991.
[31] L. C. Ballejos and J. M. Montagna, ‘Method for stakeholder identification in
interorganizational environments’, Requirements Eng, vol. 13, no. 4, pp. 281-297,
Sep. 2008.
[32] L. W. Smith, ‘Project clarity through stakeholder analysis’, Crosstalk The
Journal of Defense Software Engineering, p. December 2000.
[33] E. Manowong and S. Ogunlanas, ‘Strategies and Tactics for Managing
Construction Stakeholders’, in Construction Stakeholder Management, Wiley-
Blackwell, 2010, pp. 121-137.
[34] I. Sommerville, ‘Models for Responsibility Assignment’, in Responsibility and
Dependable Systems, G. Dewsbury and J. Dobson, Eds. London: Springer
London, 2007, pp. 165-186.
[35] I. Sommerville, ‘Causal Responsibility Models’, in Responsibility and
Dependable Systems, G. Dewsbury and J. Dobson, Eds. London: Springer
London, 2007, pp. 187-207.

148
[36] I. Alexander and S. Robertson, ‘Understanding project sociology by modeling
stakeholders’, IEEE Software, vol. 21, no. 1, pp. 23-27, 2004.
[37] H. Sharp, A. Finkelstein, and G. Galal, ‘Stakeholder identification in the
requirements engineering process’, in Proceedings of Tenth International
Workshop on Database and Expert Systems Applications, 1999., 1999, pp. 387-
391.
[38] J. Kontio, G. Getto, and D. Landes, ‘Experiences in improving risk management
processes using the concepts of the Riskit method’, in Proceedings of the 6th
ACM SIGSOFT international symposium on Foundations of software
engineering, Lake Buena Vista, Florida, United States, 1998, pp. 163-174.
[39] J. Li, F. Bjørnson, R. Conradi, and V. Kampenes, ‘An empirical study of
variations in COTS-based software development processes in the Norwegian IT
industry’, Empirical Software Engineering, vol. 11, no. 3, pp. 433-461, 2006.
[40] M. Morisio, C. B. Seaman, V. R. Basili, A. T. Parra, S. E. Kraft, and S. E.
Condon, ‘COTS-based software development: Processes and open issues’,
Journal of Systems and Software, vol. 61, no. 3, pp. 189-199, Apr. 2002.
[41] L. Brownsword, T. Oberndorf, and C. A. Sledge, ‘Developing new processes for
COTS-based systems’, IEEE Software, vol. 17, no. 4, pp. 48-55, 2000.
[42] M. Mosko, H. Jiang, A. Samanta, and L. Werner, ‘COTS Software Acquisition
Meta-Model’, 2000.
[43] P. Ulkuniemi and V. Seppanen, ‘Definition of a COTS software component
acquisition process framework: the case of a telecommunications company’,
2002, pp. 48-54.
[44] M. Ochs, D. Pfahl, G. Chrobok-Diening, and B. Nothhelfer-Kolb, ‘A COTS
Acquisition Process: Definition and Application Experience’, 2000, pp. 335–343.
[45] J. Baik, N. Eickelmann, and C. Abts, ‘Empirical software simulation for COTS
glue code development and integration’, in Proceedings of 2001 Computer
Software and Applications Conference, 2001, pp. 297-302.
[46] B. Boehm, D. Port, Ye Yang, and J. Bhuta, ‘Composable process elements for
developing COTS-based applications’, in Proceedings of 2003 International
Symposium on Empirical Software Engineering, 2003, 2003, pp. 8-17.
[47] T. Baker, ‘Lessons Learned Integrating COTS into Systems’, in COTS-Based
Software Systems, vol. 2255, Springer Berlin / Heidelberg, 2002, pp. 21-30.
[48] M. Morisio and M. Torchiano, ‘Definition and Classification of COTS: A
Proposal’, London, 2002, vol. Lecture Notes In Computer Science; Vol. 2255, pp.
165 - 175.
[49] D. Carney and F. Leng, ‘What do you mean by COTS? Finally, a useful answer’,
IEEE Software, vol. 17, no. 2, pp. 83-86, 2000.
[50] ISO/IEC-IEEE, ‘ISO/IEC/IEEE Standard for Systems and Software Engineering
- Software Life Cycle Processes’, IEEE STD 12207-2008, 2008. .
[51] T. Gantner and T. Häberlein, ‘GARP — The Evolution of a Software
Acquisition Process Model’, Software Quality — ECSQ 2002, 2002.
[52] G. Getto, T. Gantner, and T. Vullinghs, ‘Software Acquisition: Experiences with
Models and Methods’, 2000.
[53] K. Chaves Weber, E. E. Ramalho de Araujo, D. Scaler, E. L. Pereira de Andrade,
A. R. Cavalcanti da Rocha, and M. A. Montoni, ‘MPS Model-Based Software
Acquisition Process Improvement in Brazil’, in Proceedings of 6th International
Conference on the Quality of Information and Communications Technology,
2007, pp. 110-122.

149
[54] M. A. Montoni, A. R. Rocha, and K. C. Weber, ‘MPS.BR: a successful program
for software process improvement in Brazil’, Software Process: Improvement and
Practice, vol. 14, no. 5, pp. 289-300, 2009.
[55] M. Ochs, D. Pfahl, G. Chrobok-Diening, and B. Nothhelfer-Kolb, ‘A COTS
Acquisition Process: Definition and Application Experience’, 2000, pp. 335–343.
[56] B. W. Boehm and R. Ross, ‘Theory-W software project management principles
and examples’, IEEE Transactions on Software Engineering, vol. 15, no. 7, pp.
902-916, 1989.
[57] H. Barki, S. Rivard, and J. Talbot, ‘Toward an Assessment of Software
Development Risk’, Journal of Management Information Systems, vol. 10, no. 2,
pp. 203-225, Oct. 1993.
[58] J. Ropponen and K. Lyytinen, ‘Components of software development risk: how
to address them? A project manager survey’, IEEE Transactions onSoftware
Engineering, vol. 26, no. 2, pp. 98-112, 2000.
[59] R. C. Schmidt, K. Lyytinen, M. Keil, and P. E. Cule, ‘Identifying software
project risks: An international Delphi study’, Journal of Management Information
Systems, vol. 17, no. 4, pp. 5-36, 2001.
[60] D. S. Kusumo, L. Zhu, M. Staples, and H. Zhang, ‘A Systematic Mapping Study
on Off-The-Shelf-based Software Acquisition’, in Proceedings of ACIS 2011,
Sydney, 2011.
[61] K. P. B. Webster, K. M. de Oliveira, and N. Anquetil, ‘A risk taxonomy proposal
for software maintenance’, in Proceedings of the 21st IEEE International
Conference on Software Maintenance, 2005. ICSM’05, 2005, pp. 453 - 461.
[62] T. Aven and V. Kristensen, ‘Perspectives on risk: review and discussion of the
basis for establishing a unified and holistic approach’, Reliability Engineering &
System Safety, vol. 90, no. 1, pp. 1-14, Oct. 2005.
[63] K. Georgieva, A. Farooq, and R. R. Dumke, ‘Analysis of the Risk Assessment
Methods – A Survey’, in Software Process and Product Measurement, A. Abran,
R. Braungarten, R. R. Dumke, J. J. Cuadrado-Gallego, and J. Brunekreef, Eds.
Springer Berlin Heidelberg, 2009, pp. 76-86.
[64] B. Freimut, S. Hartkopf, P. Kaiser, J. Kontio, and W. Kobitzsch, ‘An industrial
case study of implementing software risk management’, in Proceedings of the 8th
European software engineering conference held jointly with 9th ACM SIGSOFT
international symposium on Foundations of software engineering, Vienna,
Austria, 2001, pp. 277-287.
[65] P. L. Bannerman, ‘Risk and risk management in software projects: A
reassessment’, Journal of Systems and Software, vol. 81, no. 12, pp. 2118-2133,
Dec. 2008.
[66] M. Keil, L. Li, L. Mathiassen, and G. Zheng, ‘The influence of checklists and
roles on software practitioner risk perception and decision-making’, Journal of
Systems and Software, vol. 81, no. 6, pp. 908-919, Jun. 2008.
[67] M. J. Carr and S. L. Konda, ‘Taxonomy-Based Risk Identification’, Software
Engineering Institute, no. June, pp. 1-24, 1993.
[68] R. C. Williams, J. A. Walker, and A. J. Dorofee, ‘Putting risk management into
practice’, IEEE Software, vol. 14, no. 3, pp. 75-82, 1997.
[69] B. Freimut, S. Hartkopf, P. Kaiser, J. Kontio, and W. Kobitzsch, ‘An industrial
case study of implementing software risk management’, SIGSOFT Softw. Eng.
Notes, vol. 26, no. 5, pp. 277–287, Sep. 2001.

150
[70] K.-S. Na, J. T. Simpson, X. Li, T. Singh, and K.-Y. Kim, ‘Software development
risk and project performance measurement: Evidence in Korea’, Journal of
Systems and Software, vol. 80, no. 4, pp. 596-605, Apr. 2007.
[71] T. Moynihan, ‘How experienced project managers assess risk’, IEEE Software,
vol. 14, no. 3, pp. 35 -41, Jun. 1997.
[72] P. Brereton and D. Budgen, ‘Component-based systems: a classification of
issues’, Computer, vol. 33, no. 11, pp. 54-62, 2000.
[73] P. Vitharana, ‘Risks and challenges of component-based software development’,
Communication of the ACM, vol. 46, no. 8, pp. 67-72, 2003.
[74] L. Rose, ‘Risk Management of COTS Based Systems Development’, in
Component-Based Software Quality, 2003.
[75] B. Boehm, D. Port, Y. Yang, and J. Bhuta, ‘Not All CBS Are Created Equally:
COTS-Intensive Project Types’, COTS-Based Software Systems, 2003.
[76] Y. Yang, B. Boehm, and D. Wu, ‘COCOTS risk analyzer’, in Fifth International
Conference on Commercial-off-the-Shelf (COTS)-Based Software Systems, 2006,
2006, p. 8 pp.
[77] G. Kotonya and A. Rashid, ‘A strategy for managing risk in component-based
software development’, in Proceedings of 27th Euromicro Conference, 2001, pp.
12-21.
[78] I. Sommerville, R. Lock, T. Storer, and J. Dobson, ‘Deriving Information
Requirements from Responsibility Models’, in Advanced Information Systems
Engineering, vol. 5565, P. Eck, J. Gordijn, and R. Wieringa, Eds. Berlin,
Heidelberg: Springer Berlin Heidelberg, 2009, pp. 515-529.
[79] S. Olander and A. Landin, ‘Evaluation of stakeholder influence in the
implementation of construction projects’, International Journal of Project
Management, vol. 23, no. 4, pp. 321-328, May 2005.
[80] G. M. Winch, ‘Managing Project Stakeholders’, in The Wiley Guide to
Managing Projects, Hoboken, NJ, USA: John Wiley & Sons, Inc., 2004, pp. 321-
339.
[81] A. Blyth, ‘Using Stakeholders, Domain Knowledge, and Responsibilities to
Specify Information Systems’ Requirements’, Journal of Organizational
Computing and Electronic Commerce, vol. 9, no. 4, pp. 287-296, Dec. 1999.
[82] V. Choudhury and R. Sabherwal, ‘Portfolios of Control in Outsourced Software
Development Projects’, Information Systems Research, vol. 14, no. 3, pp. 291-
314, 2003.
[83] A. Tiwana and M. Keil, ‘Control in Internal and Outsourced Software Projects’,
Journal of Management Information Systems, vol. 26, no. 3, pp. 9-44, Dec. 2009.
[84] A. Tiwana and M. Keil, ‘Does peripheral knowledge complement control? An
empirical test in technology outsourcing alliances’, Strategic Management
Journal, vol. 28, no. 6, pp. 623–634, 2007.
[85] S. Easterbrook, J. Singer, M.-A. Storey, and D. Damian, ‘Selecting Empirical
Methods for Software Engineering Research’, in Guide to Advanced Empirical
Software Engineering, F. Shull, J. Singer, and D. I. K. Sjøberg, Eds. London:
Springer London, 2008, pp. 285-311.
[86] R. K. Yin, Case Study Research: Design and Methods, 4th ed. Los Angeles, CA:
Sage Publications, Inc, 2008.
[87] C. B. Seaman, ‘Qualitative methods in empirical studies of software
engineering’, IEEE Transactions on Software Engineering, vol. 25, no. 4, pp.
557-572, 1999.

151
[88] J. Carver, C. B. Seaman, and R. Jeffery, ‘Using Qualitative Methods in Software
Engineering’, Los Angeles, CA, USA, 2004.
[89] C. Wohlin, P. Runeson, M. Höst, M. C. Ohlsson, B. Regnell, and A. Wesslén,
Experimentation in software engineering: an introduction. Kluwer Academic
Publishers, 2000.
[90] J. W. Creswell, Research design: Qualitative, quantitative, and mixed methods
approaches, vol. 3rd. Sage Publications, 2009.
[91] D. Budgen, M. Turner, P. Brereton, and B. Kitchenham, ‘Using Mapping Studies
in Software Engineering’, 2008, pp. 195–204.
[92] B. Kitchenham and S. Charters, ‘Guidelines for performing Systematic
Literature Reviews in Software Engineering’, Keele University and Durham
University Joint Report, EBSE 2007-001, 2007.
[93] K. Petersen, R. Feldt, S. Mujtaba, and M. Mattsson, ‘Systematic Mapping
Studies in Software Engineering’, University of Bari, Italy, 2008.
[94] ‘Zotero’, http://www.zotero.org/, 2011.
[95] R. Wieringa, N. Maiden, N. Mead, and C. Rolland, ‘Requirements engineering
paper classification and evaluation criteria: a proposal and a discussion’,
Requirements Eng, vol. 11, no. 1, pp. 102-107, Nov. 2005.
[96] CMMI Product Team, ‘CMMI for Acquisition, Version 1.2’, SEI, Carnegie
Mellon, CMU/SEI-2007-TR-017 ESC-TR-2007-017, Nov. 2007.
[97] S. Elgazzar, A. Kark, E. Putrycz, and M. Vigder, ‘COTS Acquisition: Getting a
Good Contract’, COTS-Based Software Systems, 2005.
[98] J. S. Seibel, T. A. Mazzuchi, and S. Sarkani, ‘Same vendor, version-to-version
upgrade decision support model for commercial off-the-shelf productivity
applications’, Systems Engineering, vol. 9, no. 4, pp. 296-312, 2006.
[99] J. Holck, M. K. Pedersen, and M. H. Larsen, ‘Open Source Software
Acquisition: Beyond the Business Case’, presented at the ECIS 2005,
Regensburg, Germany, 2005.
[100] L. Morgan and P. Finnegan, ‘Open innovation in secondary software firms: an
exploration of managers’ perceptions of open source software’, SIGMIS
Database, vol. 41, no. 1, pp. 76-95, 2010.
[101] L. Briand, S. Carrière, R. Kazman, and J. Wüst, ‘COMPARE: A Comprehensive
Framework for Architecture Evaluation’, Object-Oriented Technology:
ECOOP’98 Workshop Reader, 1998.
[102] J. Holck, M. H. Larsen, and M. K. Pedersen, ‘Managerial and Technical Barriers
to the Adoption of Open Source Software’, COTS-Based Software Systems, 2005.
[103] C. Albert and L. Brownsword, ‘Meeting the Challenges of Commercial-Off-The-
Shelf (COTS) Products: The Information Technology Solutions Evolution
Process (ITSEP)’, COTS-Based Software Systems, 2002.
[104] W. Aigner, P. Regner, T. Wiesinger, and J. Kung, ‘Supporting public software
acquisition workflows - implications for data models’, in Proceedings of 15th
International Workshop on the Database and Expert Systems Applications, 2004,
pp. 1016-1022.
[105] M. Haglind, L. Johansson, and M. Rantzer, ‘Experiences integrating
requirements engineering and business analysis. An empirical study of operations
and management system procurement projects’, in Proceedings of Third
International Conference on Requirements Engineering,1998, pp. 108-117.
[106] A. Heiskanen, M. Newman, and J. Similä, ‘The social dynamics of software
development’, Accounting, Management and Information Technologies, vol. 10,
no. 1, pp. 1-32, Jan. 2000.
152
[107] G. Shaffer and G. McPherson, ‘FAA COTS Risk Mitigation Guide: Practical
Methods For Effective COTS Acquisition and Life Cycle Support’, Federal
Aviation Administration, 2002.
[108] L. D. Ball, I. G. Dambolena, and H. D. Hennessey, ‘Identifying early adopters of
large software systems’, SIGMIS Database, vol. 19, no. 1, pp. 21-27, 1987.
[109] T. Helokunnas and M. Nyby, ‘Collaboration between a COTS Integrator and
Vendors’, Software Quality — ECSQ 2002, 2002.
[110] E. Engström and P. Runeson, ‘Software product line testing – A systematic
mapping study’, Information and Software Technology, vol. 53, no. 1, pp. 2-13,
Jan. 2011.
[111] EBSE RG, ‘Template for a Mapping Study Protocol’,
http://www.dur.ac.uk/ebse/resources/templates/MappingStudyTemplate.pdf, Apr-
2009. [Accessed: 05-Mar-2011].
[112] B. A. Kitchenham, D. Budgen, and O. Pearl Brereton, ‘Using mapping studies as
the basis for further research - A participant-observer case study’, Information
and Software Technology, vol. 53, no. 6, pp. 638-651, Jun. 2011.
[113] D. Greer and R. Conradi, ‘Software project initiation and planning - an empirical
study’, IET Software, vol. 3, no. 5, pp. 356-368, 2009.
[114] D. J. Reifer, V. R. Basili, B. W. Boehm, and B. Clark, ‘Eight lessons learned
during COTS-based systems maintenance’, IEEE Software, vol. 20, no. 5, pp. 94
- 96, Oct. 2003.
[115] D. Carney, S. A. Hissam, and D. Plakosh, ‘Complex COTS-based software
systems: practical steps for their maintenance’, Journal of Software Maintenance:
Research and Practice, vol. 12, no. 6, pp. 357–376, 2000.
[116] Yi Ding and N. Napier, ‘Measurement Framework for Assessing Risks in
Component-Based Software Development’, in Proceedings of the 39th Annual
Hawaii International Conference on System Sciences, 2006, vol. 9, p. 230b.
[118] N. Admodisastro and G. Kotonya, ‘Architectural Analysis Approaches: A
Component-Based System Development Perspective’, in High Confidence
Software Reuse in Large Systems, vol. 5030, H. Mei, Ed. Springer Berlin /
Heidelberg, 2008, pp. 26-38.
[119] Vu Tran and Dar-Biau Liu, ‘A risk-mitigating model for the development of
reliable and maintainable large-scale commercial-off-the-shelf integrated
software systems’, in Proceedings of Reliability and Maintainability Symposium,
1997, pp. 361-367.
[120] L. Rose, ‘Risk Management of COTS Based Systems Development’, in
Component-Based Software Quality, vol. 2693, Springer Berlin / Heidelberg,
2003, pp. 352-373.
[121] D. Port and Y. Yang, ‘Empirical Analysis of COTS Activity Effort Sequences’,
in COTS-Based Software Systems, vol. 2959, R. Kazman and D. Port, Eds.
Springer Berlin / Heidelberg, 2004, pp. 169-182.
[122] B. Boehm, D. Port, Y. Yang, and J. Bhuta, ‘Not All CBS Are Created Equally:
COTS-Intensive Project Types’, in COTS-Based Software Systems, vol. 2580, H.
Erdogmus and T. Weng, Eds. Springer Berlin / Heidelberg, 2003, pp. 36-50.
[123] J. Li, R. Conradi, O. P. N. Slyngstad, M. Torchiano, M. Morisio, and C. Bunse,
‘Preliminary Results from a State-of-the-Practice Survey on Risk Management in
Off-the-Shelf Component-Based Development’, in COTS-Based Software
Systems, vol. 3412, X. Franch and D. Port, Eds. Springer Berlin / Heidelberg,
2005, pp. 278-288.

153
[124] S. Mahmood and A. Khan, ‘An industrial study on the importance of software
component documentation: A system integrator’s perspective’, Information
Processing Letters, vol. 111, no. 12, pp. 583-590, Jun. 2011.
[125] L. Osterweil, ‘Unifying Microprocess and Macroprocess Research’, in Unifying
the Software Process Spectrum, vol. 3840, M. Li, B. Boehm, and L. Osterweil,
Eds. Springer Berlin / Heidelberg, 2006, pp. 68-74.
[126] P. L. Bannerman, ‘Risk and risk management in software projects: A
reassessment’, Journal of Systems and Software, vol. 81, no. 12, pp. 2118-2133,
Dec. 2008.
[127] D. Hillson, ‘Using a Risk Breakdown Structure in project management’, Journal
of Facilities Management, vol. 2, no. 1, pp. 85-97, 2003.
[128] M. Keil, P. E. Cule, K. Lyytinen, and R. C. Schmidt, ‘A framework for
identifying software project risks’, Communication of the ACM, vol. 41, no. 11,
pp. 76-83, 1998.
[129] P. Runeson and M. Höst, ‘Guidelines for conducting and reporting case study
research in software engineering’, Empir Software Eng, vol. 14, no. 2, pp. 131-
164, Dec. 2008.
[130] L. A. Cox, ‘Limitations of Risk Assessment Using Risk Matrices’, in Risk
Analysis of Complex and Uncertain Systems, vol. 129, Boston, MA: Springer US,
2009, pp. 101-124.
[131] B. Kitchenham, L. Pickard, and S. L. Pfleeger, ‘Case studies for method and tool
evaluation’, IEEE Software, vol. 12, no. 4, pp. 52-62, 1995.
[132] J. E. Cook, L. G. Votta, and A. L. Wolf, ‘Cost-effective analysis of in-place
software processes’, IEEE Transactions on Software Engineering, vol. 24, no. 8,
pp. 650-663, 1998.
[133] W. M. Garrabrants, A. W. Ellis, L. J. Hoffman, and M. Kamel, ‘CERTS: a
comparative evaluation method for risk management methodologies and tools’, in
Proceedings of the Sixth Annual Conference on Computer Security Applications,
1990, pp. 251-257.
[134] S. Lichtenstein, ‘Factors in the selection of a risk assessment method’,
Information Management & Computer Security, vol. 4, no. 4, pp. 20-25, Oct.
1996.
[135] M. G. Mendonca and V. R. Basili, ‘Validation of an approach for improving
existing measurement frameworks’, IEEE Transactions on Software Engineering,
vol. 26, no. 6, pp. 484 -499, Jun. 2000.
[136] K. M. Eisenhardt, ‘Building Theories from Case Study Research’, The Academy
of Management Review, vol. 14, no. 4, pp. 532-550, Oct. 1989.
[137] J. Voas, ‘Maintaining component-based systems’, IEEE Software, vol. 15, no. 4,
pp. 22 -27, Aug. 1998.
[138] Gero J.S. and Fujii H., ‘A computational framework for concept formation for a
situated design agent’, Knowledge-Based Systems, vol. 13, no. 6, pp. 361-368,
2000.
[139] K. C. Lam, D. Wang, P. T. K. Lee, and Y. T. Tsang, ‘Modelling risk allocation
decision in construction contracts’, International Journal of Project Management,
vol. 25, no. 5, pp. 485-493, Jul. 2007.
[140] M. Jorgensen, ‘Practical guidelines for expert-judgment-based software effort
estimation’, IEEE Software, vol. 22, no. 3, pp. 57-63, 2005.
[141] M. Keil and E. Carmel, ‘Customer-developer links in software development’,
Communication of the ACM, vol. 38, no. 5, pp. 33-44, 1995.

154
[142] P. Brereton, ‘The software customer/supplier relationship’, Communication of
the ACM, vol. 47, no. 2, pp. 77-81, 2004.
[143] S. Sawyer, ‘Software development teams’, Communication of the ACM, vol. 47,
no. 12, pp. 95–99, Dec. 2004.
[144] P. Waterson, S. Weibelzahl, and D. Pfahl, ‘Software Process Modelling’, in
Software Process Modeling, D. S. T. Acuña and D. N. Juristo, Eds. Springer US,
2005, pp. 111-139.
[145] K. Ballurio, B. Scalzo, and L. Rose, ‘Risk Reduction in COTS Software
Selection with BASIS’, COTS-Based Software Systems, 2002.
[146] M. John, F. Maurer, and B. Tessem, ‘Human and social factors of software
engineering: workshop summary’, SIGSOFT Softw. Eng. Notes, vol. 30, no. 4, pp.
1–6, Jul. 2005.
[147] H. Sharp and H. Robinson, ‘Some social factors of software engineering: the
maverick, community and technical practices’, SIGSOFT Softw. Eng. Notes, vol.
30, no. 4, pp. 1–6, May 2005.
[148] B. Boehm, ‘Value-based software engineering: reinventing’, SIGSOFT Softw.
Eng. Notes, vol. 28, no. 2, p. 3–, Mar. 2003.
[149] P. Grünbacher, S. Köszegi, and S. Biffl, ‘Stakeholder Value Proposition
Elicitation and Reconciliation’, in Value-Based Software Engineering, S. Biffl, A.
Aurum, B. Boehm, H. Erdogmus, and P. Grünbacher, Eds. Springer Berlin
Heidelberg, 2006, pp. 133-154.
[150] B. Boehm, P. Grunbacher, and R. O. Briggs, ‘Developing groupware for
requirements negotiation: lessons learned’, IEEE Software, vol. 18, no. 3, pp. 46 -
55, May 2001.
[151] J. Fjermestad and S. R. Hiltz, ‘Group Support Systems: A Descriptive Evaluation
of Case and Field Studies’, Journal of Management Information Systems, vol. 17,
no. 3, pp. 115-159, Dec. 2000.
[152] L. C. Ballejos and J. M. Montagna, ‘Modeling stakeholders for information
systems design processes’, Requirements Eng, vol. 16, no. 4, pp. 281-296, Nov.
2011.
[153] S. L. Lim, D. Quercia, and A. Finkelstein, ‘StakeNet: using social networks to
analyse the stakeholders of large-scale software projects’, in 2010 ACM/IEEE
32nd International Conference on Software Engineering, 2010, vol. 1, pp. 295 -
304.
[154] S. L. Lim, D. Damian, and A. Finkelstein, ‘StakeSource2.0: using social
networks of stakeholders to identify and prioritise requirements’, in 2011 33rd
International Conference on Software Engineering (ICSE), 2011, pp. 1022 -1024.
[155] S. L. Lim and A. Finkelstein, ‘StakeRare: Using Social Networks and
Collaborative Filtering for Large-Scale Requirements Elicitation’, IEEE
Transactions on Software Engineering, vol. 38, no. 3, pp. 707 -735, Jun. 2012.
[156] H. Zhu, M. Zhou, and P. Seguin, ‘Supporting Software Development With
Roles’, IEEE Transactions on Systems, Man and Cybernetics, Part A: Systems
and Humans, vol. 36, no. 6, pp. 1110 -1123, Nov. 2006.
[157] S. T. Acuña and N. Juristo, ‘Assigning people to roles in software projects’,
Software: Practice and Experience, vol. 34, no. 7, pp. 675–696, 2004.
[158] S. T. Acuna, N. Juristo, and A. M. Moreno, ‘Emphasizing human capabilities in
software development’, IEEE Software, vol. 23, no. 2, pp. 94 - 101, Apr. 2006.
[159] A. Wise, A. G. Cass, B. S. Lerner, E. K. McCall, L. J. Osterweil, and S. M. S. Jr,
‘Using Little-JIL to Coordinate Agents in Software Engineering’, in Engineering

155
of Software, P. L. Tarr and A. L. Wolf, Eds. Springer Berlin Heidelberg, 2011,
pp. 383-397.
[160] M. André, M. G. Baldoquín, and S. T. Acuña, ‘Formal model for assigning
human resources to teams in software projects’, Information and Software
Technology, vol. 53, no. 3, pp. 259-275, Mar. 2011.
[161] C. Amrit, ‘Coordination in software development: the problem of task
allocation’, SIGSOFT Softw. Eng. Notes, vol. 30, no. 4, pp. 1–7, May 2005.

156
Appendix A: Glossary

Off-the-shelf (OTS): a commercially available or open source piece of software that


other software projects can reuse and integrate into their own products.

Acquirer: a person or organisation that acquires or procures a system or software


product (which may be part of a system) from a supplier.

Developer: a person or organisation that performs development activities (including


requirements analysis, design, testing through acceptance) during the software life cycle
process.

Off-the-shelf-based (OTS) custom software project: a type of software projects


consists of OTS-based software development and acquisition processes.

OTS-based software development: software development that integrates OTS


software to develop a system. For OTS-based custom software, development is
performed to cover remaining requirements when OTS products cannot by themselves
meet the requirements.

OTS-based software acquisition: software acquisition of software that itself uses OTS
software platforms or components. For OTS-based custom software, custom
development is performed to cover remaining requirements when OTS products cannot
by themselves meet the requirements.

Risk control: actions implementing risk management.

Risk impact: loss associated with not meeting the expected outcome.

Risk perception: the judgment that people make when they asked to characterise and
analyse risks.

157
Appendix B: Mapped papers for OTS-based software acquisition
mapping study

Mapped papers for software acquisition process categories

Process Type of paper (total mapped papers/references)


Empirical Experience Conceptual Standard
framework
Planning organizational 1
strategy [2]
Implementing
organization’s process
Determining the
software requirements
Identifying potential 1
suppliers [11]
Preparing contract
requirements
Evaluating proposals 5
and selecting the [15][32][28][29]
supplier [31]
Decision-making 3 2 1
[8][24][26] [9][18] [22]
Process life cycle 10 2 2 [16][33]
[6][19][20][17][34] [1][25]
[4][36][37][38][39]
Architectural decision 1
[3]
Modelling and 3 2
simulation [5][23][30] [12][13]
Managing relationship 2 1
between developer and [14][27] [21]
acquirer
Software acquisition 5 1 1
improvement [4][7][19][17][34] [10] [35]

158
Mapped papers for OTS-based software acquisition process categories

Process Type of paper (total mapped papers/references)


Empirical Experience Conceptual
framework
Planning organizational 1
strategy [47]
Implementing 1
organization’s process [47]
Determining the software 1 13
requirements [59] [50][51][60][72][74][75][80]
[81][85][92][42][47][48]
Identifying potential 1
suppliers [47]
Preparing contract 1
requirements [47]
Evaluating proposals and 2
selecting the supplier [47][50]
Decision-making 5 2
[41][83][87] [56][65]
[88][93]
Process life cycle 3 5
[78][79][83] [55][44][63][95][94]
Architectural decision 1 5
[53] [43][54][64][90][91]
OTS selection 6 25
[57][59][61] [45][49][51][52][54][58][65]
[83][86][89] [66][67][69][70][71][72][73]
[74][76][77][80][82][84][92]
[42][48][62][68]
Managing relationships 1
between developers and [53]
acquirers
Managing relationships 1
between OTS adoption and [40]
acquirers’ organisations
Managing relationships 1
between OTS vendors and [46]
developers

References:

[1] J. Mogilensky, “Future approaches to software procurement,” Baltimore,


Maryland, United States: ACM, 1990, pp. 543-550.
[2] W. Richmond, P. Nelson, and S. Misra, “An empirical analysis of software life
spans to determine the planning horizon for new software,” Information
Technology and Management, vol. 7, Apr. 2006, pp. 131-149.
159
[3] L. Briand, S. Carrière, R. Kazman, and J. Wüst, “COMPARE: A Comprehensive
Framework for Architecture Evaluation,” Object-Oriented Technology:
ECOOP’98 Workshop Reader, 1998.
[4] T. Gantner and T. Häberlein, “GARP — The Evolution of a Software Acquisition
Process Model,” Software Quality — ECSQ 2002, 2006.
[5] T. Häberlein and T. Gantner, “Process-Oriented Interactive Simulation of Software
Acquisition Projects,” EurAsia-ICT 2002: Information and Communication
Technology, 2002.
[6] P. Regner, T. Wiesinger, J. Küng, and R. Wagner, “Software Acquisition Based on
Business Models,” Electronic Government, 2004.
[7] I. Garcia, C. Pacheco, and P. Sumano, “Use of Questionnaire-Based Appraisal to
Improve the Software Acquisition Process in Small and Medium Enterprises,”
Software Engineering Research, Management and Applications, 2008.
[8] P. Gupta, S. Gould, and B. Pola, ““To Pirate or Not to Pirate”: A Comparative
Study of the Ethical Versus Other Influences on the Consumer’s Software
Acquisition-Mode Decision,” Journal of Business Ethics, vol. 55, Dec. 2004, pp.
255-274.
[9] M.A. Serva, S.A. Sherer, and J.C. Sipior, ““When Do You ASP?” The Software
Life Cycle Control Model,” Information Systems Frontiers, vol. 5, Apr. 2003, pp.
219-232.
[10] L. Ibrahim and A. Pyster, “A single model for process improvement,” IT
Professional, vol. 6, 2004, pp. 43-49.
[11] D. Ladd, “A Software Procurement and Security Primer,” IEEE Security &
Privacy, vol. 4, 2006, pp. 71-73.
[12] H. Alfaraj and Shaowen Qin, “Business Process Modeling for Software
Acquisition: A Literature Review,” 2008, pp. 1-3.
[13] C. Conwell, R. Enright, and M. Stutzman, “Capability Maturity Models support of
modeling and simulation verification, validation, and accreditation,” 2000, pp.
819-828 vol.1.
[14] M. Haglind, L. Johansson, and M. Rantzer, “Experiences integrating requirements
engineering and business analysis. An empirical study of operations and
management system procurement projects,” 1998, pp. 108-117.
[15] M. Jorgensen, “How to Avoid Selecting Bids Based on Overoptimistic Cost
Estimates,” IEEE Software, vol. 26, 2009, pp. 79-84.
[16] “IEEE Recommended Practice for Software Acquisition,” IEEE Standard 1062,
1998 Edition, 1998.
[17] K. Chaves Weber, E. Ramalho de Araujo, D. Scaler, E. Pereira de Andrade, A.
Cavalcanti da Rocha, and M. Montoni, “MPS Model-Based Software Acquisition
Process Improvement in Brazil,” 2007, pp. 110-122.
[18] C. Beath and G. Walker, “Outsourcing of application software: a knowledge
management perspective,” 1998, pp. 666-674 vol.6.
[19] Sha Wong, “Software acquisition management experience learnt in a multi
discipline and multi contract project environment,” 2000, pp. 239-247.
[20] B. Farbey and A. Finkelstein, “Software acquisition: a business strategy analysis,”
2001, pp. 76-83.
[21] W. Aigner, P. Regner, T. Wiesinger, and J. Kung, “Supporting public software
acquisition workflows - implications for data models,” 2004, pp. 1016-1022.
[22] F. Shull, “Who Needs Evidence, Anyway?,” IEEE Software, vol. 24, 2007, pp.
10-11.
[23] S.J. Choi and W. Scacchi, “Modeling and simulating software acquisition process
160
architectures,” Journal of Systems and Software, vol. 59, Dec. 2001, pp. 343-354.
[24] P. Nelson, A. Seidmann, and W. Richmond, “Software Acquisition: The
Custom/Package and lnsource/Outsource Dimensions,” Elsevier, 1998, pp. 341-
367.
[25] J.E. Tardy, “Strategies for software acquisition,” Journal of Systems and Software,
vol. 18, Jul. 1992, pp. 281-285.
[26] T. Rands, “The key role of applications software make-or-buy decisions,” The
Journal of Strategic Information Systems, vol. 1, Sep. 1992, pp. 215-223.
[27] A. Heiskanen, M. Newman, and J. Similä, “The social dynamics of software
development,” Accounting, Management and Information Technologies, vol. 10,
Jan. 2000, pp. 1-32.
[28] E. Paschetta and A. Tsoukiàs, “A real-world MCDA application: evaluating
software,” Journal of Multi-Criteria Decision Analysis, vol. 9, 2000, pp. 205-226.
[29] F. Fabbrini, M. Fusani, G. Lami, E. Sivera, and E. Sivera, “A SPICE-based
software supplier qualification mechanism in automotive industry,” Software
Process: Improvement and Practice, vol. 12, 2007, pp. 523-528.
[30] T. Häberlein, “Common structures in system dynamics models of software
acquisition projects,” Software Process: Improvement and Practice, vol. 9, 2004,
pp. 67-80.
[31] O. Demirors, E. Demirors, and A. Tarhan, “Managing instructional software
acquisition,” Software Process: Improvement and Practice, vol. 6, 2001, pp. 189-
203.
[32] A.A. April and D. Al-Shurougi, “Software Product Measurement for Supplier
Evaluation,” FESMA-AEMES Software Measurement Conference, 2000, pp. 18–
20.
[33] ISO/IEC-IEEE, “ISO/IEC/IEEE Standard for Systems and Software Engineering -
Software Life Cycle Processes,” IEEE Standard 12207-2008, 2008.
[34] M.A. Montoni, A.R. Rocha, and K.C. Weber, “MPS.BR: a successful program for
software process improvement in Brazil,” Software Process: Improvement and
Practice, vol. 14, 2009, pp. 289-300.
[35] CMMI Product Team, CMMI for Acquisition, Version 1.2, Carnegie Mellon: SEI,
2007.
[36] J. Schreiber, Beschaffung von Informatikmitteln. Pflichtenheft, Evaluation,
Entscheidung, Paul Haupt Verlag, 2000.
[37] Software Program Managers Network, “Program Managers Guide to Software
Acquisition.”
[38] “ISPL: Information Service Procurement Library. Managing Acquisition
Processes,” 1999.
[39] The Opengroup, Beyond the Contract: An Analysis of the business impact of IT
procurement best practices, 1999.
[40] L.D. Ball, I.G. Dambolena, and H.D. Hennessey, “Identifying early adopters of
large software systems,” SIGMIS Database, vol. 19, 1987, pp. 21-27.
[41] L. Morgan and P. Finnegan, “Open innovation in secondary software firms: an
exploration of managers' perceptions of open source software,” SIGMIS Database,
vol. 41, 2010, pp. 76-95.
[42] N.A.M. Maiden, C. Ncube, and A. Moore, “Lessons learned during requirements
acquisition for COTS systems,” Communication of the ACM, vol. 40, 1997, pp.
21-25.
[43] T. Ihme, “A Model for Recording Early-Stage Proposals and Decisions on Using
COTS Components in Architecture,” COTS-Based Software Systems, 2003.
161
[44] R.J. Adams and S. Eslinger, “Best Practices for the Acquisition of COTS-Based
Systems: Lessons Learned from the Space System Domain,” COTS-Based
Software Systems, 2004.
[45] F. Sudaman and C. Mingins, “BiCom: An Evaluation Framework for COTS
Components,” COTS-Based Software Systems, 2003.
[46] T. Helokunnas and M. Nyby, “Collaboration between a COTS Integrator and
Vendors,” Software Quality — ECSQ 2002, 2006.
[47] S. Elgazzar, A. Kark, E. Putrycz, and M. Vigder, “COTS Acquisition: Getting a
Good Contract,” COTS-Based Software Systems, 2005.
[48] V. Sai, “COTS Acquisition Evaluation Process: The Preacher’s Practice,” COTS-
Based Software Systems, 2003.
[49] F. Ye and T. Kelly, “COTS Product Selection for Safety-Critical Systems,” COTS-
Based Software Systems, 2004.
[50] S. Lauesen, “COTS tenders and integration requirements,” Requirements
Engineering, vol. 11, Apr. 2006, pp. 111-122.
[51] J. Pablo Carvallo, X. Franch, and C. Quer, “Defining a Quality Model for Mail
Servers,” COTS-Based Software Systems, 2003.
[52] A. Bader, C. Mingins, D. Bennett, and S. Ramakrishan, “Establishing Trust in
COTS Components,” COTS-Based Software Systems, 2003.
[53] J. Holck, M.H. Larsen, and M.K. Pedersen, “Managerial and Technical Barriers to
the Adoption of Open Source Software,” COTS-Based Software Systems, 2005.
[54] C. Albert and L. Brownsword, “Meeting the Challenges of Commercial-Off-The-
Shelf (COTS) Products: The Information Technology Solutions Evolution Process
(ITSEP),” COTS-Based Software Systems, 2002.
[55] R. Hornstein and J. Willoughby, “Realizing the Potential for COTS Utilization: A
Work in Progress,” COTS-Based Software Systems, 2002.
[56] M.D. Feblowitz and S.J. Greenspan, “Scenario-Based Analysis of COTS
Acquisition Impacts,” Requirements Engineering, vol. 3, Mar. 1998, pp. 182-201.
[57] I. Gorton and A. Liu, “Streamlining the Acquisition Process for Large-Scale COTS
Middleware Components,” COTS-Based Software Systems, 2002.
[58] C. Ncube and J. Dean, “The Limitations of Current Decision-Making Techniques
in the Procurement of COTS Software Components,” COTS-Based Software
Systems, 2002.
[59] B.H. Far, V. Mudigonda, and A.H. Elamy, “A General Purpose Software
Evaluation System,” 2008, pp. 290-295.
[60] D. Rickman, “A new process for requirements understanding,” 2001, pp. 4D4/1-
4D4/6 vol.1.
[61] A. Liu and I. Gorton, “Accelerating COTS middleware acquisition: the i-Mate
process,” IEEE Software, vol. 20, 2003, pp. 72-79.
[62] N. Maiden and C. Ncube, “Acquiring COTS software selection requirements,”
IEEE Software, vol. 15, 1998, pp. 46-56.
[63] V. Tran, “Component-based integrated systems development: a model for the
emerging procurement-centric approach to software development,” 1998, pp. 128-
135.
[64] W. Lemahieu, M. Snoeck, F. Goethals, M. De Backer, R. Haesen, J.
Vandenbulcke, and G. Dedene, “Coordinating COTS applications via a business
event layer,” IEEE Software, vol. 22, 2005, pp. 28-35.
[65] N. Schneidewind, “Cost framework for COTS evaluation,” 1999, pp. 100-101.
[66] P. Ulkuniemi and V. Seppanen, “COTS component acquisition in an emerging
market,” IEEE Software, vol. 21, 2004, pp. 76-82.
162
[67] S. Lauesen, “COTS tenders and integration requirements,” 2004, pp. 166-175.
[68] P. Ulkuniemi and V. Seppanen, “Definition of a COTS software component
acquisition process framework: the case of a telecommunications company,” 2002,
pp. 48-54.
[69] R. Kohl, “Establishing guidelines for suitability of COTS for a mission critical
application,” 1999, pp. 98-99.
[70] Jinfang Sheng and Bin Wang, “Evaluating COTS Components Using Gap
Analysis,” 2008, pp. 1248-1253.
[71] X. Illa, X. Franch, and J. Pastor, “Formalising ERP selection criteria,” 2000, pp.
115-122.
[72] C. Ncube and N. Maiden, “Guiding parallel requirements acquisition and COTS
software selection,” 1999, pp. 133-140.
[73] F. Navarrete, P. Botella, and X. Franch, “How agile COTS selection methods are
(and can be)?,” 2005, pp. 160-167.
[74] J. Carvallo and X. Franch, “On the Use of Requirements for Driving Call-for-
Tender Processes for Procuring Coarse-grained OTS Components,” 2009, pp. 287-
292.
[75] E. Hitt, “Rebuilding the requirements process for aging avionics,” 2000, pp.
4A3/1-4A3/6 vol.1.
[76] F. Navarrete, P. Botella, and X. Franch, “Reconciling Agility and Discipline in
COTS Selection Processes,” 2007, pp. 103-113.
[77] B. Kizzort, “Selection of components for OTS component-based systems,” 2002,
pp. 6-2651-6-2659 vol.6.
[78] J. Verville and A. Halingten, “A six-stage model of the buying process for ERP
software,” Industrial Marketing Management, vol. 32, Oct. 2003, pp. 585-594.
[79] P. Poon and Y.T. Yu, “Investigating ERP systems procurement practice: Hong
Kong and Australian experiences,” Information and Software Technology, vol. In
Press, Corrected Proof.
[80] C. Rolland, “Requirements engineering for COTS based systems,” Information
and Software Technology, vol. 41, Nov. 1999, pp. 985-990.
[81] A. Bruseberg, “The design of complete systems: Providing human factors
guidance for COTS acquisition,” Reliability Engineering & System Safety, vol. 91,
Dec. 2006, pp. 1554-1565.
[82] M. Wybo, J. Robert, and P. Léger, “Using search theory to determine an
applications selection strategy,” Information & Management, vol. 46, Jun. 2009,
pp. 285-293.
[83] C.P. Salter and D.M. Buede, “A lifecycle-based method for the acquisition of
commercial-off-the-shelf (COTS) technology to support organizational processes,”
Systems Engineering, vol. 4, 2001, pp. 287-304.
[84] J. Miller and H.C. Yeoh, “COTS acquisition process: incorporating business
factors into COTS vendor evaluation taxonomies,” Software Process:
Improvement and Practice, vol. 11, 2006, pp. 601-626.
[85] L. Chung and K. Cooper, “Defining goals in a COTS-aware requirements
engineering approach,” Systems Engineering, vol. 7, 2004, pp. 61-83.
[86] A. Mohamed, G. Ruhe, and A. Eberlein, “Optimized mismatch resolution for
COTS selection,” Software Process: Improvement and Practice, vol. 13, 2008, pp.
157-169.
[87] M. Keil and A. Tiwana, “Relative importance of evaluation criteria for enterprise
systems: a conjoint study,” Information Systems Journal, vol. 16, 2006, pp. 237-
262.
163
[88] J.S. Seibel, T.A. Mazzuchi, and S. Sarkani, “Same vendor, version-to-version
upgrade decision support model for commercial off-the-shelf productivity
applications,” Systems Engineering, vol. 9, 2006, pp. 296-312.
[89] M. Ochs, D. Pfahl, G. Chrobok-Diening, and B. Nothhelfer-Kolb, “A COTS
Acquisition Process: Definition and Application Experience,” 11th ESCOM
Conference, Publications, 2000, pp. 335–343.
[90] M. Vidger, An Architecture for COTS Based Software, National Research Council
Canada Institute for Information Technology, 1998.
[91] M. Vigder, Inspecting COTS Based Software Systems. Verifying an Architecture to
Support Management of Long-Lived Software Systems, National Research Council
Canada Institute for Information Technology, 1998.
[92] C. Ncube and N.A.M.P. Maiden, “PORE: Procurement-oriented requirements
engineering method for the component-based systems engineering development
paradigm,” 1999, pp. 1–12.
[93] J. Holck, M.K. Pedersen, and M.H. Larsen, “Open Source Software Acquisition:
Beyond the Business Case,” Regensburg, Germany: 2005.
[94] M. Mosko, H. Jiang, A. Samanta, and L. Werner, “COTS Software Acquisition
Meta-Model,” Proceedings of COTS Workshop, 2000.
[95] B.C. Meyers and P. Oberndorf, Managing software acquisition: open systems and
COTS products, Addison-Wesley Longman Ltd., 2001.

164
Appendix C: Mapped papers for risks of OTS-based and custom software development and acquisition

For each risk category, there is a possibility that a paper was mapped to a risk category more than one, because the paper had more than one risk
factor. For example, in “Planning” risk category for risks of OTS-based software acquisition, paper [17] has two risk factors: underestimated cost
estimation and multi-OTS products from different vendors may result in complicated licensing arrangements.

Mapped papers for risks of OTS-based software development and acquisition

Risk category Risks of OTS-based software development (total mapped Risks of OTS-based software acquisition (total
papers/references) mapped papers/references)
Empirical Experience Critical framework Empirical Experience Critical framework
Planning 2 2 3
[45][53] [5][36] [17][17][36]
Requirements 24 6 5 2 2 3
[12][4][54][4][54][52][54][46] [55][14][8][22] [5][36][48][57][23] [28][44] [59][22] [17][36][17]
[52][53][44][49][45][51][52][53] [14][15]
[4][54][52][53][53][53][53][53]
Design 2 4
[14][55] [2][11][13][50]
Integration and 19 3 5 4
testing [4][54][39][35][53][4][54][6] [55][20][14] [41][50][56][3][5] [17][17][17][36]
[52][53][45][4][54][4][54][4]
[54][4][54]
System lifecycle 1 3 [1][19][38]
[53]
165
Maintenance 7 1 5 3
[4][54][53][52][4][54][53] [55] [5][37][5][3][3] [17][36][17]
Project closure 1
[54]
Software 2
[32][60]
OTS product 7 5 10 2
[52][6][6][49][6][49][53] [55][16][18] [3][3][3][7][21] [17][36]
[18][10] [36][40][50][36][50]
Cost 1 1
[3] [32]
Environment 2 2 1 1
[6][6] [16][16] [28] [17]
People 2 2 1
[6][52] [55][16] [17]
Systems 3 1
Engineering [9][9][9] [28]
Vendor 6 2 1 2 3
relationships [54][4][53][4][54][53] [55][55] [36] [28][28] [36][60][60]
Project 2 1
management [28] [28] [17]
Contract 1
[28]
Legal 1
[30]

166
Mapped papers for risks of custom-based software development and acquisition

Risk category Risks of custom-based software development (total mapped Risks of custom-based software acquisition (total
papers/references) mapped papers/references)
Empirical Experience Critical framework Empirical Experience Critical framework
Planning 1 2 1
[25] [61][33] [31]
Requirements 5 1 1 1 1
[29][24][42][24][42] [58] [31] [42] [25]
Design 1 1
[42] [31]
Integration and 1
testing [26]
System lifecycle 1
[31]
Project closure 1
[31]
Software 1 1
[31] [43]
Cost 2
[27][27]
Environment 1
[26]
People 1 1 1
[42] [31] [42]
Vendor relationships 6 2 3 2 1
[24][42][24][24][26][26] [25][25] [27][27][27] [25][47] [34]
Project management 4
[26][26][26][42]
167
Contract 1 [24] 1
[27]
Legal 1 1
[27] [27]

References:

[1] U. Lindqvist and E. Jonsson, ‘A map of security risks associated with using COTS’, Computer, vol. 31, no. 6, pp. 60-66, 1998.
[2] Guijun Wang and G. Cone, ‘A method to reduce risks in building distributed enterprise systems’, in Proceedings of Fifth IEEE
International on the Enterprise Distributed Object Computing Conference, 2001, pp. 164-168.
[3] Vu Tran and Dar-Biau Liu, ‘A risk-mitigating model for the development of reliable and maintainable large-scale commercial-off-the-
shelf integrated software systems’, in Proceedings of the Reliability and Maintainability Symposium 1997, pp. 361-367.
[4] Jingyue Li, O. P. N. Slyngstad, M. Torchiano, M. Morisio, and C. Bunse, ‘A State-of-the-Practice Survey of Risk Management in
Development with Off-the-Shelf Software Components’, IEEE Transactions on Software Engineering, vol. 34, no. 2, pp. 271-286, 2008.
[5] G. Kotonya and A. Rashid, ‘A strategy for managing risk in component-based software development’, in Proceedings of 27th Euromicro
Conference, 2001, pp. 12-21.
[6] J. Bhuta, S. Mallick, and S. V. Subrahmanya, ‘A Survey of Enterprise Software Development Risks in a Flat World’, in Proceedings of
First International Symposium on Empirical Software Engineering and Measurement, 2007, pp. 476-478.
[7] J. D. Smith, ‘An Alternative to Technology Readiness Levels for Non-Developmental Item (NDI) Software’, in Proceedings of the 38th
Annual Hawaii International Conference on the System Sciences, 2005, p. 315a.
[8] V. N. Tran and D.-B. Lin, ‘Application of CBSE to projects with evolving requirements-a lesson-learned’, in Proceedings of Sixth Asia
Pacific Software Engineering Conference, 1999, pp. 28-37.
[9] B. M. Horowitz and J. H. Lambert, ‘Assembling off-the-shelf components: “Learn as you Go” systems engineering’, IEEE Transactions
on Systems, Man and Cybernetics, Part A: Systems and Humans, vol. 36, no. 2, pp. 286-297, 2006.
[10] R. Fulton, ‘Assuring certifiability of outsourced software development - a DER’S perspective’, in Proceedings of 24th Digital Avionics
Systems Conference, 2005, vol. 2, p. 5 pp. Vol. 2.
[11] B. Boehm and J. Bhuta, ‘Balancing Opportunities and Risks in Component-Based Software Development’, IEEE Software, vol. 25, no. 6,
pp. 56-63, 2008.
168
[12] P. Gruenbacher, ‘Collaborative requirements negotiation with EasyWinWin’, in Proceedings of 11th International Workshop on the
Database and Expert Systems Applications, 2000, pp. 954-958.
[13] A. Egyed, N. Medvidovic, and C. Gacek, ‘Component-based perspective on software-mismatch detection and resolution’, IEEE Software,
vol. 147, no. 6, pp. 225-236, 2000.
[14] V. Tran, Dar-Biau Liu, and B. Hummel, ‘Component-based systems development: challenges and lessons learned’, in Proceedings of
Eighth IEEE International Workshop on the Software Technology and Engineering Practice, 1997. [incorporating Computer Aided
Software Engineering], 1997, pp. 452-462.
[15] J. Akkanen, A. J. Kiss, and J. K. Nurminen, ‘Evolution of a software component - experiences with a network editor component’, in
Proceedings of Sixth European Conference on the Software Maintenance and Reengineering, 2002, pp. 119-125.
[16] W. Lam and A. J. Vickers, ‘Managing the risks of component-based software engineering’, in Proceedings of Fifth International
Symposium on the Assessment of Software Tools and Technologies, 1997, pp. 123-132.
[17] Yi Ding and N. Napier, ‘Measurement Framework for Assessing Risks in Component-Based Software Development’, in Proceedings of
the 39th Annual Hawaii International Conference on System Sciences, 2006, vol. 9, p. 230b.
[18] P. Maki-Asiala and M. Matinlassi, ‘Quality Assurance of Open Source Components: Integrator Point of View’, in Proceedings of 30th
Annual International Computer Software and Applications Conference, 2006, vol. 2, pp. 189-194.
[19] R. J. Ellison and C. Woody, ‘Supply-Chain Risk Management: Incorporating Security into Software Development’, in Proceedings of
43rd Hawaii International Conference on System Sciences, 2010, pp. 1-10.
[20] S. Blom, M. Book, V. Gruhn, and R. Laue, ‘Switch or Struggle: Risk Assessment for Late Integration of COTS Components’, in
Proceedings of Second International Workshop on the Incorporating COTS Software into Software Systems: Tools and Techniques, 2007,
p. 1.
[21] L. Merola, ‘The COTS software obsolescence threat’, in Proceedings of Fifth International Conference on Commercial-off-the-Shelf
(COTS)-Based Software Systems, 2006., p. 7 pp.
[22] V. Tran, B. Hummel, Dar-Biau Liu, Thu Anh Le, and J. Doan, ‘Understanding and managing the relationship between requirement
changes and product constraints in component-based software projects’, in Proceedings of the Thirty-First Hawaii International
Conference on the System Sciences, 1998, vol. 6, pp. 132-142 vol.6.
[23] Y. Yang, Jesal Bhuta, B. Boehm, and D. N. Port, ‘Value-based processes for COTS-based applications’, IEEE Software, vol. 22, no. 4, pp.
54-62, 2005.
[24] Jiangping Wan, Dejie Li, and K. Kuang, ‘Analysis of the Business Risks for the Software Outsourcing between Hongkong and
Guangdong’, in Proceedings of 4th International Conference on Wireless Communications, Networking and Mobile Computing, 2008, pp.
1-4.
169
[25] C. M. Lott, ‘Breathing new life into the waterfall model’, IEEE Software, vol. 14, no. 5, pp. 103-105, 1997.
[26] T. Moynihan, ‘How experienced project managers assess risk’, IEEE Software, vol. 14, no. 3, pp. 35-41, 1997.
[27] B. A. Aubert, S. Dussault, M. Patry, and S. Rivard, ‘Managing the risk of IT outsourcing’, in Proceedings of the 32nd Annual Hawaii
International Conference on System Sciences, 1999, vol. Track7, p. 10 pp.
[28] L. M. Abdullah and J. M. Verner, ‘Outsourced strategic IT systems development risk’, in Proceedings of Third International Conference
on the Research Challenges in Information Science, 2009, pp. 275-286.
[29] S. Serich, ‘Prototype stopping rules in software development projects’, IEEE Transactions on Engineering Management, vol. 52, no. 4,
pp. 478-485, 2005.
[30] T. R. Huber, ‘Reducing business and legal risks in software reuse libraries’, in Proceedings of Third International Conference on
Software Reuse: Advances in Software Reusability, 1994, pp. 110-117.
[31] Xiangnan Lu and Yali Ge, ‘Risk analysis in project of software development’, in Proceedings of the Engineering Management
Conference, Managing Technologically Driven Organizations: The Human Side of Innovation and Change, 2003, pp. 72-75.
[32] J. A. McDermid, ‘COTS: the expensive solution?’, in Proceedings of IEE Colloquium on the COTS and Safety Critical Systems (Digest
No. 1997/013), 1997, pp. 1/1-1/4.
[33] M. Jorgensen, ‘How to Avoid Selecting Bids Based on Overoptimistic Cost Estimates’, IEEE Software, vol. 26, no. 3, pp. 79-84, 2009.
[34] H. M. Edwards, G. M. Mallalieu, and J. B. Thompson, ‘Some insights into the maintenance of legacy systems within small manufacturing
and distribution organisations in the UK’, in Proceedings of The Twenty-Third Annual International of Computer Software and
Applications Conference, 1999, pp. 13-20.
[35] A. A. Kvale, J. Li, and R. Conradi, ‘A case study on building COTS-based system using aspect-oriented programming’, Santa Fe, New
Mexico, 2005, pp. 1491-1498.
[36] P. Vitharana, ‘Risks and challenges of component-based software development’, Communication of the ACM, vol. 46, no. 8, pp. 67-72,
2003.
[37] A. Stuckenholz and A. Osterloh, ‘Safe component updates’, Portland, Oregon, USA, 2006, pp. 39-48.
[38] G. Brændeland and K. Stølen, ‘Using model-based security analysis in component-oriented system development’, Alexandria, Virginia,
USA, 2006, pp. 11-18.
[39] S. Mahmood and A. Khan, ‘An industrial study on the importance of software component documentation: A system integrator’s
perspective’, Information Processing Letters, vol. 111, no. 12, pp. 583-590, Jun. 2011.
[40] F. Redmill, ‘Analysis of the COTS debate’, Safety Science, vol. 42, no. 5, pp. 355-367, Jun. 2004.
[41] M. Hepner, R. Gamble, M. Kelkar, L. Davis, and D. Flagg, ‘Patterns of conflict among software components’, Journal of Systems and
Software, vol. 79, no. 4, pp. 537-551, Apr. 2006.
170
[42] C. Schmidt, P. Dart, L. Johnston, L. Sterling, and P. Thorne, ‘Disincentives for communicating risk: a risk paradox’, Information and
Software Technology, vol. 41, no. 7, pp. 403-411, May 1999.
[43] S. A. Sherer, ‘Purchasing software systems : Managing the risk’, Information & Management, vol. 24, no. 5, pp. 257-266, 1993.
[44] J. Miller and H. C. Yeoh, ‘COTS acquisition process: incorporating business factors into COTS vendor evaluation taxonomies’, Software
Process Improvement Practice, vol. 11, no. 6, pp. 601-626, 2006.
[45] Y. Yang and B. Boehm, ‘Improving process decisions in COTS-based development via risk-based prioritization’, Software Process
Improvement Practice, vol. 12, no. 5, pp. 449-460, 2007.
[46] A. Mohamed, G. Ruhe, and A. Eberlein, ‘Optimized mismatch resolution for COTS selection’, Software Process Improvement Practice.,
vol. 13, no. 2, pp. 157-169, 2008.
[47] H. Saiedian, ‘Practical recommendations to minimize software capability evaluation risks’, Software Process Improvement Practice vol.
8, no. 3, pp. 145-156, 2003.
[48] G. Brændeland and K. Stølen, ‘A Semantic Paradigm for Component-Based Specification Integrating a Notion of Security Risk’, in
Formal Aspects in Security and Trust, vol. 4691, T. Dimitrakos, F. Martinelli, P. Ryan, and S. Schneider, Eds. Springer Berlin /
Heidelberg, 2007, pp. 31-46.
[49] J. Li, F. Bjørnson, R. Conradi, and V. Kampenes, ‘An empirical study of variations in COTS-based software development processes in the
Norwegian IT industry’, Empirical Software Engineering, vol. 11, pp. 433-461, 2006.
[50] N. Admodisastro and G. Kotonya, ‘Architectural Analysis Approaches: A Component-Based System Development Perspective’, in High
Confidence Software Reuse in Large Systems, vol. 5030, H. Mei, Ed. Springer Berlin / Heidelberg, 2008, pp. 26-38.
[51] D. Port and S. Chen, ‘Assessing COTS Assessment: How Much Is Enough?’, in COTS-Based Software Systems, vol. 2959, R. Kazman
and D. Port, Eds. Springer Berlin / Heidelberg, 2004, pp. 183-198.
[52] D. Port and Y. Yang, ‘Empirical Analysis of COTS Activity Effort Sequences’, in COTS-Based Software Systems, vol. 2959, R. Kazman
and D. Port, Eds. Springer Berlin / Heidelberg, 2004, pp. 169-182.
[53] B. Boehm, D. Port, Y. Yang, and J. Bhuta, ‘Not All CBS Are Created Equally: COTS-Intensive Project Types’, in COTS-Based Software
Systems, vol. 2580, H. Erdogmus and T. Weng, Eds. Springer Berlin / Heidelberg, 2003, pp. 36-50.
[54] J. Li, R. Conradi, O. P. N. Slyngstad, M. Torchiano, M. Morisio, and C. Bunse, ‘Preliminary Results from a State-of-the-Practice Survey
on Risk Management in Off-the-Shelf Component-Based Development’, in COTS-Based Software Systems, vol. 3412, X. Franch and D.
Port, Eds. Springer Berlin / Heidelberg, 2005, pp. 278-288.
[55] L. Rose, ‘Risk Management of COTS Based Systems Development’, in Component-Based Software Quality, vol. 2693, Springer Berlin /
Heidelberg, 2003, pp. 352-373.

171
[56] K. Ballurio, B. Scalzo, and L. Rose, ‘Risk Reduction in COTS Software Selection with BASIS’, in COTS-Based Software Systems, vol.
2255, J. Dean and A. Gravel, Eds. Springer Berlin / Heidelberg, 2002, pp. 31-43.
[57] D. Wu and Y. Yang, ‘Towards an Approach for Security Risk Analysis in COTS Based Development’, in Software Process Change, vol.
3966, Q. Wang, D. Pfahl, D. Raffo, and P. Wernick, Eds. Springer Berlin / Heidelberg, 2006, pp. 124-131.
[58] M. Feather and S. Cornford, ‘Quantitative risk-based requirements reasoning’, Requirements Engineering, vol. 8, pp. 248-265, 2003.
[59] S. Elgazzar, A. Kark, E. Putrycz, and M. Vigder, ‘COTS Acquisition: Getting a Good Contract’, in COTS-Based Software Systems, vol.
3412, X. Franch and D. Port, Eds. Springer Berlin / Heidelberg, 2005, pp. 43-53.
[60] M. A. Serva, S. A. Sherer, and J. C. Sipior, ‘“When Do You ASP?” The Software Life Cycle Control Model’, Information Systems
Frontiers, vol. 5, pp. 219-232, 2003.
[61] E. Rosendahl and T. Vullinghs, ‘Performing Initial Risk Assessments in Software Acquisition Projects’, in Software Quality — ECSQ
2002, vol. 2349, J. Kontio and R. Conradi, Eds. Springer Berlin / Heidelberg, 2006, pp. 146-155.

172
Appendix D: The framework inputs

The participant of case 1Dev is the developer (the acquirer is the stakeholder of the participant), the participant of case 2AcqBank is the acquirer
(the developer is the stakeholder of the participant), the participant of case 3AcqTelco is the acquirer (the developer is the stakeholder of the
participant), the participant of case 4AcqGovt is the acquirer (the developer is the stakeholder of the participant).

Input of the framework Case 1Dev Case 2AcqBank Case 3AcqTelco Case 4AcqGovt
Risk Developer Acquirer Developer Acquirer Developer Acquirer Developer Acquirer
Item RI RC RI RC RI RC RI RC RI RC RI RC RI RC RI RC
Selection effort ill-estimated Hi Hi Lo Lo Hi Hi Hi Hi Hi Hi Hi Hi Hi Hi Hi Hi
Lo
Not adaptable to requirement
Hi (was Hi Hi Lo Lo Hi Hi Hi Hi Lo Lo Hi Hi Hi Hi
changes
Hi)
Hi
Requirements not negotiable Hi Hi Hi Hi Lo Lo Hi Hi Hi Hi Hi Hi Hi Hi Lo (was
Lo)
Complicated multi-OTS
Hi Lo Hi Hi Hi Lo Hi Lo Lo Lo Hi Hi Lo Hi Lo Lo
components arrangement
Hi Hi
Insufficient OTS component
Hi (was Hi Hi Hi Hi Hi Lo Hi Hi Lo Lo Hi Hi Lo (was
documents
Lo) Lo)
Lack of OTS-driven
requirements engineering Hi Hi Hi Hi Hi Hi Hi Hi Hi Hi Hi Hi Lo Hi Lo Hi
process
Maintenance planning Hi Lo Lo
Lo Hi Lo Lo Lo Lo Lo Hi Hi Lo Hi Lo Hi
unfeasible (was (was (was
173
Lo) Hi) Hi)
Hi
Upgrade unfeasible Lo Lo Lo Hi Lo Lo Lo Lo Lo Lo Lo Lo Hi Hi Lo (was
Lo)
Lack of information on provider Hi Hi Hi Hi Hi Hi Hi Hi Hi Hi Hi Hi Hi Hi Lo Lo
Lo
Lack of support Hi (was Lo Hi Hi Hi Lo Lo Hi Hi Hi Hi Hi Hi Hi Hi
Hi)
Hi
Reduced control of future
Hi Lo Hi Hi Lo Lo Lo Lo Hi Hi Hi Hi Lo Hi (was Lo
evolution of the system
Lo)
Legend: RI: risk impact, RC: risk control, Hi: high, Lo: low.

174
Appendix E: Cross-case analysis for data collection 1 and 2, excluded inputs to the framework and evaluation
criteria for risk assessment

Question Participant’s answers in


Case 1Dev Case 2AcqBank Case 3AcqTelco Case 4AcqGovt
Q1.2.1 Shared understanding Shared understanding Meeting the requirements Managing risk of
development
Shared responsibility
Focusing and prioritising
Stakeholder commitment tasks
Q1.2.2 Discussion User Acceptance Testing No formal analysis
(UAT)
Agreement based on a
Proof of Concept (POC) scheduled meeting of tasks
before implementation of the project.
Q1.2.3 Both developer and stakeholder Both project managers (of Both developer and Acquirer
developer and acquirer) stakeholder
Q1.2.4 A project manager and system Member of Proof of Concept Web development and A project coordinator
analyst (POC) and operation coordinator of application
implementation
Q2.2.1 Brainstorming and discussion Different approaches and Collective risk planning and Scheduled meeting and no
perceptions between developer involvement in formal risk management
participant and its developer OTS component selection

175
Q2.2.2 and Using past project data Documented in the project Integrated with budget Risk identification:
Q2.2.3 contract (not including management brainstorming and
Risk identification: brainstorming detailed technical aspects) discussion
and discussion Based on cost/revenue
Risk identification: analysis Risk analysis and
Risk analysis and prioritization: brainstorming and discussion prioritization: follow-up
discussion and using past project Risk identification: based on the discussion
data to rank risk Risk analysis and brainstorming and
prioritization: planning and discussion
review meeting
Risk analysis: what-if
scenario

Risk prioritization: based on


audit, loss revenue and cost
Q2.2.4 The one who has responsibility on Stakeholder having Defined in the software Technical and non-
risks knowledge (on a risk) contract technical understanding of
risk
Defined in Scope of Work Defined in the project Acquirer organization's
contract policy about stakeholder role Communication to achieve
Risk outside responsibility of and responsibility on system common understanding
acquirer and developer must be development
discussed with respective Resource and capacity
stakeholder Negotiation
Terms of reference
Q2.2.5 Detailing Scope of Work (after Defined in software contract Describing level of each Common understanding of
using the framework) stakeholder and shared the definition of control.
responsibility (to avoid
tendency to determine Detailed communication
responsibility to other instruments, e.g. meeting
176
stakeholder) and Terms of reference

Adding cost/benefit analysis


and comparison
Q2.2.6 The use of the framework as risk The use of the framework to The use of the framework to Common understanding of
expectation and assessment method understand risk control, understand risk control, the definition of control and
impact and responsibility impact and responsibility impact (of processes and
risks)
The use of the framework as
risk expectation and
assessment method

The use of the framework to


check relationships between
risks

The expectation to modify


the input of the framework
Q2.2.7 After Design Review Meeting The beginning of the project The beginning of the project It can be integrated to
(requirement and OTS software process
In the project tender selection)
(The use of the framework in the
project tender is better than after To integrate to existing risk
design review meeting) management, the framework
should be added with other
attribute (e.g. cost)
Q2.2.8.a Cannot predict what the other Cannot predict what the other Cannot predict what the Cannot predict what the
stakeholder’s opinions about risk stakeholder’s opinions about other stakeholder’s opinions other stakeholder’s opinions
control and impact risk control and impact about risk control and about risk control and
because of different interests impact impact unless both
177
stakeholders have a
common understanding
Q2.2.8.b This framework (in Chapter 6) is This framework (in Chapter This framework (in Chapter This framework (in Chapter
better compared with the method 6) is better compared with the 6) is better compared with 6) is better compared with
used in Chapter 5 method used in Chapter 5 the method used in Chapter the method used in Chapter
because the framework 5 because the framework 5 because the framework
Using the framework, stakeholder compared actual risk compared actual risk provided structured,
should be able to determine risk perceptions perceptions and use simple methodical and measured
control level of risk control and approach to analyse risk
impact control and impact

178

You might also like