Received: 18 November 2019 Revised: 14 March 2020 Accepted: 20 April 2020

DOI: 10.1002/smr.2269

RESEARCH ARTICLE - EMPIRICAL

Toward ontology-based risk management framework for

software projects: An empirical study

Temitope Elizabeth Abioye M.Sc.1 | Oluwasefunmi Tale Arogundade1 |

Sanjay Misra2,3 | Adio T. Akinwale1 | Olusola John Adeniran4

1
Department of Computer Science, Federal
University of Agriculture, Abeokuta, Ogun, Abstract
Nigeria Software risk management is a proactive decision-making practice with processes,
2
Department of Computer Engineering, Atilim
methods, and tools for managing risks in a software project. Many existing tech-
University, Ankara, Turkey
3
Department of Electrical and Information niques for software project risk management are textual documentation with varying
Engineering, Covenant University, Ota, Ogun, perspectives that are nonreusable and cannot be shared. In this paper, a life-cycle
Nigeria
4
approach to ontology-based risk management framework for software projects is
Department of Mathematics, Federal
University of Agriculture, Abeokuta, Ogun, presented. A dataset from literature, domain experts, and practitioners is used. The
Nigeria
identified risks are refined by 19 software experts; risks are conceptualized, modeled,
Correspondence and developed using Protégé. The risks are qualitatively analyzed and prioritized, and
Oluwasefunmi T. Arogundade, Department of
aversion methods are provided. The framework is adopted in real-life software pro-
Mathematics, Federal University of
Agriculture, Abeokuta, Nigeria. jects. Precision recall and F-measure metrics are used to validate the performance of
Email: arogundadeot@funaab.edu.ng
the extraction tool while performance and perception evaluation are carried out
using the performance appraisal form and technology acceptance model, respec-
tively. Mean scores from performance and perception evaluation are compared with
evaluation concept scale. Results showed that cost is reduced, high-quality projects
are delivered on time, and software developers found this framework a potent tool
needed for their day-to-day activities in software development.

KEYWORDS

hierarchical risk classification breakdown structure (HBRS), performance appraisal form (PAF),
software development life cycle (SDLC), software ontology-based risk management (SORM),
software risk ontology (SRO), technology acceptance model (TAM)

1 | I N T RO D UC TI O N

A software project is a process involving many activities such as domain analysis, requirement specification, communication with developers
and end users, designing and production of various artifacts, evaluating and testing of software products, and installation and maintenance of
application at the end user's site.1
The demands on software development are increasing daily due to an increase in knowledge of information technology. More people are
aware of the importance of information systems and how tasks can be carried out easily by adopting them. Also, the consideration of the
world as a global village is one of the major quests for information technology, which has led to a global increase in the demand for informa-
tion systems. With the increase in demand for information systems, it is important to know that software development is a complex task due
to various artifacts and the phases involved in its implementation. Although software projects management is gaining more recognition from
industry, military, finance, and academia, yet many projects are uncompleted and could not be delivered because of its failure to meet up
with the initial requirements. Some projects that are delivered failed to meet up with the scheduled time, which invariably leads to an

J Softw Evol Proc. 2020;e2269. wileyonlinelibrary.com/journal/smr © 2020 John Wiley & Sons, Ltd. 1 of 24
https://doi.org/10.1002/smr.2269
2 of 24 ABIOYE ET AL.

increase in the budgeted cost. These failures have to do with the facts that software projects are complicated undertakings, relying on human
knowledge and interactions.2 In lieu of this, the delivery time of the finished product is mostly delayed, the required specified systems are a
mirage, and the need to produce software at a reasonable cost is rarely achievable. In this context, high quality and productivity are critical
conditions for software development.3 To deal with these problems, it becomes essential to incorporate risk management into the software
development process. For maximal effectiveness, risk management process should be introduced throughout the phases of the software
development process.
Risk management is a means to improve project performance via systematic identification, appraisal, and management of project-related
risks.4 Like this, software risk management is a procedure used in the control and monitoring of project risks, which involves processes, methods,
and specialized tools. Boban et al5 stated that “Risk is inherent in the delivery of software projects; therefore, the risk is said to be quality of
chances of losing something with the effect on projects.” Because each project to be undertaken possesses an extent of risks invariably, all pro-
jects have significant risk level.6 Software project risks are boosted with a high rate of network and media connectivity. The development of hi-
tech equipment has increased the possibility of high risk for many establishments. Risk can be described as the possibility of loss and uncertainty;
thus, it involves the result of combining the negative effect if happens and its impact, and it is without an exact value. Existing studies have
highlighted the various factors affecting software development, which include quality, cost, scope, and time to market. Often, we assume that
everything will go well as it is planned in building successful software projects; it is not always so due to the aforementioned factors. These factors
militating against software projects attractiveness are called risks.7
The type of risk management method used in this study is referred to as proactive or anticipatory risk management. It involves looking for-
ward to any system failure, the causes of the failure, the risks involved and how to deal with them, and the adequate aversion methods for such
category of risks. Risks cannot be eliminated from the software project, but it can only be reduced to some extent and then managed. Therefore,
the risk management process is very crucial to the success of any software development, and it is a strategic aspect of all software projects.5
Software development is a knowledge-driven sector that depends on employees' expert knowledge to produce an output, and expert
knowledge is majorly tacit.8 Therefore, reuse and sharing of expert knowledge are vital processes in software development. Because expert
knowledge is mainly tacit,9 accessing this knowledge at the right time is difficult and depends on the willingness of the expert to share
knowledge at that point in time.10 Inaccessibility to the right knowledge causes software experts to make uncertain decisions, which invari-
ably have a negative impact on software development. Also, software project performances are faced with some risk factors due to little or
no work experience in the side of the software developers.11 Another risk types could be generated due to client's inexplicit software
requirements specification documentation.12 In the development of software systems, there are various development process models to
adopt in order to achieve a good result. A particular process model should be chosen and strictly adhere to in the implementation of soft-
ware projects; failure to do this could result to the discovery of new risk factors, which can lead to the failure of the project. Risk manage-
ment plays a vital role for software experts, and they are expected to be furnished with knowledge in this domain so that they can perform
effectively.13,14 The major issue in software risk management is the effective reusability of the risk knowledge repository, which can assist in
resolving many challenges faced in the development of software systems. Consequently, the objective of this study is to improve software
development through the effective usage of ontology-based software risk management framework that fosters knowledge sharing, extraction,
and reuse.
In the present work, the knowledge repository is built using tacit knowledge acquired through interview of software experts and extracted
explicit knowledge from literature and organizational documents. In this research, the proposed software risk management approach is of great
help to the selected software developers in the development of efficient software systems. It offered great assistance in risk identification, analy-
sis, and mitigation.
Considering the great value placed on software projects, it is important to carry out a well-planned and ordered procedure to categorize soft-
ware risks using spiral software development life cycle (SDLC) model so that risks can be prioritized and mitigated and the residual risk can be
documented and accepted.15
The remainder of this study is structured as follows: Section 2 presents SDLC phases and model, risk management process, software risk
ontology (SRO) development, and its scope as the literature review and background to the study. Section 3 presents a research framework
design and architectural framework as the research methodology. Section 4 presents software ontology-based risk management (ORM)
(SORM) development and SRO dynamic extraction tool. Empirical validation, implementation of the framework on software projects, ontology
validation, and field trial evaluation are presented in Section 5. Section 6 discusses the result and threat to validity. Finally, Section 7 presents
a conclusion.

2 | BACKGROUND AND LITERATURE REVIEW

The following subsections provide background and literature review. In Section 2.1, various terms and processes used in this paper are provided.
In Section 2.2, related works on software ontology risk management are presented.
ABIOYE ET AL. 3 of 24

2.1 | Background

The background study includes software risk ontologies, the SDLC phases and model used in this research, risk management process, SRO devel-
opment, and the ontology scope.

2.1.1 | Ontology and informal retrieval

In recent years, the development of ontologies—the formal specification of shared conceptualization16 in a domain and relations among them—
had become common on the World Wide Web. Many disciplines now develop standardized ontologies that domain experts can use to share and
annotate information in their fields. Medicine, for example, has produced large, standardized, structured vocabularies such as SNOMED and the
semantic network of the Unified Medical Language System.
Information retrieval (IR) is a technology that has to do with the representation, storage, and organization of, and access to information
items.17 The objective of IR is to help the users to locate relevant and vital knowledge quickly and with ease. Because timely delivery of well-
specified software systems, which falls within the specified budget, is important for organizational productivity, then the speed with which soft-
ware experts can access and reuse knowledge and equally share information plays a vital role in the delivery of zz. This field of technology is rec-
ognized as an essential area in the timely delivery of successful software systems. Therefore, the use of IR in accessing information pertaining to
software risks based on ontology is important to this study.

2.1.2 | Software risk management ontologies

Ontologies18 are the connector that joins knowledge and subprocesses together, which pave way for the content-oriented interface. Recently,
ontologies development—explicit formal specifications of the words in a domain and its relationship—is frequent on the World Wide Web. Most
of the technical problems combating shared and reusable knowledge-based software risks are heterogeneous hardware platforms; programming
language and protocols are well-taken care of with the use of ontologies. Ontologies provide three levels of communication to take care of these
challenges. They are representation language format, agent communication protocol, and specification of the content of shared knowledge, which
can be done using ontology.19,20
Ontology, therefore, is a shared formal conceptualization of a domain that provides a good platform for solving some of the existing problems
of risk management in a software project. The reasons for integrating ontology with software projects risk management include the following:

i Ontology serves as a standard vocabulary that will promote a shared understanding of key concepts of risks in software projects.
ii It promotes knowledge reusability.
iii It acts as a platform to formalize risks in software development such that experts in various places can access the information.
iv It promotes a good platform for risk management in a software project that will support professional purposes.21

Meanwhile, there is a need to understand the risk management process and its importance in software projects development. The fundamen-
tal reasons organizations implement a risk management process for their software projects are to minimize the negative impact on their business
processes and to serve as a sound foundation for decision making. The demands for software projects are increasing daily due to an increase in
knowledge of information technology. In this context, high quality and productivity are critical conditions for software development.3 To deal with
these challenges, it becomes essential to incorporate risk management process into the software development process and at the initial phase of
the software development process. A software project has five phases: requirement analysis, the design, implementation and unit testing, integra-
tion and system testing, and operation and maintenance. However, software project development may involve the usage of these phases at the
same time. Therefore, risk management is the same in respect of the SDLC that is being assessed. Risk management is an iterative and important
process that should be incorporated into all the SDLC phases.22 Generally, risk management involves the following steps:

i Risk identification—This phase lays emphasis on the threat in a project, and its aim is to be proactive by preempting what might go wrong in
the project. Risks discovered in the previous projects should be noted.
ii Risk analysis—This involves the quantitative and qualitative evaluation of risks as the case may be so as to ascertain the impact on the pro-
ject. In the case of software projects, qualitative risk analysis is normally used due to the fact that software risks cannot be evaluated
numerically.
iii Risk planning—This concerns mitigation/aversion plans for prioritizing risks. The purpose is to reduce the probability or the effect of risks on
the project before it happens.
4 of 24 ABIOYE ET AL.

iv Risk monitoring—The managed risks need to be properly monitored as the project progresses so as to avert any sudden risks that might
occur.

2.1.3 | Software development life cycle

Software development life cycle is a series of activities organized in sequential order of building desirable or successful software systems. It con-
sists of various tasks like feasibility study, requirement elicitation, requirements analysis, the design, implementation, system testing, deployment,
and maintenance. SDLC involves models and standard methodologies that must be adhered to by the software developers in order to build suc-
cessful systems.23 Sound knowledge of SDLC is a key point in achieving desirable software systems.24 Numerous SDLC models are available
depending on the type of system to build. For this study, a new adaptive SDLC based on the spiral model is used. The spiral model is a merger of
the iterative development process model and the traditional model with an emphasis on risk reduction. This model divides software projects into
smaller bits with associated variance risks. The development starts in the midpoint of the spiral where the project is small, which allows the project
to be easily managed with low risk. Though the project kicks off small with fewer risks to handle; thereafter, the work widens in the next iteration
to handle more risks. It is worthy to note that the prototype of the software project is developed at the end of each iteration. In this study, five
phases shall be considered, which are (1) requirement analysis, (2) the design, (3) implementation and unit testing, (4) integration and system test-
ing, and (5) deployment and maintenance.

2.1.4 | SRO development

Ontology development or a domain model is a crucial step in software risk knowledge-based development process and has gained the attention
of many people in the information technology sector.25 As often said, there is no one “correct” way or methodology for developing ontologies;
the best way is to develop it based on the intended application and the anticipated extensions in mind. SRO development is founded on extensive
interviews with software experts, extracts from literature, and organizational documents. The risks are gathered through a semistructured inter-
view with 10 software experts: the least years of experience among these experts' range between 5 and 10 years. These mentioned channels are
selected in making sure that all the risks affecting the successful completion of software projects are well represented and also to provide a means
of having a well-developed and populated software risk knowledge based, which is reusable without loss of important ontological concepts. The
steps taken in building SRO involve the extraction of important terms from text and definition of the concept taxonomies, relations, attributes,
and instances.
Ontology development is an iterative process, which requires a lot of skills and expertise, and it is time-consuming because of so many pro-
cesses involved. The main approach used in the development of SRO is based on the prescriptions stated by McGuinness and Boyce20 and
Natalya and McGuinness.25

2.1.5 | Determine the scope of ontology

Determination of ontology scope plays a major role for a better understanding of the intended uses and the users.26 The need and the purpose of
SRO are to provide the software professional and the domain experts with a shared conceptualization of rich software risks and to provide the
aversion methods to the identified risks. The main users of SRO include the following:

i Knowledge-driven application: It is an application that will exploit knowledge on risks and risk management process to realize the objective
of SRO.
ii Software expert: This is a software developer seeking knowledge on the risks that might affect software project being undertaken.
iii Domain expert: This is a person in charge of maintaining the relationship and concepts in SRO. The work of the domain expert includes
matching the SRO as new risks are discovered by the software expert through their day-to-day activities.

2.2 | Literature review

Rene Robin and Uma27 showcased the development of an ontology for risk management by considering only the risk identification process. The
risk classes and subclasses are categorized and modeled using the technical development process. Another study by Julio et al28 presented the
usage of specialized tools as indicators for capturing information, which will identify and analyze risks in a particular environment. The main risk
ABIOYE ET AL. 5 of 24

management process is not mentioned in the study. Also, in a work by Hijazi et al29 theoretical list of software risks at each phase of the SDLC is
compiled without formal representation. In addition, there is a description of the software project management framework by De Amescua et al30
that comprises a static structure, which is shown in an entity-relationship diagram. The framework proposed in this study provides the project
manager who is in charge of all the activities of the development of the software access to a knowledge base. The project managers can store and
retrieve information on software project risk management related to all the software projects being undertaken in their organizations. The
authors31 investigated the extent to which the suggested value proposed by ISO9126 quality model for software products matched this standard
and the procedure in software quality management and risk management correlates to user value prescriptions and experiences. The research car-
ried out by Baumeister and Llg32 presented an improved activity-based approach, which has its foundation on business main cost data, which can
readily be incorporated into management accounting systems. More so, this approach has definite cost forecasting and control, which is connected
with risk analysis. In a recent study,33 the authors discussed the sematic documentation approach, which involves the use of multiple ontologies
together with documents having ontology syntax in making the content of these documents understandable by the computer, which promote
reusability. The aim of the study is to have quick access to information in recorded data in documents and spreadsheets having to do with scope,
time, and cost management in software projects. Another study34 about current trends in software project management proposed communication
risk as a superrisk in the development of software projects. They emphasized the communication risks related to another category of risks in a sur-
prising way and that the software project is a complex undertaken, which involves the cognitive nature of its development. Therefore, communica-
tion risk can solve the problem of freedom provision and flexibility to software experts and the risk related to freedom and flexibility.
A study35 presented 127 risk management practices for agile methods because the agile software development process does not have partic-
ular activities to manage risks. They identified risk management practices with the aid of analytic hierarchical process, a multicriteria process
coupled with modeling, which is used as a scientific ranking method. These are linked to components and subcomponents to enhance the ranking
and statistical analysis. The result showed improvement in risk management in agile software projects, which enhances the delivery of more suc-
cessful projects. In a research by Tavares et al36 the usage of qualitative risk management analysis on Scrum software projects is discussed. They
found out that the failure of software projects depends majorly on improper use of and/or lack of risk management process. A work37 presented
the impact of conceptual factors such as project characteristics, project risk management team, risk identification approaches, and project quality
on the level of project risk and the relationship among these factors. Furthermore, it analyzed the effect of project risk and the extent of the effect
of residual performance risk on the last subjective and objective project performance for software experts to establish effectiveness and adequacy
of the proposed risk management practices.
Comparing this research work with some selected related works, we discovered that Rene Robin and Uma used ontology to develop their
knowledge repository. The risk classes and subclasses are categorized and modeled using technical development process.27 Also, Hijazi et al29
used SDLC in their risk identification process. Meanwhile, Tavares et al36 used qualitative risk analysis only on Scrum software projects. Bastos33
used ontologies to develop their knowledge repository for recorded data for reusability in order to enhance the development of software projects.
Likewise, Dey et al34 identified risks but did not use spiral SDLC model. The research carried out by Sariglannidis and Chatzoglou37 analyzes the
effect of project risks and the extent of the effect of residual performance risk on software project performance for software experts. Finally,
Tavares et al35 used statistical risk analysis and ranking for their risk analysis and risk prioritization methods. It is worth to note that all these
works only touched one aspect of risk management process, which is risk identification, except Julio et al,28 Tavares et al,35,36 and Sariglannidis
and Chatzoglou37 who carried out a risk analysis. Other aspects of risk management such as risk prioritization and risk aversion are not dealt with
except in this study. The comparative analysis of this study with the related works is presented in a simplified form in Table 1.
Knowing these facts about the successful development of software projects fosters the incorporation of risk management practices, which
must be used iteratively in software projects.

3 | R E S EA R C H M E T H O D O L O G Y

The aim of this work is to develop a generic ORM framework that is reusable and can be shared, which will serve as a guide for developing effi-
cient and high-performance software projects. SORM supports a practical principle as it tries to give practical answers and output38 in the area of
software project development. A design approach that follows build and process procedure, as opposed to the design science research methodol-
ogy, according to previous studies39,40 shall be considered. The benefit and beauty of this study are that it proffers practical solutions to chal-
lenges facing the development of quality software products. On the basis of this, more attention is given to the framework design, which is the
backbone for the implementation of SORM.

3.1 | Framework design process

The design of this research stands to answer four questions as depicted in Figure 1.
6 of 24 ABIOYE ET AL.

TABLE 1 Comparative analysis

Catered for
ontology for Risk identification Catered for risk Catered for
the knowledge using spiral SDLC Catered for risk prioritization risk aversion
Authors repository model analysis process process process
Rene Robin and Yes No, technical No No No
Uma27 development
process is used.
Sariglannidis and No No, risk identification Yes, assessment of No No
Chatzoglou37 did not follow impact of residual
SDLC. risks is
carried out.
Hijaz et al29 No, textual Yes, risk identification No No No
documentation is done through
of risks is done. SDLC.
Tavares et al36 No No Yes, qualitative risk No No
analysis
Tavares et al35 No No Yes, statistical risk Yes No
analysis
Bastos et al33 Yes No No No No
34
Dey et al No No, risk identification No No No
is carried out.
Julio et al28 No No, specialized tool is No, specialized tool No No
used to capture is used to analyze
risk in that risk in that
environment environment
alone. alone.
This study Yes Yes, risk identification Yes, qualitative risk Yes Yes
is carried out using analysis is done.
spiral SDLC model.

Abbreviation: SDLC, software development life cycle.

FIGURE 1 Research design framework

Question 1: How do we organize the risk knowledge of software project in a way that it can be used for accessing and managing the risk through-
out the system life cycle?

a Search for data—The authors used literatures on software project risks.

b Brainstorm—Software experts with experience not less than 5 years on the field are consulted to get more information on software risks.
c Model—The authors conceptualized and used protégé ontology editor 4.3.0 to build software risk knowledge repository.
d Output—Implementation of ontology-based software risks is established.
ABIOYE ET AL. 7 of 24

Question 2: How does the incorporation of risk management process in software project development affect the outcomes of the project?

a Source—The authors engaged in literature search for the right risk management process.
b Procedure—Risk management process: risk identification, risk analysis, risk prioritization, and risk planning (mitigation/aversion plans for priori-
tizing risks) are the steps followed to implement this phase.
c Output—Implementation of ORM for software projects is developed.

Question 3: How do we analyze the possible effects of risks on software projects?

a Identify method—The authors used a literature search to come up with standard methods that are appropriate for this study. Qualitative
method of risk analysis is used.
b Identify tools—Hierarchical risk classification breakdown structure (HBRS) is used in breaking down the risks into different categories and sub-
categories, whereas analytical hierarchical process (AHP) is used in taking the decision because the qualitative method of risk analysis is
adopted. Therefore, HBRS and AHP are the tools identified by the authors.
c Procedure—Pairwise comparison using a questionnaire is used.
d Output—Prioritized risks with the appropriate aversion method.

Question 4: How do we validate the effectiveness of the ontology risk management framework?

a Development—Implementation of SORM prototype.

b Evaluation (case study)—The prototype is given to software experts to use for their new software projects.
c Output—Reports from software experts using technology acceptance model (TAM) and performance appraisal form (PAF).

On the basis of the research design process approach, this work is done through the knowledge gained from software experts during the
interview, organization materials, selected web materials, and internet resources.

3.2 | Architectural framework

According to the research methodology, adopted by authors,41 SORM framework development involved four major domains: SRO develop-
ment, the extraction of the risk knowledge base, the establishment of the SORM framework, and the dynamic ontology extraction tool. The
conceptual model for this framework is depicted in Figure 2. This work adopted six steps in the development of software project risk ontol-
ogy and ORM framework. These include requirement collection/software risk identification, risk classification, the preliminary taxonomy of
risks, expert checking, dynamic ontology extraction tool, and framework validation. SRO is the backbone of this framework. It is built by com-
bining questionnaire, brainstorming by the experts, and literature search to extract and collate risks and weed out those risks that are not part
of the software development. Ontology validation is carried out using structured questions, expert interview, and case study demonstra-
tion.19,42,43 More importantly, this study used information retrieval algorithm to develop the dynamic ontology extraction tool to supplement
and update the ontology. These activities help in building the software risk knowledge base. Selected software experts involved in the work-
shop are to give a better view of the software risk profile and add any newly discovered risks encountered during their day-to-day activities.
The SORM is developed using adaptive spiral model SDLC and risk management process. The development of ontology is knowledge inten-
sive, which comprises tactic and explicit knowledge, and therefore, it involves engaging software experts who have been on the job for up
to 5 years.

3.2.1 | Identification of software risks

Developing an ORM for software projects requires the combination of tacit and explicit knowledge to form the knowledge repository of the
system. Explicit knowledge is easily accessed from the organization documents and literature, whereas tacit knowledge resides in human heads,
and therefore, it is difficult to capture and reuse. Structured questionnaires are given to software experts during an interactive session to cap-
ture and elicit tacit knowledge. Software risks identified in this work are majorly from the literature review and a few from experts during
the interview.
8 of 24 ABIOYE ET AL.

F I G U R E 2 Conceptual model for the

proposed framework

3.2.2 | Risk classification

This phase involves grouping or classifying risks into its various categories. This stage is carried out through a review of existing classifica-
tion system and expert interview. HBRS is used. HBRS is a prominent tool that can be used to manage risks, and it involves hierarchical
charts of risks based on categories and subcategories that help in classifying potentials and their causes. HBRS represents risks either top–
down or bottom–up. Top–down starts from the goal, that is, from the highest level to the finer level, whereas bottom–up is vice versa. This
tool ensures that relevant risks are not left out, and it equally helps to uncover where most of the risks are localized and their interdepen-
dencies among one another. HBRS facilitates the monitoring and evaluation of risk and aversion methods that are suitable for the discov-
ered risks.34 In a nutshell, HBRS assists in the prioritization of risks, which invariably helps in providing an appropriate method of risk
mitigation. This phase is important in ontology for creating classes, subclasses, and the risk factors with the relationship that exist
among them.

3.2.3 | Preliminary taxonomy of risks

Taxonomy of risks involves the process of giving a unified name to risk group/class. The naming of the risk classes and subclasses follows spi-
ral model SDLC. We considered five main phases of the SDLC in this work, which are requirement analysis phase, design phase, implementa-
tion and testing phase, integration and system testing phase, and deployment and maintenance phase. Review of existing taxonomy and
expert interview are the prominent tools that are used in this phase to build feasible SRO that can be validated.

3.2.4 | Experts checking

This is the review of risk classification by the software developer so as to ascertain the conformity of the risk taxonomy with the existing
ones. In this phase, literature review, review of existing taxonomy, expert interview, and the conceptualization44 are used for the final modifi-
cation of risk concepts. In the implementation of ORM framework for a software project, the aforementioned extraction tools could also be
adopted by the software developers.
ABIOYE ET AL. 9 of 24

3.2.5 | Organization of concepts into hierarchy

Software projects risk ontology framework is broken down into risk classes, risk subclasses, and risk factors/indexes (risk index is the instance of
risk events). In order to have a feasible ontology, HBRS is used to organize each concept into their appropriate position in the hierarchy. Figure 3
shows the hierarchical SRO. FACT++ is first used to validate the ontology; software experts are involved in the usage of the ontology by querying
the ontology, and the results of the query confirm the validation of the ontology.

3.2.6 | Framework evaluation

This work is evaluated using case studies of software projects through PAF, and also, TAM is used for the perception evaluation. The dynamic
extraction tool is evaluated using extracted validity test that comprises three major metrics, which are precision, recall, and F-measure.

3.3 | SDLC-based software risks classification

In this study, HBRS is used in the management of software risks. A top–

down hierarchical chart is used in the classification of risks. Looking at Figure 3, Appendix B represents software risks, which is the focus of
this research. The first tier in the hierarchy is the five phases of the SDLC representing software risk classes/categories. The second tier is the
software risk subclasses/subcategories based on each of the five phases. A total of 20 software risk subclasses are identified. Five risk subclasses
are for requirement analysis class, which include feasibility study, requirement elicitation, requirement analysis, requirement validation, and
requirement documentation. Six risk subclasses are uncovered at the design class, and they are examining requirement document, choosing
an architectural design, programming language choice, physical model construction, design verification, and design specification. Under the

FIGURE 3 Hierarchical breakdown of

software risk
10 of 24 ABIOYE ET AL.

implementation and unit testing class, two risk subclasses are identified, and they are coding and unit testing. Integration activity, integration test-
ing, and system testing are the three risk subclasses identified at integrating and system testing risk class. Finally, four risk subclasses are uncov-
ered at the deployment and maintenance risk class. These include installation, operation, acceptance testing, and maintenance.
The third tier in the software risks hierarchy corresponds to risk indexes. This is an instantiation of software risks, which gives an insight into risk
factors. Seventy-seven risk factors are identified under 20 risk subclasses, which include and not limited to underestimation of project time, scope,
and other resources, unrealistic schedule and budget, unclear project scope, inaccurate requirement, and conflicting user requirement. Pairwise com-
parison is carried out according to Saaty45 for the determination of risk weight that is used in the prioritization of risk classes and subclasses. Detail
calculation can be provided through a request from the author. For risk indexes/factors prioritization, Likert's scale is used for pairwise comparison.
This scale ranges from 1 to 5 with these descriptions: strongly disagree, disagree, undecided, agree, and strongly agreed, respectively. The number
ticked by the software experts corresponds to the risk factor weight, which is used in prioritizing the risk factors. Risk analysis is carried out manually,
which produced the risk weight for each risk. The risk weight is used to prioritize the software risks, and appropriate aversion methods are provided.

4 | D E V E L O P M E N T OF SO R M F R A M E W O R K

The first step in this approach is the risk identification process. Most of the risks identified are obtained from literature review29 and expert inter-
view. The identified risks are arranged using HBRS. The qualitative risk analysis of SDLC could provide very strong information about software
risks; AHP is used to achieve this purpose. AHP is a multicriteria decision-making tool that permits subjective and objective reasoning in the pro-
cess of making a decision. AHP permits the major involvement of experts in reaching a compromise, and it allows project managers to reason
before making a decision. According to Saaty,45 AHP is made up of three things. They are decomposition, comparative judgment, and synthesis of
priorities. Because the qualitative analysis of risk is involved in this study, APH allows software experts in picking a number from the scale, which
is a subjective judgment unlike the quantitative risk analysis where actual figure must be provided for calculation. It helps in capturing of specific
objectives in the form of weighted criteria, which can be used in scoring software project risks. Therefore, the usage of AHP is good for qualitative
risk analysis, especially in the area of risk prioritization. The prioritization of software risks leads to the next phase in risk management, which is
risk mitigation/aversion method process. These processes and relationship are needed in building a good SRO knowledge repository. On the basis
of the purpose of this study, provision of reusable and shared software risk management framework that will serve as a reference for building suc-
cessful and high-performance software systems in the future, SORM should be able to give the list of risks at each phase of SDLC in a prioritized
way with the appropriate aversion methods. This is made possible using AHP.
Analysis of software risks is done manually using the expert choice.46 Likelihood of risk determination for classes and subclasses is done using
the AHP. This involves a comparison of risk classes pairwise to get the likelihood of their occurrences, comparing the risk subclasses pairwise to
get the likelihood of their occurrences and synthesizing the results across the hierarchy so as to determine the probability of failure of risk classes.
The risks in each hierarchical tier are compared as pairs with respect to their importance in making the decision in this study. A verbal scale, AHP
that allows the software experts to put subjectivity, experience, and knowledge in an intuitive and natural way, is used. After comparison matrices
are done, relative weights are gotten for the various software risks. The qualitative risk analysis is carried out using a questionnaire.
The qualitative risk calculations can be done using Equations 1–5.

4.1 | Normalized table and consistency ratio calculation for requirements analysis

Table 2 and blank Table 3 are provided by the researcher to software experts, and they are filled with the help of the researcher and returned.
a = column sum for each column
b = likelihood of risk
n = size of the square matrix

TABLE 2 Scale of relative importance for pairwise47

Intensity Definition Explanation

1 Equal importance
3 Moderate importance
5 Essential or strong importance
7 Demonstrated importance
9 Extreme importance
2, 4, 6, 8 Immediate values
Two activities contribute equally to the object.
Slightly favor one over another
Strongly favor one over another
Dominance of the demonstrated importance in practice
Evidence favoring one over another of the highest possible order of affirmation
When compromise is needed
ABIOYE ET AL. 11 of 24

TABLE 3 Pairwise comparison matrix on risk subclass of requirement analysis using information in Table 1

Feasibility Requirement Requirement Requirement Requirement

Risk class study elicitation analysis validation documentation
Feasibility study 1 3 3 4 5
Requirement elicitation 1/3 1 2 2 3
Requirement analysis 1/3 ½ 1 2 3
Requirement validation 1/4 ½ ½ 1 1
Requirement 1/5 1/3 1/3 1 1
documentation
Column sum (a) 2.117 5.33 6.833 10 13

Table 3 is derived by dividing each cell by column sum to get a normalized table.

X
a= ðsum of the entire columnÞ ð1Þ

X ðsum of number across the rowÞ

b= ð2Þ
n

X
ʎmax = ðcolumn sum x like hood of riskÞ ð3Þ

ʎmax – n
Consistency Index ðCIÞ = ð4Þ
ðn− 1Þ

CI
Consistency Ratio ðCRÞ = x 100 ð5Þ
RI

If the consistency ratio is below 10%, it is acceptable; otherwise, the data are rejected.
On the basis of these calculations, risk classes and subclasses are prioritized using the value derived under the likelihood of risk column in nor-
malized matrix table. The result of risk analysis gives a better understanding of risk mitigation/aversion methods to apply. The four risk mitigation
methods are avoidance, reduction, transfer, and absorption.
Table 2 is a ranking scale that is used for software risks pairwise comparison evaluation. The scale values are assigned to the risk classes and
subclasses based on the software expert's decision. The ranking scale is from 1 to 9. This table provides information for the objective and subjec-
tive judgment of the software developers, and it plays a vital role in getting the software risks prioritized.
Table 3 is a sample of an assessed questionnaire by the software developers.
Table 4 is a normalized cell table derived through calculation from Table 3.
Table 5 is important in decision making by the researcher. The purpose of this table is to enable the researcher to accept or reject the
assessed questionnaire by the software developers. The point is that if any of the CR is greater than 10% and the assessment is not consistent,
then such decision is rejected.

TABLE 4 Normalized matrix to determine the likelihood of risk derived

Feasibility Requirement Requirement Requirement Requirement Likelihood of

Risk class study elicitation analysis validation documentation risk (b)
Feasibility study 0.472 0.563 0.439 0.400 0.385 0.4518
Requirement 0.157 0.188 0.293 0.200 0.231 0.2138
elicitation
Requirement analysis 0.157 0.094 0.146 0.200 0.231 0.1656
Requirement 0.118 0.094 0.073 0.100 0.077 0.0924
validation
Requirement 0.094 0.063 0.049 0.1 0.077 0.0766
documentation
Column sum 2.117 5.33 6.833 10 13
12 of 24 ABIOYE ET AL.

TABLE 5 Consistency ratio (CR) of each risk class of the select experts

Risk classes CR (%)

Requirement analysis 3.42
Design 2.90
Implementation and unit testing 0
Integration and system testing 2.02
Operation and maintenance 3.88

4.2 | Development of dynamic SRO extraction tool

Software risk ontology development involved many experts interview and documents analysis. Due to this fact, its update is a vital research issue
in order to maintain the reusability of this ontology. A dynamic ontology extraction tool is developed in this study to solve this problem so as to
supplement and renew the ontology. Thus, IR algorithm-based dynamic risk profile analysis tool is developed. IR models are implemented to give
the analytical capacity to measure document relevancies.48 The usage of IR algorithms can assist the summarization of the document through the
extraction of important terms.49 Therefore, the document could be represented as a set of important term vectors and is equally referred to as a
set of semantically relevant keywords. More importantly, document relevancies are calculated based on these important terms. Through the
timely analysis of IR algorithm, the ontology extraction tool can support the dynamic update of ontology.
The similarity can be calculated by the cosine value of two keyword vectors: kwi and kwj, as seen in Equation 9.50 Equations 6–8 define
the term weighting calculation of each keyword. Also, the coefficient 0.5 is used to normalize the word frequency suggested by Salton and
Buckley.51

wdj,kw = ð0:5 + 0:5 x tf dj, kw Þ x idf kw w   ð6Þ

frekw
dj,kw = 0:5 + 0:5 x xloglog n N
fre kw

frekw
tf dj, kw = ð7Þ
fre

N
idf kw = loglog ð8Þ
nkw

      Pn
i = 1 wdi, kwj x wdi, kwk
similarity kw j !, kwk ! = Cosine kw j !, kwk ! similarity kw j !, kwk ! = qffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffi
Pn 2 qffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffi
Pn 2 ð9Þ
i = i di, kwj x
w i = i wdi, kwk

wdi, kw is the term weight of keyword kw in document di; frekw is the frequency of each keyword; max fre is the maximum frequency of keyword
Psj
kw in document di; i = 1 fre is the total frequency of every keyword in each document; N is the total document numbers of the knowledge base;
nkw is the total numbers of the document in which certain keyword appears.
The design of SRO extraction tool is to dynamically extract important risk factors so as to supplement the SRO. Hence, extraction validity
needs to be validated. In this study, primary metrics—precision, recall, and the F-measure (harmonic mean)—are used to confirm the extraction
validity, and these are described in Equations 10–12, respectively. These metrics have been used by researchers in the field of information
retrieval.52

Relevant Extracted Risk concepts

Precision = ð10Þ
Extracted Risk concepts

Relevant Extracted Risk concepts

Recall = ð11Þ
Relevant concepts

Precision  Recall
F −measure = 2  ð12Þ
Precision + Recall

The F-measure score simply means the weighted average of the precision and recall; and it is at best value when the score is 1, and it is the worst
value when the score is 0.
ABIOYE ET AL. 13 of 24

The design is divided into two phases—user interface and the software risk knowledge extraction machine. The latter comprises the risk con-
cepts, which is being populated by the SRO and the supplement tool. The risk concepts make up the software risk knowledge base; this is where
the risks concepts, its relationship, and the rules that govern them reside.
An application interface is developed for the software risks ontology. This is done to assist a prospective user who is seeking knowledge on
the risks facing the successful completion of software projects to access the information with accuracy. This application is implemented using the
Java programming language (NetBeans IDE). With this graphical user interface, users are given an opportunity of searching for specific terms per-
taining to software risks, its classification, aversion methods, and risks with high rating. The user interface is where the user is able to query
the ontology via the knowledge base, and if the information matches, there is a response to user query; otherwise, the user is asked to resend
his query.

5 | E M P I R I C A L V A L I D A T I O N O F SO R M

The complete process of implementations of our model on real projects from industry is presented in the following subsections.

5.1 | Implementations on industrial projects

The effectiveness of this framework is validated by implementing SORM framework for software projects undertaken in four different organiza-
tions. These organizations are selected because of easy accessibility to software developers currently working on new projects, which made it
possible for the proposed framework to be used for the development. The companies are the information technology unit of Leadwell Assurance
Plc, Lagos. This company is established in 1970, and it offers various insurance policies such as burglary insurance, fire insurance, health insurance,
life insurance, and all-risk policy; LightUp Software Limited, Ibadan, is established few years back with the purpose of providing solutions to infor-
mation technology problems of stakeholders. The remaining two companies are Datacom Limited and ATB Consulting both in Lagos, Nigeria.
The scope of this implementation includes (1) software project risk ontology development and extraction of the software risk knowledge
base; (2) implementation of the proposed ontology-based software risk management framework; and (3) modification of ontology extraction
tool.19 In this study, the interview is conducted with some software experts to review the risks discovered and how feasible is the application of
this framework. Lastly, this study validated this framework through an expert interview using the PAF and TAM using perception form. The
dynamic extraction tool is equally validated using primary metrics—precision, recall, and F-measure.

5.1.1 | Requirement collection/identification of software risks

Software risks knowledge base is formed through dataset from literature and organizational documents; this knowledge repository is used as a
reference in the project undertaken. There is interaction with the software expert as the projects are ongoing to extract the tacit knowledge from
these domain experts.

5.1.2 | Risk classification

Because the software identified in this research used spiral model SDLC as means of classifying the identified risks, few risk factors discovered
during the case study are fixed at their appropriate subclasses such as lack of proper interaction with the user, building the software based on
assumptions. Therefore, HBRS is used for successful risk classification in this regard. Figure 3 depicted the software risks hierarchical breakdown
structure.

5.1.3 | Experts checking

From the previous discussion, several tools could be used to get important concepts from the software risks knowledge base. At this phase, docu-
ment analysis and expert interview are the prominent tools used. A knowledge base is built through risks gathered from the software being devel-
oped coupled with the previous knowledge base. A total number of 77 risk factors are gathered under the five phases of the SDLC. After the
software risk knowledge and information are merged together, risk profiles analysis is done. In this research, the risk profile is gathered, which
forms the SRO knowledge base. Risk profiles, which are also known as risk events, include risk event description, risk cause and its consequence,
14 of 24 ABIOYE ET AL.

risk analysis, and mitigation strategies. In lieu of this, software expert is given a task to carry out a survey to assemble risk events on the project
being undertaken as at that time. A total of 56 risk profiles are assembled from 10 different projects.
The risk profile together with their associated risk factors is refined through expert interview. The software experts involved in this review
have a minimum of 5-year working experience and are consulted based on their experience of being on the job (software development and analy-
sis) consistently for the last 5 years. Having done all these, the extraction of well-represented software risk knowledge base is perfected.

5.1.4 | SRO development

Because SDLC is used in risk identification, the same process is applied at this step. HBRS is used to break down the risks into their various cate-
gories. According to Figure 3, the apex is the overall goal while the first tier represents five software risk categories, the second tier represents
19 subcategories, and 77 risk factors/indexes are represented at the third tier of the hierarchical structure (check Appendix B). Figure 4 depicted
the onto-viz of SRO development.

5.1.5 | Implementation of the ORM framework for software projects

After the identification of risks and the development of feasible SRO, next is the analysis of these risks. In order to get the classes and subclasses
weight, AHP is used for this purpose. AHP allows for maximum pairwise comparison on risk classes and subclasses. The structure of the question-
naire is available through request. Likewise, Likert's scale is used to know the risk rating for the risk factors. The analysis of these risks gave birth
to the prioritization of risk. With the aid of various risk weight derived from the AHP questionnaire, adequate aversion methods are provided.
This questionnaire is designed for software experts with a minimum of 5 years of experience. A total of 25 questionnaires are distributed; all
are returned, while six out of it are not consistent. The questionnaires are filled by the experts though the researcher is present to render assis-
tance in explaining any area that seems difficult to understand due to the complexity of the structured questions. Because software project can-
not be assigned value to, only qualitative analysis of risks is done.

5.2 | Implementation and assessment of SRO extraction tool

The beauty of this research is the extraction of important risk concepts and the future update of the SRO knowledge base. In order to achieve this
objective, the SRO extraction tool is dynamically designed to fulfill this purpose. The implementation of this tool is done in collaboration with the

FIGURE 4 Software risk OWL-viz

ABIOYE ET AL. 15 of 24

software expert team using scenario for the tool. On the basis of the nature of software risks, the risk classes and subclasses are static, while the
risk indexes are dynamic and need updating by the software expert team. Therefore, the tool is designed for the selected software developers to
retrieve these concepts through the risk subclass and any valid risk concept. The prototype extraction tool for the selected software experts is
shown in Figure 5. To validate the performance of the extraction tool, four retrievals are carried out to test the extraction validity. Table 6
depicted the extraction performance of the risk concept test retrievals. This study pays more attention to using the tool for risk concepts extrac-
tion rather than relevant risk profiles extraction. The precision recall and T-measure rate52 are adopted and adapted to assess the performance of
the proposed extraction tool using Equations 10–12.
Subsequently, few records are used to test the performance of extraction tool. Two different similarity thresholds are chosen; the low similar-
ity threshold is taken as 0.60, and the high similarity threshold is taken as 0.70 so as to examine the validity of the tool. After the analysis, it could
be seen from Figure 5 that the valid extracted rate of software risk concepts is majorly 70% and above for both precision and recall metrics.
Looking at the weighted average of these two metrics (F-measure), the result shows that the extracted rate is mostly over 70%.
With this validation process using primary metrics, the extraction tool performance has been demonstrated. Equally important is the fact that
the retrieved result could be future-able risk factors to be added to the previous risk indexes. On the basis of this, dynamic information could
assist in the identification of likeable risk and thereby supplement the SRO.

5.3 | Ontology validation and framework usage

Ontology can only be useful if it can give consistent responses to real-world questions. This section listed out some questions, which are used in
validating the ontology. The tool used for developing and querying the SRO is protégé editor 4.3.0 (Build 304) and FACT++. The protégé ontology
has a modular design, and it provides basic functionality and numerous plug-ins depending on what to do. For this work, protégé plug-in, which
targets Web Ontology Language53 and RDF, is chosen. FACT++ is used to check for the consistencies in the ontologies and for submitting queries
in order to verify their validity. The queries are expressed in DL Query Language. The capability of the SRO is tested to respond to user queries

F I G U R E 5 Response to query from

the risk extraction interface

TABLE 6 Test retrieval results for prototype extraction tool

Relevant extracted risk Extracted risk Relevant risk

S/N Query concepts concepts concepts Precision Recall F-measure
1 Design activity 5 6 7 0.8333 0.7143 0.7692
2 Coding activity 6 7 9 0.8571 0.6667 0.7500
3 Operation and maintenance phase, 3 4 5 0.7500 0.6000 0.6667
how risky is it?
4 Risks that should be avoided 15 19 21 0.7895 0.7143 0.7500
16 of 24 ABIOYE ET AL.

TABLE 7 Demographic of software expert assessors

Participant's no. Field/background Roles Years of experience

1 Insurance Software analyst 5–10
2 Academic Software engineer 11–20
3 IT consultancy Software expert 5–10
4 Finance Software developer 5–10
5 Finance Software analyst 20–30
6 Insurance Software analyst 5–10
7 Datacom System analyst >30
8 Academic System programmer 5–10
9 Academic Chief system analyst 11–20
10 Computing Centre Database administrator 5–10

by posing instances of the competency questions from DL query interface. The graphical user interface is implemented using the Java program-
ming language. The management and the reasoning of the ontology are facilitated using Jena API and FACT++ reasoner, respectively. Figure 5
shows the response to a query from the risk extraction interface.
The framework is made available for the software experts before the commencement of the new software projects. A short tutorial on the
usage of the framework is presented, and questions raised by the experts are answered. They are encouraged to follow the step-by-step proce-
dure of the framework to know the efficiency of the tool. Table 7 shows the demographic of software experts who used this framework.
On the basis of the report from the software experts, more time is spent on requirement analysis and the design classes and subclasses
although it pays off at the end because the software projects are detailed and with the right specifications. Knowing the risks at each phases of
the software development with the right mitigation strategies actually afforded them the opportunity to submerge many of the risks that would
have hindered speedy and accurate completion of the software projects. Retrieving and studying risk factors at each phase of the SDLC have
immensely aided the timely completion of the software projects even at no extra cost.

5.4 | Field trial evaluation

The performance and perception evaluation of this framework are done using the PAF and TAM. TAM, according to Venkatesh and Bala,54 is
widely used as one of the prominent information system theories and gives a better view of the information system user's acceptance.26 Ten soft-
ware experts are given PAF and perception form for the evaluation of this framework. For PAF, six specialized indicators are chosen to measure
the success rate of a software project, which are cost, time, quality, scope, complexity, and functional requirement. PAF used special rating against
these indicators, which are “5—exceptional,” “4—exceeds expectation,” “3—meet expectation,” “2—below expectation,” and “1—need expectation”
according to Tanjila and Grundy.55 Likewise, for perception form, nine constructs are used to describe perceived case of use and perceived useful-
ness. The same scale of grading used for PAF is equally used for perception evaluation.
The performance evaluation score is obtained by dividing the summation of individual scores by number of participants as shown in
Equation 13.

Xn = 10 scores
PE = ð13Þ
n=0 n

Analysis of these results is compared with evaluation concepts scale55 as seen in Table 8. Performance and perception evaluation carried out by
the 10 experts are shown in Tables 9 and 10, respectively.
Evaluation concepts scale judges based on the quality of performance. The labels used are “outstanding,” “good,” “satisfactory,” “marginal,”
and “poor.” Although there are other evaluation scales such as behavioral frequency scale and standard scale, the evaluation concept scale has
been found to be suited for this work. The overall score is given in this table alongside the interpretation.
Performance appraisal form analysis is carried out by finding the average of each indicator for the selected 10 experts, and the final result is
compared with evaluation concept scale to finalize the performance evaluation carried out as seen in Table 9. Perception evaluation analysis is
carried out by finding the average of each indicator for the selected 10 experts, and the final result is compared with evaluation concept scale to
finalize the perception evaluation result as seen in Table 10.
ABIOYE ET AL. 17 of 24

TABLE 8 Evaluation concept scale55

Overall score Interpretation

0–0.99 Poor
1–1.99 Marginal
2–2.99 Satisfactory
3–3.99 Good
4–5 Outstanding

TABLE 9 Summary of software expert evaluation using selected indicators with mean value

Cost Duration Complexity Quality Scope Functional requirement

2 3 4 3 3 3
4 3 3 4 3 3
4 3 2 2 4 3
4 3 4 3 3 3
3 3 2 3 3 4
3 3 3 5 3 4
3 4 2 4 4 4
4 3 3 3 3 3
2 4 3 3 4 3
3 3 3 3 2 4
Mean 3.20 3.20 2.90 3.30 3.20 3.30

TABLE 10 Perception evaluation results

UA RD
4 3
3 3
4 3
3 3
4 3
3 2
2 3
3 3
2 3
5 3
Mean 3.3 2.9
BA UF AC PA SA US CA
3 4 4 4 3 3 3
3 2 3 4 4 4 3
3 4 3 2 3 3 3
3 4 4 5 5 4 4
3 4 4 5 4 4 3
3 4 4 5 4 4 4
3 3 3 4 4 3 3
4 3 3 4 4 3 4
3 4 4 3 4 3 3
3 3 4 4 4 3 5
3.1 3.5 3.6 4.0 3.9 3.4 3.5

6 | R E S U L T A N D DI SC U S S I O N

The problem with the success of a software project is the lack of proactive risk management procedure. This work implemented the proposed
ORM framework for software project through spiral model SDLC on 10 different software projects. Protégé Web Ontology Language editor 4.3.0
(Build 304) is used to build the SRO knowledge repository while ontology consistency is checked with the aid of FACT++ reasoner, which is also
used to submit DL query.
The approach used in this study presented risks at each SDLC phases. The performance and perception evaluation of this study are carried
out using PAF and TAM with an extensive interview with the software experts.
The outcomes of these evaluations are compared with the evaluation concept scale, and the following deductions are made:
For performance evaluation, the resulting outcome is compared with the evaluation concept scale. It is discovered that “cost” has a mean value
of 3.20, which means that it has a rating “good.” This shows that no additional cost is incurred during the execution of the software projects that used
SORM framework, and this could be seen from the assessment of the experts. For “duration” (time) and “scope,” the mean grading is the same as that
18 of 24 ABIOYE ET AL.

of cost, which is 3.20. Looking through the PAF assessment according to Table 9, all the expert scored “duration” 3 except for two experts that scored
4, and this shows that the projects meet up with the scheduled duration and are delivered on time. Meanwhile, only one expert scored 2 for “scope,”
whereas other experts scored 4 and 3; yet the deployed system meets up with the right requirement specification. “Time” and “cost” do have an
impact on the quality of a project, but as it can be seen here, the fact that these projects are delivered on time and at a budgeted cost does not have
a negative impact on the quality of the delivered system. As seen in Table 9, the quality means that the score is 3.30, which are higher than that of
cost and time; this shows that the delivered software projects are of high quality. Finally, the “functional requirement” as an indicator equally scored
the same value as “quality.” This implies that the deployed systems are able to perform all the specified functions.
For perception evaluation, the comparison of the result with the evaluation concept scale as seen in Table 10 shows that all the nine
constructs exceed 3.0 except “risk documentation,” which is 2.90, and this signifies that more risk factors can be added to this framework in the
future. Meanwhile, a tool for updating SRO has been provided during this work. The exceptional rating is given to “portability,” and it signifies that
the framework can be used on any platform. Also, “supportability” is rated 3.9; it implies that the software experts found this framework useful in
their day-to-day activities and their intention of using the framework during their work is given. “Understandability” means the score is 3.3, which
means that the framework is comprehensive enough and can be easily interpreted by all. Also, “usefulness” and “changeability” construct scores
are 3.5; this shows that software experts found that this framework has an essential tool in achieving goals in the course of their duties.
“Accessibility” as a construct has to do with the ease with which software experts are able to use the framework. The mean score allotted to this
construct is 3.6, and it signifies that the framework is easily accessed by the experts. “Buildability” as a construct deals with how easy or direct
the usage of SORM framework helps in meeting the basic requirement needed for the development of successful software systems. Buildability
has a mean score of 3.1, and it is interpreted as “exceed expectation.” On the basis of this score, we conclude that SORM framework is
straightforward in guiding professionals in the development of successful software systems.
In lieu of the aforementioned, the following suggestions are made.

i This framework will help experts to avoid rework, which usually causes delay in the delivery of project, budget extension, and even the poor
quality of software projects.
ii ORM framework for software project will help in converting tacit knowledge to explicit knowledge, which may be used as a reference by
software developers in the future.
iii This framework being ontology based will promote knowledge sharing and reuse in the software development domain.

6.1 | Threat to the validity

Threats to validity are based on four categories discussed in Opdahl and Sindre,56 which are conclusion, internal, construct, and external validity.
These four shall be discussed relative to this work.

6.1.1 | Conclusion validity

Conclusion validity has to do with the capability to infer a rightful conclusion about the connection that exists between treatment and outcome.56
In this research, conclusion validity deals with the connection between risk identification methods and its outcome in terms of a number of risks
identified at various phases of software life cycle or the evaluation responses by the experts after the usage of this framework. The important
factor that needs to be considered is whether the number of software experts that evaluated the framework is appropriate to provide an
acceptable conclusion deduced from this work.
The outcome of the performance evaluation carried out is of a notable advantage for the risk management framework in cost reduction, taken
an effect size of 3.20 (mean value for cost from Table 8). α is used to denote Class I error, whereas β denotes Class II error; these relationships
hold for Equations 14 to 16 as calculated in Appendix A.

AB
N= ½53 ð14Þ
ðE=SÞ2

 
1 1
A= + ð15Þ
q1 q0

 2
B = Zα + Zβ ð16Þ
ABIOYE ET AL. 19 of 24

Yields N = 3.13, which means that a minimum of three experts would be needed to perform the evaluation. Because 10 experts are involved in
this regard, we have enough number of assessors for our conclusion.

6.1.2 | Internal validity

Internal validity involves the connection that exists between treatment and outcome. It checks if the experimental outcome has to do with speci-
fied treatment or with unspecified factors. Internal validity talks about the degree at which a study affirms the credibility between treatment and
outcome. The treatment in this study is the presentation of this framework to the developers to use in new software projects about commerce,
while the control group is the implementation of projects through the old process. Some of the threats to internal validity are as follows:
Expert selection bias is possible if one expert is smarter than the other. Our well organized and designed experimental procedure prevented
this threat. A workshop is organized for software experts purposely for risk identification, and questionnaire is administered to them for assess-
ment under the same time limits and other conditions. The risks are jointly identified and categorized by experts. Also, some risks are removed by
experts.
The research work might have caused invalid conclusion if experts had previous knowledge about the usage of this framework rather, the
tutorial on the operation of the framework presented by the researcher to the experts is the knowledge available to them. The outcome of the
performance evaluation shows that the framework is generally acceptable by the experts.
Bias in tutorials would occur if the framework is not presented in a manner easily operable. This framework is portable, and it is presented in a
format that is understandable by the experts. The researcher took time to demonstrate the operation of the framework to the expert in order to
enhance the easy operation.
Training time is short, and it took 15 min to familiarize the expert with the protégé editor. The fact that the participants are all software
experts made this possible. Task time took 18 days. Because the framework serves as a guide to building a successful software project, this means
that the task time and project duration are the same. At every phase of project development, the framework needs to be considered to check for
risks that must be guided against.
Researcher expectation would occur if the experts had perceived that the risk management framework is developed by the student and that
positive assessment of the framework will help in meeting up with a time limit. This threat is too complex to manage, but it is not mentioned nei-
ther verbally nor inaction that the research must be submitted within a time frame.56

6.1.3 | Construct validity

Construct validity asks the question of whether it is acceptable to draw from the observation made in the experiment to the empirical constructs,
which the researcher is observing. In our performance evaluation, effectiveness and easiness of use of this framework are observed, and this is
rated by the scores allotted by experts to software success selective indicators. Hence, two threats are considered.
Whether the use of techniques in this research is an ideal portray of empirical methods examined (qualitative risk analysis using AHP). This
work overcame this threat by using multiple techniques in identifying software risks. Workshop (brainstorming session), use of a questionnaire,
organizational documents, and review of literature are used to gather the risks. The use of multiple techniques ensures that no risk is left behind.
The second threats is whether the evaluation scale used (evaluation concept scale) is a representative of the metrics used for measuring
effectiveness in software risk management framework? There are other evaluation scales such as behavioral frequency scale and standard scale
that can be used to score the output of this work. Because the effectiveness of this framework is the measure of success of this research, the two
above-mentioned metrics are unsuitable for the measurement.

6.1.4 | External validity

External validity involves generalization—is there a possibility of generalizing from the experimental setting to real-life practice? The problem here
is whether the result of this work can be generalized to what is obtainable in software industry.57
Some of the threats to consider are as follows:
The use of nonexperts instead of professionals: According to Tanjila and Grundy,55 the factor that determines subject performance in research is
not nonexperts versus professionals rather and their level of competence. However, this threat does not affect this work because software
experts having a minimum of 5-year experience are engaged in this research.
Lack of motivation: In an experiment, when students know that they stand not to gain or lose anything, the interest might be lost. In this work,
participants are highly motivated because they saw that the framework has a potent tool in their day-to-day activities in software development.
20 of 24 ABIOYE ET AL.

7 | CO NC LUSIO N

This research aimed to prove how the development of SORM framework could improve the on-time delivery of quality software project using
case studies demonstration and also to equally reduce the impacts of risks on the software project being undertaken. The contacted software
experts are able to incorporate knowledge reuse in risk management through the development of SORM framework and software risk knowledge
based.
The proposed SRO is developed using literature, organizational documents, and expert interview. Case studies are used to validate this study,
and this affirmed that this conceptual structure could assist the software developers in knowledge reuse during risk management processes in the
course of software development, which can reduce risks pose to software projects. Moreover, the use of explicit knowledge through the acquisi-
tion of tacit knowledge is applicable to software development processes and thereby leading to high performance in the SDLC. The conversion of
tactic knowledge to explicit knowledge led to knowledge reuse.
In conclusion, the ORM framework for software project could be applied to other software development because the model proposed here is
with reference to general software risk management standard procedure. Likewise, this framework is validated through case studies with exten-
sive experts' interview using PAF, and the perception evaluation is carried out using TAM. This framework is therefore recommended to software
developers so as to meet up with all the prerequisite in the delivery of high-performance software projects.

ORCID
Oluwasefunmi Tale Arogundade https://orcid.org/0000-0001-9338-491X
Sanjay Misra https://orcid.org/0000-0002-3556-9331

RE FE R ENC E S
1. Cotterell M, Hughes B. Software Project Management. London: International Thomson Computer Press; 1995.
2. Charette RN. Why software fails? {software failure}. IEEE Spectrum. 2005;42(9):42-49.
3. Ricardo A, Ana C, Cruz N, Paula G, Gleidson B, Fabiano B. ODE: ontology-based software development environment. CACIC, 2003; 1124-1135.
4. Chapman C. Project risk analysis and management—PRAM the generic process. Int J Proj Manag. 1997;15(5):273-281.
5. Boban M, Pozgaj Z, Seric H. Strategic for successful software development risk management. Management. 2003;8(2):77-91.
6. Brandon D. Project Management for Modern Information System. USA: Idea Christian Brothers University, USA: Group Inc (IGI); 2006.
7. Surbhi A, Vinay C. Decision support system for software risk analysis during software development. Inter J Curr Trends Sci Tech Latest Trends. 2012;2
(1):29-35.
8. Ryan S, O'Connor RV. Development of a team measure for tacit knowledge in software development teams. J Syst Softw. 2009;82:229-240.
9. Ryan S, O'Connor R. Acquiring and sharing tacit knowledge in software development teams: an empirical study. Inform Software Tech. 2013;55(9):
1614-1624.
10. Islam MZ, Ahmad ZA, Mahtab H. The mediating effects of socialization on organizational contexts and knowledge sharing. J Knowl Glob. 2010;3(1):
31-48.
11. David H. Using a risk breakdown structure in project management. J Facil Manag. 2003;2(1):85-97.
12. Davis AM., Dieste O, Hickey AM, Juristo N, Moreno AM. Effectiveness of requirements elicitation techniques: empirical results derived from a
systematic review. 14th IEEE international requirements engineering conference (RE'06); 2006.
13. Dikmen I, Birgonul MT, Anac C, Tah JHM, Aouad G. Learning from risks: a tool for post-project risk assessment. Autom Construct. 2008;18(1):42-50.
14. Tah JHM, Carr V. Knowledge-based approach to construction project risk management. J Comput Civ Eng. 2001;15(3):170-177.
15. Krishna BC, Subrahmanyam K, Anjaneyulu SSN, Kim T-H. A novel Dr. KSM approach for information security and risk management in healthcare sys-
tems. J BioSci Biotechnol. 2015;7(4):11-16.
16. Gruber TR. A translation approach to portable ontology specification. Knowl Acquis, ScienceDirect, Elsevier. 1993;5(2):199-200.
17. Ricardo BY, Berthier RN. Modern Information Retrieval. New York: Addison Wesley; 1999 1999.
18. Staab S, Studer R, Schnurr HP, Sure-Vetter Y. Knowledge processes and ontologies. IEEE Intell Syst. 2001;16(1):26-34.
19. Pandit A, Zhu Y. An ontology-based approach to support decision making for the design of ETO (engineer-to-order) products. Autom Construct. 2007;
16(6):759-770.
20. Boyce S, Pahl C. Developing domain ontologies for course content, in educational technology & society. Int Forum Educ Techno Soc. 2007;10(3):
275-288.
21. Afolabi I, Daramola O, Adio T. Developing domain ontology for Nigerian history. Aust J Basic Appl Sci. 2014;8(4):1-3.
22. Ashbaugh DA. Security Software Development: Assessing and Managing Security Risk. New York, USA: Auerbach Publications; 2008.
23. Stellman A, Greene J. Applied Software Project Management. USA: O'Reilly Media; 2005.
24. Thayer R, Yourdon E. Software Engineering Project Management. 1st ed. New York: Pearson; 2001.
25. Natalya FN, McGuinness DL. Ontology Development 101: A Guide to Creating Your First Ontology. Stanford, CA: Stanford University; 2000.
26. Abdullahi SN, Indulska M, Sadiq S. Compliance management ontology—a shared conceptualization for research and practice in compliance manage-
ment. Inform Syst Front. 2016;18(5):1-4.
27. Rene Robin CR, Uma GV.Design and development of an ontology for risk management in software project management. International Symposium on
Computing, Communication, and Control (ISCCC 2009) Proc.of CSIT, 2011; 1, 253-254.
28. Julio M, Critine G, Hermano M. Defining indicators for risk assessment in software development projects. CLEI Electr J. 2013;16(1):10-14.
29. Hijazi H, Alqrainy SA, Muaidi H, Khdour T. Risk factors in software development phases. Eur Sci J. 2014;10(3):1857-7431.
30. De Amescua A, García J, Velasco M, et al. A software project management framework. Inform Syst Manage J. 2004;21(2):78-85.
ABIOYE ET AL. 21 of 24

31. Sheriff MA, Georgiadou E. Relating software quality models and process methods to user value. Int J Hum Capital Inform Technol Prof (IJHCITP). 2013;4
(2):27-42.
32. Baumeister A, Llg M. Activity driven budgeting of software projects. Int J Hum Capital Inform Technol Prof (IJHCITP). 2010;1(4):14-30.
33. Bastos EC, Barcellos MP, Falbo RA. Using semantic documentation to support software project management. J Data Semant. 2018;7:107-132.
34. Dey PP, Amin M, Sinha BR, Jawad S, Al Any L, Badkoobehi H. Current trends in software project management. Adv Soc Sci Educ Hum Res. 2018;120:
22-26.
35. Tavares BG, Sanches da Silva CE, Diniz de Souza A. Practices to improve risk management in agile projects. Int J Software Eng Knowl Eng. 2018;29(2):
1-19.
36. Tavares BG, Sanches da Silva CE, Diniz de Souza A. Risk management analysis in Scrum software projects. Int Trans Oper Res. 2017;26:1884-1905.
37. Sariglannidis L, Chatzoglou PD. Software development project risk management: a new conceptual framework. J Software Eng Appl. 2011;4:293-305.
38. Saunders MNK, Lewis P, Thornhill A. Understanding research philosophy and approaches to theory development. In: Research Methods for Business
Students. 8th ed. Harlow, England: Pearson; 2019:128-170.
39. Amaral JN. About Computing Science Research Methodology. Alberta, Canada: Edmonton; 2011.
40. Holz HJ, Applin A, Haberman R, Joyce D, Purchase H, Read C.Research methods in computing. In: Working Group Reports on ITiCSE on Innovation
and Technology Computer Science Education, 2006; 38(4), 96.
41. Tserng HP, Yin SYL, Dzeng RJ, Wou B, Tsai MD, Chen WY. A study of ontology-based risk management framework of construction projects through
project lifecycle. Autom Construct, Elsevier Journal. 2009;18:994-1008.
42. El-Diraby TE, Zhang J. A semantic framework to support corporate memory management in building construction. Autom Construct. 2006;15(4):
504-521.
43. El-Diraby TE. Web-services environment for collaborative management of product life-cycle costs. J Construct Eng Manag. 2006;132(3):300-313.
44. Department of Energy Quality Managers, & Software Quality Assurance Subcommittee. Reference document SQAS21.01.00–1999 (accessed 26.01.16).
45. Saaty TL. The Analytic Hierarchy Process. New York, USA: McGraw-Hill; 2000.
46. Forman EH, Saaty TL. Expert Choice. Pittsburgh, PA: RWS Publications; 1983.
47. Saaty TL. The Analytic Hierarchy Process, Planning, Piority Setting, Resource Allocation. New York: McGraw-Hill; 1980.
48. Lin KY, Soibelman L. Promoting transactions for A/E/C product information. Autom Construct. 2006;15(6):746-757.
49. Rezgui Y. Ontology-centered knowledge management using information retrieval techniques. J Comput Civ Eng. 2006;20(4):261-270.
50. Salton G. Automatic Text Processing: The Transformation, Analysis, and Retrieval of Information by Compute. New York: Addison Wesley; 1989.
51. Salton G, Buckley C. Term-weighting approaches in automatic retrieval. Inf Process Manag. 1988;24(5):513-523.
52. Hripcsak G, Rothschild AS. Agreement, the F-measure, and reliability in information retrieval. J Am Med Inform Assoc. 2005;12(3):296-298.
53. Web Ontology Language (OWL). http://www.w3.org/2015/sw/WebOnt/
54. Venkatesh V, Bala H. Technology acceptance model 3 and a research agenda on interventions. Decis Sci, A Journal of Decision Science Institute.
2008;39(2):273-315.
55. Tanjila K, Grundy J. Performance appraisal of software testers. Inform Software Tech. 2013;56(5):495-505.
56. Opdahl AL, Sindre G. Experimental comparison of attack trees and misuse cases for security threat identification. Inform Software Tech. 51:916-932.
57. Stalhane T.A comparison of two approaches to safety analysis based on use cases. International conference on conceptual modelling, Lecture Notes In
Computer Science (LNCS), 4801, 423-437.

How to cite this article: Abioye TE, Arogundade OT, Misra S, Akinwale AT, Adeniran OJ. Toward ontology-based risk management
framework for software projects: An empirical study. J Softw Evol Proc. 2020;e2269. https://doi.org/10.1002/smr.2269

APP E NDIX A. : THREAT TO VALIDITY

AB
N= ½52 ðA1Þ
ðE=SÞ2

In threats to validity Values 0.05 and 0.2 for α and β are set probability levels used in conclusion validity to confirm whether the sample size is
enormous to justify the conclusions.
The standard normal deviate for α = Zα = 1.960.
The standard normal deviate for β = Zβ = 0.842.

 
1 1
A= + A = 4:080 ðA2Þ
q1 q0

 2
B = Zα + Zβ B = 7:849 ðA3Þ
22 of 24 ABIOYE ET AL.

The effect size for this work is 3.20,

AB
N= = 3:13:
ðE=SÞ2

APP E NDIX B .: SOFTWARE RISK SUBCLASSES WITH THEIR RISK FACTORS

S/N Risk factors in

Feasibility study activity
1. Under estimation of project time, scope, and other resources
2. Unrealistic schedule
3. Unrealistic budget
4. Unclear project scope
5. Insufficient resources
Requirement elicitation activity
6. Unclear requirement
7. Inaccurate requirement
8. Conflicting user requirement
9. Ignoring the nonfunctional requirement
10. Unclear description of the real environment
11. Gold plating
Requirement analysis activity
12. Infeasible requirement
13. Inconsistent requirement
14. Nontraceable requirement
Requirement validation activity
15. Misunderstood domain-specific terminology
16. Misexpressing user requirements in natural language
Requirement documentation activity
17. Inconsistent requirement data and requirement document
18. Nonmodifiable requirement document
Requirement document examination
19. Requirement document unclear for developers
Chosen architectural design method
20. Improper architectural design method choice
Chosen programming language
21. Improper choice of programming language
Physical model construction activity
22. Complicated design
23. Large complex system
24. Unavailable expertise for reusability
Design verification activity
25. Difficulties in verifying design to requirement
26. Many feasible solution
27. Wrong design
ABIOYE ET AL. 23 of 24

S/N Risk factors in

Design specification activity
28. Difficulties in allocating functions to components
29. Extensive specification
30. Omitting data processing functions
Design activity documentation
31. Incomplete design document
32. Large design document
33. Unclear design document
34. Inconsistent design document
Coding activity
35. Nonreadable design document
36. Developing the wrong user interface
37. Programming language does not support architectural design
38. Modules are developed by different programmers
39. Complex, ambiguous, inconsistent code
40. Large amount of repetitive code
41. Inexperienced programmers
42. Too many syntax errors
43. Technology change
Unit testing activity
44. High fault rate in newly design components
45. Code is not understandable by reviewers
46. Lack of complete automated tools
47. Not all faults are discovered in unit testing
48. Testing is monotonous, boring, and repetitive
49. Coding drivers and stubs
50. Poor regression testing
Integration activity
51. Difficulties in ordering components' integration
52. Integrate wrong version of components
53. Omissions or oversights
Integration testing activity
54. A lot of bugs emerged during the integration
55. Data loss across an interface
56. Difficulties in localizing errors
57. Difficulties in repairing errors
System testing activity
58. Unqualified testing team
59. Limited testing resources
60. Inability to test in the operational environment
61. Testers rely on process myths
62. Testing cannot cope with requirement change
63. Nontestable system
Installation activity
64. Problems in installation
65. The effect on the environment
66. Environmental change

(Continues)
24 of 24 ABIOYE ET AL.

S/N Risk factors in

Operation activity
67. Emergence of new requirement
68. Difficulties in using the system
Acceptance testing activity
69. User resistance to change
70. Missing capabilities
71. Too many software faults
72. Testers under performance
73. Suspension and resumption problems
74. Insufficient data handling
Maintenance activity
75. The software engineer cannot reproduce the problem
76. Problems in maintainability
77. Budget contention