Professional Documents
Culture Documents
fully edited. Content may change prior to final publication. Citation information: DOI
10.1109/ACCESS.2020.3007874, IEEE Access
ABSTRACT Most organizations use IT projects as vehicles to implement the IT strategy which contributes to achieving
organizational strategic objectives and goals. Agile project management has been a potential solution to deliver successful IT
projects instead of the traditional waterfall approach. This potential has resulted in organizations adopting agile project
management to deliver IT projects on time and realise benefits. However, auditors especially those whose experience lies with
more traditional and system development life cycle controls are struggling with how to audit agile projects. The problem
addressed in this article is lack of an audit framework for auditing agile projects to ensure IT project success. The purpose of
this article is to present the proposed conceptual framework for auditing agile projects that are implemented using Scrum
methodology. The framework introduces audit processes in each process of the Scrum methodology. The quantitative research
method used online survey questionnaire to validate the conceptual framework amongst IT professionals in South Africa. The
data were analyzed using SPSS 26.0. The research revealed that there was a significant correlation between the identified audit
processes and agile project success. This research emphasizes on the need to take into account auditing agile project from its
initiation to closure. This article contributes to the body of knowledge with regard to project auditing. Auditors can use the
proposed framework to audit agile projects which are implemented using Scrum methodology to ensure successful completion
of IT projects.
INDEX TERMS Agile project, Agile methods, Project audit, Project success, Software quality assurance,
Information Technology.
I. INTRODUCTION controls, are struggling with how to audit agile projects [49].
Information technology (IT) projects are vehicles that Project auditing throughout the project life cycle helps to
organizations use to implement the IT strategy, which identify project risks earlier, trigger timely corrective actions
contributes to achieving organizational strategic objectives and improve project performance, which increases the chances
and goals. In monetary value, from 2014 to 2020, US$24 of delivering a successful project [32]. There are limited
751 billion was spent on IT projects worldwide; of this research studies on how to audit agile projects, meaning that
US$17 820 billion was wasted on failed IT projects [30]. In auditing agile projects is still a new area in research. Literature
the period 2015 to 2020, South Africa spent R417.2 billion reveals that there is a lack of an audit framework for auditing
on IT projects, R221.1 billion of which was wasted on failed agile projects [14], [41], [49].
IT projects [30]. These statistics indicate that IT projects are In this study a conceptual framework for auditing agile
still failing and not delivering the strategic organizational projects was developed and validated among 77 IT
goals and objectives. Agile project management has been a professionals in South Africa. The main objective of this
potential solution to deliver successful IT projects instead of article is to present the proposed conceptual framework for
the traditional waterfall approach [4], [7], [30]. 71% of auditing agile projects. The framework has audit processes
organizations worldwide are adopting agile project which can be used by auditors to audit agile projects to ensure
management to improve the success rates of IT projects [40]. the successful delivery of these projects.
The adoption of agile project management has resulted in The article is organized as follows: Section 2 provides a
organizations increasing their productivity, improving literature review on the agile concept, agile methods, software
efficiency and delivering projects on time [50], [51]. quality assurance and agile project auditing. Section 3 presents
However, auditors, especially those whose experience lies a discussion of a conceptual framework for auditing agile
with more traditional and system development life cycle projects. The research methodology is dealt with in Section 4.
This work is licensed under a Creative Commons Attribution 4.0 License. For more information, see https://creativecommons.org/licenses/by/4.0/.
This article has been accepted for publication in a future issue of this journal, but has not been fully edited. Content may change prior to final publication. Citation information: DOI
10.1109/ACCESS.2020.3007874, IEEE Access
Results are discussed in Section 5 and the findings are management as “an iterative lifecycle designed based on
presented in Section 6. Finally, conclusions and short delivery cycles to deal with uncertainty in scope and
recommendations for future research are made in Section 7. rapid change in requirements”. Furthermore, they [28] argue
that agile project management is value driven rather than
II. LITERATURE REVIEW plan driven. Agile project management is guided by the
Agile Manifesto, which consists of four values and twelve
A. DEVELOPMENT, VALIDATION AND VERIFICATION principles to improve software development processes [5].
The term ‘development’ has various meanings to different The four values are: (i) Individuals and interactions over
people and can be explained in different contexts. processes and tools; (ii) Working software over
Development is defined as “transformation from current comprehensive documentation; (iii) Customer collaboration
states to desired one” [39]. [54] defined validation as “a over contract negotiation; and (iv) Responding to change
global term encompassing both verification and evaluation”. over following a plan. The Agile Manifesto is the baseline
Verification is about examining the system against its for researchers and practitioners to introduce a variety of
specifications [54]. Validation is concern with examining agile methods that all comply with the spirit of agile.
either a particular design meets its intended purpose [54]. In
this study, validation means verifying and evaluating the D. AGILE METHODS
conceptual framework to meet a research objective. The philosophy of the agile method is designed around a set
Development means a process of creating something new of core concepts, including agile practices, measurement
that adds value and solve a certain problem. metrics and decision making [24]. The core characteristics of
agile are self-organizing teams, fast execution, value driven
B. CONCEPTUALIZATION OF INFORMATION and business oriented. Agile practice is basically promoting
TECHNOLOGY PROJECTS the concept of self-organizing (self-directed) teams that are
Information Technology projects are organizational empowered to organize their work on their own [8], [22].
investments that require money, time, people, technology, Each of the agile methods shares the values described in the
facilities, equipment, and processes [42]. Information Agile Manifesto.
technology (IT) projects are implemented in organizations to Agile encompasses various methods, including Adaptive
provide benefits such as to create strategic business value in Software Development (ASD), Agile Unified Process (AUP),
return for IT investment, to achieve greater efficiency and to Crystal Methods, Dynamic System Development
improve service delivery to customers [32]. Methodology (DSDM), eXtreme Programming (XP), Feature
However, there are some of the challenges facing IT Driven Development (FDD), Lean Software Development
projects include unclear requirements, requirements gap in the (LSD), Kanban, Scrum and Scrum ban [7], [8], [22]. Cao et al.
final product, project scope creep, poor communication [8] explain that all the agile methods share the same practice
between project team members, project infrastructure delivery and have a common approach, but each method has its unique
delay, undefined project dependencies, poor stakeholders strategy and characteristics.
management, inadequate support from the top management,
lack of project management methodology, lack of project audit TABLE 1
AGILE METHODS MOST USED IN ORGANIZATIONS
processes throughout the project life cycle, project go-live
PMI Pulse of the VersionOne (2018)
failure, lack of post-project support and maintenance plan, Agile method
Profession (2017)
lack of project management best practices, poor risk Lean 67% 2%
assessment and management, cost overruns, IT project Scrum 55% 54%
XP 42% 1%
managers do not assign Key Performance Indicators (KPIs) to
project benefits [32], [40].
IT projects differ from other projects because they involve The survey by PMI Pulse of the Profession [40], [51]
using hardware, software, and networks to create a final reveals that Scrum is the most widely used agile method, as
product, service or result [42]. shown in table 1. Therefore, this study focused on Scrum to
develop a framework for auditing agile projects.
C. AGILE CONCEPT
The agile concept was introduced by a group of developers E. SCRUM OVERVIEW
in 2001. Agile project management is an approach that is Scrum is a project management framework that is applicable
used to design and deliver software that has the greatest value to any type of project in any industry including IT [9]. In
to the customer [9]. There are various definitions of agile Scrum, projects are managed with small teams in a series of
project management. According to Wideman [52], agile iterations called sprints. The primary focus is on delivering
project management is “a conceptual project management products that satisfy customer requirements in small iterative
framework for undertaking software development projects in shippable increments. There are three core roles in scrum that
which the emphasis is moved from planning to execution”. ensure the project objectives are met. The core roles are the
Lemétayer and Sheffield [28] define agile project product owner, scrum master and scrum team. On scrum, the
This work is licensed under a Creative Commons Attribution 4.0 License. For more information, see https://creativecommons.org/licenses/by/4.0/.
This article has been accepted for publication in a future issue of this journal, but has not been fully edited. Content may change prior to final publication. Citation information: DOI
10.1109/ACCESS.2020.3007874, IEEE Access
roles of the project manager are distributed which entails that the backbone of the project. The SQA practices are different
project management functions are shared between scrum in each agile method [48], [29].
team, scrum master and product owner. The emphasis in In scrum methodology, the most commonly used SQA
scrum is on self-organization and self-motivation where the practices are code reviews, test driven development (TDD)
team assumes greater responsibility in making a project and close collaboration [17], [31], [43], [49]. Code reviews
successful. According to [9] and [21] Scrum framework help to improve quality and knowledge sharing among scrum
comprises of the following processes: project developers. Test driven development Close
• Product Vision: The statement describing the vision for collaboration is the process where customers and the scrum
the product which is aligned with the organization’s development team meet together on a daily basis to gain
business needs. understanding on what is being developed by the developers.
• Release Planning: The purpose of a release planning is This practice of SQA ensures that requirements are correctly
to establish the schedule for the product releases. The understood by stakeholders and that the final product meets
meeting is conducted to determine the length of the the customer business needs.
iterations and sprint meetings. This includes release plans
for multiple sprints. G. AGILE PROJECT SUCCESS
• Product Backlog: The list of the product requirements Project success has different perceptions since its evolution
(user stories) are included in the product backlog. The to present [32]. Project success includes project success
user stories are selected for development based on the criteria which are the measures for determining whether a
priority levels assigned by the customer. Items listed in project is successful or not. In the traditional view the
the product backlog support the delivery of the product measures of project success are scope, time, cost, quality,
vision. resources, customer satisfaction, stakeholder satisfaction,
• Sprint Backlog: Sprint backlog includes the list of realised benefits [32]. These success measures are referred in
requirements selected from the product backlog that are project audits or post-project reviews.
included in a particular sprint. Sprint backlog has sprint According to [51] the measures of agile project success are
(iteration) which comprises of sprint planning meeting, customer satisfaction, business value delivered, velocity,
daily scrum meeting, sprint review meeting and sprint budget against actual cost, planned against actual stories per
retrospectives. The duration of sprint iteration is two iteration, planned against actual stories release dates, iteration
weeks to one month which are used to build a potentially burndown, defects in to production, burn-up chart, defects
shippable product increment to a customer. overtime, cycle time, release burndown, Work-in-process
• Shippable Product (Product Deliverable): At the end (WIP), defects resolution, customer retention, estimation
of each sprint, the product deliverable that met accuracy, earned value, test pass/fail overtime, revenue/sales
acceptance criteria is shipped to the customer. impact, cumulative flowchart, product utilization, individual
hours per iteration and scope change in a release.
F. SOFTWARE QUALITY ASSURANCE ON AGILE
In this study the agile project success measures were
PROJECTS
designed using the VersionOne [51] agile project success
Software quality assurance (SQA) is defined as “a planned
criteria because they have been used in measuring agile project
and systematic approach to the evaluation of the quality of
and adherence to software product standards, processes, and success.
procedures” [17]. SQA is vital during all the phases of the
software development process to achieve the required and H. AGILE PROJECT AUDITING
higher level of quality in terms of product and process. The Most organizations are adopting agile project management
software product quality attributes, such as functional to improve the success rates of IT projects. This adoption of
suitability, performance, efficiency, compatibility, usability, agile requires organization changes in culture, mindset,
reliability, security, maintainability and portability [37], are behaviors and structure [19]. One of these organizational
analyzed to achieve SQA [43]. SQA is the key to successful changes is how agile projects are audited. Auditing agile
IT projects. projects is necessary to increase the chances of delivering
Agile methods rely on agile values and guiding principles successful IT projects in the organization [36], [41]. Project
to deliver high-quality software [5]. This claim has increased auditing is currently undergoing a fundamental change due
the usage of agile in the software and IT industries [4], [7], to emerging technologies associated with the Fourth
Industrial Revolution. This implies that the methods for
[31]. Agile methods for software development have been
auditing agile projects require a different audit process [3].
adopted by many organizations to improve the quality of
Auditors should be involved in the entire software
software products. The quality in agile projects influences
development process.
project success [1], [2]. SQA in agile involves quality in the Various studies have been done on agile project auditing.
software development process and quality of the product [47]. [35] proposed an auditing model for ISO 9001 traceability
The quality assurance activities in software development are requirements that is applicable in the agile (XP) environment.
The model is based on evaluation theory and includes the use
This work is licensed under a Creative Commons Attribution 4.0 License. For more information, see https://creativecommons.org/licenses/by/4.0/.
This article has been accepted for publication in a future issue of this journal, but has not been fully edited. Content may change prior to final publication. Citation information: DOI
10.1109/ACCESS.2020.3007874, IEEE Access
This work is licensed under a Creative Commons Attribution 4.0 License. For more information, see https://creativecommons.org/licenses/by/4.0/.
This article has been accepted for publication in a future issue of this journal, but has not been fully edited. Content may change prior to final publication. Citation information: DOI
10.1109/ACCESS.2020.3007874, IEEE Access
project audit’, ‘agile project success’, ‘software quality Agile values and
[19], [20]
2
guiding principles
assurance in agile project’, ‘project governance’, “agile
Software quality [17], [26], [31], [37], 9
software development’. Major databases were used to search assurance in agile [43], [44], [46], [48],
for related articles, such as those provided by major project [53]
publishers: IT project 4
[23], [30], [32], [42]
management
• Scopus (www.scopus.com), Total 47
• Elsevier (www.sciencedirect.com),
• Emerald (www.emeraldinsight.com), Phase 3: Identifying and naming concepts
• Springer (www.springerlink.com), This phase involved reading and reread the selected data and
• Wiley (www.wiley.com). discover concepts from literature. The seven concepts were
The keyword ‘auditing agile project” was searched from identified after analysing the dimensions found in phase 2. The
websites such as PWC (www.pwc.com), Deloitte identified concepts were:
(www.deloitte.com). The books on agile project management Concept 1: Agile values and guiding principles
were also selected. A total of 47 data sources focused on the Concept 2: Agile project management
research keywords were identified. Concept 3: Agile methods
Concept 4: Project governance
Phase 2: Extensive reading and categorizing of the selected Concept 5: Agile project audit
data Concept 6: Software quality assurance in agile project
This phase aimed at reading each article and categorizing the Concept 7: Agile project success
selected data. Descriptive dimensions were used to classify the
papers. The content of the paper was further accessed by Phase 4: Deconstructing and categorizing concepts
means of descriptive analysis: (i) how is the distribution of The aim of this phase was to deconstruct each concept to:
publications of articles related to auditing agile project across (i) identify its main attributes, characteristics, assumptions and
time period, (ii) what research methodological are applied? role; (ii) organize and categorize the concepts according to
(iii) which dimensions of auditing agile project are addressed? their features, and ontological, epistemological and
Descriptive analysis: The distribution of publications across methodological. The results of deconstructing and
the time period was assessed. The body of literature identified categorizing concepts are as shown in table 3.
47 papers which were published in the period (1998-2019).
The assessment of specific papers on auditing agile project TABLE 3
found that the first published papers were in year 2013 [25], RESULTS OF DECONSTRUCTING AND CATEGORIZING
CONCEPTS
[35]. Other publications found in year 2018 [34] and 2019 [3], Name of Categorizing References
[14], [20], [36], [41]. A total of eight papers were published Description of concept
concepts concept role
on auditing agile projects from 2013 to 2019. None of these Agile project management Ontological [19], [20]
is guided by four agile concept
eight papers have come with the framework for auditing agile values and twelve
projects. Research methodologies applied were differentiated Agile values principles to improve
and guiding software development
as: (i) surveys, (ii) case studies, (iii) theoretical and conceptual principles processes
papers, (iv) literature reviews, and (v) modelling papers. In the Agile project It is a conceptual project Methodological [9], [22],
management management framework concept [24], [51],
47 identified papers, survey account for 26 of the papers, while for undertaking software [52]
3 modelling papers, nine literature reviews, three theoretical development projects.
Agile These are approaches to Methodological [4], [5], [10],
and conceptual papers, and case studies account for 6 of the methods project management that concept [12], [16],
papers. The dimensions addressed on auditing agile project are promote continuous [21], [27],
iteration of development [29], [50]
as shown in the table 2. and testing throughout the
software development
TABLE 2 lifecycle. For example,
Scrum methodology.
CATEGORIES OF THE SELECTED DATA
Project It oversees Methodological [1], [6], [18],
governance implementation of agile concept [33]
Number of project activities and
Dimension Reference papers provides a framework for
Project governance [1], [6], [18], [33] 4 decision making
Agile project success [2], [7], [8], [28],[32], 7 throughout the project.
[47], [49] Agile project Audit agile project to Methodological [3], [14],
audit ensure compliance, concept [16],[25],
Agile project audit [3], [14], [25], [34], [35], 7 identify risks, internal [34], [35],
[36], [41] controls and success [36], [40]
Agile methods and [4], [5], [10], [12], [16], 9 completion.
software development [21], [27], [29], [50] Software It is a planned and Methodological [17], [26],
Agile project 5 quality systematic approach to concept [31], [37],
[9], [22], [24], [51],[52] assurance in the evaluation of the [43], [44],
management
agile project quality of and adherence [46], [48],
to software product [53]
This work is licensed under a Creative Commons Attribution 4.0 License. For more information, see https://creativecommons.org/licenses/by/4.0/.
This article has been accepted for publication in a future issue of this journal, but has not been fully edited. Content may change prior to final publication. Citation information: DOI
10.1109/ACCESS.2020.3007874, IEEE Access
standards, processes, and reviews which were incorporated into the final survey
procedure.
Agile project Agile project success Methodological [2], [7], [8], questionnaire before administering it to a larger sample size.
success criteria have been used in concept [28],[32],
measuring agile project [47], [49]
success. The conceptual framework for auditing agile projects is
discussed in the next section.
Phase 5: Integrating concepts
This phase aimed at integrating and grouping together III. CONCEPTUAL FRAMEWORK FOR AUDITING AGILE
concepts that have similarities to a new concept. This phase PROJECTS
reduced the number of concepts and allowed to have This section presents a conceptual framework for auditing
reasonable number of concepts. The integrated concepts are as agile projects (Fig. 3), which was developed and validated
among IT professionals in South Africa.
shown in Fig. 2.
Phase 6: Synthesis, resynthesis and making it all make The conceptual framework for auditing agile projects can
sense assist project auditors in auditing agile projects. The
This phase aimed at synthesizing the identified concepts and framework may improve the success rates of IT projects in the
drawing conclusion. Agile project management has to be organization.
adhered to agile values and guiding principles [19], [20].
Project governance monitors agile project progress throughout
Agile encompasses various methodologies, but most of the
its implementation and provides a framework for decision
organizations use scrum methodology [50], [51]. This
making. Agile project audit provides assurance and enables
research reviews scrum methodology to come up with the
the sponsor to have confidence that the project governance is
working, the project is being managed as intended and be audit processes. Scrum has five processes such as product
successfully completed as well as examine compliance, risk vision, product backlog, release planning, spring backlog and
management and internal controls. There are various agile potentially shippable product to customer. Each process has
methods such as scrum, eXtreme programming, Kanban. number of activities which are implemented in iteration
Auditing agile project has to focus in a particular agile method manner. These processes were also used to identify
because each agile method has different processes. This study components of the framework and audit process items. The
focused on scrum methodology which is the commonly used conceptual framework in Fig. 3 comprises with the following
method in agile project management [50], [51]. components:
• Product vision audit: Audit processes which can be used
Phase 7: Validating the conceptual framework to audit agile projects are examine the created product
Validating the conceptual framework used subject matter vision [9], establish a steering committee, identify
experts. Agile project management experts from the stakeholders, develop epics (user stories) and user story
University of Johannesburg’s Department of Applied acceptance criteria and train the Scrum team and
Information Systems reviewed the framework and provided stakeholders on agile values and guiding principles [6],
feedback. The comments were used to improve the [14], [18], [20], [25], [33].
framework. The experts also reviewed the questionnaire items • Product backlog audit: The audit processes examine
to ensure its content validity. The experts provided useful developed product requirements, minutes of the meeting
where product requirements were reviewed and approved
This work is licensed under a Creative Commons Attribution 4.0 License. For more information, see https://creativecommons.org/licenses/by/4.0/.
This article has been accepted for publication in a future issue of this journal, but has not been fully edited. Content may change prior to final publication. Citation information: DOI
10.1109/ACCESS.2020.3007874, IEEE Access
and train the Scrum master, Scrum team and stakeholders distributed using convenience sampling. Only 77 samples
on product requirements [9], [14], [20], [25], [34]. were valid and completed by IT professionals from South
• Release backlog audit: The audit processes examine the Africa who were involved in managing and implementing
plan and schedule for multiple sprints and the minutes of agile projects in the organization.
the meeting where the length of the iterations was
determined and release plans that were reviewed and B. RELIABILITY TESTS
approved [9], [25]. Cronbach’s alpha coefficient was used to test the reliability
• Sprint backlog audit: The audit processes are examine of the questionnaire and was calculated for the five audit
developed sprint requirements, examine minutes of the processes. The results in table 4 indicate that Cronbach’s
meeting where sprint requirements were reviewed and alpha coefficient was above 0.7, which means that there was
approved, sprint objectives were defined, sprint tasks internal consistency and good reliability [13].
were outlined and estimates were calculated, hold daily
standups, develop a Scrum team task schedule plan, TABLE 4
CRONBACH’S ALPHA RELIABILITY TEST RESULTS
change requests for sprint tasks, Scrum board to monitor Audit process No. of Cronbach’s
progress, and examine sprint progress reports, sprint items alpha
retrospective reports, product acceptance criteria, Coefficient
burndown chart and effective use of the product [9], [34], Product vision audit 12 0.879
[35]. Product backlog audit 9 0.924
• Product audit: The audit processes examine product Release backlog audit 4 0.811
deliverables acceptance report, formal working Sprint backlog audit 16 0.929
deliverables agreement, project retrospective, project
Product audit 5 0.748
closed contract and product support and maintenance
agreement [9], [14], [25].
• Steering committee: The steering committee is a project V. RESULTS
governance mechanism which oversees agile project The survey results are presented in descriptive analysis and
progress, provides project support and guidance, correlation analysis.
monitors project performance, controls project
implementation activities and provides a framework for A. DESCRIPTIVE ANALYSIS
decision making throughout the implementation of the A total of 77 complete responses were received from IT
agile project [6], [18], [32], [33]. professionals in South Africa, as shown in table 5. A profile
• Adherence to agile values and guiding principles: The of the respondents indicated that 37.7% were project
Agile Manifesto consists of two components, namely managers, 15.6% were Scrum developers and 13% were
values and guiding principles [5]. In managing and business analysts.
implementing agile projects, project stakeholders should
TABLE 5
adhere to agile values and guiding principles [20]. RESPONDENT PROFILE
Type of respondent Frequency Valid per cent
IV. METHODOLOGY Sponsor 1 1.3
Project manager 29 37.7
A. DATA COLLECTION Product owner 2 2.6
In the literature there are various validation techniques Chief Scrum master 8 10.4
Program Scrum master 1 1.3
applied to qualitative and quantitative research methods. Scrum developer 12 15.6
Validation techniques used in qualitative research include Business analyst 10 13.0
case studies, ethnography, focus groups and interviews. Tester 4 5.2
While quantitative research applied survey questionnaires, Stakeholder 2 2.6
experiments, observations, content analysis. This research Engineer 1 1.3
User 7 9.0
adopted questionnaire as a validation technique because
Total 77 100.0
large amounts of information can be collected from a larger
sample size at a relative low cost. Questionnaire also
As illustrated in table 6, most of the respondents had been
provides the same questions to respondents and maintains
using an agile approach in managing and
standardization in gathering information [15]. The research
implementing projects for between 3 and 6 years. Most of
used a quantitative research method and data were collected
the project managers were involved in managing agile
through a structured survey questionnaire. The questionnaire
projects in their organizations.
aimed at answering the research questions in relation to
auditing agile projects in the organization. The survey TABLE 6
instrument was pre-tested to assess content validity, the AGILE APPROACH IN MANAGING AND IMPLEMENTING
results were reviewed, and changes were incorporated into PROJECTS
the final questionnaire. The online survey questionnaire was
This work is licensed under a Creative Commons Attribution 4.0 License. For more information, see https://creativecommons.org/licenses/by/4.0/.
This article has been accepted for publication in a future issue of this journal, but has not been fully edited. Content may change prior to final publication. Citation information: DOI
10.1109/ACCESS.2020.3007874, IEEE Access
Period of using agile approach in managing and 2-4 4-8 8-12 > 12
weeks weeks weeks weeks
implementing projects
<5 6 5 1 0 12
Average number of
Less than Between 3 More than sprints
Total Between 5 4 23 0 32
3 years and 6 years 6 years
5 and 10
sprints
Sponsor Between 2 5 23 0 30
0 0 1
1 10 and 15
Project Between 0 0 1 0 1
Respondents’ role in agile project
4 18 7 29
manager 15 and 20
Product owner 0 1 1 2 > 20 1 0 0 1 2
Chief Scrum Total 14 14 48 1 77
0 5 3 8
master
management
Program
Scrum master
0 1 0 1 There are various agile methods used in managing projects.
Scrum Scrum (75%) was the most common agile method used by
2 8 2 12
developer organizations in South Africa, as shown in Fig. 4. According
Business
5 4 1 10 to VersionOne [51], 54% of their respondents used Scrum
analyst as a method of managing projects. These results mean that
Tester 0 4 0 4
Stakeholder 1 1 0 2
most organizations worldwide, including South Africa, use
Engineer 0 1 0 1 Scrum in managing and implementing projects. However,
User 3 4 0 7 more South African organizations (75%) use Scrum as a
Total 15 48 14 77 method compared to the rest of the world [51].
TABLE 7
SIZE OF SCRUM TEAM AGAINST AVERAGE NUMBER OF
SPRINTS IN A RELEASE
Size of Scrum team
Between 5 Between
<5 > 15
& 10 10 & 15
members members
members members Total
<5 6 5 0 1 12
Average number of sprints
Between 5 1 28 3 0 32
FIGURE 4. Agile methods used in managing projects
& 10
Between 1 27 2 0 30 As illustrated in Fig. 5, most respondents indicated that the
10 & 15 reasons for adopting agile were accelerated software
delivery in their organization (67%), enhanced delivery
Between 0 0 1 0 1
predictability (59%) and enhanced ability to manage
15 & 20 changing abilities (54%). On the other hand, reasons for
> 20 0 1 1 0 2 adopting agile were less about improving engineering
Total 8 61 7 1 77 discipline (6%), managing a distributed team (11%) and
increasing software maintainability (25%). The reasons for
adopting agile for South Africa differ from those of the
The results in table 8 indicate that the average duration of a
global survey [51], but both surveys have identified reasons
Scrum sprint was between 8 and 12 weeks. The average
for adopting agile.
number of sprints in a release was between 5 and 10 sprints.
This means that when the average duration of Scrum sprints
was between 8 and 12 weeks, the average number of sprints
in a release was between 5 and 10 sprints.
TABLE 8
AVERAGE DURATION OF SCRUM SPRINTS AGAINST AVERAGE
NUMBER OF SPRINTS IN A RELEASE
Average duration of Scrum sprints Total
This work is licensed under a Creative Commons Attribution 4.0 License. For more information, see https://creativecommons.org/licenses/by/4.0/.
This article has been accepted for publication in a future issue of this journal, but has not been fully edited. Content may change prior to final publication. Citation information: DOI
10.1109/ACCESS.2020.3007874, IEEE Access
Daily standup 68 86
Sprint/iteration planning 67 80
Retrospectives 61 80
Sprint/iteration review 59 80
Short iterations 40 67
Releasing planning 33 57
Story mapping 15 38
Frequent releases 13 50
This work is licensed under a Creative Commons Attribution 4.0 License. For more information, see https://creativecommons.org/licenses/by/4.0/.
This article has been accepted for publication in a future issue of this journal, but has not been fully edited. Content may change prior to final publication. Citation information: DOI
10.1109/ACCESS.2020.3007874, IEEE Access
B. CORRELATION ANALYSIS
A Pearson correlation analysis test was performed to
determine whether there was any significant relationship
between the audit processes and agile project success.
Cohen [11] suggests guidelines to interpret the relationship
values between 0 and 1. According to Pallant [38], these
guidelines apply whether or not the r value is positive or
negative. r = 0.100 to 0.290 indicates a small or weak
relationship, r = 0.300 to 0.490 indicates a medium or
moderate relationship and r = 0.500 to 1.000 indicates a
large or strong relationship.
A correlation analysis test was then performed between the
proposed audit processes and extent to which agile projects
implemented are successful in the organization. These audit
processes were for the product vision, product backlog, release
backlog, sprint backlog and product deliverables of the scrum
methodology. The analysis aimed at determining whether
there was a relationship between agile project success and
those audit processes.
FIGURE 8. Agile project success criteria The items in each audit process were derived from the
literature reviews and scrum methodology processes. In this
Agile project success was measured using the weights in study, measures were designed from the literature review on
percentage (%) for 22 success criteria against audit agile project management and scrum methodology processes.
processes as shown in table 9. These weights (%) were A 5-point Likert scale ranging from “Strongly disagree” to
obtained from respondents’ perceptions on relationship “Strongly agree” was used to measure the 46 items. In the
between agile project success criteria and audit processes correlation analysis project success was measured using the
(46 items). Table 9 results indicated that Customer/ User extent to which the audit processes were implemented better
satisfaction (72%) had the highest weight (%) between agile in managing agile projects. The respondents were asked to use
project success criteria and audit processes followed by the most recent agile project that they were involved in. The
Revenue/ Sales impact (49%), Budget Vs. Actual cost respondents used the Likert scale to rank the audit processes
(39%), and Individual hours per iteration/week (39%). that can be used to audit agile projects. The ranking was done
TABLE 9
according to the importance of the audit process.
ANALYSIS RESULTS FOR AGILE PROJECT SUCCESS CRITERIA Correlation analysis results are discussed in the following
AND AUDIT PROCESSES sections.
This work is licensed under a Creative Commons Attribution 4.0 License. For more information, see https://creativecommons.org/licenses/by/4.0/.
This article has been accepted for publication in a future issue of this journal, but has not been fully edited. Content may change prior to final publication. Citation information: DOI
10.1109/ACCESS.2020.3007874, IEEE Access
This work is licensed under a Creative Commons Attribution 4.0 License. For more information, see https://creativecommons.org/licenses/by/4.0/.
This article has been accepted for publication in a future issue of this journal, but has not been fully edited. Content may change prior to final publication. Citation information: DOI
10.1109/ACCESS.2020.3007874, IEEE Access
This work is licensed under a Creative Commons Attribution 4.0 License. For more information, see https://creativecommons.org/licenses/by/4.0/.
This article has been accepted for publication in a future issue of this journal, but has not been fully edited. Content may change prior to final publication. Citation information: DOI
10.1109/ACCESS.2020.3007874, IEEE Access
team tasks schedule (r = 0.442, N = 73, p < .01), a sprint blocks list (r = 0.343,
plan N = 73, p < .01), existence of a sprint burndown chart to
SB4: There are
assigned agreement monitor the progress of the Scrum team (r = 0.446, N = 73,
by Scrum team to p < .01) and existence of a sprint retrospective report (r =
show their
.284* 0.015 0.459, N = 73, p < .01). This finding is in line with findings
commitment to from [9], [25] which indicated that minutes of daily standup
deliver the product
requirements to all meetings should be documented, to have sprint blocks list,
the sprints using a sprint burndown chart to monitor the progress of the
SB5: There are Scrum team. These results indicate that these audit
minutes of the sprint processes play an important role in influencing agile project
planning meeting
that defined sprint .524** 0.000
success.
objectives, outline On the other hand, there was a significant weak correlation
sprint tasks and between extent to which agile projects implemented are
calculate estimates
successful in the organization and existence of a signed
SB6: There is a
change request for .311** 0.007 agreement by the Scrum team to show their commitment to
sprint tasks deliver the product requirements to all the sprints (r = 0.284,
SB7: An updated N = 73, p < .05). This result implies that the signed agreement
and maintained
product backlog
.436** 0.000 by the Scrum team to show their commitment to deliver the
continuously product requirements to all the sprints plays a positive role in
SB8.a: There are influencing agile project success.
minutes of daily .442** 0.000 A strong significant correlation was indicated between
standup meetings
extent to which agile projects implemented are successful in
SB8.b: There are
.119 .314
the organization and existence of minutes of the sprint
sprint progress
planning meeting that defined sprint objectives, outlined sprint
reports
SB8.c: There is a tasks and calculated estimates (r = 0.524, N = 73, p < .01).
.343** 0.003 These findings align with the findings from [9] which
sprint blocks list
SB8.d: There is a indicated that it is important to have minutes of the meeting
sprint burndown
chart to monitor
for sprint planning. This result means that existence of minutes
.446** 0.000
progress of the of the sprint planning meeting that defined sprint objectives,
Scrum team outlined sprint tasks and calculated estimates has a significant
SB8.e: There is a role in influencing agile project success.
scrum board used by
scrum team to track .142 .231 There was no correlation between agile project success
its progress during and audit process SB3, SB8.b, SB8.e, SB9, SB11.a and
sprint SB11.b.
SB9: There are
.194 .100
sprint review reports
SB10: There is a 5) PRODUCT DELIVERABLES AUDIT PROCESSES
.459** 0.000
sprint retrospective Table 14 presents correlation results for the product
SB11.a: There are deliverables audit processes.
product acceptance .072 .544
criteria TABLE 14
SB11.b: Customer is CORRELATION ANALYSIS RESULTS FOR PRODUCT
.157 .185 DELIVERABLES AUDIT PROCESSES
using the product
*. Correlation is significant at the 0.05 level (2-tailed). Extent to which agile projects
**. Correlation is significant at the 0.01 level (2-tailed). Audit process implemented are successful in the
N=73 organization
Pearson Sig.
correlation (2-tailed)
Moderate significant correlations were found between PR1: There are
extent to which agile projects implemented are successful in stakeholders' product
.196 .096
the organization and agile project success and the deliverables acceptance
reports
development of sprint requirements documents which are PR2: There is a formal
aligned with the product backlog requirements (r = 0.456, working deliverables
N = 73, p < .01), existence of minutes of the meeting that Agreement that
.089 .453
reviewed and approved sprint requirements documents documents the
(r = 0.424, N = 73, p < .01), existence of change requests successful completion of
the sprint
for sprint tasks (r = 0.311, N = 73, p < .01), the updating PR3: Establishment of
and maintenance of the product backlog (r = 0.436, N = 73, .454** .000
project retrospective
p < .01), existence of the minutes of daily standup meetings PR4: Closed a contract .449** .000
This work is licensed under a Creative Commons Attribution 4.0 License. For more information, see https://creativecommons.org/licenses/by/4.0/.
This article has been accepted for publication in a future issue of this journal, but has not been fully edited. Content may change prior to final publication. Citation information: DOI
10.1109/ACCESS.2020.3007874, IEEE Access
This work is licensed under a Creative Commons Attribution 4.0 License. For more information, see https://creativecommons.org/licenses/by/4.0/.
This article has been accepted for publication in a future issue of this journal, but has not been fully edited. Content may change prior to final publication. Citation information: DOI
10.1109/ACCESS.2020.3007874, IEEE Access
requests for the sprint tasks and minutes of daily standups are
important audit processes in implementing a successful agile VII. CONCLUSIONS
project. These findings were highlighted in the studies of
Deloitte [14], Qasaimeh and Abran [35] and Newmark et al. A. CONTRIBUTIONS OF THE STUDY
[34]. The theoretical contribution of this study is firstly that it
Other research findings are that the sprint blocks list, provides a conceptual framework for auditing agile projects.
burndown chart to monitor the progress of the Scrum team, the The framework introduces audit processes which have not
sprint retrospective and the project retrospective are also been used in auditing agile projects. Secondly, there are
important audit processes in achieving a successful agile limited studies on how to audit agile projects and auditing
project outcome [20], [25], [34]. agile projects is a new area of research. Thirdly, none of the
project management best practices and standards has
provided guidance on agile project auditing. This study
C. WEAK AGILE AUDIT PROCESSES
contributes to the body of knowledge on agile project
The research found that the establishment of a steering
auditing.
committee to oversee the implementation of the project
The practical contribution of this study firstly, it provides
activities and provide a decision-making framework
a tool on how to audit agile projects using scrum
throughout the project life cycle is a success factor for the
methodology. The project management practitioners from
delivery of projects [6], [18], [33].
both public and private sector including auditors can use the
The key research findings are that training for the Scrum
framework to audit agile projects to improve success rates of
master and Scrum team on product requirements is an
agile projects. Secondly, this study contributes knowledge to
important audit process that influences agile project success.
the agile project management curriculum of the education
The product backlog approval document is an important audit and training institutions. These institutions can incorporate
process in achieving agile project success. These findings the concept of auditing agile projects in their project
were highlighted in the study by Deloitte [14] on auditing agile management curriculum to create competent agile project
projects. auditors in the IT industry.
The research found that training the Scrum master and
Scrum team on product requirements is an important audit B. LIMITATIONS OF THE STUDY
process in delivering a successful agile project. These findings This study was limited because of the small sample size (77)
support those from studies by Guerrero et al. [20] and drawn from South Africa. Further research is needed to
Newmark et al. [34]. Ensuring that there are minutes of the collected data globally in a larger sample size. Secondly, this
meetings that determined the length of the iterations and sprint study focused on the Scrum agile method because it is the
meetings, and minutes of the meetings that reviewed and method commonly used in South Africa. There are various
approved the release plan is an important audit process in agile methods, and further studies may devise audit
achieving a successful agile project outcome. The research processes for different agile methods. A quantitative method
also found that a signed agreement by the Scrum team to show using a survey questionnaire was used for data collection.
their commitment to deliver the product requirements to all the Further studies can use a qualitative method or mixed
sprints was one of the important audit processes in agile method and compare the findings. There are limited studies
project success. These findings corroborate those of Deloitte on how to audit agile projects using scrum methodology.
[14]. The overall audit processes that contribute to agile This study cannot be generalized to projects from other
project success are as shown in figure 9. industries since the domain investigated was IT project
management. Further studies are needed on how the
proposed framework can be generalized in other industries.
The survey questionnaire did not include type of institutions
(either public or private), this is among the limitations of this
study.
C. FUTURE RESEARCH
Some of the limitations of this study highlight the areas for
further research. As agile project management is widely
used, further research is needed to validate the conceptual
framework for auditing agile projects in more countries
FIGURE 9. Audit processes contribute to Agile project success
worldwide. Further research is also needed to compare the
proposed work and justify performance and needs of work
Based on the above findings, the audit processes in the
with recent studies and methods.
strong group are very important in achieving a successful agile
project outcome. The audit processes in the moderate group
Appendix A: List of all audit processes for Auditing Agile
are important and those in the weak group are somewhat
project
important in agile project success.
This work is licensed under a Creative Commons Attribution 4.0 License. For more information, see https://creativecommons.org/licenses/by/4.0/.
This article has been accepted for publication in a future issue of this journal, but has not been fully edited. Content may change prior to final publication. Citation information: DOI
10.1109/ACCESS.2020.3007874, IEEE Access
This work is licensed under a Creative Commons Attribution 4.0 License. For more information, see https://creativecommons.org/licenses/by/4.0/.
This article has been accepted for publication in a future issue of this journal, but has not been fully edited. Content may change prior to final publication. Citation information: DOI
10.1109/ACCESS.2020.3007874, IEEE Access
[4] S. Ambler, “The agile scaling model (ASM): Adapting agile methods [24] M. Karlesky and M.V. Voord, “Agile project management for
for complex environments”, IBM Corporation, Somers, NY, USA. burning your gantt charts,” in Proc. of Embedded Sys. Conference,
RAW14204-USEN-00. Dec. 2009. Boston, MA, USA, 2008, pp. 247-267.
[5] K. Beck, M. Beedle, A. van Bennekum, W. Cunningham, M. Fowler, [25] D. H. Kim, D. S. Kim, C. Koh, and H. W. Kim, “An information
J. Grenning, J. Highsmith, A. Hunt, R. Jeffries, J. Kern, B. Marick, system audit model for project quality improvement by the agile
R.C. Martin, S. Mellor, K. Schwaber, J. Sutherland, and D. Thomas, methodology,” Int. J. of Info. and Ed. Technology, vol. 3, no. 3,
“Manifesto for agile software development”. [Online]. Available pp.295-299, 2013.
http://agilemanifesto.org, Accessed on: Feb. 20, 2018. [26] J. Li, N.B. Moe, and T. Dybå, “Transition from a plan-driven process
[6] C. Biesenthal and R. Wilden, “Multi-level project governance: trends to Scrum: A longitudinal case study on software quality,” in Proc. of
and opportunities,” Int. J. of Proj. Manag., vol. 32, pp.1291-1308, the 2010 ACM-IEEE Int. Symp. on Empirical Software Eng. and
2014. Measurement, New York, NY, USA, 2010, pp. 13:1-13:10.
[7] D. Cao and T. Chow, “A survey study of critical success factors in [27] M. Lindvall, V. Basili, B. Boehm, P. Costa, K. Dangle, F. Shull, R.
agile software projects,” J. of Systems and Software, vol. 81, no. 6, Tesoriero, L. William, and M. Zelkowitz, “Empirical findings in
pp. 961-971, 2008. agile methods. Extreme programming and agile methods XP/agile
[8] D. Cao, M. Djordjevic, V. Nikolic, and D. Stankovic, “A survey study universe,” Lecture Notes in Computer Science, vol. 2418, pp. 197-
of critical success factors in agile software projects in former 207, 2002.
Yugoslavia IT companies,” J. of Systems and Software, vol. 86, no. [28] J. Lemétayer and J. Sheffield, “Factors associated with the software
6, pp. 1663-1678, 2013. development agility of successful projects,” Int. J. of Proj. Manag.,
[9] D. Canty, Agile for project manager. Brooken Sound Parkway, NW, vol. 31, no. 3, pp. 459-472, 2013
USA: Taylor & Francis Group, 2015. [29] U. Malik, H. Nasir, and A. Javed, A. “An efficient objective quality
[10] E. Cardozo, A. Neto, A. Barza, C. França, and F. da Silva, “Scrum model for agile application development,” Int. J. of Computer Appl.,
and productivity in software projects: A systematic literature review” vol. 85, no. 8, pp.19-24, 2014.
in Proc. of the 14th Int. Conference on Eval. and Assess. in Soft. Eng. [30] C. Marnewick, Rebooting information technology project
(EASE ‘10), Keele, UK, 2010, pp. 131-134. management. [Online]. Available
[11] J. Cohen, Statistical power analysis for the behavior sciences. https://www.uj.ac.za/newandevents/Documents/2018/Prof%20C%2
Hillsdale, NJ, USA: Lawrence Erlbaum Associates, 1988. 0Marnewick%20-%20Inaugural%20Address.pdf, Accessed on: Feb.
[12] F.K. Chan and J.Y. Thong, “Acceptance of agile methodologies: A 12, 2019.
critical review and conceptual framework,” Decision Support Sys., [31] S. Mehreen and A. Fahim, “Evaluation of quality assurance factors
vol. 46, no. 4, pp. 803-814, 2009. in agile methodologies,” Int. J. of Advanced Computer Sc., vol. 2, no.
[13] L.J. Cronbach, “Coefficient alpha and the internal structure of test,” 2, pp. 73-78, 2012.
Psychometrika, vol. 16, no. 3, pp. 297-334, 1951. [32] E. Mkoba and C. Marnewick, “IT project success: A conceptual
[14] Deloitte. Auditing Agile projects Your grandfather’s audit won’t framework for IT project auditing assurance,” in Proc. of Annual
work here! [Online]. Available Conference of the South African Inst. of Comp. Scientists and
https://www2.deloitte.com/content/dam/Deloitte/us/Documents/risk/ Information Technologists, Sep. 2016, vol. 26, pp. 1-8,
us-risk-auditing-agile-projects.pdf, Accessed on: Oct. 14, 2019. doi: 10.1145/2987491.2987495.
[15] M. Denscombe, The Good Research Guide for small-scale social [33] R. Müller, Project governance. Surrey, UK: Gower, 2009.
research projects, 4th ed. Berkshire, UK: Open University Press, [34] R. Newmark, G. Dickey and W. Wilcox, “Agility in audit: Could
2010. scrum improve the audit process,” American Acc. Ass., vol. 12,
[16] T. Dingsøyr, S. Nerur, V. Balijepally, and N. Moe, “A decade of agile no. 1, pp. A18-A28, 2018.
methodologies: Towards explaining agile software development,” [35] M. Qasaimeh and A. Abran, “An audit model for ISO 9001
The J. of Sys. and Soft., vol. 85, no. 6, pp. 1213-1221, 2012. traceability requirements in agile-XP environments,” J. of Software,
[17] R. Dovleac and C. Suciu, “Quality assurance within the agile system vol. 8, no. 7, pp. 1556-1567, 2013.
development life-cycle,” Int. J. of Econ. and Manag. Sys., vol. 3, [36] IIA, Auditing projects in the early stages. [Online]. Available:
pp.182-187, 2019. https://www.iia.org.uk, Accessed on: Dec. 3, 2019.
[18] R. Garland, Project governance: A practical guide to effective project [37] ISO/IEC 25010, Systems and software engineering - System and
decision making. London, UK: Kogan, 2009. software product Quality Requirements and Evaluation (SQuaRE) -
[19] P. Gibbons, The science of successful organizational change: How System and software quality model, Geneva, Switzerland: ISO/IEC,
leaders set strategy, change behavior, and create an agile culture. 2011.
Upper Saddle River, NJ, USA: Pearson Education, 2015. [38] J. Pallant, SPSS survival manual: A step by step guide to data analysis
[20] A. Guerrero, R. Fresno, A. Ju, A. Fox, P. Fernandez, C. Muller, and using IBM, 5th ed. Cross Nest, NSW, Australia: Allen & Unwin, 2013.
A. Ruiz-Cortés, “Eagle: A team practices audit framework for agile [39] R. Pearson, Rethinking Gender Matters in Development, in Allen, T.
software development,” in Proc. of the 27th ACM ESEC/FSE’19, & Thomas, A. (eds.). Poverty and Development into the 21st Century.
Estonia., 2019, pp. 1139-1143, doi:10. 1145/3338906.3341181. Oxford: Oxford University Press, 2000.
[21] Guide to the Scrum Body of Knowledge, A guide to the scrum body [40] PMI Pulse of the Profession, PMI pulse of the profession, 9th global
of knowledge (SBOK™ Guide). Arizona, USA: Scrum Study, 2014. project survey: Success rates rise, transforming the high cost of low
[22] R. Hoda, S. Marshal, and J. Noble, “Agile project management,” in performance, Newton Square, PA, USA: Project Management
Proc. of Comp. Sc. Research Student Conference, New Zealand, Institute, 2017.
2008, pp. 218-221. [41] PWC. Agile project delivery confidence mitigate project risks
[23] M. Huemann, “Management audits of projects and programme: How and deliver value to your business. [Online]. Available:
to improve project management and programme management https://www.pwc.com/gx/en/actuarial-insurance-
quality,” in Proc. of III IPMA ICEC International Expert Seminar, services/assets/agile-project-delivery-confidence.pdf, Accessed on:
Bilbao, Spain, 2004, pp.349-350. Dec. 3, 2019.
This work is licensed under a Creative Commons Attribution 4.0 License. For more information, see https://creativecommons.org/licenses/by/4.0/.
This article has been accepted for publication in a future issue of this journal, but has not been fully edited. Content may change prior to final publication. Citation information: DOI
10.1109/ACCESS.2020.3007874, IEEE Access
This work is licensed under a Creative Commons Attribution 4.0 License. For more information, see https://creativecommons.org/licenses/by/4.0/.