See discussions, stats, and author profiles for this publication at: https://www.researchgate.

net/publication/317132380

Improving software quality using Six Sigma DMAIC-based approach: a case

study

Article  in  Business Process Management Journal · July 2017

DOI: 10.1108/BPMJ-02-2017-0028

CITATIONS READS

11 293

2 authors, including:

Anjali Awasthi
Concordia University Montreal
149 PUBLICATIONS   3,840 CITATIONS   

SEE PROFILE

Some of the authors of this publication are also working on these related projects:

Call for Book Chapters: Fleet Management and Planning for Sustainable Connected Mobility Systems View project

Call for Book Chapters : Sustainable City Logistics Planning View project

All content following this page was uploaded by Anjali Awasthi on 06 May 2021.

The user has requested enhancement of the downloaded file.

 Business Process Management Journal
Improving software quality using Six Sigma DMAIC-based approach: a case study
Racha Karout, Anjali Awasthi,
Article information:
To cite this document:
Racha Karout, Anjali Awasthi, (2017) "Improving software quality using Six Sigma DMAIC-based
approach: a case study", Business Process Management Journal, Vol. 23 Issue: 4, pp.842-856,
https://doi.org/10.1108/BPMJ-02-2017-0028
Permanent link to this document:
https://doi.org/10.1108/BPMJ-02-2017-0028
Downloaded by Concordia University At 16:48 13 November 2017 (PT)

Downloaded on: 13 November 2017, At: 16:48 (PT)

References: this document contains references to 30 other documents.
To copy this document: permissions@emeraldinsight.com
The fulltext of this document has been downloaded 289 times since 2017*
Users who downloaded this article also downloaded:
(2017),"Six Sigma model of transfer of development capability", Business Process Management
Journal, Vol. 23 Iss 4 pp. 857-872 <a href="https://doi.org/10.1108/BPMJ-02-2017-0026">https://
doi.org/10.1108/BPMJ-02-2017-0026</a>
(2017),"Lean process design for a radiology department", Business Process Management
Journal, Vol. 23 Iss 4 pp. 779-791 <a href="https://doi.org/10.1108/BPMJ-02-2017-0025">https://
doi.org/10.1108/BPMJ-02-2017-0025</a>

Access to this document was granted through an Emerald subscription provided by emerald-
srm:149884 []
For Authors
If you would like to write for this, or any other Emerald publication, then please use our Emerald
for Authors service information about how to choose which publication to write for and submission
guidelines are available for all. Please visit www.emeraldinsight.com/authors for more information.
About Emerald www.emeraldinsight.com
Emerald is a global publisher linking research and practice to the benefit of society. The company
manages a portfolio of more than 290 journals and over 2,350 books and book series volumes, as
well as providing an extensive range of online products and additional customer resources and
services.
Emerald is both COUNTER 4 and TRANSFER compliant. The organization is a partner of the
Committee on Publication Ethics (COPE) and also works with Portico and the LOCKSS initiative for
digital archive preservation.
 *Related content and download information correct at time of download.
Downloaded by Concordia University At 16:48 13 November 2017 (PT)

BPMJ
23,4
842
Received 9 February 2017
Revised 14 April 2017
Accepted 24 April 2017
Downloaded by Concordia University At 16:48 13 November 2017 (PT)
Business Process Management
Journal
Vol. 23 No. 4, 2017
pp. 842-856
© Emerald Publishing Limited
1463-7154
DOI 10.1108/BPMJ-02-2017-0028
The current issue and full text archive of this journal is available on Emerald Insight at:
www.emeraldinsight.com/1463-7154.htm
Improving software quality using
Six Sigma DMAIC-based
approach: a case study
Racha Karout and Anjali Awasthi
Concordia University, Montreal, Canada
Abstract
Purpose – Managing quality is a vital aspect in software development world, especially in the current business
competition for the fast delivery of feature rich products with high quality. For an organization to meet its
intended level of excellence in order to ensure its success, a culture of quality should be built where every
individual is responsible of quality and not just the software testing team. However, delivering software
products with very few bugs is a challenging constraint that is usually sacrificed in order for a company to meet
other management constraints such as cost, scope and scheduling. The paper aims to discuss these issues.
Design/methodology/approach – In this paper, the authors present a Six Sigma DMAIC-based framework
for improving software quality. Different phases of DMAIC methodology are applied for quality improvement in
one of the largest software applications for “RK” company (name anonymized) in Canada where critical to
quality aspects are identified, production bugs classified and measured, the causes of the large number of
production bugs were specified leading to different improvement suggestions. Several metrics were proposed to
help “RK” company control its software development process to ensure the success of the project under study.
Findings – This paper shows how companies can use a systematic approach such as DMAIC to eliminate
errors and improve efficiency. It helps them to identify and implement improvements that leads to an
increased confidence in the quality of the product produced at all levels.
Originality/value – By applying DMAIC at “RK” company the authors were able to demonstrate how
DMAIC can help organizations improve the quality of their software products. As a result, reduce cost and
cycle times, achieve customer satisfaction and improve profit margin.
Keywords Six Sigma, DMAIC, Customer satisfaction, Software quality
Paper type Research paper
1. Introduction
To compete in today’s world, businesses require a structured approach, disciplined
thinking, and the engagement of everyone in the organization (Evans and Lindsay, 2005).
The software industry is no exception. In recent years, software has become an increasingly
important component in consumer products and capital goods industries. More and more
products contain multiple software components (embedded systems), and depend on it for
many of their functions. The service industry has also not remained untouched with
software playing an important role in many service industries such as telecoms, banking
and insurance (Issac et al., 2010). As a result, the criticality of software quality in decision
making cannot be ignored (Awasthi and Varman, 2003). Poor software quality results in
delayed, failed, abandoned or rejected software project. Even those software projects
already implemented may need expensive ongoing maintenance and corrective releases or
service packs for assuring good software quality (Chow and Cao, 2008). Moreover,
IT companies continue to be challenged in overcoming intractable software quality
problems and operational complexity while also trying to deliver software as per the agreed
scope, schedule and budget (Nanda and Robinson, 2011).
IEEE (1991) defines software quality as the degree to which a system, component or
process meets specified requirements, in other words the degree to which a system,
component or process meets customer or user needs or expectations (as cited in Galin,
2004). Chang et al. (2006) mentioned that major software quality attributes are
functionality, reliability, usability and maintainability. There should not be issues that
affect its ability to maintain or reestablish its level of performance. The software should be
 easy to use and properly structured that would make it easy to maintain. However, Improving
with today’s market competition and the need for rapid delivery, software quality is often software quality
sacrificed, thus leading to failure of the software projects. Also, the use of traditional using Six Sigma
methodology such as waterfall with the current market pace, continuously changing
customer requirements, rapidly evolving technology in the current software industry DMAIC
plays a major role in reduced software test coverage, and as a consequence poor software
quality. Moreover, people jump to solutions without fully understanding the problem or 843
finding the root cause of poor quality.
Several approaches have been used to improve software quality. For example, Basili and
Caldiera (1995) suggested a quality improvement paradigm for the development of strategic
capabilities based on a control tool (the goal-oriented approach to measurement that
addresses the support of the improvement process with quantitative information) and an
organizational tool (an infrastructure aimed at capitalization and reuse of software
experience and products). Dawson (1994) suggested a continuous improvement approach
Downloaded by Concordia University At 16:48 13 November 2017 (PT)

for a software project at Motorola. Several improvements were considered such as better
project management, configuration management, structured requirements analysis and
design, development tools, testing and acceptance criteria and metrics. Subramanian et al.
(2007) argued that capability maturity model levels influence the choice of information
system implementation factors such as training, executive commitment, simplicity and
prototyping which in turn impacts software quality and performance. Hossain et al. (2013)
mentioned agile technique to develop and implement software quickly in close cooperation
with the customer in an adaptive way.
In this paper, we propose a Six Sigma-based approach for improving software quality in
organizations. Six Sigma has gained widespread acceptance in many industries as a
beneficial approach to product and process improvement (Gack and Robison, 2003). It can be
best described as a business process improvement approach that seeks to find and eliminate
causes of defects and errors, reduce cycle times and cost of operations, improve
productivity, better meet customer expectations, and achieve higher asset utilization and
returns on investment in manufacturing and service processes (Evans and Lindsay, 2005).
The objective of Six Sigma is to increase the profit margin, improve financial condition by
minimizing the defects rate of product. It increases the customer satisfaction, retention and
produces the best class product from the best process performance (Kabir et al., 2013). It is
based on a simple problem solving methodology – DMAIC, which stands for “define,
measure, analyze, improve and control.” The strengths of DMAIC are that it is a systematic
approach for eliminating errors and improving efficiency. This allows organizations to
identify and implement improvements that leads to an increased confidence in the quality of
the product produced at all levels: team, management, marketing and most importantly the
customer especially in the current competitive market that demands short cycle time, fast
software releases with feature rich and high-quality software products. Despite the defined
strengths, there are some weaknesses as well. It requires the total cooperation of the
organization at all levels. It also relies on good data for understanding process performance,
thus, it requires a considerable effort to be made to collect accurate data which makes it time
consuming and costly.
The rest of the sections are organized as follows. In Section 2, we present the literature
review. Section 3 contains the solution approach. The numerical application is provided in
Section 4. Finally, Section 4 concludes the paper and provides future research directions.

2. The DMAIC methodology

The DMAIC is a process improvement cycle of Six Sigma program as well as an effective
problem solving methodology (Hung and Sung, 2011). The five steps involved in the DMAIC
methodology are described as follows.
 BPMJ 2.1 Define
23,4 After a Six Sigma project is selected, the first step is to clearly define the problem. One must
describe the problem in very specific operational terms that facilitate further analysis.
The purpose of this phase is to set the project goals based on the knowledge of the
organization, customer critical to quality (CTQ) and the process that needs to be improved.

844 2.2 Measure

This phase focuses on how to measure the internal processes that impact CTQs. It requires
understanding the causal relationship between process performance and customer value.

2.3 Analyze
Too often, we want to jump to a solution without fully understanding the nature of the
problem and identifying the source, or “root cause,” of the problem. The analyze phase
focuses on why defects, errors or excessive variation occur.
Downloaded by Concordia University At 16:48 13 November 2017 (PT)

2.4 Improve
Once the root cause of a problem is understood, the analyst or team needs to generate ideas
for removing or resolving the problem and thereby improve the CTQs.

2.5 Control
The control phase focuses on how to maintain the improvements, and includes putting tools
in place to ensure that the key variables remain within the maximum acceptable ranges
under the modified process (Evans and Lindsay, 2005). According to Evans and Lindsay
(2005), any control system has three components: a standard or goal, a means of measuring
accomplishments and comparison of actual results with the standard, along with feedback
to form the basis for corrective action.
Table I presents the various techniques employed in this study at different stages of
DMAIC for improving software quality in organizations.

3. Case study
A case study was done at “RK” company by applying DMAIC methodology to one of its
software development projects. “RK” Company is one of the fast growing companies in
Canada. It is a medium size company whose main focus is building software applications
using the latest technologies. It has been in development for the past few years.
The development team is divided into six sub-teams, where each team is specialized in
developing a specific area. In total the project has 35-40 developers. There is also a system

Phase Tools used Justification of usage

Define Critical to quality (CTQs) To determine the metrics that are most important to customers
SIPOC To establish the boundaries of the business process
Measure Pareto charts To prioritize the problem solving work
Analyze 5-Why technique To isolate the causes from the symptoms
Cause and effect diagram Identify the root cause of the problem
Interrelationship diagram Identify the relationship between different causes
Improve Quality function deployment To help improve the design phase
To gain client satisfaction
Table I. Control Measurement metrics To help identify whether the improvements applied are going in
Techniques used the right direction
in DMAIC To help identify areas that needs additional focus
 test team that consists of 12-14 engineers that test the product manually. A software Improving
developer engineers in test team was recently added to create automated test cases for the software quality
project. Their current focus is creating test tools. using Six Sigma
DMAIC
3.1 Software development at RK
The methodology currently followed at “RK” for software quality management of the
project is waterfall. Waterfall model is a sequential software development process in which 845
progress is regarded as flowing increasingly downwards through a list of phases that must
be executed in order to successfully build a computer software (Royce, 1970). These phases
are requirements analysis, system design, implementation, testing and maintenance.
Moving to the next phase can happen only when its preceding phase is completely done.
Waterfall model is good for small projects where requirements are clearly defined and
detailed at the first stage. However, waterfall model is not suitable for moderate to large
projects. The level of uncertainty and risk is very high. It is idealized and does not match
Downloaded by Concordia University At 16:48 13 November 2017 (PT)

reality well. Also, the software is delivered late in project (Munassar and Govardhan, 2010),
and as a result the bugs are not found until the end of the software life cycle which leads to
an increased cost to fix those faults. Moreover, waterfall is not a good model for complex
and object-oriented projects. Another disadvantage of waterfall model is that it is not
suitable for projects where requirements change all the time which leads to a high risk.

3.2 DMAIC methodology for software quality improvement

At “RK” company, the emphasis is on applying DMAIC framework on the described project
in order to identify the root causes for the large number of bugs found in production and
provide improvement suggestions to ensure the success of the project. Production bugs can
be defined as the bugs found after the application code has been released to clients.
Interviews were carried out with the software development manager and the quality
assurance manager to collect information about the process followed and the issues the
team has. Also, Production bugs were collected from the company reporting system and
were classified by type, severity and yearly release. The collected information will be used
throughout the DMAIC phases.
3.2.1 Define phase. The goal is to reduce the number of bugs found in production as low
as possible and to be able to find them as early as possible in the iteration cycle. As a result the
cost will be radically reduced. In this phase we started by identifying the critical to quality
CTQs outputs of the project under study. Also, since the methodology that was being followed
is waterfall, bugs should have been found in the testing phase. As a result, the boundaries of
the current testing process were identified through the use of the SIPOC diagram.
3.2.1.1 CTQs. The most important quality attributes that contribute to customer
perceptions for the project under study are the following:
• Security: security relates to the ability of software to prevent prohibited access and
withstand deliberate attacks intended to gain unauthorized access to confidential
information, or to make unauthorized access (Chang et al., 2006). Since the project
under study is based on security concept, this attribute is the main key for customers.
• Reliability: Quyoum et al. (2010) explained that software reliability is an important facet
of software quality. Software reliability is the probability of the failure free operation of
a computer program for a specified period of time in a specified environment.
• Assurance: this is related to the knowledge and courtesy of employees, and their ability
to convey trust and confidence (Evans and Lindsay, 2005). For example, the ability of the
project support team to answer questions, have the capabilities to do the necessary work
to fix customer issues in a timely manner and be polite and pleasant during this time.
 BPMJ • Efficiency: Chang et al. (2006) explain that it relates to how the software optimally
23,4 uses system resources. It includes the time behavior as the ability of software to
provide appropriate responses, processing time and throughput rate when
performing its function under stated conditions.
• Maintainability: which refers to the ease with which a software can be understood,
modified and retested. The easier the software can be maintained, the easier it is to
846 isolate defects or their causes, correct defects or their causes, maximize the software
useful life, maximize its efficiency, reliability and safety as well as meet new
requirements and cope with a changed environment.
3.2.1.2 Process definition – SIPOC diagrams. Table II presents the “SIPOC”
diagram for the software testing process that is currently implemented for the project
under study.
3.2.2 Measure phase. Concentrating on the stated goal in the define phase, the data
Downloaded by Concordia University At 16:48 13 November 2017 (PT)

collection focused on the number of bugs found in production for the past releases in
2012-2014 since the frequency of releases were yearly. Each release includes new features
and bug fixes. The number of bugs was identified by severity, yearly release and the type of
issues found. It focused on the population in order to achieve more accurate results. Several
Pareto charts were created to summarize and display the relative importance of the
differences between groups of data such as the yearly releases, severity of bugs found and
the type of errors identified in production:
(1) Pareto chart based on type of errors: the issues found in production were classified
by type such as functional, backend services, help information, graphical user
interface, run time error, translation, scalability and configuration. Figure 1 shows
the Pareto chart based on the type of errors.
From Figure 1, we can conclude that “RK” company needs to focus on improving
the functional and backend services.

Supplier Input Process Output Customer

Organization Test manger tool Create manual test cases for New feature test suite Project
new planned features manager
QA team High-level Execute test cases for each New feature test results Development
requirements fully implemented new manager
document feature
QA manager System Report bugs for new features List of identified bugs QA manager
requirements for new feature
document
Development Project stable Verify bug fixes for new Modified build after Organization
team build features bug fixes
Development Testing tools Retest all new features when Regression test results
manager all new planned features are
fully implemented
Project Testing Perform regression testing Updated test cases
manager environment
Test plan Report bugs found in Updated regression test
regression suite to include priority
Regression test Verify bug fixes for one test cases of newly
Table II. suite regression testing implemented features
Project SIPOC Legacy system Perform manual test case
diagram knowledge maintenance (clean up)
 600
Pareto Chart
120%
Improving
500 100% software quality
400
300
80%
60%
using Six Sigma
200
100
40%
20%
DMAIC
0 0%
al

n
ro
e

ilit
io

io

tio
n

847
ic

fa

er
tio

at

at

ab

ra
rv

er
m

sl
nc

u
al
Se

nt

an
tim
r

fig
Sc
fo
Fu

rI

Tr

on
d

In

se

un
en

C
p

lU Figure 1.

R
ck

el
H

ca
Ba

hi

Pareto chart based

p
ra
G

on type of errors
Number of bugs Cumulative Percentage 80% Marker

(2) Pareto chart based on the yearly releases: the faults found in production were also
Downloaded by Concordia University At 16:48 13 November 2017 (PT)

classified by yearly releases. Figure 2 identifies the errors found per year.
It can be seen from Figure 2 that the release in 2013 had the highest number of
bugs followed by the release in 2014. This shows that the “RK” company needs to
concentrate their testing and bug fixes on the features that got released and the
affected functionalities in these two years.
(3) Pareto chart based on severity of the bug: the bugs found in production were also
classified by severity. Figure 3 focuses on this classification.
Looking into Figure 3, we notice that medium and major bugs have the highest
ratio. However, this result does not imply that the focus should not be on critical issues
as well since this type of severity should not even be found in production and could be
showstoppers.
3.2.3 Analyze phase. In this phase we analyzed the results obtained from the measurement
phase. Also, several techniques were used to identify the root cause of the large number of
bugs found in production.
Results of Pareto chart. Analyzing the collected data in the measure phase and applying the
Pareto chart helped in identifying the “vital few” from trivial many in order to identify the areas
that need more intensive focus to improve the quality of software. Pareto chart based on the

Pareto Chart
1,000 200%

500 100%
Figure 2.
0 0% Pareto chart based on
2013 2014 2012
yearly releases
Number of Bugs Cumulative Percentage 80% Marker

Pareto Chart
1,000 150%
100%
500
50%
0 0% Figure 3.
Medium Major Minor Critical Pareto chart based
on severity
Number of Bugs Cumulative Percentage 80% Marker
 BPMJ type of errors (“Figure 1”) indicates the need to focus on fixing the functional and backend
23,4 services issues. The Pareto chart based on yearly releases (“Figure 2”) indicates the need to
concentrate testing and bug fixes on features that got released and affected functionalities. The
Pareto chart based on the severity of the bugs (“Figure 3”) indicates that medium and major
bugs have the highest ratio and should be addressed immediately as well as the critical bugs
since this type of severity should not even be found in production.
848 Results of 5-Why technique. The 5-Why was applied to the project under study as follows:
• Why there are too many bugs found in production in each release?

This is because it was not tested properly.

• Why it was not tested properly?

Because many regression and performance tests were cut in many cases.
Downloaded by Concordia University At 16:48 13 November 2017 (PT)

• Why test cases were cut?

Because test schedule is not adequate for the number of tests that should be done.
• Why there was not enough time to execute all necessary tests?

Because developers never deliver on time and there is a large regression on legacy
features that need to be executed.
• Why developers never deliver on time?
Because of the many features that are added at each release and in some cases client
requirements or new features are added or UI changes are done in the middle of
implementation leading to a change in the scope and the current methodology does not
properly adapt to the requirements change.
Results of cause and effect diagram. Cause and effect diagram is applied to the project
under study to group together the issues identified from the interviews conducted with the
QA manager and the development manager for the project under study. It allowed us to
identify the root cause of the problem as shown in Figure 4.
Results of interrelationship diagram. The interrelationship diagram is applied for the
project under study as shown in Figure 5 in order to identify the potential causal
relationships that might lie behind a problem that continues to recur despite attempts to
resolve it.

Requirements Methodology Flow Test schedule

Scope creeps Manual

Not followed Many legacy regression
Long sprints
No single owner Many new features

Continuous change Late delivery

Change adaptation
Poor quality
Not detailed No explanation limited
Obsolete
No Fix
Training
No frequent update
Friction
Figure 4. No Updates
Cause and effect Investigation
diagram
Documentation Communication Tools
 Improving
Large number of bugs found in production/Poor Quality software quality
using Six Sigma
DMAIC

849
Requirements Methodology Flow Testing Schedule
Downloaded by Concordia University At 16:48 13 November 2017 (PT)

Documentation Communication Tools Figure 5.

Interrelationship
diagram

From the interrelationship diagram we can conclude that the currently followed
methodology is the root cause of poor software quality for the project under study.
3.2.4 Improvement phase. In this phase, several improvement suggestions were
provided for the project under study in order to achieve high-quality product. The main
suggestion for “RK” company was to move from waterfall process to “Agile Methodology.”
3.2.4.1 How can “RK” company implement Scrum method. Schwaber and Beedle (2002)
suggested that the team should consist of five to nine members and if more people are
available, several sub-teams can be formed (as cited in Abrahamsson et al., 2002). This is
already the case of the project under study. The project team is already formed of six sub-
teams, which facilitates the switch from waterfall methodology to agile methodology using
Scrum method and reduces resistance to change.
Following Schwaber and Beedle (2002) suggestion that was mentioned in Abrahamsson
et al. (2002), “RK” company will be adopting scrum for an existing project where the
development environment and technology to be used exists, but the project team has
problems related to complex technology and changing requirements that were identified in
the analysis phase. In this situation, scrum should be started with daily stand-up meetings.
The objective of the first sprint should be to demonstrate any piece of user functionality on
the selected technology. The team should work on solving the impediments of the project
that will enable the team to progress.
The team should apply sprint procedure in order to adapt to changing environmental
variables. Every sprint should begin with the sprint planning meeting in which the product
owner and the team discuss which stories will be moved from the product backlog into the
sprint backlog.
During the sprint, the team should have a daily scrum meeting for 15 minutes during
which the team discuss solutions to challenges and report progress to the product owner.
At the end of the sprint, a review meeting should be held to present the work done to the
product owner in order to determine if the work done has met its acceptance criteria.
Also, the team should have a retrospective meeting to discuss what went good and what
needs to be improved and provide improvement suggestions.
In addition, “RK” company needs to keep in mind that one of the key quality principles of
Six Sigma is participation and team work. In any organization, the person who best
understands his or her job and how to improve both the product and the process is the one
 BPMJ performing it. When managers give employees the tools to make good decisions and the
23,4 freedom and encouragement to make contributions, they virtually guarantee that better
quality products and production processes will result (Evans and Lindsay, 2005).
This quality principle coincides with agile methodology. As mentioned by Moe et al. (2009),
when adopting agile methods in an organization based on traditional, plan-driven
development model, the focus of decision-making moves from the project manager to the
850 software development team, and the decision-making process changes from individual and
centralized to shared and decentralized. Thus, leadership is shared and important decisions
on what to do and how to do it are made through an interactive process involving many
people who influence each other, not just a single person (as cited in Moe et al., 2012).
To succeed in the decision making and reach the required level of maturity in agile
development, the team for the project under study needs to evolve and become an
experienced team that:
Downloaded by Concordia University At 16:48 13 November 2017 (PT)

• collaborates on projects by communicating and being committed;

• cares about customers and software quality;
• allows requirements to change;
• shares knowledge;
• manages source code and tests using tools, methods and metrics supported by
infrastructure appropriate for agility;
• self-organizes at a sustainable pace;
• standardizes and continuously improves agile practices; and
• generates perceived outcomes for customers and management (Fontana et al., 2014).
After stabilizing the change of moving from waterfall to agile using Scrum method, “RK”
company can work on optimizing the process by looking into other tools for directly
improving service delivery and catalyzing continuous improvement. The suggested tool is
called “Kanban.”
3.2.4.2 How can “RK” company implement Kanban. “RK” company can benefit from
Kanban system in addition to the suggested agile Scrum method to limit the team’s
work-in-progress (WIP) to set capacity and to balance the demand on the team against the
throughput of their delivered work. By doing this they can achieve sustainable pace.
Since each software team has different situation, the team of the project under study need
to experiment or in another way evolve their process to best suit their needs.
Following the steps that Anderson (2010) defined in his book:
• First, “RK” company needs to define the start and end point for control. It is
necessary to decide where to start and end process visualization, and in doing so,
define the interface points with upstream and downstream partners.
• The next step is to identify the types of work that arrive at that point and any others
that exit within the workflow that will need to be limited.
• The third step is to draw wall cards to show the activities that happen to the work
rather than specific functions or job descriptions. During the first few weeks the team
may make changes to the board until it stabilizes to fit their needs and criteria.
• The next step is to demand analysis. For each type of work identified, the team for
the project under study should make a study of the demand based on historical data
to make a quantitative study. Then the Kanban system can be designed and
resourced appropriately to cope with this demand.
 • Once the team has an understanding of the demand, they can decide how to allocate Improving
capacity within the Kanban system to cope with that demand. software quality
• Each visual card representing a discrete piece of customer-valued work has several using Six Sigma
pieces of information on it. The design of the card is important. The information on DMAIC
the cards must facilitate the pull system and empower individuals to make their own
pull decisions.
851
• The team needs to align the design of the Kanban system and card wall with the
decision made earlier to limit the boundary of WIP control.
However, changing and optimizing the process is not sufficient for the project under study
to achieve the required level of quality. The project has a lot of legacy features and
complicated functionalities that makes it impossible to cover in regression with the current
method of testing. Not to forget that the software system will continue to grow in
Downloaded by Concordia University At 16:48 13 November 2017 (PT)

advancements and complexity as new features and enhancements are presented with each
iteration. This introduces many challenges on the quality assurance system test team.
Verifying the added features are functioning as required, ensuring that those changes
did not break any of the previous functionalities and validating bug fixes is nearly
impossible to test manually.
“RK” company can meet its intended level of quality by automating test cases where test
execution time is much faster than manual test run, thus leading to maximizing test coverage.
3.2.4.3 How can “RK” company start automating. It is crucial for the automation test
team to have a very good understanding of the product they need to automate test cases for.
Based on this knowledge, several decisions can be made:
(1) The first decision to be made is to determine what test cases to automate and those
that need to be tested manually. It is impossible to automate every possible scenario.
Some types of bugs can be found only while someone is carefully watching the screen
and running the application. These are the types of bugs that humans are vastly
better at detecting than computers are (Page et al., 2009). Also test cases that are
performed once or very few times might not be worth the cost and effort for
automating. However, test cases that will be repeated many times or subject to human
error are very good candidates for automation. Test cases that need to be run with
different configurations or on different platforms can be executed faster if automated.
(2) To benefit from automation as much as possible, it is better for the automation team to
be involved as early as possible in the software development life cycle and run the test
more often. Being involved from the beginning allows the automation team to find
bugs as early as possible, thus reducing the cost of fixing bugs radically. However,
since the project under study has a lot of legacy features, it is impossible to automate
all those features and still catch up with the development team as they will be adding
new features at the same time. In this case, the automation team need to consider
automating new features in order to catch those bugs for the new functionalities as
early as possible and focus on automating core functionalities of legacy features. Also,
they need to focus on the areas where most production bugs are found.
(3) The next step for the automation team is to select the suitable tool or tools to use in
automating test cases. Selecting the right tool is a crucial decision in automation.
The automation team for the project under study needs to consider several key
points that will help to make the right decision:
• The tool should support the platforms and technology that is used for the project
implementation.
 BPMJ • The team might also consider to use the tool that uses the same programming
23,4 language that the development team uses.
• The test tool should be stable enough as the automation team need to avoid false
positive results as much as possible. This is the case when the test fails even
though the targeted functionality is working correctly.
• The richness of the tool features and at the same time the ease of use of the tool is
852 another key point that the automation team needs to consider as it will affect the
effort and time needed to learn how to use it.
• The tool should also have most of the features if not all that supports the
verification of the functionalities for the project under test.
• Another key point to consider is the flexibility of the tool to be able to reuse,
maintain and centralize the test code as much as possible as well as the ability of
Downloaded by Concordia University At 16:48 13 November 2017 (PT)

the tool to support any change in the user interface in order to avoid another type
of false positive of the test result.
(4) Another consideration is that the automation team lead should know the level of
experience and skills for each team member in order to properly distribute the
automation testing effort across the team members as the complexity of the test
cases will have different levels.
Another improvement suggestion that “RK” company can apply or any software
organization is to improve the software quality and CTQ performance at design phase.
This can be done by using a quality tool “quality function deployment” (QFD) that can
be considered as a preventive action that results in a reduction in the cost.
3.2.4.4 How can “RK” company implement QFD. A set of matrixes is used to relate the
voice of the customer to the product’s technical requirements and component requirements.
This matrix is called the “house of quality.”
“RK” company needs to follow the below steps to build the house of quality:
• Identify customer requirements: QFD starts with establishment of objectives, which
represent the answer to “What?” what is desired in order to reach the new service’s
development? These objectives derive from client’s requirements and are called the
“voice of the customer” (Bernal et al., 2009).
• Identify technical requirements: technical requirements are design characteristics
that describe the customer requirements as expressed in the language of the designer
or engineer. Essentially, they are the “How” by which the company will respond to
the “What” – customer requirements (Evans and Lindsay, 2005).
• Develop a relationship matrix: a relationship matrix should be developed between
customer requirements and the technical requirements. Relations between the client
and design requirements are not always 1:1; there are complex relationships and
varying levels of strength. A single design requirement may have an influence on
several of the client’s requirements (Bernal et al., 2009).
• Add key competitor evaluation and key selling points: this step identifies importance
ratings for each customer requirement and evaluates competitor’s existing products
or services for each of them. Customer importance ratings represent the areas of
greatest interest and highest expectations as expressed by the customer. Competitive
evaluation highlights the absolute strengths and weaknesses in competing products.
• Evaluate technical requirements of competitive products and services and develop
targets: this step is usually accomplished through intelligence gathering or product
 testing and then translated into measureable terms. These evaluations are Improving
compared with the competitive evaluation of customer requirements and technical software quality
requirements. If a competing product is found to best satisfy a customer using Six Sigma
requirement but the evaluation of the related technical requirements indicates
otherwise, then either the measures used are faulty or else the product has an image DMAIC
difference, which affects customer perceptions. On the basis of customer
importance ratings and existing product strengths and weaknesses, targets for 853
each technical requirement are set.
• Selecting requirements to be deployed in the remainder of the process:
the technical requirements that have a strong relationship to customer needs,
have poor competitive performance, or are strong selling points are identified
during this step. These characteristics have the highest priority and need to be
“deployed” throughout the remainder of the design and development
process to maintain a responsiveness to the voice customer (Evans and
Downloaded by Concordia University At 16:48 13 November 2017 (PT)

Lindsay, 2005).
3.2.5 Control phase. The next step for “RK” company is to start implementing the suggested
improvements and measure the improvement performance in order to take the necessary
actions that will ensure that the process is under control. Some of the metrics that “RK”
company can use are:
• Tracking WIP: to do this, we need a cumulative flow diagram that shows quantities
of WIP at each stage in the system. If the Kanban system is flowing correctly, the
bands on the chart should be smooth and their height should be stable.
• Lead time: lead time can indicate how predictably the organization delivers. If an item
was expedited, how quickly did it get from the order into production? If it was of
standard class, was it delivered within the target time?
• By measuring the average time of how long items take until items reach production
from the time they get approved, can help measure how predictably “RK” company
delivers. It is expected that the average lead time is similar in each cycle.
• Throughput: throughput should be reported as the number of items or some
indication of their value that were delivered in a given time period, such as one
month. Throughput should be reported as a trend over time.
• Initial quality: defects represent opportunity cost and affect the lead time and
throughput of the Kanban system. It makes sense to report the number of escaped
defects as a percentage against the total WIP and throughput. Overtime, we want to
see the defect rate fall to close to zero.
• Defect removal efficiency: it indicates the percentage of software errors found before
software release.
It is defined as DRE ¼ E/(E+D)
E is the number of errors found before delivery of the software to the end user.
D is the number of defects found after delivery.
As D increases, DRE decreases (i.e. becomes a smaller and smaller fraction) (Jayanthi and
Florence, 2013).
Ideally “RK” company wants the defect removal efficiency to be 1, which means
that there are no defects found after delivery. However, achieving zero defects after
delivery is nearly impossible. For this, it is necessary for the test team to find as many
bugs as possible before delivery and not to drop the measure of defect removal efficiency
below 0.95.
 BPMJ 4. Conclusions and future works
23,4 In this paper, we addressed the problem of software quality management and proposed a
Six Sigma DMAIC-based approach. A case study is conducted using the proposed approach
at a Canadian organization called “RK” company. The CTQ aspects are identified,
production bugs classified and measured, the causes of the large number of production bugs
were specified leading to different improvement suggestions.
854 However, there are multiple approaches to applying Six Sigma in order to improve the
quality of software products. For example, Redzic and Baik (2006) created a team charter to
identity the problem, measured the process and established its baseline based on historical
phase containment effectiveness data. The improvement suggestions were based on the
conducted analysis such as the use of Monte Carlo simulation to account for risk and
uncertainty as well as a cost benefit analysis. On the other hand, Jayaraman et al. (2013) also
used a project charter to define the business problem, presented an MIS report to measure
the number of customer defects compared to another MIS report to measure the defects
Downloaded by Concordia University At 16:48 13 November 2017 (PT)

found during the development life cycle of the project. Cause and effect diagram, hypothesis
testing and Pareto chart were used to identify that a higher number of bugs were reported
by the customers during acceptance testing compared to those that were identified during
the software development life cycle. Conversely, our approach focused mainly on the large
number of bugs reported by customers after the application code was released to clients and
focused on reducing this number in order to diminish its impact.
In the software development industry, there is always need for improvement. Recent
studies show that Agile and Lean strategies are more effective than traditional strategies on
average. However, the success rate for software projects is still low compared to other
industries. There is still work needed on finding ways to reduce cycle times especially when
implementing large features that would increase the time needed for development and testing,
thus leading to a reduced predictability for project release dates. Also, software companies still
face problems to find the proper tools and methods that would help facilitate the design phase
especially for features with complicated implementation logic and the necessity to keep the
software product easy to use at the same time. This causes the development team to go back to
design phase in the middle of implementation and as a consequence, it leads to increased cycle
time and reduced predictability. Moreover, Agile and Lean strategies, require team members
to be experienced enough to be able to make decisions with the least cost of failure, which is
not the case in most software teams where team members expertise vary. Another possible
area that requires improvement is the ability to find reliable and easy to use automation test
tools that will not cause false positive test results that would require a lot of investigation and
maintenance from the test automation team that will also lead to an increased cycle time.

References
Abrahamsson, P., Salo, O., Ronkainen, J. and Warsta, J. (2002), “Agile software development methods
review and analysis”, available at: www.vtt.fi/inf/pdf/publications/2002/P478.pdf (accessed
February 17, 2015).
Anderson, D. (2010), Kanban Successful Evolutionary Change for Your Technology Business, Blue Hole
Press, Sequim, WA.
Awasthi, A. and Varman, R. (2003), “Investigating the influence of information technology on decision
making”, Journal of Advances in Management Research, Vol. 1 No. 1, pp. 74-87.
Basili, V.R. and Caldiera, G. (1995), “Improve software quailty by reusing knowledge and experience”,
Sloan Management Review, Vol. 37, pp. 55-64.
Bernal, L., Dornberger, U., Suvelza, A. and Byrnes, T. (2009), “Quality function deployment (QFD) for
services”, available at: www.vgu.edu.vn/fileadmin/pictures/studies/MBA/Handbook_QFD_
Services.pdf (accessed February 17, 2015).
 Chang, C.W., Wu, C.R. and Lin, H.L. (2006), “A simplified measurement scheme for software quality”, Improving
Journal of Information and Optimization Science, Vol. 27 No. 3, pp. 723-732. software quality
Chow, T. and Cao, D.B. (2008), “A survey study of critical success factors in agile software projects”, using Six Sigma
The Journal of Systems and Software, Vol. 81 No. 6, pp. 961-971.
Dawson, S.P. (1994), “Continuous improvement in action applying quality principles to software”,
DMAIC
Information Systems Management, Vol. 11 No. 1, pp. 31-39.
Evans, J.R. and Lindsay, W.M. (2005), An Introduction to Six Sigma & Process Improvement, 855
South-Western, OH.
Fontana, R.N., Fontana, I.M., Garbuio, P.A.D.R., Reinehr, S. and Malucelli, A. (2014), “Processes versus
people: how should agile software development maturity be defined?”, The Journal of Systems
and Software, Vol. 97 No. C, pp. 140-155.
Gack, G.A. and Robison, K. (2003), “Integrating improvement initiatives: connecting Six Sigma for
software, CMMI®, personal software process (PSP) (SM), and team software process (TSP)
(SM)”, Software Quality Profesional, Vol. 5 No. 4, pp. 5-13.
Downloaded by Concordia University At 16:48 13 November 2017 (PT)

Galin, D. (2004), Software Quality Assurance from Theory to Implementation, Pearson Education.
Hossain, A., Kashem, M. and Sultana, S. (2013), “Enhancing software quality using agile techniques”,
IOSR Journal of Computer Engineering, Vol. 10 No. 2, pp. 87-93.
Hung, H.C. and Sung, M.H. (2011), “Applying Six Sigma to manufacturing process in the food industry
to reduce quality cost”, Scientific Research and Essays, Vol. 6 No. 3, pp. 580-591.
IEEE (1991), “IEEE std 610.12-1990 – IEEE standard glossary of software engineering terminology”,
The Institute of Electrical and Electronics Engineers, IEEE Software Engineering Standards
Collection, New York, NY, February.
Issac, G., Rajendran, C. and Anantharaman, R.N. (2010), “Determinants of software quality: customer’s
perspective”, Total Quality Management & Business Excellence, Vol. 14 No. 9, pp. 1053-1070.
Jayanthi, R. and Florence, M.L. (2013), “A study on software metrics and phase based defect removal
pattern technique for project management”, International Journal of Soft Computing and
Engineering, Vol. 3 No. 4, pp. 151-155.
Jayaraman, P., Kannabiran, K. and Kumar, V. (2013), “A Six Sigma approach for software process
improvement and its implementation”, International Journal of Mining, Metallurgy & Mechanical
Engineering, Vol. 1 No. 3.
Kabir, E., Boby, M.I. and Lutfi, M. (2013), “Productivity improvement by using Six Sigma”, Engineering
and Technology, Vol. 3 No. 12, pp. 1056-1084.
Moe, N.B., Aurum, A. and Dybå, T. (2012), “Challenges of shared decision-making: a multiple case
study of agile software development”, Information and Software Technology, Vol. 54 No. 8,
pp. 853-865.
Moe, N.B., Dingsøyr, T. and Dybå, T. (2009), “Overcoming barriers to self-management in software
teams”, IEEE Software, Vol. 26 No. 6, pp. 20-26.
Munassar, N.M.A. and Govardhan, A. (2010), “A comparison between five models of software
engineering”, International Journal of Computer Science, Vol. 7 No. 5, pp. 94-101.
Nanda, V. and Robinson, J.A. (2011), Six Sigma Software Quality Improvement: Success Stories from
Leaders in the High Tech Industry, McGraw-Hill, New York, NY.
Page, A., Johnston, K. and Rollison, B. (2009), How We Test Software at Microsoft, Microsoft Press,
Washington, DC, available at: http://keripo.com/static/academia/upenn/cis099/How-We-Test-
Software-at-Microsoft_Alan-Pa.pdf (accessed June 1, 2017).
Quyoum, A., Din Dar, M.U. and Quadri, S.M.K. (2010), “Improving software reliability using software
engineering approach – a review”, International Journal of Computer Applications, Vol. 10 No. 5,
pp. 41-47.
Redzic, C. and Baik, J. (2006), “Six Sigma approach in software quality improvement”, IEEE Fourth
International Conference on Software Engineering Research, Management and Applications,
August, pp. 396-406.
 BPMJ Royce, W.W. (1970), “Managing the development of large software systems: concepts and techniques”,
23,4 Proceedings of IEEE Wescon, August, pp. 1-9.
Schwaber, K. and Beedle, M. (2002), Agile Software Development with SCRUM, Prentice Hall, Upper
Saddle River, NJ.
Subramanian, G.H., Jiang, J.J. and Klein, G. (2007), “Software quality and IS project performance
improvements from software development process maturity and IS implementation strategies”,
The Journal of Systems and Software, Vol. 80 No. 4, pp. 616-627.
856
Further reading
Cao, L., Mohan, K., Xu, P. and Ramesh, B. (2009), “A framework for adapting agile development
methodologies”, European Journal of Information Systems, Vol. 18 No. 4, pp. 332-343.

Corresponding author
Downloaded by Concordia University At 16:48 13 November 2017 (PT)

Racha Karout can be contacted at: rak_20@hotmail.com

For instructions on how to order reprints of this article, please visit our website:
www.emeraldgrouppublishing.com/licensing/reprints.htm
Or contact us for further details: permissions@emeraldinsight.com

View publication stats