You are on page 1of 6

2017 3rd International Conference on Science in Information Technology (ICSITech)

Software Development Evaluation Process Using


CMMI-Dev on Limited Resources Company
I Made Sugi Ardana, Suharjito
Computer Science Department,
BINUS Graduate Program – Master of Computer Science, Bina Nusantara University,
Jakarta, Indonesia
sugiardana@gmail.com, suharjito@binus.edu

Abstract—The quality of products is very important for a development processes. But apparently there are still some
software development industry. The product quality is strongly projects that are not in accordance with the expectation of the
influenced by the software development process. Software company and the client. For example, there is a delay of
development company with limited resources is hard to improve implementation and there are many changes in the requirement
the quality of the software produced. The method to be taken is without being followed by increased prices resulting in
to improve the software development process to achieve maturity increased production costs and reduce the company's revenue.
level 2. In this study, CMMI-Dev (CMMI for Development) There are some guidelines to help software developer results in
version 1.3 with continuous representation is used as a reference. a process in accordance with needs. CMMI (Capability
SCAMPI-C with PST Tool was used to conduct the assessment of
Maturity Model Integration) is one of the guides that focus on
the 5 (five) process areas of CMMI level 2. Data were collected by
interviews and observations of project documentation and the
the software development process.
process of software development. Unapplied practice from To survive in the competition of making software, software
assesment then be analyzed using Ishikawa Analysis to find root development company creates software development strategies
of the problem refers to the SPF (Software Process Framework) by optimizing processes so that software development is not
and subsequently use Pareto Analysis to determine the too dependent on people, but rather on processes that have
improvements priority. The process which have already good are been standardized. If related to the maturity level of CMMI,
in the process area PP (Project Planning), CM (Configuration
then the level that wants to be achieved is a level 2 (Managed)
Management), and most of the processes in the area of PMC
where at this level has been ascertained that the process is
(Project Monitoring and Control). Software development
processes which are need to be improved in the process area with
planned and executed according to the policy (policy), the
priority PPQA (Process and Product Quality Assurance), REQM project team consists of people who are competent so that the
(Requirements Management), then PMC (Project Monitoring results can be measured, involving relevant stakeholders, the
and Control). The prior improvement from the SPF category project is monitored, controlled, reviewed, and evaluated, as
were in the order of procedure category and then process well as job status can be determined at any time [2, 4].
category. In this study, CMMI will be used to evaluate the software
development process in limited resources company. From the
Keywords—Software Process Framework; CMMI-Dev;
SCAMPI; Ishikawa Analysis; Pareto Analysis
results of the evaluation can be suggested any
recommendations regarding the process that needs to be
enhanced and which processes are already good and should be
I. INTRODUCTION maintained.
As the time goes by, the software development will
continue to evolve as the complexity of feature additions, II. RELATED WORKS
change business processes, error recovery, and application of
new technologies. Thus, the high quality of the software A. Software Development
become a vital need [1]. To get a high-quality software, a good
Software development is a series of activities aimed to
process quality of making software is required. Software
produce a software product that suits the requirement. There
Engineering Institute (SEI) said that the quality of a system or
are four basic activities that are common to the development of
product determined by the quality of the process and the
software, namely, software specification, software
maintenance [2]. The most important thing that affect the
development, software validation and software evolution [5].
success or failure of software development is the process of
For different type of systems require different development
making the software [3].
processes. Increasing the hardware performance resulted the
Software development industry that has limited resources need for software development to be increasingly large and
continuously strives to improve the quality and delivery time of complex.
software being made. This is done by improving software

978-1-5090-5864-8/17/$31.00 ©2017 IEEE 319


2017 3rd International Conference on Science in Information Technology (ICSITech)

By using an informal approach to face an increasingly


complex development, the development of software gets
increasingly ineffective. This resulted in greater costs, more
time, as well as lower quality. To overcome the ineffectiveness,
of that sparked an idea to do software development with
engineering approach, resulting a more efficient, standardized
and scalable software development.

B. SDLC (System Development Life Cycle)


SDLC is a cyclical process of system development. The
entire process of system development is done through many
steps from the beginning until the formation of the product
through the analysis, design, implementation and maintenance.
Along with the development of software engineering, many Fig. 2. Operational Framework.
models have been developed to assist the software
development process, such as the waterfall, spiral, prototyping, D. CMMI (Capability Maturity Model Integration)
XP, and RAD. Although many models will evolve, but it
CMMI published by the SEI (Software Engineering
generally follows the basic stages of the SDLC in waterfall
Institute) of Carnegie Mellon University, is a model approach
development model, which consists of the analysis and
in the assessment of the level of maturity and ability of an
definition of requirements, system and software design,
organization [2]. CMMI is based on three concepts: Process
implementation and unit testing, integration and system testing,
Area (PA), goal, and practice. There are two types of goal,
operation and maintenance [5] as in Fig. 1.
which are Specific Goals (SG) and the Generic Goal (GG).
There are two types of practice, which are Specific Practices
(SP) and the Generic Practice (GP). Specific Goals are a series
of activities or goals of the process area. Specific Practice is the
steps that are used to achieve goals [2]. Achievements and
fulfillment of the SG on the PA is very influential in the
maturity level achieved by the organization. Each PA will
consist of several SG and SG will consist of several SP [2].

E. CMMI-Dev (CMMI for Development)


CMMI-Dev is a model framework issued by the SEI
(Software Engineering Institute) of Carnegie Mellon University
in late 2001 and published in August 2006 to replace a similar
Fig. 1. Basic Stages of SDLC. concept that is CMM that has been used for the assessment
process since the 1990s. CMMI-Dev was made to avoid the use
of various CMM models separately. Therefore, the model that
C. Software Development Process Framework has been integrated in the CMMI for Development is the
To obtain good results when developing software, the SEI codification of CMM models [2]. CMMI-Dev consists of 22
issued an operational framework that refers to the Capability Process Areas which are: CAR (Causal Analysis and
Maturity Model, called SPF (Software Process Framework). Resolution), CM (Configuration Management), DAR (Decision
The SPF stated that the framework forms the link between the Analysis and Resolution), IPM (Integrated Project
elements of the operation, namely policy, standard, process, Management), MA (Measurement and Analysis), OPD
procedure, training, tools [6]. Framework that is formed will (Organizational Process Definition), OPM (Organizational
be used to define and design the information, where it can be Performance Management), OPF (Organizational Process
used to help establish the necessary documents. Policy is used Focus), OPP (Organizational Process Performance), OT
as a rule directing, guiding, or limit the operation. Standard is (Organizational Training), PI (Product Integration), PMC
to provide operational definitions or criteria for acceptance of (Project Monitoring and Control), PP (Project Planning),
the product or process. Process is to describe what is done in PPQA (Process and Product Quality Assurance), QPM
the operation in accordance with the policy and standards. (Quantitative Project Management), RD (Requirements
Procedure describing instruction steps in implementing a Development), REQM (Requirements Management), RSKM
process. Training is to give individuals a knowledge and skills, (Risk Management), SAM (Supplier Agreement Management),
including training for the organization's policies, standards, TS (Technical Solution), VAL (Validation), VER
processes, procedures, and tools. Tool is to provide the (Verification) [2].
required assistance in supporting policy, standards, processes,
There are four categories of process area which are: Process
procedures, and training. The relation of these elements, in
Management, Project Management, Engineering, and Support.
establishing an operational framework is shown in Fig. 2.
The relationship between process areas, categories, and
maturity level can be seen in Table I [2]. In the organizations
with maturity level 2, the job status can be known by

320
2017 3rd International Conference on Science in Information Technology (ICSITech)

management at certain points, for example, on major milestone TABLE II. COMPARISON OF SCAMPI CLASS
(major milestones) and on completion of a major task (major Class A Class B Class C
tasks) [2]. Evaluation team size 3–4 1–2
8 – 10
Evaluation time 10 days 3-4 days 1-2 days
TABLE I. PROCESSS AREA, CATEGORY, AND MATURITY LEVEL Min data source 3 2 1
Process Maturity Reliability and success High Medium Low
Category
Area Level Effort and cost needed High Medium Low
CAR Support 5 Direct interview Yes No
CM Support 2 Yes
SCAMPI A B C
DAR Support 3
IPM Project Management 3
MA Support 2
OPD Process Management 3 III. METHODOLOGY
OPF Process Management 3
OPM Process Management 3 A. Research Object
OPP Process Management 5 Based on the comparison table class of SCAMPI, for
OT Process Management 4
PI Process Management 3
SCAMPI C, the minimum number of data sources is 1 (one)
PMC Engineering 3 [1]. In this study the data source will be taken from two (2)
PP Project Management 2 software development project that has been completed.
PPQA Project Management 2
QPM Support 2 To select the projects that will be evaluated, which
RD Project Management 4 represents a typical software development project of the
REQM Engineering 3 enterprise, interviews will be conducted the Director of the
RSKM Project Management 2 company.
SAM Project Management 3
TS Project Management 2
VAL Engineering 3
B. Framework of Thinking
VER Engineering 3 Based on a literature review, in order to increase the
success of the project development, we should improve the
process used. Increasing development process can refer to the
F. SCAMPI best practices of CMMI-Dev.
SCAMPI stands for Standard CMMI Appraisal Method for Operations of the software development refers to the
Process Improvement. SCAMPI is composed of three classes software development process framework. Framework is
namely class A, class B, and class C, which is distinguished by already adapted to the implementation of the process areas of
the level of accuracy and the effort produced. SCAMPI-A is the CMMI. To determine which parts need to be improved, we
the most rigorous method and the only method that can need a method to measure it.
generate value (score) and judgement necessary to lead a
certified appraisal. Usually costs incurred for scampi A pretty To perform these measurements, will be using SCAMPI
big. SCAMPI-B is a method that is less formal than the method. In this research, the measurement is only to determine
SCAMPI-A, the activities are fewer than SCAMPI-A. By which parts need to be improved and not to measure the score
applying the SCAMPI-B, organization can predict the results of maturity level. As explained respectively, then SCAMPI
obtained when doing SCAMPI-A. This method does not need a class C to be chosen [7].
certified appraisal lead. SCAMPI-C is shorter, flexible, and less Based on the results of these measurements, it determined
expensive with SCAMPI SCAMPI-A and-B. Decision from the weaknesses and what improvements need to be done in
SCAMPI-C is usually used to measure the readiness of the accordance with the reference CMMI-Dev and SPF. Pareto
organization before applying CMMI. By doing SCAMPI analysis then will be used to make the priority of improvements
appraisal-C, it can easily get a gap of a process that has been [10].
carried out by the organization in comparison with the best
practices of the CMMI. Scope of SCAMPI-C can also be C. Research Stage
customized with the objective appraisal [7, 8, 9].
Stages of research in Fig. 3, started by defining the issues
For SCAMPI class C, the need to do the appraisal reduces that will be examined along with the boundary and scope. Then
the amount of effort and cost, but it reduces the accuracy and determine the objective and benefits of the research. After that
reliability as well but the accuracy and reliability are also doing study literature and taking case to be investigated.
lower. SCAMPI-C even allows reducing the number of
projects that will be investigated to zero and just do the Data collection is done after preparing the data collection
standard documented appraisal process, not the implementation instruments. With reference to the CMMI-Dev level 2, there
of the project [7]. The comparison between the three SCAMPI are 7 Process Areas had to be evaluated. But in this study only
classes can be seen in Table II. 5 Process Area from collected data will be examined, which
relates directly to the software development process, namely
the PP (Project Planning) which consists of a total of 3 Specific
Goal and 14 Specific Practice, PMC (Project Monitoring and

321
2017 3rd International Conference on Science in Information Technology (ICSITech)

Control) , consists of a total of 2 Specific Goal and 10 Specific IV. RESULT AND DISCUSSION
Practice, REQM (Requirement Management), consists of a
total of 1 Specific Goal and 5 Specific Practice, CM A. Data Collection
(Configuration Management), consisting of a total of 3 Specific Interviews were conducted for the selection of projects that
Goal and 7 Specific Practice, PPQA (Process and Product will be evaluated. Based on the results of interviews regarding
Quality Assurance), consists of a total of 24 Specific Specific the selection of projects to be evaluated, then taken software
Goal and Practice. development projects for a group insurance in the insurance
After sufficient data has available, then conduct CMMI company ABC and insurance company XYZ. The data being
assessment by performing data processing of the results of collected in the form of project documents and observations
these observations by using PST Tool. From this assessment results on the process of software development on these
will be obtained CMMI process that appropriate and the projects.
process that needs to be improved. For the processes that need 1) Profile of Software Development Company
to be improved, will be analyzed using Ishikawa diagrams and Software development company being used in this study
refer to the SPF to find the root of the problem. To determine specializes in making software for insurance company. Initially
the priority of process improvements, will be used Pareto the software development process is highly dependent on the
diagram, with reference to the SPF. Finally, establish employee so that when there are employees who resigned, the
recommendations based on the results of data processing as a company desperate. Likewise, the status of the project, known
whole. only by employees who manage the project. Learning from this
experience, the company started making policies to planning,
Start reporting and monitoring of the project. Thus, the development
of a project can be seen at regular intervals, including when
there are issues or risks that must be escalated to management
Research Problem CMMI Assessment Assesment level.
Result
The management of the company hope the future of
Research Objective
Ishikawa Analysis Assesment software development is not too dependent on people, but
Result rather on processes that have been standardized. By referencing
Study Literature to the maturity level CMMI, then the level that want to be
Pareto Analysis Assesment achieved is level 2 (Managed). At this level has been
Result
Setup Data ascertained that the process is planned and executed according
Collection
Instrument Establish
to the policy, the project team consists of people who are
Recommendation competent so that results can be measured, involving relevant
Data Collection stakeholders, the project is monitored, controlled, reviewed,
End and evaluated, as well as job status can be determined at any
time.
Currently, an employee could have a different role
Fig. 3. Research Stage. according to the phases of the project and its involvement in
the project. After the software project closing, an employee is
still engaged to support, because there are no support division,
D. Data Collection Method
which is a different team with software developers team. An
Data collection methods in this study consist of interviews, employee who became a systems analyst in a project could also
observation of the software development process, and analysis be a programmer in another project. Likewise, a system analyst
of the documents associated with software development in a phase of defining requirements, then will be a programmer
process at software development company with limited when development phase and to provide guidance to the user
resources. during implementation phase. This resulted in limited resources
working on a project.
E. Data Processing Method
2) Software Development Project for Insurance Group in
Data obtained from interviews, observation and document
PT. ABC
analysis will be used as a reference to perform data analysis.
PT. ABC previously had experienced failure in
The analysis will be done based on the procedures used to
implementing insurance software group from the previous
carry out an appraisal of the process area by using the
vendor. This causes a very high in overseeing the work of
SCAMPI-C method. By doing the assessment will be obtained
vendors. Software being created integrating the 12 departments
which practice is in a good state and practice that need to be
that conduct insurance group operations.
improved. For practice that needs to be improved will look for
the root of the problem by using the Ishikawa Analysis. Often each department which is the owner of each module
Furthermore, for the prior of improvements will be analyzed have diametrically opposed interests. Sometimes the
using Pareto analysis. Finally, will be made recommendations requirements that have been approved by a department could
for improvement. be changed when discussion on other related modules from
other departments. Replacement of user in each department as

322
2017 3rd International Conference on Science in Information Technology (ICSITech)

well be the next obstacle. If one user resigned and his TABLE IV. FINDING ON EACH SPF CATEGORY
replacement had worked at another insurance company before,
usually brings flow and business processes from the previous SPF Category Number of Findings
company. Procedures 10
3) Software Development Project for Insurance Group in Processes 8
PT. XYZ Policies 6
PT. XYZ also had a bad experience with a previous vendor, Standards 1
where the previous software cannot accommodate the needs of
the business of PT. XYZ. Besides, due to vendor’s dispersion,
a lack of adequate support arises in the event of a software D. Results of Pareto Analysis
problem. Conflict of interests between departments are also
Based on the analysis Ishikawa for each category of SPF,
faced by the software development project in PT. XYZ
the number of finding for each category can be summarized to
although not as hard as in PT. ABC, as previously it has been
be used for pareto analysis based on SPF category. The finding
using desktop software with integrated database. At the time of
is sorted from the most to the least finding, then each category
software development at PT. XYZ, the company had an
is calculated of count, cumulative and cumulative percentage
integrated web-based software, so it does not create from the
finding and can be seen as in Table V.
scratch. Defining the software requirement has also become a
bit easier. It done by gap analysis between existing software
and user needs. TABLE V. PARETO ANALYSIS BY CATEGORY SPF

Number of
B. Results of Process Area Assessment using SCAMPI-C SPF Category
Findings
Cumulative Cumulative %
To assess the process area using PST Tool from Dr. Procedures 10 10 40
Kneuper, artefacts on selected projects is used to replenish the Proocesses 8 18 72
tools. In this study, the artefacts are available in the form of Policies 6 24 96
project documents. For process area with its practice that no Standards
documents as object evidence, the practice has not been 1 25 100
successfully applied to the project and will be found with
yellow background. If it is not yet implemented, will be found Based on data in Table V then can be made Pareto diagram
with red background. All finding will be counted for each as shown in Fig. 4.
process area. Number of finding for each process area can be
seen in Table III. The most finding is in the PPQA (11
findings) and the least finding in the CM (2 findings).

TABLE III. FINDING ON EACH PROCESS AREA

Number of
Process Area
Findings
Process and Product Quality Assurance (PPQA) 11
Requirement Management (REQM) 6
Project Monitoring and Control(PMC) 3
Project Planning (PP) 3
Configuration Management (CM) 2 Fig. 4. Pareto Diagram By SPF Category.

Based on the Pareto diagram in Fig. 4, the priority of


C. Results of Ishikawa Analysis process improvement by SPF category is procedures category
Using Ishikawa Analysis for each finding that was and then followed by process category.
associated with the SPF, each finding is analyzed whether in
Based on the Ishikawa analysis for each process area, the
the category of SPF Procedures, Processes, Policies, or
number of finding for each process area can be summarized to
Standards. All finding will be counted for each category.
be used for pareto analysis based on process area. The finding
The number of finding for each category are summarized in is sorted from the most to the least finding, then each process
Table IV. The most findings are in the procedures category (10 area is calculated of count, cumulative and cumulative
findings) and the least finding in the standards category (1 percentage finding and can be seen as in Table VI.
finding).

323
2017 3rd International Conference on Science in Information Technology (ICSITech)

TABLE VI. PARETO ANALYSIS BY PROCESS AREA V. CONCLUSIONS


Number After evaluation of the software development process at the
Cumulative
Process Area of Cumulative
% company with limited resources using SCAMPI-C based on
Findings
Process and Product 11 11 44 CMMI-Dev version 1.3 with level 2 and analyzed with
Quality Assurance Ishikawa analysis and Pareto analysis, it can be concluded that
(PPQA) some software development process already good and some
Requirement 6 17 68 need to be improved. Software development processes which
Management (REQM) already good were process contained in the area of PP (Project
Project Monitoring and 3 20 80
Control(PMC)
Planning), CM (Configuration Management), and PMC
Project Planning (PP) 3 23 92 (Project Monitoring and Control).
Configuration 2 25 100 Software development process which need to be improved
Management (CM) were processes contained in the area of Process and PPQA
(Product Quality Assurance), REQM (Requirements
Based on data in Table VI then can be made pareto diagram Management), and a small portion in the area of the PMC
as shown in Fig. 5. (Project Monitoring and Control).
Priority of software development process improvement is
recommended using the following order. Start with establish
procedures of recording module testing, procedures of testing
results sign off, procedure of testing planning. Then conduct
training on how to perform QA, using best practice of
experience accomplished. Then establish procedure of defining
software requirement, procedure of defining software
requirements planning, procedure of definition software
requirements evaluation. Then conduct training on software
requirements definition, bi-direction software requirements
tracking. The last is establish procedures of project monitoring
and controlling and conduct training on project monitoring and
controlling.
Fig. 5. Pareto Diagram By Process Area.

Based on the Pareto diagram in Fig. 4, the priority of REFERENCES


process improvement by process area sequentially are PPQA [1] W. Widodo, "Evaluasi Proses Pengembangan Perangkat Lunak pada
(Process and Product Quality Assurance), REQM Virtual Team Development Menggunakan CMMI Versi 1.3," Jurnal
(Requirement Management), and PMC (Project Monitoring Informatika, vol. 10, no. 1, pp. 1140-1148, Jan 2016.
and Control). [2] CMMI Product Team, "CMMI for Development Version 1.3," Carnegie
Mellon University, Hanscom, 2010.
E. Recommendation [3] J.-C. Liou, "On Improving CMMI in an Immature World of Software
Development," Journal of Information Science and Engineering, 2011.
By combining pareto analysis by SPF category and pareto [4] Kautsarina, "Penilaian Tingkat Kematangan Tiga Proses Area Level 2
analysis by process area, can be establish recommendation on CMMI Versi 1.2 pada Small Independent Software Vendor di Indonesia
process improvements that can be done in the company. (Studi Kasus: Inovasia)," Widyariset, vol. 14, no. 3, pp. 665-674, 2011.
[5] I. Sommerville, Software Engineering Ninth Edition, Boston: Addison-
The first recommendation is to establish procedures of Wesley, 2011.
recording module testing, procedures of testing results signoff, [6] T.G. Oslon et.al, "A Software Process Framework for the SEI Capability
and procedure of testing planning. Then conduct training on Maturity Model: Repeatable Level," Software Engineering Institute,
how to perform QA, using best practice of experience Pittsburgh, 1993.
accomplished. After that establish procedure of defining [7] R. Kneuper, CMMI Inproving Software and System Developmen
software requirement, procedure of defining software Processing Using Capability Maturity Model Integration (CMMI-Dev).,
requirements planning, and procedure of definition software Santa Barbara: Rockynook, 2008.
requirements evaluation. [8] W. Hayes, G. Miluk, L. Ming and M. Glover, Handbook for Conducting
Standard CMMI Appraisal Method for Process Improvement (SCAMPI)
Next recommendation is to conduct training on software B and C Appraisals, Version 1.1, Hanscom: Carnegie Mellon University,
requirements definition, bi-direction software requirements 2015.
tracking. The last recomendation is establish procedures of [9] "PIID and SCAMPI Tool (PST)," 2016. [Online]. Available:
http://www.kneuper.de/English/PIID-SCAMPI-Tool/. [Accessed 12
project monitoring and controlling then conduct training on May 2016]
project monitoring and controlling.
[10] D. Haughey, "Pareto Analysis Step by Step," 2016. [Online]. Available:
https://www.projectsmart.co.uk/pdf/pareto-analysis-step-by-step.pdf.
[Accessed 01 August 2016].

324