PICMET 2009 Proceedings, August 2-6, Portland, Oregon USA © 2009 PICMET

An Approach to Characterize a Software Process

Markus Suula, Timo Mäkinen, Timo Varkoi
Tampere University of Technology, Pori, Finland

Abstract--Process improvement is a cyclic activity consisting

of several phases. According to the Quality Improvement
Paradigm, the first phase is characterization. This paper
presents a developed methodology for characterizing a software
process. The methods include software process assessment and
modeling. Using an assessment driven process modeling
methodology we can create a descriptive model that
characterizes the current process, and a prescriptive model
containing improvement suggestions. The selected processes are
reviewed and the process components are gathered according to
a software engineering process meta-model. The SPICE
assessment model is utilized in process reviews. In a case study
the methodology is applied to characterize the software process
of a small enterprise. The current process is defined and its
process capability and product quality are determined. The
enterprise’s process guide, the produced work products and the
projects workload are analyzed. This paper reports the
experiences of using the methodology and presents a way to
analyze the implications caused by the differences between the
existing process guidance and the actually implemented process.

I. INTRODUCTION
Figure 1. Process Architecture of an Iterative Software Process
It is widely accepted, that the quality in a software product
is largely determined by the quality of the process that is used
to develop and maintain it [10]. Process improvement means
understanding existing processes and changing these
processes to increase product quality and/or reduce costs and
development time [9]. In this paper, we discuss approaches to
achieve the understanding of processes. The discussion is
based on a case study of a small software enterprise.
Process is a collection of activities that takes one or more
kinds of input and creates an output that is of value to the
customer [10]. The activities can be grouped into workflows
like in a widely known software development methodology,
the Rational Unified Process (RUP) [4]. RUP is iterative and
incremental process divided into four phases: 1. inception, 2.
elaboration, 3. construction, 4. transition. Fig. 1 depicts the Figure 2. Process improvement cycle according to the Quality Improvement
architecture of a RUP-like imaginary software process and Paradigm [1]
how different activities spread in the process.
According to the Quality Improvement Paradigm (QIP) Process capability is a characterization of the ability of a
[1], process improvement is a cyclic activity consisting of six process to meet current or projected business goals [2]. It is
phases (see Fig. 2). The first phase is characterization for determined through a systematic assessment and analysis of
understanding the current status of processes. The Product selected processes within an organization against a target
Focused Software Process Improvement methodology capability. During a process assessment an organizational
(PROFES) [7] divides the characterization phase into four unit’s processes are evaluated against a process assessment
steps: 1. verify commitment, 2. identify product quality model, such as the International Standard ISO/IEC 15504-5
needs, 3. determine current product quality, and 4. determine [3] or the Capability Maturity Model integration (CMMI) [8].
current process capability. The process assessment models contain reference goals and
practices, and the result of an assessment denotes, how well
the operations of an organizational unit match to the goals
and practices described by an assessment model.

1103
 PICMET 2009 Proceedings, August 2-6, Portland, Oregon USA © 2009 PICMET
The context of our study is a small software enterprise
that undertook a project to build a monitoring and reporting
system for an industrial customer. Our aim was to experiment
an approach for Assessment Driven Process Modeling
(ADPM) [5], which we are developing to support the fourth
step of the Characterize phase in PROFES. The enterprise’s
interest was to get practical experiences of certain software
tools for process management, which they considered in the
first place for their own use, and in the second place as part of
their business. To fulfill both of the objectives, we planned a
quasi experiment in which ADPM and the software tools
were utilized.
During the study we noticed that the state of the software
project was not very strong. There were remarkable overruns,
and it seemed that, despite of the existence of standard
processes and the project plan, the project moved ahead on an
ad hoc basis. Although the outcome of assessment or process
modeling characterizes the current software practices of an
organizational unit, they do not describe how well the results
of the practices meet the business goals. An assessment as a
means to determine process capability as it has been defined
in ISO/IEC 15504 [2], does not imply that the business goals
are met.
In this paper we propose an approach to characterize
process capability according to its definition in the
International Standard, ISO/IEC 15504-1 [2]. The inspiration
for the illustration comes from the architecture of RUP (Fig.
1). The capability of current process of a small software
enterprise is characterized using the approach.
The second section introduces the research context
meaning the enterprise, its process guide and the studied
project. The third section describes the methods used in the
study. Results of the study are presented in the fourth section.
The fifth section summarizes this study.
II. RESEARCH CONTEXT
The software enterprise where the study was carried out
has a defined standard process called Standard Process Set
(SPS). The performed process was defined by studying one of
the enterprise’s projects. One of our team members also
worked in the studied project as a programmer so we could
get in-depth information about the project and the performed
process.
A. The Enterprise
The enterprise was founded in Finland in 1994. The wide
product repertoire included products for information
management systems and industrial solutions. The enterprise
became advanced IBM business partner in 1998 so it was
1104
natural that the products were based on IBM technologies.
Process improvement has always been part of the enterprise’s
strategy and Software Process Improvement (SPI) is based on
the ISO/IEC 15504 standard (SPICE) [3]. Their customer
projects follow the waterfall model. Nowadays the enterprise
has reduced its product repertoire and has concentrated in
production monitoring and reporting systems (PMRS) and
enterprise content management (ECM). The products are base
on IBM technologies e.g. DB2, DB2 Content Manager,
Websphere Portal and Lotus Domino. The enterprise has
expanded its market area to Scandinavia, the United
Kingdom and the Baltic countries. SPI activities are
considered to be crucial in order to gain growth in the
international market.
B. Standard Process Set
In the year 2004 a study [6] was carried out in the
enterprise to improve their processes towards the ISO/IEC
15504 capability level 2, where a process is performed and
managed. The planning of the study consisted of an analysis
of the latest process capability assessments conducted in
2001-2002. The main focus was on the management practices
so the management process group had the first priority for
improvement. Second priority was given to the customer-
supplier process group and third priority to the engineering
process group. Based on the assessment report from 2001, the
enterprise had recognized and listed ten individual software
processes. Those ten processes were used to create a standard
process for the enterprise. During the process improvement
study the number of processes was increased to nineteen to
complete the SPS.
The idea of the SPS is that it is customized to the needs of
a project. The nineteen processes of the SPS are divided into
five groups. SPS is a folder structure where each of the
processes has their own folder containing the process content.
Table 1 lists the SPS processes and the process related
content grouped into four groups: 1. Guidance, 2. Checklists,
3. Templates, 4. Misc. The group “Misc” consists of pictures,
tables and other miscellaneous content. Roles are in
alphabetical order and the abbreviations used are explained in
Table 2.
We can see from Table 1 how much SPS provides support
for the processes. Engineering process group has the most
documents (30) but the least documents per process - avg.
three (3) documents per process. It should be noticed that 40
per cent of the ENG documents belong to the TS-process and
that the test plan document template is the same for IT, FT
and ST processes. Project management is the most supported
process with sixteen (16) documents. It is clear that the
prioritization made when establishing the SPS came true.
 PICMET 2009 Proceedings, August 2-6, Portland, Oregon USA © 2009 PICMET

TABLE 1. THE COMPONENTS OF STANDARD PROCESS SET

Templates
Checklists
Guidance
Standard Process Set -component Roles

Misc.
Customer-Supplier process group CUS
SM – SALES AND MARKETING 3 1 5 8 CD, MD, S
DL – DELIVERY 0 1 2 0 MD, PM, Sec
CR – CUSTOMER REQUIREMENT SPECIFICATION 0 1 1 0 CD, D, PM
PS – PRODUCTION SERVICE 0 1 1 0 PM, SM
CUS sum 3 4 9 8
Engineering process group EG
FS – FUNCTIONAL SPECIFICATION 1 1 2 0 CD,D,PM
TS – TECHNICAL SPECIFICATION 3 2 6 1 CD,D,PM
SI – SOFTWARE IMPLEMENTATION 0 1 0 0 P, PM
MT – MODULE TESTING 0 1 1 0 D, P, PM
IT – INTEGRATION TESTING 0 1 1 0 D, P, PM
FT – FUNCTIONAL TESTING 0 1 2 0 D, PM
ST – SYSTEM TESTING 0 1 1 0 D, P, PM
RT – REQUIREMENT TESTING 0 1 0 0 P, PM
PL – PILOTING 0 1 0 0 P, PM, TE
MS – MAINTENANCE SERVICE 0 2 0 0 MD, PM, SM
EG sum 4 12 13 1
Support process group SUP
DN – DOCUMENTATION 4 1 3 2 DD
QS – QUALITY ASSURANCE 0 1 1 0 CD, MD, PM
CM – CONFIGURATION MANAGEMENT 0 1 1 0 CD, MD, PM
SUP sum 4 3 5 2
Management process group MA
PM – PROJECT MANAGEMENT 1 1 9 5 CD, MD, PM
Organization process group ORG
PE – PROCESS ESTABLISHMENT 3 1 1 2 MD
SPS sum 15 21 37 18

TABLE 2. ROLE ABBREVIATIONS

Abbreviation Role
CD Chief Designer
D Designer
DD Document Developer
MD Managing Director
P Programmer
PM Project Manager
S Sales
Sec Secretary
SM Service Manager
TE Test Engineer

C. The Project
In the studied project a client had ordered a production
monitoring and reporting system (PMRS). The PMRS was
designed to integrate with the client’s already existing
information system. The system was implemented with a new
tool that the enterprise had no experience with. Requirement
elicitation started in April 2006; client approved the customer
requirement specification in June 2006. Implementation
project began in the summer of 2007 and the system was
estimated to be released in September 2007.
The project was divided into ten phases which were
planned to be carried out according to the waterfall model. In
most of the phases a major document was to be created
(Table 3). In addition, a final report was also planned to be
written by the project manager in the end of the project. The
project team consisted of the following roles: project
manager, requirements engineer, chief designer, designer,
production environment designer, specialist in customer’s
industry, programmer, and tester. One person could have
multiple roles e.g. project manager was also a specialist,
requirements engineer and tester.
There were two major releases in the project: first in
October which was an implementation review, and the
second in November 2007 which was supposed to be the final
release. The client was not satisfied with either the first
release in October or the release in November. Changes were
made until October 2008 when client accepted the release
1.0.

1105
 PICMET 2009 Proceedings, August 2-6, Portland, Oregon USA © 2009 PICMET

TABLE 3. PROJECT PHASES, PHASE DOCUMENTS AND ACCOUNTABLES FOR CREATING THE DOCUMENTS
Project phase Document Accountable
(0. Requirement elicitation) Customer requirement specification Project manager,
Chief designer,
Designer
1. Project kick-off and planning Project plan Project manager
2. Analysis Functional specification Chief designer,
Project manager
3. Design Technical specification Chief designer, Designer
4. Implementation and unit testing - -
5. Integration testing in production environment Integration testing plan Chief designer, Designer
6. System testing in production environment System testing plan Project manager, Designer
7. Approval test by the supplier Approval test report Project manager
8. Training and deployment - -
9. Approval test by the client - -
10. Maintenance - -

It was stated in the project plan, that the project should
follow the SPS. It was declared that the project manager is
responsible for ensuring that the SPS processes are followed.
The documentation process was emphasized in the project
plan: every project phase and document should be reviewed
or inspected internally. The change management part of the
project plan stated, that depending on the change request’s
magnitude and impact to the system decisions were made
either internally (small changes that had minor affect on the
schedule or work load) or between the supplier and the client
(major changes that increase the work load significantly).
The project team recorded their work into the enterprise's
work hour tracking system by filling a form illustrated in Fig.
3. Process and task are selected from a static list which is the
same for all projects, i.e. the task is not project specific. The
description field can be used to specify what exactly was
done. The tracking system cannot be used to track the
progress of individual, project specific tasks.
III. METHODS
The study was carried out by taking part in the
enterprise’s activities. In this study the enterprise’s process
guide and the work products and workload of the sample
project were analyzed. Information about the sample project
was gathered in process information gathering workshops
participated by both the project team and the modeling team.
The chosen processes were reviewed and process components
were gathered according to the software process engineering
meta-model. The process components were tasks, roles, work
products and tools. Problems and notices were also gathered.
Earlier assessed processes were chosen to be reviewed
utilizing the SPICE-standard. Modeling was based on an
assessment driven process modeling methodology [5].
In this study the product was the project itself not the
system that the project produced. The product quality was
determined by comparing project team members work hour
data to the project plan’s work breakdown structure (WBS)
which was a Gantt chart. The work hour data was gathered
from the work hour tracking system. The work hour data was
transferred into Microsoft Excel which was used to create
diagrams. The diagrams were scaled and combined with an
open source software Paint.NET using the RUP process
architecture diagram (Fig. 1) as a template. Releases and
project schedule were added also to the diagram.

Figure 3. The Work Hour Form
Project specific tasks are kept in a spreadsheet task list
containing the following information about a task: state of the
task, target/module, task description, criticality and urgency.
The state of the task was color coded according to readiness
per cent in a way that red meant 0-25% readiness, orange 25-
50%, yellow 50-75% and green 75-100%. Criticality and
urgency were numeric values from zero to five.
Process Modeling
Preparation
In the start-up session we introduced the modelling team
and the ADPM process to the software enterprise and the
goals and phases of the ADPM process were described in
general. We proposed a set of processes and capability levels
to be reviewed which was approved by the enterprise’s
management. Next we scheduled the process review
workshops. The modeling team was also authored to access
the work products of the project

1106
 PICMET 2009 Proceedings, August 2-6, Portland, Oregon USA © 2009 PICMET
Execution
The performed process was determined by reviewing
selected processes of the project, namely ENG.1
Requirements Elicitation, ENG.4 Software Requirements
Analysis, ENG.5 Software Design, ENG.6 Software
Construction, ENG.8 Software Testing and MAN.3 Project
Management. The processes were selected according to the
previous assessments. Processes were reviewed in six three
hour workshops. The enterprise’s project team was invited to
every workshop but only the key persons in the process to be
reviewed were required to attend. There were two persons in
the modeling team: a chairman and a secretary. The chairman
led the conversation and drew preliminary models which
Figure 4. State Diagram of a Workshop
In the first workshop we reviewed the process ENG.6
Software Construction. The review started with a vast base
practice ENG.6.BP2 Develop Software Units which took the
whole three hours of the first workshop. In the second
workshop the review of the ENG.6 process was continued. In
the beginning the modeling team presented the models
created in the first workshop and the project team commented
them. Comments were written down and the models were
altered accordingly. In the second workshop base practices
ENG.6.BP1 Develop Unit Verification Procedures,
ENG.6.BP4 Verify Software Components and ENG.6.BP3
Ensure Consistency were reviewed. Processes ENG.4
Software Requirements Analysis and ENG.8 Software
Testing were in the agenda of the third workshop. Time run
out before the ENG.8 process was reviewed but the ENG.4
process was completed. In the fourth workshop process
ENG.1 Requirements Elicitation was reviewed. In the fifth
workshop the previously missed or superficially reviewed
base practices were revised thoroughly. Process attribute PA
1107
were projected with an overhead projector to the wall. The
models were drawn with OmniGraffle tool. The symbols used
in the models were similar to the Eclipse Process Framework
(EPF) symbols. The secretary wrote down notes about the
conversation.
The workshops followed the process presented in Fig. 4.
The agenda of each workshop consisted of two or three
SPICE processes, of which one was chosen in the beginning
of the workshop to be reviewed. The process review
proceeded with one base practice at a time. Each base
practices was divided into tasks. Then the work products
(inputs and outputs), roles, subtasks, tools, notices and
problems related to the tasks were gathered.
2.2 Work Product Management emphasizing ENG.6 process
was reviewed in the sixth workshops.
Finalization
Processes ENG. 5 Software Design, ENG.8 Software
Testing and MAN.3 Project Management were not reviewed
because of the lack of time, but the processes were discussed
during the reviews of the other processes. Project
management was talked about in every workshop but
especially in the sixth workshop. The modeling produced two
types of models: 1. descriptive models, 2. prescriptive
models. The descriptive models describe the performed
process of the studied project - how things were done. The
prescriptive models were derived from the descriptive models
by adding components that could solve the detected
problems. The components of the solution were primarily
based on the SPS, but in the case that SPS could not offer a
solution, the SPICE standard and other sources were used.
 PICMET 2009 Proceedings, August 2-6, Portland, Oregon USA © 2009 PICMET
IV. RESULTS
The implementation project under study took twelve
months more than planned and the work hours were exceeded
by 300 per cent. Fig. 5 shows the workload of the
implementation project grouped into seven categories: 1.
Requirements, 2. Analysis & Design, 3. Implementation, 4.
Testing, 5. Delivery, 6. Project Management, and 7. Others.
The group “Others” covers the hours from the following
processes: documentation, sales and marketing, and
administration. The studied project consists of two separate
subprojects which were the requirement elicitation project
and the implementation project. The requirement elicitation
project is not considered in Fig. 5.
Figure 5. Workload chart
1108
Based on Fig. 5 the system delivered in October 2007 was
almost untested. The amount of testing starts to increase after
the first release. After the release in February 2008 there are
two peaks. These peaks are so called implementation rushes.
Tasks piled up because the project team was not anymore
fully available for the project. The project was planned to
follow the waterfall model but it became an iterative project.
We discovered that the requirements changed a lot during
the project which explains multiple releases and why the
amount of project management increases after the project
schedule was exceeded. The customer requirement
specification (CR) was valid throughout the project. The
problem was in the functional specification (FS). The FS did
not define the ‘whats’ of the CR into ‘hows’ sufficiently, e.g.
there was a requirement which stated that the system must be
easy to use. This requirement was not clarified in the FS. The
customer based many requirement change requests to this
usability requirement. There were practically no analysis &
design after the first release even though there was a lot of
requirement changes and implementation. FS was not
updated according to the changes and it became obsolete fast.
SPS doesn’t offer a change management practice but it
requires constant maintenance of the documents.
A system test plan (ST) was created based on the first
version of the FS so also the ST became obsolete fast. All the
testing in the project was explorative and unguided.
According to Sommerville [9] in the waterfall approach the
project cost i.e. working hour distribution is 15%
specification, 25% design, 20% development, 40%
integration and testing, and in iterative development 10%
specification, 60% iterative development and 30% system
testing. In the studied project testing had ca. 10% proportion
(Fig. 6). The insufficient testing can be assumed to be one of
the key factors for large amount of implementation work. It is
said that 50% of the implementation work is rework for
fixing defects so in the studied project one fifth of the costs
are rework. There is no way to completely eliminate defects
but it is important to remember that it gets more expensive
the later the defect is detected.
Figure 6. Workload Distribution
 PICMET 2009 Proceedings, August 2-6, Portland, Oregon USA © 2009 PICMET
V. CONCLUSION AND FUTURE WORK
The study indicated that neither the process guide nor the
project plan was followed in the studied project. Many of the
problems in the studied project could have been avoided by
following the process guide. The studied project can be
considered failed because its workload was 300 per cent more
than planned and the deadline was exceeded by a year. The
biggest problems in the studied project were a constant flood
of changes and a lack of change management system. It was
discovered that the process guide needs to be updated, and
then utilized properly.
The results of this study were presented to the project
managers, chief designers and management of the enterprise.
The case study motivated the management to really invest
into SPI activities. The enterprise has now hired a part time
quality manager. The management and the project managers
found the produced workload chart (Fig. 5) informative and
useful. Project managers said that similar charts could help to
follow the progress of the project if charts that represent the
current status of the project were constantly available or
could be automatically generated. Project managers also
stated that it is a problem that they can’t follow the progress
of individual tasks. These emerged issues initiated a project
to improve the work hour tracking system in the enterprise.
1109
REFERENCES
[1] Basili, V.R., Caldiera, G., Rombach, H.D.; “The Experience Factory,”
Encyclopedia of Software Engineering, John Wiley & Sons, pp. 469-
476, 1994.
[2] ISO/IEC 15504-1, “Information Technology - Process Assessment -
Part 1: Concepts and vocabulary,” 2004.
[3] ISO/IEC 15504-5, “Information Technology - Process Assessment -
Part 5: An Exemplar Process Assessment Model,” 2006.
[4] Jacobson, I., Booch, G., Rumbaugh, J.; “The Unified Software
Development Process,” Addison-Wesley, 1998.
[5] Mäkinen, T., Varkoi, T.; “Assessment Driven Process Modeling for
Software,” Process Improvement. Proceedings of the PICMET'08
Conference, Cape Town, South Africa, 6 p, 27-31 July 2008.
[6] Nurkkala, R.; “Improving Project Management In a Small Software
Organization,” Master’s Thesis, Tampere University of Technology,
Department of Information Technology, 73 p, 2004.
[7] PROFES, “PROFES User Manual, Final Version,” Retrieved
20.10.2008 World Wide Web,
http://virtual.vtt.fi/virtual/proj1/profes/UserManual.html
[8] Software Engineering Institute, “CMMI for Development (CMMI-
DEV) Version 1.2 (CMU/SEI-2006-TR-008),” Software Engineering
Institute, Carnegie Mellon University, August 2006.
[9] Sommerville, I.; “Software Engineering. Seventh Edition,” Pearson
Education Limited, 2004.
[10] Zahran, S. (1998), Software Process Improvement. Pearson Education,
1998