You are on page 1of 7

Gathering Performance Requirements

1 Introduction

The Quality of Service attributes like performance, availability, reliability and supportability
are becoming an increasingly important measure of project success. Several studies have
indicated that the inability to meet the desired/required performance is one of the main reasons
for software deployment failures. A study conducted by Infosys Technologies Ltd. on several
poorly performing Internet applications indicated that more than 50% of the problems were
caused by a lack of clear performance goals when the software was designed and developed.
Further, root cause analysis revealed that these defects were caused by incomplete performance
specifications and poorly understood constraints under which the system operates. This has
been reiterated in Tom Gilb’s principle, "Projects without clear goals will not achieve their
goals clearly."
[1] So, what are the factors that hinder the process of a comprehensive performance
requirements gathering?
While there are hordes of techniques for functional requirements gathering and analysis, there
seems to be no industry-standard approach or technique for gathering performance
requirements. There are some notational models like RT-UML and Coleman’s Use Case
Template.
[2] That extends a simple use case with a field for representing nonfunctional requirements.
However, restricting this exercise to be purely based on the use case view misses the essence of
some important dimensions of performance such as fluctuations in the workload and the
constraints under which the system will operate. Thus, a shop-usable approach is not readily
available. The other factor that impedes the creation or adoption of a standardized approach is
the inherent variability in context of each software development. Some of these are driven by
the different software solution patterns, like client server application, Enterprise Application
Integration, Web-based, or real time; and their development scenario like re-engineering, new
development, maintenance, etc. In this article we describe a systematic approach for gathering
performance requirements for an Internet application, and illustrate it with a case-study of re-
engineering of an Internet Banking application from ASP to ASP.NET.

2 Problem Formulations

Consider the following statement that was extracted from one real-life project document that
laid out the performance objectives of the software system - "The ability to handle a large
workload with minimal processor time, and to increase the maximum achievable workload
utilizing standard hardware scaling techniques." There are several pitfalls in this description of
the required system performance that lead to ambiguities in its interpretation, and verification.
For example, it does not quantify the workload the system needs to meet, or the expected
processor cycles per unit of workload, or the maximum workload till the system is expected to
scale linearly. For the architects and designers, incomplete specification makes it difficult for
them to ascertain whether the architecture will meet the desired goals or analyze the tradeoffs
with respect to other quality of service requirements such as security, or usability that the
system is expected to meet. For the IT managers, it could manifest as uncertainties in timelines,
budget, available resources etc. The above requirements specification should be more precise
in terms of the workload to be supported (for example, number of concurrent users an d/ or
expected transaction throughput), the maximum acceptable processor utilizations and the
forecasted workload. These requirements could then be verified in the subsequent lifecycle
stages. For example, performance testing would validate the system’s adherence to these
targets for the operational workload. As mentioned by Keepence et al [3], any system
requirement should be Specific, Measurable, Attainable, Realizable and Traceable; he refers
this as the SMART criteria.

3 Problem Solution

The performance of a system is a function of its operational workload, the underlying hardware
and software infrastructure and the application’s persistent data volume. We choose to define
these as:
(a) Deployment View to capture the hardware and software environment, constraints and
service-level agreements for the deployment environment.
(b) Operational Workload View to capture the different types of workloads that the system is
subjected to, the pattern and intensity of the requests during the different workloads intervals,
the desired responsiveness and the growth projections.
(c) Persistent Data View to capture the current data size and the data growth over a period of
time. These views capture the key attributes from business and technological standpoint.
Though we define these views, it is difficult to generalize the set of parameters and the level of
granularity required to suit every situation. For example, the performance requirement
document for a software system designed to be used by tellers in a bank may need to capture
the users, and their think time. However, for an Internet online shopping site, the workload is
best quantified as the expected number of request arrivals per unit of time, and its arrival
pattern. In the following case study, we demonstrate the process of capturing performance
requirements using these three views for an Internet application that was being re-engineered.

4 Case Studies

The case study is of an Internet Banking application that was undergoing a major re-
engineering initiative due to certain business and technological drivers. The user interface and
communication layers of the current application that were based on Microsoft technologies
ASP, COM, VC++ and VBScript, was re-engineered to use Microsoft .NET framework. One
critical piece in this re-engineering exercise, was defining the performance objectives of the
new system in a manner that it can be verified in the downstream project lifecycle stages.
However, the only performance requirement that was quantified by the business at the start of
the project was that the re-engineered application should be as good as the existing application
in terms of its responsiveness.
Clearly, the situation required us to baseline the current system’s performance along with the
existing operational workload and data volumes. This could be used as a baseline to validate
the performance of the re-engineered system. The next section details the process adopted to
capture the several attributes of these three views mentioned in the previous section.
Figure 1: Performance Requirements Gathering

4.1 Gathering Data for Existing System

The exercise began by identifying an analyst as a single point of contact. The analyst was
responsible for organizing interviews with the relevant stakeholders, and ensuring that their
responses were provided on time. These stakeholders were developers of the application
development and maintenance team, business analysts, system and database administrators,
and program manager of the re-engineering initiative. Before kick starting the interviews, a
briefing session was organized to detail the purpose of the performance requirement gathering
exercise. This session helped obtaining relevant documentation about the existing system from
the stakeholders, it provided an opportunity to clarify stakeholders’ queries, concerns, and in
communicating the need for their commitment, in providing support throughput this exercise.
The next steps were to create and circulate questionnaires with the objective of gathering data
and information required for creating the three views described earlier. Fig. 1 illustrates the
relationship between the stakeholder profiles and the different views for which they can
provide their inputs.

4.1. A Deployment View

The existing system architecture document forms a good starting point for the creation of the
deployment view. However, most often this needs to be supplemented with a questionnaire to
the system administrators and IT infrastructure hosting and management team to capture more
details. Some sample questions extracted from this case study are given below:
What is the bandwidth of the Internet link between the end users and the Web infrastructure?
What is the bandwidth of network link (LAN) between the Web and application servers and
between the application servers and the database server?
Provide the configuration of each Web server, application server and the database server in
terms of:
(I) Number of processors
(II) Type of processor (processor speed)
(III) Memory capacity
(IV) Hard disk: disk size, number of disk controllers and RAID level.

Is there any infrastructure change/upgrade planned for reengineered application?


Are the vital server resources like processor utilization, memory utilization, network
utilization and resources utilization for key processes monitored on Web, application and
database servers? If yes, then can this data for the peak workload durations be provided
to us?
What are the maximum acceptable utilizations for the resources mentioned above?
Are the server logs like the Web server access logs available?
A snapshot from a sample deployment view is shown below:

Figure 2: Snapshot from sample Deployment View

4.1. B Operational Workload View


Operational Workload View captures the information related to the business transactions of the
application. The first step in the creation of this view is to identify all the critical business
transactions from the functional specification documentation, or with the help of the
development team. Further questionnaires are sent to the business stakeholders to get expected
response times, transaction volume projections for the future, and other known trends or
seasonality in the workload. However, the creation of this view will require analysis of
workload data spanning the over past few months and/ or several geographically distributed
users. We will describe the approach for this analysis in section 4.2. Some sample questions
extracted from this case study are given below:
List the different usage periods such as Heavy, Medium, Light and Very light load and identify
the time intervals during which they occur.
List the business transactions that occur during the above usage periods.
Provide the arrival rate (number of transactions/hour) for each business transaction in the
above periods
Provide any growth projections expected for each transaction for the growth periods (12, 18
months) for each of the above scenario.
What is the expected total number of users of the system?
What is the expected maximum number of logged-in users?
Other than the workload mentioned above, are there any other applications/transactions
that generate additional load on the servers?
List the business criticality of each business transaction.
Please provide the current or the expected average, maximum and 90-percentile of response
time for the above transactions.
What are the response time limits of the external systems on which the application is
dependant?
A snapshot from a sample operational workload view is shown below:

Figure 3 Snapshot from sample Operational Workload View


4.1. C Persistent Data View
This view captures the current data volumes and expected growth. It is very important to
understand the data retention policies in order to plan for purge or archival cycles. A
questionnaire was circulated to the DBAs to get the information on the database size and
organization (both physical and logical). Some sample questions extracted from this exercise
are given below:
What are the main data entities that are used in the application (this may correspond
directly to a database table or a set of related tables)?
What is the current database size?
What is the size of some of the largest tables in the database?
What is the data archival and purge frequency?
What is the expected database growth in the next few years?
A snapshot from a sample Persistent Data View is shown below:

Figure 4 Snapshot from a sample Persistent Data View


The responses to the questionnaire were reviewed and taken up for further analysis. However,
it is important to realize that stakeholders can answer not all the questions. In the case study,
some of the important inputs that could not be provided by the stakeholders were: Arrival
pattern of requests and the percentile distribution of arrival rate for different workloads.
Operational workload mix determined by the users' navigation pattern in the site. Current and
expected response times of the transactions in-terms of average, maximum and 90th percentile
values. The next section details the techniques we used to capture this missing information.

4.2 Analysis Using Statistical and Analytical Techniques

In order to understand the arrival pattern and the percentile distribution of workload and the
workload mix we used existing historic data from Web server access logs of the existing
application. The log files for the representative time periods were analyzed to get a percentile
distribution table of the different arrival rate to the site. For more details on the analysis
technique please refer to [4]. Further, the mix and the intensity of the different components in
the workload can also be determined. In the case study, the transaction mix was found to be
almost constant for each hour of the day. Using the business workload forecast and
contingency factor, the growth factors for two different growth rates spanning twenty-four
months were identified.
The request execution time for the various requests recorded in the access logs gives an
indication of the current processing time of the various transactions. These numbers were
recorded as the baseline for comparison against the re-engineered application's transaction.
Now, with all the required information, it was time to document and communicate the results
of the analysis. The results were presented in a manner that was easy to understand because of
several graphs, and charts. For example graphs could be used to represent the arrival rate of the
top 10 transactions, transaction mixes, and the growth rate over the period of next two years
etc. Sample data for the growth projection is given in the table below.

Figure 5: A snapshot from sample growth projections data


5 Conclusions
Capturing the performance goals in a systematic manner is a pre-requisite for building
performance into an application right from the start of application development. Unfortunately,
lack of a well-defined approach has made it a difficult proposition for many requirements
analysts. In this article, we discussed a shop-usable approach and illustrated it with the case
study of re-engineering of an Internet banking application.
The performance requirements document produced in the case study, with the help of the three
views, provided specific and measurable performance requirements. Some of the ways in
which this helped in subsequent lifecycle stages were as follows:
We were able to establish the current capacity baseline by correlating the system resource
utilization from the Deployment View and arrival rate of the business transactions from the
Operational Workload View. It provided us the ability to set performance targets for the re-
engineered system, by base lining the current response times of the system. We had specific
throughput requirements for performance validation of the re-engineered system. We were able
to apply growth factors to workload, which helped us in determining the required capacity
forecast due to growth projections. The data volume requirement for the new system helped in
testing the system against close to real life data volumes and also for planning for database
organization accordingly. We could identify business transactions in the existing system that
had unacceptable response times, and re-examine the application design and code when they
were taken up for re-engineering. This approach proved to be quite effective in identifying the
required Quality of Service attributes and can be applied in any similar situation.