Welcome to Scribd, the world's digital library. Read, publish, and share books and documents. See more
Download
Standard view
Full view
of .
Save to My Library
Look up keyword
Like this
5Activity
0 of .
Results for:
No results containing your search query
P. 1
Outsourced Product Testing - Practical Measurement

Outsourced Product Testing - Practical Measurement

Ratings: (0)|Views: 2,116|Likes:
Published by Mindtree Ltd
The success of an outsourced product testing effort is a function of a number of factors, not all of which are easily measurable. Traditional metrics used for this have tended to focus on objective attributes such as 'productivity per tester' but have generally ignored more subjective factors such as 'quality of communication between client and vendor teams'. This paper describes a methodology used at Aztecsoft named 'Service Measurement Methodology' that seeks to address these shortcomings of traditional approaches. We outline limitations of traditional approaches and describe the 'Test Quality Index (TQI)', a metric that addresses many of these limitations.

http://www.aztecsoft.com/
The success of an outsourced product testing effort is a function of a number of factors, not all of which are easily measurable. Traditional metrics used for this have tended to focus on objective attributes such as 'productivity per tester' but have generally ignored more subjective factors such as 'quality of communication between client and vendor teams'. This paper describes a methodology used at Aztecsoft named 'Service Measurement Methodology' that seeks to address these shortcomings of traditional approaches. We outline limitations of traditional approaches and describe the 'Test Quality Index (TQI)', a metric that addresses many of these limitations.

http://www.aztecsoft.com/

More info:

Published by: Mindtree Ltd on Jun 26, 2008
Copyright:Attribution Non-commercial

Availability:

Read on Scribd mobile: iPhone, iPad and Android.
download as PDF, TXT or read online from Scribd
See more
See less

09/27/2010

pdf

text

original

 
 
Page
1 of 10
 
itest.aztecsoft.com
 
Practicalmeasurement of OutsourcedProduct Testing
 
Sanjay Kulkarni,Aztecsoft itest
 The information contained in this document is proprietary &confidential to Aztecsoft Limited.
Winner of the
TestingLeadership Award
at QAI Software TestingConference 2007
 
 
Page
2 of 10
 
itest.aztecsoft.com
ABSTRACT
The success of an outsourced product testing effort is a function of a number of factors, not all owhich are easily measurable. Traditional metrics used for this have tended to focus on objectiveattributes
such as “productivity per tester”
but have generally ignored more subjective factors such as
“quality of communication between client and vendor teams”.
This paper describes a methodologyused at Aztecsoft
named „Service Measurement Methodology‟
that seeks to address these shortcomingsof traditional approaches. We outline limitations of traditional approaches and describe
the “
TestQuality Index (TQI)
, a metric that addresses many of these limitations.Over the past few years we have collected large quantities of data to test the efficacy of the approachdescribed here. We conclude by presenting this data and comparing the efficacy of our approach with
that of traditional approaches such as “Customer Satisfaction Index”
.
INTRODUCTION
The last few years have seen an exponential growth in the level of testing services outsourced toIndia
1
. Simultaneously there has also been a drastic change in the
nature
of test services offered fromIndia. These trends are a result of a number of factors that include:
-
Lower cost
-
Closing of gap between the abilities of onshore and offshore capabilities
-
Faster release cycles as a result of round-the-clock product development achieved throughdistribution of work across multiple geographiesManaging this exponential growth in levels of business without diluting the quality of service is a majorchallenge that vendors in this space have had to grapple with. One prerequisite for even attempting toaddress this challenge is to be able to
measure
the quality of service. Indeed, it could be argued that
this ability may be the differentiator between companies that have “made it” versus those that have
fallen by the wayside.There have been a number of attempts at designing such metrics. Most of these have tended to focuson objective metrics such as:
-
Productivity per person day for activities such as test case design and/or test case execution
-
Number of defects identified during the test case review
-
Percentage of test coverageThese metrics have been useful in improving specific processes and in precisely quantifyingdeliverables but have been lacking in the ability to capture the quality of service delivered. In
addition, they fail to capture important “soft” attributes such as “quality
 
of communication”.
 
At Aztecsoft we designed “
Service Measurement
Methodology”, an assessment methodology that
attempts to factor in tangible as well as intangible observations about a project in attempting tomeasure the quality of service provided on the project. This methodology was developed over the pastcouple of years and is currently used extensively in Aztecsoft Independent Testing projects.Section
„Conceptualize the “Service Measurement Methodology”‟
describes how we went aboutdesigning this methodology, Section
„Design a tool to measure‟
enumerates the elements of thismethodology and describes artifacts making up this methodology in detail.
Section „Structure of TQI‟
explains actual structure of TQI. We conclude by looking at the methodology in action
section Case
 
 
Page
3 of 10
 
itest.aztecsoft.com
Study: Customer Response
contains a case study of how this methodology was applied on a particularproject.
CONCEPTUALIZING
THE “SERVICE MEASURE
MENT METHODOLOY
 
Our approach to designing a measurement standard for the quality of a testing project involved lookingclosely at all aspects of such projects. Right at the beginning of our exercise we went through thefollowing steps:1.
 
For each test service offered by us we listed out the factors that would help determinewhether a project belonging to that type of service was a success or a failure. The idea was to
later build a “
measurement methodology
” based on the project‟s performance along all these
dimensions.2.
 
Identify the desirable outcomes of independent test projects. Examples of this include levelsof team satisfaction, number of show-stopper bugs after the product being released to market)Based on this we had
designed the tool to implement „Service measurement methodology‟
 a.
 
Define key success factors to meet project objectivesb.
 
Design questions to reflect key success factorsc.
 
Presentations to stakeholders and obtain feedback
DESIGN A TOOL TO MEASURE
There was need of a tool to monitor project performance of test only projects. The details of ourapproach are:
a.
 
Define key success factors to meet project objectives
Defining key success factors ensures successful execution of projects. We have incorporated the keysuccess factors efficiently in the TQI framework. The test manager plays an important role inidentifying key success factors for the project. For example, in automation projects, understandingtest automation requirements and getting early confirmation on POC.We identified the key success factors for each of our testing service offering in order to designquestions.
b.
 
Design questions to reflect key success factors
This was a critical step while designing the TQI model. We had a debate on choosing the mechanism tocapture the performance of key success factors. O
ne is bivalence („Y‟ or „N‟) and
the other is multi-valence (range of 1-10). There are certain benefits of multi-valence such as flexibility in approach andavailability of intermediate performance as against the True or False results. For example, if we haveto check whether test case coverage is sufficient? It is very difficult to say coverage is sufficient as it issubjective.After a lot of deliberation, the structure took its final shape. After a general consent on bivalencestructure while designing standard test service, offerings like test strategy, test planning, test casedesign, test case execution, ad-hoc testing, and bug tracking mechanics were considered. Anotherimportant offering is to automate the test case to reduce the cost of testing iterations and to reduceredundant activities among the test team members.

Activity (5)

You've already reviewed this. Edit your review.
1 hundred reads
1 thousand reads
jamesjchou liked this
hyund123 liked this
Ahmad liked this

You're Reading a Free Preview

Download
/*********** DO NOT ALTER ANYTHING BELOW THIS LINE ! ************/ var s_code=s.t();if(s_code)document.write(s_code)//-->