Creating Sanity Amidst Test Methodology Madness

Right-sizing your test methodologies

CONFIDENTIAL: For limited circulation only

© 2011 MindTree Limited

Slide 1

Agenda

Rightsizing?

hen Identify the symptoms

hat Key aspects to right-size

ow MindTree Tools & Frameworks

Processes should work for you. You should not work for processes.
CONFIDENTIAL: For limited circulation only © 2011 MindTree Limited Slide 2

Poll Question #1

● Are your processes taking too much effort and time?
● Yes ● No ● Not able to measure

CONFIDENTIAL: For limited circulation only

© 2011 MindTree Limited

Slide 3

When to Rightsize?

hen

Identify the symptoms

• • • • •

Test processes taking too much effort and time? Metrics not helping you take decisions? Not able to justify the investments in testing? Test process not helping you identify the critical defects early? Difficulty in quantifying the quality in your test processes?

Acknowledge the need. Don’t wait until you loose your sleep.
CONFIDENTIAL: For limited circulation only © 2011 MindTree Limited Slide 4

What to Rightsize?

hat

Key aspects to right-size

• • • • •

Quality goals / objectives Software life cycle methodologies Engagement models Governance mechanism (metrics, reviews) Tools and frameworks

Tailor your processes for right fit.
CONFIDENTIAL: For limited circulation only © 2011 MindTree Limited Slide 5

Poll Question # 2

● Do you know what metrics should you be tracking for your testing
needs?

● Yes ● No ● I do, but not sure if they are exhaustive

CONFIDENTIAL: For limited circulation only

© 2011 MindTree Limited

Slide 6

Test Metrics Analysis and Decisions (TMAD) Model

Test Metrics Analysis and Decisions (TMAD) Model

ow MindTree Tools & Frameworks

Metrics Based Governance

• • •

• •

Business goals Quality goals Governance • Metrics, reviews, baselines Forecast • Risks, best practices Corrective & preventive actions Enablers

Statistical Models

Historical Data

Org / Industry Baselines

CONFIDENTIAL: For limited circulation only

© 2011 MindTree Limited

Slide 7

Test Metrics Analysis and Decisions (TMAD) Model
Metrics Analysis
Proposed Metrics Goal Associated Queries (RCA) Undesired Behaviour (Risks)

Defect Containment Effectiveness

1. Improve testing quality 2. Prevent defect leakage

1. Was the criteria to classify defect detection phase clear from a leakage perspective? 2. Are there clear entry and exit criteria between phases 3. Was the team's knowledge adequate to uncover leaked defects? 4. Was the scope of testing for each phase clear? 5. Do we have requirement traceability matrix?

1. Blame game 2. Manipulation of defect classification criteria

Metrics Decision Table
Decision Support (Scope) 1. 1. Improve test case review effectiveness 2. Improve process compliance 3. Improve product and test design knowledge 2. 3. 4. 5. Corrective Actions (feedback) Clear entry and exit criteria between phases Process adherence Effective knowledge management Update test suite Get test strategy reviewed by customers Raw Data Requirements Suggestions

1. Defects detected in test phase 2. Leaked defects

Do RCA for every release and identify preventive actions

CONFIDENTIAL: For limited circulation only

© 2011 MindTree Limited

Slide 8

Test Quality Index (TQI) Model

● Bivalent questionnaire; Y or N
responses

● “Productivity per tester” vs. ● Contextual metrics vs.

Identify areas of improvement

1

“quality of deployed resources and resource deployment”
Improve scores with each deliverable

comprehensive score; across 30 process areas

3

Customize the scored checklist

2

● Scoring based on self
assessment
Winner of the Testing Leadership Award at QAI Software Testing Conference
CONFIDENTIAL: For limited circulation only © 2011 MindTree Limited Slide 9

Poll Question # 3

● Do you have an integrated framework which helps you rightsize the
test processes and methodologies?

● Yes ● No ● We have point solutions, but not an integrated solution

CONFIDENTIAL: For limited circulation only

© 2011 MindTree Limited

Slide 10

MindTest™ - Integrated Test Methodology
Processes Templates Guidelines Checklists Best Practices

Lifecycle Processes

Customization
Tools Technologies Domain Type of Testing Stage of Testing

Analysis

Audits

TQI

TMAD

Continuous Processes

Feedback

Test Estimates

Metrics & Measurement

ProjectSpace
Knowledge Management

ProcessNet
Lifecycle Processes

Test Dashboard
Test Measurement and Metrics - ROI
© 2011 MindTree Limited

Radar
Defect Management

TestLink
Test Management

CONFIDENTIAL: For limited circulation only

Slide 11

Case Study
The Customer
The customer is the world's leading specialist in air transport communications and information technology (IT) solutions.

Business Impact • 90% coverage on test cases executed, a significant
improvement over the earlier metric

• • •

100% achievement on acceptable open defects for severity 1 and 2 Customer overhead of test case reviews reduced by 60% Constant review of the test case execution helped us to evaluate and report how far/near is the application from the acceptance criteria at different phases of testing

The Business Challenge • No testing guidelines • Inadequate process • Lack of measures for impact of testing • No single snapshot of engagement level governance

Enabled development team to prioritize the defects that needs to be fixed for the immediate build


Customer is now able to ascertain the quality of the application
Reduced risks and enabled effective decision making through test dashboard

MindTree Solution
Executed in-depth analysis to understand

• • •

Quality objectives and alignment between test process and application's business goals Used Test Metrics Analysis and Decisions (TMAD) Model and Test Quality Index (TQI) to come up with appropriate metrics and processes

“MindTree's expertise in metrics is paramount in managing the engagement and seeing if process changes have made the hoped for improvement and if the application is ready for release. The metrics are also a great way to communicate progress across all the groups in this engagement.” Senior Manager, Quality Integration

Collected key metrics as governance enablers to measure and drive decisions based on the health of the projects

CONFIDENTIAL: For limited circulation only

© 2011 MindTree Limited

Slide 12

In Short…

You don’t have to search for clues in the labyrinth

Identify the need

Evaluate the key aspects

Optimize the process

Measure and improve

CONFIDENTIAL: For limited circulation only

© 2011 MindTree Limited

Slide 13

MindTree Testing Services Overview

Testing Size and Focus
• Dedicated unit of 2200+ specialists focused on testing • Contributes 25% of the MindTree’s global business

Testing Depth and Breadth
• Tested over 2000 applications & products from chip to cloud • Services ranging from point solutions to long term CoEs • Domain specific testing services and solutions

Testing Process and Delivery Excellence
• Testing pillars of excellence - MindTest™, TestLabs, TestingAcademy • Virtual practices and CoEs for effective delivery

Thought Leadership

• Part of ISO working group in defining new testing standard – ISO / IEC 29119 • President of India’s leading testing forum STeP-IN

1500+

700+

20+

95%

delivering PREDICTABLE QUALITY through INDEPENDENT TESTING
CONFIDENTIAL: For limited circulation only © 2011 MindTree Limited Slide 14

MindTree Workshop

ARE YOU TRACKING THE RIGHT TEST METRICS FOR YOUR BUSINESS DECISIONS?
Complimentary Workshop at your Doorstep
REGISTER NOW
www.mindtree.com/events/testingwebinar/index.html or email to Balaji_Thesma@mindtree.com

CONFIDENTIAL: For limited circulation only

© 2011 MindTree Limited

Slide 15

Sylvia Veeraraghavan Sylvia_Veeraraghavan@mindtree.com www.mindtree.com
CONFIDENTIAL: For limited circulation only © 2011 MindTree Limited Slide 16

Sign up to vote on this title
UsefulNot useful

Master Your Semester with Scribd & The New York Times

Special offer for students: Only $4.99/month.

Master Your Semester with a Special Offer from Scribd & The New York Times

Cancel anytime.