Gthompson 220909

Implementing Test Maturity Model integrated
(TMMi) Workshop
Presented by:
Geoff Thompson
22nd September 2009

SIGiST
Listen | Challenge | Understand | Interpret | Create

Private & Confidential Experimentus Ltd 85 Tottenham Court Road London W1T 4TQ T: +44 (0)870 770 6099 www.experimentus.com
Agenda
 What is TMMi
 Understanding of the TMMi level 2 process areas
 TMMi Quick Assessment
 A view of potential TMMi level
Copyright © 2009 Experimentus Ltd

What is TMMi?
3
TMMi Model Purpose
 A standard reference model to assess test process

Maturity
 To identify:
– Testing strengths and best practices
– Project / test process issues and risks
– A realistic programme of test process improvements
– Facilities governance, measurement and accreditation of

processes
4
TMMi Model Scope
 Supports software testing in systems- and software-

engineering
 Addresses all levels of testing
– Low- and high-level dynamic testing are in scope
 Provides a generic framework as a reference model for

test process improvement
 Does not provide a process for test process
improvement
5
The TMMi Maturity Level Definitions
6
TMMi Structure
Maturity Level
Indicates Contains
Testing Process
Process Area
Capability
Achieves Contains
Specific &
Goal Generic Practice
Describes
Implementation
7
Quick Assessment
8
Background
 An indicative look at your test processes

 Only works if you answer honestly
 Focussed on Level 2
 An indication of the outcome of a TMMi assessment
 The process areas we will be reviewing:
– Test policy and strategy
– Test planning
– Test monitoring and control
– Test design and execution
– Test environment
9
Test Policy and Strategy – Improvement Goals
 Test policy, aligned with the business (quality) policy, defined and
agreed
 An organisation-wide test strategy established and deployed,
 Goal-oriented test process performance indicators established
10
Test Policy and Strategy – Activity Questions
 A test policy, aligned with the business (quality) policy, is defined

and agreed by the stakeholders
 An organisation wide or programme wide test strategy is

established and deployed, identifying and defining test levels
 Test process performance indicators defined and documented
 Test process performance indicators are deployed
11
TMMi Quick Assessment (1)
 Evaluation Dimensions
– Approach: commitment and ability to implement activity
– Deployment: how the activity is institutionalised
– Results: the effectiveness of the activity performed
12
TMMi Quick Assessment (2) – Process
 Scoring based on all three evaluation dimensions
 Activities performed in a Process Area scored between 0 and 10
 Activity score is the average of the 3 dimensions
 Process Area score is the average of all scores of the key
activities for that Process Area
13
Sample score sheet
Activities Score Test Policy and Strategy Level 2
Organisation: Acme Software Co Ltd Date: 20/05/2009

Activities performed 1 Score
Dimension 0 1 2 3 4 5 6 7 8 9
0
1. A test policy is defined and documented Approach X
Deployment X 3.3
10
Results X
2. A test strategy is defined and documented Approach X
Deployment X 12
4
Results X
3. Test process performance indicators are Approach X
defined and documented
Deployment X 7.7
23
Results X
4. Test process performance indicators are Approach X
deployed
Deployment X 6
2
Results X
Total Score 17
Result (= total score / 4) 4.25
14
TMMi Quick Assessments – Result
 For every Process Area:

– Average score :
 Up to 2.99 “Not Satisfied”
 Between 3 and 4.99 “Partly Satisfied”
 Between 5 and 7.99 “Largely Satisfied”
 Over 8 “Fully Satisfied”
15
Test Planning – Improvement Goals
 Risk assessment performed to identify critical areas for testing
 Test approach established and agreed
 Well-founded test estimates established and maintained
 Test plan established and maintained
 Commitments to test plan established and maintained
16
Test Planning: Activity Questions
 Risk assessment performed to identify critical areas for testing
 Test approach based on risks is developed and agreed
 Test estimates established / agreed according to documented

procedure
 Test plan developed / agreed, ensuring that testing is well

managed and progress reported
 Test project commitments made to external groups are

established / maintained
17
Test Monitoring and Control – Improvement Goals
 Actual progress and performance is monitored against test plan
 Actual product quality monitored against plan and expectations
 Corrective actions managed to closure when progress or quality

deviate significantly from plan or expectations
18
Test Monitoring and Control – Activity Questions
 Progress monitored against test plan
 Product quality measured and reported in a standardised way
 Changes to plan or expectations are controlled / managed
effectively
 Data from previous projects used to shape future projects

19
Test Design and Execution – Improvement Goals
 During test analysis and design test approach is translated into

tangible test conditions and test cases
 Undertake all tasks required to prepare for test execution
 Manage all incidents using the defined work flow
20
Test Design and Execution – Activity Questions
 Test cases specified according to documented set of techniques
 Test execution preparation tasks documented and completed
 Test execution performed using documented procedures
 Test incidents reported / managed using documented procedures
21
Test Environment – Improvement Goals
 Environment requirements analysed, specified and verified as

achievable and appropriate
 Environments and data are built, tested and provided according

to requirements
 Environments and data appropriately supported
22
Test Environment – Activity Questions
 Environment specified early in the project according to

documented procedure
 Higher test levels performed in environment that is „real-life‟
 Management and control of environment is by documented

procedures
 Build, availability and usage of environment is co-ordinated using

documented procedure
 Environment incidents reported according to a documented

procedure
23
The Result
Do you agree with the result?

How is everyone else doing?
 Experimentus obtains accreditation for its Assessment Method
 The need to understand the industry
 Survey data collected during Q4 2008
 More than 100 respondents
 Many industry sectors
 Respondents from 4 continents
25
Survey - Overall Results
TMMi level 2
27.5%
TMMi level 1
72.5%
26
A Reminder of Level 1
 Level 1 means
– Little process
– Delivery of good testing

depends on people who
know what they are
doing
– The classic hero culture

of the indispensible
27
By Industry Sector
TMMi level 2
TMMi level 1
28
What are you going to do next?

Assessment Results
 An assessment against TMMi Levels 2 and 3
Key Process Areas Maturity Achievement
3.4 Control & Monitoring of Testing Process
Actively Performed
3.3 Testing Integrated into the SDLC
Mostly Deployed
3.2 Test Technical Training Program

Partially/Not Deployed
3.1 Testing Organisation
2.3 Test Design Techniques & Methods
2.2 Test Planning & Execution
2.1 Testing & Debugging Goals & Policies
0% 20% 40% 60% 80% 100% 0% 20% 40% 60% 80% 100%
30
Thank you
www.experimentus.com
+44(0)8707 706099
info@experimentus.com
31
About Experimentus....
 An IT services consultancy providing organisations with software quality

management expertise
 Key solutions aimed at optimising the approach to software quality

management
• Process assessment and certification

• Trusted advice and delivery
• Process improvement
• Metrics
 Accredited TMMi method, assessors and lead assessors
 Proven track record in using TMMi to help organisations improve

32
Acknowledgements
 Test Maturity Model integrated (TMMi) – TMMi Foundation
 CMM / CMMI – Software Engineering Institute (SEI)
 TMMi Assessment Method, Experimentus Ltd

Gthompson 220909

Uploaded by

Document Information

Original Title

Copyright

Available Formats

Share this document

Share or Embed Document

Sharing Options

Did you find this document useful?

Is this content inappropriate?

Copyright:

Available Formats

Gthompson 220909

Uploaded by

Copyright:

Available Formats

Implementing Test Maturity Model integrated

22nd September 2009

Listen | Challenge | Understand | Interpret | Create

Copyright © 2009 Experimentus Ltd

 A standard reference model to assess test process

– Project / test process issues and risks

– A realistic programme of test process improvements

– Facilities governance, measurement and accreditation of

 Supports software testing in systems- and software-

 Provides a generic framework as a reference model for

 An indicative look at your test processes

 An organisation-wide test strategy established and deployed,

 Goal-oriented test process performance indicators established

 A test policy, aligned with the business (quality) policy, is defined

 An organisation wide or programme wide test strategy is

 Test process performance indicators defined and documented

 Test process performance indicators are deployed

– Approach: commitment and ability to implement activity

– Deployment: how the activity is institutionalised

– Results: the effectiveness of the activity performed

 Scoring based on all three evaluation dimensions

 Activities performed in a Process Area scored between 0 and 10

 Activity score is the average of the 3 dimensions

 Process Area score is the average of all scores of the key

activities for that Process Area

Activities Score Test Policy and Strategy Level 2

Organisation: Acme Software Co Ltd Date: 20/05/2009

 For every Process Area:

 Up to 2.99 “Not Satisfied”

 Between 3 and 4.99 “Partly Satisfied”

 Between 5 and 7.99 “Largely Satisfied”

 Over 8 “Fully Satisfied”

 Risk assessment performed to identify critical areas for testing

 Test approach established and agreed

 Well-founded test estimates established and maintained

 Test plan established and maintained

 Commitments to test plan established and maintained

 Risk assessment performed to identify critical areas for testing

 Test approach based on risks is developed and agreed

 Test estimates established / agreed according to documented

 Test plan developed / agreed, ensuring that testing is well

 Test project commitments made to external groups are

 Actual progress and performance is monitored against test plan

 Actual product quality monitored against plan and expectations

 Corrective actions managed to closure when progress or quality

 Progress monitored against test plan

 Product quality measured and reported in a standardised way

 Changes to plan or expectations are controlled / managed

 Data from previous projects used to shape future projects

 During test analysis and design test approach is translated into

 Undertake all tasks required to prepare for test execution

 Manage all incidents using the defined work flow

 Test cases specified according to documented set of techniques

 Test execution preparation tasks documented and completed

 Test execution performed using documented procedures

 Test incidents reported / managed using documented procedures

 Environment requirements analysed, specified and verified as

 Environments and data are built, tested and provided according

 Environments and data appropriately supported

 Environment specified early in the project according to

 Higher test levels performed in environment that is „real-life‟

 Management and control of environment is by documented

 Build, availability and usage of environment is co-ordinated using