Put Test in the Drivers Seat Apply a Test Management Dashboard

Kristian Fischer, Kristian.Fischer@PaConsulting.com PA Consulting Group Tuborg Boulevard 5 DK-2900 Hellerup Denmark EuroSTAR, November 2008

Abstract
This article describes why test managers and testers have to change their attitudes and transform themselves into winners, not victims. Furthermore, it explains how a simple one page test management dashboard with a handful of selected KPIs enables test managers to predict delays and use tactics to avoid them. The dashboard also provides daily progress information to enable stakeholders to make key decisions bringing the test team to be where they want to be in the future In the drivers seat.

PA Knowledge Limited 2008. Put Test in the Driver's Seat - EuroSTAR November 2008 Page 1

Put Test in the Drivers Seat Apply a Test Management Dashboard

Test managers are too often the victims rather than the victors. Traditionally they have been considered the black sheep of projects, who: Raise problems without providing solutions Find issues which cause deadlines to uncontrollably slip to the right and Always moan about not having enough time or enough (and clear) requirements. Unless we see a shift in attitude from a reactive recovery to a proactive delivery mentality, test managers will continue to be victims, and IT projects will continue to fail. So how do we create this test manager transition from self-pitying victims to victors securing project success in the future?

No more Mr. Black Sheep

A test manager is not a test manager anymore. The test managers responsibilities are growing in numbers, and the test manager must now bridge the gap between IT and the business, support requirements creation, secure alignment with business processes and the solution, and secure that the final result meets the standards expected. In other words, the test manager needs to be able to embrace a wide variety of areas within the project areas which typically have not been within test manager scope. No more Mr. Black Sheep. It is time to settle the score. The test manager needs to step up to the challenge and: Move away from the line of fire and transform them into what is needed at that Transform / move specific time in the project. And why not begin at project start-up? Be proud of what they are. And make everyone know that. The attitude has to change! Apply a test management dashboard. It will Apply Attitude dashboard create value for the project and make the test manager and the rest of the project proud.

PA Knowledge Limited 2008. Put Test in the Driver's Seat - EuroSTAR November 2008 Page 2

Move away from the line of fire

Instead of waiting for the development teams to start providing testable code, test managers should actively join the project at start-up. They have to move away from the heat away from the end of the food chain and towards the delicious starters. The mindset needs to move from reactive recovery to proactive delivery. With the right mindset and setup, this will: Support requirements creation - test managers will know if the requirements are testable, and if not how to make them testable. Furthermore, they will know the quality of requirements before any code has been developed Secure alignment with business processes and software solution by being part of the project form beginning to end, test managers will have first-hand knowledge on the business processes and requirements and the final solution. Any discrepancy will not go unnoticed Provide early testing benefits - early testing will provide valuable information on what works and what does not. It also helps find the earliest acceptable system (EAS) Provide early warnings of delays - early testing will find defects early, and this may reveal warnings of delays.

You are what you are proud of

It is not enough to engage in the project from the beginning to end. For test managers to succeed in these tasks, there is a need for a shift in attitude from all project participants from project owner to project members to test managers. For the sake of test managers, testers and the project. There are several ways to help change the attitude. One is for management to acknowledge the work done and value created by test managers and testers. Test should become a strategic tool in every organisation that deals with IT. The management should furthermore emphasize to the organisation that test no longer is a place for amateurs, who think test is something everyone can do or who cannot find other parts of the project to work in. Test has professionalized, and perhaps even in such a degree that test managers are often more capable of their work than project managers, who still seem to struggle to get rid of the amateur helmet. Another is - as mentioned above - for test managers to engage in more than just the obvious test stuff in projects. If test managers stay at the end of the food chain, things will not taste any better. But if test managers start to show their expertise, knowledge and professionalism across the project, the attitudes will inevitable change. Applying a test management tool has proven to help change the attitude and show the value test and test managers create in and for the project.

PA Knowledge Limited 2008. Put Test in the Driver's Seat - EuroSTAR November 2008 Page 3

Provide confidence in the project by applying a test management dashboard

A simple one page test management dashboard with a handful of selected KPIs on the front cover (and data in the underlying sheets) enables test managers to: Provide daily progress information and quality information Create transparency in the testing field to all stakeholders Predict delays and use tactics to avoid their occurrence Build trust in test and the project Help to change the victim-attitude. This again enables stakeholders to make key decisions, bringing the test team and the test manager to be where they want to be in the future. The benefit of the test management dashboard is the ability to highlight the main KPIs in a clear and comprehensible manner. It is therefore essential that the dashboard is: Easy to use and understand Easy to update Easy to expand or change, i.e. flexible. If based on Excel, it is furthermore an inexpensive tool to use for many organisations and most people know their way around Excel. But anything goes as long, as it meets the objectives to support requirements creation, align business process with solution, provide early testing benefits, and provide early warnings of delay.

Project EuroSTAR

Week 46

EAS

Overall Status

Test Execution Plan for December Release

200 180 160 140 120

Risk

100 80

Countdown Timer
TIT test in progress 116 days to SIT 130 days to TIT to UAT 137 days to UAT 161 days to GO-LIVE

60 40 20 0

Week 40

Week 41

Week 42

Week 43

Week 44

Week 45

Week 46

Week 47

Week 48

Benefit
Achivements Last week
-TIT test <90% delivered -FT for FA3 initiated -FA2 test completed

Risk Matrix

Next week
-Complete FA4 test preparation -Initiate test on FA4 -Complete FT for FA3

Probability
3

Issue
2

Top Concerns
Time Constraints Not clear enough requirements Not enough time available for testers to test Self-pitying test managers

High
1

Low
0 0

2
1 2

Defect Status Count of Defect ID 18 16 14 12 10 8 6 4 2 0 Change Defect Request

Status
Delivered Closed Analysis Allocated

Week 49

Good Idea
1

Low Impact

High

Defect Type

An example of a test management dashboard created in Excel

0 0 1 2

PA Knowledge Limited 2008. Put Test in the Driver's Seat - EuroSTAR November 2008 Page 4

Picking the right KPIs for the dashboard

Picking the right KPIs on the front page is an iterative process. The test manager should take outset in the basic elements, management always wants to see. During the project, the need for information will change and so must the dashboard. The above illustrated dashboard has its basic elements evolving around: Overall status - a subjective and effective way to show what way the wind blows (how are we doing and how are we feeling today?) Early Acceptable System (EAS) shows the main functions, which can form an acceptable solution when tested (what are the main and most critical functions, which can go live after a test?) Countdown to important dates puts emphasis and focus on important dates (when are we going into UAT?) Test execution plan shows number of tests planned, run, passed by week (what do we want to test and can we?) Achievements shows the main achievements. Boosts the confidence and shows attitude (what are the main achievements this week?) Concerns lists the major concerns, project manager or steering group should be aware of (what do we need to handle and what has been solved?) Risk Matrix the classic. Rate the risks to find out where to focus (what risks should we focus on and how does the overall risk picture look like?) Defect status Defects found and closed by week/day normalised against level of effort (are we looking the right places, can testers keep up testing, can developers keep up fixing defects?). Elements which are very suitable, but did not make it on the dashboard are: Requirements (changes) - major and minor requirement changes (do we know they are coming, what they are and are we ready for them?) Defects status II defects per function or activity compared to total defects (where are the defects found and should we change focus?) Exit Criteria shows the defined exit criteria (what are the exit criteria and are we meeting them?) Functional hierarchy grouping of logical functional areas (what areas are we developing and what are we testing?) Test activity plan a schedule on estimates compared to actual (can we make it in time or are we looking at early warnings of delay?) People on the project an overview of the people in the project (do we have the people we need to perform the tasks at the time needed?) Any others?
December Release
1. Area 1 Area 2
1.1 1.2 1.3 1.4 1.5 1.6 1.7 2.1 2.2 2.3
1.1.1 1.2.1 1.3.1 1.4.1 1.6.1 1.7.1
2.1.1 2.2.1

2.4

2.5

2.3.1 - Set

2.4.1

2.5.1

1.1.2

1.2.2

1.3.2

1.4.2

1.6.2

1.7.2

2.2.2

2.3.2 - Get

2.4.2

2.5.2

1.2.3

1.3.3

1.4.3

1.6.3

1.7.3

2.2.3 - xx

2.4.3

2.5.3

1.4.4

1.6.4

1.7.4

2.2.4 - xx

2.4.4

1.4.5

PA Knowledge Limited 2008. Put Test in the Driver's Seat - EuroSTAR November 2008 Page 5

The tool is not everything

Although the dashboard is simple and effective, attention needs to be given to the process of: Defining Gathering and Updating the KPIs. Furthermore, a dashboard owner (management) needs to be assigned as well as a person responsible for keeping the dashboard alive and kicking, who most likely will be the test manager. And finally, we need the whole test team to make the dashboard known in the project and organisation.

It all boils down to three things

Bringing it all together, it boils down to three things: Transform/move the test managers need to lead when appropriate and participate at all times in the project Attitude the test managers need to make their voices heard and stand up for their procession. Change the attitude test is for professionals Dashboard the test managers can make their work and value obvious by showing and telling using a test management dashboard. Now it is all up to you as a test manager to change the losing streak.

PA Knowledge Limited 2008. Put Test in the Driver's Seat - EuroSTAR November 2008 Page 6