The Pathologies of Failed Test Automation Projects.v11.STAREast 2013 External

The Pathologies of failed
Test Automation Projects
Michael Stahl, Intel

Apr 2013
Based, in part, on work done with Alon Linetzki, Best Testing (www.best-testing.com)
2 On the Menu…
• Automation failure patterns
• What can we do?
• Summary
Based, in part, on work done with Alon Linetzki, Best Testing (www.best-testing.com)
© Michael Stahl, 2013, all rights reserved

Disclaimers
 Names and brands referenced herein may be claimed as the property of

third parties
 The views expressed in this presentation are solely my own, and do not in
any manner represent the views of my employer
 Information in this presentation is provided “AS IS” without any warranties
or representations of any kind
A common automation story…
4
 Once upon a time…

I can automate
my work!...
Cool! Mr. Otto Mate

Can you make it…
Otto’s Team

5
Piece of
cake!
Perfect!
Can you add…
Mr. Otto Mate

What a wonderful
world!...
Otto’s Team
Mr. Mann a. Ger

6
Arggghhh!
Fix Fix Fix
Otto! The
last release
Otto’s Team fails!... Mr. Otto Mate
… and mates
Makes sense… Sir!

I need time!
I need help!
Mr. Mann a. Ger

7
Mr. Otto Mate

… and mates
… and mates
… and mates
8
Let’s
Mr. Oto Mate
REDESIGN!!! … and mates
#$&@***!!!
… and mates
Mr. Mann a. Ger … and mates

9
A pattern emerges…

Pattern #1: Mushrooming

A pattern…
11
Single User / Developer
Simple Tool
Stage 1 – Small & Local

A pattern…
12
Multiple Users / Single Developer
Enhanced Tool
Stage 2 – Generalization
A pattern…
13
Multiple Users &

Developers
Complicated Tool
Stage 3 – Staffing
A pattern…
14
Multiple Users &

Developers
Test Case
Management
Stage 4 – Non-core features

A pattern… Arghhh!
15
Stage 5 – Overload
Pattern #2:
The Competition
16

Pattern #2 – The Competition (ver. 1)
17
Team A
Team B

18
Team A
Team B
Team C
Team D !!!
19
Team A
Team B
!!! …
!!!
Team C …
!!!
Team D …
Pattern #3: The Night Run Fallacy

Pattern #3 – The Night Run Fallacy
21

22
Night Time
Test Time

23
Add a snapshot – or a movie snippet – of PAVE
Corporate Truism:
It’s easier to get budget for machines than for more testers
24
Add a snapshot – or a movie snippet – of PAVE

25
Test Automation Truism:

Machines create work for more testers
The Tragedy of the Commons
multiple individuals…
will ultimately deplete a shared limited resource…
even when it is not in their long-term interest
Garrett Hardin, Science, 1968

Pattern #4:
Going for the Numbers

28

Pattern #5 – Going for the Numbers
29

30
Robustness is Invisible

Pattern #5:
The Magician Apprentice Syndrome
Pattern# 5 – The Magician Apprentice
32

Recap: The Patterns
3
Mushrooming
3
The Competition
The Night time Fallacy
The Magician Apprentice

34 So...
What can we do?

35 Are the patterns…
Unavoidable ? ?

36 Counter measures

37 Mushrooming

Alert Signals & Counter Measures
38
How to use this information?

 GPS
 Locate the stage you are at

 Get directions for the way out
 Map
 Startright and avoid
the wrong turns

Alert Signals
39

Alert Signals
40

Alert Signals
41
Failing the project
Failing the project
Failing the project
Failing the project
Failing the project

Alert Signals: Stage 1
Small, local, feature-centered
42
 Single feature test tool?

 The creator is the user?
 “Skunk works”?
 Key words:
 “Tool”
 “Utility”

Counter Measures: Stage 1
Small, local, feature-centered
43
Ensure the following… and relax:

 Code control
 Documentation
 User Manual (Usage line…)
 “Green” in the code
 High level design

Generalization
44
 Additional features?
 Multiple users?
 Automation web site?
 >25% of the tester’s time?
 Key words:
 “Use by other testers”
 “Common Libraries”

Generalization
45
“The hardest part of building a software system is deciding

precisely what to build... No other part of the work so cripples
the resulting system if done wrong. No other part is more
difficult to rectify later”
- Fred P. Brooks (author of “The Mythical Man-Month”)

Generalization
46
 Stage 1 measures
 Strategy
 Architecture
 Lightweight PM
 Version control
 Scope control
 Bugs & Requests database

Institutionalization and staffing
47
 Requests for additional heads?

 Automation F2F?
 Tool-related delays in execution?
 Key words:
 “Tool Owner”; “Automation team”
 “Framework”; “Infrastructure”
 “Roll back”
 “Bug fix release”
48

49
 Stage 1, 2 counter measures

 Management level decision time
 Programming language
 Framework focus
 Code and release management
 Acceptance tests
 Skillset development
 Metrics

50
Metrics suggestions
 Automation framework quality
 Number of false fails
 Framework’s test results; Bug trends
 ROI
 Number of runs
 Invested effort by type (new,
maintenance, rewrite)
 Number of bugs found by Automation (?)

Change of focus: Technology  Management
51
 Generic features?
 Test log wading?
 False fails?
 Key words:
 “test suite / cycle generation”
 “robustness enhancement”
 “setup issues”

52
 Stage 1, 2, 3 counter measures

 Build VS Buy
 Re-architect
 Core / Non-Core
 Solid infrastructure
 Influence upstream
 Testability hooks
 Design for Test (automation)

 Build VS Buy?
(hint: Buy)
See: http://www.stickyminds.com/s.asp?F=S17601_COL_2
Stage 4 – Non-core features

54
Build (VS Buy) if…

 Competitive edge
 Existing expertize
 Core competency
 Cheaper; Faster
 Good use of resources
 Acceptable risk
 Long term support
Main source: Allen Eskelin
http://www.informit.com/articles/article.aspx?p=21775

Maintenance overload; Re-design
55
 Maintenance & logistics overload?

 Limitations overplay?
 Loss of credibility?
 Stage 1 initiatives?
 Key words
 “Did it fail in manual test?”
 “Architecture limitation”
 “refactoring”; “redesign”
 “…I can write a small program…”

56
Loss of credibility

57
Options:
 Continue… 
 Give up problematic areas
 Partial return to Stage 1
 “We value Robustness over New
Features”
 Prepare for re-design – with a new
map...
How to Use this information?
58
 GPS
 Locate the stage you are at
Be alert for
 Get directions for the way Alerts
out
Implement
Identify your
Counter
stage
Measures
Analyze your
situation

How to Use this information?
59
 Map
 Start right and avoid
the wrong turns Plan your trip
Implement
Be alert for
Counter
Alerts
Measures
Analyze your
situation

60 The Competition

61
 Salvageable up to stage 2
 Stage 3 and up:
 Merge the teams
 EOL both tools
 Start a 3rd

62
 Plugin Architecture

63
 Fix the main problem

 Accept, encourage Stage 1 stuff…
 Maintaining vigil:
 Avoid ver.1 pattern

64 The Night Run Fallacy

65 6 6
5 5
 Never forget test time, test efficiency

 Balance test skills VS automation skills
 Allocate machines or machine time

66 Going for the Numbers

67
 Change how you count “Done”

68 The Magician Apprentice Syndrome

69
http://www.youtube.com/watch?v=zlnz1rSJj7Y

70
Automation should not necessarily mimic

humans…
… it should get the job done

71
http://www.youtube.com/watch?v=qDVYIT85ntg

72
 Holistic approach VS “test after test”

 Re-think, re-strategize, re-evaluate
before automating
 Identify Actions, Keywords (KDT)

73 Summary

The Patterns
7
Mushrooming
4
The Competition
The Night time Fallacy
The Magician Apprentice

Summary
75
 The patterns are pervasive

 Almost inevitable
 Awareness is key (engineers; managers)
 Driven by organizational and human nature
 The solution is only partially technical

Thank You!
Questions time…
michael.stahl@intel.com
www.testprincipia.com

The Pathologies of Failed Test Automation Projects.v11.STAREast 2013 External

Uploaded by

Document Information

Original Title

Copyright

Available Formats

Share this document

Share or Embed Document

Sharing Options

Did you find this document useful?

Is this content inappropriate?

Copyright:

Available Formats

The Pathologies of Failed Test Automation Projects.v11.STAREast 2013 External

Uploaded by

Copyright:

Available Formats

The Pathologies of failed

Test Automation Projects

Michael Stahl, Intel

© Michael Stahl, 2013, all rights reserved

 Names and brands referenced herein may be claimed as the property of

 Once upon a time…

Cool! Mr. Otto Mate

© Michael Stahl, 2013, all rights reserved

Mr. Otto Mate

© Michael Stahl, 2013, all rights reserved

Makes sense… Sir!

Mr. Mann a. Ger

Mr. Otto Mate

Mr. Mann a. Ger … and mates

© Michael Stahl, 2013, all rights reserved

© Michael Stahl, 2013, all rights reserved

Single User / Developer

Stage 1 – Small & Local

Multiple Users / Single Developer

Multiple Users &

Multiple Users &

Stage 4 – Non-core features

© Michael Stahl, 2013, all rights reserved

© Michael Stahl, 2013, all rights reserved

© Michael Stahl, 2013, all rights reserved

© Michael Stahl, 2013, all rights reserved

© Michael Stahl, 2013, all rights reserved

Add a snapshot – or a movie snippet – of PAVE

Add a snapshot – or a movie snippet – of PAVE

© Michael Stahl, 2013, all rights reserved

Test Automation Truism:

© Michael Stahl, 2013, all rights reserved

© Michael Stahl, 2013, all rights reserved

© Michael Stahl, 2013, all rights reserved

© Michael Stahl, 2013, all rights reserved

© Michael Stahl, 2013, all rights reserved

© Michael Stahl, 2013, all rights reserved

© Michael Stahl, 2013, all rights reserved

What can we do?

© Michael Stahl, 2013, all rights reserved

© Michael Stahl, 2013, all rights reserved

© Michael Stahl, 2013, all rights reserved

© Michael Stahl, 2013, all rights reserved

How to use this information?

 Locate the stage you are at

© Michael Stahl, 2013, all rights reserved

© Michael Stahl, 2013, all rights reserved

© Michael Stahl, 2013, all rights reserved

Failing the project

Failing the project

Failing the project

Failing the project

Failing the project

© Michael Stahl, 2013, all rights reserved

 Single feature test tool?

© Michael Stahl, 2013, all rights reserved

Ensure the following… and relax:

© Michael Stahl, 2013, all rights reserved

© Michael Stahl, 2013, all rights reserved

“The hardest part of building a software system is deciding

© Michael Stahl, 2013, all rights reserved

© Michael Stahl, 2013, all rights reserved

 Requests for additional heads?