Exploratory Test Management Outline

1. 2. 3. 4. 5.

Introduction Test Management and Techniques ET Planning, Exec. and Documentation ET Styles ET Management

Exploratory Test Team Management

Risk-Based Test Management Session Based Test Management

ET Workshop v. 1.20 Test Management

2002 Amland Consulting

5-1

Exploratory Test Management

These slides are distributed under the Creative Commons License.
In brief summary, you may make and distribute copies of these slides so long as you give the original author credit and, if you alter, transform or build upon this work, you distribute the resulting work only under a license identical to this one.

For the rest of the details of the license, see http://creativecommons.org/licenses/bysa/2.0/legalcode.

ET Workshop v. 1.20 Test Management

2002 Amland Consulting

5-2

1. 2. 3. 4. 5.

5. Exploratory Test Management

5.1 Exploratory Test Team Management 5.2 Risk-Based Test Management 5.3 Session Based Test Management

ET Workshop v. 1.20 Test Management

2002 Amland Consulting

5-3

Introduction Test Management and Techniques ET Planning and Documentation ET Styles ET Management

Traditional Test Teams

are built on
Available knowledge and skills Business needs Employees ambitions

provide a career path through

Execution coordination planning Defect reporting analysis Strategy

This might not work for Exploratory Testing

ET Workshop v. 1.20 Test Management

2002 Amland Consulting

5-4

1. 2. 3. 4. 5.

How to build and lead an effective test team

Slides and Questionnaire from Lloyd Roden, Grove Consultants, Lloyd@grove.co.uk

ET Workshop v. 1.20 Test Management

Copyright 2002 Lloyd Roden, Grove Consultants (Lloyd@grove.co.uk)

2002 Amland Consulting 5-5

Introduction Test Management and Techniques ET Planning and Documentation ET Styles ET Management

The questionnaire

Things to note No right or wrong answer Try not to think too much Helps us assess our strengths Can be used with other management questionnaires (Belbin, 16PF etc) Plot values on the graph
Copyright 2002 Lloyd Roden, Grove Consultants (www.grove.co.uk)
2002 Amland Consulting 5-6

ET Workshop v. 1.20 Test Management

Introduction Test Management and Techniques ET Planning and Documentation ET Styles

ET Management

The questionnaire - how to complete

X - Axis
X __ __ __ __ X __ __ X __ X __ X __ __ X __ 6
Formal Retiring Business Like Guarded Organised Introvert Logical Focused Cool Insensitive __ X __ __ X __ __ X __ __ __ __ X __

Friendly Approachable Casual Open Unstructured Social Intuitive Random Warm Perceptive
ET Workshop v. 1.20 Test Management

Copyright 2002 Lloyd Roden, Grove Consultants (www.grove.co.uk)

2002 Amland Consulting 5-7

Introduction Test Management and Techniques ET Planning and Documentation ET Styles

ET Management

The questionnaire - how to complete

Y - Axis
__ __ __ __ X __ __ __ X __ __ __ __ 2 Indirect Accepting Leisurely Thoughtful Relaxed Patient Cautious Receptive Co-operative Analytical

To the point Challenging Quick Insistent Lively Impatient Adventurous Confronting Competitive Strong Minded
ET Workshop v. 1.20 Test Management

X __ X __ X __ __ X __ X __ __ X __ X __ X __
5-8

Copyright 2002 Lloyd Roden, Grove Consultants (www.grove.co.uk)

2002 Amland Consulting

Introduction Test Management and Techniques ET Planning and Documentation ET Styles ET Management

The Grid
10 9 8 7

6
0 1 2 3 4 5 4 3 2 1 0
ET Workshop v. 1.20 Test Management

10

X (6,2)

Copyright 2002 Lloyd Roden, Grove Consultants (www.grove.co.uk)

2002 Amland Consulting 5-9

Introduction Test Management and Techniques ET Planning and Documentation ET Styles ET Management

The Model
The Pragmatist The Pioneer

The Analyst
ET Workshop v. 1.20 Test Management

The Facilitator

Copyright 2002 Lloyd Roden, Grove Consultants (www.grove.co.uk)

2002 Amland Consulting 5-10

Introduction Test Management and Techniques ET Planning and Documentation ET Styles ET Management

Key words for the Pragmatic Style Tester

Likes Dislikes

strategic / goals positive results / brief practical efficiency tasks

ET Workshop v. 1.20 Test Management

indecision vagueness time-wasting unproductive

Copyright 2002 Lloyd Roden, Grove Consultants (www.grove.co.uk)

2002 Amland Consulting 5-11

Introduction Test Management and Techniques ET Planning and Documentation ET Styles

ET Management

The Pragmatic style tester will

be good for setting and monitoring short/long term goals for the team be good at documenting factual test reports remain positive through pressure be keen to adopt Most Important Tests first principle be a strong driving force - ensure a task is done want to implement efficiency into the team be self-motivated and task oriented will make quick decisions enjoy challenging testing tasks

ET Workshop v. 1.20 Test Management

Copyright 2002 Lloyd Roden, Grove Consultants (www.grove.co.uk)

2002 Amland Consulting 5-12

Introduction Test Management and Techniques ET Planning and Documentation ET Styles ET Management

Key words for the Pioneer Style Tester

Likes Dislikes

new / ideas change openness results/efficiency involving others risks

ET Workshop v. 1.20 Test Management

standards detail norm paper-work

Copyright 2002 Lloyd Roden, Grove Consultants (www.grove.co.uk)

2002 Amland Consulting 5-13

Introduction Test Management and Techniques ET Planning and Documentation ET Styles

ET Management

The Pioneer style tester will

be good at ad-hoc testing / bug hunting / errorguessing/ exploratory testing be good at challenging and improving things to make more efficient and effective enjoy GUI type testing/lateral tester have good ideas be good at brainstorming Test Conditions share ideas about different ways to approach testing identify and take necessary risks when required have creative test ideas - how to find more faults

ET Workshop v. 1.20 Test Management

Copyright 2002 Lloyd Roden, Grove Consultants (www.grove.co.uk)

2002 Amland Consulting 5-14

Introduction Test Management and Techniques ET Planning and Documentation ET Styles ET Management

Key words for the Analysing Style Tester

Likes Dislikes

accuracy attention to detail proof standards reliable all alternatives

ET Workshop v. 1.20 Test Management

new / change untested / risks brief / speed letting go

Copyright 2002 Lloyd Roden, Grove Consultants (www.grove.co.uk)

2002 Amland Consulting 5-15

Introduction Test Management and Techniques ET Planning and Documentation ET Styles

ET Management

The Analysing style tester will

be good at defining and documenting test cases be good at producing test standards and procedures analyse problems and finding root cause produce work which is accurate and complete enjoy logical tests scenarios provide proof when faults are found document thorough test reports complete work regardless of what it takes challenge requirements

ET Workshop v. 1.20 Test Management

Copyright 2002 Lloyd Roden, Grove Consultants (www.grove.co.uk)

2002 Amland Consulting 5-16

Introduction Test Management and Techniques ET Planning and Documentation ET Styles ET Management

Key words for the Facilitating Style Tester

Likes

Dislikes

networking positive team oriented consensus / sharing building bridges status quo

pressure / deadlines confrontation isolation dictated

ET Workshop v. 1.20 Test Management

Copyright 2002 Lloyd Roden, Grove Consultants (www.grove.co.uk)

2002 Amland Consulting 5-17

Introduction Test Management and Techniques ET Planning and Documentation ET Styles

ET Management

The Facilitating style tester will

be good in a RAD environment or a buddy test team often ask opinion before raising issues be good at documentation co-operate well with other departments often see the other side be good at defusing us v them syndrome be popular make things happen - eventually! will provide support in testing to other team members

ET Workshop v. 1.20 Test Management

Copyright 2002 Lloyd Roden, Grove Consultants (www.grove.co.uk)

2002 Amland Consulting 5-18

Introduction Test Management and Techniques ET Planning and Documentation ET Styles ET Management

Tester Style - patterns

usually operate within a certain boundary what if you are on the line/centre
flexible between styles can be difficult to manage

opposites repel
maybe reasons for team tension!

analysts & pragmatists tend towards Tasks facilitators & pioneers tend towards People
ET Workshop v. 1.20 Test Management

Copyright 2002 Lloyd Roden, Grove Consultants (www.grove.co.uk)

2002 Amland Consulting 5-19

Introduction Test Management and Techniques ET Planning and Documentation ET Styles ET Management

The Ideal Test Team

A MIXTURE IS THE BEST BUT IT DEPENDS...

ET Workshop v. 1.20 Test Management

Copyright 2002 Lloyd Roden, Grove Consultants (www.grove.co.uk)

2002 Amland Consulting 5-20

1. 2. 3. 4. 5.

5. Exploratory Test Management

5.1 Exploratory Test Team Management 5.2 Risk-Based Test Management 5.3 Session Based Test Management

ET Workshop v. 1.20 Test Management

2002 Amland Consulting

5-21

1. 2. 3. 4. 5.

Ordinary Project Plan including Testing

Specification Design Coding Test

Plan:

In the Real World: Specification Design Coding Test

ET Workshop v. 1.20 Test Management

2002 Amland Consulting

5-22

1. 2. 3. 4. 5.

Incremental Project Plan with Testing

Spec. Design Coding Test

Plan:

Spec. Design

Coding

Test

Spec. Design

Coding

Test

In the Real World:

Spec. Design Coding Test?

Spec.

Design

Coding

Test?

Spec.

Design

Coding

Test......

ET Workshop v. 1.20 Test Management

2002 Amland Consulting

5-23

1. 2. 3. 4. 5.

Why Risk Based Testing?

Its about controlling:
Time Resources New Technology Lack of knowledge Lack of experience Quality Demands Scope ???
ET Workshop v. 1.20 Test Management 2002 Amland Consulting 5-24

1. 2. 3. 4. 5.

What is Risk?

A risk is an unwanted event that has negative consequences.

Shari Lawrence Pfleeger
In other words, a risk is a problem waiting to happen.
ET Workshop v. 1.20 Test Management

Shari Lawrence Pfleeger (2000)

2002 Amland Consulting 5-25

1. 2. 3. 4. 5.

What is Risk Management?

Plans to avoid these unwanted events or, if they are inevitable, minimize their negative consequences.
Shari Lawrence Pfleeger
ET Workshop v. 1.20 Test Management

Shari Lawrence Pfleeger (2000)

2002 Amland Consulting 5-26

1. 2. 3. 4. 5.

Problems vs. Risk

Problem (Issue):
Something that has or will happen

Risk
Something that might happen (in the future)

ET Workshop v. 1.20 Test Management

2002 Amland Consulting

5-27

1. 2. 3. 4. 5.

Types of Risk
Project / Process Risk
Resourcing Planning Contracts Etc.

Business / Product Risk

Stability Performance Quality / Errors (Quality Risk = Potential Errors) Etc.
ET Workshop v. 1.20 Test Management 2002 Amland Consulting 5-28

1. 2. 3. 4. 5.

Business Risk Why is it so difficult? (1)

Flaw in the Pentium chip (1994)
6 million PCs sold, potential USD300 loss per chip Intels risk impact USD 1.8 billion Intel:
Average computer user: wrong answer every 27.000 years Heavy user: every 270 years Conclusion: The flaw is not meaningful to most users

IBM:
Average computer user: a problem every 24 days A large company (500 PCs): 20 problems per day! Conclusion: stopped selling Pentium PCs

IBMs assessment 400.000 times worse than Intels!

ET Workshop v. 1.20 Test Management

Pfleeger (2000)
2002 Amland Consulting 5-29

1. 2. 3. 4. 5.

Business Risk Why is it so difficult? (2)

Evaluating the risk of an accident at a small ammonia storage plant (1988 to 1990) 11 EU countries (national teams) + several private firms
The national teams assessment varied by a factor of 25.000, reaching wildly different conclusions.
ET Workshop v. 1.20 Test Management

Pfleeger (2000)
2002 Amland Consulting 5-30

1. 2. 3. 4. 5.

How to identify risk?

1. A loss associated with the event
2. The likelihood that the event will occur 3. The degree to which we can change

the outcome

ET Workshop v. 1.20 Test Management

2002 Amland Consulting

5-31

1. 2. 3. 4. 5.

What is Risk-Based Testing?

Risk-based testing carries at least two major meanings:
Risk-Based Test Management
To determine what things to test next (prioritisation)

Doing Risk analysis for the Purpose of Finding Errors

ET Workshop v. 1.20 Test Management

Lessons Learned in Software Testing, Cem Kaner et al 2001b

2002 Amland Consulting 5-32

1. 2. 3. 4. 5.

RBT: The purpose of finding errors

Make a prioritised list of risks.
Perform testing that explores each risk. As risks evaporate and new ones emerge, adjust your test effort to stay focused on the current crop.
ET Workshop v. 1.20 Test Management

James Bach 1999a

2002 Amland Consulting 5-33

1. 2. 3. 4. 5.

Risk-Based Test Management

1. Define all requirements to be tested 2. Based on risk assessment prioritise the requirements 3. Plan and define tests according to requirement prioritisation (coverage to be defined in test plan) 4. Execute test according to prioritisation and acceptance criteria (as defined in the test plan)
ET Workshop v. 1.20 Test Management 2002 Amland Consulting 5-34

1. 2. 3. 4. 5.

Risk-Based Test Management

Statistical Risk Analysis (Hans Schaefer) Failure Mode and Effect Analysis (FMEA) Software Reliability engineering, John D. Musa Risk based test strategy (TMap, Iquip.nl)) Decision Theory (Bayesian Belief Nets) Rational Unified Process (RUP) ...and many, many more
ET Workshop v. 1.20 Test Management 2002 Amland Consulting 5-35

1. 2. 3. 4. 5.

RBT Management - Statistic Risk Analysis Hans Schaefer

Statistical Risk Analysis Theory

Equation:

Re(f) P(f)*C(f)
Re(f) - Risk Exposure of function f P(f) - Probability of a fault in function f C(f) - Cost related to a fault in function f
ET Workshop v. 1.20 Test Management

Schaefer 1998, Amland 1999

2002 Amland Consulting 5-36

Introduction Test Management and Techniques ET Planning and Documentation ET Styles ET Management

RBT Management - Statistic Risk Analysis - Hans Schaefer

Statistical Risk Analysis Matrix

Cost Probability Weight (1-5) Weight (1-5) Risk 1 3 Wieghted Exposure Weighted Indicator 1 Indicator 2 Average 1 3 2 1 2 2 2 4,5 4 4 4,5 12

Function (1 - 3) A B C 2 1 3

ET Workshop v. 1.20 Test Management

Schaefer 1998, Amland 1999

2002 Amland Consulting 5-37

Introduction Test Management and Techniques ET Planning and Documentation ET Styles

ET Management

Statistical Risk Based Testing

Plan: Identify Elements to be Tested
Logical or physical Functions, Modules etc.

Identify Risk Indicators

What is important to predict the probability of faults?

Identify Cost (consequence) of faults Identify Critical Elements

I.e. functions, tasks, activities etc. based on Risk Analysis (Indicators and Cost)

Execute
Improve the Test Process and Organization: Schedule and Track
ET Workshop v. 1.20 Test Management

Schaefer 1998, Amland 1999

2002 Amland Consulting 5-38

Introduction Test Management and Techniques ET Planning and Documentation ET Styles

RBT Management - Statistic Risk Analysis

Example Statistical Risk Analysis Matrix

Cost C(s) C(c) Avrg. * Probability New Desgn Func. Qual. Size = Re Com- Weigh. Risk plexity Sum Exposure

ET Management

5
Interest Calc.

5
3

1
3

3
3 37 111

Close Account
Customer Profitablty

31

62

1,5

41

61,5

Other Probability Factors might include: Function Points, Frequency of Use etc.
ET Workshop v. 1.20 Test Management

Schaefer 1998, Amland 1999

2002 Amland Consulting 5-39

Introduction Test Management and Techniques ET Planning and Documentation ET Styles

ET Management

Risk Based Testing - Reporting

510 10 1

High Low Medium

440 439

INTERFACE RISK 11 2 2

Probability

TECHNICAL

370 369

302

Low

Low

Medium

Medium

High

High

BUSINESS RISK Consequence

ET Workshop v. 1.20 Test Management

2002 Amland Consulting

5-40

1. 2. 3. 4. 5.

5. Exploratory Test Management

5.1 Exploratory Test Team Management 5.2 Risk-Based Test Management 5.3 Session Based Test Management

ET Workshop v. 1.20 Test Management

2002 Amland Consulting

5-41

1. 2. 3. 4. 5.

Charter/ Mission

Notes, Risks, Issues, Questions and Errors

Execution: Pairs & Sessions

Debriefing

Introducing the Test Session

1) Charter 2) Time Box

3) Reviewable Result
4) Debriefing

vs.
5-42

ET Workshop v. 1.20 Test Management

From Rapid Software Testing, copyright 1996-2002 James Bach

2002 Amland Consulting

1. 2. 3. 4. 5.

Charter/ Mission

Notes, Risks, Issues, Questions and Errors

Execution: Pairs & Sessions

Debriefing

Charter summary
Architecting the Charters i.e. Test Planning Brief information / guidelines on:
Mission: Why do we test this? What should be tested? How to test (approach)? What problems to look for?

Might include guidelines on:

Tools to use Specific Test Techniques or tactics to use What risks are involved Documents to examine Desired output from the testing
ET Workshop v. 1.20 Test Management 2002 Amland Consulting 5-43

Introduction Test Management and Techniques ET Planning and Documentation ET Styles ET Management

Charter/ Mission

Notes, Risks, Issues, Questions and Errors

Execution: Pairs & Sessions

Debriefing

Time Box: Focused test effort of fixed duration

Short: 60 minutes (+-15) Normal: 90 minutes (+-15) Long: 120 minutes (+-15)
Brief enough for accurate reporting. Brief enough to allow flexible scheduling. Brief enough to allow course correction. Long enough to get solid testing done. Long enough for efficient debriefings. Beware of overly precise timing.

ET Workshop v. 1.20 Test Management

From Rapid Software Testing, copyright 1996-2002 James Bach

2002 Amland Consulting 5-44

Introduction Test Management and Techniques ET Planning and Documentation ET Styles

Charter/ Mission

Notes, Risks, Issues, Questions and Errors

Execution: Pairs & Sessions

Debriefing

ET Management

Reviewable Result: The session sheet

Charter
#AREAS

Start Time Tester Name(s) Breakdown

Test Notes Bugs

#BUG

Issues
#ISSUE
CHARTER ----------------------------------------------Analyze MapMakers View menu functionality and report on areas of potential risk. #AREAS OS | Windows 2000 Menu | View Strategy | Function Testing Strategy | Functional Analysis START ----------------------------------------------5/30/00 03:20 pm TESTER ----------------------------------------------Jonathan Bach TASK BREAKDOWN ----------------------------------------------#DURATION short #TEST DESIGN AND EXECUTION 65 #BUG INVESTIGATION AND REPORTING 25 #SESSION SETUP 20

#DURATION #TEST DESIGN AND EXECUTION #BUG INVESTIGATION AND REPORTING #SESSION SETUP #CHARTER/OPPORTUNITY

Data Files
ET Workshop v. 1.20 Test Management 2002 Amland Consulting

From Rapid Software Testing, copyright 1996-2002 James Bach

5-45

Introduction Test Management and Techniques ET Planning and Documentation ET Styles ET Management

Charter/ Mission

Notes, Risks, Issues, Questions and Errors

Execution: Pairs & Sessions

Debriefing

Debriefing: Measurement begins with observation

The manager reviews session sheet to assure that he understands it and that it follows the protocol. The tester answers any questions. Session metrics are checked. Charter may be adjusted. Session may be extended. New sessions may be chartered. Coaching / Mentoring happens.

ET Workshop v. 1.20 Test Management

From Rapid Software Testing, copyright 1996-2002 James Bach

2002 Amland Consulting 5-46

1. 2. 3. 4. 5.

Exploratory Test Management Summary

Introduction Test Management and Techniques ET Planning, Exec. and Documentation ET Styles ET Management

Exploratory Test Team Management

Risk-Based Test Management Session Based Test Management

ET Workshop v. 1.20 Test Management

2002 Amland Consulting

5-47

Exercise 5
Based on available information;
Discuss how you would manage and track a test team working on testing StarOffice. What would your ideal test team look like? What management strategy would you use? Why? Outline a risk matrix
Select a few areas or functions Identify probability indicators How to define Consequence of an error?
ET Workshop v. 1.20 Test Management 2002 Amland Consulting 5-48

Summary (1)
Strengths:
Customer-focused, risk-focused Takes advantages of each testers strengths Responsive to changing circumstances Well managed, it avoids duplicative analysis and testing High Bug find rates

Blind Spots:
The less we know, the more we risk missing Limited by each testers weaknesses (can be mitigated with careful management) This is skilled work, juniors arent very good at it
ET Workshop v. 1.20 Test Management

Copyright 1996 2002 Cem Kaner

2002 Amland Consulting 5-49

Summary (2)
Exploratory testing in pairs, is a very effective test approach Use it as a complementary testing approach (it depends) Skilled testers can become very good exploratory testers. Novice testers need mentoring to do exploratory testing.
ET Workshop v. 1.20 Test Management 2002 Amland Consulting 5-50

Learn more
Test Training
www.testingeducation.org

Exploratory Testing:
Rapid Software Testing, by James Bach, www.satisfice.com james@satisfice.com Black Box Software Testing, by Cem Kaner, www.kaner.com kaner@kaner.com Amland Consulting, www.amland.no, stale@amland.no

People Issues, test techniques, inspections:

Grove Consultants, www.grove.co.uk + many, many more

ET Workshop v. 1.20 Test Management

2002 Amland Consulting

5-51

Presentation - Summary
1.

Introduction Test Management and Techniques ET Planning, Exec. and Documentation ET Styles ET Management

Introduction: Exploratory Testing and Risk, Why and when to use Exploratory Testing. Test Management, Test Techniques and Test Process ET Planning, Documentation and Execution. How to do Exploratory Testing Exploratory Test Styles How to do Exploratory Testing, continued Exploratory Test Management How to Manage Exploratory Testing and Teams

2.

3.

4.

5.

ET Workshop v. 1.20 Test Management

2002 Amland Consulting

5-52

References/Bibliography (1)
Amland, S., 1999. Risk Based Testing and Metrics, EuroSTAR '99 Preceding, Barcelona, Spain, http://www.amland.no/articles Amland, S., 2000. Risk-based testing: Risk analysis fundamentals and metrics for software testing including a financial application case study, Journal of Systems and Software, Volume 53, Issue 3, 15.9.2000, Elsevier publishing, UK. Argus, C. & Johnson, B., Ad Hoc Software Testing; Exploring the Controversy of Unstructured Testing, http://www.testingcraft.com/ad_hoc_testing.pdf Bach, J., 1999a. Risk-Based Testing. How to conduct heuristic risk analysis, Software Testing & Quality Engineering Magazine, November/December 1999, vol. 1, issue 6, http://www.stqemagazine.com. See also satisfice.com Bach, J. , 1999b. What is Exploratory Testing? www.satisfice.com Bach, Jonathan, 2000, Session-Based Test Management, STQE Magazine, 11/00, www.satisfice.com Beizer, Boris, Software Testing Techniques, Van Nostrand Reinhold. NY. 1990. Beizer, Boris, 1995, Black Box Testing, Wiley

ET Workshop v. 1.20 Test Management

2002 Amland Consulting

5-53

References/Bibliography (2)
Bereza-Jarocinski, B., 2000, Test and Decision Theory, Enea Data AB, bogb@enea.se, +46 50 709 714 293 (www.bbj.com.pl) Black, Rex, 1999. Managing the Testing Process, Microsoft Press. Buwalda, Hans, Janssen, Dennis, and Pinkster, Iris, 2001. Integrated Test Design & Automation Using The TestFrame Method, Addison Wesley, Copeland, Lee, Exploratory Planning, Sep. 3, 2001, StickyMinds.com Collard, Ross, 2002, to be published, rcollar@attglobal.net . Fenton, N.E. & Pfleeger, S.L., 1997. Software Metrics, a rigorous & practical approach, 2nd edition, International Thomson Computer Press. Fewster, M., Graham, D., 1999, Software Test Automation: Effective Use of Text Execution Tools, Addison-Wesley Gerrard, P., 1999, Risk-Based Testing, TEST Congress, London, UK, www.evolutif.co.uk Gilb, T., Principles of Software Engineering Management, Addison-Wesley, Wokingham

ET Workshop v. 1.20 Test Management

2002 Amland Consulting

5-54

References/Bibliography (3)
Kaner, C., Falk, Nguyen, 1999, Testing Computer Software, John Wiley & Sons Kaner, C., 2000. Rethinking Software Metrics, Evaluating measurement schemes, Software Testing & Quality Engineering Magazine, March/April 2000, vol. 2, issue 2. Kaner, C., 2001a, Black-Box Software Testing, training at Satisfice July 2001. Kaner, C., Bach, J., Pettichord, B., 2001b, Lessons Learned in Software Testing, John Wiley & Sons; http://www.testinglessons.com/, ISBN: 0471081124 Kaner, C., Bach, J. 2001c, Exploratory Testing in Pairs, presentation at STAREast, Orlando, FL, www.kaner.com Karolak, Dale Walter, Software Engineering Risk Management, IEEE Computer Society Press, 1996. Keefer, G., 2002. Extreme Programming Considered Harmful for Reliable Software Development, www.avoca-vsm.com Keith, Geordie, 2002, All Hands on Deck: How we used our whole company to test, STQE Magazine, vol. 4, issue 4, July / August 2002. Lyndsay, James and Eeden, Niel van, 2002, Adventures in Session-Based Testing, paper at StickyMinds.com presented at EuroSTAR 2002
ET Workshop v. 1.20 Test Management

2002 Amland Consulting

5-55

References/Bibliography (4)
Marick, Brian, 1995, The Craft of Software Testing, Prentice Hall PTR, ISBN 013-177411-5 Myers, Glenford, 1979. The Art of Software Testing, John Wiley & Sons Neumann, P.G., Computer-Related Risks, Addison-Wesley/ACM Press, ISBN 0201-55805-X, 1995, http://www.csl.sri.com/users/neumann/ Nguyen, 2000, Testing Applications on the WEB, John Wiley & Sons Ottevanger, Dr. I.B., 1999. A Risk-Based Test Strategy, Presented at STARWest 1999, http://www.sqe.com/startwest. Part of TMAP, IQUIP Informatica B.V. PO Box 263, 1110 AG Diemen, The Netherlands (www.iquip.nl). Pfleeger, S.L., 2000. Risky Business: what we have yet to learn about software risk management, Journal of Systems and Software, issue 11, 2000, Elsevier Publishing, UK. Robinson, H., Microsoft, Exploratory Modeling, http://www.testingcraft.com/exploratory-robinson.html, home page: http://www.geocities.com/model_based_testing/ Roden, Lloyd, 2001. How to build and lead an effective test team, tutorial EuroSTAR 2001, Grove Consultants, www.grove.co.uk, Lloyd@grove.co.uk
ET Workshop v. 1.20 Test Management 2002 Amland Consulting 5-56

References/Bibliography (5)
Schaefer, H., 1998. Surviving under time and budget pressure, keynote STAR West '98, schaefer@c2i.net, http://www.sqe.com/starwest, http://home.c2i.net/schaefer/ Vga, J., Amland, S., 2002. High Speed Web Testing, contribution to the book Software Quality and Testing in Internet Times, Editor: Dirk Meyerhoff et al, Springer-Verlag, Berlin Heidelberg, Germany 2002, ISBN:3-540-42632-9 Whittaker, J. and Jorgensen, A., 1999. Why software fails. ACM Software Engineering Notes, July. http://se.fit.edu/papers/ and www.stickyminds.com Whittaker, J. and Jorgensen A., 2002, How to Break Software: A Practical Guide to Testing, Addison-Wesley, ISBN: 0-201-79619-8, presented at EuroSTAR 2000 vstedal, E. . and Stlhane, T., 1992. A goal oriented approach to software testing, Reliability Engineering and System Safety. Elsevier Science Publishers Ltd., UK.
ET Workshop v. 1.20 Test Management 2002 Amland Consulting 5-57

Links
www.stickyminds.com www.satisfice.com www.kaner.com www.testingeducation.org www.pettichord.com www.amland.no home.c2i.net/schaefer/ www.bbj.com.pl http://www.stqemagazine.com http://www.testinglessons.com/ http://www.context-driven-testing.com/ www.grove.co.uk
ET Workshop v. 1.20 Test Management 2002 Amland Consulting 5-58

1. 2. 3. 4. 5.

Contact Details
Stle Amland, Amland Consulting Hulda Garborgsv. 2, N-4020 STAVANGER Norway Phone: +47 905 28 930 Fax: +47 51 58 55 24
e-mail: stale@amland.no WEB: www.amland.no
ET Workshop v. 1.20 Test Management 2002 Amland Consulting 5-59