WK1

Keynote Presentation
Wednesday 10/24/2007 8:45 AM JUMP TO: Biographical Information The Presentation

The Five "Doings" of Software Testing

Presented by: Mark Fewster and Dorothy Graham, Grove Consultants

Presented at: The International Conference on Software Testing Analysis and Review October 22-26, 2007; Anaheim, CA, USA

330 Corporate Way, Suite 300 , Orange Park, FL 32043 888-268-8770 904-278-0524 sqeinfo@sqe.com www.sqe.com

Dorothy Graham
Dorothy Graham is the founder of Grove Consultants in the UK, which provides advice, training and inspiration in software testing, testing tools and Inspection. Originally from Grand Rapids Michigan, she has lived and worked in the UK for over 30 years. Dorothy is co-author with Tom Gilb of "Software Inspection", Addison-Wesley, 1993, co-author with Mark Fewster of "Software Test Automation", Addison-Wesley, 1999, and co-author with Rex Black, Erik Van Veenendaal and Isabel Evans) of Foundations of Software Testing:: ISTQB Certification, Thomson, 2007. Dorothy was Programme Chair for the first EuroSTAR Conference in 1993. She has been on the boards of conferences and journals in software testing, and has been an active member of the British Computer Society's Specialist Interest Group in Software Testing since 1989. She was a founder member of the Software Testing Board of the Information Systems Examination Board (ISEB). She was a member of the working party that developed the ISTQB Foundation Syllabus. Dot is a popular and entertaining speaker at conferences and seminars worldwide. She was awarded the European Excellence Award in Software Testing in 1999.

Mark Fewster
Mark has 20 something years of industrial experience in software testing. Since joining Grove Consultants in 1993, he has provided consultancy and training in software testing, particularly in the application of testing techniques and test automation. He has published papers in respected journals and is a popular speaker at national and international conferences and seminars. Mark is co-author of the book "Software Test Automation with Dorothy Graham, published by Addison-Wesley. In 2006 he received the Mercury BTO Innovation in Quality Award.

STAR West 2007

The Five Doings of Software Testing

Prepared and presented by

Grove Consultants Mark Fewster

Llwyncynhwyra, Cwmdu Llandeilo, SA19 7EW, UK Tel: +44 1558 685180 email: mark@grove.co.uk

and

Dorothy Graham
40 Ryles Park Road Macclesfield SK11 8AH, UK Tel +44 1625 616279 email: dorothy@grove.co.uk

www.grove.co.uk
Grove Consultants, 2007

Five doings
What do we do when we are testing? - searching (for defects) - checking (software or aspects) - assessing (software or system) - measuring (software quality) - sampling (software)

Searching
in testing we search for - defects and clues to possible defects - evidence about what works examples outside testing - searching for something that has been lost - searching for evidence (detective, archaeologist) - fishing - treasure hunt (competition)

Characteristics of searching
searching is easier - if you know what to search for - if you know where to search - with a map - with a bright light can be difficult to know when to stop motivation is significant - affects effectiveness / efficiency - want to find defects / want to be searching

Knowing what to search for

knowing what helps select techniques - rounding errors -> boundary value analysis - combinations -> decision tables, all pairs - user navigation -> state transition knowing what helps select tools - low coverage -> data generation, random testing - performance -> load generation, monitoring - memory leaks -> dynamic analysis - coding defects -> static analysis

Knowing where to search

most complex - hides more defects - more chance of error most critical - most important, greatest impact new - not previously tested changed - correctness dependent - affected by changes elsewhere difficult / easy - harder to get right, more trivial mistakes most used - needs to be more reliable most visible - failures more damaging

Maps/models for searching in testing

Specification
Input Conditions Valid username F Valid password Account in credit Output Conditions Login accepted F Restricted access Tags A

Code
T T T F T T - F T F T T - T F B C D

Condition Month

Valid Tag Partition 6 -> 12 V1

Invalid Tag Partition <6 >12 non-int X1 X2 X3 X4 X5 X6 X7 X8 X9 X10 X11

Valid Tag Boundary 6 12 1 31 1 30 -6 207

Invalid Tag Boundary XB1 XB2 XB3 XB4 XB5 XB6 XB7 XB8 XB9

VB1 5 VB2 13 1 VB3 0 VB4 32 VB5 0 VB6 31 VB7 -7 VB8 208

Day (Month= 1 -> 31 7, 8, 10, 12) Day (Month= 1 -> 30 6, 9, 11) No. days

V2

<1 >31 non-int

V3

<1 >30 non-int

-6 -> 207 V4

< -6 >207

Feature Complexity updates 25 report 17 query 15 purge 12 input 10

Bright light for searching

a bright light - makes it easier to see what youre looking for knowing how to recognise a bug - makes it easier to see what youre looking for understanding the application / specification understanding the business domain / user needs
training, coaching, experience, discussions

having tools to assist in searching - makes searching easier, quicker, more reliable

Knowing when to stop searching

can always find more bugs - given more time and resource testing is resource constrained - run out of time, effort, machine resource stop when sufficient evidence found - that system will work successfully - that system cannot be used (in current state)

Motivation for searching

want to be searching? - enjoy searching, being thorough - appreciate value in confirming presence / absence want to find bugs? - in own work often no! - in others work often yes! improving motivation - team activity - rewards

Searching lessons for testing

know what you are searching for - and choose appropriate techniques e.g. test techniques, software maps/models know where to search - dont try to test everything be selective decide (in advance) criteria for stopping consider motivation for searching - not always a problem but can become one

Checking
in testing we check - software is working (a form of positive testing) - software against spec. / standards / checklists examples outside testing - camping checklist - report - car (MOT) - multiple choice exam

Characteristics of checking
multiple definitions - examine, investigate, make inquiry for accuracy, quality of progress - a (rapid) control to ensure accuracy, progress (check list) outcome - binary - yes / no, pass / fail

others (less appropriate) -pause, restrain, control - slow growth or progress - impede an opponent

implies - objectivity check against

predefined thing different people, same result

- brevity check this against that,

no more

- presumption software works check it works rather

than check it fails

Objectivity of checking
checking against something - checklist - standard - specification not opinion - outcome is pass/fail, ok/not ok predetermined - known, trusted, understood - used before? - others know what is and what is not checked - can be improved must be credible - must exist - otherwise cannot check - be respected / valued

Brevity of checking
cover checklist / standard / spec. - no more easy to do - with appropriate skills / knowledge quick to do - less thinking possibly can stop early - stop on first checklist failure (reason not to ship) - if software can be rejected further investigation - becomes assessing not checking

Checking - lessons for testing

use checking whenever appropriate - its quicker and cheaper (than other forms of test) - more objective (fewer disagreements) e.g. entry / exit criteria - quick and painless check with objective result - criteria previously agreed with all relevant parties improve - incrementally improve checklists, standards, etc.

Assessing
in testing we assess - the overall quality of the software / system - whether it is ready for release examples outside testing - buying a car - choosing a house - essay exam - hiring people - art work

Characteristics of assessing
definitions - estimate value of - determine amount of (tax, fine, damages, etc.) - judge the worth, importance of, evaluate outcome - on a scale, not binary - may not be conclusive implies - subjectivity use personal or

professional opinion different people, different result

- open ended no clear finish line - no prior knowledge of state of software might be good, could be

poor need to be prepared

Subjectivity of assessing
assessment using - personal opinion - professional opinion - consensus outcome - outcome may need supporting evidence - persuasive arguments different people - may give different assessment variable criteria - assessment criteria not all predetermined - may change (unwittingly) over time - not obvious to others what is and what is not assessed - can be challenged

Open-endedness of assessing
how much to assess? - broad scope - all aspects or just the most important? - which are the most important? how thorough? - equal depth - greater depth for most important areas proprietary knowledge - training, experience can take time/resource - more thinking, balancing - maybe more to consider than planned (e.g. impact of many defects) further investigation - focus more specific - more thorough

Assessing lessons for testing

assess after checking & only if needed clear assessment criteria - creditability of assessors need knowledge and expertise and be respected a more subjective view - dont stray into personal preference - justify and support your conclusions exploratory testing is assessment - regression testing is checking

Measuring
in testing we measure - quality of the software examples outside testing - table - car - driving skill - cabinets or shelves - dress-making

Characteristics of measuring
assign a value to something - one or more numbers - a qualitative measure (e.g. good, ok, poor) quality of measure can vary - simple guess to scientific accuracy appropriateness of metrics can vary - many aspects to measure, not all are relevant easier to measure to what has been measured before

Quality of measure
guess (professional estimate) - approximate, quick, better if visible - based on experience, comparison, information more detailed assessment - takes time, effort, resource - more accurate extensive assessment - much more time, effort, resource - more accurate, precise

Appropriateness of measure
depends on the objective - what do you want to know? - how many different measures are needed? in testing - we want to know the goodness of the software but often report the badness (e.g. number of bugs
found) historical: measure of old version of software

- we can predict (estimate) what we dont know yet

Measuring lessons for testing

software quality has many aspects - functionality, quality characteristics, defects one measure doesnt fit all! choose the quality of your measurement - gold / silver / bronze testing reporting - report goodness e.g. tests passed, predict future defect levels easier if youve done similar measure before

Sampling
in testing we are sampling - the software by sampling inputs test cases examples outside testing - election exit polls - product manufacturer - market surveys

Characteristics of sampling
choice of sample is critical - sample size generally, the larger the better - demographics must be representative we infer conclusions for the larger set - these can be wrong we can improve our sampling - use tools to increase sample size - use techniques (e.g. pre-qualify a larger sample)

Choice of sample
sample size - larger samples give more accurate results but cost more demographics - being representative is key techniques help - can change over time in testing - sample size often constrained by time / resource - demographics biased to most important

Inferring conclusions from sample

creditability of conclusions depend - on creditability of sample - on understanding of bigger picture what other issues could affect the outcome in testing - are the chosen tests representative? - do we understand how the software will be used?

Sampling lessons for testing

state sample size and selection criteria - helps communicate testing is not a perfect process justify selection criteria for your samples use sampling in reviews / inspections - having a more accurate measure of a sample can improve overall measure of the whole ensure good samples - representative at high level, random within? - use techniques to improve sample quality - use tools / technology to increase sample size

The Five Doings of Testing

Summary
testing involves activities that we do elsewhere - searching, checking, assessing, measuring, sampling understanding how we do these activities - can help improve the way we do them learn from similar tasks outside testing - can help avoid misunderstandings - can improve those activities within testing