March 31 – April 2, 2009

Managed Test Automation
Hans Buwalda, CTO, LogiGear Corporation
"Managing Mr. Playback . . ."

Introduction

• Who is here?
industries? involvement in testing? "Mr. Playback" wants to know...

Managed Test Automation | © 2009 by LogiGear Corporation. All rights reserved. Made available under the EPL v1.0

Objectives
• Implement effective and maintainable automated tests • Managing testing and test automation • Main focus: concepts, techniques and experiences
To a lesser extend: in-depth treatments, full coverage of possible issues

• We will use the "Action Based Testing" method as a guideline today, as an example of an approach:
Representative of keyword-based approaches Incorporates most of the key ideas and concepts Home match for your teacher

• Not everything will be equally interesting, or accessible, to everybody
Managed Test Automation | © 2009 by LogiGear Corporation. All rights reserved. Made available under the EPL v1.0

Outline of this Tutorial
4 1
INTRODUCTION AND OVERVIEW ORGANIZATION & MANAGEMENT

TEST DEVELOPMEN T

2

3

TEST AUTOMATION

Managed Test Automation | © 2009 by LogiGear Corporation. All rights reserved. Made available under the EPL v1.0

Introduction and Overview

Managed Test Automation | © 2009 by LogiGear Corporation. All rights reserved. Made available under the EPL v1.0

Testing Under Pressure
DEADLINE

specification

development

testing

Managed Test Automation | © 2009 by LogiGear Corporation. All rights reserved. Made available under the EPL v1.0

Testing Under Pressure
DEADLINE

specification

development

testing

Develop tests in time: • Test design • Auditing, acceptance • Preparations • Automation
Managed Test Automation | © 2009 by LogiGear Corporation. All rights reserved. Made available under the EPL v1.0

The Challenges for a Test Process
• Testing should be effective • Testing should be efficient • Testing should be under control • Testing should be fun

Managed Test Automation | © 2009 by LogiGear Corporation. All rights reserved. Made available under the EPL v1.0

Organize it Well
EXTERNAL Customers Vendors Government Agencies Publicity End User Departments Marketing/ Sales Test Development Quality Assurance Technology/ Infrastructure INTERNAL

Management

Test Automation

System Development After Sales/ Help Desk Production

Managed Test Automation | © 2009 by LogiGear Corporation. All rights reserved. Made available under the EPL v1.0

Existential Questions
• Why test? • Why not test? • Why automate tests? • Why not automate tests?

???

Managed Test Automation | © 2009 by LogiGear Corporation. All rights reserved. Made available under the EPL v1.0

Why test?
• People expect us to do • Somebody wants us to • Increases certainty and control
Showing absence of problems

• Finds faults, saving time, money, damage
Showing presence of problems

Managed Test Automation | © 2009 by LogiGear Corporation. All rights reserved. Made available under the EPL v1.0

Why not test?
• It costs time and money • You might find problems . . . • We forgot to plan for it • We need the resources for development • It is difficult • It's hard to manage
Managed Test Automation | © 2009 by LogiGear Corporation. All rights reserved. Made available under the EPL v1.0

Why Automate Tests?
• It is more fun • Can save time and money
potentially improving time-to-market

• Reduces involvement of valuable specialists
"cans" application business and application knowledge

• Consolidates a structured way of working
when established as integral part of system development process

• Speeds up development
shorter turnaround between changes more security
Managed Test Automation | © 2009 by LogiGear Corporation. All rights reserved. Made available under the EPL v1.0

Why not Automate?
• Can rule out the human elements
promotes "mechanical" testing might not find "unexpected" problems

• More sensitive to good practices
pitfalls are plentiful

• Creates more software to manage • Needs/uses technical expertise in the test team • Tends to dominate the testing process
at the cost of good test development

Managed Test Automation | © 2009 by LogiGear Corporation. All rights reserved. Made available under the EPL v1.0

Different Points of View
1. Does the system comply with the requirements
the “positive” angle

2. Are there any problems we should know about
the “negative” angle

3. Will the system work in practice
the “pragmatic” angle

Managed Test Automation | © 2009 by LogiGear Corporation. All rights reserved. Made available under the EPL v1.0

What are Typical Problems
• No clear direction
• what, when, how, who, ...

• Focus on tools and technology • Test design is not well thought through
• main driver for success of failure

• Automation lacks architecture, is not transparent and hard to manage • Testing is underestimated or avoided
• testing is difficult and expensive • management is often in denial • it looks unattractive to spend money on testing

• Test automation is underestimated
• test automation can be very difficult • "use the head first, before using the hands"

Managed Test Automation | © 2009 by LogiGear Corporation. All rights reserved. Made available under the EPL v1.0

The 5% Challenge for Test Automation
• No more than 5% of all test cases should be executed manually

• No more than 5% of all efforts around testing should involve automating the tests

Managed Test Automation | © 2009 by LogiGear Corporation. All rights reserved. Made available under the EPL v1.0

Approaches to Automate Test Execution
• Record & Playback • Scripting • Keywords

see also: "Software Test Automation", Mark Fewster and Dorothy Graham

Managed Test Automation | © 2009 by LogiGear Corporation. All rights reserved. Made available under the EPL v1.0

1. Record and Playback
select window "Logon" enter text "username", "administrator" enter text "password", "testonly" push button "Ok" select window "Main" push button "New Customer" expect window "Customer Information" select field "First Name" type "Paul" select field "Last Name" type "Jones" select field "Address" type "54321 Space Drive" . . .
Managed Test Automation | © 2009 by LogiGear Corporation. All rights reserved. Made available under the EPL v1.0

1. Record and Playback
• Can help well to generate early versions of individual steps in a larger automated script • Also useful to get to know a test tool, or the UI of the system under test • In its pure form maintainability of generated test scripts can become a major liability • Puts the testers in the automation role
"loose a good tester, gain a poor programmer"

Managed Test Automation | © 2009 by LogiGear Corporation. All rights reserved. Made available under the EPL v1.0

2. Scripting
Test Case Design

TEST DESIGNER

Test Case Automation

AUTOMATION ENGINEER

Test Case Execution

MR. PLAYBACK
Managed Test Automation | © 2009 by LogiGear Corporation. All rights reserved. Made available under the EPL v1.0

2. Scripting
... Function EnterCustomer ( FirstName, LastName, Address ) Click("New Customer"); ExpectWindow("New Customer"); EnterField("First Name", FirstName); EnterField("Last Name", LastName); EnterField("Address", Address); ... Click("OK"); End Function ... Function Main Logon(); EnterCustomer("Mary", "Jones", "123 Palm Drive"); EnterCustomer("Paul", "Franklin", "321 Regent Street"); ... LogOff(); End Function ...
Managed Test Automation | © 2009 by LogiGear Corporation. All rights reserved. Made available under the EPL v1.0

2. Scripting
• Structured process
test cases are designed first and then automated automation is managed as a programming task done by a specialized automation engineer

• Engineering approach
emphasis on structure and re-use similar to regular programming projects common tasks (like "logon") are factored out data can be made external to the programming

• High quality, limited coverage
approach is good (see above), but is essentially a waterfall approach efforts are substantial, burdening engineering capacity hard to keep manageable, in particular in large projects sensitive to changes in the behavior of the application under test

Managed Test Automation | © 2009 by LogiGear Corporation. All rights reserved. Made available under the EPL v1.0

3. Keywords, Action Based Testing
number name quantity

new product

P-9009
number

Sledge Hammer
quantity

5

add quantity add quantity add quantity

P-9009 P-9009 P-9009
number

20 3 6
quantity

check quantity

P-9009

34

function action_AddQuantity .... end
… add quantity check quantity new product …

function action_CheckQuantity .... end function action_NewProduct .... end

Managed Test Automation | © 2009 by LogiGear Corporation. All rights reserved. Made available under the EPL v1.0

3. Keywords, Action Based Testing
• Describe test cases as a series of actions with keywords and arguments, in a sheet like format • Externalize those descriptions out of the scripting • Let the testers (not the automation engineer) define the keywords, as by-product of the test design • Organize the test cases into "test modules", each with a clear and differentiated scope • Focus the automation efforts on the actions, not on the test cases
Managed Test Automation | © 2009 by LogiGear Corporation. All rights reserved. Made available under the EPL v1.0

3. Keywords, Action Based Testing
Test Development Plan
Break down

Test Module 1
Test Objectives

Test Module 2
Test Objectives

Test Module N

...

Test Objectives

Test Cases

Test Cases

Test Cases

Automate

Actions

ACTION AUTOMATION
Managed Test Automation | © 2009 by LogiGear Corporation. All rights reserved. Made available under the EPL v1.0

Example of a Test Module, "high level"

test objective test objective

Managed Test Automation | © 2009 by LogiGear Corporation. All rights reserved. Made available under the EPL v1.0

Example of a Test Module, "low level"
TEST MODULE VERSION Product Maintenance 1.0

...
TEST CASE requirement test objective click enter click check message Test Case 03 add 003
source

Use an invalid number A number must be divisable by 9
control destination

main
window

new
control

new product
value

new product
source

number
control

P9001

new product
text

process

Invalid product number.

Managed Test Automation | © 2009 by LogiGear Corporation. All rights reserved. Made available under the EPL v1.0

Comparing Formats
Most values are implicit. The tester has to figure them out during execution…. Execution instructions are repeated in multiple test cases

classic format

Enter a user id that is greater than 10 characters, enter proper information for all other fields, and click on the "Continue" button Enter a User Id with special character's), enter proper information for all other fields and click on the "Continue" button Enter the information, with a password of 4 characters and click on the "Continue" button

There should be an error message stating that "User Id must be less than 10 characters". An error message should be displayed indicating that "User Id cannot contain some special characters". Check for an error message saying: "Password must contain at least 5 characters".

keywords
user id message

check registration dialog check registration dialog check registration dialog

aaaaabbbbbc
user id

User Id must be less than 10 characters
message

résoudre
password

User Id cannot contain some special characters
message

test

Password must contain at least 5 characters

Managed Test Automation | © 2009 by LogiGear Corporation. All rights reserved. Made available under the EPL v1.0

Common Terminology
• "Keywords"
usually base level, interacting with the user interface related to control classes (like textboxes, lists and menu's) examples: "click", "enter", "select"

• "Business Templates"
typically high level, making automation scripts re-usable examples: "login", "enter transaction"

• "Actions"
can be at multiple levels (interface, business oriented, test logic, ...) defined as part of the test design (not per se to re-use existing automation) a method like Action Based Testing allows earlier defined actions to be used to define higher level action words
Managed Test Automation | © 2009 by LogiGear Corporation. All rights reserved. Made available under the EPL v1.0

Potential benefits of keywords
• More tests, better tests
more breadth more depth

• • • • • •

Fast, results can be quickly available
the design directly drives the automation

Easier to involve business subject matter experts
the action format allows for easy readability by non-technical users

Less efforts for automation
"script free" in most cases

Automation more stable and maintainable
limited and manageable impact of changes in the system under test

Develop tests more early in the life cycle
deal with execution details later

...

Managed Test Automation | © 2009 by LogiGear Corporation. All rights reserved. Made available under the EPL v1.0

Risks of the Keyword Methods
• Sometimes hailed as panacea, complications can be underestimated
often treated as a technical "trick"

• The method needs understanding and experience to be successful
pitfalls are many, and can have a negative effect on the outcome

• Lack of structure can lead to lack in manageability
in particular in the test design

• Automation might take more time and costs than expected • Impact of changes might be higher than expected • Results can be disappointing and boring
Managed Test Automation | © 2009 by LogiGear Corporation. All rights reserved. Made available under the EPL v1.0

Combining Approaches . . .
• Use keywords for the automation-ready description of test cases

• Use scripting to set up structured automation for the actions

• Use record and playback to record actions
Managed Test Automation | © 2009 by LogiGear Corporation. All rights reserved. Made available under the EPL v1.0

Some "Paradigm Shifts" . . .
• From testing to: test development • From process oriented to: product oriented • From being part of a waterfall to: agile test development • From automation-centric to: test-centric • From selective automation to: integral automation • From requirement-based to: scenario-based

Managed Test Automation | © 2009 by LogiGear Corporation. All rights reserved. Made available under the EPL v1.0

Test Development
ORGANIZATION & MANAGEMENT

TEST DEVELOPMEN T

TEST AUTOMATION

Managed Test Automation | © 2009 by LogiGear Corporation. All rights reserved. Made available under the EPL v1.0

Recent Software Error

Six months before Global Surveyor fell silent, engineers sent up incorrect software commands that disabled its solar panels . . .

source: www.cnn.com/2007/TECH/space/04/13/mars.surveyor.ap/index.html

Managed Test Automation | © 2009 by LogiGear Corporation. All rights reserved. Made available under the EPL v1.0

State your Objectives . . .
... TO-3.51 The exit date must be after the entry date ...
// Test Case: Billy Goodfellow // Test Objective: TO-3.51 Function EnterBilly Press "New Employee" Enter Field "Name", "Goodfellow" Enter Field "First Name", "Billy" Enter Field "Entry Date", "2002-10-02" Enter Field "Exit Date", "2002-10-01" ... Press "Process" Find Text "Exit date must be after entry date" if Found then Result passed else Result failed, "No error message" ... End Function
Managed Test Automation | © 2009 by LogiGear Corporation. All rights reserved. Made available under the EPL v1.0

State your Objectives . . .
... TO-3.51 The exit date must be after the entry date ...

test objective

TO-3.51
name entry date exit date

enter employment check error message

Bill Goodfellow 2002-10-02 2002-10-01 The exit date must be after the entry date.

Managed Test Automation | © 2009 by LogiGear Corporation. All rights reserved. Made available under the EPL v1.0

Traceability of Results
Requirements

Test Objectives

Test Cases

Test Results
Managed Test Automation | © 2009 by LogiGear Corporation. All rights reserved. Made available under the EPL v1.0

Possible Project Life Cycle
Plan of Approach

Management

Test Development Plan
intake

IT Department Automation: - design - research - development - testing - support

Test Module Development Business Users

Development

Test Module Acceptance Stake Holders Execution and Follow Up

Focus on: - actions - interface handling - technology - never on tests

Tech Specialists

Managed Test Automation | © 2009 by LogiGear Corporation. All rights reserved. Made available under the EPL v1.0

Examples of Time Lines
System under test becomes available in phases
Sep Oct Nov Dec Jan Feb Mar Apr

Test Development Plan Test Development Automation Execution and follow up

System under test becomes end of the cycle
Mar Apr May Jun Jul Aug Sep Oct Nov Dec Jan Feb Mar

Test Development Plan Test Development Automation Execution and follow up

Managed Test Automation | © 2009 by LogiGear Corporation. All rights reserved. Made available under the EPL v1.0

The Three “Holy Grails” of Test Design

The right breakdown into test modules

The right approach per module

The proper level of test specification

Managed Test Automation | © 2009 by LogiGear Corporation. All rights reserved. Made available under the EPL v1.0

Grail 1: Breakdown, Divide and Conquer
Mission, Requirements, System Design, Use Cases, Priorities, Planning, Budget, Stakeholders, ...

Identify test modules • priority and planning • what, when, who, how

Test Module 1 Analysis Development Execution Maintenance

Test Module 2 Analysis Development Execution Maintenance

Test Module N Analysis

...

Development Execution Maintenance

Managed Test Automation | © 2009 by LogiGear Corporation. All rights reserved. Made available under the EPL v1.0

Grail 1: Example breakdown
• UI oriented tests
does function key F4 work does listbox xyz the right values is the tab order correct

• • • • • • •

Do individual functions work
like transactions in a financial system

Alternate paths in use cases
like cancel a transaction

End-to-end tests Simulating business processes Tests with specific automation needs
like multi station tests

Tests of non-UI functions High ambition level tests (aggressive tests)

Managed Test Automation | © 2009 by LogiGear Corporation. All rights reserved. Made available under the EPL v1.0

Question ???
• Can you give a set of breakdown criteria for tests?

Managed Test Automation | © 2009 by LogiGear Corporation. All rights reserved. Made available under the EPL v1.0

Grail 1: Breakdown Criteria
• Straightforward Criteria
Architecture of the system under test (which part are we testing) Functionality and other requirements Kind of test (navigation flow, quality attributes, ...) Ambition level (smoke test, regression, requirement based, aggressive, …)

• Additional Criteria
Stakeholders Complexity of the test Technical aspects of execution Planning and control Risks involved

Managed Test Automation | © 2009 by LogiGear Corporation. All rights reserved. Made available under the EPL v1.0

14.39-

Grail 1: What is a good breakdown?
• Well differentiated and clear in scope • Balanced in size and amount • Reflects the level of tests • Modules are stand-alone, not dependent on each other • Fitting the priorities and planning of the project
Managed Test Automation | © 2009 by LogiGear Corporation. All rights reserved. Made available under the EPL v1.0

Example Test Development Plan

Managed Test Automation | © 2009 by LogiGear Corporation. All rights reserved. Made available under the EPL v1.0

Review Process with Stake Holders
START

Test Team sends draft Module to Stake Holder

Stake Holder reviews: - coverage - correctness

no changes needed? yes Stake Holder returns notes: - additions - corrections

Stake Holder returns notice of approval

Test Team receives and processes notes

Test Team marks the Module as "Final"

END

Managed Test Automation | © 2009 by LogiGear Corporation. All rights reserved. Made available under the EPL v1.0

Grail 2: Approach per Test Module
• Organize the test design process around the test modules • Plan the test module:
when to develop: is enough specification available when to execute: make sure the functionality at action level is well-tested and working already

• Process:
analysis of requirements formulation test objectives creating test cases

• Identify stakeholders and their involvement:
users, subject matter experts developers auditors

• Choose testing techniques if applicable:
boundary analysis, decision tables, transition diagrams, soap opera testing, ...
Managed Test Automation | © 2009 by LogiGear Corporation. All rights reserved. Made available under the EPL v1.0

Grail 2: Eye on the ball, Scope
• Always know the scope of the test module • The scope should be unambiguous • The scope determines many things:
what the test objectives are which test cases to expect what level of actions to use what the checks are about and which events should generate a warning or error (if a “lower” functionality is wrong)

Managed Test Automation | © 2009 by LogiGear Corporation. All rights reserved. Made available under the EPL v1.0

Grail 2: Examples of Testing Techniques
• • • • Equivalence class partitioning
any age between 18 and 65

Boundary condition analysis
try 17, 18, 19 and 64, 65, 66

Error guessing
try Cécile Schäfer and test the sorting of the name list

Exploratory
"Exploratory testing is simultaneous learning, test design, and test execution", James Bach, www.satisfice.com

Model-based
see Harry Robinson: www.model-based-testing.org see Better Software Magazine (STQE Magazine), March 2003

Error seeding
deliberately injecting faults in a test version of the system, to see if the tests catch them handle with care, don't let the bugs get into the production version

Managed Test Automation | © 2009 by LogiGear Corporation. All rights reserved. Made available under the EPL v1.0

Grail 2: More Testing Techniques
• State transitions • Decision tables • Soap Opera Testing

Recommended: • "Why Software Fails", James Whittaker and Alan Jorgensen, and • "Testing Computer Software", Cem Kaner, Jack Falk and Hung Nguyen

Managed Test Automation | © 2009 by LogiGear Corporation. All rights reserved. Made available under the EPL v1.0

State Transition Diagram
initial

doors closed
signal tampering signal incorrect correct

first check

alert state
time-out

second check
correct incorrect

doors open

self destruct

Question: what test cases would you create to cover these situations?
Managed Test Automation | © 2009 by LogiGear Corporation. All rights reserved. Made available under the EPL v1.0

State Transition Table
Car Safety System, Model 007, Luxury Version*
NR 1 2 3 4 5 6 7 8 9 10 STATE doors closed doors closed first check first check doors open second check second check self destruct alert state alert state EVENT signal tampering correct code incorrect code close signal correct code incorrect code anything time out signal OUTPUT NEXT beep first check hi-lo alert state green doors open hi-lo second check beep doors closed green doors open hi tone self destruct boom oblivion hi tone self destruct beep first check

*certain insurance restrictions might apply, please check with your agent
Managed Test Automation | © 2009 by LogiGear Corporation. All rights reserved. Made available under the EPL v1.0

Example of a Decision Table
dialog data correct large order online customer show error and resume give discount 20% give discount 3% notify marketing send to priority execution display price info n y n n y n y y y n y y y

x x x x x x x x x x x

TEST CASE 1 TEST CASE 2 TEST CASE 3 TEST CASE 4 TEST CASE 5
Managed Test Automation | © 2009 by LogiGear Corporation. All rights reserved. Made available under the EPL v1.0

Question ???
• What is wrong with the following pictures?

Managed Test Automation | © 2009 by LogiGear Corporation. All rights reserved. Made available under the EPL v1.0

No Y2K Problems in Auckland Airport??

Managed Test Automation | © 2009 by LogiGear Corporation. All rights reserved. Made available under the EPL v1.0

Anything wrong with this instruction ?

You should change your battery or switch to outlet power immediately to keep from losing your work.

Managed Test Automation | © 2009 by LogiGear Corporation. All rights reserved. Made available under the EPL v1.0

New Arrival in the Realm Of Techniques

Equivalence Island
Managed Test Automation | © 2009 by LogiGear Corporation. All rights reserved. Made available under the EPL v1.0

Soap Opera Testing

The “Mechanical Approach” for Test Development • Start with a (preferably long) list of requirements • Make a test case for every requirement • Use a standardized test technique to translate the requirements into test cases • Hire (many) people to perform the tests manually • ….

Managed Test Automation | © 2009 by LogiGear Corporation. All rights reserved. Made available under the EPL v1.0

Some Pitfalls With a Too Mechanical Approach • No fun at all • Inhibiting creativity • Coverage is focused at single requirement level • Any defects should probably have been found in an earlier test • Suggests false sense of control • Test set hard to maintain • Doesn’t catch mistakes in the requirements
Managed Test Automation | © 2009 by LogiGear Corporation. All rights reserved. Made available under the EPL v1.0

Soap Operas
Ashley hears about Jack's deposit when he thought he had to go. Victoria lectures her father about what's wrong with him and Nikki but Victor advises her that it's none of her business Olivia learns Dru has no regrets about leaving and takes great satisfaction in having Lily as her companion. Dru then asks Olivia why she is raking Malcolm over the coals. Stopping by Gina's, Nikki spots Brad and sits with him, admitting she doesn't want to be alone tonight. Victor stops by Mack's party at the Crimson Lights. Ashley takes a home pregnancy test. Worried about Billy, Raul makes call and J.T. claims he doesn't know where Billy is. Raul rushes over and finds Billy out cold in the snow. Raul worries when he can't find a pulse. . . .
Managed Test Automation | © 2009 by LogiGear Corporation. All rights reserved. Made available under the EPL v1.0

Some Other Examples
Pension Fund William starts as a metal worker for Industrial Enthropy Incorporated in 1955. During his career he becomes ill, works part time, marries, divorces, marries again, gets 3 children, one of which dies, then his wife dies and he marries again and gets 2 more children…. World Wide Transaction System for an International Bank A fish trade company in Japan makes a payment to a vendor on Iceland. It should have been a payment in Icelandic Kronur, but it was done in Yen instead. The error is discovered after 9 days and the payment is revised and corrected, however, the interest calculation (value dating)…
Managed Test Automation | © 2009 by LogiGear Corporation. All rights reserved. Made available under the EPL v1.0

Properties of Soap Operas
• Using a recurring theme, with “episodes”

• About “real life”

• But condensed

• And more extreme

Managed Test Automation | © 2009 by LogiGear Corporation. All rights reserved. Made available under the EPL v1.0

Soap Operas for Testing
• Define a scope of the test to develop • Identify with the business environment • Identify which elements would make things difficult • Draft test cases (typical some dozen lines) • Write them down in modules
Managed Test Automation | © 2009 by LogiGear Corporation. All rights reserved. Made available under the EPL v1.0

Soap “Count the Goodies”
In this episode, the Goodies have won $750,000 in the lottery. They decide to buy that big house on the corner of the street that they’ve always dreamed about. Father, Bing Goody, goes to the banking office to make the necessary arrangements. Of course, the bankruptcy that Bing had two years ago complicates the approval, but Jim, the guy at the bank, is very helpful and even sells Bing on the idea of a beautiful vacation property in Mexico. Bing has a brilliant idea. Why not ask neighbor Jones if he is interested in coowning the vacation property? Like everything else with Mr. Jones, his credit is perfect.
Managed Test Automation | © 2009 by LogiGear Corporation. All rights reserved. Made available under the EPL v1.0

The Story Continues…
One week later: After a small party with some new friends, only about $720,000 of the prize money is left, so the mortgage arrangements need to be changed. Jim, Mr. Jones, and Bing decide to meet the next morning to make the necessary adjustments. Since it was a great party, they are not at their best when they meet, so many mistakes are made. They correct some immediately, others they miss and will need to fix later. They find that an additional second mortgage on Mr. Jones’s home is needed in order to still qualify for the vacation home.
Managed Test Automation | © 2009 by LogiGear Corporation. All rights reserved. Made available under the EPL v1.0

What Will Happen Next?

See it all . . . . . . in the next test module ! !

Managed Test Automation | © 2009 by LogiGear Corporation. All rights reserved. Made available under the EPL v1.0

What This Soap Can Test
• • • • • • • • • • • Entering a customer with a big family Mortgage arrangements including a first and a second house Property abroad Both a primary residence and a vacation property Weighing of the income of a second owner Co-ownership on the second house only Establishing a second mortgage to finance a vacation property Bankruptcy two years ago Modification of the down payment Correcting errors upon entry Corrections after the application has been processed

Managed Test Automation | © 2009 by LogiGear Corporation. All rights reserved. Made available under the EPL v1.0

“Killer Soaps”

let me wash this thing...

• Aggressive tests, aimed at finding hidden problems • Run when everything else has passed • Ask experienced users and subject matter experts for ideas

Managed Test Automation | © 2009 by LogiGear Corporation. All rights reserved. Made available under the EPL v1.0

Test Analysis / Test Creation

Test Analysis - what do we want - what do we need separate and
TEST OBJECTIVES

Test Creation - confrontation with reality - put it to the test

relate

TEST CASES

Managed Test Automation | © 2009 by LogiGear Corporation. All rights reserved. Made available under the EPL v1.0

Questions to Answer with a Test Collection

mechanical

soaps

soaps + objectives

1

Does the system comply with the requirements

*** * **

* *** **

*** *** ***

Are their any problems 2 (defects and/or failures) we should know about 3 Will the system work in practice

Managed Test Automation | © 2009 by LogiGear Corporation. All rights reserved. Made available under the EPL v1.0

Grail 3: Specification Level, choosing actions
• Scope of the test determines the specification level • As high level as appropriate, as little arguments as possible
Use default values for non-relevant arguments

• Clear names (usually verb + object works well) • Manage the Actions • Document the Actions • By-product of the test design

Managed Test Automation | © 2009 by LogiGear Corporation. All rights reserved. Made available under the EPL v1.0

Grail 3: What is usually not good . . .
• Too high level to understand the test: “Test all”
start test everything end

• Low-level actions (automation) in a high-level test:
type press key type click button set code "Hans" "<tab>" "Buwalda" "OK" Fc122x XX33

• Hard to understand “insiders only” language • Tons of arguments not relevant to the test
85 arguments for an action have sensible default values for common arguments like zip codes, phone numbers etc
Managed Test Automation | © 2009 by LogiGear Corporation. All rights reserved. Made available under the EPL v1.0

Designing Actions/Specification Level
• Scope of the test determines the specification level • As high level as appropriate, as little arguments as possible
Use default values for non-relevant arguments

• Clear names (usually verb + object works well) • Manage the Actions
register them, keep them organized

• Document the Actions
intention, arguments, assumptions, conditions, implementation notes, ...

Managed Test Automation | © 2009 by LogiGear Corporation. All rights reserved. Made available under the EPL v1.0

Levels of Actions
• High level actions to capture functions in a business process
hide unneeded details
• always hunt down details

usually implemented with lower level actions

• Low level actions to access the system interface
either built-in in a action based tool like TestArchitect or tailor created as part of an interpretation “harness”, using scripting and/or record&playback

Managed Test Automation | © 2009 by LogiGear Corporation. All rights reserved. Made available under the EPL v1.0

Example of “Going High Level”

Managed Test Automation | © 2009 by LogiGear Corporation. All rights reserved. Made available under the EPL v1.0

Example of “Going High Level”

section key navigate key navigate wait for controls loaded check breadcrumb check focus capture cost values

Get cost values
key

F7 5
window

view account info
text

# view account info breadcrumb
window field

view account info
start cost

cancel
cost in cost out net cost

>> starting cost

>> total cost in

>> total cost out

>> total net cost

Managed Test Automation | © 2009 by LogiGear Corporation. All rights reserved. Made available under the EPL v1.0

Example of “Going High Level”

Managed Test Automation | © 2009 by LogiGear Corporation. All rights reserved. Made available under the EPL v1.0

Example of “Going High Level”
section key navigate key navigate locate page tab wait for controls loaded check breadcrumb select enter value click wait for controls loaded capture sequence num Capture initial number
key

F7 3
page tab

Scan Criteria
window

search
text

# search breadcrumb
window control value

search
window

scan direction
control

Backward
value

search
source

business date match
control

# bus date

search
window

go

search results
store as

>> seq num

Managed Test Automation | © 2009 by LogiGear Corporation. All rights reserved. Made available under the EPL v1.0

Example of “Going High Level”

Managed Test Automation | © 2009 by LogiGear Corporation. All rights reserved. Made available under the EPL v1.0

Test Automation
ORGANIZATION & MANAGEMENT

TEST DEVELOPMEN T

TEST AUTOMATION

Managed Test Automation | © 2009 by LogiGear Corporation. All rights reserved. Made available under the EPL v1.0

Components of Automation
• Tests and test data
what needs to happen to prove or disprove the functionality under test best to keep this separate from the technical part of the automation

• Programming/scripting
technical specification of the individual steps of the automation

• Interface information
what is where on screens, web pages, etc shield volatile physical details

• UI and non-UI automation functions
the actual (usually tool provided) commands to work with the interfaces of the system under test

Managed Test Automation | © 2009 by LogiGear Corporation. All rights reserved. Made available under the EPL v1.0

Automation: Implementing Actions
• In a keyword driven approach the automation focuses on the actions:
as opposed to programming test cases in general this will lead to fewer and shorter scripts etc in the automation

• In the Action Based Testing method actions are implemented in three ways:
built-in in the tool, like “click”, “enter”, “check” etc defined in an “action definition”, using other (lower level) actions
• similar to functions in a programming language

programmed in a programming or scripting language

Managed Test Automation | © 2009 by LogiGear Corporation. All rights reserved. Made available under the EPL v1.0

Example of an "Action Definition"

action is implemented in the action definition

arguments used in the main test are represented by their names

"low level" actions to do real work: 1. built-in or 2. scripted

Managed Test Automation | © 2009 by LogiGear Corporation. All rights reserved. Made available under the EPL v1.0

Automation Overview
Test Modules
• test objectives • test cases

Actions
• shield (volatile) technical details of an interface • provide "logical" names for controls and their classes • split up in lower level actions • provided defaults values for arguments

Action Definitions

Interface Definitions (UI, SQL, OLE, ...)

Low Level Actions 90% + Test Architect Built-in Additional Scripting (C#, VB, ...)

Managed Test Automation | © 2009 by LogiGear Corporation. All rights reserved. Made available under the EPL v1.0

Doing the Count Down . . .

6000
250

3000
200
20 22

Test Cases

Actions

Programmed Actions

In a good application of the keywords approach a large increase in test cases (like doubling the amount) should result in a modest increase in actions, and a minor increase, if any, in programmed actions.
Managed Test Automation | © 2009 by LogiGear Corporation. All rights reserved. Made available under the EPL v1.0

Programming
• Often called "scripting", but much like regular programming • Traditional playback tools mostly have built-in scripting languages
often Basic look-a-likes, WinRunner's TSL is a C look-a-like

• More and more tools support existing programming languages
like Visual Basic

• Regular languages can in some cases do the job too
in particular for non-gui's for example scripting languages like Python and Ruby but also Java, C++, VB, C#, etc

Managed Test Automation | © 2009 by LogiGear Corporation. All rights reserved. Made available under the EPL v1.0

Functionality of an Action Interpreter

Function Interpret
Interpreter Report

While not end of file do Read next line Split the line into arguments Look up the action in the "action list" Execute the action Print the results of this line in the report End of the while loop End

Managed Test Automation | © 2009 by LogiGear Corporation. All rights reserved. Made available under the EPL v1.0

“Atomic and Generic”
• In ABT key recommendations for programmed actions are to make them:
atomic, only one small task per action generic, maximum re-use

• Avoid programming test cases or high level actions
split them up in well-defined low level actions

Managed Test Automation | © 2009 by LogiGear Corporation. All rights reserved. Made available under the EPL v1.0

Automation: example test lines

System Under Test

Test Lines

Managed Test Automation | © 2009 by LogiGear Corporation. All rights reserved. Made available under the EPL v1.0

Automation, step 1: make "Action Functions"

identify the UI element

perform the operation

identify the UI element

perform the check

Managed Test Automation | © 2009 by LogiGear Corporation. All rights reserved. Made available under the EPL v1.0

Automation, step 2: "Divert" to the action functions

call this function for each line in the test module

Managed Test Automation | © 2009 by LogiGear Corporation. All rights reserved. Made available under the EPL v1.0

Interface Information and Interface Handling
• Handling interfaces is the essential challenge
automation works "outside in" interfaces are not necessarily designed with test tools in mind

• Common interfaces
GUI Web API's, with methods and/or functions message protocols, on basis of TCP/IP or SS7 embedded software databases with SQL files (like in batch runs)

Managed Test Automation | © 2009 by LogiGear Corporation. All rights reserved. Made available under the EPL v1.0

Identifying Elements of an Interface
• All elements have properties
can be used for identification and verification

• Some elements can be easily identified
like buttons, checkboxes, frames

• Other elements have only volatile and/or ambiguous properties
like text boxes, list boxes, combo boxes can differ across runs, or across versions/builds

• Text labels tend to be "invisible"
only bitmap

Managed Test Automation | © 2009 by LogiGear Corporation. All rights reserved. Made available under the EPL v1.0

Interface Definitions
• Quickly define how interfaces must be accessed

• Commonly used for GUI's, however applicable to other interfaces as well

• Typically kept in a table or spreadsheet
separate from the test modules

Managed Test Automation | © 2009 by LogiGear Corporation. All rights reserved. Made available under the EPL v1.0

Components of Interface Definitions
• Interface entity
window web page text screen 3270 page (typically based on the HLLAPI standard) class member api function message database table

• Interface element
button text box link field argument ....
Managed Test Automation | © 2009 by LogiGear Corporation. All rights reserved. Made available under the EPL v1.0

Example Interface Definition
window control value

enter enter enter enter click

add dialog add dialog add dialog add dialog
source

number name description quantity
control

1234567 saw Cut through things 500

add dialog

ok

INTERFACE ENTITY interface entity setting interface element interface element interface element interface element interface element interface element

add dialog title
ta name

Add A New Item
ta class label

number name description quantity
ta name

textbox textbox textbox textbox
ta class

Item Number Name Description Quantity On Hand
caption

ok cancel

button button

Process Cancel

Managed Test Automation | © 2009 by LogiGear Corporation. All rights reserved. Made available under the EPL v1.0

Some Tips to Get Stable Automation
• Make the system under test automation-friendly
developers are not always inclined to do that, but it pays off ask to add specific property values to the GUI interface controls for automated identification
• like “accessible name” in .Net and Java, or “id” in Web controls

• • •

Use "active" timing
wait based on conditions in the system under test, avoid fixed “sleeps”

Test your automation
develop separate tests for the actions themselves

Use automation to identify differences between versions of the system under test
test in a loop if all control can still be found

Keep an eye on the test design
unsuccessful automation virtually always can be traced back to poor test design

Managed Test Automation | © 2009 by LogiGear Corporation. All rights reserved. Made available under the EPL v1.0

"Accessible Name" property

Managed Test Automation | © 2009 by LogiGear Corporation. All rights reserved. Made available under the EPL v1.0

Testing the automation . . .
• The automation is a service provider to the testers:
provide working actions provide accurate up to date interface definitions

• In each project consider having the following folders: Automation Tests Action Tests Interface Tests
Managed Test Automation | © 2009 by LogiGear Corporation. All rights reserved. Made available under the EPL v1.0

Cross Platform Testing
• • No longer absolute dominance of Windows and IE Diversity in:
operating systems
• Windows (XP, Vista, Windows 7), Linux (variety of distributions), Mac

widget toolkits
• .Net, InfraGistics, QT, Java, STK+, Flex, ...

browsers
• Internet Explorer, FireFox, Safari, Chrome,... • within browsers: Ajax, JavaScript, Java, VB Script, Flash, ...

non-traditional platforms
• Windows Mobile, Android, Symbian, RIM, ... • Netbooks!

non-client platforms
• Cloud computing, SOA, ...

This trend will continue:
even if Windows and/or IE keep dominating the alternatives can't be ignored much longer, diversity is the name of the game

Managed Test Automation | © 2009 by LogiGear Corporation. All rights reserved. Made available under the EPL v1.0

Cross Platform Automation
• The key ingredient for success is design
test design is even more the decisive factor automation needs to be layered and flexible tooling needs to plug in into the overall model
UI LEVEL TESTS HIGH LEVEL TESTS API LEVEL TESTS

HIGH LEVEL ACTIONS

LOW LEVEL ACTIONS

GENERIC CONTROLS AND OPERATIONS

PLATFORM 1

. . .

PLATFORM n

Managed Test Automation | © 2009 by LogiGear Corporation. All rights reserved. Made available under the EPL v1.0

Non-UI Testing
• Examples
application programming interfaces (API’s) embedded software protocols files, batches databases command line interfaces (CLI’s) multi-media

• Impact is mainly on the automation
test design should in most cases be transparent towards the specific interfaces

Managed Test Automation | © 2009 by LogiGear Corporation. All rights reserved. Made available under the EPL v1.0

Multiple System Access
Test Modules, driving either one or multiple interfaces

Automation Scheme

protocol access

UI access

API access

System (part) Under Test

database access

Managed Test Automation | © 2009 by LogiGear Corporation. All rights reserved. Made available under the EPL v1.0

Device Testing
Testing Host
ABT Automation

Device
Mini Agent

Interface Info

Software Under Test

Managed Test Automation | © 2009 by LogiGear Corporation. All rights reserved. Made available under the EPL v1.0

Testing a Service Architecture or Protocol

Automation

SEND

RECEIVE Interface Definitions stash (optional)

CLOUD, SOA, ...

Managed Test Automation | © 2009 by LogiGear Corporation. All rights reserved. Made available under the EPL v1.0

Testing the Cloud, testing the SOA
• A lot of business as usual for testing • Test everything always
puts an extra need on high coverage automation not necessarily time to "maintain" not-working automation

• Where is what, and is it safe
is data where you think it is, and what you think it is are protocols and safety measures working

• Various perspectives
unit level, service level, client level, user level testing functions, testing interfaces, testing protocols

• Expect the unexpected, "jungle testing" • We still have to learn
Managed Test Automation | © 2009 by LogiGear Corporation. All rights reserved. Made available under the EPL v1.0

"Jungle Testing"
• Expect the unexpected
prone to unexpected requests prone to unexpected situations in data prone to attacks how does a generic design respond to a specific unexpected event?

• Difference in thinking
coding bug: implementation is different from what was intended/specified jungle bug: system does not respond well to an unexpected situation

• To prepare
study the matter (common hack attacks, ...) make a risk analysis make time to discuss it (analysis, brainstorm) involve people who can know use "exploratory testing" (see James Bach's work on this) use an agile approach for test development applies equally to testing, requirements, design
• testing should focus on the specific attacks
Managed Test Automation | © 2009 by LogiGear Corporation. All rights reserved. Made available under the EPL v1.0

From the Headlines

• Bugs can be data driven • They can happen in real time
the earlier you see them the better

• There will be material and immaterial damage • Your tests won't be perfect
sometimes unexpected simply means unexpected learn, generalize, prevent the bad bug, and its family, for the future 20/20 hindsight is an acceptable technique...
Managed Test Automation | © 2009 by LogiGear Corporation. All rights reserved. Made available under the EPL v1.0

Handling Batch Processes
Test Module
Transfer Check Balance Check Balance Jones Jones Williams Williams -100 100 100
1

Batch Input File
Jones Williams
2

100

1

Batch Run
2

"Delayed Checks"
3

Batch Output File
Jones Williams -100 100

Check Balance Check Balance

Jones Williams

-100 100

Managed Test Automation | © 2009 by LogiGear Corporation. All rights reserved. Made available under the EPL v1.0

Multi-media: The "Play List" Approach
• The testers focus on a play list
items are played/shown while the machine tells what it is the tester should see/hear the tester acknowledges/falsifies the system stores checksums of those items that were ok
• efficient and relatively easy to store and handle

Checksums can be kept for files, bitmaps etc

• Next time only the changed items go on the play list again
when the checksum is different from the previous run/version in most systems this will only be a limited amount of items

Managed Test Automation | © 2009 by LogiGear Corporation. All rights reserved. Made available under the EPL v1.0

Performance Testing
• The topic is complex, many components and phenomena • To formulate tests in actions can be surprisingly straightforward • Often performance testing isn't testing, but more close to research • The three controls you can/should address:
hardware (equipment, infrastructure, etc) software (programs, database models, settings, etc) demands (1 second can be 10 times more than 2 seconds)
See also: "Web Load Test Planning", Alberto Savoia, StickyMinds.com
Managed Test Automation | © 2009 by LogiGear Corporation. All rights reserved. Made available under the EPL v1.0

Data Driven Testing
• Separate test logic from the data • Have the data, including expected outcomes, in a separate file or table
can also be generated during test execution, for example extracted from a database

• Use placeholders in the test case • Works for both scripted and keyword driven approaches

Managed Test Automation | © 2009 by LogiGear Corporation. All rights reserved. Made available under the EPL v1.0

Data Driven Testing, Example

Managed Test Automation | © 2009 by LogiGear Corporation. All rights reserved. Made available under the EPL v1.0

Topics Test Management

ORGANIZATION & MANAGEMENT

TEST DEVELOPMEN T

TEST AUTOMATION

Managed Test Automation | © 2009 by LogiGear Corporation. All rights reserved. Made available under the EPL v1.0

Organization
• Much of the success is also gained or lost in how you organize the process
who does test design who does automation what to outsource, what to keep in-house

Write a plan of approach for the test development and automation
scope, assumptions, risks, planning methods, best practices tools, technologies, architecture stake holders, including roles and processes for input and approvals team ... Test design is a skill . . .

Assemble the right team
testers, lead testers automation engineer(s) managers, ambassadors, ...

Automation is a skill . . . Management is a skill . . .

Managed Test Automation | © 2009 by LogiGear Corporation. All rights reserved. Made available under the EPL v1.0

Plan of Approach (Example)
• Test assignment • Approach • Activities • Products • Test Environment • Planning • Risks • Project organization • External dependencies • Costs and benefits
Managed Test Automation | © 2009 by LogiGear Corporation. All rights reserved. Made available under the EPL v1.0

Typical Time Allocation
TEST DEVELOPMENT

efforts

AUTOMATION

time
Managed Test Automation | © 2009 by LogiGear Corporation. All rights reserved. Made available under the EPL v1.0

Embedding QA
• Clear division of tasks and responsibilities:
testers (who will now become “test developers”) users, power users, subject-matter specialists test automation engineers (do they work for you?) developers other stakeholders

• Decision making and communication
who's call is it to pass a system for release what happens with bugs who does which testing what whistles to blow/not blow, when

• Flexibility
testing is a "2nd order dependency" activity firmness on the integrity of provided information (but be practical)
Managed Test Automation | © 2009 by LogiGear Corporation. All rights reserved. Made available under the EPL v1.0

Question ???
• What are good skills for a tester?

Managed Test Automation | © 2009 by LogiGear Corporation. All rights reserved. Made available under the EPL v1.0

Skills for the Team
• Typical skills needed
testing (real testers) subject matter
• beware of "guru's", don't allow single individuals to monopolize key knowledge

technology (often underestimated and then weak spot) engineering practices (like configuration management) soft skills (communication, managerial, diplomacy)

• Acquiring the skills
training, reading, practicing, coaching interest, curiosity hiring, borrowing overall picture counts, not everyone needs to know everything

Managed Test Automation | © 2009 by LogiGear Corporation. All rights reserved. Made available under the EPL v1.0

The Tester as a Detective
• Looking for the bad guys (bugs) • Specific personality traits:
authority, respected inconspicuous, not threatening able to look past excuses and stories mindful of details
Mr. Authority

Mr. Inconspicuous

Mr. No Nonsense

Mr. Details

Managed Test Automation | © 2009 by LogiGear Corporation. All rights reserved. Made available under the EPL v1.0

Tester Needs Hunting Instincts

Managed Test Automation | © 2009 by LogiGear Corporation. All rights reserved. Made available under the EPL v1.0

Off-shoring
• Can a keyword driven approach be done offshore? • If yes, what would it take? • If no, why not?

???

Managed Test Automation | © 2009 by LogiGear Corporation. All rights reserved. Made available under the EPL v1.0

Off-shoring Keyword Automation
• Potential
can get a lot more tests at a lower cost
• potential of double cost benefit: 1. off-shore, 2. automation • risk of double backlash: 1. bad design, 2. bad automation

flexibility, particularly for "bursty" projects
• ramp up quickly, scale back quickly

• Pitfalls
little "ownership" with the system under test and the subject matter under test a keyword method is a subtle approach (see the rest of this tutorial...). This makes outsourcing challenging: mind the "Automation Cliff" . . . very hard to go back when things turn out disappointing

• What to do
keep local ownership provide clear direction, constant attention and coaching supervise, supervise, supervise ask to create example test modules first and review them carefully
Managed Test Automation | © 2009 by LogiGear Corporation. All rights reserved. Made available under the EPL v1.0

Organization and Project Level

ACTIVITIES AT ORGANISATION LEVEL

ACTIVITIES AT PROJECT LEVEL

ACTIVITIES AT PROJECT LEVEL

ACTIVITIES AT PROJECT LEVEL

Managed Test Automation | © 2009 by LogiGear Corporation. All rights reserved. Made available under the EPL v1.0

Beyond the projects: Test Governance
TEST BUSINESS POLICIES AND DIRECTIONS
(significance, position, organization)

TEST COMPETENCE MANAGEMENT (improvement focus)
pilot projects training coaching resource mediation test quality assurance test environments testware management vendor contacts

TEST PROJECT MANAGEMENT (solution focus)
Test Strategy & Planning Test Design Test Development Test Execution Test Maintenance

initiation, know how, support, ...

experiences, products, people, ...

Managed Test Automation | © 2009 by LogiGear Corporation. All rights reserved. Made available under the EPL v1.0

management & control

Test Policy

• Corporate point of view on testing (or software quality)

• Not a common topic in many organisations

• However can be useful to have

Managed Test Automation | © 2009 by LogiGear Corporation. All rights reserved. Made available under the EPL v1.0

Policy Model
Business Environment

Organisation Environment (Including Policies)

IT Organisation And Policies

Organisation and Policies of Testing
Managed Test Automation | © 2009 by LogiGear Corporation. All rights reserved. Made available under the EPL v1.0

How to Get It
• Think about it yourself • Identify the stake holders • Use existing policy documents (if any) • Discuss it:
Within the testing department Externally (IT, “business”, stakeholders)

• If possible start low profile and end high profile • Policies should be recognized and understood • Consider the use of an “incremental” document
Managed Test Automation | © 2009 by LogiGear Corporation. All rights reserved. Made available under the EPL v1.0

Typical Issues (1)
• What is the significance of testing and what may it cost • Connection to critical success factors of the company • Do we have problems • When should testing be done in the life cycle • Who should be involved (test development, assessment, reporting) • Who is responsible
Managed Test Automation | © 2009 by LogiGear Corporation. All rights reserved. Made available under the EPL v1.0

Typical Issues (2)
• How to deal with testing expertise • Centralized / decentralized • Methods and tools • Migration to a desired situation • Automation or not

Managed Test Automation | © 2009 by LogiGear Corporation. All rights reserved. Made available under the EPL v1.0

Examples of policy statements
• Testing and test automation are regarded as business critical • Testware (developed tests and automation) is to be treated as business asset • User friendliness is a major priority in testing • Test Automation is used as much as possible • QA is responsible for timely and accurate information on system quality, independent of other stakeholders • System Development is responsible for system quality • Releasing a system is a business responsibility
Managed Test Automation | © 2009 by LogiGear Corporation. All rights reserved. Made available under the EPL v1.0

Homework . . .
• • • • • • • • • • • • • • Cem Kaner, Hung Nguyen and Jack Falk, Testing Computer Software 2nd Edition, Wiley Cem Kaner, James Bach, and Bret Pettichord, Lessons Learned in Software Testing Dorothy Graham and Mark Fewster, Automating Software Testing, Addison Wesley Hans Buwalda, Action Figures (on model-based testing), STQE Magazine, March 2003 Hans Buwalda, and Maartje Kasdorp, Getting Automated Testing Under Control, STQE Magazine, November Issue 1999 Hans Buwalda, Dennis Janssen and Iris Pinkster, Integrated Test Design & Automation, Addison Wesley Hans Buwalda, Soap Opera Testing (article), STQE Magazine, February 2005 Hans Buwalda, Soap Opera Testing (keynote), STAR East 2000 Hans Buwalda, Testing with Action Words, Abandoning Record and Playback, Eurostar 1996 Hans Buwalda, The Three Holy Grails of Test Development, Quality Week 2001 Hans Buwalda, QA All Stars, Building Your Dream Team, Better Software, September 2006 Hans Buwalda, The 5% Solution, Software Test & Performance Magazine, September 2006 Hung Nguyen, Michael Hackett, Brent K. Whitlock, "Happy About Global Software Test Automation“, Happy About, 2006 Hung Nguyen, Robert Johnson and Michael Hackett, Testing Applications on the Web 2nd Edition, Wiley

Managed Test Automation | © 2009 by LogiGear Corporation. All rights reserved. Made available under the EPL v1.0

Sign up to vote on this title
UsefulNot useful