NYC Technology Forum

Introduction to Test Automation

11/2/07

Bill Rinko-Gay
Solutions Director, Software Quality Management

John Stephens
Practice Director, Software Quality Management

All rights reserved

Not to be reproduced without permission
 Objectives

 Get familiarized with basic concepts of Test

Automation

 Understand what it takes to implement

effective Test Automation

 Introduce techniques to emphasize

maintainability

2
What is Test Automation?
 Benefits of Automated Testing
Examples

 Run automated tests unattended (overnight)

 Reliable, consistent testing

 Increased speed in test execution

 Increased test coverage

4
 General Misconceptions About
Automated Testing

 Automation is simply “record and playback”

 Automation is the “silver bullet” to solve all testing

problems

 Return on Investment will be achieved immediately

 Automation will replace all manual test cases

5
 What is Test Automation?

 An enhancement on manual testing

 Utilizes technology to provide repeatable,

consistent testing of the Application Under
Test (AUT)

 Reduces costs and time for regression

testing

6
 Primary Goal of Automated Testing

 To reduce number of tests to be run

manually, not to eliminate manual testing
altogether

Automated Testing is an enhancement upon

existing methodology and manual process

7
So, You Want to Implement Test
Automation…
 Challenges
in Automated Testing

 Test Automation is a development effort!

 Automation is time consuming to create (and maintain, if

poorly implemented)

 Limitations of the testing tools and developer expertise

 Automation of the wrong tests for the given constraints

 Unrealistic expectations
9
 Test Automation
Checklist
 Criteria for Automation
 Tool – AUT compatibility
 Test Automation Requirements
 Implementation Design
 Stable, isolated test environment
 Test data
 Implementation and coding standards
 Maintenance
10
 What to Automate?
Choosing Ideal Manual Test Cases
 Criteria for Automating a Test Case
Why Automate?

Criticality Test Cases that are business-critical and/or high risk

Repetitiveness Test Cases that are to be performed repeatedly

Test Cases that are tedious to perform manually

Tedious

Test Cases that require considerable amount of time

to perform manually
Time Consuming

Selecting the right tests to automate is very critical to

12 effective implementation!!
 Test Automation
Checklist
 Criteria for Automation
 Tool – AUT compatibility
 Test Automation Requirements
 Implementation Design
 Stable, isolated test environment
 Test data
 Implementation and coding standards
 Maintenance
13
 Tool - AUT Compatibility

 Conduct Proof of Concept of Tool on AUT

 Navigate through functional areas to test

 Verify Object Recognition of AUT

 Determine whether tool selection is valid

14
 Test Automation
Checklist
 Criteria for Automation
 Tool – AUT compatibility
 Test Automation Requirements
 Implementation Design
 Stable, isolated test environment
 Test data
 Implementation and coding standards
 Maintenance
15
 Test Automation Requirements
Definition

The set of criteria that the automated

test suite must meet to be successful -
the requirements for an automation
project.

16
 Test Automation Requirements
What are the Success Criteria for the project?
 Meeting the goals and objectives of the automation effort

 Tool capabilities

 Capabilities and skills of the test automation team

Example Requirement: The test tool shall test the login

functionality of a web application, on a Windows 2000
platform, and run unattended simultaneously on three
PCs.

17
 Planning for Test Automation

Test Automation Test Automation

Requirements Design/Approach

Test Automation Requirements help drive and determine

the approach and design of the implementation.

18
 Test Automation
Checklist
 Criteria for Automation
 Tool – AUT compatibility
 Test Automation Requirements
 Implementation Design
 Stable, isolated test environment
 Test data
 Implementation and coding standards
 Maintenance
19
 Test Automation Methodology

Analysis
&
Strategy

Design

Construction

Execution &
Results
Management

20
 Implement a Test Automation
Architecture

 Automation Architecture encompasses the

environment for execution and development of an
automated test suite. It is the overall scheme in which
the test scripts will be developed.

21
Designing the Test Automation
Architecture
Maximizing Maintainability
 Automation Architecture Design
Options
 Capture/replay

 Framework-based

 Data-driven

 Keyword-driven

 Hybrid

23
 Design Goals

 Evaluate feasible design options for implementation

 Anticipate likely changes to AUT

 Consider skill level of team members developing and/or

inheriting the automated scripts

 Design of automated tests should address test

automation requirements

24
 Typical Design Components

Compiled module containing application-specific routines

Function Library and functions

A script that initiates automated test execution

Driver Script

A script that performs a discrete portion of a business process to

Subscripts be tested

A script that sets up and standardizes test automation

Startup Script environment

Data sources (such as spreadsheets) used to store test data for use
Data Table during test execution
25
 Test Automation
Checklist
 Criteria for Automation
 Tool – AUT compatibility
 Test Automation Requirements
 Implementation Design
 Stable, isolated test environment
 Test data
 Implementation and coding standards
 Maintenance
26
 Automation Test Environment

 Stable and isolated environment is important for

reliable execution of automated tests

 GUI must be identical to Production

27
 Test Automation
Checklist
 Criteria for Automation
 Tool – AUT compatibility
 Test Automation Requirements
 Implementation Design
 Stable, isolated test environment
 Test data
 Implementation and coding standards
 Maintenance
28
 Test Data Strategy
 Create a test environment that is restorable to a known
constant, such that the test cases can be repeated each
time

 Identify the procedure required for backup and recovery

of data

Test Data is critical to reliable automated testing!!

29
 Creating Test Data
Various Methods

 Create test data through application by

doing data entry

 Import test data from production

environment, if available

 Develop automated test scripts to generate

test data as prerequisite

30
 Test Automation
Checklist
 Criteria for Automation
 Tool – AUT compatibility
 Test Automation Requirements
 Implementation Design
 Stable, isolated test environment
 Test data
 Implementation and coding standards
 Maintenance
31
 Coding Standards

 Specify coding practices and naming

conventions

 May be based on standards in place for

project’s AUT developers

32
 Test Automation
Checklist
 Criteria for Automation
 Tool – AUT compatibility
 Test Automation Requirements
 Implementation Design
 Stable, isolated test environment
 Test data
 Implementation and coding standards
 Maintenance
33
 Maintenance
Emphasize robustness and flexibility

 Start small and build on

 Build modular components

 Stay within scope of automation requirements

 Adhere to defined standards and conventions

34
 Maintenance
Building an Automation Test Team

Ideal characteristics of an Automation Engineer:

 Must be a good tester

 Has programming background

 Must be aware of maintenance and reliability

 Must be motivated to develop and create automated test

scripts

35
 Learning Objectives
Revisited

 Get familiarized with basic concepts of Test

Automation

 Understand what it takes to implement

effective Test Automation

 Introduce techniques to emphasize

maintainability

36
37