Test Workflow Test Workflow

Michael Fourman

2001-10-27

Cs2 Software Engineering

Introduction Introduction
Verify the result from implementation by testing each build Plan the tests in each iteration Integration tests for every build within the iteration System tests for the end of the iteration Design and implement tests by creating test cases that specify what to test test procedures that specify how to perform the tests executable test components to automate the tests (if possible) Perform the tests and systematically handle the test results Defective builds are sent to other workflows (e.g., design and implementation) for defect fixing
Cs2 Software Engineering

2001-10-27

Context Context

Test System Procedures Plans Expected Results Results Evaluations Defects Test Components

Tester

System/Component Under Test

2001-10-27

Cs2 Software Engineering

Test Focus Test Focus

Core Workflows
Requirements

Phases
Inception Elaboration Construction Transition

Analysis

Design

Implementation

Test
Preliminary 1 2 n n+1 n+2 m m+1

Iterations
2001-10-27 Cs2 Software Engineering

Testing in the Software Life Cycle Testing in the Software Life Cycle
In general, whenever there is an implementation result, there is a test Test at each build Inception phase: initial test planning, prototype testing Elaboration phase: test architectural baseline Construction phase: significant testing at each build Transition phase: re-test fixes and regression tests Regression tests: in a new build, re-apply tests from old builds to make sure nothing broke in new build Evolve test model create new test cases for next build refine prior test cases into regression tests remove obsolete tests and corresponding test procedures and Software Engineering components 2001-10-27 Cs2

Artifacts Artifacts
Test model artifacts Test cases Test procedures Test components Test subsystem packages for complex test models Other artifacts Test Plan Defects Test Evaluation
Cs2 Software Engineering

2001-10-27

Test Model Test Model

Describes how executable components (e.g., builds) in the implementation model are tested by integration tests and system tests Describes how specific system aspects are to be tested e.g., usability testing of user interface or user manual

2001-10-27

Cs2 Software Engineering

Test Case Test Case

Specifies one way to test the system What to test (requirements) With which inputs Expected results Test conditions and constraints

2001-10-27

Cs2 Software Engineering

Test Case Test Case

Examples Test a use case or a scenario in a use case (black box) actor interaction, pre-conditions, action sequences, post-conditions Test a use-case realization (white box) verify interaction between components Installation tests, configuration tests, negative (abuse) tests, stress tests Organize similar tests in input/result matrix Family of test for different paths through a use case

2001-10-27

Cs2 Software Engineering

Test Cases Test Cases

Special tests of system as a whole Installation tests verify that the system can be installed on the customer platform and that it executes correctly when installed Configuration tests verify that the system works correctly in different configurations Negative tests (abuse tests) try to cause the system to fail Test outside the intended design, e.g., wrong configuration, wrong input, etc. Reveals weaknesses in design Stress tests show problems with insufficient or shared resources
Cs2 Software Engineering

2001-10-27

Test Procedures Test Procedures

Specify how to perform one or several test cases or parts of them Instructions for tester in manual test case Instructions for interacting with test automation tool to create executable test components, then instructions to integrate and execute those test components One test procedure may cover several test cases One test case may require several test procedures Test procedure instructions often reflect flow-ofevents descriptions, including input values, how to enter input values, and how to verify results

2001-10-27

Cs2 Software Engineering

Test Components Test Components

Automate one or several test procedures or parts Provide test inputs, control and monitor test execution, and possibly analyze and report results Developed with scripting languages, programming languages, or test automation tools test drivers, test harnesses, and test scripts Complex test cases and procedures may require formal analysis, design, and implementation modeling of the test components

2001-10-27

Cs2 Software Engineering

Test Plan Test Plan

Describes testing strategies, resources and schedule Strategy What kind of tests to perform for each iteration Test objectives Required level of test and code coverage Test acceptance criteria Trade-off cost of test vs. improved quality and reduced risk Test Plan may be defined in conjunction with the model artifacts from other workflows analysis (black box) design and implementation (white box) Tests may include artifact reviews before implementation tests
Cs2 Software Engineering

2001-10-27

Test Results Test Results

Results are the output data of the test and/or the behavior of the system during test Analysis of results: pass or defect Defect: a system anomaly (software fault or a defect found in artifact review) capture symptom in sufficient detail to allow problem resolution capture defect in problem and resolution tracking system Test evaluation: summary of overall test results test-case coverage, code coverage defects and defect status

2001-10-27

Cs2 Software Engineering

Test Workflow Activities and Workers Test Workflow Activities and Workers
Plan Test Design Test Evaluate Test

Test Engineer

Integration Tester

Perform Integration Test Perform System Test

System Tester

Component Engineer
2001-10-27

Implement Test
Cs2 Software Engineering

Workers Workers
Test designer: responsible for integrity of the test model plan tests: test goals and schedule select and define test cases, test procedures evaluate test results Component engineer: responsible for development of test components that automate some of the test procedures consider independent test

2001-10-27

Cs2 Software Engineering

Workers Workers
Integration tester: perform integration tests and report defects may be same person as system integrator in implementation workflow System tester: perform system tests and report defects focus on externally observable behavior consider users, beta customers, etc. as testers

2001-10-27

Cs2 Software Engineering

Activity: Plan Tests Activity: Plan Tests

Describe testing strategy Inputs: use case model, supplemental requirements, design model Develop test cases and procedures that have a minimum of overlap test the most important test cases test requirements associated with the highest risk Estimate resources needed for the testing effort, including human and system resources Schedule the testing effort
Cs2 Software Engineering

2001-10-27

Activity: Design Integration Test Cases Activity: Design Integration Test Cases
Derive test cases from use-case realizations design (component interaction) Develop tests with minimal overlap that test an interesting path or scenario through use case Primarily consider interaction diagrams Combinations of actor input, output and system start state that lead to interesting scenarios that employ the classes that participate in the diagrams

2001-10-27

Cs2 Software Engineering

Activity: Design System Test Cases Activity: Design System Test Cases
Test system as a whole Test combinations of use cases instantiated under different conditions Hardware configurations System loads Number of actors Database size etc. Prioritize combinations that are required or are likely to function in parallel are likely to influence each other if they function in parallel involve multiple processes frequently use system resources or use them in complex and unpredictable ways
Cs2 Software Engineering

2001-10-27

Activity: Design Regression Test Cases Activity: Design Regression Test Cases
Modify test cases from earlier builds to be used as regression tests in subsequent builds Make more flexible and resilient to changes to the system under test Specifically address previously found defects Automate where possible

2001-10-27

Cs2 Software Engineering

Activity: Identify and Structure Test Procedures Activity: Identify and Structure Test Procedures
Work test case by test case and suggest testing procedures for each one Reuse existing test cases May need to modify them to be reusable Align test procedures to subsystems under test Minimize overlap Isolates change to test resulting from change to subsystem

2001-10-27

Cs2 Software Engineering

Activity: Implement Test Activity: Implement Test

Create test components from test procedures With test automation tools Perform or specify actions using test automation tools Record actions as a test component, such as a Visual Basic script When programming explicitly Design and implement using test procedures as specifications Provide input and output data generation, visualization, interpretation and management utilities Use spreadsheets and database applications
Cs2 Software Engineering

2001-10-27

Activity: Perform Integration and System Test Activity: Perform Integration and System Test
Perform the integration tests relevant to a build Manually perform test procedures or execute test components Compare test results with the expected test results Investigate test results that deviate from the expected Faulty system and/or faulty test? Report defects to the component engineer responsible for the components likely to contain the fault Report defects to the test designers For use in evaluation of overall test results System tests are performed the same way as integration tests
Cs2 Software Engineering

2001-10-27

Activity: Evaluate Test Activity: Evaluate Test

Test designers prepare metrics of test acceptance with respect to system quality level Test completeness and coverage Percentage of test cases executed Percentage of code and paths tested Reliability Categorize defect severity Analyze trends of discovered defects and tests passed with expected results o Compare with prior experience to predict amount of testing still required to reach an acceptable defect level

2001-10-27

Cs2 Software Engineering

Activity:Evaluate Test Activity:Evaluate Test

Suggest actions Suggest further tests Relax test criteria Deliver parts of the system that passed; rework and retest parts that failed Document test completeness, reliability and actions

2001-10-27

Cs2 Software Engineering

Summary Summary
Test Model Test cases specify what to test Test procedures specify how to perform test cases Test components automate the test procedures Test Plan Test Evaluation Defects to be fed into other core workflows

2001-10-27

Cs2 Software Engineering