Fundamentals of Test Execution

Session 1: Fundamentals of Test Execution

© 2007, Cognizant Technology Solutions. All Rights Reserved. The information contained herein is subject to change without notice.

C3: Protected

About the Author
Created By: Rajarajan(225558); Balasubramanian(226466); Elangovan(208633); Venkatakrishnan(102083); Rohit(158977);Vishnu Priya(179360); Mathuram(105686) 3-12 years of Experience in Software Testing Version 5.0 and 22 Feb 2011

Credential Information: Version and Date:

2

Icons Used
Hands on Exercise

Questions

Tools

Coding Standards

Test Your Understandin g

Reference

Try it Out

A Welcome Break

Contacts

3

Fundamentals of Test Execution Session[1]: Overview  Introduction: This session explains the various activities performed during the test execution phase and the process to log the test results and the defect management methodology 4 .

Fundamentals of Test Execution Session[1]: Objective  Objective: After completing this chapter. you will be able to: » Describe various activities involved during Test Execution » Describe and interpret a Test Execution Plan » Describe Test Environment and the need for it » Explain Build Management and Verification Process » Describe the process of Test Execution » Describe Triaging and Defect Management » Differentiate Test Execution for Fresh development and Maintenance Testing 5 .

Test Execution: An Introduction  The Test Execution involves: » Initial Planning for Test Execution » Build Verification Process » Test Case Execution » Test Log Creation » Defect Reporting and Tracking » Defect Triaging » Updating Traceability Matrix » Re Testing of Defects » Regression Testing » Test Execution Status reporting Planning Activities for Test Execution Fig: Test Execution Process 6 .

Test Execution: An Introduction(Contd.)
 Follows the Test Design phase in STLC  Process of executing a series of test cases as defined in the test plan to validate if the application meets the requirements  The application is tested and the defects are reported to the development team  The fixes provided by the development team are retested and ensured that they do not recur, the fix does not introduce any new defects  Can be manual or automated using a tool

7

Planning for Test Execution
 Below activities need to be taken care before the start of Test execution
» Test Cases Review and Sign Off
• All the test cases that are identified for the specific level and type of testing need to be approved and signed off by the respective stake holders

» Test Cycles
• The number of rounds the testing will be conducted and the scope of testing at each level need to be identified in the Test plan and agreed upon by all the stake holders

8

Planning for Test Execution(Contd.)
» Test Lab set up
• The methodology for deploying the build in the respective test environment need to be devised and approved by the development, testing teams and the customer • Test Lab set up also includes creation of a test suite with the test cases which need to be executed in a specific sequence
– – – – Creation of Cycles for each level of testing Identification of Smoke/Build verification test cases Identification of Test Suite(collection of test cases) per cycle Frequency of Build delivery per cycle

» Task allocation / Run plan
• The allocation of the test cases to be executed need to be done based on the skill set and the availability of the testers

9

Planning for Test Execution: Execution Run Plan Execution Run Plan 10 .

Sample Execution Run Plan Sample 11 .Sample Execution Run Plan Execution Run Plan .

Planning for Test Execution(Contd.) » Risks and Mitigation Process • The potential risks that are identified for Test execution during the course of the project are revalidated and the availability of the mitigation and contingency plan for each of these risks need to be ensured » Escalation Process • The escalation process if an unforeseen event occurs during test execution need to be arrived and agreed upon by the stake holders 12 .

Need for a Test Environment  Testing an application hosted in the desktops/Servers used by the developers for coding purposes is not recommended for the below reasons » Both testers and developers will have access to the code » Behavior of the application may be unstable if the developers work on the code while the testers are testing » The developers may have used simulated programs during coding and the testers (being unaware of this) may also use the same for testing. This may result in huge failure when the application is deployed in the production environment » Parallel or overlapping data update by various teams 13 .

Need for a Test Environment (Contd.)  Mirror the production environment as closely as possible  The availability of the test environment exclusively for various types and levels of testing is always recommended but is subject to every individual project like » Entire test environment is exclusive » Database is the same but the hardware used is different » A sub set of the data will keep changing for every cycle and the master data remains the same 14 .

Tools like Eclipse. Crystal Reports) » Hardware required(Ex. server and application » Appropriate level of access to external application if the testing scope covers for the same 15 .Test Environment Configuration  Configuring of Test Environment can include setting up of the following: » Software required(Ex. Special interfaces like Handheld Device. Touch Screen) » Server to host the application in the desired configuration » Database setup/Server Setup to run the application » Appropriate level of access for test team to access Database.

) » Application to be tested » Test management tool set up » Access to the remote desktops and the connectivity to them » Contact Details of the customer’s helpdesk should be available during the testing if the environment is slow » Environment availability – All the environments may not be available full time during test execution.Test Environment Configuration (Contd. There may be exceptions like • An application will be running only during US business hours • An application will be down for maintenance during US public holidays 16 .

Test Environment: Test Management Tool Set Up  Test Management Tool is a software used to plan and execute testing activities like capturing requirements and test cases. managing defects. collecting metrics and analyzing the reports  Version of the tool or the instance agreed upon in the Test Plan needs to be installed in the test environment desktops  Project has to be created in the tool to manage the test activities and right level of access should be given to all test and development users  All requirements should be uploaded into the tool by Business Analyst / Testing team 17 .

Test Environment: Test Management Tool Set Up (Contd.)  Cycles for execution needs to be created in the tool under the project » Cycles agreed upon in the Test Plan is considered and may vary during the test execution » Cycle period or dates should be mentioned and it should be mapped with the test cases that we plan to execute  The tool may also be integrated with the third party tool 18 .

Test Environment: Database Set Up  The test environment should be as close to the production environment in terms of » Database(DB) set up » Levels of users defined and the no. of users defined for each level » Volume of data in the database » Type of data  Detailed Environment Requirement has to be addressed and tracked in a separate document like  If there are multiple test environments. the above mentioned points need to be ensured in all the environments 19 .

it is always better to ensure if the credentials provided works and has the appropriate levels of access » In some cases. This may require us to reschedule the run plan for execution  Batch Job dependencies (if there are any) need to be identified and planned accordingly 20 . the credentials provided may be valid only for a limited period of time.Test Environment: Database and Test User Set Up  Test Users set up » The Appropriate roles identified as test data for testing need to be created and obtained from the customer for all the instances of test environments » Once the test data is received.

Test Environment: Test User set up Rights to access DB Server Application Server Testers External Application 21 .

Countries applicable for the application » » » Secondary data can also be obtained from the customer if the application (previous version) already exists in Production If it is a new version of application. Secondary data can be created and customized during testing Test team is responsible for ensuring the test data availability before test execution • Secondary / Transaction data 22 .Test Data Set Up in the Test Environment  Primary / Secondary Data » Primary Data is generally referred to the master data and is obtained from customer. Example for Primary data tables are Employee Data.

Test Data Management in Test Environment  Test data should be available before the test execution  The data used for every cycle of testing may vary like » The entire database is refreshed » A subset of data is retained and reused » Only the master data is retained  Tools can also be used for creating test data  Batch queries should be created for insertion/ Updation / Deletion of data in the tables  Ownership of data has to be clearly defined among and within the teams 23 .

Test Data Management in Test Environment (Contd.)  Sample Queries for cleaning up and inserting the data » » » » »  Delete from empdetail Delete from emp Insert into emp(table columns)(table Values) Insert into empdetail(table columns)(table Values) Update can be done as required Note: While doing deletion. secondary/transaction data should be deleted first and then primary but it is vice versa during insertion 24 .

Build Management  For every cycle of test execution. the development team shares a controlled version of software (Build) which needs to be deployed in the respective test environment and tested  Each build is associated with a version number and a release note attached  The release notes of the build primarily includes information of the features of the software included in the build and the deployment details 25 .

Build Management (Contd. Program Manager)  The version of the build is incremented for every change in the build 26 .)  Frequency of build delivery to the project team is as per the Test plan and in consensus with the stake holders of the project  The files/DB files/Batch Programs used to generate a build are controlled by a configuration tool/ team assigned for it  Build Manager is responsible for communicating to the stake holders on the release of the build to test with release notes (In Some Projects.

Build Verification Process Development Team Build the compiled code into software Build Manager Adds the release notes Release Notes Test Team Build Verification using Smoke / Sanity Test BVT Failed BVT Pass Test Team Test Execution 27 .

compiled and built as a single unit by the developers  The Build Manager will add the release notes to this version of the software build and release it for testing  The Testing team runs the identified Smoke and Sanity test cases on the build.Build Verification Process (Contd. the build will be rejected 28 . once it is ready  If the smoke/sanity test cases pass with the desired results.)  The units of software developed is integrated. build will be accepted and the testing team will proceed with the Test Execution  If the smoke/sanity test fails with critical defects that the team is not able to proceed with testing.

Importance of Release Notes  Release notes includes details on » Version of the build » Features included / not included in the build » Defects that are fixed / not addressed in the build  Testers need to test only those features that are mentioned clearly in the release notes 29 .

Test Your Understanding Test Your Understandin g 30 .

Questions on Build Management Release notes should be sent by ? a) Build Manager b) Program Manager c) Test Manager What is the percentage of BVT test cases have to be passed ? a) 85% b) 90% c) Depends on the project BVT means ? a) Smoke b) Regression c) SIT 31 .

Test Execution Test Case  Test Log document to log actual results Identification of test cases assigned to each tester for the day Tester understands the dependency between test cases and identifies the sequence among them Tester ensures the prerequisite of each test case is met before execution Tester uses the test data identified for the test case during test execution Executes the test case / Retest if the defect is fixed Does the Application behave as desired ? Log Test Results with proof of execution Update Traceability Matrix Log defect / Update status of defect 32 .

Test Case Execution and Logging Results Execute Step 1 of a test case Does the Application behave as desired ? Log defect / Update status of defect Log Test Results for the test step with proof of execution Is the next test step available and executable with the defect raised for the previous step in this test case? Execute next test case Execute the next step of the same test case 33 .

needs to be mentioned clearly. the application is not behaving as desired and hence the test step is marked as “Fail” While logging the “Actual result” the behavior of the application. the actual behavior of the application is logged in the “Actual result” section of the corresponding test step If the actual result is the same as the “Expected result”.Test Case Execution and Logging Results   Each test case is executed step by step After executing the test step. the application behaves as desired and hence the test step is marked as “Pass” If the actual result is the not same as the “Expected result”. This will help the following: » Other testers in the team to understand the impact of failure of this test case on the test cases that they would execute » Development team to identify the exact issue and addressing it » Retrospection of the cause to introduce this defect    34 .

a defect needs to be raised and the testing can be continued for the consecutive test steps of the same test case  This may not be possible in all cases. if the “From” and “To” dates are not editable  A test case is considered to be pass only if all the test steps in that particular test case pass 35 .Test Case Execution and Logging Results (Contd. While applying leave. While applying leave. then the tester may proceed to execute the next test case Example In the Leave Management system.)  If a particular test step fails. a defect needs to be logged in the defect log and it is recommended to execute the consecutive test steps if possible Example In the Leave Management system. if the “Type of Leave” combo box does not have the label name as expected.

to be compliant with the regulations 36 . irrespective of the behavior of the application Test log sheet is a copy of test case document with additional details like Actual results. Pass/Fail. Defect ID and the Tester name who executed it Test Log is a chronological record of the all the information about the test case execution » » » » » What are the test cases executed In What order it was executed Who executed those test cases(Tester) Status of the each step of the test cases (PASS/FAIL) This can be created manually or using test management tool    For some specific applications it is mandatory to log the test results physically in the test case document.Logging of the Actual Results in the Test Log  The actual result of a test case and the details of execution are logged in a document called Test Log/ Test Management tool.

Sample Test Log  Sample Test Log Sheet 37 .

Sample Test Case Run Status  Sample of the Test Case run status Test Case Run Status 38 .

Logging a Defect  Testers raise defects when the expected result and the actual behavior of the application do not match  The defect management methodology defined for every testing project would be different  The testers need to follow the one defined in the Test Plan of that particular project  The defect logging can be done in a simple excel document or any defect management / test management tool adopted by the team 39 .

This can be auto generated if a test management / defect management tool is used » Defect Description • A textual description provided to describe the defect to the other stakeholders of the project » Steps to reproduce the defect • The list of steps to be followed to reproduce the defect » Test Case Id • The Id of the test case executed to find the defect is provided for reference • The Test data used to reproduce the defect may also be provided here for reference 40 .Logging a Defect (Contd. Usually. a naming convention decided by the testing team is followed.)  Key Elements of a defect includes: » Defect Id • A unique Id is assigned to each defect raised during testing.

Logging a Defect (Contd. This is usually assigned during the defect triage meeting » Module Name • The module of the application in which the defect was found is mentioned here 41 . the name of the developer is provided here.) » Assigned to • Each defect will be assigned to a developer to fix.

Logging a Defect (Contd.)  Key Elements of a defect include » Severity • • • • The impact of the defect in the software is called Severity Every project will define the severity levels of defects in the Test Plan Severity of a defect is generally set by the Testing team while logging defects A few samples of the severity levels of a defect are 42 .

Examples for Defects–Severity: Critical All pages are blank while logged into the application 43 .

Examples for Defects–Severity: High “Half a Day” option is not available for “Leave Start Date” 44 .

Examples for Defects–Severity: Medium The calendar icon does not allow to click 45 .

Examples for Defects–Severity: Low The label is incorrect 46 .

less critical defects can have the target resolution time anywhere between 8-48 hrs 47 . fixing one defect may rectify many of the related defects • Each of these priority levels will have a time associated with it.Logging a Defect  Key Elements of a defect include » Priority • Priority of a defect is generally determined by the delivery manager of the project as defined in the Test Plan based on the following – – – – Severity of the defect Capacity management of the development team Time required to fix a defect Impact of fixing a defect on the other defects – sometimes. Critical defect may have a target resolution time of 2-6 hrs. to enable the developers to fix within that time– For Example.

there may be cases where the defect is not repeatable • A clear screenshot of the defect with the relevant area highlighted will help the development team and the business to understand the defect better • It is always better to highlight the area where the defect had occurred in the screen shot » Defect Status • The status of a defect would change from stage to stage during the defect life cycle • The defect life cycle is defined in the Test Plan for a project » Test data used to identify the defect • This will help the other stake holders to – Reproduce the defect – Identify if the issue is with choosing the test data » Version of the build in which the defect is identified » Test Environment details in which the defect occurred 48 .) » Capture screenshot of the defect • Though the details of the defect are documented.Logging a Defect (Contd.

Sample Defect Life Cycle Sample Defect Life Cycle (Defect Status and the process may vary from Project to Project as per Test Plan) Test Execution If Defect found? NEW Is Valid Defect? REJECTED / CANCELLED Fix in this release? DEFERRED OPEN Fix of defects Defect found in Retest? Triage Meeting REOPEN FIXED Dev Team Test Team CLOSED 49 .

Best Practices While Logging Defects  Log defects with: » Clear description of the defect – as is in the application » Appropriate tone of the language » Screenshot(s) with • • • • Clear marking of the defect area Call outs Timestamp Sequence of screen shots for each step if required » Highlight if there are any similar/ related defects earlier in the remarks » Mention the other areas/features of testing affected due to this defect in the remarks » Even if the defect is observed once and is not reproducible. log the defect with more details on when the defect occurred and when it is not reproducible 50 . log it and the development team and the business will decide to consider it as a defect or just an observation » In the above case.

Updating the Traceability Matrix  The traceability matrix created during the Requirements Analysis and the Test Design phase is updated with the defect details  The defect id is mapped against the corresponding failed test case id  This will help in tracking which are the requirements that are misinterpreted / misunderstood by the teams  Only the defect id is mapped here and the details of the defects are obtained from the defect log 51 .

the defect status is changed to closed and the test log is updated appropriately  In some cases. even after the fix  Otherwise. Hence the related modules also need to be retested 52 .Re-testing of defects  Defects that are fixed by the development team are retested by the testing team to ensure that the application behaves as desired after the fix  The status of the defect is changed to Re-Open and is assigned to the developer if the application does not behave as desired. fixing of a defect would introduce new defects in the same module or any other related module of the application.

Regression Testing  If the application under test is a maintenance application. the enhancement to the application may introduce defects in the existing functionality of the application  Needs to be done to ensure that the application is in tact and behaves as before even after the enhancement/ defect fix  There will generally be a set of regression test cases identified or automated test suite already available 53 .

Best Practices During Test Execution  Identify the business critical/ error prone areas and focus on those. These can be identified from/using: » » » » Knowledge of the application Business knowledge Programming knowledge End user experiences  Test the application as an end user  Try and relate to similar scenarios while testing  Collaborate with other testers/leads/SMEs in the project to understand » Implications of the defects on other modules » Behavior of the defect if tested with other credentials » Behavior of the defect in various environments – Entire test environment/hardware/software 54 .

Test Execution: Reports  Types of Reports » Daily Execution Report – End of Day » Test Summary Report (No / No Go Decision) – End of Execution  These reports will be circulated to the stakeholders including » » » » » » » Onsite Test Manager Test Leads Dev Manager Program Manager Onsite Dev Managers Dev Module Leads Client  Data for reports can be prepared manually or taken from the tool if a test management tool is used 55 .

of test cases passed Vs No. of test cases failed Vs. of test cases executed No. of test cases executed Defect report • • • • • By By By By By Status Severity Priority Module age of the defects 56 . No.Test Execution Status – Daily Execution Report  The Status of test execution needs to be communicated to the stakeholders of the project in the agreed frequency as per the Test Plan  Below are some of the key information which will be shared with the stakeholders during test execution » » » » % of test cases executed No.

End of Test Report  At the end of every test cycle. the testing team will share the end of test report with the stake holders  This will help the other stake holders of the project to understand » Status of test execution » Summary of defects found grouped by • • • • Severity Status Age of defects Modules of the application in which the defects occur » Summary of the modules that are impacted due to the fixes made to the defects » Changes introduced in the business due to the defects found by the testing team (if any) » Stability of the application » Risks involved in moving to the next cycles of testing or UAT with the current state of the application 57 .

Execution Status.Sample Daily Execution Status Report  Daily Execution Report consists of Defect Status. Defect List and the Downtime Tracker Daily Execution Status Report Sample 58 .

Test Execution: Comparison ‘App Dev’ Vs ‘App Maintenance’ Test Environment Application Development Test Data Tools & Process Test Execution  The Test Environment has to be set  Test Data has to be identified for Test Management tool needs to Focus is more on System afresh all the test cases. as applicable be set up for the project Testing. System Integration Defect Management related set Testing and retesting of up needs to be done afresh defects Application Maintenance  The server set up for the Test  Test data may be identified from Environment would generally be the existing production environment Only the additional need for the available  Only the new build of the application test data required need to be has to be deployed in the existing identified and obtained before environment execution If there are any specific changes based on the enhancement. those needs to be taken care The access to the DB. App server and other external interfaces may be available already Only those that are additionally required need to be obtained before execution  Project would have been already set up in the Test Management tool New folders and other essentials need to be set for the current release only Defect Management and the process is already set and hence just need to be followed Focus is more on Regression testing of the application to ensure the enhancement does not introduce any defects into the existing functionality 59 .

Test Your Understanding Who will take decision to go for UAT ? a) Dev Manager Manager b) Client c) Test Which tracker is used for down times ? a) Down time b) Environment c) Defect When the Test Summary Report will be send? a) End of SIT b) End of Week c) Both 60 .

Fundamentals of Test Execution Session 1: Summary  Process of executing a series of test cases as defined in the test plan to validate if the application meets the requirements  Can be manual or automated using a tool  The following need to be planned before the Test Execution » Test cases review / Sign Off » Test cycles » Test Lab set up » Task Allocation / Run plan » Risks and mitigation process » Escalation process 61 .

)  Configuring of Test Environment includes the following: » Software required(Ex. Touch Screen) » Server to host the application in the desired configuration » Database setup/Server Setup to run the application » Appropriate level of access for test team to access Database. Special interfaces like Handheld Device.Fundamentals of Test Execution Session 1: Summary (Contd. Tools like Eclipse. Crystal Reports) » Hardware required(Ex. server and application » Appropriate level of access to external application if the testing scope covers for the same » Identifying and accommodating the batch job dependencies 62 .

Fundamentals of Test Execution Session 1: Summary (Contd.)  The release notes of the build primarily includes information of the features of the software included in the build and the deployment details Build verification Test validates if the build is good to proceed with testing and does not have any major showstoppers Test Log is a chronological record of the all the information about the test case execution What are the test cases executed In What order it was executed Who executed those test cases(Tester) Status of the each step of the test cases (PASS/FAIL) This can be created manually or using test management tool   » » » » » 63 .

)     Testers raise defects when the expected result and the actual behavior of the application do not match The defect management methodology defined for every testing project would be different The testers need to follow the one defined in the Test Plan of that particular project The defect logging can be done in a simple excel document or any defect management / test management tool adopted by the team 64 .Fundamentals of Test Execution Session 1: Summary (Contd.

Fundamentals of Test Execution Session 1: Summary (Contd.)  Key Elements of a defect includes: » » » » » » » » » » » » » Defect Id Defect Description Steps to reproduce the defect Test Case Id Assigned To Module Name Severity Priority Screenshot of the defect Defect Status Test data used to identify the defect Version of the build in which the defect is identified Test Environment details in which the defect occurred     Traceability Matrix has to be updated with the defect details post the test execution Defects that are fixed by the development team are retested by the testing team to ensure that the application behaves as desired after the fix Regression Testing needs to be done to ensure that the application is in tact and behaves as before even after the enhancement/ defect fix During and post the Test execution. 65 . various reports are generated to share information on the status of the execution and the application defects with the stake holders of the project.

All trademarks. and trade names in this course are the marks of the respective owner(s). 66 . service marks. The materials that can be accessed from linked sites are not maintained by Cognizant Academy and we are not responsible for the contents thereof.Fundamentals of Test Execution Session 1: Source  CSTE CBoK  ISTQB  Project experience Disclaimer: Parts of the content of this course is based on the materials available from the Web sites and books listed above.

© 2007. . The information contained herein is subject to change without notice. Cognizant Technology Solutions.You have completed the Session 1 Fundamentals of Test Execution . All Rights Reserved.

Sign up to vote on this title
UsefulNot useful