This action might not be possible to undo. Are you sure you want to continue?
A Perspective Seattle Area Software Quality Assurance Group John Napier, CSQE, CSTE BEST Consulting
Value of Professional Associations Value of Software Certification programs QAI’s “Effective Methods of Software Testing”
The opinions expressed in this presentation are my own, and not those of: BEST Consulting The Boeing Company Quality Assurance Institute SASQAG
. .. No common language “In my last job…” No individual professional context No organizational professional context Can’t see the forest...What its like in the trenches.
What’s a person to do? Establish a personal professional context Get to know “the forest” Help create a common language Help bring best practices into your organization .
The Quality Assurance Institute What’s QAI? 4/16/2012 .
benchmark data and experiences Software quality focus Certification Programs Certified Quality Analyst Certified Software Quality Engineer .The Quality Assurance Institute An international association of 1000 organizations banded together to share best practices.
QAI Certified Software Test Engineer (CSTE) Mandatory Skills Optional Skills Test Planning Risk Analysis Use of Test Tools Measurement Test Performance Test Case Design Defect Management Establishing and Maintaining a Test Environment .
ASQC* Certified Quality Software Engineer (CQSE) Experience Professional Association Software Engineering Body of Knowledge Exam Many other ASQC Certification Programs *American Society for Quality Control .
Effective Methods of Software Testing Carolyn Fairbanks Q-Star Consultants 10-14 March 1997 Seattle 4/16/2012 .
Effective Methods of Software Testing There are really only two issues in software testing: Requirements Organizational Culture 4/16/2012 .
Techniques .Techniques Test Case Design .Techniques Test Conditions .QAI Test Materials Organization Test Strategy Test Plan Supporting Materials Metrics Test Toolbox .
When Testing will Occur 4. Tradeoffs . Who will Conduct the Testing 6. Type of Project 2. Critical Success Factors 5. Type of Software 3.Test Strategy Definition 1.
Types of Project Traditional Client/Server Prototyping Object Oriented Purchased / Contracted .1.
Types of Software Batch Process Control Graphic User Interface Object Oriented Procedure Control Expert System Multi-Media Advanced Math Models Diagnostic Software Sensor and Signal Simulation DB Management Data Presentation Decision Analysis Pattern & Image Computer Operations Software Development Tools .2.
When will Testing Occur? Requirements Phase Activities Design Phase Activities Build Phase Activities Test Phase Activities Installation Phase Activities Maintenance Phase Activities .3.
Critical Success Factors Correctness Authorization Data Integrity Audit Trail Continuity of Processing Service Levels Access Control Methodology Reliability Ease of Use Maintainable Portable Coupling Performance Ease of Operations .4.
Testing Concerns Matrix
Critical Success Factors
Life Cycle Stages
Requirements Design Etc
Complete, Clear, Correct,
Correctness Testable, Feasible
7. Tradeoffs (What can be given up?)
Schedule Cost / Resources Scope Technology Quality / Critical Success Factors
Some QAI Philosophy
Test Life Cycle, Test Workbench
Testing Relationship Definitions Testing Lifecycle Concept Operational or Business Need Define Requirements Design System Acceptance Test System Test Integration Test Build System Unit Test .
Testing Relationship Definitions Continuous Life Cycle Testing Verify Operational or Business Need Validate Acceptance Test Verify Define Requirements Validate System Test Verify Design System Validate Integration Test Verify Build System Validate Unit Test .
Testing Relationship Definitions Test Support Activities TEST OBJECTIVES Verify Validate Operational or Business Need Acceptance Test Verify Validate TEST PLANNING Define Requirements Verify System Test Validate TEST EXECUTION (DYNAMIC) & TEST REPORTING Design System Integration Test Verify Validate TEST EXECUTION (STATIC) Build System Unit Test .
QAI Definitions “Test Workbench” Concept Problems Products for Test Test Quality Control Test Processes Validated Test Results Test Toolbox Test Measures .
2. 3. Set Test Objectives Develop Test Plan Execute Tests Summarize and Report Results .The QAI Test Process 1. 4.
Step 1 .Quality Control .Organize Test Team Task 2 .Set Test Objectives Task 4 .Set Test Objectives Task 1 .Perform Risk Assessment Task 3 .
Set Test Objectives Task 1 .5 .Organize Test Team Define Test Responsibilities / Tasks Determine People to Perform Tasks Worksheets 4.
Perform Risk Assessment Dimensions of Risk Criticality / Mission Impact System Size.Set Test Objectives 2. Scale and Complexity System Environment and Stability Reliability and Integrity Technology Integration Impact of Failure Likelihood of Failure Simple or Complex Scoring Approach .
Set Test Objectives Using: Prioritize Each System Objectives Low Test Strategy Medium High Risk Score Analysis Team Assignments Worksheet 6 Set Objectives Brainstorming Relate Objectives to Function / Output Relate Objectives to Events / System Input .Set Test Objectives 3.
Set Test Objectives Quality Control Objectives QC Checklist Objectives Incident Report (Change Request) …lots of checklists! .
Develop Unit Test Plan 9.Develop Test Plan 1. QC Unit Test Plan . Define Tests 4. Define Structural Attributes 3. Formulate System Test Plan 7. Define Admin Requirements 6. QC System Test Plan 8. Create Test / Function Matrix 5. Define Business Processes 2.
Define Business Functions Describe what the system should do …stands alone as an activity …is controllable …can be a transition …can be described by a process .Develop Test Plan 1.
Business Function Matrix Function Description Evaluation Criteria Worksheet 8 .
Develop Test Plan 2. Define Structural Attributes Reliability Efficiency Integrity Usability Maintainability Testability Flexibility Portability Reusability Interoperability Worksheet 9 .
Structural Attribute Matrix Attribute Priority Description Evaluation Criteria .
Develop Test Plan 3. techniques and test types Associate evaluation criteria with techniques and types of tests Determine type of test which satisfies each objective . Define Tests Using Test Objectives Worksheet Business Functions Worksheet Structural Attributes Worksheet Define Tests Associate test factors.
Type of Test Techniques Functional Requirements Regression Error Handling Manual Support Intersystems Control Parallel Structural Stress Execution Recovery Operations Compliance (to process) Security .
Test Data Sources Test Data / Test Desks Production Data Simulation / Prototyping Review / Confirmation Instrumentation / Tracing Test Script .
At a high level .Test Definition Each Test is defined in terms of: Controls Inputs Outputs Procedures ...
7 17. Function #1 Function #2 25 5.Develop Test Plan 4.30 45 . Create Function / Test Matrix Tests Function Test 1 Test 2 Ect.
14 . Define Admin Test Requirements Schedule Budget Resources Testing Materials Test Training Tests to be Conducted Defect Estimation Worksheets 13.Develop Test Plan 5.
Test Description See Test Plan Example .Develop Test Plan 6.Plan Section 3 . Formalize Test Plan Section I .General Information Section 2 .Specifications and Evaluation Section 4 .
Develop Test Plan 7. Test Plan Quality Control Quality Control Checklist Test Plan Incident Report (Change Request) …even more checklists! .
Develop Unit Test Plan Using: System Test Plan Program Initiation Document Program Design / Specification Documents Develop Test Plan Administration Subfunction / Test Matrix Interface / Test Matrix Worksheets 20.21.22 .Develop Test Plan 8.
Develop Test Plan 9. Unit Test Plan Quality Control Checklists .
Develop Test Cases Task 3 .Select Test Tools Task 2 .Execute Tests Task 1 .Step 3 .Execute Tests Task 4 .Quality Control .
Examples Decision Tables Automated capture / playback Checklists Program Analyzer .QAI Tool Definition What’s a Test Tool? …any vehicle which assists in performing a test.
Select Test Tools Using: System Test Plan Unit Test Plan Tool Selection Methodology .Execute Tests Step 1 .
Develop Test Cases Test Condition Expected Result Test Procedures Worksheet 23 .Execute Tests Step 2 .
Execute Tests Step 3 .Execute Tests Test Activities Log Regression Testing Full Partial Worksheet 24 .
Perform Data Reduction Task 3 .Record Defects Task 2 .28 .27.Summarize and Report Test Results Task 1 .Develop Findings and Recommendations Task 4 .26.Finalize Test Report Task 5 .Test Report Quality Control Test Defect Report Test Defect List Test Defect Log Worksheets 25.Step 4 .
Supporting Material Metrics Toolbox Techniques Test Condition Analysis Test Design Analysis 4/16/2012 .
Test Metrics Method Goal .Question .Metric .
Budget Selected Test Metrics .QAI Recommend Test Metrics Defect Reports Requirements Implemented Backlog of Defects (Not Corrected) Status of Project. Tests. Schedule.
QAI Test Metrics Selection Types 1 4 3 20 2 General Subject Involvement in Testing Extent of Testing Resources Consumed in Testing Effectiveness of Testing Assessment of Testing .
The QAI Test Toolbox Data Modeling Boundary Value Analysis Capture/Playback Cause-Effect Graphing Change Control Trackers Checklists Checkpoint Review Code Comparison Compiler-based Analysis Complexity-based Analysis Compliance Checkers Control Flow Analysis Correctness Proofs Coverage Based Analysis Data Dictionary Decision Tables Defect Trackers Desk Checking Equivalence Partitioning Error Guessing .
The QAI Test Toolbox File Comparison Flowcharts Help Desk Trackers Inspections Instrumentation Integrated Test Facility Mapping Model Animation Model Balancing Network Analyzers Performance / Timing Analyzers Population Analysis Risk Matrix Scoring Snapshots Symbolic Execution System Logs Test Case Generator Test Data Generator Test Scripts .
The QAI Test Toolbox Test Stubs and Harnesses Test Suite Managers Tracing Walkthroughs .
The QAI Test Toolbox Test Tools Types of Functional and Structural Tests X X Boundary Analysis Population Analysis Checklists X X X .
Business Process Analysis 4.Expected Result . Test Transactions Test Condition .Test Process . Population Analysis 3. Specification Decomposition 2.Test Condition Analysis Sources 1. Structural Analysis 5.
Identify Test Conditions 1. Population Analysis Files and Tables Screens Field/Data Element Test Condition . System Specifications or Production Environment 2.Expected Result .Test Process Worksheet 4 .
Test Process .Expected Result .Identify Test Conditions 3. Test Transaction Types Field Record File Relationships Error Use (Outputs) Search Match / Merge Stress Control Attributes States Procedures Control Flow Test Condition .
Structural Analysis …decision points.Expected Result .Test Process . Test Condition .Identify Test Conditions 4.. Business Cases …business scenarios… 5..
Design Test Cases Design Test Cases Develop Scripts / Sets Expand Test Conditions .
Design Test Cases Expand Conditions into Test Cases using Boundary Value Analysis Equivalence Partitioning Decision Tables Code Coverage Statement Coverage Condition Coverage Expand Test Conditions .
Design Test Cases Test Scripts Actor/Scene …Sam scowls at the report... Dialog “Where did this come from?” Detailed Test Script Descriptions Script Reuse Expand Test Conditions .
QAI Summary .Ten Key Test Concepts Use strategic test questions to set direction Develop Test Objectives Use a cross-functional “test team” Prepare a test plan. then execute it Utilize both verification and validation methods Collect and analyze defects Prepare test management reports .
Ten Key Test Concepts Make test efforts reusable Feed the continuous improvement process with information from testing Continuously improve the process .QAI Summary .
Incorporating the Process Models within Models …Make them your own 4/16/2012 .
This action might not be possible to undo. Are you sure you want to continue?