Test Management, Standards, and

Environment

CSTE Domain 7
Marta Heinrich
April 11, 2001
 CSTE Domain 7
Consists of 3 Areas
Test Management
Test Standards
Test Environment
 Test Management

Consists of:
Test Manager Skills Test Technology
Test Objectives Staffing
Test Competency Management of Staff
Test Performance
 Test Manager
A Good Test Manager should have the following skills:
The ability to understand The ability to maintain a
and evaluate software test organization that is
test processes, standards, strong, independent,
policies, tools, training, formal, and unbiased
and measures
The ability to lead,
The ability to recruit and communicate, support,
retain outstanding test and control
professionals
Time to provide the care
needed to manage test
groups
 Test Manager
A Test Manager is responsible for:
Ensuring tests are Reviewing analysis
designed and design artifacts
Ensuring tests are Chairing the test
executed in a timely readiness review
manner
Managing the test
Test planning and effort
estimation Overseeing acceptance
Designing the test tests
strategy
 Test Objectives
Establish objectives for the enterprise,
organization, and project
Should restate the project objectives from
the project plan
Should determine whether those project
plan objectives have been achieved
How to create was covered in Domain 10 –
Planning Process
 Test Competency

Establishing the organization’s competency

goals
What goals does the company have for testing?
No defects?
100% of Customer Requirements?
85% uptime of software?
Etc…
 Test Performance

Monitoring test performance for adherence

to the plan, schedule and budget, and
reallocating resources as required to avert
undesirable trends
 Test Performance
Use of Entrance and Exit Criteria help
monitor test performance through the
stages of the SDLC
The criteria ensure the work is complete,
complies with applicable standards, and all
necessary activities are completed before
work moves to the next stage
Entrance/Exit criteria are detailed in
Domain 10 – Planning Process
 Test Performance
Two methods to measure performance:
Estimate
Person doing work estimates the % of work done
Milestone
Define specific milestones that must be met

Methods of performance measurement should

meet these criteria
Objective
Measure performance in accomplishing real task
Efficient
 Test Technology
Maintaining organizational awareness of,
and competency with, emerging software
and testing technologies

Publications, seminars, conferences,

newsgroups, websites, are all ways to
keep up on emerging technology
 Test Technology
The following technological developments are
causing organizations to revise their
approach to testing:
Integration (with day-to-day operations)
System chains
The domino effect
Reliance on electronic evidence
Multiple users
 Staffing
Acquiring, training, Consider the following
and retaining a people for inclusion on
competent test staff the test team:
Ratio of testers to Professional Testers
developers Users
1:3 or 1:4 Data Administrators
Too many: IS staff
counter-productive Contractors/Consultants
Too few: Security Administrators
under-resourced
 Staffing

The following steps can be used to form

best team:
Identify potential test team members
Recruit test team members and develop
tentative assignments
Define individual work
assignments
 Management of Staff

Keeping staff appropriately informed, and

effectively utilizing the test staff
 Management of Staff
Keep progress updated through reporting:
Test Result Summary
Risk report
Coverage report
‘Bug’ report
Schedule progress
Tester Assignments
Keep test data accurate and timely
Avoid too many meetings
 Management of Staff
Two ways to configure a team
Skills-based
Allows more flexibility
Each member focuses on an area of specialization
Good when multiple projects are going on at once
Project-based
Cuts down on interruptions and shifting tasks
Time is billed to only 1 project and decreases
reporting time
 Test Standards – Area 2
External Standards
Familiarity with and adoption of industry test
standards from organizations
Internal Standards
Development and enforcement of the test
standards that testers must meet
 External Standards
IEEE
Institute of Electrical and Electronics Engineers
Founded in 1884
Have an entire set of standards devoted to
Software
Testers should be familiar with all the standards
on the attached IEEE document (included an
abstract for each one)
 External Standards
Other Standards:
ISO – International Organization for Standards
SPICE –Software Process Improvement and
Capability Determination
NIST – National Institute of Standards and
Technology
DoD – Department of Defense
Detailed information on all included in IEEE
handout
 Internal Standards
The use of Standards…
Simplifies communication
Promotes consistency and uniformity
Eliminates the need to invent yet another
solution to the same problem
Provides continuity
Presents a way of preserving proven practices
Supplies benchmarks and framework
 Test Environment – Area 3

Test Process Engineering

Tool Development and/or Acquisition
Acquisition of a Test Bed/Test Lab/Test
Environment
 Test Process Engineering

Developing test processes that lead to

efficient and effective production of testing
activities and products
 Test Process Engineering
Activities a test manager should perform
during the test process
1. Monitor product quality
2. Perform an appropriate test process
3. Coordinate the test schedule with all other
activities in the project
4. Collaborate with the project team and
stakeholders
5. Study and learn
6. Lead the test team
Tool Development/Acquisition

Acquiring and using the test tools, methods,

and skills needed for test development,
execution, tracking, and analysis (both
manual and automated tools including test
management tools
Tool Development/Acquisition
The test manager plays a key role in the
identification, selection, and acquisition of
tools
The introduction of a software tool, and the
scheduling of these events, must be
determined in advance
When determining cost, determine ‘TRUE’
cost
Tool Development/Acquisition
Tool usage should be initiated in response to an
expressed management goal
Difficulties can arise in these areas:
Organizational obstacles
Problems arising from the tools
Obstacles in the computer environment
Ways to overcome the difficulties
Organizational
Tools
Obstacles
 Tool Development/Acquisition
Sequence of events for tool acquisition
1. Set Goals 6. Review of candidates
2. Set Tool Objectives 7. Score candidates
3. Acquisition Plan 8. Select Tool
4. Selection Criteria 9. Procure Tool
5. Identify candidate 10. Evaluation Plan
tools
 Tool Development/Acquisition
Sequence of events continued
11. Toolsmithing plan 16. Modifications
12. Training plan 17. Training
13. Tool received 18. Use in the Operating
14. Acceptance Test Environment
15. Orientation 19. Evaluation Report
20. Determine if Goals
are met
 Test Bed/Test Lab/Test
Environment
Designing, developing, and acquiring a test
environment that simulates ‘the real world’,
including capability to create and maintain
test data
 Test Lab
Testers should perform controlled tests in an
environment where the outcomes are known,
the platforms are the same for each iteration
of testing, etc.

The best test lab is created by planning ahead

of time, and taking all factors into
consideration prior to designing.
 Test Lab
Steps to create a test lab
Select and plan the lab area
Size, lighting, layout, climate control, fire safety and
prevention, power, static, facilities
Determine test lab inventory
Platforms, hardware, additional software,
consumables, tools, furnishings, reference materials,
inventory list
Miscellaneous
Pad budget for costs, assign test lab admin, security
for lab, change control process, include maintenance
and upgrade costs in budget
 Test Lab
Three major issues to address
The platforms are not representative of the
platforms in the real world
The software will be expected to work on
platforms not included in the test lab
The supporting software on various platforms is
not comprehensive
 Test Data
Creates representative processing conditions
using test transactions
The key aspect of test data is optimizing testing
through selecting the most important test
transactions
To be effective, test data should use transactions
having a wide range of valid and invalid input
data
Determine expected results for test data to
compare with actual results
 Test Data
Perry’s 9 Step process for the creation and
use of test data
1. Identify test resources
2. Identify test conditions
3. Rank test conditions
4. Select conditions for testing
5. Determine correct results of processes
 Test Data
Perry’s 9 Step process continued …
6. Create test transactions
7. Document test conditions
8. Conduct test
9. Verify and correct
NOTE: When the software changes,
the test data needs to be updated as well.
 Domain 7
Consists of 3 Main Areas
Test Management
Manager Skills, Objectives, Test Competency, Test
Performance, Test Technology, Staffing,
Management of Staff
Test Standards
External and Internal
Test Environment
Test Process Engineering, Tool Development and
Acquisition, Test Lab/Test Data
 References
Black, Rex. Managing the Testing Process, Microsoft
Press, 1999.
Kaner, Polk, and Nguyan. Testing Computer
Software, Wiley Computer Publishing, 1999
Kit, Edward. Software Testing in the Real World,
Addison Wesley, 1997
Perry, William E. Effective Methods for Software
Testing, Wiley Computer Publishing, 2000
Raynus, Joseph. Software Process Improvement with
CMM, Artech House, 1999.
http://www.stickyminds.com
http://www.data-dimensions.com/Testers’Network