ISYS6338– Testing and System Implementation

Session 6: Other Players in Testing

Project
 Learning Outcomes

• LO 2: Design the testing management plan for a software

• LO 3: Design the testing implementation plan for a software
 References

• Black, Rex. (2009). Managing the testing process : practical

tools and techniques for managing hardware and software
testing. 03. Wiley. Indianapolis. ISBN: 9780470404157.
 Sub Topics

• Partners in Testing Project

• Outsourcing in Testing Project
Partners in Testing Project
 Needs for Partners

• You might want to get business analysts, help desk or

technical support, target customers or users, or sales and
marketing staff involved in testing to build their confidence in
the quality of the system that’s about to be released— or the
testing that your team or some other team performed on that
system. In some cases, the motivation is political, to make up
for credibility deficits in the development team or your own
team of testers. In other cases— especially customer
involvement in acceptance testing— the motivation is
contractual, since many outsource development efforts
include a period of acceptance testing by the customers prior
to final payment.
Partners in Testing Project

Testing
Sales Offices
Service
Users &
Providers
Vendors User-
Surrogates

Partners
in Testing
 Vendors

• When vendors are producing key components for your

systems, clear-headed assessments of component quality and
smart testing strategies are key to mitigating those risks.

• Most vendors take a narrow, targeted view of their testing:

they might test very deeply in certain areas, but they aren’t
likely to test broadly.

• What is the risk to your system’s quality related to your

vendors’ components?
– Component irreplaceability.
– Component essentiality.
– Component/system coupling.
– Vendor quality problems.
 Testing Service Providers

• Testing service providers include any organization that offers

test services to clients. Most of the time these services come
at a fee, but some organizations provide some specialized
services at no charge to the client. The provider can offer
these services on-site (insourcing), off-site (outsourcing), or
both.
• A testing service provider brings several key strengths to the
table. The most important is expertise in test project
management and test engineering.
• Another advantage is that testing service providers can often
begin running tests for you more quickly than you could do it
yourself.
• A testing service provider, whether lab or consultancy or
whatever, might also offer expert consulting and training
 Sales Office

• If you sell a product internationally, you might have a local

sales office or a sales partner (such as a distributor) in various
regions.

• As fellow employees, the staff members in a sales office have

the same goals and objectives you do —they have a stake in
ensuring that the test effort is successful.

• Unfortunately, these sales and marketing people might not

have technical sophistication or a particular skill with testing. If
you are responsible for the results of the testing and want
specific items tested, you will need to spell out these details.
Any test cases you give to nontechnical colleagues must be
precise and unambiguous.
 Users & User-Surrogates

• This category includes business analysts, help desk, customer

support, and technical support personnel along with actual
target customers and users.

• Most commonly, these folks participate in alpha, beta, pilot, or

acceptance testing efforts. (One of our clients, though, invites
its customers’ most-expert users of its system to participate in
system test, using their own unique data and workflows).

• As mentioned previously, testing by users and user-surrogates

can result from considerations of credibility, contractual
obligation, or from a need to broaden test coverage.
Outsourcing in Testing Project
 How Outsourcing Affects
Testing
• Over the last 20 years, outsource development of one or more
key components in the system has come to dominate
software and hardware systems engineering. The trend
started in hardware in the 1990s. RBCS clients like Dell,
Hitachi, Hewlett-Packard, and other computer systems
vendors took advantage of cheap yet educated labor overseas
to compete effectively in an increasingly commoditized
market.
• Outsourcing spread slowly into software in the 1990s. The
near-simultaneous bursting of the telecom and dot-com
bubbles in 2000 combined with the Y2K and Euro-conversion
wind-downs, making the software industry and its customers
more conservative and price-conscious. By the end of 2002,
when after three years into a spectacular IT downturn
computer science enrollments in the United States fell to less
than half of their 1999 levels, price had become the primary
determinant inmost IT project decisions. Mass outsourcing of
 How Outsourcing Affects
Testing (Cont.)
• As with hardware, commoditization plays a role with software
outsourcing. Software as a Service (SaaS) represents one facet
of commoditized software, and it is clearly a form of pay-as-
you-go outsourcing. The use of open source packages
represents another facet of commoditized software, being a
form of pay-nothing-as-you-go outsourcing (albeit requiring
potentially expensive self-support).

• We also need to consider a from the outside looking in

perspective; i.e., how you manage testing when a constraint is
distribution of some of the testing, all of the testing, and
perhaps even the entire development effort.
 Outsourcing Scopes

• Outsource the development, but retain the testing in-house.

• Outsource the development and the testing to one company

only.

• Outsource the development and the testing to two different

companies.

• Outsource the development and/or the testing, each to

multiple companies.
 Reasons for Outsourcing

• The desire to realize labor and other cost savings.

• The expertise, capital equipment, or geographical advantages
of the outsource organization.
• The need for system or product certification by a qualified
service provider.
• The inability to handle the work in-house, either due to a
temporary spike in workload or a one-off project.
• Organizational or peer pressure on decision-makers to
outsource, whichis the ‘‘everybody’s doing it’’ argument.
• Dissatisfaction with the in-house team’s capability, whether
for service, cost, attitude, quality, or some other reason, which
is the ‘‘it couldn’t be worse, at least it’s cheaper’’ argument.
 Success Factors
for Outsourcing
• You have to select the right testers for the testing tasks, with
the right skills. This includes test management skills for large
chunks of testing work that will be outsourced.
• The outsourced testers must have access to the right
equipment, the right tools, the right infrastructure, and so
forth. If the assumption is that the outsourced testing will use
equipment, tools, and infrastructure that you have in-house,
this will not happen by magic, but will require careful planning
and management of the logistics of that.
• The outsourced testers must have the ability to adapt their
work to your project and your organization.
• The outsourced testers must have sufficient independence to
tell the straight truth about their results.
 Issues for Outsourced Project

• The system being built: This includes not only the transfer of
test releases, release notes, code, and other supporting files,
under proper configuration management control and as part
of a solid release engineering process, but also the change
management and project management elements.
 Issues for Outsourced Project
(Cont.)
• The test system: Due to reasons of expense or security, it is
often necessary to share test environments with the
outsource development and, if you have them, outsource
testing partners. This includes the obvious elements of the
test environment, but also access to cohabiting software,
affected or linked systems or databases, and other connected
elements that will be present in the customer or production
environment. You’ll need also to be able to exchange various
testware items, such as test data, cases, and test tools, both to
audit the test work of the outsource organizations and to
avoid problems with gaps and overlap in the creation of the
testware. Finally, you also need to coordinate the test
processes, at least the major touchpoints that cross-
organizational boundaries.
 Issues for Outsourced Project
(Cont.)
• Information flows: A tremendous amount of information
should flow across organization boundaries as part of testing
in an outsourced project. This includes project documents,
quality risk analysis documents, estimates and plans, work
assignments, bug reports, and, of course, test results reports.
It also includes less-formalized but equally critical information
flows like emails, project discussion enablers like wikis and
newsgroups, and the like.
 Risks for Outsourced Project

• Political instability, especially when using outsource

organizations in developing regions
• Time zones, language barriers, and other communication
issues.
• Overtime and other off-hours constraints due to lack of access
to the facility, lack of power or other infrastructure during off-
hours at the facility, or physical safety issues associated with
being at the facility during off-hours.
• Infrastructure problems and inadequacies, such as unreliable
Internet connectivity, poor roads and airports, difficulties
obtaining potable water and safe food during site visits, and
so forth.
• Skills availability, as mentioned in the earlier section.
• Unforeseen and sometimes abruptly exposed organizational
weakness, including loss of key players due to extreme rates
 Conclusion about
Outsourcing for Testing
• As discussed in this section, outsourcing does pose a number
of challenges to testing and quality. However, none of those
challenges are fundamentally that much different or harder to
manage than the risks and activities on non-outsourced,
collocated projects. The differences are those of degree more
than of kind. So, the diligent test manager can succeed with
outsourced projects if she manages the overall testing process
with the same level of attention to detail that she would use to
manage any other testing effort.
• Software outsourcing matures, testers and test managers are
well positioned to become essential project participants.
Outsourcing might be cheaper than doing projects in-house,
but it’s harder, much more complex, and thus riskier. Since
testing, ultimately, is composed of risk management activities,
I suggest you bring your risk management skills to bear. You
might well find that you are the key to outsourcing success in
Thank