The Benefits of Outsourced Testing

Why choose to Outsource Testing?

For the majority of organisations, the testing effort is often cyclical, with high and low peaks of
activity. However, because of this it is not efficient or practical to maintain a fully resourced test
team all year round, leaving excess capacity during the low peaks. One solution is to use a
specialist outsourcing testing company to provide the help you need, either in the form of attaining
in-house assistance (consultants onsite), or, as discussed in this article, outsourcing this function
completely.
Other reasons for outsourcing their testing includes: to cut costs, to speed testing, to improve their
testing, to save office space, acquire test environment facilities, and release in-house staff to take
on development activities.

The Benefits of Outsourcing

The use of outsourced test teams has increased dramatically in recent years with the recognition
that companies can benefit from a wealth of testing expertise that can add to, and complement, in-
house knowledge.
The main advantage is that instead of maintaining a fully resourced test team all year round, when
test activity peaks core permanent employees can either be supplemented by outsourced testing
personnel, or the excess work is outsourced completely. Companies can now contact service
suppliers to avail of end-to-end software quality services, for use at any point during development,
to shorten time-to-market cycles without taxing internal resources. Additionally, they can benefit
from contracting in testers who will be specialists in the field.
Utilising an outside test support services firm can provide the ability to utilise resources in a "just-
in-time" fashion. Ideally, the outsource vendor can provide all of the services that may be required
for the system testing. The energy expended in building the relationship can help the client achieve
a level of confidence in the quality of the product, which would historically require a larger internal
team.
The ability to then call on those resources when they are needed, and then have the ability to give
them back when they are no longer needed, provides a significant cost saving. In addition, it allows
everybody focus on what they do best, thereby maximising everyone's resources.
There are a number of additional benefits that can be achieved, including -
ƒ These specialists will seamlessly increase the value of your company and allows for more
efficiency because both partners in this outsource relationship will be focused on what they
do best - a software developer needs to focus on making great software.
ƒ The use of independent test teams can validate that the existing testing and procedures are
up to the mark.
ƒ As a complementary service, it can improve web-enabled software quality, accelerate
development cycles, and reduce cost of building software.
ƒ Experienced & skilled testers can be acquired for short periods of time.
ƒ The alliance enables the organisation to perform a greater range of testing services than is
currently possible on it’s own.
ƒ This solution dramatically improves software reliability while accelerating time-to-market
and reducing the cost of development.
ƒ Compatibility lab testing can provide for testing your application against various
combinations of Hardware, Operating Systems & Internet Browsers – particularly important
for eBusinesses, but one that might not be achievable by the company alone. [For example,
they may have a server farm that you cannot afford or justify for the testing, no matter how
important].
 ƒ Return on Investment - learn how to find costly and embarrassing problems before your
customers find them.

The First Steps to Successful Outsourcing

I would recommend that you consider the following points when deciding whether to outsource a
project or application for testing.
1. The nature of the project
Firstly, consider the nature of the project or application that you are considering outsourcing.
If the project is an enhancement version of a stable product, with existing test scripts that could be
used by the outsourcing partner, it would be ideal to outsource.
However, if it is the first release of a totally new application, it may not be ideal to outsource the
testing. For example, if there are major problems with the application, testing is not going to solve
them, and if your testing partner encounters a large number of serious errors the test coverage will
probably be lessened.
Lastly, carefully consider what you expect to achieve by outsourcing. In all likelihood, outsourcing
will never (and probably should never) completely do away with the necessity to have in-house
expertise, so you should consider the loss of knowledge and hands-on experience that your test
team will lose.
2. What testing do you want them to do?
It is also important to consider what types of testing that you actually require to be performed. For
example, if stress testing is important, an outsourced testing firm may be more experienced and
suited towards performing this. Or, you may wish to perform a certain aspect of the testing that is
very complex and that you believe you would not sufficiently benefit from by outsourcing.
3. The Practicalities
Who is going to be fixing the bugs? If your company is going to, the process and procedures by
which the error reports are going to be exchanged is critical. If your error management system
cannot be remotely updated [i.e. by the outsourced partner reporting the bugs directly] then I
would suggest either not outsourcing at all, or else implementing an error management tool which
can cater for remote logging, and can be under your direct control. It is important that you retain
control (or at least possession) of the bug database for future reference.
Secondly, how are you going to hand over new builds of the s/w? If you have to burn CD’s, unless
there is a quick way of transferring the new builds over to the outsourced testing partner, valuable
testing time may be lost.

Choosing an Outsourced Testing Partner

When choosing to utilise outsourced testing services, it is important to check the following:
• Their past experience – specifically in testing the same type of project
• Their current (not historical) client list
• Bios of their key personnel that are going to be working on your project (and not just Bio’s
of their best people).
• Detailed evidence of the existence of testing processes and procedures, particularly for
preparing test cases/scripts; executing test cases/scripts; and (most importantly) their
entire review process.

Risk Planning for Outsourced Testing

What’s the big deal - after all, they will actually do the work, won’t they? I mean, what could go
wrong?
As the extent of the outsourcing grows, however, so too will the challenge of managing
relationships with them. But if companies fail to do this effectively, projects could be jeopardised.
The danger is that weak outsourcing partners can stall entire projects. So, how do you provide for
this situation?
Apart from the contractual elements to formalise the relationship, you must naturally consider how
you can actually verify their progress. Additionally, you need also to be sure that the work is being
performed to a high level of quality.
In my opinion, the only way to ensure this is to hold regular reviews and assessments with the
partner. (See next section on reviewing).
If possible, send a representative (one of your testers) to be part of their team. In addition to
providing you a direct insight into the actual progress and quality, the representative can assist the
outsourcer either by having previous experience of that application, or at the very least, an
informal channel of communication.

Managing Outsourced Testing Projects

One of the key ingredients for success when outsourcing is the relationship. There must be trust
between both parties. This trust is necessary in order to communicate the value points and the
methods of communication that are critical for supporting the flexibility that using a test support
services firm requires.
You should take a collaborative approach, and work together – after all, you are both working towards
the same goal – releasing a quality product.
I would recommend the following:
Firstly, define the channels of communication. Appoint a person from your company, to act as the first
level of contact between your company and the outsourced partner, who will deal directly with an
agreed specified person on their side (ideally the test manager for the project).
These two people are the first-level team for resolving problems as they arise, and additionally your
representative also provides and archives all information passed to the outsourcing partner.
Secondly, agree escalation procedures, and specify to whom the problem is to be escalated to - on each
side.
Set out milestones – but beware when setting milestones, that YOU are the ones who state whether the
outsourced testing company has reached the milestone or not – for example, if one milestone is the
production of the test plan, if you are not happy with the test plan then you must be in a position to state
that they have NOT reached the milestone. Otherwise you could end up in a position whereby they gave
you a test plan that you considered inadequate, yet they are looking for payment for reaching the
milestone.
To cater for this, and other concerns, it is vitally important to specify formal review points – stages at
which progress can be measured, and where issues can be highlighted.
Formal Review Points:
Test Plan Insist that you must sign off on the test plan before test preparation can commence.
Review Hold a round table review (by phone or videoconferencing as available) and ensure
the test plan clearly sets out the approach to be taken, identifying what levels and
types of testing are to be applied and the methods, techniques, tools and resources to
be used, as well as the review points, schedule, test coverage required, and signoff
criteria. [See here for more details about test plan contents].

If you are not satisfied with the test plan, do not signoff. Ensure that one of the
criteria for proceeding to the next stage is your signoff.
Test Case Ensure that all test cases are reviewed and signed off by your own business
Review analysts/product specialists, to ensure that they are of sufficient quality, depth,
accuracy and coverage.
Insist on modifications to the test cases until you are satisfied with their quality.
Remember – the testing will probably only be as good as the test cases – you should
assume that anything that is not defined as a test scenario will not be tested, no
matter how obvious it may be to you.
Execution An execution readiness review is performed to ensure that the outsourcing company
Readiness are ready to move into the Test Execution phase. This readiness review checks to
Review ensure that the test entrance criteria, as specified in the Test Plan, have been met.
Execution Ensures that regular detailed progress information, including full bug details and
Progress test case pass/fail statistics are passed to you throughout the execution phase.
Reporting During this phase it is important that you can analyse the bugs that are being
logged and match the bugs to specific test cases. This can then serve as an
independent check of the test execution progress.
In parallel you can perform some testing yourself, to double-check the test cases that
are passing.
Build If you hand over software that is not of ‘testable’ quality, the outsourced partner
Acceptance is not going to be able make much progress and neither of you are going to
Tests benefit at all.
Therefore, you should insist that ‘Build Acceptance Tests’ are performed on each
build/drop of software - either perform these yourself, or get the outsourcing
partner to perform them. These tests are simple checks to ensure that the
software is of ‘testable’ quality in that they are going to able to test the various
functions and not be stuck by a very obvious showstopper bug. If they cannot
proceed with the testing there is no point blaming them.
Bug As I am sure you are aware, waiting for bugs to be fixed can be a serious
Turnaround bottleneck for all testing projects, and possibly even more so for an outsourced
Times project. Therefore, you should actively monitor bug turnaround times for
bottlenecks.
It is probably also more appropriate that somebody from your company be the
one escalating/hassling the developers to fix the bugs, rather than an external
person trying to.
Exit Report Ensure they prepare an Exit Report, which will summarise the testing and
& Final includes the following details:
Signoff. • The full bug list
• List of any outstanding bugs
• List of any uncompleted test cases
• A copy of all Test spreadsheets
• A copy of all test results & statistics
Do not sign off on this document until you are satisfied as to the accuracy of the
contents, particularly the test coverage metrics. Ask to see log files or any other
evidence of test execution having taken place.
Final Deliverables
A final consideration to take into account is that you may have to test this project yourself in the future.
However, this could be more difficult because you do not have the benefit of the knowledge and hands-
on experience of actually testing the project in the first place.
In order to minimise this risk it is important to specify as a condition of the project that upon completion
of the testing, you should be handed back all test documentation.
For example, you should include the following as specific deliverables:
• The final test plan document
• All updated and revised test cases and expected results
• All test data requirements
• Testing pre-conditions, including test sequence, test environment setup and initialisation
• The Bug database
• All scripts etc.
In this way if you have to test it yourself at least you will not be starting from scratch.
Conclusion
Once you have picked a good partner, and gone through the first project together, outsourced testing can
be an excellent option. But it must be noted that to make a success of it, it requires continuous
monitoring and assessment, and careful choice of projects to outsource.
There must be trust between both parties. This trust is necessary in order to communicate the value
points and the methods of communication that are critical for supporting the flexibility that using a test
support services firm requires.
You should also take into account that the old adage ‘garbage in, garbage out’ is particularly true in
these situations. If you handover a product or information for testing that is not of ‘testable’ quality, it is
not going to help either of you.
Upon completion of each project you should perform a final acceptance test on all returned software –
but be open about this process and give constructive feedback. In addition, you should also assess each
other’s performance, particularly with regard to bottlenecks, and be open to accepting criticism of your
own company’s performance.
Keep in mind that if you do decide to outsource some or all of your testing, a lot of effort will still be
required by yourselves in order to support and enable the outsource partner to maximise performance
(for example, reviewing test cases etc.)
Previously testing often focussed on functional issues and risks as being the main concerns. However,
with the advent of eBusiness development, this focus has changed. Non-functional issues such as cross
browser compatibility / configuration, usability, performance and security have become more and more
important considerations when planning test projects. It is also becoming more prevalent that companies
are increasingly making use of outsourced testing to get the expertise quickly without expending all of
their capital.
Specialisation breeds a continued need for companies to focus on core competencies, and by using
strategic outsourcing to fill gaps helps fast-growing companies in the high-tech industry keep focused on
their initial goals and keeps resources allocated to the areas that need them most.