You are on page 1of 4


To learn how to select a Testing Tool and to study the benefits and costs associated with Automated Testing Tools.

"Automated Testing" is automating the manual testing process currently in use. This requires that a formalized "manual testing process", currently exists in the company or organization. Automation is the use of strategies, tools and artifacts that augment or reduce the need of manual or human involvement or interaction in unskilled, repetitive or redundant tasks. Following types of testing can be automated:

Functional Regression Exception or Negative Stress Performance


- testing that operations perform as expected. - testing that the behavior of the system has not changed. - forcing error conditions in the system. - determining the absolute capacities of the application and operational infrastructure. - providing assurance that the performance of the system will be adequate for both batch runs and online transactions in realtion to business projections and requirements. - determining the points at which the capacity and performance of the system become degraded to the situation that hardware or software upgrades would be required.


Reliable: Tests perform precisely the same operations each time they are run, thereby eliminating human error Repeatable: You can test how the software reacts under repeated execution of the same operations. Programmable: You can program sophisticated tests that bring out hidden information from the application. Comprehensive: You can build a suite of tests that covers every feature in your application.

Reusable: You can reuse tests on different versions of an application, even if the user interface changes. Better Quality Software: Because you can run more tests in less time with fewer resources. Fast: Automated Tools run tests significantly faster than human users. Cost Reduction: As the number of resources for regression test are reduced.


Proficiency is required to write the automation test scripts. Debugging the test script is major issue. If any error is present in the test script, sometimes it may lead to deadly consequences. Test maintenance is costly in case of playback methods. Even though a minor change occurs in the GUI, the test script has to be rerecorded or replaced by a new test script. Maintenance of test data files is difficult, if the test script tests more screens.


Cost of software tools Cost of hardware Cost of salaries for automated testing enginners Cost of training people Percents from the pay off for equipments and software tools used in training process Maintenance expenses for equipments and tools Expenses for the analysis of the behavior of the product based on the test data generated or on the results of the previous tests Costs involved in automation environment design and implementation Cost of operating the automated tests


Following are the seven steps to select and deploy a test tool in an organization. Step - 1: Identify your test suite requirements among the generic requirements discussed. Add other requirements, if any.

Step - 2: Make sure experiences discussed in previous sections are taken care of. Step - 3: Collect the experiences of other organizations, which used similar test tools. Step - 4: Keep a checklist of questions to be asked to the vendors on cost / effort / support. Step - 5: Identify list of tools that meet the above requirements and give priority for the tool, which is available with the source code. Step - 6: Evaluate and shortlist one / set of tools and train all test developers on the tool. Step - 7: Deploy the tool across test teams after training all potential users of the tool.


1) Meeting Requirements: a) There are many tools available in the market today but rarely do they meet all the requirements of given product or a given organization. Evaluating different tools for different requirements involves lot of effort, money and time. Huge delay is involved in selecting and implanting test tools. b) Test tools may not provide backward or forward compatibility with the product-under-test (PUT). c) Test tools may not go through the same amount of evaluation for new requirements. For example some tools had Y2K-problem. d) A number of test tools cannot distinguish between a product failure and a test failure. This increases analysis time and manual testing. The test tools may not provide the required amount of trouble-shooting/debug/error messages to help in analysis. For example, in case of GUI testing, the test tools may determine the results based on messages and screen coordinates at runtime. Hence, if the screen elements of the product are changed, it requires the test suite to be changed. The test tool must have some intelligence to proactively find out the changes that happened in the product and accordingly analyze the results. 2) Technology Expectations: a) In general, test tools may not allow test developers to extend / modify the functionality of the framework. So, it involves going back to the tool vendor with additional cost and effort. Very few tools available in market provide source code for extending functionality or fixing some problems. Extensibility and customization are important expectations of a test tool.

b) A good number of test tools require their libraries to be linked with product binaries. When these libraries are linked with the source code of the product, it is called as the "instrumented code". This causes portion of testing be repeated after those libraries are removed, as the results of certain types of testing will be different and better when those libraries are removed. For example, the instrumented code has a major impact on the performance testing since the test tools introduce an additional code and there could be a delay in executing the additional code. c) Finally, test tools are not 100% cross-platform. They are supported only on some O.S. platforms and the scripts generated from these tools may not be compatible on other platforms. Moreover, many of the test tools are capable of testing only the product, not the impact of the product/test tool to the system or network. When there is an impact analysis of the product on the network or system, the first suspect is the test tool and it is uninstalled when such analysis starts. 3) Training Skills: Test tools require plenty of training, but very few vendors provide the training to the required level. Organization-level training is needed to deploy the test tools, as the users of the test suite are not only the test team but also the development team and other areas like SCM (Software Configuration Management). Test tools expect the users to learn new language/scripts and may not use standard languages/scripts. This increases skill requirements for automation and increases the need for a learning curve inside the organization. 4) Management Aspects: A test tool increases the system requirement and requires the hardware and software to be upgraded. This increases the cost of the already-expensive test tool. When selecting the test tool, it is important to note the system requirements and the cost involved in upgrading the software and hardware needs to be included with the cost of the tool. Migrating from one test tool to another may be difficult and requires a lot of effort. Not only is this difficult, as the test suite that is written cannot be used with other test tools but also because of the cost involved. As the tools are expensive and unless the management feels that the returns on investment (ROI) are justified, changing tools are generally not permitted. Deploying a test tool requires as much effort as deploying a product in a company. However, due to project pressures, test tools effort at deploying gets diluted, not spent. Thus, later it becomes one of the reasons for delay or for automation not meeting expectations. The support available on the tool is another important point to be considered while selecting and deploying the test tool.