Best Practices in Unit Testing Automation

Unit testing and Automation:

Once the source code has been generated it must be tested to uncover (and correct) as many errors as possible before
delivery. In order to achieve this, testing must be conducted systematically & test cases must be designed using
disciplined techniques so that they have a high likelihood of finding errors. Unit testing is one such testing technique
which reveals errors in our application at early stage of software development process.

Unit testing is a process of verification and validation of individual units of source code in isolation. The key idea is that
each piece of code needs its own tests and the best person to test that code is the developer working on it. Hence, unit
testing is typically performed by developers in a development cycle, rather than in the QA phase. By performing unit test,
programmers can catch bugs early before they cause significant problems or delays in the later phases. However, manual
process of writing test cases, running tests, monitoring the test results, and pinpointing further areas for testing consumes
more time and effort. Hence automation is essential.

First step towards the automation is usage of unit testing frameworks. They can be considered as intermediate between
manual testing techniques and automated unit testing tools. These frame-works provide an interface to write and execute
manually written tests. They define how to structure your test cases and provide the tools to run them. Though, effort
and time required here is also higher, testing with frameworks ensures more coverage, reusability and repeatability of
tests. Further, these tests can be easily managed and maintained throughout the product life cycle.

Some of the commonly used unit testing frameworks are,

JUnit -a simple unit testing framework for java standalone application
Cactus -an extension of JUnit for unit testing server-side java code (Servlets, EJBs etc)
CppUnit -a unit testing framework for C/C++ applications
NUnit -a unit-testing framework for all .Net languages

WiproUT is an automated java unit testing tool developed and owned by Tools Group, Wipro Technologies. It
automatically generates and executes Junit, Cactus test cases by taking compiled source code as an input.

Strategy:

Effective testing is important, so is the strategy. By reviewing the Test Specification prior to testing, one can assess the
completeness of tests and testing tasks. An effective test plan and procedure will lead to the orderly construction of the
software and the discovery of errors at each stage in the construction phase.

Points to be considered here are,

1. As a first step, applicability and scope of unit testing has to be defined

2. The objectives of testing like test effectiveness, test coverage, the cost to find and fix defects should be clearly
stated within the test plan in measurable terms.
3. Selection of right tool is another important step at this stage. Below parameters can be considered while
selecting a tool,
a. Project environment and technology
b. Amount of code to be written newly
c. Amount of code that undergoes frequent changes
d. Lifespan of product being developed
e. Team size and effort estimated for the project
f. Budget
 List of tools available at Tools_list can be referred for some of the commonly used unit testing tools.
4. Regression testing definition, frequency, actions to be stated in test plan
5. Test plan should be implemented effectively in the project with the tool selected
6. Formal technical reviews should be conducted periodically to assess the test strategy and test cases

Unit test case design:

Test cases should be designed to ensure that individual units of program are structurally correct and function as expected.
These tests should adequately exercise functionality of all internal components, program logic and loop conditions. Such
test cases later become the foundation for Regression testing practice (regular, repeatable verification of the code base).
Unit test design typically involves writing test harnesses, specifying input data and supplying stubs/mock objects for
missing functions.

Unit testing automation tools provide us unit testing capabilities in simplified manner. They come with the capability to
generate test cases in a high level at the click of a button, by using code as input. To test the complex logic implemented
code, tool generated tests can be customized to test all paths, conditions and logic.

Steps to design test cases in tool:

1. Define a project specific test configuration by defining,

-The scope of testing (what needs to be tested)
-Number of assert statements per test
-Timeout value for tests
-Test runner (testing framework to be used for execution) etc
2. Share the created configuration with the entire team
3. Generate the test cases using defined configuration fie.
4. Review the tests generated and update with application-specific input values
5. Try to pass all feasible boundary values as test input values to uncover more errors. Consider negative input
values s too
6. Identify parameterizable fields in a application and generate parameterized tests (Data driven tests)
7. Associate the data source with parameterized tests.

Unit test case execution and Defect closure:

Knowing the internal working & specified function that a product has been designed to perform, tests can be executed to
demonstrate each function is fully operational while at the same time searching for errors in each function. Post test
execution closure of defects has to be done with highest priority,

The general steps in closing violations are,

Isolate code along with unit test which has problem,

Test isolated code to confirm it fails,
Fix the bug,
Run test to verify it succeeds.

Tools provide an effective interface to run all tool generated/customized test cases with centralized testing, debugging
and reporting environment. On executing tool generated/user defined/legacy tests it reports problems like functionality
related issues, regression related issues, unexpected runtime exceptions organized by error categories and severities.

Recommended steps to execute test cases and close defects using tool:

1. Run the test cases generated/modified

 2. Verify and validate the test execution results reported by tool
3. Isolate the failures reported by tool, prioritize and fix them by editing the offending source-code. As a good
practice, fix the problems related to exceptions first, then functionality problem followed by others
4. Re-run the tests to ensure the correctness
5. Baseline these tests for further regression testing
6. Run the base-lined set of tests on regular basis to validate code behavior with changes
7. Track this activity at project level through review meetings

Coverage analysis:

Code coverage analysis measures the degree to which the source code has been tested. It is the way to verify
effectiveness of source code as well as test cases. In other words, the coverage of code is the best indication of how well
the code has been tested. Coverage analysis results can also be used to identify unexecuted code of program (dead code).

Most of the unit testing tools offer both customizable annotated representation of covered/ uncovered code and sharable
coverage statistics. Coverage analyzer can be executed post test case execution to get the coverage details in which
coverage at different levels including line coverage and decision coverage can be tracked.

Recommended actions in case of lesser coverage:

1. Clean up the code by removing/modifying the unexecuted lines of code

2. Review tests and test input values to include missed conditions/scenarios
3. Parameterize the test input by associating with data-sources like excel, databases to pass wide range of input
values
4. Use user defined objects, stubs and mock objects to handle units with domain specific objects and dependencies

Comparison of some commonly used tools:

Features of tools
Technol
Tools Project specific Automatic Test Test Data Coverage Build Integratio
ogy Reports
configuration Generation Execution Sourcing Analysis integration n with IDE
WiproUT        
Java
JTest        
*Junit,
*Cactus x x  x x   
C++Test    x    
C/C++
*CppUnit x x  x x   
.Test    x    
.Net
*NUnit x x  x x   

* Frameworks

Conclusion:

All of us are interested in improving the quality of the applications we develop. However, formal manual testing
methodologies are simply impractical to implement for 100% of code coverage. Tool-assisted process offers both an
efficient and effective ways to identify and locate defects –without requiring painstaking tasks.
Further, the best tools alone aren’t enough; well-defined processes aren’t enough. What’s required is,
A lifecycle approach to quality,
Structured planning and design of the environment,
Consistent use of best practices.