System Testing

The query occurs periodically during production tests as to whether it is safe to terminate testing

and deliver the product with the assumption that it will attain its reliability goals (Goel and

Okumoto, 1981). Increased warranty prices, consumer frustration, and loss of market share result

from launching the product too early. Similarly, continued research beyond what is expected

results in losses: lack of time to launch, loss of market reach, and, eventually, loss of market

share. For the developer, the problem becomes that of determining the best time to finish the

product testing and publication (Donavan and Murphy, 2001).

Consequence of Implementing a System with Limited or No Testing

Medical instruments are potentially dangerous, including certain radiation therapy equipment

and infusion pumps. Implanted devices face a special danger, since while only one user is

affected by a particular malfunction, a bug in a device's functionality may create errors over the

entire user community. Around 1990 and 2000, over 200,000 users were hit by safety recalls of

pacemakers and implantable cardioverter-defibrillators attributable to firmware (i.e. software)

concerns, representing 41 percent of the recalled devices and increasing in frequency.

The FDA's Maude database reports nearly 30,000 fatalities and nearly 600,000 accidents from

system malfunction in the 20-year span from 1985 to 2005. In a research carried out by the FDA

between 1992 and 1998, 242 out of 3,140 product recalls (7.7 percent) were discovered to be due

to defective software. Of these, 192 were triggered by faults during software servicing, almost 80

percent. The real occurrence of software errors in medical devices is actually much higher than

these statistics indicate, as indicated by a research conducted by GAO.

Ignoring System Testing

Under the following conditions, system testing can be ignored. 

 Testing Deadlines.

 Completion of execution of test cases.

 Completion, to a certain extent, of practical and code coverage.

 The rate of bugs drops below a certain threshold and no bugs of high importance are

found.

 Decision by Management. 

It is obvious that the decision to begin testing or to stop is a normal trade-off: (a) If testing ends

so early, several bugs persist. We therefore encounter the expense of subsequent fixing of bugs

and losses as a result of the disappointment from consumers. (b) If the test proceeds up to the full

time allowable, then the expense of the test effort and the depletion of the consumer initiative

shall be sustained (Dalal Mallows, 1988).

Reference

Dalal, S. R., & Mallows, C. L. (1988). When should one stop testing software? Journal of the

American Statistical Association, 83(403), 872-879.

https://www.tandfonline.com/doi/abs/10.1080/01621459.1988.10478676

Donovan, J., & Murphy, E. (2001). When to stop development testing: An infrequently used

stopping rule revisited. Quality Engineering, 13(3), 367-376.

https://www.tandfonline.com/doi/pdf/10.1080/08982110108918664

Goel, A. L., & Okumoto, K. (1981). When to stop testing and start using software? ACM

SIGMETRICS Performance Evaluation Review, 10(1), 131-138. Retrieved from

https://dl.acm.org/doi/abs/10.1145/1010627.807921