Aditi Technologies

Basic Concepts of Software Testing

Software Testing
Basic Concepts and Industry awareness

Page 1 of 60

Aditi Technologies

Basic Concepts of Software Testing

Table of Contents 1. Introduction.................................................................................................4 2. Types of Testing...........................................................................................4 2.1. White Box Testing...................................................................................4 2.2. Black Box Testing....................................................................................4 2.3. Unit Testing............................................................................................5 2.3.1. Benefits ...........................................................................................5 2.3.2. Encourages change ...........................................................................5 2.3.3. Simplifies Integration ........................................................................5 2.3.4. Documents the code .........................................................................5 2.3.5. Separation of Interface from Implementation .......................................6 2.3.6. Limitations .......................................................................................6 2.4. Integration testing .................................................................................6 2.4.1. Purpose ...........................................................................................6 2.5. Performance Testing................................................................................7 2.5.1. Technology ......................................................................................7 2.5.2. Performance specifications .................................................................7 2.5.3. Tasks to undertake ...........................................................................8 2.6. Stress Testing ........................................................................................8 2.7. Security Testing......................................................................................9 2.7.1. Security Testing Techniques ...............................................................9 2.8. Usability Testing.....................................................................................9 2.9. Stability Testing....................................................................................10 2.10. Acceptance Testing..............................................................................10 2.11. Installation Testing..............................................................................10 2.12. Alfa Testing........................................................................................11 2.13. Beta Testing.......................................................................................11 2.14. Product Testing...................................................................................11 2.15. System Testing...................................................................................12 2.16. Regression Testing...............................................................................12 2.17. Compatibility Testing...........................................................................13 2.18. Test Cases, Suits, Scripts and Scenario..................................................13 2.19. Defect Tracking...................................................................................14 2.20. Formal Verification...............................................................................14 2.20.1. Validation and Verification ..............................................................14 2.21. Fuzz Testing.......................................................................................15 2.21.1. Uses ............................................................................................15 2.21.2. Fuzz testing methods .....................................................................15 2.21.3. Event-driven fuzz ..........................................................................16 2.21.4. Character-driven fuzz ....................................................................16 2.21.5. Database fuzz ...............................................................................16 3. Manual Testing...........................................................................................17 3.1. Facts...................................................................................................17 3.2. Software Crisis .....................................................................................17 3.3. Software Myths ....................................................................................17 3.3.1. Management Myths .........................................................................17 3.3.2. Developers Myths ...........................................................................17 3.3.3. Customer’s Myth .............................................................................18 3.3.4. What do we do? ..............................................................................18 3.4. Software Quality Assurance: ..................................................................18 3.4.1. Verification: ...................................................................................18 3.4.2. Validation: .....................................................................................18 3.5. Software Life Cycle Models: ...................................................................18

Page 2 of 60

Aditi Technologies

Basic Concepts of Software Testing

3.6. What makes a good Software QA engineer? .............................................18 3.7. Testing: ..............................................................................................19 3.7.1. Why Testing? .................................................................................19 3.8. Test Life Cycle .....................................................................................19 3.9. Testing Techniques ...............................................................................19 3.10. Test Plan: ..........................................................................................19 3.10.1. Test Specification: .........................................................................20 4. Testing Procedure.......................................................................................20 4.1. Bug Tracking .......................................................................................20 5. Testing Tools and Software..........................................................................23 5.1. Load and Performance Test Tools ...........................................................23 5.2. Java test Tools......................................................................................23 5.3. Link Checking Tools...............................................................................27 5.4. Perl Testing Tools..................................................................................28 5.5. Web Functional and Regression Testing Tools............................................28 5.6. Web Site Security Test Tools...................................................................33 5.7. Web Site Management Tools...................................................................37 5.8. Other Web Testing Tools........................................................................45 6. Testing FAQ ..............................................................................................50

Page 3 of 60

Black Box Testing Testing of a function without knowing internal structure of the program. Page 4 of 60 . White box testing does not account for errors caused by omission. stability. There are many approaches to software testing. 2. Types of Testing 2. glass-box and clear-box. but effective testing of complex products is essentially a process of investigation. and closed-box. White Box Testing White box testing is also known as glass box. Synonyms for black-box include: behavioral. white box testing uses specific knowledge of programming code to examine outputs. He or she can then see if the program diverges from its intended goal. In other words Testing is nothing but CRITICISM or COMPARISION.2. and all visible code must also be readable. Black-box and white-box are test design methods. portability. Although most of the intellectual processes of testing are nearly identical to that of review or inspection. testing can never completely establish the correctness of computer software. Black-box test design is usually described as focusing on testing functional requirements. The quality of the application can and normally does vary widely from system to system but some of the common quality attributes include reliability. maintainability and usability. Introduction Testing is a process used to help identify the correctness. where the "questions" are things the tester tries to do with the product. Unlike black box testing.1. Black-box test design treats the system as a "black-box". With that in mind. so it doesn't explicitly use knowledge of the internal structure. and it focuses specifically on using internal knowledge of the software to guide the selection of test data. This is a software testing technique whereby explicit knowledge of the internal workings of the item being tested are used to select the test data. completeness and quality of developed computer software. not merely a matter of creating and following rote procedure. opaque-box. the word testing is connoted to mean the dynamic analysis of the product—putting the product through its paces. Here comparison in the sense comparing the actual value with expected one. One definition of testing is "the process of questioning a product in order to evaluate it". Synonyms for white-box include: structural. functional. The test is accurate only if the tester knows what the program is supposed to do. and the product answers with its behavior in reaction to the probing of the tester. clear box and open box testing.Aditi Technologies Basic Concepts of Software Testing 1. White-box test design allows one to peek inside the "box". structural. 2. Refer to the ISO standard ISO 9126 for a more complete list of attributes and criteria.

The idea is to write test cases for every non-trivial function or method in the module so that each test case is separate from the others if possible.) can use any test design methods.Aditi Technologies Basic Concepts of Software Testing While black-box and white-box are terms that are still in popular use.Encourages change Unit testing allows the programmer to re-factor code at a later date. but others wish we'd stop talking about boxes altogether. This provides the benefit of encouraging programmers to make changes to the code since it is easy for the programmer to check if the piece is still working properly. Page 5 of 60 . 2. but it's still discouraged. Some call this "gray-box" or "translucent-box" test design. In practice.2.3. Behavioral test design is slightly different from black-box test design because the use of internal knowledge isn't strictly forbidden. and their influence is hard to see in the tests once they're implemented. Note that any level of testing (unit testing. By testing the parts of a program first and then testing the sum of its parts will make integration testing easier. This type of testing is mostly done by the developers.Benefits The goal of unit testing is to isolate each part of the program and show that the individual parts are correct. etc. It provides a written contract that the piece must satisfy. One has to use a mixture of different methods so that they aren't hindered by the limitations of a particular one. 2. many people prefer the terms "behavioral" and "structural". and make sure the module still works correctly (regression testing).1.3.Documents the code Unit testing provides a sort of "living document" for the class being tested. This isolated testing provides four main benefits: 2.3.3. a unit test is a method of testing the correctness of a particular module of source code.Simplifies Integration Unit testing helps eliminate uncertainty in the pieces themselves and can be used in a bottom-up testing style approach.3. Unit testing is usually associated with structural test design. It is important to understand that these methods are used during the test design phase. Unit Testing In computer programming. 2.4. system testing. 2. but this is because testers usually don't have well-defined requirements at the unit level to validate. Clients looking to learn how to use the class can look at the unit tests to determine how to use the class to fit their needs. it hasn't proven useful to use a single test design method.3.

groups them in larger aggregates. and delivers as its output the integrated system ready for system testing.Aditi Technologies Basic Concepts of Software Testing 2.5.6. are exercised through their interfaces using Black box testing.4. 2. the software developer abstracts an interface around the database connection. These "design items". The overall idea is the "building block" approach in which verified assemblages are added to a verified base which is then used to support the Integration testing of further assemblages. It follows unit testing and precedes system testing. 2. it only tests the functionality of the units themselves. across procedure calls or process activations. in order to test the class.Purpose The purpose of Integration testing is to verify functional. individual subsystems are exercised through their input interface. Therefore.Limitations It is important to realize that unit-testing will not catch every error in the program. 2.3. the tester finds herself writing code that interacts with the database.1.4. This is a mistake. This results in loosely coupled code. i. because a unit test should never go outside of its own class boundary. Integration testing Integration Testing is the phase of software testing in which individual software modules are combined and tested as a group. As a result. testing a class can frequently spill over into testing another class.3. In addition. takes as its input modules that have been checked out by unit testing. and then implements that interface with their own Mock Object. applies tests defined in an Integration test plan to those aggregates. Page 6 of 60 . Simulated usage of shared data areas and inter-process communication is tested. By definition. All test cases are constructed to test that all components within assemblages interact correctly. performance and reliability requirements placed on major design items. thus minimizing dependencies in the system. it may not be trivial to anticipate all special cases of input the program unit under study may receive in reality.Separation of Interface from Implementation Because some classes may have references to other classes. A common example of this is classes that depend on a database. Unit testing is only effective if it is used in conjunction with other software testing activities. it will not catch integration errors. assemblages (or groups of units). performance problems and any other system-wide issues. for example. success and error cases being simulated via appropriate parameter and data inputs.e.

g. However. are available to perform such tests. The test result shows how the performance varies with the load. that while the average response time might be acceptable. response time. Performance testing can serve different purposes. etc.2. including Compuware Corporation's QACenter Performance Edition.1.Aditi Technologies Basic Concepts of Software Testing 2. performance testing is frequently used as part of the process of performance profile tuning. Sometimes the results can reveal oddities. will result in the overall system running faster. It is sometimes a difficult task to identify which part of the system represents this critical path. 2. a separate PC acts as a test conductor. The idea is to identify the “weakest link” – there is inevitably a part of the system which. Various tools. The usual sequence is to ramp up the load – starting with a small number of virtual users and increasing the number over a period to some maximum. coordinating and gathering metrics from each of the injectors and collating performance data for reporting purposes. In performance testing. It can demonstrate that the system meets performance criteria. Usually. i.5. it is often crucial (and often difficult to arrange) for the test conditions to be similar to the expected actual use. e. Performance Testing In software engineering. in order to see what happens when an acceptable load is exceeded –does the system crash? How long does it take to recover if a large load is reduced? Does it fail in a way that causes collateral damage? 2.e. and some test tools come provided with (or can have add- Page 7 of 60 .Performance specifications Performance testing is frequently not performed against a specification. In the diagnostic case. Performance testing can be combined with stress testing. It can compare two systems to find which performs better. there are outliers of a few key transactions that take considerably longer to complete – something that might be caused by inefficient database queries. software engineers use tools such as profilers to measure what parts of a device or software contribute most to the poor performance or to establish throughput levels (and thresholds) for maintained acceptable response time. given as number of users vs.Technology Performance testing technology employs one or more PCs to act as injectors – each emulating the presence or numbers of users and each running an automated sequence of interactions (recorded as a script. or as a series of scripts to emulate different types of user interaction) with the host whose performance is being tested. Tools in this category usually execute a suite of tests which will emulate real users against the system. if it is made to respond faster..5.5. no one will have expressed what the maximum acceptable response time for a given population of users is. performance testing is testing that is performed to determine how fast some aspect of a system performs under a particular workload. Or it can measure what parts of the system or workload cause the system to perform badly.

and even done in different parts of the country. network overhead.Aditi Technologies Basic Concepts of Software Testing ons that provide) instrumentation that runs on the server and reports transaction times. Set up of a configuration of injectors/controller Set up of the test configuration (ideally identical hardware to the production platform). quiet network (we don’t want results upset by other users). If there can also be a statement of what constitutes the maximum allowable 95 percentile response time. or investigation of critical path and recommendation of corrective action. where they had found the system spent most of its time.5. There is an apocryphal story of a company that spent a large amount optimizing their software without having performed a proper analysis of the problem. but even having the most efficient idle loop in the world obviously didn’t improve overall performance one iota! Performance testing almost invariably identifies that it is parts of the software (rather than hardware) that contribute most to delays in processing users’ requests. 2. etc.Tasks to undertake Tasks to perform such a test would include: • • • • • • Analysis of the types of interaction that should be emulated and the production of scripts to do those emulations Decision whether to use internal or external resources to perform the tests. Performance testing can be performed across the web. database access times.6. router configuration. Running the tests – probably repeatedly in order to see whether any unaccounted for factor might affect the results. It can also be done in-house. 2. since it is known that the response times of the internet itself vary regionally. in order to observe the results. either pass/fail. a web server may be stress tested Page 8 of 60 . They ended up rewriting the system’s ‘idle loop’. often to a breaking point. It is always helpful to have a statement of the likely peak numbers of users that might be expected to use the system at peak times. Without such instrumentation one might have to have someone crouched over Windows Task Manager at the server to see how much CPU load the performance tests are generating. Stress Testing Stress Testing is a form of testing that is used to determine the stability of a given system or entity. then an injector configuration could be used to test whether the proposed system met that specification. which can be analyzed together with the raw performance statistics. deployment of server instrumentation. Analyzing the results.3. It involves testing beyond normal operational capacity. although routers would then need to be configured to introduce the lag what would typically occur on public networks. For example.

Data corruption and application Defacement. Data theft. If usability testing uncovers difficulties. Usability Testing Usability testing is a means for measuring how well people can use some humanmade object (such as a web page.7. compromising usability and functionality. the detection of application vulnerabilities requires independent evaluation of your specific application's features and functions by experts. i. Security Testing Application vulnerabilities leave your system open to attacks. including making things work with people. 2. Usability testing focuses on a particular object or a small set of objects. usability testing measures the usability of the object. A designers' primary function should be more than appearance. An external security vulnerability review by Third Eye Testing will give you the best possible confidence that your application is as secure as possible. manipulating parts.1.Security Testing Techniques • • • • • • • • Vulnerability Scanning Network Scanning Password Cracking Log Views Virus Detect Penetration Testing File Integrity Checkers War Dialing 2. to discover errors and areas of improvement. then developers should improve the design and test it again. or a device) for its intended purpose. "Caution: simply gathering opinions is not usability testing -. bots. While automated tools can help to eliminate many generic security issues. and various denial of service tools to observe the performance of a web site during peak loads. 2.8. or interpreting feedback. Designers commonly focus excessively on creating designs that look "cool". whereas general human-computer interaction studies attempt to formulate universal principles." Page 9 of 60 . Stress testing a subset of load testing. During usability testing.Aditi Technologies Basic Concepts of Software Testing using scripts.7.you must arrange an experiment that measures a subject's ability to use your document. the aim is to observe people using the product in as realistic a situation as possible.e. Also see testing. such as people having difficulty understanding instructions. a document. software testing. a computer interface. performance testing. Downtime. Security within an application or web service is crucial to avoid such vulnerabilities and new threats. This is often caused by pressure from the people in charge. forcing designers to develop systems based on management expectations instead of people's needs.

For example. wherein the person performs a list of tasks using the product being tested while observers watch and take notes. the test subjects should be given the instructions and a box of parts. The idea is that if the software works as intended and without issues during a simulation of normal use. Rather.). and pre. 2. usability testing involves watching people trying to use something for its intended purpose. illustration quality. The focus in this type of testing is less on simple problems (spelling mistakes. software will not run etc. cosmetic problems) and show stoppers (major problems like the software crashing. 2.9. and what people like. or realistic situation.Aditi Technologies Basic Concepts of Software Testing Rather than showing users a rough draft and asking. Page 10 of 60 . and the toy's design all affect the assembly process. The test scripts will emulate real-world usage of the system.and post-test questionnaires are also used to gather feedback on the product being tested. when testing instructions for assembling a toy. Users of the system will perform these tests which. stability testing is an attempt to determine if an application will crash. developers have derived from the User Requirements Specification. to test the attachment function of an e-mail program. 2. In the pharmaceutical field. paper prototypes. it refers to a period of time during which a multi-dose product retains its quality after the container is opened. so that developers can see problem areas. Setting up a usability test involves carefully creating a scenario. Instruction phrasing. a scenario would describe a situation where a person needs to send an e-mail attachment. For example. Developers should have worked out these issues during unit testing and integration testing. ideally. Acceptance Testing User acceptance testing (UAT) is one of the final stages of a software project and will often occur before the customer accepts a new system. the focus is on a final verification of the required business function and flow of the system. Installation Testing Installation testing (in software engineering) can simply be defined as any testing that occurs outside of the development environment. Test designers will draw up a formal test plan and devise a range of severity levels. and ask him or her to undertake this task. Results of these tests will allow both the customers and the developers to be confident that the system will work as intended. Stability Testing In software testing.10. to which the system should conform. "Do you understand this?". it will work just the same in production.11. The aim is to observe how people function in a realistic manner. The technique popularly used to gather data during a usability test is called a think aloud protocol. Several other test instruments such as scripted instructions.

13. Product testing experts design the test process to take advantage of the economies of scope and scale that are present in a software product. to use it / test it and come back with feedback or bugs. the software is released to a limited audience who would finally form the end users. Deployment plans in such circumstances may include back-out procedures whose use is intended to roll the target environment back in the event that the deployment is unsuccessful. This is often known as the second stage of alpha testing. however no serious or critical bugs would exist. Product Testing Software Product development companies face unique challenges in testing. Ideally. the generation of that setup program itself and its efficacy in a variety of machine and operating system environments can require extensive testing before it can be used with confidence. Whilst the ideal installation might simply appear to be to run a setup program. 2. This process helps in determining whether the final software meets its intended purpose and whether the end users would accept the same. In-house developers often test the software in what is known as 'ALPHA' testing which is often performed under a debugger or with hardware-assisted debugging to catch bugs quickly. 2. A beta release is very close to the final release. 2. This technique is known as black box testing. testing is usually required before release to the general public. particularly where software is to be released into an already live target environment (such as an operational web site) installation (or deployment as it is sometimes called) can involve database schema changes as well as the installation of new software. Beta Testing Many a time. It can then be handed over to testing staff for additional inspection in an environment similar to how it was intended to be used. the deployment plan itself should be tested in an environment that is a replica of the live environment.12. Alfa Testing In software development. The product handed out as a Beta Release is not bug free. In distributed systems.14. Only suitably organized and executed test process can contribute to the success of a software product.Aditi Technologies Basic Concepts of Software Testing Such testing will frequently occur on the computer system the software product will eventually be installed on. A factor that can increase the organizational requirements of such an exercise is the need to synchronize the data in the test deployment environment with that in the live environment with minimum disruption to live operation. Page 11 of 60 .

as its input.Aditi Technologies Basic Concepts of Software Testing These activities are sequenced and scheduled so that a test activity occurs immediately following the construction activity whose output the test is intended to validate. integrated system to evaluate the system's compliance with its specified requirements. The purpose of Integration testing is to detect any inconsistencies between the software units that are integrated together called assemblages or between any of the assemblages and hardware. all of the "integrated" software components that have successfully passed Integration testing and also the software system itself integrated with any applicable hardware system(s). Alpha testing and Beta testing are sub-categories of Regression testing. The purpose of Integration testing is to detect any inconsistencies between the software units that are integrated together called assemblages or between any of the assemblages and hardware. System testing is more of a limiting type of testing.). 1990. all of the "integrated" software components that have successfully passed Integration testing and also the software Regression itself integrated with any applicable hardware Regression(s). Regression testing is testing conducted on a complete. NY.15. Regression testing is more of a limiting type of testing. NY. and as such. 1990.16. Alpha testing and Beta testing are sub-categories of System testing. System testing is testing conducted on a complete. System Testing According to the IEEE Standard Computer Dictionary. According to the IEEE Standard Computer Dictionary.). integrated system to evaluate the system's compliance with its specified requirements. and as such. 2. Regression Testing Regression Testing is typically carried out at the end of the development cycle. as its input. During this testing. System testing falls within the scope of Black box testing. System testing takes. where it seeks to detect both defects within the "inter-assemblages" and also the system as a whole. where it seeks to detect both defects within the "inter-assemblages" and also the system as a whole. As a rule. IEEE Standard Computer Dictionary: A Compilation of IEEE Standard Computer Glossaries. New York. all bug previously identified and fixed is tested along with it's impacted areas to confirm the fix and it's impact if any. New York. Regression testing takes. Regression testing falls within the scope of Black box testing. should require no knowledge of the inner design of the code or logic (IEEE. should require no knowledge of the inner design of the code or logic (IEEE. IEEE Standard Computer Dictionary: A Compilation of IEEE Standard Computer Glossaries. As a rule. Page 12 of 60 . 2.

(c) credible. or even a test scenario. These steps can be stored in a word processor document. The optional fields are a test case ID. They are usually different from test cases in that test cases are single steps and scenarios cover a number of steps. Test Cases.Aditi Technologies Basic Concepts of Software Testing 2. Suits. (d) complex. test step or order of execution number. spreadsheet. Test scripts should be written for modules with the highest risk of failure and the highest impact if the risk becomes an issue. They can be as simple as a diagram for a testing environment or they could be a description written in prose. In a database system. They may also be called a test script. It definitely contains a section where the tester identifies the system configuration used during testing. depth. Page 13 of 60 . A group of test cases may also contain prerequisite states or steps. and (e) easy to evaluate. Test suites and scenarios can be used in concert for complete system tests. you may also be able to see past test results and who generated the results and the system configuration used to generate those results. along with various additional pieces of information. It can occasionally be a series of steps but with one expected result or expected outcome. related requirement(s). A scenario test is a test based on a hypothetical story used to help a person think through a complex problem or system. Compatibility Testing One of the challenges of software development is ensuring that the application works properly on the different platforms and operating systems on the market and also with the applications and devices in its environment. database or other common repository. Scripts and Scenario Black box testers usually write test cases for the majority of their testing activities.17. The test suite often also contains more detailed instructions or goals for each collection of test cases. and descriptions. and descriptions of the following tests. A test case should also contain a place for the actual result.18. Most companies that use automated testing will call the code that is used their test scripts. operating system and browser versions. The ideal scenario test has five key characteristics. and regression testing. It is (a) a story that is (b) motivating. system. Collections of test cases are sometimes incorrectly termed a test plan. The most common term for a collection of test cases is a test suite. author. 2. A test case is usually a single step. Larger test cases may also contain prerequisite states or steps. and its expected result. These past results would usually be stored in a separate table. Compatibility testing service aims at locating application problems by running them in real environments. Most white box tester write and use test scripts in unit. and check boxes for whether the test is automatable and has been automated. test category. thus ensuring you that the application is compatible with various hardware.

Petri nets. (by inspection. Verification: "Are we building the product right?” i.19. Validation is the complementary aspect. formal verification is the act of proving or disproving the correctness of a system with respect to a certain formal specification or property. System types that are considered in the literature for formal verification include finite state machines (FSM). Often one refers to the overall checking process as V & V. and reasoning with the aid of automatic theorem provers such as HOL or Isabelle.20. Validation: "Are we building the right product?” i.e. but the two concepts can be used in conjunction. testing. abstract interpretation.. abstraction refinement.. The main approaches to implementing formal verification include state space enumeration. 2. 2. does the product conform to the specifications. which is more closely related to exploratory testing. Formal Verification In the context of hardware and software systems.20. for a software product one can inspect the source code (static) and run against specific test cases (dynamic).. symbolic state space enumeration. combinatorial circuits. digital circuits with internal memory.e. timed automata and hybrid automata. cryptographic protocols. but not the same as session-based testing. using formal methods. Page 14 of 60 . The verification process consists of static and dynamic parts.g. 2. and abstractions of general software components.Aditi Technologies Basic Concepts of Software Testing Scenario testing is similar to. evaluating and prioritizing these defects is a difficult task.1. such as linear temporal logic (LTL) or computational tree logic (CTL). and tracking them to closure. defect tracking is the process of finding defects in a product. Validation and Verification Verification is one aspect of testing a product's fitness for purpose. labeled transition systems (LTS) and their compositions. Usually formal verification is carried out algorithmically. Defect tracking is important in software engineering as complex software systems typically have tens or hundreds of thousands of defects: managing. or recording feedback from customers). does the product do what the user really requires. The properties to be verified are often described in temporal logics. Defect Tracking In engineering. E. processalgebraic methods. Defect tracking systems are computer database systems that store defects and help people to manage them.

fuzz testing can only be regarded as a proxy for program correctness.Aditi Technologies Basic Concepts of Software Testing Validation usually can only be done dynamically. The advantage here is that the cost of generating the tests is relatively low. Uses Fuzz testing is often used in large software development projects that perform black box testing.21. developers need to reproduce errors in order to fix them. or possibly from a mechanism in an embedded system. almost all fuzz testing makes a record of the data it manufactures. and fuzz testing is one of the techniques which offer a high benefit to cost ratio. Fuzz testing is thought to enhance software security and software safety because it often finds odd oversights and defects which human testers would fail to find. rather than behaving correctly.. so that if the computer fails dramatically. the product is tested by putting it through typical usages and atypical usages ("Can we break it?"). Thus. • Character driven inputs are from files or data streams. such as relational databases.e. For example. and free of preconceptions about system behavior. and even careful human test designers would fail to create tests for. The basic idea is to attach the inputs of a program to a source of random data. If the program fails (for example. 2. with fuzz test failures actually being more useful as a bug-finding tool than fuzz test passes as an assurance of quality. fuzz testing is not a substitute for exhaustive testing or formal methods: it can only provide a random sample of the system's behavior. • Database inputs are from tabular data. The great advantage of fuzz testing is that the test design is extremely simple. or by failing in-built code assertions). third party testers have used fuzz testing to evaluate the relative merits of different operating systems and application programs. Fuzz Testing Fuzz testing is a software testing technique. i. 2. by crashing. rather than a direct measure. Fuzz testing is also used as a gross measurement of a large software system's quality. the test data is preserved. usually before applying it to the software.1. Fuzz testing methods As a practical matter. However. For this reason. 2. Modern software has several different types of inputs: • Event driven inputs are usually from a graphical user interface.21.21.2. There are at least two different forms of fuzz testing: Page 15 of 60 . These usually have a budget to develop test tools. and in many cases passing a fuzz test may only demonstrate that a piece of software handles exceptions without crashing. then there are defects to correct.

5. when the character data exceeds the available buffer space. Often the same schema descriptions can be used to automatically generate fuzz databases. Simple fuzz usually uses a pseudo random number generator to provide input. often the database is more tolerant of odd data than its client software. simply providing error status can cause resource problems or a crash.3.Aditi Technologies Basic Concepts of Software Testing • • • Valid fuzz attempts to assure that the random input is reasonable. To succeed in a fuzz-tested environment. The classic source in UNIX is the random data generator. because input and comparison constraints reduce the invalid data in a database. Robust error detection systems will report only the most significant or most recent error over a period of time. The most common problem with an event-driven program is that it will often simply use the data in the queue. Database fuzz is controversial. 2. Since major customer and enterprise management Page 16 of 60 . and a general-purpose interface is available to users. Another is that decode tables or logic may be incomplete. Database fuzz The standard database scheme is usually filled with fuzz that is random data of random sizes. Some IT shops use software tools to migrate and manipulate such databases.21. 2. without even crude validation. Event-driven fuzz Normally this is provided as a queue of data-structures. decode every possible binary value. software must validate all fields of every queue entry.21. One common problem with a character driven program is a buffer overrun. This problem tends to recur in every instance in which a string or number is parsed from the data stream and placed in a limited-size area. or conforms to actual production data. The queue is filled with data structures that have random values. Character-driven fuzz Normally this is provided as a stream of random data. By using all of these techniques in combination. 2. not handling every possible binary value. and then ignore impossible requests. fuzz-generated randomness can test the un-designed behavior surrounding a wider range of designed system states.21. One of the more interesting issues with real-time event handling is that if error reporting is too verbose.4. However. A combined approach uses valid test data with some proportion of totally random input injected. Fuzz testing may use tools to simulate all of these domains.

Aditi Technologies

Basic Concepts of Software Testing

software is starting to be open-source, database-based security attacks are becoming more credible. A common problem with fuzz databases is buffer overrun. A common data dictionary, with some form of automated enforcement is quite helpful and entirely possible. To enforce this, normally all the database clients need to be recompiled and retested at the same time. Another common problem is that database clients may not understand the binary possibilities of the database field type, or, legacy software might have been ported to a new database system with different possible binary values. A normal, inexpensive solution is to have each program validate database inputs in the same fashion as user inputs. The normal way to achieve this is to periodically "clean" production databases with automated verifiers.

3. Manual Testing
3.1.
• • •

Facts
In India itself, Software industry growth has been phenomenal. IT field has enormously grown in the past 50 years. IT industry in India is expected to touch 10,000 crores of which software share is dramatically increasing.

3.2.
• • •

Software Crisis
Software cost/schedules are grossly inaccurate. Cost overruns of several times, schedule slippage’s by months, or even years are common. Productivity of people has not kept pace with demand. Added to it is the shortage of skilled people. Productivity of people has not kept pace with demand Added to it is the shortage of skilled people.

3.3.
• • • • • •

Software Myths
Software Management is different. Why change or approach to development? We have provided the state-of-the-art hardware. Problems are technical If project is late, add more engineers. We need better people.

3.3.1.Management Myths

3.3.2.Developers Myths
• • • • We must start with firm requirements Why bother about Software Engineering techniques, I will go to terminal and code it. Once coding is complete, my job is done. How can you measure the quality...it is so intangible.

Page 17 of 60

Aditi Technologies

Basic Concepts of Software Testing

3.3.3.Customer’s Myth
• • A general statement of objective is good enough to produce software. Anyway software is “Flex-ware”, it can accommodate my changing needs.

3.3.4.What do we do?
• • • • Use Software Engineering techniques/processes. Institutionalize them and make them as part of your development culture. Adopt Quality Assurance Frameworks : ISO, CMM Choose the one that meets your requirements and adopt where necessary.

3.4.

Software Quality Assurance:

The purpose of Software Quality Assurance is to provide management with appropriate visibility into the process being used by the software project and of the products being built. • Software Quality Assurance involves reviewing and auditing the software products and activities to verify that they comply with the applicable procedures and standards and providing the software project and other appropriate managers with the results of these reviews and audits.

3.4.1.Verification:
• • Verification typically involves reviews and meetings to evaluate documents, plans, code, requirements, and specifications. The determination of consistency, correctness & completeness of a program at each stage.

3.4.2.Validation:
• • Validation typically involves actual testing and takes place after verifications are completed The determination of correctness of a final program with respect to its requirements.

3.5.
• • • •

Software Life Cycle Models:
Prototyping Model Waterfall Model – Sequential Spiral Model V Model - Sequential

3.6.

What makes a good Software QA engineer?

The same qualities a good tester has are useful for a QA engineer. Additionally, they must be able to understand the entire software development process and how it can fit into the business approach and goals of the organization.

Page 18 of 60

Aditi Technologies

Basic Concepts of Software Testing

Communication skills and the ability to understand various sides of issues are important. In organizations in the early stages of implementing QA processes, patience and diplomacy are especially needed. An ability to find problems as well as to see 'what's missing' is important for inspections and reviews.

3.7.
• • • • • •

Testing:
An examination of the behavior of a program by executing on sample data sets. Testing comprises of set of activities to detect defects in a produced material. To unearth & correct defects. To detect defects early & to reduce cost of defect fixing. To avoid user detecting problems. To ensure that product works as users expected it to.

3.7.1.Why Testing?
• • • • To To To To unearth and correct defects. detect defects early and to reduce cost of defect fixing. ensure that product works as user expected it to. avoid user detecting problems.

3.8.
• • • • • • • •

Test Life Cycle
Identify Test Candidates Test Plan Design Test Cases Execute Tests Evaluate Results Document Test Results Casual Analysis/ Preparation of Validation Reports Regression Testing / Follow up on reported bugs.

3.9.
• • •

Testing Techniques
Black Box Testing White Box Testing Regression Testing

These principles & techniques can be applied to any type of testing.

3.10.

Test Plan:

A Test Plan is a detailed project plan for testing, covering the scope of testing, the methodology to be used, the tasks to be performed, resources, schedules, risks, and dependencies. A Test Plan is developed prior to the implementation of a project to provide a well defined and understood project roadmap.

Page 19 of 60

Testing Procedure The following are some of the steps to consider: • • Obtain requirements. 4. A Test Specification is produced as the first step in implementing a Test Plan. required standards and processes (such as release processes. set up or obtain test input data Obtain and install software releases Perform tests Evaluate and report results Track problems/bugs and fixes Retest as needed Maintain and update test plans. reporting requirements. integration. etc. It provides a repeatable. software. functional design. Determine test approaches and methods . error classes Prepare test plan document and have needed reviews/approvals Write test cases Have needed reviews/inspections/approvals of test cases Prepare test environment and test-ware.Aditi Technologies Basic Concepts of Software Testing 3. milestones Determine input equivalence classes. timelines. obtain needed user manuals/reference documents/configuration guides/installation guides. etc. etc. usability tests. boundary value analyses. coverage analyzers. Obtain budget and schedule requirements. change processes. set priorities. and determine scope and limitations of tests.) Identify application's higher-risk aspects. Determine test environment requirements (hardware. communications. comprehensive definition of a testing campaign. system. Bug Tracking What's a 'test case'? Page 20 of 60 . and internal design specifications and other necessary documents. test cases. problem/bug tracking. those responsible for tasks. prior to the onset of manual testing and/or automated test suite development.1. test environment. Determine project-related personnel and their responsibilities. set up logging and archiving processes.) Determine test-ware requirements (record/playback tools. and labor requirements Set schedule estimates. functional. Test Specification: A Test Specification defines exactly what tests will be performed and what their scope and objectives will be. etc. set up test tracking processes.) Determine test input data requirements Identify tasks. and test ware through life cycle • • • • • • • • • • • • • • • • • • 4.1. test tracking. load.10.unit.

ID.g. relevant hardware specifics Test case name/number/identifier One-line bug description Full bug description Description of steps needed to reproduce the bug if not covered by a test case or if the developer doesn't have easy access to the test case/test script/test tool Names and/or descriptions of file/data/messages/etc. to determine if a feature of an application is working correctly. test conditions/setup. etc.. Bug identifier (number. platform. used in test File excerpts/error messages/log file excerpts/screen shots/test tool logs that would be helpful in finding the cause of the problem Severity estimate (a 5-level range such as 1-5 or 'critical'-to-'low' is common Was the bug reproducible? Tester name Test date Bug reporting date Name of developer/group/organization the problem is assigned to Description of problem cause Description of fix Code section/file/module/class/method that was fixed Date of fix Application version that contains the fix Tester responsible for retest Retest date Retest results • • • • • • • • • • • • • • • • • • • • • • • • • • Page 21 of 60 .) The application name or identifier and version The function. and expected results. objective. etc. input data requirements. since it requires completely thinking through the operation of the application. and determinations made regarding requirements for regression testing to check that fixes didn't create problems elsewhere. system. fixes should be re-tested. test case name. The following are items to consider in the tracking process: Complete information such that developers can understand the bug. etc. action. 'New'. and reproduce it if necessary. A test case should contain particulars such as test case identifier. object. For this reason.Aditi Technologies Basic Concepts of Software Testing • • A test case is a document that describes an input. or event and an expected response. get an idea of it's severity. screen. After the problem is resolved.) Current bug status (e. If a problem-tracking system is in place. feature. Note that the process of developing test cases can help find problems in the requirements or design of an application. where the bug occurred Environment specifics. steps. it's useful to prepare test cases early in the development cycle if possible. A variety of commercial problem-tracking/management software tools are available (see the 'Tools' section for web resources with listings of such tools). it should encapsulate these processes. module. 'Released for Retest'. What should be done after a bug is found? • The bug needs to be communicated and assigned to developers that can fix it.

In many • • • • • • • Page 22 of 60 .it's tough to maintain and modify code that is badly written or poorly documented. etc. And the use of objectoriented techniques can complicate instead of simplify a project unless it is well engineered. known and unknown dependencies among parts of the project are likely to interact and cause problems. like anyone else. developers need to know when bugs are found and how to get the needed information. hardware requirements that may be affected. can make mistakes. effects on other projects. Why does software have bugs? • • Miscommunication or no communication . Time pressures .scheduling of software projects is difficult at best. In some fast-changing business environments. Egos . If there are many minor changes or any major changes. enormous relational databases. work already completed that may have to be redone or thrown out. For instance. Enthusiasm of engineering staff may be affected. When deadlines loom and the crunch comes.the customer may not understand the effects of changes. the result is bugs. testers need to know when retesting is needed. In this case. we'll wing it' o 'I can't estimate how long it will take. mistakes will be made. Changing requirements . or may understand and request them anyway . and sheer size of applications have all contributed to the exponential growth in software/system complexity. and reporting/summary capabilities are needed for managers. Programming errors .people prefer to say things like: o 'no problem' o 'piece of cake' o 'I can whip that out in a few hours' o 'it should be easy to update that old code' Instead of: o 'that adds a lot of complexity and we could end up o making a lot of mistakes' o 'we have no idea if we can do that. management must understand the resulting risks. often requiring a lot of guesswork. client-server and distributed applications.as to specifics of what an application should or shouldn't do (the application's requirements). data communications.redesign. Poorly documented code . until I take a close look at it' o 'we can't figure out what that old spaghetti code did in the first place' If there are too many unrealistic 'no problems'. rescheduling of engineers. continuously modified requirements may be a fact of life. Windows-type interfaces. Software complexity .the complexity of current software applications can be difficult to comprehend for anyone without experience in modern-day software development. the result is bugs. and QA and test engineers must adapt and plan for continuous extensive testing to keep the inevitable bugs from running out of control.Aditi Technologies Basic Concepts of Software Testing • • • • Regression testing requirements Tester responsible for regression tests Regression testing results A reporting or tracking process should enable notification of appropriate personnel at various stages. and the complexity of keeping track of changes may result in errors.programmers.

Monitors applications as soon as installed no coding is needed. GUI interface emphasizing ease of use. Java test Tools Java Development Tools Java coverage. Recorded GUI elements. it should be hard to read'). EJBs. Includes a natural user interface. presents data in easy-to-use customizable dashboards which enable deep. etc. class libraries. and other server technologies as well.Aditi Technologies Basic Concepts of Software Testing • organizations management provides no incentive for programmers to document their code or write clear. Load and Performance Test Tools AppPerfect DevSuite 5.2. Extensive documentation. regression testing for JAVA VM upgrades. Code Analyzer. and contract (black box) test tool from TreborSoft. Tag Libs. execution and management of automated Java/Swing application tests. and a component recognition algorithm that takes into account a variety of attributes. and server application's modules. AppPerfect DevSuite Suite of testing. Java/J2EE Profiler and other modules. qftestJUI Record/playback test tool from Quality First Software for creation. regression. In fact. and there's job security if nobody else can understand it ('if it was hard to write. it's usually the opposite: they get points mostly for quickly turning out code. Introscope Performance monitoring tool from Wily Technology. and clone detection tools from Semantic Designs. Testing Tools and Software 5.visual tools. and monitoring products for java development from AppPerfect Corp. GJTester Java unit. Cactus A simple open-source test framework for unit testing server-side java code (Servlets. scripting capabilities. scripting tools. tuning. 5. 'Transaction Tracer' can provide detailed tracing of execution paths and component response times for individual transactions in production systems. without implementing test clients. Test private and protected functions. Enables test case and test script development without programming. profiler. user actions and associated data are automatically integrated into an editable tree view reflecting the hierarchical structure of the application's GUI. RMI. often introduce their own bugs or are poorly documented. metrics. Filters. resulting in added bugs. etc. Useful for testing CORBA. intuitive views of interrelation between system components and application infrastructure. understandable code. Includes: Unit Tester.1. Software development tools .). Included 'LeakHunter'identifies potential memory leaks. Intent is to allow fine-grained continuous testing of all files Page 23 of 60 . compilers.

Koalog Code Coverage Code coverage analyzer for Java applications from Koalog SARL. Enables quick composition of a performance test suite. From Apache Software Foundation.Aditi Technologies Basic Concepts of Software Testing making up an application: source code but also meta-data files (such as deployment descriptors. By Mike Clark/Clarkware Consulting. licensed under the BSD License. (Abbot = "A Better 'Bot'). For use by developers implementing unit tests in Java. HTML. using Ant. etc) through an in-container approach. Typically use within your IDE. The framework may be invoked directly from Java code or accessed without programming through the use of scripts via 'Costello'. Site includes a large collection of extensions and documentation. Free Open Source Software released under the IBM Public License and hosted on SourceForge. LaTex. Supports customizing existing rules and defining custom rules. and session merging to allow compilation of overall results for distinct executions. a script editor/recorder. Intended for use where there are performance/scalability requirements that need re-checking while refactoring code.available under the GNU Lesser General Public License. Integrates with Ant and JUnit. Jemmy A Java library that is used to create automated tests for Java GUI applications. Abbot Java GUI Test Framework Testing framework by Timothy Wall provides automated event generation and validation of Java GUI components.awt. jfcUnit Framework for developing automated testing of Java Swing-based applications at the UI layer (as opposed to testing at lower layers. or from the command line. QStudio for Java Java code inspection tool from QA Systems allows automation of a major portion of code inspection process. Contains Page 24 of 60 . which can then be run automatically and independent of other JUnit tests. CSV. Couples advanced static analysis capabilities to ISO 9126 quality standard framework. It uses JUnit and extends it. Free Open Source Software from SourceForge site. improving upon the very basic functions provided by the java. Also available as plugins for JBuilder and Eclipse. capability of working directly on Java method binaries (no recompilation). Provides recording and playback capabilities. Integrates with leading Java Development Environments and platforms. predefined (XML. for early detection of software defects and automatic assessment of code quality. Suitable for use both by developers for unit tests and QA for functional testing.Robot class. TEXT) or custom report generation. Free . Includes: in-process or remote coverage computation. for which JUnit may be sufficient).Framework to write repeatable java unit tests A regression testing framework written by Erich Gamma and Kent Beck. JUnitPerf Allows performance testing to be dynamically added to existing JUnit tests. JUnit .

file. classes. J_SQA for ObjectOriented software quality measurement. JMSAssert. JCover test coverage analyzer. tree node expanding. J_Diagrammer for Java code logic analysis. package. JCover Java code test coverage analysis tool from Codework Limited. at all levels and between all levels. virtual machines. and dependencies for Java. Fifty different audits and metrics. PDF. Cyclomatic Number. can 'reason' about selective need for regression testing Java classes. method. class. Halstead Measures. or jar file. components. Clover Code coverage tool for Java from Cenqua. PerformaSure Low-overhead. source distribution. Seamless integration with projects using Apache ANT. etc. it can be used separately as well as together with the NetBeans IDE. C.. reView Java source code visualization tool from Headway Software. JBench Freeware Java benchmarking framework to compare algorithms. JPretty reformats Java code according to specified options. control flow analysis and diagramming. JemmyTest is a program written in Java which uses the Jemmy API to test applications. and J_Playback for GUI operation capture and automatic playback. a tool and technique for writing reliable software. configurable. TrueJ Source code audit and metrics tool from BlueBay systems. JEvolve. JSynTest. or via a Swing GUI. compiler-style output. JBuilder. methods. Reverse engineer and automatically lay out and view code. integrates with a variety of editors/IDE's. package and produces reports in multiple formats. View coverage data in XML. HTML.e. an intelligent Java code evolution analyzer that automatically analyzes multiple versions of a Java program and shows how various classes have evolved across versions. JVerify Java class/API testing tool uses an invasive testing model allowing access to internals of Java objects from within a test script and utilizes a proprietary OO scripting language. and other IDE's.. and C++ applications. Coverage API provided. user-friendly performance diagnosis tool from Quest Software for Page 25 of 60 . Gathers coverage measures of branches.. J_Test for test coverage analysis and test case minimization. highly scalable. integrates with build tools for quality gate and reporting. Fully integrated plugin for NetBeans. J_DocGen for Java code static analysis. for speed. JBrowser class browser. Inc. button pushing. Works with source or compiled files. a Java source analyzer to generate code comments and metrics such as inheritance depth.). Shows all dependencies. application. statements. a syntax testing tool that automatically builds a Java-based test data generator. etc. . Panorama for Java Visual environment containing six integrated java tools from ISA. Java Tool Suite from Man Machine Systems Includes JStyle. etc.Aditi Technologies Basic Concepts of Software Testing methods to reproduce all user actions which can be performed on Swing/AWT components (i. text typing. J_Structure for Java code structure analysis and diagramming. Coverage difference comparison between runs. Available as binary distribution (including documentation).

(Note: some other tools in these listings also handle testing. or are planning to add such capabilities. Check listed web sites for current information. Integrates with several Java IDE's. OS. Includes Cyclomatic Complexity. It automatically generates and executes JUnit tests and checks whether code follows 400 coding standards and can automatically correct for many. to diagnose and resolve performance bottlenecks. maintainability. Has online advisor for quality improvement.) Page 26 of 60 . and network metrics. VTune Intel's performance tuning tool for applications running on Intel processors. JProfiler freeware version available. management. includes Java support. Sun's Java Test Tools As of February 4. written in Java. Enhanced Cyclomatic Complexity. code coverage analyzer and code analysis for Java. 2000 Sun discontinued accepting orders for these products. OptimizeIt Profiler. and clarity. complexity. Hundreds of easily-confugured run-time. automatic unit testing and standards compliance tool for Java. TCAT for Java Part of Software Research's TestWorks suite of test tools. and JProbe Threadalyzer for finding deadlocks. JProbe Developer Suite Collection of Java debugging tools from Quest Software. and race conditions. servlets. LProbe Coverage code coverage tool. or load testing of java applets. coverage analysis. Krakatau Metrics for Java Software metrics tool from Power Software includes more than 70 OO. Jtest ParaSoft's Jtest is an integrated. thread debugger. thread and event analysis. and code coverage tool suite from Borland (formerly from VMGear). stalls. and applications. testability. LOC metrics and MOOD metrics. Traces and reconstructs execution path of end-user transactions across all components of a clustered multi-tieer J2EE system.Aditi Technologies Basic Concepts of Software Testing distributed J2EE applications. Halstead Software Science metrics. includes JProbe Profiler and JProbe Memory Debugger for finding performance bottlenecks and memory leaks. Includes suggestions for optimization techniques. and size metrics related to reusability. procedural. DevPartner Java Edition Compuware's (formerly NuMega) debugging/productivity tool to detect and diagnose Java bugs and memory and performance problems.

Xenu's Link Sleuth Freeware link checker by Tilman Hausherr. employs a simple. Site Map. Ideal for dynamic sites requiring frequent link checking. DNS servers and SSL certificates. Site Audit Low-cost on-the-web link-checking service from Blossom Software. Excel. used to test and validate critical website components. can be automatically scheduled. Linkalarm Low cost on-the-web link checker from Link Alarm Inc. previews files when fixing broken links and before orphan removal. Web Link Validator Link checker from REL Software checks links for accuracy and availability. CSV. InfoLink Link checker program from BiggByte Software. HTML. Link Checking Tools HiSoftware Link Validation Utility Link validation tool. Runs as often as every hour. fixes broken links with an easy. Automatically-scheduled reporting by e-mail.. replaces links but does not reformat or restructure HTML code. For Windows. 3-click process. and FTP protocols. Export to text. detects and reports redirected URL.3. customizable reports. RTF. Freeware 'REL Link Checker Lite' version available for small sites. can also produce a graphical site map of entire web site. free trial period available. multiple page list and site list capabilities. free version or low-cost pro version. includes FTP link checking. several report formats available. Link Checker Pro Link check tool from KyoSoft. Handles HTTP. For Windows. supports SSL websites. Page 27 of 60 . such as internal and external links. provides multiple-level undo/redo for all operations. finds broken links or paths and links with syntactic errors. HTTPS. Alert Linkrunner Link check tool from Viable Software Alternatives. updates links to moved and renamed files. partial testing of ftp and gopher sites. familiar interface for managing files. domain names. evaluation version available. ChangeAgent Link checking and repair tool from Expandable Language.Aditi Technologies Basic Concepts of Software Testing 5. For Windows platforms. SiteAnalysis Hosted service from Webmetrics. for Windows. or as infrequent as once a week. For Windows. Identifies orphan files and broken links when browsing files.

CyberSpyder Link Test Shareware link checker by Aman Software. allowing for customizable queries and reports.0 validator programs for AWK or PERL by H. organize and Page 28 of 60 . Windows. For Windows. ability to check password-protected areas. reports URLs changed since last checked. results can be exported to database. Distributed under Gnu General Public License. binary avaialable for Linux. Handles one URL at a time. MOMspider Multi-Owner Maintenance Spider. originally by Dave Raggett. For all UNIX flavors. but old versions still available as freeware. Web Functional and Regression Testing Tools IeUnit IeUnit is an open-source simple framework to test logical behaviors of web pages. Big Brother Freeware command-line link checker for Unix. Includes cross referenced and hyperlinked output reports. LinkLint Open source Perl program checks local/remote HTML links. For Windows. Windows. LinkScan Electronic Software Publishing Co. Churchyard. Includes capabilities for central management of large multiple intranet/internet sites. 'what's new' reporting. reports of orphan files and files with mismatching case. site has much documentation and related info.5. HTMLchek for awk or perl Old but still useful HTML 2. HTML TIDY Free utility available from SourceForget. relocations. 5. For automatic fixing of HTML errors. test resumption at interruption point. Available as source code or binaries. released under IBM's Common Public License. timeouts.net. Utilizes the HTTP 'HEAD' request instead of the 'GET' request so that it does not require retreival of the entire html page. This site contains an interesting discussion on the use of META tags. Available as source code. ID/Password entries.0 or 3. randomized order checking. can check for bad links due to specified problems such as server-not-found. doc-not-found.'s link checker/site mapping tool. Has not been updated in recent years. PERL script for a web spider for web site maintenance. capabilities include automated retesting of problem links.Aditi Technologies Basic Concepts of Software Testing changed-link checking. Discontinued. by Francois Pottier. via the WWW Consortium (the folks who set web standards). for UNIX and PERL. unauthorized-access. Not updated in recent years. page size analysis. Mac. Validates hyperlinks for all major protocols. support of proxy servers for remote URL checking.4. capabilities include specified URL exclusions. Not updated in recent years. It helps users to create. 5. configurable. link checker. Results stored in database. HTML syntax error checking. formatting disorganized editing. support for all standard server-side image maps. Perl Testing Tools W3C Link Checker Link checker PERL source code. and finding problem HTML areas.

KUMO Editor Toolset from Softmorning LTD for creation and editing of web macros and automated web tests. built-in exception handling and reporting facility. retry on error. cookies. Enables creation of scenarios from spreadsheets. files. Supports keyword-driven testing. Page 29 of 60 . Records browser interaction by element instead of screen coordinates. etc. and Test Writing Assistant . supports ASP. SSL. servlets.Web browser plug-in module to assist the test writing process. and monitoring products from AppPefect Corp. JStudio SiteWalker Test tool from Jarsch Software Studio allows capture/replay recording. robust handling of page modifications. eastern and western European languages. provision to add GUI. JSP. Multilingual for Asian. HTML. Includes Script Editor. but with extended API). Includes a test runner with GUI interface. Supports multiple OS's and browsers. Supports handling dynamic content created by JavaScript. autocomplete. Test Complete Enterprise Automated test tool from AutomatedQA Corp. Page objects navigator allows browsing of hierarchy of web objects in a page. QEngine Test tool from AdventNet enables functional testing of Web sites and Web-based applications. actiWate Java-based Web application testing environment from Actimind Inc. page titles and HTML element properties. integrates with a variety of IDE's. Scripting uses Jython. data-driven Test wizard to fetch script data from external source.Aditi Technologies Basic Concepts of Software Testing execute functional unit tests. Macro recorder transforms any click to a C# directive. includes web functional testing capabilities. Works with a variety of browsers and OS's. automatic recording of any Web browser events and translates into an Python editable scripts. Works with Internet Explorer. AppPerfect DevSuite Suite of testing.EXE files to enable running web macros on demand and integration into other software frameworks. Includes syntax-coloring editor with intellisense. fail definitions can be specified for each step of the automated workflow via JavaScript. Can be used to unit test any individual component with an HTTP interface (JSP. HtmlUnit etc. Shareware for Windows/MSIE. PHP. Free and professional versions available. JavaScript's Document Object Model enables full access to all document elements. Application Map Editor to view and edit the map object properties. QEngine Web Test Studio Web functional test tool from AdventNet. Freeware. Secure recording on password fields.DLL and . Record and playback capability. HTML forms.) or it can be used to create a suite of HTTP level functional or regression tests. Implemented in JavaScript for the Windows XP platform with Internet Explorer. records using page elements controls symbolically rather than with raw screen coordinate. that includes a web functional testing module. Can export created . HTML-based test result reports can be generated. Database and File checkpoints and verify database tables. and loop. ASP. CGI. Advanced framework for writing test scripts in Java (similar to open-source frameworks like HttpUnit. tuning. run-time debugging features. WebInject Open source tool in PERL for automated testing of web applications and services. Test data from any database or Excel spreadsheet can be mapped to enter values automatically into HTML form controls. For Windows and MSIE.

Utilizes a distributed testing model and consists of three parts: STC Administrator. SimpleTest Open source unit testing framework which aims to be a complete PHP developer test solution. Can group tests into a testsuite that again can be part of a bigger testsuite. Free including source code. Page 30 of 60 . etc. such as bitmap or text matching. Flash. Includes an Intelligent. and Runner executes the test case document. C++ or C++/MFC. Handles Applets. and a Data-Driven. until recording capabilities are added. XMLbased test script code is editable with user's preferred XML editor. Includes all of the typical functions that would be expected from JUnit and the PHPUnit ports. STC Master and STC Master Service. Has an XMLbased scripting capability to enable modifying captured scripts or creating new scripts. Adaptable and MultiThreaded Playback Engine. validation of table contents. Controls are recorded as individual objects independent of screen positions or resolution. with its HTML objects recognition. Active-X controls. HTML/DOMAware and Object Mode Recording Engine. Enables 'Persistent Acceptance Testing' that activates tests each time a web application is used. automates repetitive tasks for Web site testing (and standard Windows applications). Provides a high-level API for navigating a web application combined with a set of assertions to verify the application's correctness including navigation via links. can be inserted during a recording. has some JWebUnit functionality as well. and can be adapted to any reporting style or requirements. 90-day evaluation copy available. cookie testing and form submission. Includes capability to expand scope of macros by editing and adding loops. playback window/size can be different than in capture. jWebUnit Open source Java framework that facilitates creation of acceptance tests for web applications. The simple navigation methods and ready-to-use assertions allow for more rapid test creation than using only JUnit and HttpUnit. test case documents are always synchronized with the application. Utilizes HttpUnit behind the scenes. Canoo WebTest Free Java Open Source tool for automatic functional testing of web applications. Key features besides capture/playback include automatically detecting and capturing standard and custom content errors. (300+ commands). form entry and submission. but all recorded items are validated and logged 'on the fly'. Reports information needed to troubleshoot problems. Includes a WinTask Scheduler. Special validation points. Test results are reported in either plain text or XML format for later presentation via XSLT. WinTask Macro recorder from TaskWare. animated bitmaps. This includes web page navigation.Aditi Technologies Basic Concepts of Software Testing Site Test Center Functional and performance test tool from Alliance Software Engineering. TestSmith Functional/Regression test tool from Quality Forge. TestCaseMaker/Runner Test case document driven functional test tool for web applications from Agile Web Development. Fuzzy matching capabilities. TestAgent Capture/playback tool for user acceptance testing from Strenuus. Maker creates test case documents. scripts have to be developed manually. ensure robustness of scripts with Synchronization commands. Editable scripts can be recorded in SmithSript language or in Java. LLC. and other typical business web application features. but also adds mock objects. etc. Standard reporting XSLT stylesheets included. branching statements.

Utilizes Collections. Free for most uses. Easy development and maintenance . Supports HTTPS/SSL. and generates a detailed test report. PAMIE Free open-source 'Python Automated Module For Internet Explorer' Allows control of an instance of MSIE and access to it's methods though OLE automation . Allows creation of completely new test scenarios without ever having performed that test before. Includes a scenario debugger including single step. and parsing of response codes or parsing page content for expected or unexpected strings. tab/comma delimited file formats. Perl dbi database access. recorded sessions can be saved as XML and reopened later.GUI Unique test automation tool from Omsphere LLC. screen names. and more. Written in ActivePerl. Includes capability for automated scripting. Records HTTP messages by acting as a Web proxy. For Windows.Aditi Technologies Basic Concepts of Software Testing MITS. Plugins can define test types and provide additional report capabilities. allowing inheritance of all Perl functionality including regular expressions. data driven scenarios. and assertions to responses in order to validate a scenario during its playback. Advanced object binding reduces script changes when Web-based apps change. Next-generation scripting language eliminates problems created by syntax or other language errors. For Windows platforms. Allows the attachment of extraction or replacement rules to any HTTP message content. Create scenarios using the point and click interface. Badboy Tool from Bradley Software to aid in building and testing dynamic web based applications. extensible IDE).no need to keep track of GUI maps for each window. It can test thousands of test scenarios without use of any scripts. Perl module that allows a user to automate use of IE via Perl scripts. has an intelligent state machine engine that makes real-time decisions for navigating through the GUI portion of an application. many Perl cpan library functions. break points and response introspection. Combines sophisticated capture/replay ability with performance testing and regression features.. Methods. Testers enter test data into a spreadsheet used to populate objects that appear for the particular test scenario defined. HTTP requests and responses are fully displayed in order to inspect and customize their content. all without changing tool. broken links and the structure of the crawled web. allowing creation of more scripts in less time. reporting various metrics. Includes a Task API for building custom test tasks. PureTest Free tool from Minq Software AB. Free Demo and Test Script available. Events and Properties exposed by the DHTML Object Model. HTTP-WebTest A Perl module which runs tests on remote URLs or local Web files containing Perl/JSP/HTML/JavaScript/etc. etc). includes an HTTP Recorder and Web Crawler. This module comes with a set of default Page 31 of 60 .'Simple Automated Module For Internet Explorer'. or logic associated with the engine. Solex Web application testing tool built as a plug-in for the Eclipse IDE (an open. testware architecture (object names. The Web Crawler is useful for verifying consistency of a static web structure. Multi-platform written in Java. Uses IE's built in COM object which provides a reference to the DOM for each browser window or frame. source code avalable. dynamic Web applications. Supports unlimited set of ODBC-compatible data sources as well as MS Excel. QA Wizard Automated functional web test tool from Seapine Software. SAMIE Free tool designed for QA engineers . This module can be used "as-is" or its functionality can be extended using plugins.

Records any combination of browsing. databases. and web-based applications. Can extract web data and save as CSV file or process the data via a script. or XML. thus automatically supports advanced features such as SSL. written in Java. Utilizes multiple validation methods. Maps and tests all possible paths through a dynamic site. For Windows and MSIE. Can handle data input from text files. and JUnit is used as the testing library. Works by remote controlling the browser. basic http authentication. component location and mouse movements for accurate replay. Works as a proxy server. TestWeb Test tool from Original Software Group Ltd. utilizes a new approach to recording/playback of web browser scripts. can enforce over 200 HTML.net/Open Source Development Network. but can be easily extended with third-party plugins. Rational Functional Tester Page 32 of 60 . assists user during the recording with visual feedback. The 'TestPartner Visual Navigator' can create visual-based tests. WML and XHTML coding standards or customized standards. from SourceForge. form filling. IntelliScripting logic removes the reliance on specific browser window sizes. dynamic testing. JavaScript. cookies and automatic page redirection. and webserver load and capacity analysis. including form submission. an XML DOM. for easier script maintenance. Linux.org. Can run load tests based on the tool's analysis of web server log files. includes an HTTP proxy recorder to automate test script generation. load. content validation. Allows creation of rules for automatic monitoring of dynamic page content. Ideally suited for automated unit testing of web sites when combined with a Java unit test framework such as JUnit. A command line interface allows for easy integration with other test software. WebKing Web site functional. HttpUnit Open source Java program for accessing web sites without a browser. or MS VBA can be used for customized scripting. tables. designed and implemented by Russell Gold. Inc that uses a 'Test Enabled Web Browser' test engine that provides browser-based client side quality checking. and a mechanism for playing tests back from the GUI and command line. and links. For Windows. Power users can manually edit a recorded macro. Jython is used as the scripting language. and allows Java test code to examine returned pages as text. eValid Web test tool from Software Research. Open-source project maintained by Ilya Martynov. Playback can run in background while other tasks are performed on the same machine. page performance tuning. HTTP-Redirects and cookies. or containers of forms. and static analysis test suite from ParaSoft. supports hyperlinks targeted at new instances of browser.Aditi Technologies Basic Concepts of Software Testing plugins. Solaris. clicking. CSS. iOpus Internet Macros Macro recorder utility from iOpus Inc. Java. MaxQ Free open-source web functional testing tool from Tigris. Includes ServletUnit to test servlets without a servlet container. Emulates the relevant portions of browser behavior. script testing and information gathering. 508 compliance. It analyses the underlying intentions of the script and executes it by direct communication with web page elements. Compuware TestPartner Automated software testing tool from Compuware designed specifically to validate Windows. automates repetitious aspects of web site testing.

6. Assessment module can be used by auditors and compliance officers to conduct comprehensive audits. and debugging scripts.NET and J2EE applications. Java or traditional client/server-based applications. proxies. Winrunner Functional/regression test tool from Mercury. includes support for testing Web. HTML. 'Data Bank Wizard' simplifies creation of data-driven tests. includes support for testing Web. and web-based applications. Java. hanging. includes site monitoring and load testing capabilities.NET. test plans. Java. For Windows and Linux. test process management capabilities. scripting language. C++. browsers. 5. SilkTest Functional test tool from Segue for Web. . etc. and IDE for developing. PDAs and cell phones. editing. recovery system for unattended testing. and record/playback. Enables data-driven testing. compiling. QuickTest Pro Functional/regression test tool from Mercury. Web Site Security Test Tools Watchfire's AppScan Tool suite from Watchfire automates web application security testing. etc. direct database access and validation. choice of scripting languages and editors. test planning and management. etc. and offers recommendations for fixing detected security flaws. etc. produces defect analyses. ERP. Includes full VBA script development environment and options such as javascript. DHTML. in order to induce component crashes. ERP. Codenomicon HTTP Test Tool Tool for discovering and eradicating security-related flaws in HTTP implementations via robustness testing. and denial-ofservice situations which may affect component/application security. and other applications. Java beans. and to validate compliance with security requirements. and more. Evaluation version available. running. Tool systematically generates very large numbers of protocol messages containing exceptional elements simulating malicious attacks.) SecurityMetrics Appliance Page 33 of 60 . Compuware's QARun QARun for functional/regression testing of web. (HTTP implementations may be utilized in web servers. protocol analyzers. network appliances. Features include: test creation and customization. XML. e-Test Suite Integrated functional/regression test tool from Empirix for web applications and services and . DOMbased testing and validation.Aditi Technologies Basic Concepts of Software Testing IBM's (formerly Rational's) automated tool for testing of Java. Handles ActiveX. Java.

Scheduled vulnerability scanning. Includes instructions to help immediately remedy security problems. CVE standards support. MacAnalysis database is updated regularly with a "Vulnerability Update" function to update database of more than 100 new holes per month. MacAnalysis Low cost Network Security audit tool from Lagoon Software. updated bi-monthly. Provides detailed reports and vulnerability fix recommendations. Inc. including SANS/FBI Top 20 Vulnerabilities Scan. real-time IDS event analysis. Freeware. firewalls. plug-in facility for third party apps. and browser checkup tool. remediation management of vulnerabilities. Runs on Macintosh. can audit UNIX. data theft or system destruction. Operates as a Layer 2 Bridge no network configuration needed. Also available is 'Tiger Analytical Research Assistant' (TARA). an upgrade to the TAMU 'tiger' program . Automatically downloads latest IDS attack signatures. Lightning Console Security management tool from Tenable Network Security for multiple security and network administrators across multiple organizations. inside and outside the firewall. Supports the FBI/SANS Top 20 Consensus. SARA 'Security Auditor's Research Assistant' Unix-based security analysis tool from Advanced Research Corp. vulnerability assessment scripts and program enhancements nightly. SANS/ISTS certified. Qualys Guard Online service that does remote network security assessments. remote self scan and API facilities. asset management. Qualys Free Security Scans Several free security scan services from Qualys. websites.a set of scripts that scan a Unix system for security problems. network security scan. Can automatically schedule vulnerability assessment of designated IP addresses during low traffic times. provides proactive 'Managed Vulnerability Assessment'. routers. Win servers STAT Scanner Page 34 of 60 . and more for security vulnerabilities which may lead to interrupted service. based on the SATAN model. Mac. Perimeter Check SecurityMetrics 'Perimeter Check' service analyzes external network devices like servers. network topology discovery and organizational executive reporting for hundreds of administrators via easy-to-use web interface.Aditi Technologies Basic Concepts of Software Testing Integrated software and hardware device includes Intrusion Detection and Prevention Systems and Vulnerability Assessment.

Customizable reporting. reports specify severity levels of problems. adapting penetration strategy based on previous results. Can scan a variety of operating systems. Capabilities include: scan and analyze an entire network domain and/or a single machine. open-source remote network security auditing tool. customizable security reports for management and technical personnel. Learns about network as it scans. Runs on many UNIX flavors. web server requests-per-second benchmark. and Nessus Attack Scripting Language. by Renaud Deraison. Utilizes a root-cause and path-analysis engine to illustrate exact sequence of steps taken to uncover vulnerabilities. based on 'never trust the version number' and 'never trust that a given service is listening on the good port'. whereas the client is a frontend designed to collect the results.Security testing tool from SAINT Corporation. product runs on Windows. Scans systems and services on the network and safely simulates common intrusion or attack scenarios. enabling identification of exactly where to correct vulnerabilities to enforce corporate security policies. CVE compatible. NetRecon Network vulnerability assessment scanner from Symantec. Updated regularly. An updated and enhanced version of the SATAN network security testing tool. denial-of-service checks. NMap Network Mapper Page 35 of 60 . select or ignore specific vulnerabilities via configuration files. and a wide variety of other tests. Nessus is made up of two parts : a server and a client. correct vulnerabilities across the network with 'AutoFix' function. Port scans. and SAINTbox scanner appliance. track vulnerability trends via analyses comparing current and previous assessments. retest corrected vulnerabilities immediately. Works with a variety of operating systems. Secure-Me Automated security test scanning service from Broadbandreports. analysis reports of vulnerabilities with detailed information relating to the name/description/risk level of each vulnerability. Also available is 'WebSAINT' self-guided scanning service. eliminate vulnerabilities using recommended solutions with links to related websites and knowledge-base articles.Aditi Technologies Basic Concepts of Software Testing Tool from Harris Corp. the server (nessusd) manages the 'attacks'. Includes more than 1000 tests in 23 vulnerability categories. Utilizes comprehensive updatabale vulnerability databases to automatically detect vulnerabilities. Single machine or full network scans. Nessus Security Scanner Free. Limited free or full licensed versions available. SAINT Security Administrator's Integrated Network Tool . 45 common web server vulnerability checks. for security analysis of Windows/UNIX/Linux and other resources. Includes DoS testing.com for individual machines.

etc. NetIQ Security Analyzer Multi-platform vulnerability scanning and assessment product. to uncover and report systems vulnerabilities that might be open to attack. DMZ and externally visible assets. what services (ports) they are offering. Automatic update service allows updating with latest security tests. software product. Foundstone Vulnerability management software tools from McAfee/Network Associates can provide comprehensive enterprise vulnerability assessments. Performs scheduled and selective probes of network services. scanners. Page 36 of 60 . intrusion detection resources. Runs on most flavors of UNIX as well as Windows. bv-Control for Internet Security Secutiry scanning tool from BindView. what type of packet filters/firewalls are in use. designed to rapidly scan large networks or single hosts. servers. intrusion detection tools. hubs. Remotely evaluates security of network perimeter. firewalls. routers. and any other devices with an IP address on a network. or managed service. etc. operating systems. printers. Includes a Software Developer's Kit to allow custom security test additions. prepares remediation advice. remediation information. includes capabilities for checking and remediation of security problems of servers. publications. electronic law. etc. CERIAS Security Archive Purdue University's 'Center for Education and Research in Information Assurance and Security' site. Provides flexible risk management reports. 'hotlist' section includes extensive collection of links. CyberCop ASaP Managed vulnerability assessment service from McAfee/Network Associates. Uses raw IP packets in novel ways to determine what hosts are available on the network. Also includes an FTP site with a large collection of (mostly older) security-related utilities. organized by subject. what operating system (and OS version) they are running. Available as a hardware appliance. workstations.Aditi Technologies Basic Concepts of Software Testing Free open source utility for network exploration or security auditing. desktops. provides automated vulnerability assessment for measuring online security risks. Scans any device on network regardless of platform. For Windows/Solaris/Linux. to hundreds of security information resources and tools. etc. InternetScanner Tool from Internet Security Systems. Systems are analyzed on demand or at scheduled intervals. and many other characteristics. routers.

Maxamine Knowledge Platform Integrated site managment tool from Maxamine Inc. for maintaining. Accessibility. (Includes various types of security tools. combined utilities/tools. Capabilities include: high-level summary reports. actively monitors and evaluates the health of a site's live applications utilizing synthetic business transactions. Free and professional versions available.Aditi Technologies Basic Concepts of Software Testing trend analyses and comprehensive data sets to support policy enforcement. and authoring/publishing/deployment tools that include significant site management or testing capabilities. to enable detailed analysis of what is going on behind the scenes of Web operations. Limitedcapability free version also available. Includes link checker. Recorded transactions can then be analyzed when production errors occur and fed back into SilkPerformer for real-world performance testing scenarios. and trend analyses using intuitive browser-based reports. Integrates performance Page 37 of 60 . and site monitoring capabilities.7. database servers. ManageEngine Applications Manager Site management tool from AdventNet. combining site content. webSensor Integrated site measurement and analysis tool from Moniforce BV. not just for testing.) 5. Web Site Management Tools (This section includes products that contain: site version control tools. analyzing web sites. WebCEO Tool from Radiocom Ltd. traffic analysis. server management and optimization tools. site scores. Information Assurance. FPT/publishing. and online service available. for monitoring compliance standards related to Privacy. Passive monitoring records real user sessions and reliably detects and logs all error situations encountered by real-users. Inc. Desktop product for multiple platforms. WYSIWYG editor. and Web Governance. promoting.roll-up of results from different business units or global Web sites for overall views of site compliance. Similar managed service also available.) COAST WebCentral Site management tool from Coast Software.org. Inc.. SilkCentral Performance Monitor Tool from Segue Software.. works with a variety of web servers. Top 75 Security Tools Listing of 'top 75' network security tools from survey by Insecure. service types. structure and visitor traffic data into one searchable data base. and OS's. complete inventory of Web sites and Comparison Inventory Reports to compare different versions of a Web site.

Other capabilities include search engine optimization. etc.Aditi Technologies Basic Concepts of Software Testing and availability data with site traffic data. ContentStudio E-catalog management tool from TechniCon Systems with Win Explorer-type interface with drag and drop functionality. Web Site Monitoring Performance Monitoring . Legacy-to-Web Tools can "bulk-load" online catalog from legacy Page 38 of 60 . includes modules to check HTTP and ICMP/ping for checking availability of HTTP and HTTPS sites. and more. Cuevision Network Monitor Monitoring tool from Cuevision for monitoring website. and repair. response errors. for performance and availability monitoring. GFI Network Server Monitor Server management tool from GFI Software Ltd. Captures and monitors real user sessions. For Windows. network errors. automatically reboot servers. sends email notification if site is slow or down. Alerts via email. server. RealiTea Web application management solution that provides detailed visibility into availability and functionality issues to enable efficient problem identification. customizable views of success/failure rates for key online business processes and other critical metrics. and diagnostics. and popup. restart services. pagers. bandwidth usage. Site Manager to create and manage your web site. RedDot CMS Web content managment system from RedDot Solutions includes modules such as SmartEdit. suited to large systems. real-time executive dash board. Add-on capabilities include a 'Dashboard' to provide real-time. and network. checks network and servers for failures and fixes them automatically. visitor segmentation. services. Asset Manager to securely centralize images. applications. and 'Real Scripts' automatically generated from recorded user sessions for use in specified other load testing tools. SMS. Freeware version of GFI Network Server Monitor is also available. etc. network management. restart apps and services. run scripts. page aborts. click path analysis. page sizes.Free open-source website performance monitoring and uptime notification application in PERL. including server errors. capabilities include notifications via email. providing context and correlation data for application failure analysis. net send. PROGNOSIS Comprehensive tool from Integrated Research Ltd. from AllScoop. response times. isolation. Web Compliance Manager to manage integrity and accessibility. eliminates need for programmers and special production staff to maintain catalogs.

Statistical Reports. which hosts the management system application and the client's administrative interfaces and pushes the final assembled pages to client Web servers. PHP-Code Insertion. scheduler. Dynamic Menu. User Administration. Database Backup. verifying content and functional integrity. FTP Assistant for Webmasters FTP client from Byteway. can manage all components across complex Web-enabled applications by measuring and monitoring performance.com for web site updating. site cleanup. comprehensive Communications Gateway for inbound and outbound data. Installer easily installs system on your server. CrownPeak CMS Content management service from CrownPeak Technology. etc. Provides complete software developers environment. TestWEB Monitor Web site monitoring tool from Original Software checks 5 or more URL's up to every 30 seconds. Works on multiple platforms. Free for use on small sites. Constructioner Light Edition available as Freeware. such as option compatibilities on a single product. Tonic Suite of web site management/monitoring/test tools from Tonic Software. detecting. high-level functions save template coding time.Aditi Technologies Basic Concepts of Software Testing product data. reporting and fixing problems before users see them. All can be integrated without writing code. Design/administrate database connected PHP web applications in combination with individual webdesign. Capabilities include defining intra-configuration rules. Trellian InternetStudio Page 39 of 60 . and a robust API. HTML and GIF optimization. upload resuming. and more. SpinPike Flexible and scalable content management system from SavvyBox Systems. WebLight HTML validator and link checking tool from Illumit LLC. link verification. drag-and-drop. based on database-driven. Multilingualism. load testing. Receive instant notification by email whenever one of the sites is unavailable. spatial relationships between products. Content and Table Management (WYSIWYG-Editor). template-based dynamically-created content. Search. low cost for large sites. Constructioner Website development software with integrated content management system from Artware Multimedia GmbH. additional features include manual or customizable automated control. WYSIWYG editor. Includes: Ready-to-use Backoffice. Message Board.

V. e-commerce site designer/generator. XML-content-based multichannel delivery. authoring tools. simply create an account and paste one line of javascript into your pages.capabilites/support include scalability. version control. payment processing. scalable content management platform for content contributors distributed across an organization. enables users to update their web site via e-mail. browser-based. etc. Serena Collage Content management tool from Serena. administration. site mapper. Use any e-mail client capable of producing HTML to format your content or use HTML tags in a plain text e-mail. styles. product info. customer relationship management. image mapper. Documentum Enterprise content management product from EMC Corp. integration with a variety of servers. accessibility. privacy compliance. HTML validator. renaming the files to match the convention.Aditi Technologies Basic Concepts of Software Testing Suite of web site management utilities from Trellian including site upload/publishing tools. and website standards compliance. Works with content from any platform or application. link checker. security. and more. prices. text editor. multi-lingual support. FlexWindow Tool from Digital Architects B. invalid chars). Update news flashes. Easy to install. Can automatically run external programs. length. etc. approval workflow. e-mail. support for more than 50 document formats. web servers. HTML encryptor/optimizer. searchability. and more. and DB servers. spell checker. Basic accounts are free. Alerts by cell phone. notifications. pager. Watchfire WebXM Content analysis tool from Watchfire analyzes sites for slow loading pages. Alchemy Eye System management tool from Alchemy Lab continuously monitors server availability and performance. Web500 CMS Web content management and site maintenance solution from Web500. security.. image handling. Runs with a variety of platforms. activity tracking. . business process automation. HTML editor. For Windows. HTML Rename Site Migration/Batch processing tool from Expandable Language that enforces file naming conventions (case. Enables collaboration. more. templates. advertisements. then Page 40 of 60 . broken links. and log events. site spider. globalization. Add-on modules allow capabilities such as WAP. stories. e-commerce.

portals and scanned images. version control. management of workflow. Enables services such as library services. Enterprise web content management system that enables quickly and efficiently building. integrates with other OpenView products to provide a variety of management and monitoring services and capabilities. index/search. reuse and access. managing and publishing Web content and supporting from one to thousands of Web sites. It can constantly monitor any number of servers and ports. securing and sharing digital and paper-based documents and reports. based on native XML and XSL technologies. Eliminates problems encountered when moving files between Windows. and maintaining highly dynamic web sites. and customizable workflow. Manages Web content. production. services for capturing. Enables end-user emulation of major business-critical applications as well as a single integrated view of the complete Internet infrastructure. personalization. staging. and services for collaborative environments and for digital asset and records management. Content Management Server Windows based content mgmt tool from Microsoft (formerly 'nResolution' from nCompass Labs). anticipate capacity shortfalls. including creation. documents. workflow. IP Check Server monitoring tool from Paessler GmbH. CGI scripts return errors or the SQL server is not reachable. and archiving. Inc. conversion services. Designed to help IT staff efficiently predict. Stellent Universal Content Management System Content management tool from Stellent. where all content and content types are stored for management. Other modules provide additional services such as: services for creating. allows a distributed team of non-technical content experts to manage every aspect of site content. editing. and indexing content by means of a browser window or via MS Word. and manage and report on service level agreements. isolate. security. replication and administration. Broadvision Suite of content and publishing management tools from Broadvision Inc. Alerts webmasters if a webserver is not working correctly because the web service is down. 'Stellent Content Server' uses a web-based repository. For Windows..Aditi Technologies Basic Concepts of Software Testing correcting the links to those files automatically. and UNIX systems and publishing to CD-ROM. Page 41 of 60 . Enables scheduling of content refreshes. content development. HP OpenView Internet Services Internet services monitoring/management tool from HP. deploying. digital assets. Mac. For Mac or Windows. tracking of revisions. diagnose and troubleshoot problem occurrences. publishing. Rhythmyx Content Manager Web content management product from Percussion Software.

SSI. full-featured editor from Chami. Also has keyword count for pages vs competitor's pages. server-side and live preview modes. submit URL's. Enhanced logging utilizing custom generated script uploaded to host. includes 'anti-caching technology' ensuring more reliable logging by blocking caching of site's pages by visitor's browsers. downloading and online editing of files. GUI support of W3C's HTML Tidy. Requires MSIE. Perl. XHTML. WebSite Director Page 42 of 60 . WebLog Manager In addition to logging capabilities. Pro plugins available to paid registered users. upload. Ruby. CSS. validate. 'Active Protect MetaTags' allowing only search engine spiders to see front-page meta-tags. XHTML and XML authors to edit. From Monocle Solutions. Java. and 'Active Deny Access' to prevent particular IP addresses from accessing web site. JavaScript.Aditi Technologies Basic Concepts of Software Testing HTML-Kit Free. improve search engine ranks. support more than 50 host file systems. manage. 'Active Protect Images' to prevent other sites from "borrowing" images. Over 400 free plugins available for extending and customizing HTML-Kit. Finds errors and provides suggestions on how to create standards compliant pages. AgentWebRanking Freeware tool from AADSoft to monitor site's search engine position. Capabilities include notification by email and automatic reboot of web server. PERL. click paths. checks search engine placement including position. Requires MSIE and Windows. provides notification by displaying a message or sending an e-mail. meta tag creator. for Windows. external. rank on page. on web server and Windows client. For Windows. FTP Workspace for uploading. and much more. XSL. runs on both Lotus Domino and IBM WebSphere. WebCheck Windows application that runs in background and periodically checks a site for availability and correctness. dragand-drop files. searches for keywords. XML. Python. A1Monitor Utility from A1Tech for monitoring availability of web servers. can specify search depth. IBM Workplace Web Content Management IBM's web content management product for Internet. Uses a highly customizable and extensible integrated development environment while maintaining full control over multiple file types including HTML. automatically resume interrupted transfers. preview and publish web pages. Requires UNIX. auto or manual submit of URL's to search engines. lookup help. page. and the ability to use hundreds of optional free add-ins through its open plugins interface. WS_FTP Pro FTP/web publishing tool from Ipswitch. extranet and portal sites.com designed to help HTML. intranet. seamless integration with the CSE HTML Validator. format. and update websites. Includes internal. Validate XML documents using its DTD and/or check for well-formedness. Searches top engines for keywords.

Includes a 'Database Import Wizard'. E-mail/pager notifications. metrics. WebTrends Analysis Suite Web site management tool from NetIQ includes log analysis. HTML validation. and can automatically convert output to CALS or HTML table models or to XML. For Windows. scalable web content management solution that enables collaboration. Unicenter TNG w/Web Management Option Site management application from Computer Associates includes access and security control. allows defining and applying existing workflow and approval rules to web content management process.Aditi Technologies Basic Concepts of Software Testing Web-content workflow management system with browser-based interface includes configurable workflow management. Equalizer Load balancing server appliance and site management tool from Coyote Point Systems. XMetal XML development tool from BlastRadius for XML-based web site authoring and validation. For Windows. alerting. Evaluation copy available. PowerMapper From Electrum Multimedia. standards-based environment. link problems. access control. For Windows. for customizable automated site mapping. link analysis and quality control. real-time server monitoring of server response time. requires Windows and MSIE. and publishing control tool. monitoring. UNIX. proxy server traffic analysis and reporting. Page 43 of 60 . logging. e-mail submission of web content. version control. Site/C 'Set-and-forget' utility from Robomagic Software. link checking. etc. Interwoven Team Site Web development. logging capabilities. works with many servers. Web based interface for load balancing administration. and e-mail notifications. For Windows. content management and site visualization. for periodic server monitoring for web server connection problems. Supports multiple platforms and browsers. server failure detection. server management. Macromedia Web Publishing System Macromedia's complete. OS's. and platforms. and publishing to websites within a centrally controlled. number of pending requests. management. network management. For MS and Netscape web servers. monitoring and recovery.

security. etc. Includes version control. graphics management. OpenDeploy Interwoven's configurable control system for deploying from development to production environments. Meta manager. HomeSite Macromeida's web site project management/authoring/validator tool.. scheduled and on-demand reporting. connections. management. TeamSite Interwoven's collaborative web site production control. and management product for enterprise-wide internet and intranet projects. Date stamper. administration. links. has updateable rulebase. Supports a wide variety of web attributes. image tag checker. link checking. etc. browser interface. and static files. Includes published API for creating custom monitors. publishing. XML repositories. Page 44 of 60 . Validates against various HTML versions. API's. Runs on servers and monitors server performance. Monitors mimic users' end-to-end actions. For Windows or Unix. and servers. For Windows. page in screen is validated with results displayed below. browser extensions. Server side compatible with many available web servers. global search-and-replace. layout editor. From Talicom. Visual site structure editor. Link checker. link checker. Content Management Server Vignette Corporation's product for web site collaborative content. variable lock controls. HTML-to-Text converter. file uploading. For Windows. Includes automated deployment. For Windows. error info is linked to error in html.Aditi Technologies Basic Concepts of Software Testing SiteScope Mercury's product for agentless site monitoring and maintenance. NetObjects Fusion Site authoring/management tool. file edit merging. Microsoft FrontPage Microsoft's web site authoring and site management tool. Support for managing content stored in databases. customizable reports. staging/publishing control. and encryption capabilities. For Windows and Unix. comparison capabilitie. includes site management capabilities. spell checker. thumbnail viewer. provides notifications of problems. and maintenance. databases. logs. HTML PowerTools HTML validator.

Fiddler An HTTP Debugging tool by Eric Lawrence. how it interprets page's robot exclusion rules and more. Extreme Web Analyzer Web site performance analysis tool by Dale Feiste for use in a development or controlled test environment. to which links it extracts. can specify the screen width(s). system resource usage. Consists of 3 components. and compress the result into a very small file to replay. Enter URL and receive back report with screenshots taken in real time on real hardware. SiteTimer service . for automated recording.Find out how long it takes various connection types to get a page. Instruments in memory the target application at runtime so no changes are required to application under test.com Service from HREF Tools to check web page appearance in different Browser/OS combinations. Free. it displays test user's computer screen along with a picture-in-picture window displaying the test participant's face and audio.reports on how a search engine may view a webpage. A Recorder records and synchronizes video and data.Aditi Technologies Basic Concepts of Software Testing 5. upload to a bug tracking system. and system configuration information.8. TestGen Free open-source web test data generation program that allows developers to quickly generate test data for their web-services before publicly or internally releasing the web service for production. The predictable network performance of this environment enables reliable modeling of trace data. from TechSmith Corp. Allows user to save and review relevant information for bug reports. The Manager component includes integrated editing functionality for assembly of important video clips to share with stakeholders. EngineViewer and SiteTimer Free basic services: EngineViewer . Repro Manual testing 'helper' tool that records desktop video. A Remote Viewer enables geographically dispersed observers to watch usability tests from any location. and MS Excel for charting and modeling. Acts as an HTTP Proxy running on port 8888 of local PC. Other Web Testing Tools Morae Usability test tool for web sites and software. check all the graphical links to ensure they're correct. Perl. creating a digital record of system activity and user interaction. where network traffic is captured over a LAN. and share with others. For Windows. examine server's HTTP headers. Any application which accepts an HTTP Proxy can be configured to run through Page 45 of 60 . font magnification(s) and page position(s) for the proofs. web standards and elastic design. analyzing and sharing of usability data. For anyone interested in css. URL2image. from how it breaks down the HTML. more. system operations in 7 different categories. requires a system capable of running Ethereal and/or Sniffer. Remote Viewer observers can set markers and add text notes.

Provides performance enhancing tips. Will send a popup when one of the servers stops responding.) in small text files. includes additional search-enginerelated capabilities. FunnelIT Free utility from ByStorm Software helps streamline test reporting and analysis by gathering Windows platform environment/configuration information (running tasks. loaded modules. FREEping Free ping software utility from Tools4ever which will ping all your Windows-based servers (or any other IP address) in freely-definable intervals. tune-up wizard. component details. and "fiddle" with incoming or outgoing data. Free. system file listings. IP Traffic Test and Measure Network traffic simulation and test tool from Omnicor Corp. Free. Professional. device drivers. environment settings. etc. Full-featured version (not free) has configurable reporting. Sandra 'System ANalyser. and other versions available in multiple languages. can generate TCP/UDP connections using different IP addresses. and allows inspection of the HTTP data.). modules. for Windows. OS version. ODBC sources. Available as an executable intended to leave no footprint. Services information. VisitorVille Site traffic monitoring tool from World Market Watch Inc. that depicts website visitors as animated characters in a virtual village.NET event-based scripting subsystem. and much more. data creation or capture and replay. processor information. html. etc. set breakpoints. Network Adaptor Information. memory details. Installed Software and Installed XP Updates. and includes a simple but powerful JScript. Logs all HTTP traffic between between computer and the Internet.Aditi Technologies Basic Concepts of Software Testing Fiddler. more. etc. Diagnostic and Reporting Assistant' utility from SiSoftware. manage and monitor throughput. ports. services. users can watch their web traffic as if they're watching a movie. Also allows comparisons among results from multiple systems. file system and memory bandwidth benchmarking. Reporting via save/print/fax/email in text. mainboard. and delay. Path Application Manager Page 46 of 60 . XML. drives. loss. Designed to be much simpler than using NetMon or Achilles. Includes CPU. SetSim Pro Northern Webs' free on-the-web meta tag checker. Drive Free Space/Size/Used info. and also allows gathering TCP/IP Address/Network Information. physical memory information. virtual memory settings. processes. Provides large variety of information about a Windows system's hardware and software.

such as "Referrer" and "User Agent". HTTP Interceptor A real-time HTTP protocol analysis and troubleshooting tool from AllHTTP. Dummynet Flexible tool developed by Luigi Rizzo. Can be used on user's workstations. Browser-based Documents (including those without a 'view source' command) to extract precise information about the DOM elements in an HTML source. testing. View all headers and data that travel between your browser and the server. and troubleshooting with code coverage capabilities. Macintosh. SpySmith can also spy on Windows objects. PocketSOAP Packet-capture tool by Simon Fell. binaries and source available. Page 47 of 60 .com. and multipath effects. Linux. in most versions of every browser ever released. Co-Advisor's current focus is on HTTP intermediaries such as firewalls. Co-Advisor Tool from The Measurement Factory for testing quality of protocol implementations. CoAdvisor can test for protocol compatibility. For Windows. robustness. Uses pattern recognition technology to peer deep inside customized or COTS applications. freeware. any version. RAMP Section 508 and W3C Accessibility Guidelines tool from Deque Systems that automates analysis and remediation of non-compliant web functionality. For Windows. and XML switches. available as on-line service. or source code. Interceptor also allows changing of select request headers on-the-fly. Can log captures to disk. delays. It allows the user to peek inside I. any operating system. Browser Cam Service for web developers and testers that creates screen captures of web pages loaded in any browser. Enables enhancement of QA. captures and displays packet data between local client and specified web server. filters. with GUI. Other info: runs on FreeBSD packages. Linux RPMs.Aditi Technologies Basic Concepts of Software Testing Application Monitoring and management tool from Winmoore. compliance. analogous to running an MRI scan. binaries. originally designed for testing networking protocols. Inc.E. packet losses. Free 90-day trial. SpySmith Simple but powerful diagnostic tool from Quality Forge. Allows viewing of web page appearance on Windows. especially useful when testing web sites and web-based applications. caching proxies. Also available is PocketXML-RPC and PocketHTTP. security. Split-screen display and dual logs for request and response data. or on FreeBSD machines acting as routers or bridges. Windows (on-demand). and other quality factors. can be used in testing to simulate queue and bandwidth limitations.

HiSoftware AccVerify Tool for testing site Accessibility & Usability. can also programmatically fix most common errors found. any message definitions. Searchability. and any network. throughput. etc.) Tool can learn what request/response scenarios are being tested for future tests and can work with any protocol. etherpeek. Inc. Custom Checks and Test Suites to meet organization's standards. MITs. contains over 2000 test files and an associated test report (also in XML). client/server. For automated testing of Web. round trip times. Freeware. mailboxes. Runs on Windows platforms. and mainframe applications.GUI XML Conformance Test Suite XML conformance test suites from W3C and NIST.comm Tool from Omsphere LLC for simulating virtually any software interface (internal or external). for Windows. Allows developers to test down to the unit level by simulating the internal software interfaces (message queues. Can crawl a site and report errors. Overall Site Quality. etc). allows monitoring web sites for changes or misuse of your intellectual property in metadata or in the presented document. freeware. retransmissions. and as source code. HP Net Metrix. Certify Test automation management tool from WorkSoft. and various graphs. Privacy and Intellectual Property policy verification. can be used by setting browser to use it as a proxy server and then can monitor all traffic to and from browser. HiSoftware Web Site Monitor Tool allows user to monitor your server and send alerts. Page 48 of 60 . Ethernet. Available for various UNIX flavors. For windows. For managing and developing test cases and scripts. or WinDump. such as those produced by tcpdump. link validation. bytes. Works with all text-based IP protocols. Allows testing without pitfalls associated with live connections to other systems (TCP/IP. snoop. Also available: MITS. tcptrace Tool written by Shawn Ostermann for analysis of TCP dumpfiles. and generating test scripts.Aditi Technologies Basic Concepts of Software Testing TcpTrace Tool by Simon Fell acts as a relay between client and server for monitoring packet data. The test report contains background information on conformance testing for XML as well as test descriptions for each of the test files. ProxyTrace Tool by Simon Fell acts as a proxy server to allow tracing of HTTP data. This is a set of metrics for determining conformance to the listed W3C XML Recommendation. window advertisements. FTP. freeware. Runs on Windows. Can produce various types of output with info on each connection seen such as elapsed time. and segments sent and received.

and developers can then debug their respective code independently of one another no matter if the code units reside on the same servers or on different servers or on any combination thereof. time period. Remote code can pass messages and dialogs directly to your local machine via Team Remote Debugger component. Runs on Solaris or Linux. errors. Govt.0/1. client. allows for entry of custom headers. capability to read in existing database table structures to aid in data generation. For Windows. ActiveX Exe. etc. MRTG Multi Router Traffic Grapher . XHTML. T-SQL. WebMetrics Web usability testing and evaluation tool suite from U. Thread. handles HTTP 0. Datatect Test data generator from Banner Software generates data to a flat file or ODBC-compliant database. VJ. MTS.free tool utilizing SNMP to monitoring traffic loads on network links. For UNIX. Windows. produces a range of configurable reports that breaks down info into network and server speeds. XML. CFML ). Delphi. VB. without ever attaching to process. Hypertrak Suite of software protocol analyzers from Triometric accurately calculates end-to-end download speeds for each transaction. Freeware. Included with software are examples of using the control from within Visual Basic. COM.S. comparison to SLA's. WebBug Debugging tool from Aman Software for monitoring HTTP protocol sends and receives. data generation interface to Segue SilkTest.1. HTML2TXT Conversion utility that converts HTML as rendered in MS Internet Explorer into ASCII text while accurately preserving the layout of the text. URL.9/1. includes capabilities such as scripting support that allows user to write VBScripts that modify data to create XML output. written in any language ( ASP. COM+. For UNIX. generates reports as web pages with GIF graphics on inbound and outbound traffic. not just samples. Page 49 of 60 .Aditi Technologies Basic Concepts of Software Testing Web Optimizer Web page optimizing tool from Visionary Technologies intelligently compresses web pages to accelerate web sites without changing site's appearance. wide variety of data types and capabilities for custom data types. Team Remote Debugger Debugging tool from Spline Technologies allows tracing of any number of code units of any kind ( ASP. Windows. Source code available. DLL. Visual C++. and HTML. CFML ) residing on multiple shared and dedicated servers at the same time. VC++. T-SQL. performance for each server. NIST. CSS. and Javascript and includes GIF and JPEG optimizer techniques. Removes unnecessary information in HTML.

What is 'Software Testing'? 3. What is the 'software life cycle'? 17. 'if the user is in interface A of the application while using hardware B. Organizations vary considerably in how they assign responsibility for QA and testing. What are 5 common problems in the software development process? 12. Also common are project teams that include a mix of testers and developers who work closely together. Will automated testing tools make testing easier? 1.) 2. What is 'good code'? 14. 3. Sometimes they're the combined responsibility of one group or individual.Aditi Technologies Basic Concepts of Software Testing 6. Why is it often hard for management to get serious about quality assurance? 5. How can new Software QA processes be introduced in an existing organization? 7. then D should happen'). and ensuring that problems are found and dealt with. The controlled conditions should include both normal and abnormal conditions. Testing should intentionally attempt to make things go wrong to determine if things happen when they shouldn't or things don't happen when they should.monitoring and improving the process. What are some recent major computer system failures caused software bugs? by Page 50 of 60 . Testing FAQ 1. Why does software have bugs? 6. It is oriented to 'detection'. What is 'good design'?Top of the Document 15. What is verification and Validation? 8. and does C. What's an 'inspection'? 10. What is 'Software Testing'? Testing involves operation of a system or application under controlled conditions and evaluating the results (eg. It will depend on what best fits an organization's size and business structure. What kinds of testing should be considered? 11. (See the Books section for a list of useful books on Software Quality Assurance. What is 'Software Quality Assurance'? Software QA involves the entire software development Process . What is SEI? CMM? CMMI? ISO? Will it help? 16. What are some recent major computer system failures caused by software bugs? 4. making sure that any agreed-upon standards and procedures are followed. What is a 'walkthrough'? 9. with overall QA processes monitored by project managers. It is oriented to 'prevention'. What is software 'quality'? 13. What is 'Software Quality Assurance'? 2.

forced shutdown of 100 power plants. The error was found and corrected after examining millions of lines of code. portions of the project could be salvaged. Software testing was one of the five major problem areas according to a report of the commission reviewing the project. the trains were started by altering the control system's date settings. a software bug was determined to be a major contributor to the 2003 Northeast blackout. News stories in the fall of 2003 stated that a manufacturing company recalled all their transportation products in order to fix a software problem causing instability in certain circumstances. Articles about the incident stated that it took two weeks to fix all the resulting errors. if any. which was unable to correctly handle and report on an unusual confluence of initially localized events. and that the total cost of the incident could exceed $100 million. Reportedly the original contract allowed for only 6 weeks of acceptance testing and the system was never tested for its ability to handle a rate increase. retailer was reportedly hit with a large government fine in October of 2003 due to web site errors that enabled customers to view one anothers' online orders. The bug was reportedly in one utility company's vendor-supplied power monitoring and management system. News reports in September of 2000 told of a software vendor settling a lawsuit with a large mortgage lender. In July 2004 newspapers reported that a new government welfare management system in Canada costing several hundred million dollars was unable to handle a simple benefits rate increase after being put into live operation. and economic losses estimated at $6 billion. according to mid-2004 news reports.S. The company found that many of their newer trains would not run due to their inability to recognize the date '31/12/2000'. A bug in site management software utilized by companies with a significant percentage of worldwide web traffic was reported in May of 2004. According to news reports in April of 2004. In January of 2001 newspapers reported that a major European railroad was hit by the aftereffects of the Y2K bug. news reports revealed the intentional use of a software bug as a counter-espionage tool. This eventually resulted in major industrial disruption in the country that used the stolen flawed software. A major U.Aditi Technologies Basic Concepts of Software Testing • • • • • • • • • • Media reports in January of 2005 detailed severe problems with a $170 million high-profile U. the worst power system failure in North American history.S. In early 2004. in the early 1980's one nation surreptitiously allowed a hostile nation's espionage service to steal a version of sophisticated industrial software that had intentionally-added flaws. The failure involved loss of electrical power to 50 million customers. the vendor had reportedly delivered Page 51 of 60 . The company found and reported the bug itself and initiated the recall procedure in which a software upgrade fixed the problems. government IT systems project. that additional problems resulted when the incident drew a large number of e-mail phishing attacks against the bank's customers. Millions of bank accounts were impacted by errors due to installation of inadequately tested software code in the transaction processing system of a major North American bank. According to the report. Studies were under way to determine which. The bug resulted in performance problems for many of the sites simultaneously and required disabling of the software until the bug was fixed.

It was determined that spacecraft software used certain data in English units that should have been in metric units. rescheduling of engineers. The school district decided to reinstate it's original 25-year old system for at least a year until the bugs were worked out of the new system by the software vendors. The problem went undetected until customers called up with questions about their bills. January 1998 news reports told of software problems at a major U.000 customers. Among other tasks.programmers.Aditi Technologies Basic Concepts of Software Testing • • • • an online mortgage processing system that did not meet specifications.S. like anyone else. was delivered late.S. etc. futures exchange. Among those affected was the electronic trading system of the largest U. 5. If there are many minor changes or any major changes. client-server and distributed applications. This is illustrated by an old parable: In ancient China there was a family of healers.the complexity of current software applications can be difficult to comprehend for anyone without experience in modern-day software development. work already completed that may have to be redone or thrown out. Why does software have bugs? • • Miscommunication or no communication .000 erroneous report cards and students left stranded by failed class registration systems. Multi-tiered applications. Why is it often hard for management to get serious about quality assurance? • Solving problems is a high-visibility process. In October of 1999 the $125 million NASA Mars Climate Orbiter spacecraft was believed to be lost in space due to a simple data conversion error. the district's CIO was fired. major problems were reported with a new computer system in a large suburban U. public school district with 100. the orbiter was to serve as a communications relay for the Mars Polar Lander mission. Software complexity . effects on other projects. and sheer size of applications have all contributed to the exponential growth in software/system complexity. and didn't work. Bugs in software supporting a large commercial high-speed data network affected 70.000+ students.redesign. data communications.the enduser may not understand the effects of changes. which failed for unknown reasons in December 1999.000 business customers over a period of 8 days in August of 1999.as to specifics of what an application should or shouldn't do (the application's requirements). telecommunications company that resulted in no charges for long distance calls for a month for 400. known and unknown dependencies among • • Page 52 of 60 . problems included 10. 4. one of whom was known throughout the land and employed as a physician to a great lord. preventing problems is lowvisibility. In early 2000. or may understand and request them anyway . Several investigating panels were convened to determine the process failures that allowed the error to go undetected.S. hardware requirements that may be affected. Programming errors . can make mistakes. enormous relational databases. Changing requirements (whether documented or undocumented) . which was shut down for most of a week as a result of the outages.

etc. maintainable code. In many organizations management provides no incentive for programmers to document their code or write clear. Software development tools . understandable. management must understand the resulting risks. In this case. How can new Software QA processes be introduced in an existing organization? • A lot depends on the size of the organization and the risks involved. resulting in added bugs. it's usually the opposite: they get points mostly for quickly turning out code. feedback to developers. Also see information about 'agile' approaches such as XP. often requiring a lot of guesswork. compilers. also in Part 2 of the FAQ. class libraries. step-at-a-time process.people prefer to say things like: o 'no problem' o 'piece of cake' o 'I can whip that out in a few hours' o 'it should be easy to update that old code' instead of: o 'that adds a lot of complexity and we could end up making a lot of mistakes' o 'we have no idea if we can do that. and testers. When deadlines loom and the crunch comes.scheduling of software projects is difficult at best. egos . and QA and test engineers must adapt and plan for continuous extensive testing to keep the inevitable bugs from running out of control . until I take a close look at it' o 'we can't figure out what that old spaghetti code did in the first place' If there are too many unrealistic 'no problem's'. serious management buy-in is required and a formalized QA process is necessary.see 'What can be done if requirements are changing continuously?' in Part 2 of the FAQ. the result is bugs. For large organizations with high-risk (in terms of lives or property) projects. with a goal of clear. managers. complete. developers. depending on the type of customers and projects. In some fast-changing business environments. mistakes will be made. management and organizational buy-in and QA implementation may be a slower. often introduce their own bugs or are poorly documented. the result is bugs. A lot will depend on team leads or managers. Where the risk is lower. Time pressures .Aditi Technologies Basic Concepts of Software Testing • • • • • • parts of the project are likely to interact and cause problems. For small groups or projects. continuously modified requirements may be a fact of life. testable requirement • • • Page 53 of 60 . 6. Enthusiasm of engineering staff may be affected. QA processes should be balanced with productivity so as to keep bureaucracy from getting out of hand. scripting tools.it's tough to maintain and modify code that is badly written or poorly documented. and there's job security if nobody else can understand it ('if it was hard to write. The most value for effort will often be in (a) requirements management processes. we'll wing it' o 'I can't estimate how long it will take.visual tools. a more ad-hoc process may be appropriate. it should be hard to read'). In fact. Poorly documented code . and the complexity of coordinating changes may result in errors. and ensuring adequate communications among customers.

client and server applications on a network.continuous testing of an application as new functionality is added. to test particular functions or code modules.testing of combined parts of an application to determine if they function together correctly. Typically done by the programmer and not by testers. code. What is verification and validation? • Verification typically involves reviews and meetings to evaluate documents. walkthroughs.based on knowledge of the internal logic of an application's code. Tests are based on coverage of code statements. The 'parts' can be code modules. What is a 'walkthrough'? • A 'walkthrough' is an informal meeting for evaluation or informational purposes. and a recorder to take notes. Unit testing . or that test drivers be developed as needed. Incremental integration testing . 7. requires that various aspects of an application's functionality be independent enough to work separately before all parts of the program are completed. Little or no preparation is usually required. and (c) postmortems/retrospectives. Tests are based on requirements and functionality. • • Page 54 of 60 . most problems will be found during this preparation. (b) design inspections and code inspections. Not always easily done unless the application has a well-designed architecture with tight code. 10. may require developing test driver modules or test harnesses. and inspection meetings. plans. typically with 3-8 people including a moderator. What kinds of testing should be considered? • • • Black box testing . paths. The result of the inspection meeting should be a written report. issues lists. 9. Validation typically involves actual testing and takes place after verifications are completed. branches. The term 'IV & V' refers to Independent Verification and Validation.Aditi Technologies Basic Concepts of Software Testing specifications embodied in requirements or design documentation. What's an 'inspection'? • An inspection is more formalized than a 'walkthrough'. Attendees should prepare for this type of meeting by reading thru the document. as it requires detailed knowledge of the internal program design and code. requirements. done by programmers or by testers. individual applications. Integration testing .not based on any knowledge of internal design or code. or in 'agile'-type environments extensive continuous coordination with end-users. and the purpose is to find problems and see what's missing. White box testing . conditions. not to fix anything. 8. The subject of the inspection is typically a document such as a requirements spec or a test plan. reader. and specifications. This can be done with checklists.the most 'micro' scale of testing.

Also used to describe such tests as system functional testing while under unusually heavy loads.black-box type testing that is based on overall requirements specifications. such as interacting with a database. This type of testing is especially relevant to client/server and distributed systems. surveys. video recording of user sessions. Regression testing . Recovery testing . may require sophisticated testing techniques.similar to system testing.testing an application under heavy loads. Load testing . Page 55 of 60 . For example. or other catastrophic problems. heavy repetition of certain actions or inputs. covers all combined parts of a system. It can be difficult to determine how much re-testing is needed. Functional testing . if the new software is crashing systems every 5 minutes.term often used interchangeably with 'stress' and 'load' testing. especially near the end of the development cycle. or based on use by end-users/customers over some limited period of time.term often used interchangeably with 'load' and 'performance' testing. partial. etc. Stress testing . using network communications. and other techniques can be used. bogging down systems to a crawl. or interacting with other hardware. etc.Aditi Technologies Basic Concepts of Software Testing • • • • • • • • • • • • • • • etc. Usability testing .testing how well software performs in a particular hardware/software/operating system/network/etc.) System testing . This doesn't mean that the programmers shouldn't check that their code works before releasing it (which of course applies to any stage of testing. such as testing of a web site under a range of loads to determine at what point the system's response time degrades or fails. environment.typically used interchangeably with 'recovery testing' Security testing . Install/uninstall testing . the 'macro' end of the test scale. or systems if appropriate. Sanity testing or smoke testing .testing how well the system protects against unauthorized internal or external access.black-box type testing geared to functional requirements of an application. the software may not be in a 'sane' enough condition to warrant further testing in its current state. applications. Programmers and testers are usually not appropriate as usability testers.testing how well a system recovers from crashes. input of large numerical values.testing of full. Failover testing . Clearly this is subjective. Acceptance testing . involves testing of a complete application environment in a situation that mimics real-world use. and will depend on the targeted end-user or customer. willful damage. End-to-end testing . User interviews. this type of testing should be done by testers. or upgrade install/uninstall processes.testing for 'user-friendliness'.typically an initial testing effort to determine if a new software version is performing well enough to accept it for a major testing effort. Compatability testing . Automated testing tools can be especially useful for this type of testing. Ideally 'performance' testing (and any other 'type' of testing) is defined in requirements documentation or QA or Test Plans.final testing based on specifications of the end-user or customer. Performance testing . or corrupting databases. hardware failures.re-testing after fixes or modifications of the software or its environment. large complex queries to a database system.

Stick to initial requirements as much as possible . testing. not paper. Context-driven testing . detailed.require walkthroughs and inspections when appropriate.clear. plan for adequate time for testing and bug-fixing. intranet capabilities. etc.be prepared to defend against excessive changes and additions once development has begun. attainable. Comparison testing . and be prepared to explain consequences. minor design changes may still be made as a result of such testing. testers may be learning the software as they test it. cohesive. Mutation testing . bug fixing.start testing early on.comparing software weaknesses and strengths to competing products. continuous coordination with customers/end-users is necessary. • • • • Page 56 of 60 . If changes are necessary. and documentation. culture. by deliberately introducing various code changes ('bugs') and retesting with the original test data/cases to determine if the 'bugs' are detected.e-mail. In 'agile'-type environments. groupware.allow adequate time for planning.similar to exploratory testing. This will provide them a higher comfort level with their requirements decisions and minimize excessive changes later on. the testing approach for life-critical medical equipment software would be completely different than that for a low-cost computer game. networked bug-tracking tools and change management tools. changes. insure that information/documentation is available and upto-date .testing when development and testing are essentially completed and final bugs and problems need to be found before final release. personnel should be able to complete the project without burning out. complete. 11. User acceptance testing . design.testing driven by an understanding of the environment.often taken to mean a creative. and intended use of software.a method for determining if a set of test data or test cases is useful. but often taken to mean that the testers have significant understanding of the software before testing it. work closely with customers/end-users to manage expectations.testing of an application when development is nearing completion. Adequate testing .Aditi Technologies Basic Concepts of Software Testing • • • • • • • • Exploratory testing . Beta testing . not by programmers or testers. testable requirements that are agreed to by all players.. Proper implementation requires large computational resources. promote teamwork and cooperation. Alpha testing . use protoypes if possible to clarify customers' expectations. Realistic schedules . What are 5 common problems in the software development process? • Solid requirements . they should be adequately reflected in related schedule changes. For example.preferably electronic. If possible. re-testing. not by programmers or testers.determining if software is satisfactory to an enduser or customer. Use prototypes to help nail down requirements. informal software test that is not based on formal test plans or test cases. Communication . 'Early' testing ideally includes unit testing by developers and built-in testing and diagnostic capabilities. make extensive use of group communication tools . re-test after fixes or changes. Typically done by end-users or others. Typically done by end-users or others. Ad-hoc testing .

indentations. Coding style should be consistent throught a program (eg. quality is obviously a subjective term. What is software 'quality'? • Quality software is reasonably bug-free. use of brackets. use as many characters as necessary to be adequately descriptive (use of more than 20 characters is not out of line). and is maintainable. Function descriptions should be clearly spelled out in comments preceding a function's code. customer acceptance testers. magazine columnists. but everyone has different ideas about what's best. Management/accountants/testers/salespeople. can be used to check for problems and enforce standards. avoid abbreviations.Aditi Technologies Basic Concepts of Software Testing 12.) In adding comments. an application should include documentaion of the overall program function and flow (even a few paragraphs is better than • • • • • • • • • • • • Page 57 of 60 . meets requirements and/or expectations. less than 50 lines is preferable. less than 100 lines of code is good. It should be kept in mind that excessive use of standards and rules can stifle productivity and creativity. the development organization's. etc. these may or may not apply to a particular situation: Minimize or eliminate use of global variables.vertically and horizontally. There are also various theories and metrics. avoid abbreviations. use as many characters as necessary to be adequately descriptive (use of more than 20 characters is not out of line). or what is too many or too few rules. and is readable and maintainable. naming conventions. 'Peer reviews'. Each line of code should contain 70 characters max. It will depend on who the 'customer' is and their overall influence in the scheme of things. A wideangle view of the 'customers' of a software development project might include end-users. such as McCabe Complexity metrics. Organize code for readability. a common rule of thumb is that there should be at least as many lines of comments (including header blocks) as lines of code. What is 'good code'? • 'Good code' is code that works.use both upper and lower case. Some organizations have coding 'standards' that all developers are supposed to adhere to. However. customer management. future software maintenance engineers. customer contract officers. Use whitespace generously .use both upper and lower case. One code statement per line. Function and method sizes should be minimized. stockholders. be consistent in naming conventions. For C and C++ coding. No matter how small. err on the side of too many rather than too few comments. is bug free. be consistent in naming conventions. 'buddy checks' code analysis tools. Use descriptive function and method names . etc.the accounting department might define quality in terms of profits while an end-user might define quality as user-friendly and bug-free. etc. Each type of 'customer' will have their own slant on 'quality' . delivered on time and within budget. Use descriptive variable names . • 13. here are some typical ideas to consider in setting rules/standards.

developed by the SEI. and configuration management processes are in place. requirements management. to minimize complexity and increase maintainability. For C++.characterized by chaos. or if possible a separate flow chart and detailed program documentation. Level 1 . realistic planning. It is geared to large organizations such as large U. some common rules-of-thumb include: The program should act in a way that least surprises the user It should always be evident to the user what can be done next and how to exit The program shouldn't let the users do something stupid without warning them. What is 'good design'? • 'Design' could refer to many things. easily modifiable. understandable. Good functional design is indicated by an application whose functionality can be traced back to customer and end-user requirements. 14. Few if any processes in place. Level 3 .) For C++. now called the CMMI ('Capability Maturity Model Integration'). but often refers to 'functional design' or 'internal design'. Make extensive use of error handling procedures and status and error logging. Good internal design is indicated by software code whose overall structure is clear. make liberal use of exception handlers.software project tracking. and heroic efforts required by individuals to successfully complete projects.For programs that have a user interface. Level 2 . and works correctly when implemented. a Software Engineering Process • • • Page 58 of 60 . and maintainable. avoid too many levels of inheritance in class heirarchies (relative to the size and complexity of the application). successes may not be repeatable. keep class methods small.Aditi Technologies Basic Concepts of Software Testing • • • • nothing). It's a model of 5 levels of process 'maturity' that determine effectiveness in delivering quality software. initiated by the U. However. successful practices can be repeated. Minimize use of multiple inheritance.S.S. Organizations can receive CMMI ratings by undergoing assessments by qualified auditors. it's often a good idea to assume that the end user will have little computer knowledge and may not read a user manual or even the on-line help. Defense Department contractors. less than 50 lines of code per method is preferable. periodic panics. many of the QA processes involved are appropriate to any organization.standard software development and maintenance processes are integrated throughout an organization. • • • 15. For C++. and if reasonably applied can be helpful. Defense Department to help improve software development processes. is robust with sufficient error-handling and status logging capability. What is SEI? CMM? CMMI? ISO? IEEE? ANSI? Will it help? • • SEI = 'Software Engineering Institute' at Carnegie-Mellon University. and minimize use of operator overloading (note that the Java programming language eliminates multiple inheritance and operator overloading. CMM = 'Capability Maturity Model'.

and it applies to many kinds of production and manufacturing organizations.among other things. (c)Q9004-2000 . TickIT. 32% of organizations were U. ISO = 'International Organization for Standardization' . processes. production. It includes aspects such as initial concept. documentation planning. To be ISO 9001 certified. functional design. test planning. federal contractors or agencies. Also see http://www. servicing.) The median size of organizations was 100 software engineering/maintenance personnel.S. For those rated at Level 1.the focus is on continuous process improvement. 'IEEE Standard of Software Unit Testing (IEEE/ANSI Standard 1008). and other processes. installation. 27% were rated at Level 1. a third-party auditor assesses an organization. Other software development/IT management process assessment methods besides CMMI and ISO 9000 include SPICE. coding.Quality Management Systems: Guidelines for Performance Improvements. and quality is consistently high. and others. Level 5 .S. the most problematical key process area was in Software Quality Assurance.it indicates only that documented processes are followed.Quality Management Systems: Requirements. and other aspects. 17. (For ratings during the period 1992-96. phase-out. The impact of new processes and technologies can be predicted and effectively implemented when required. document preparation. and certification is typically good for about 3 years.Quality Management Systems: Fundamentals and Vocabulary. requirements analysis. Level 4 . 1018 organizations were assessed. maintenance. It covers documentation. the primary industrial standards body in the U. publishes some software-related standards in conjunction with the IEEE and ASQ (American Society for Quality). design. creates standards such as 'IEEE Standard for Software Test Documentation' (IEEE/ANSI Standard 829). the standards can be purchased via the ASQ web site at http://e-standards.asq. 23% at 2. 39% at 2. Perspective on CMM ratings: During 1997-2001. 6% at 4. and 0. 2% at 4. ITIL. Project performance is predictable. What is the 'software life cycle'? • The life cycle begins when an application is first conceived and ends when it is no longer in use.metrics are used to track productivity. development. Will automated testing tools make testing easier? Page 59 of 60 .iso.The ISO 9001:2000 standard (which replaces the previous standard of 1994) concerns quality systems that are assessed by outside auditors. not just software.S.org/ IEEE = 'Institute of Electrical and Electronics Engineers' .Aditi Technologies Basic Concepts of Software Testing • • • • • • • Group is is in place to oversee software processes. 16. 62% were at Level 1.. after which a complete reassessment is required. and products. In the U. testing. testing. (b)Q90002000 . and CobiT. The full set of standards consists of: (a)Q9001-2000 . and training programs are used to ensure understanding and compliance. and 5% at 5. Trillium. MOF. internal design. 'IEEE Standard for Software Quality Assurance Plans' (IEEE/ANSI Standard 730).4% at 5. updates. ANSI = 'American National Standards Institute'.ch/ for the latest information. Note that ISO certification does not necessarily indicate quality products . retesting. integration. Of those. 23% at 3. 13% at 3. Bootstrap.

bug reporting.for testing client/server and web applications under various load levels. the primary being: http://www. in an application GUI and have them 'recorded' and the results logged by a tool. or on-going long-term projects they can be valuable. etc. The 'recording' is typically in the form of text based on a scripting language that is interpretable by the testing tool. Test drivers can be in the form of automated test tools or custom-written testing software. and may be oriented to code statement coverage. Coverage analyzers . a tester could click through all combinations of menu choices. Other tools .to check that links are valid.vyomworld. and configuration management. Another common type of approach for automation of functional testing is 'data-driven' or 'keyword-driven' automated testing. client-side and server-side programs work. development. logs.these tools check which parts of the code have been exercised by a test. documentation. Other automated tools can include: Code analyzers . buttons. condition coverage. For example. If new buttons are added. and maintenance of automated tests/test cases. HTML code usage is correct. and comparing the logging results to check effects of the changes. Page 60 of 60 .since they are separate from the test drivers. This document is for internal purposes only. and for all types of platforms. documentation management. dialog box choices.Aditi Technologies Basic Concepts of Software Testing • • • • • • • • • • Possibly For small projects. adherence to standards. A common type of automated tool is the 'record/playback' type.such as via a spreadsheet . The same has then been reviewed and restructured for better reading and understanding. interpretation and analysis of results (screens.such as bounds-checkers and leak detectors. The test drivers 'read' the data/action information to perform specified tests. etc. Load/performance test tools . a web site's interactions are secure. Additionally. For larger projects. the application might then be retested by just 'playing back' the 'recorded' actions.monitor code complexity.com/testing/. Web test tools . data. or some underlying code in the application is changed. Memory analyzers . etc. etc. in which the test drivers are separated from the data and/or actions utilized in testing (an 'action' would be something like 'enter a value in a text box'). etc. The data and actions can be more easily maintained . Note that there are record/playback tools for text-based interfaces also. the time needed to learn and implement them may not be worth it.) can be a difficult task. The problem with such tools is that if there are continual changes to the system being tested. This approach can enable more efficient control. path coverage. the 'recordings' may have to be changed so much that it becomes very time-consuming to continuously update the scripts. All contents of this document have been collated from various resources.for test case management.

Sign up to vote on this title
UsefulNot useful